#cloud-init 2014-03-17
<harmw> hm, isn't there something like a codestandard on readthedocs..? I know of some openstack projects that have just that
<smoser> philips, it doesn't actually matter there.
<smoser> the '-' get converted to '_'.
<smoser> harmw, suck.
<smoser> i don't know what to do about that really. i like that there is a version string in the image, but I really wish that
<smoser> a.) i didn't screw that up
<smoser> b.) i could properly "promote" a release
<smoser> (ie, rename)
<smoser> build-release doens't do anything that shouldn't be doable in a ci build
<smoser> harmw, the '~pre1' is by design (versus '-pre1'). '~' in dpkg means "less than".
<harmw> uhm, whats wrong with committing a change in bin/build-release since that's where you keep track of version numbering :)
<harmw> and using the ~ annoyed me, I wasn't aware of a dpkg design thingy
<harmw> but whats wrong with using - instead of ~? 0.3.2-pre1 would still be the same as 0.3.2~pre1, right?
<smoser> harmw, 0.3.2-pre1 is > 0.3.2
<smoser> $ dpkg --compare-versions 0.3.2-pre1 gt 0.3.2 && echo yes, greater || echo no
<smoser> yes, greater
<harmw> ah like that
<smoser> the issue with committing a change in bin/build-release is that then you cant feed it a tag as input.
<smoser> the other issue is that then 0.3.2~pre4 can't just be "promoted" to 0.3.2 
<smoser> as it has that yucky string in it.
<smoser> i don tknow. its over thinking things i know. 
<harmw> can't we just go from 0.3.2rc1 to 0.3.2?
<smoser> what do you mean?
<harmw> nah, just thinking out loud
<smoser> harmw, i'm ok to punt on that, and just change that one thing and call it 0.3.2.
<smoser> assuming my kernel change didn't cause anything else
<harmw> it builds just fine, I haven't tested the new image though
<harmw> planning to do just that tonight
<smoser> crap.
<smoser>  /dev/null has bad perms in the tarball.
<smoser> and disk image
<harmw> and you only updated the kernel :p stupid kernel
<smoser> ?
<smoser> oh. no. it was a bug before.
<smoser> fix in hand.
<smoser> and it explains why running 'cirros-status' as root would not get you network debug info
<harmw> hm, so my images are broken as well?
<smoser> right
<harmw> (since I'm not using bin/bla)
<smoser> fixed and pushed.
<harmw> nice, now I only need to wait till midnight :p
<smoser> you weren't using bin/bla ?
<smoser> at all ?
<smoser> you probably were.
<harmw> haha, then I'm not going to get the fun out of this commit
<harmw> I was using the steps you describe in README
<smoser> you were. xmakedeves gets called from
<smoser>  * build disk images using bin/bundle
<smoser>    $ sudo ./bin/bundle -v output/$ARCH/rootfs.tar download/kernel-$ARCH.deb output/$ARCH/images
<harmw> bin/bundle, yes
<harmw> not bin/build-release
<smoser> right. build-release just does what is in the README
<smoser> its proably almost identical to yours
<harmw> indeed it is :)
<harmw> (though mine doesn't upload to LP )
<smoser> it doesn't upload
<smoser> where did you think it did that ?
<smoser> (actually, theres a bug that *i*haven't uploaded to launchpad :)
<harmw> hm well I thought to have skimmed over some bits that pushed something to LP
 * harmw should probably just stop thinking every know and then
<harmw> smoser: I see 0.3.2 is tagged as such?
<harmw> I'm auto-building in about 10 minutes
<smoser> no tag
<harmw> just ned to think of something to only build a new buildroot when it is really -really- neded
<smoser> i'mo building trunk right now.
<smoser> i'll put it on download and just not call it
<smoser> but can point you at it
<harmw> fair enough :)
<smoser> its built i386, amd64 and is 10 minutes into arm
<smoser> i suspect 20 minutes it'll finish.
<harmw> ah yes, the other archs - I'm still only building x86_64
<harmw> btw why is make br-source done witj i386 as arch?
<smoser> just because
<harmw> hmk, and its wrong to have it build with $ARCH ?
<harmw> i386/arm/x86_64 ?
<smoser> probably, yeah. they do have different packages though i think.
<smoser> i forget. 
<harmw> well having it build against x86_64 went great, haven't tried it on arm/i386 though
<smoser> oh. right. it iwll work.
<smoser> and it i think does it anyway.
<smoser> the explicit step is just to have the download done
<smoser> so that it hits cache 
<harmw> ah lol
<harmw> well its caching those downloads anyway
<harmw> any thoughts on this btw https://bugs.launchpad.net/cirros/+bug/1292973 ?
<harmw> harlowja: news from the bsd front?
<harlowja> harmw ah, let me see if i can get sean in, didn't chat last week
<harlowja> pinging now
<harmw> hm, sucks my server hasn't got enough firepower left to host a jenkins instance for c-i nightlies
<smoser> harmw, is that busted in current buildroot ?
<smoser> we should forward there if it is
<harmw> yea, it regards the 2012.05 buildroot
<harmw> or just upgrde buildroot
<harmw> which i've already done in some branch
<harlowja> harmw no response ping from sean, i'll let u know
<harlowja> will get him in here or bug him about whats happening/status
<smoser> http://paste.ubuntu.com/7109590/
<smoser> harmw, ^ . its odd. i built on an azure instance to a tmpfs, thkning it would go crazy faster.
<smoser> but it did not.
<smoser> harmw, http://download.cirros-cloud.net/pre-release/0.3.2/
<harmw> tmpfs, nice, I was thinking of going that way aswell
<harmw> my buildroot is 2G, thats gonna be a close call...
<smoser> well, 7G was within a cuple hundred meg.
<smoser> but that included everything.
<harmw> 2.1GB, without the downloads folder
<harmw> oh and its still compiling
<harmw> ah no, just finishing up
<harmw> took 33 minutes
<harmw> http://buildservice.weites.com/cirros/trunk/
<harmw> nice :>
<harmw> smoser: do you have reallife results from ARM-based cirros boots?
<smoser> no. other than lxc.
<harmw> cool, I'm expecting a cubietruck which has hardware virt stuff
<smoser> oh nice.
<smoser> canonical has some such hardware, and its possible people here have used it.
<smoser> but i'm not sure actually. i really did it for lxc. but i'd like to have it known functional on som eactual hardware
<smoser> and at some point would want arm64 stuff too
<harmw> it should arrive next week or so
<harmw> and then Ill load it up as nova-compute node
#cloud-init 2014-03-18
<smoser> cirros 0.3.2 released
<harmw> coool
<harmw> argh, smoser what's the url for the latest kernel image? (and how the hell do I get it myself)
<harmw> and isn't there a virtual flavor of 3.13 kernel?
<smoser> harmw, its -generic now.
<smoser> there is still a virtual, but it depends on the -generic
<harmw> the package is called -generic now? (and -virtual is gone, as a file)
<smoser> i gave you info on this man.
<smoser> http://paste.ubuntu.com/7069858/
<harmw> yup, but 0.3.2 still uses flav=virtual
<smoser> right. it uses precise kernel.
<harmw> but if it's -generic now, thats just fine
<smoser> so they just changd how the packaging is done.
<harmw> ok
<smoser> previously, there was a '-virtual' which actually confliced with '-generic'
<smoser> now there is a -virtual , which depends on -generic
<harmw> aha
<smoser> er... something like that
<harmw> yea well :)
<smoser> and then something also depends on -extra-drivers
<smoser> so most users will get the virtual kernel +_ the extra stuff
<harmw> damn, cirros-status should show the running kernel
<smoser> :)
<smoser> it should. you're right.
<smoser> i think we should branch 0.3
<smoser> and start 0.4
<harmw> ok, and wht will those 2 focus on?
<harmw> we could branch 0.3 and merge my buildroot+kernel update stuff with that?
<harmw> keep 0.2 the current supported version, but what should 0.4 aim for?
<smoser> 0.3.x is "stable"
<smoser> 0.4 is buildroot+kernel+mdev+ppc64+world-domination
<harmw> uhm right :| obviously
<harmw> somewhere my mind came up with 0.2 as beeing current :|
<harmw> I'd say make it happen smoser :)
<smoser> yeah. so i'd take your 'uname' info for 0.3.X, and i'd like to hvae the fix for more generic dhclient stuff there too. specifically for mtu.
<smoser> as it seems that can completely block usage 
<harmw> agreed
<smoser> altough i guess you could send config-drive user-data to run ifconfig
<smoser> and fix it
<harmw> smoser: http://paste.ubuntu.com/7115837/
<smoser> :)
<harmw> smoser: bin/build-release deletes the raw image, could you please not do that?
<smoser> why? whats in it?
<harmw> I'm uploading just that to openstack
<harmw> rather pointless i manually convert back to raw with qemu-img :)
<harmw> plus, could you include this to support Hyper-V vm's? (despite the inner workings not beeing in place just yet)
<harmw> qemu-img convert -O vpc cirros-0.3.2-pre1-x86_64-rev309.raw cirros.vhd
<harmw> smoser: you should update the Changelog as well
<harmw> since r307
<smoser> harmw, i admit that part of me doing qcow is to try to teach the world to stop being stupid.
<smoser> providing 'raw' does not help with that.
<smoser> i dont really care what layout a hypervisor uses to represent a disk as its being used.
<smoser> but qcow2 is an open def-facto standard that provides both sparseness and compression.
#cloud-init 2014-03-19
<jcmcken> hi all -- quick question, when anyone gets a minute
<jcmcken> is there a native way to specify that cloud-init configuration baked into an image can override configuration passed as user data?
<harmw> smoser: I'm feeding the raw image to glance, which hands it over to my Ceph backend
<harmw> come to think of it, isn't Glance able to convert images on the fly these days..?
<harmw> anyway, supplying a vhd to serve Hyper-V would be nice 
<smoser> harmw, glance doesn't (or shouldn't) do it.
<smoser> generally thats the philosophy with glance.
<smoser> its just a registry.
<smoser> hypervisors need to just deal with it.
<smoser> and i guess then that that means where nova provisions something "to volume" that it needs to do it too.
<smoser> id' happily cahnge to some other format if that format was as easily consumable and producable as qcow and was also sparse and compressed.
<smoser> especially if it streamed.  wget | convert-to-raw | dd of=block-devic
<smoser> that is ideal. i guess s/wget/curl/
<harmw> ah, well
<smoser> (so yeah, you touched on somewhat a religous topic for me)
<harmw> haha :)
<smoser> it is absolutely absurd to expect "image producers" (or appliance producers or whatever)
<smoser> to have to offer the same bits in 6 different formats.
<smoser> and you (as the user) to have to know which one to get.
<harmw> true, though it is conveniant
<smoser> the unfair benefit for me of qcow is that it "just works" with kvm.
<harmw> which was what I'm after :)
<smoser> so that is me being biased a bit.
<smoser> harlowja_away, please ping me when in.
<smoser> i really need to call 0.7.5 by like end of tomorrow
<smoser> so if there is stuff that you think is or should get in, please let me know.
<smoser> harmw, same for you above
<smoser> or anyone else.
<harmw> hm? ah, cloud-init?
<smoser> yeah :)
<smoser> we also talk about cloud-init here sometimes
<harmw> hehe
<harmw> well the fbsd stuff is in, and I've not worked on that for quite a while now
<harmw> mostly waiting for it to arrive in ports
<harmw> and to busy with cirros :p
<smoser> again, thanks for your help with that.
<harmw> sure np
<harmw> https://git.openstack.org/cgit/openstack/cinder/commit/?id=e066158b5235a3879fe90fa3bd813fc3363c01f5 that looks like Glance auto-converting any image type to raw volume
<harmw> or it converts the image to raw, making cow volumes possible
<harmw> (meh, Ill just have to read the source at some point time)
<harmw> smoser: just have Canonical donate me a nice HP Gen8 Microserver and we'll be another step closer to world domination :>
<smoser> harmw, NUCs are the new hotness.
<smoser> microservers crap
<harmw> I'm in for one of those as well :)
<smoser> nucs are really neat actually. they also have 'eamt' which gets you vnc to the system and remote power control an dsuch.
<smoser> but to our experiments no serial over lan
<harmw> so they include some kind of bmc?
<smoser> essentially.
<smoser> its consumer grade
<smoser> but pretty neat.
<harmw> sweet, didn't know that 
<smoser> it shares 1 NIC with the host
<harmw> ay
<smoser> its actually available on lots of system syou probably didn't know about
<smoser> ie, if you have a thinkpad of < 2 years old they all have it.
<harmw> I haven't bought hardware in ages, so... :)
<smoser> yeah, my thinkpad is 4 years old. it missed the eamt by 1 generation
<smoser> :)
<harmw> and all servers we have have here come with idrac/ilom or whatever decent thing they have
<harmw> *installed
<smoser> eamt is really kindofo hookey, but neat.  the vnc works by taking the hosts port 5900
<harmw> wtf
<smoser> ie, you cannot get to that host's IP address on 5900. 
<smoser> it shares the IP
<harmw> aargh
<harmw> thats just sux
<smoser> well, consumer
<smoser> :)
<harmw> true :)
<smoser> what do you want for $300 in a hocky puck sized server
<harmw> hehe
<harmw> 2nd nic and 2nd 2.5" hdd :)
<harmw> ideal compute nodes
<harmw> atleast in a homelab
<smoser> yeah. they're really neat.
<smoser> for the money i think they beat the pants off of hp microservers
<smoser> at least the onest that we had some of.
<harmw> ofc, but the new Gen8 microservers are way better compared to the first 3 generations
<harmw> but enough about that :) back to world domination
<harmw> when you're gonna branch cirros 0.4?
<harmw> btw smoser, how about a little tool in bin/ to change to root password when building from source?
<smoser> booooooooo
<smoser> this year is our year (cubs)
<harmw> perhaps, but here we are rather clueless on just wtf the cubs are and why they should win :>
<smoser> as above where i like to selfishly inflict my preferences of kvm on the world, i also like making people type "cubswin:)"
<smoser> someone disappointed that "cubswin:)" doesn't show anything about cirros in google
<harmw> hehe
<harmw> if I created such a tool, would you merge it?
<smoser> actually... i've dreamt of "cirros-tools" 
<smoser> as a package
<smoser> err... a separate project.
<smoser> fine to start in cirros
<harmw> hmk? 
<smoser> but that do things like:
<smoser>   cirros-util start lxc
<smoser>  cirros-util download 
<smoser> ...
<smoser>   cirros-util set-passwd cubs-lose
<smoser> it'd make testing things easier too.
<harmw> hm hm, interesting
<smoser> for lxc, though, its in lxc now. 'lxc create -t cirros' (although that there needs a feature to add user-data/meta-data)
<smoser> actually... i wonder how close 'backdoor-image' would come to working on cirros.
<smoser> i might have even tested it at one point 
<smoser> (it has code that changes passwords)
<smoser> https://code.launchpad.net/~smoser/+junk/backdoor-image
<smoser> so that might actually work as it is. 
<smoser> backdoor-iamge --user cirros cirros-image.img 
<kwadronaut> (typo)
<smoser> what is 'sl' ?
<harmw> isn't that the steamlocomotive?
<smoser> oh. funny.
<smoser> :)
<harmw> ubuntu ships with it iirc
<smoser> 26k though installed.
<smoser> wonder if there is a trim version :)
<harmw> :)
<smoser> did you ever see the "wheres chuck" meme ?
<smoser> http://www.jonobacon.org/2011/11/16/wheres-chuck/
<smoser> i had a ascii art version. that i was going to shove into cirros
<smoser> and have it show it if you did the konami code on the console
<smoser> i'd love to have some easter egg like that
<harmw> damn right :)
<harmw> I don't know this specific meme though
<harmw> but we have similar memes here :p
<smoser> https://launchpad.net/cirros
<smoser> harmw, you have almost as many points as i do!
<smoser> :)
<harmw> hehe cool
<smoser> i just branched 0.3. so now there is lp:~cirros-dev/cirros/0.3
<smoser> and lp:~cirros-dev/cirros/trunk
<harmw> ah yes, I see
<smoser> the second is the target of the ilnk 'lp:cirros'
<harmw> and the latest version got updated :)
<smoser> and your name is in the changelog
<smoser> fame and fortune will come your way soon
<harmw> o m g
<harmw> your telling ppl to just build the image themselves?
<smoser> no.
<smoser> i jsut don't knwo what to do.
<smoser> as i really dont want to amnually upload stuff to launchpad
<smoser> and download.cirros-cloud.net is actually akami CDN'd so it would be faster anyway
<smoser> *and* i can more easily get logs of those downloads.
<smoser> so i really don't want people looking at launchpad, basically.
<harmw> ok, well a link from the source page or something would be nice so ppl know where to look for prebuilt images
<smoser> this is true.
<smoser> :)
<harmw> http://bazaar.launchpad.net/~cirros-dev/cirros/trunk/view/head:/ChangeLog
<smoser> especially since cirros-cloud.net redirects you to launchpad
<harmw> most epic changelog ever
<harmw> having some decent html on cirros-cloud.net would be nice btw, instead of merely redirecting
<smoser> my wife suzanne has promised a cirros logo
<smoser> and i've wanted to have a shirt for ODS
<harmw> lol nice
<smoser> i agree on all of this.
<harmw> is this your 'whatever, I'm down with everything'-day?
<smoser> :)
<smoser> https://launchpad.net/cirros
<smoser> that should look better now.
<smoser> 2 links to download.cirros-cloud.net
<harmw> lol
<smoser> did you see this:
<smoser> https://bugs.launchpad.net/cirros/+bug/1273159
<smoser> obviously you did
<smoser> but is this right:
<smoser>  This can be worked around by adding this line to the eth0 stanza of /etc/network/interfaces
<smoser> ?
<smoser> i can't see how it is
<smoser> but maybe
<smoser> https://www.mail-archive.com/busybox@busybox.net/msg03985.html
<harmw> hm, if thats right it would require the addition of -O staticroutes to go in there as well
<smoser> yeah.
<harmw> can't verify that right now though
<smoser> well, source code makes it seem unlikel
<smoser> $ grep -r nodefaultopts .
<smoser> shows nothing
<smoser> in busybox git
<harmw> hm, I believe I've seen that -o somewhere though
<harmw> to not ask for default options
<harmw> plus, it's no_default_options
<smoser> wrt the retries.... on the metadata service
<smoser> there were issues on ec2 
<smoser> where the metadata servie woudn't come up right away.
<smoser> crazy stupid.
<harmw> hmk
<smoser> but when our ubuntu images first got there we were booting and hitting it before it was up
<smoser> and we'd just say "nothing there!"
<smoser> so ... poll and retry :-(
<harmw> hmk, so 20 retries made sense?
<smoser> i think they're probably much better now.
<smoser> well, 20 retries covers i think 60 seconds?
<smoser> somethin glike that. i think.
<smoser> ah. itmeout is 10 seconds . on the curl request
<smoser> so it could be up to 20*10 + 20*2 (the nap length)
<harmw> yea well, it's a pita having to wait 5 minutes because it takes to long to acquire an ip and trying to contact a non-existant ec2 api :p
<smoser> that is kind of silly. we can probably make it do max of 60 seconds.
<smoser> it is unreasonable in the first place for the MD to not be there.
<harmw> depends on the env, when I'm testing cirros I certainly don't do that in my openstack setup
<harmw> but just with qemu, or hyper-v 
<harmw> no ec2 api's on either of those
<smoser> right. that is reasonable.
<smoser> so i'd like to have a (non-root) way to boot the instance with metadata
<smoser> ie, like ubuntu  images do (http://ubuntu-smoser.blogspot.com/2013/02/using-ubuntu-cloud-images-without-cloud.html)
<harmw> ok, so when there is no ec2 available it firesup a local ds and uses that
<harmw> well, fires up.. it just reads in the user-data file which was already there
<harmw> perhaps the existence of such a file could (should?) make it skip contacting ec2 in the first place
<smoser> yeah, thats what it does harm.
<smoser> and you can do that with 0.3.2.
<smoser> if you seed those directorries, then it will read from them.
<harmw> cool
<harmw> then we realy need a proper seed-tool :)
<smoser> agreed.
<smoser> the thing that sucks is root
<smoser> oh. actually, kyou could still attach a config-drive disk.
<smoser> and that should work.
<smoser> and maybe i did it to allow a "nocloud"
<smoser> maybe
<smoser> yeah, actually it should
<smoser> harmw, you can just attach a nocloud disk. 
<harmw> to much hassle :>
<smoser> to avoid root
<smoser> its worth it. :)
<harmw> hehe ,well, I'm cool with using sudo to manipulate images... but yes, have a root-less solution is cool as well
<smoser> one other thing you could do in a raw image is have some place in the disk that a tool could just edit straight away
<smoser> bu that doesn't work with qcow or any other format
<harmw> hehe, raw +1 :>
<smoser> and you probably only get like 512 bytes there (this is how grub does things for its 'environment' file)
<smoser> well, it knows how to read extX filessystem, but then to just write into those places.
<harmw> might be just enough for nocloud
<smoser> wll, i dont know. limits user-data.
<smoser> one thing that is very useful, and tests a lot of it is just lxc
<harmw> true, but normal users probably have ec2 for that
<smoser> and with 14.04 you can use lxc without root
<smoser> but we'd need to make cirros work well there.
<harmw> fair enough
<smoser> its not too much hassle to attach a disk. really.
<smoser> and if you have:
<harmw> no ofc not :)
<smoser>  cirros boot kvm --user-data=foo
<smoser> then, you dont know of such hassle.
<harmw> so true
<harmw> but what if I'm testdriving in a hyper-v vm
<harmw> which is kind of a pain already
<smoser> well, then yo uhave to download the install shield installer 
<smoser> and click yes-i-accept
<smoser> and then get some DLLs from google some where
<smoser> and then reboot
<smoser> and *then* you can do it.
<harmw> ah yes, the one that requires .net4.5 and which isn't supported on a hyper-v server running Windows Server Core
<harmw> funny you
<harmw> cirros.msi :>
<smoser> windows is *so* well designed for automation
<harmw> spare me...
 * harmw manages several dozens of Windows systems
<smoser> someone pointed me at this a few days ago.
<smoser> haven't read it all
<smoser> http://www.thoughtworks.com/insights/blog/cloud-based-devops-possible-windows
<harmw> "However, it isnât all sunshine and lollipops. WinRM is actually pretty painful and fiddly to use and PowerShell is an ugly and procedural language."
<harmw> so true
<harlowja> smoser hey, just got in
<smoser> lazy west coast people
<smoser> utlemming, did you see  my comments ?
<harlowja> haha
<harlowja> smoser one that would be nice @ https://code.launchpad.net/~harlowja/cloud-init/local-before-net/+merge/211783
<harlowja> to fix the issue where cloud-init-local starts after networking
<smoser> ugh.
<smoser> well, that only fixes in sysvinit
<smoser> (not rhel6, right?)
<smoser> *and* that would then differ from other distros
<harlowja> sure, idk the ordering of other distros, since systemd and ordering isn't so easy to figure out
<smoser> right. i dont think its guanrteed
<harlowja> sure, so don't the other files also need to have this start before networking?
<harlowja> *this == cloud-init-local
<smoser> harlowja, ?
<smoser> i dont understand
<harlowja> guess the question is should the other files be adjusted also?
<smoser> harlowja, theres no way to do it in ubuntu.
<smoser> i dont think
<smoser> well, at least not more invasive than i'd lke to go at this point.
<harlowja> k, so i guess then maybe systemd needs to be adjusted, in the rhel5/6 that i'm y! using we are using those sysvinit scripts
<harlowja> brb
<harlowja> ok back
 * harlowja had to do apple security update crap
<harlowja> anyways smoser we can debate that later, i'm fine with the next release afaik
<harlowja> be nice to have sean here push the freesbsd stuff, but i can't seem to find him
<harlowja> oh, i found him online 
<harlowja> lol
<harlowja> but he's not responding, sad face
<harlowja> lol
<harlowja> harmw so sean is currently sucked back into the mail vortex :(
<harmw> haha lol
<harlowja> maybe more blackhole
<harlowja> harmw bugging him about maybe when he'll have some time free from mail
<harlowja> he says maybe end of this week :-/
<harmw> ok :)
#cloud-init 2014-03-20
<harmw> smoser: I think this one may be closed https://bugs.launchpad.net/cirros/+bug/1258964 (about downloadlinks)
<publio> anybody using this util with cloudstack?
<smoser> publio, http://shankerbalan.net/blog/cloud-init-supports-cloudstack-as-data-source/
<smoser> i've never used it, but apparently it should work.
<publio> I have the same issue as this bug https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1250398
<publio> that examples on fedora, maybe that one uses another version?
<smoser> publio, well, there is a reasonable chance 0.7.5 will fix it
<smoser> as it does not depend on boto
<smoser> it'd be good to knwo that.
<publio> so should I update to the dev branch to get it to work?
<publio> it looks like it's v0.7.5 on trusty
<smoser> publio, yeah. do you have trusty ?
<smoser> you can just give it a try and see.
<smoser> thats the easiest path fro sure
<publio> starting up right now :)
<publio> trusty doesn't seem to work either
<publio> it refuses connections to 22
<publio> can I set the user-data url directly?
<publio> nvm, was missing colon in my config..
#cloud-init 2014-03-21
<harmw> god how I hate DDoS attacks
<oobx> thanks for all of the work, fellas.
<oobx> I've been trying hard to find docs for cloud-init.  I see lots of examples.  But, no version-by-version syntax. Is that available?
<oobx> except 7.5 http://cloudinit.readthedocs.org/en/latest/
<smoser> publio, did it work ?
<smoser> oobx, config examples are in the tarballs for the version, and if you've installed a deb, also included.
<smoser> make sense ? 
<smoser> i would like version'd docs. but its jus tlow priority.
<smoser> or you could check out bzr and look at docs/ at a particular tag  level
<oobx> thanks, smoser!  I'll check the examples.
<smoser> it actually seems to me that that is the most direct way to deliver documentation.
<oobx> probably.  But, I think lots of people just google first.  The examples might be indexed by google.  But, I was looking for exact syntax, like apache docs or something.  I'm trying to use prebuilt RPMs and such for CENTOS.  So, I'm may not be using the latest version of cloud-init.  I don't want to bang my head against the wall trying directives that will never work in my version.
<oobx> i'm trying to wedge cloud-init into vcloud director guests.  So far, it's working by attaching an ISO.  I think it's using the OVF datasource.
<oobx> i'm new to cloud-init, especially on vcloud/vmware.  I do have a spot instance on Amazon.
<oobx> but, I have not gotten too far into the nitty gritty there.
<oobx> I'm actually selling vcloud services.  So, I need to have a good understanding before I recommend it to anyone.
<oobx> off to a meeting...
<smoser> oobx, i agree. i google too.
<smoser> oobx, utlemming might have more info that could help you.
<smoser> i'm not sure though.
<smoser> oobx, another amazon instance to look at will cost you a whopping $0.017 per hour!
<smoser> or you can just poke around with "nocloud" 
<smoser> http://ubuntu-smoser.blogspot.com/2013/02/using-ubuntu-cloud-images-without-cloud.htmlhttp://ubuntu-smoser.blogspot.com/2013/02/using-ubuntu-cloud-images-without-cloud.html
<oobx> thanks, smoser. I'm gonna try nocloud.  I didn't at first, because vcloud director doesn't have a gui for attaching and removing disks.  But, the fog ruby gem seems to allow upload and attach/detach of disks.  Disks would actually work better, because leaving a CD inserted in a vmware instance limits vmotioning capabilities and such.
<smoser> oobx, oh. i was just saying you can play with it on a local system.
<smoser> you can even (slowly) play with it in qemu or nested kvm.
<oobx> I really want to play with it on vcloud, where I'll have to figure out the whole flow.  What's teh best datasource for vcloud? altcloud?  I'd love to mimick ec2's http calls.  But, I can't think of a way to do that ... yet :)
<harmw> smoser: how'd you got to 181 points so sudden? accumulated from other LP projects?
<harmw> I was so close..
<harmw> :p
<smoser> in cirros ?
<smoser> oh. i know.
<smoser> i went through and did the bugs state and such.
<smoser> BOOO YAH!
<harmw> lame
<harmw> realy
<harlowja_still_a> smoser, hey out sick with not so fun case of poison oak, if u were wondering where i went
<harlowja_still_a> not fun stuff....
<harlowja_still_a> *got it outside last weekend when rock climbing
<harlowja_still_a> :(
<harlowja_still_a> damn mother nature, lol
<smoser> silly californians.
<harlowja_still_a> def
<smoser> we dont have to wrry about poison oak here.
<smoser> still snow most places
<harmw> poison oak..?
<harmw> snow?
<harlowja_still_a> no snow in bay area (ever)
<harlowja_still_a> smoser, probably poison ivy though where u are
<harlowja_still_a> *not right now of course
<harlowja_still_a> same evil chemical in poison ivy
<smoser> we have both poison ivy and poison oak
<smoser> and i've had both.
<smoser> but you sure don't have to worry about it riht now
<smoser> :)
<harlowja_still_a> ya, i'm just having the excitement of watching this run its course :-/
<harlowja_still_a> *for the Nth+1 time
<harlowja_still_a> hot water is currently my bestest friend, lol
#cloud-init 2014-03-23
<penguinRaider> hi need around here I need to add a new datasource to cloud-init. I read about the datasource object interface on the readthedocs. But how and where should I add a new datasource for cloud-init to recognize it ?
#cloud-init 2015-03-16
<nk121> hi, how can i pull down apt package like linux-image-extra-`uname -r` in cloud-init?  specifically, i need uname -r to be resolved at run time
<Odd_Bloke> nk121: You could put it in a script.
<Odd_Bloke> nk121: See https://cloudinit.readthedocs.org/en/latest/topics/examples.html#run-commands-on-first-boot
<smoser> nk121, why not just use linux-generic ?
<smoser> nk121, 
<smoser> #cloud-config
<smoser> packages: [linux-generic]
<smoser> apt_reboot_if_required: true
<smoser> i think that will do what you want it to do.
<smoser> boots, installs linux-generic, if that is a newer version that in the image (which hopefully its not), then it will reboot into that new kernel.
<Odd_Bloke> smoser: Would that pull in -extra?
<smoser> generic depends on -extra
<smoser> basically generic = virtual + extra
<smoser> Odd_Bloke, so the above is indeed what you want to use if you need "the full ubuntu kernel" in your image.
<Odd_Bloke> Oh, right, I didn't look at the fully resolved dependencies.
<Odd_Bloke> smoser: Ping re: https://code.launchpad.net/~daniel-thewatkins/cloud-init/fix-smartos/+merge/252874
<smoser> Odd_Bloke, i know. i really wanted to get that friday, but lost that day (and this morming so far) on maas images. 
<smoser> but always feel free to bother me with such reminders
<Odd_Bloke> smoser: Consider yourself bothered. ;)
<smoser> yeah. did you have another one also ?
<Odd_Bloke> smoser: I have a branch to move to v2 metadata for SmartOS, but I haven't been able to test that properly yet (because of the above issues).
<nk121> Odd_Bloke, smoser: thanks for the suggestions. i need snd-aloop loopback device from alsa. before i saw your replies i went with "linux-image-extra-virtual" but that seems to install a lot more than i need
<smoser> nk121, well, sure. yo'ure going to get a lot more thna you need.
<smoser> there are basically only 2 kernel module sets. (virtual and "every thing else")
<nk121> is the alsa loop driver part of "every thing else"?
<smoser> probably.
<smoser> at least based on your description
<nk121> also, as an aside, when i originally tried linux-image-extra-`uname -r`   and ran it, i found that once it hit that line, it didn't install any packages after it, but then proceeded to run other parts of cloudinit that relied on packages that were not installed (with more failures).     is there a way for it to stop at failure and not continue configuring a broken system?
<smoser> i think you're asking if you can jsut stop if the package install module fails ?
<smoser> ie:
<smoser> packages: [this-package-does-not-exist]
<smoser> and then that not go on ?
<nk121> well i mean, i guess in general, what does it mean for system to fail in some part of the configuration -- i feel the option should either be, stop and escalate, or continue -- but should be a choice
<nk121> well package might not exist, or maybe it can't download it / or some other unatticipate problem
<nk121> unatticipated
<smoser> nk121, current versions write a file to /run/ that reports errors in json format.
<smoser> much of the reason it just goes on, is that if it did not, in many cases, you'd never have access to the system in order to see/diagnose such failure.
<smoser> i do see your point though
<nk121> ok i guess i should check for that file at the end then.     with regards to just packags, it seems it completely stops doing the rest of the packages if it has a failure.   thats rather inconsistent
<smoser> nk121, well, that is apt's behavior
<smoser> nk121, http://paste.ubuntu.com/10611394/
<nk121> smoser: ah, i guess i assumed it was doing a for loop and installing each one by one
<smoser> that'd be a lot slower.
<nk121> smoser: thanks for your help and the discussion, much appreciated.
<suro-patz> smoser: ping
<suro-patz> I worked towards addressing https://bugs.launchpad.net/cloud-init/+bug/1225922, with https://code.launchpad.net/~suro-patz/cloud-init/vm-clone-ip-reusage-issue/+merge/252961, as per the discussion in the bug and in paste.openstack.org/show/135506/
<suro-patz> I would appreciate if you can review the fix
<harlowja> or let suro-patz know that its just not the right way :-P
#cloud-init 2015-03-17
<nk121> Hi #cloud-init. Question about upstart jobs -- is it safe to assume that after all of cloud init stuff is done, it starts the service?
<Odd_Bloke> nk121: "The service"?
<nk121> i mean the upstart job
<Odd_Bloke> nk121: Which job are you referring to?
<nk121> an upstart job included in the cloud-init configuration
<nk121> does it get started at the end of everything being run?
<nk121> it seems to, but i'm not sure when in the lifecycle its started (like is it the last thing that happens when cloud init is run?)
<nk121> (so should runcmd and user data scripts assume that the upstart job is not running)
<nk121> Odd_Bloke: ^
<Odd_Bloke> nk121: I'm not sure how the ordering is determined, I'm afraid.
<nk121> Odd_Bloke: That's ok, of course. :)
<Odd_Bloke> smoser might be able to help out, but he's on EST so won't be around for a few hours.
<Odd_Bloke> nk121: Examining the code, it looks like there isn't a defined ordering; so things will be executed in the order Python's dict implementation spits them out.
<Odd_Bloke> But it just takes me missing one line for that to be completely wrong. :)
<nk121> Yeah
<nk121> hmm, so then its probably not safe to have a dependency on write_files and runcmd (i have a script that is written out in write_files and then executed as a specific user in runcmd)
<nk121> so far, it seems to run my runcmd block after the file is created
<Odd_Bloke> nk121: Aha, OK, I think the ordering is defined by cloud.cfg.
<nk121> is that a section?
<nk121> or you mean, a file in the source
<Odd_Bloke> nk121: That'll (probably) live in /etc/cloud; it should be installed by your cloud-init package.
<nk121> hmm
<nk121> cloud_config_modules:
<nk121> # Emit the cloud config ready event
<nk121> # this can be used by upstart jobs for 'start on cloud-config'.
<nk121>  - emit_upstart
<Odd_Bloke> nk121: So I think the upstart job is written out before any of these things run.
<nk121> okay i'll have to continue this in the morning, its way past my bed time (i'm in PST). thanks for your help tonight
<Odd_Bloke> nk121: Sleep well! :)
<Odd_Bloke> smoser: Daily reminder for https://code.launchpad.net/~daniel-thewatkins/cloud-init/fix-smartos/+merge/252874 :)
<smoser> Odd_Bloke, is that python2 safe ?
<smoser> does ser.readline() return something that can be .decode()ed in python2 ?
<smoser> nk121, the job will be written pretty early in boot.
<smoser> so unless you're upstart job is set to run really early, it will be started by upstart.
<smoser> but cloud-init lets upstart take care of that.
<smoser> the 'emit upstart' isn't waht does it.
<smoser> merged
<Odd_Bloke> smoser: You can decode() anything in Python 2.
<Odd_Bloke> >>> "".decode().decode().decode()
<Odd_Bloke> u''
<smoser> yeah, just tested that locally.
<suro-patz> logger url
<suro-patz> logger: url
<harlowja> suro-patz there is no logger in most public channels; thats typically a y! concept ;)
<suro-patz> is there a archive for cloud-init, I do not see that in pipermail
<Odd_Bloke> suro-patz: It is logged to irclogs.ubuntu.com (e.g. http://irclogs.ubuntu.com/2015/03/17/%23cloud-init.html)
<Odd_Bloke> I _think_ that is updated hourly.
<suro-patz> cool! thanks Odd_Bloke
<nk121> smoser: thanks. Could you clarify/confirm: 1) Is it a safe assumption to be run a file written in a write_files section in a runcmd or user data script? 2) If an upstart job is added via cloud-init, can you assume that its been installed into /etc/init and started before code in runcmd/user_data_script is run?
<smoser> nk121, http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/config/cloud.cfg
<smoser> write-files writes the files. 
<smoser> scripts-user in cloud_final_modules is what runs your 'runcmd' provided stuff.
<smoser> so yes. 1 is safe and guaranteed.
<nk121> ok, so for 1) write_files is in init, and runcmd is in config and user-data is in final, so that seems correct
<smoser> 2 is guaranteed except for if init system is not upstart (which... in vivid it is not and wont be in 16.04)
<nk121> oh not, not run-cmd itself?
<smoser> run-cmd itself actually just writes the command to a script that will later get run. interestingly :)
<nk121> sorry i mean run-cmd doesn't run run-cmd
<nk121> ah
<nk121> ok, so you are saying though i should not assume in the long run that my upstart job will be running when my runcmd/user-data script is run as that will change in a future ubuntu
<nk121> within a stage, is that the order htey are run in?
<nk121> err, you are implying ubuntu is moving away from upstart?
<smoser> yes. within a stage they're run in that order.
<nk121> oh, systemd replacement
<smoser> yes, 15.04 will use systemd by default.
<smoser> and thus 16.04.
<nk121> okay, got it. thank you.
<smoser> i'm not planningon making cloud-init in any version magically convert upstart to systemd jobs
<smoser> but there will likelyi be in a future cloud-init a way to write systemd jobs like you can write the upstart jobs now.
<suro-patz> smoser: there?
<smoser> suro-patz, here for 3 minutes maybe
<harlowja_> hmmm, i think he just wanted your feedback on https://code.launchpad.net/~suro-patz/cloud-init/vm-clone-ip-reusage-issue
<harlowja_> if u get some time; u seem to have thoughts on that idea; so i'm gonna let u decide if thats ok to let in, lol
<suro-patz> thanks harlowja_ ! 
<harlowja_> np
<harlowja_> although i think i missed him to, lol
<harlowja_> was 10 minutes to late
<smoser> ... 
<smoser> oh man.
<smoser> that is just a mess.
<smoser> (not your code, but the whole non-deterministic network state behavior of cloud-init)
<smoser> magic fixes in 2.0
<smoser> i have to read/think more than i hve time for now. will try to look tomorrow
<suro-patz> sure smoser
<smoser> later. 
<smoser> and happy st. patricks day :)
#cloud-init 2015-03-18
<HanSolo> Hello guys
<HanSolo> what is the best way to add my elastic ip in /etc/hosts with cloud-init please
<HanSolo> ?
<smoser> HanSolo, theres nothing that can do thta for you. :-(
<Odd_Bloke> smoser: Thanks for the merge yesterday. :)
<Odd_Bloke> smoser: Here's another little one: https://code.launchpad.net/~daniel-thewatkins/cloud-init/lp-1311463/+merge/253362 :)
<smoser> Odd_Bloke, ok. thank yo ufor writing tests. 
<Odd_Bloke> :)
<HanSolo> smoser: I have to use this (ugly) kind of workaround then ? http://ternarylabs.com/2010/09/15/automatically-configure-hostname-for-new-ec2-instances/
<HanSolo> I know this is beyond scope of cloud-init sorry for asking :/
<smoser> HanSolo, its not really beyond the scope.
<smoser> i'd like to have a solution for you
<HanSolo> thank you smoser, this is still a great product
<harmw> harlowja_: ever seen this with kvm? https://bugzilla.redhat.com/show_bug.cgi?id=1202990
<harlowja_> harmw don't think i've seen that
<harlowja_> but we don't run on fedora, so thats maybe why
<harmw> ah, well, CentOS 7 showed the same
<harlowja_> ah
<harlowja_> haven't seen that
<harmw> are you aware of certain ovs specifics one should be setting?
<harlowja_> not off the top of my head
<harmw> i'm using an mtu of 1400 here btw, for what its worth
<harlowja_> harmw if u jump on #openstack-operators u may find some folks there that have some idea
<harmw> oh sure, thanks
<harlowja_> np
<harlowja_> someone there might of seen this
<harmw> I'm gonna try if I can get some results from -not- running through nova
<harlowja_> k
<harmw> eg, dropping a lot of the extra kvm parameters
<harlowja_> ya, maybe u'll figure it out
<harmw> probably not though :P
<harlowja_> lol
<stupidnic> Could somebody help me troubleshoot a long period on cloud-init modules:config
<stupidnic> I run cloud-init on a centos6 image and it get's an IP address quickly but then it just sits there for a long duration before completing
<stupidnic> http://pastebin.com/XkwiUxsN
<stupidnic> that's what the logs show
<smoser> stupidnic, i suspect /var/log/cloud-init.log has some WARN in it.
<smoser> can you paste that ?
<stupidnic> smoser: it does not
<smoser> hm..
<stupidnic> let me jump into that server again
<smoser> its poling something.
<stupidnic> that was my thought too
<stupidnic> seemed like a timeout or something
<stupidnic> cloud-init.log is zero length
<stupidnic> I can show the entire cloud-init-output.log if you like
<stupidnic> http://pastebin.com/gimfKReE
<stupidnic> nothing much really
<stupidnic> there is just that huge 500+ second delay between the ci-info output and the modules:config run
<stupidnic> I can clone the base image and turn on more debugging, I just don't know how to do that actually
<smoser> stupidnic, the log is getting swallowed somewhere.
<smoser> oh.. syslog.
<smoser> try looking in /var/log/syslog or wherever syslog messages would go.
<smoser> grep -r "cloud" /var/log 
<stupidnic> coming up empty
<stupidnic> well not empty empty
<stupidnic> but nothing new
<smoser> odd. well, anything important here ?
<smoser> ie, do you care particularly about the instance
<smoser> 2 things to try
<smoser> a.) edit /etc/cloud/cloud.cfg.d/05_logging*
<stupidnic> It's a customer instance
<smoser>  then yes, yes you do :)
<smoser> reproduce somewhere else ?
<stupidnic> but I can dump the base image out of glance
<stupidnic> and then fiddle with that
<stupidnic> and the put it back into glance and use it as a base for testing instances
<smoser> oh. can you see the console log of the instance ?
<smoser> nova console-log ...
<stupidnic> Yeah it's more of the same sadly
<smoser> that may have WARN on it too
<stupidnic> the original pastebin was taking from that
<smoser> well, debug messages woudl be useful. 
<smoser> so that 05_logging.cfg
<smoser> just comment otu:
<smoser>   - [ *log_base, *log_syslog ]
<smoser> that line.
<smoser> and then it should log directly to the file
<stupidnic> k
<stupidnic> let me dump the image and get that taken care of
<stupidnic> smoser: okay... what should I be commenting out here? just the stuff under log_cfgs: section?
<smoser> just that line.
<stupidnic> alright, done... let me load the image back into glance and spin up an instance
<stupidnic> well of course... it didn't do it this time
<stupidnic> cloud-init took 22 seconds from start to finish
<smoser> do you see log too ?
<stupidnic> yes
<stupidnic> the cloud-init.log is over 140KB in size
<stupidnic> About the only difference this time was the size of the instance and the tenant
<stupidnic> client instance was 1TB this was only 10GB
<stupidnic> possible that the size difference was the delay?
<smoser> oh. well, was it the first boot ?
<stupidnic> Yes
<smoser> yeah. cloud-init resizes the disk
<smoser> and on an older kernel, that is very slow.
<stupidnic> I thought that occurred in the init ramdisk
<smoser> on older kernels, it will rewrite the partition table in the ramdisk
<stupidnic> at least in Centos 6
<stupidnic> right
<smoser> (on newer ones it does that in cloud-init too)
<stupidnic> yeah this is a centos 6 image
<stupidnic> so 2.6.32 kernel
<smoser>  < 3.2 kernel is stupid slow resizing.
<smoser> i tihnk its actually 3.10 that is , not kidding, 1000x faster.
<stupidnic> hmmm... might be worth going with the centos lt kernels then
<stupidnic> Something to consider. I don't want to fragment my images too much though
<smoser> is not really taht simple (its hiding some of the operations i think) but, a 'resize2fs from 2G -> 1TB' will be probalby 1000x faster on nwere kernel.
<stupidnic> last thing I need is to support five variants of each OS
<smoser> yeah. i'd probalby live with it.
<stupidnic> Let me try a larger instance and see what it does
<stupidnic> I have SAN to burn ;)
<smoser> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1179610
#cloud-init 2015-03-19
<stupidnic> That's about right
<smoser> the bug report is a bit wrong. in the description.
<smoser> i thought it was all user-space
<smoser> but really the user space says "hey kernel, do magic!"
<stupidnic> and of course... now cinder decides it is going to act up
<stupidnic> smoser: yeah, it's clearly dependent on the size of the backing volume upped the size to 1TB and it's doing the same thing. At least this image has the logging enabled so I can see exactly what is taking so long
<smoser> yeah, and the log should even explicitly say "took X seconds" for that somewhere.
<smoser> one interesting thing... i made that backgroundable.
<stupidnic> Oh?
<smoser> but it turns out it doesn't matter. its so IO heavy, that nothing else is happing. 
<stupidnic> hahah
<stupidnic> That's the way it goes I guess
<smoser> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/examples/cloud-config.txt
<smoser> you can give it a try:
<stupidnic> the SAN might have the IOPS to handle that though
<smoser> resize_rootfs: noblock
<smoser> you can put that in user-data or in a /etc/cloud/cloud.cfg.d/foo.cfg file
<stupidnic> seems like cloud-config would be the more lower impact way to test it
<stupidnic> I have to run, but I should be back in about an hour to test it out and see how it does.
<stupidnic> smoser: thanks for the help and pointers
<stupidnic> I'll let you know how I make out with noblock
<stupidnic> smoser: 2015-03-18 20:16:33,573 - util.py[DEBUG]: Resizing took 482.659 seconds
<SpamapS> if I want to use configdrive on OpenStack, does cloud-init automatically try to mount and read it?
<SpamapS> smoser: ^ ?
<SpamapS> harlowja_away: ^
<harlowja_> SpamapS yes
<SpamapS> harlowja_: ok, mordred is suggesting that is not true.
<SpamapS> harlowja_: our Ubuntu trusty nodes don't seem to do it anyway.
<harlowja_> so then it starts to depend on what datasources are enabled
<harlowja_> $ cat /etc/cloud/cloud.cfg
<harlowja_> ie datasource_list:
<harlowja_>  - ConfigDrive
<harlowja_>  - Ec2
<harlowja_>  - None
<harlowja_> if configdrive not there, then its not gonna do anything :-P
<harlowja_> SpamapS go give a monty a hug, or something, lol, u both at HP, go find him or something (not sure which office he is at, lol)
<SpamapS> HAHA
<SpamapS> He's at his home in NY
<SpamapS> I'm at my home in LA
<SpamapS> so.. yeah.
<harlowja_> start walking then
<harlowja_> lol
<harlowja_> chop chop
<suro-patz> smoser: Would appreciate, if you can get some time and provide some feedback on https://code.launchpad.net/~suro-patz/cloud-init/vm-clone-ip-reusage-issue/+merge/252961
<smoser> SpamapS, i'm 99% certain it does. 
<smoser> just the other day suggsted to someone to use it and it solved their problem  
<smoser> if it doesnt, plase open bug
<SpamapS> smoser: thanks, we figured it out. What mordred was on about was that it doesn't _leave_ it mounted.
<SpamapS> smoser: also OpenStack infra can't use it because it doesn't work in precise.
 * harlowja_ my thought on that is that its 0.6.3 and openstack isn't writing the old v1 config-drive format anymore?
#cloud-init 2015-03-20
<nk121> write-files occasionally fails for me, the only i can find is
<nk121> 2015-03-20 07:35:14,540 - util.py[WARNING]: Running write-files (<module 'cloudinit.config.cc_write_files' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_write_files.pyc'>) failed
<nk121> any ideas on how to debug?
<ndonegan> Hi, currently have cloud-init setup so it will ONLY try EC2. Unfortunately as we have found out, EC2 when using a certain provider's net product is not as reliable as we'd like, so we've been seeing the "likely bad things to come!" message a bit too often :(
<ndonegan> At the moment, I've made a copy of the FallBack/None data source which can take in a "user_data" from the config and do things like stop the service that's supposed to start from starting etc.
<ndonegan> Is there any neater way of doing this?
<ndonegan> And now I spot that Fallback can use data from the config!
<zilberstein> good day #cloud-init
<zilberstein> I have a problem setting timezone on virtual machine with cloud-init
<zilberstein> I pass `timezone` key in meta-data
<zilberstein> but cloud-init reports in log the following
<zilberstein> Mar 20 11:53:57 localhost [CLOUDINIT] cc_timezone.py[DEBUG]: Skipping module named timezone, no 'timezone' specified
<zilberstein> it is specified in /etc/cloud/cloud.cfg
<zilberstein> - timezone
<smoser> zilberstein, the '-' ?
<smoser> oh. i see.
<zilberstein> smoser, what - stands for ?
<smoser> what metadata did you send ?
<smoser> it really should work:
<smoser> #cloud-config
<smoser> timezone: foo
<nk121> hi smoser, any ideas on how to debug write-files? sometimes my files write, and sometimes they don't. the yaml appears valid (it loads in python yaml parser at least). the only error message i could find: 	2015-03-20 07:35:14,540 - util.py[WARNING]: Running write-files (<module 'cloudinit.config.cc_write_files' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_write_files.pyc'>) failed
<smoser> nk121, probably /var/log/cloud-init.log has something
<smoser> a stack trace probably
<smoser> if not, you can alway srun that module by hand:
<smoser>  sudo cloud-init --debug single --frequency=always --name=write_files
<stresler> I want to start work on a new datasource that I'll eventually want to submit to here http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/files/1084/cloudinit/sources 
<stresler> My question is, what is the best way to test this? Can I drop it into the directory layout somewhere while I'm working on it, or do I need to roll a build to test it ? 
<nk121> smoser: cool, i think i found the isssue. i'm creating an acount via system_info.default_user.name (basically to change it from "ubuntu"), and the file i'm writing has owner NEWUSER:NEWUSER, which when I log into the system that user/group exists. the error from write_files is OSError: Unknown user or group: 'getpwnam(): name not found:   so perhaps that user isn't created yet?
<harlowja_> smoser seems like we are good to go
<harlowja_> [self._process.terminate,
<harlowja_>                           self._process.terminate,
<harlowja_>                           self._process.kill]
<harlowja_> oops
<harlowja_> https://openstacksummitmay2015vancouver.sched.org/event/b52d0c99f65d7713c61524b5a380812b?iframe=no&w=&sidebar=yes&bg=no#.VQyQD2avL-9
<harlowja_> that one ^
<harlowja_> lol
#cloud-init 2016-03-21
<smoser> magicalChicken, i dont expect you to make much sense out of it
<smoser> but http://paste.ubuntu.com/15443576/
<smoser> that was my quick hack attempt to do some 'fallback'.
<smoser> laregely to just test some of the other assertiions i had.
<smoser> magicalChicken, so one take awy of this is that the fallback network info should not try to rename devices that are 'virtual'
<magicalChicken> smoser: I think I understand it mostly. I get not trying to mess with the virtual devs
<rharper> smoser: in the patch you shared, there was a reference to a /tmp/my.patch (looking on our shared host) it modifies the systemd networking.service Wants target; is that still needed ?
<smoser> no. that is resolved with cloud-init-local saying Wants=networking-pre.target
<rharper> ok
<smoser> which is in my branch
<rharper> net1 ?
<rharper> I should pull and rebuild the deb, right ?
<smoser> yeah.
<smoser> let me know how it fails :)
<rharper> sure
<smoser> i dont think i've broken it, but i might ave
<rharper> heh
<smoser> i think it should work
<smoser> i probably had it working with 1191
<smoser> but the working tree on taht system has this http://paste.ubuntu.com/15466159/
<smoser> so if it doesnt work, maybe read that and see if it helps
<rharper> ok
<rharper> \o/
<rharper> 1191 works
<rharper> so happy to not see the 120 second timeout =P
<smoser> rharper, but current does not ?
<rharper> smoser: I just did a bzr pull, and I got 1191
<rharper> in net1
<rharper> so that's current AFAICT
<rharper> maybe you have local changes not pushed to trunk.net1 ?
<smoser> hm.. i have 1199 here.
<magicalChicken> I'm seeing 1199 as well
<smoser> https://code.launchpad.net/~smoser/cloud-init/trunk.net1/
<magicalChicken> It looks like it works to me, I've been working from it
<rharper> oh
<rharper> hrm
<rharper> ah, yeah
<rharper> 1199
 * rharper can't read
<rharper> 1199 works fine
<smoser> woot.
<smoser> can you push your git that works ?
<rharper> just getting that done now
<rharper> smoser: ok, pushed
 * rharper will now poke at using scripts.d instead of cloud-init injected scripts so we can test cloud-init disabled 
<rharper> magicalChicken: what are you blocked on w.r.t cloud-init-test ?
<magicalChicken> rharper: Nothing atm, since it's all passing nwo
<magicalChicken> rharper: I had been working on getting a test of cmdline ip= going but had been blocked for a bit, but I should be able to finish that now
<rharper> cool
<smoser> rharper, so...
<rharper> y
<smoser> openstack networking information -> network-yaml
<smoser> that is a needed component.
<smoser> should be fairly stand alone conversion
<smoser> could you look at that ?
<rharper> sure, do you have a pointer to their metadata yaml ?
<smoser> rharper,  https://review.openstack.org/#/c/85673/9/specs/juno/metadata-service-network-info.rst
<smoser> is some.
<smoser> i'm not sure if there is better or more complete somewhere else
<rharper> cool
<smoser> but that shoudl give you somethign to google for
<rharper> image can have custom startup scripts to get networking config from
<rharper> Config Driv
<rharper> arbitrary blobs ... we don't support that in our network config as of now;   what do? =)
<smoser> rharper, ?
<smoser> config driv ?
<rharper> smoser: in the RFC
<rharper> the goal was to support neutron network config, and allow arbitrary scripts run to "configure" things
<smoser> cloud-init does read the config drive.
<smoser> we actually do somewhat read this information
<rharper> I'm just saying that if the config-drive includes scripts that muck with the network; it's possible things won't go quite right
<smoser> no it doesnt
<smoser> i dont think
<smoser> i'm almost certain its sane
<rharper> the last use-case described is what concerns me
<rharper> * image can have custom startup scripts to get networking config from Config Drive
<smoser> i think image here means OS image
<smoser> ie, cloud-init can have custom startup scripts...
<rharper> yeah, OK
<rharper> to obtain the data
<rharper> sorta like the per-Datasource "bring up a layer to find your source" scripts
<rharper> cool
<smoser> interesting.
<smoser> serverstack does not populate network_data
<smoser> you can get a config drive like this, rharper
<smoser> nova boot --config-drive=1 --user-data=/tmp/cstack.hr20QA/ud-rendered --key-name=brickies --flavor=m1.small --nic=net-id=db6a8975-5ca2-49d6-8ca7-f8747a163e58 --image=421f67f2-c72d-4396-a678-ff4a16278fb7 xenial-20160321-210440
<smoser> --config-drive=1 is what you want
<rharper> cool
<smoser> going afk for 2 to 3 hours.
<rharper> k
<rharper> smoser: magicalChicken: well, I have run-parts based collect_scripts working now;  the downside is waiting on multi.target adds like 30 seconds to the total test-time;  I'll see if i can just do after cloud-init.final service and see if that speeds things up
<magicalChicken> rharper: 30 seconds isn't really all that long, so that's good
<rharper> magicalChicken: yeah, not terrible; but I think it was closer to 10 seconds without; so it's mostly just empty time
<rharper> yeah, I moved it to just be After=cloud-init.final
<rharper> and we're at 18 seconds
<rharper> cool
<magicalChicken> nice
<rharper> smoser: magicalChicken: ok, pushed the run-parts collect scripts to cloud-init-test master; smoser I'll start poking on the config drive conversion to network.yaml
#cloud-init 2016-03-22
<magicalChicken> smoser, rharper: Hey, I have fallback configuration working in (what I believe) is a mostly sane way
<magicalChicken> lp:~wesley-wiedenmeier/cloud-init/net-fallback
<smoser> reading
<magicalChicken> Currently I am not attempting to rename the device to eth0, as that fails usually, but the code to do that is in place, a parameter just has to be changed
<smoser> merge lines
<magicalChicken> sure, just a second
<magicalChicken> just pushed
<smoser> ok
<smoser> sitll ther ei think
<magicalChicken> there's one or two that kidna make sense for separating sections, I can pop them out too
<smoser> no. i'm complaining about the merge conflict lines
<magicalChicken> oh, whoops
<smoser> :)
<magicalChicken> haha sorry about that i'm pretty tired :)
<smoser> ok, so comments on what is see there.
<smoser> find_fallback_network_device()
<smoser> what id' like is somethign more like:
<smoser>  generate_fallback_config()
<magicalChicken> sure, that makes sense
<smoser> and that would simply return {'config': {....}, 'version': 1}
<smoser> the dictionary
<magicalChicken> okay, and we could generate the .link in distros.debian?
<magicalChicken> it might be okay to just omit the .link file for now until we can figure out the veth naming thing
<smoser> right!
<magicalChicken> haha awesome, okay
<smoser> here... let me show you the diff i have.
<smoser> http://paste.ubuntu.com/15470570/
<smoser> so basically we call init.apply_networking()
<smoser> and it decides which networking source to look at (via find_networking_config)
<smoser> selects the right one, and then calls the distro's "apply" for that.
<magicalChicken> yeah, that makes sense
<smoser> the transfer stuff is sort of unrelated
<magicalChicken> is that from trunk?
<smoser> that diff is versus  my net1
<magicalChicken> ah, okay
<magicalChicken> i can merge in from there and apply the rest of the diff
<smoser> dont worry about grabbing my paste that isnt in my branch yet
<magicalChicken> okay sure
<smoser> you get 'fallback_config()' to just return the config
<magicalChicken> okay
<smoser> make sense ?
<magicalChicken> yeah, I'll get that done real quick
<smoser> oh. and magicalChicken yrou return value should be a dict with 'config' and 'version'
<magicalChicken> Oh, right yeah
<magicalChicken> I've got the change made but something went wrong and the file didn't get written, so debugging rn
<smoser> k
<smoser> you had started taht from my net1 branch, right?
<magicalChicken> yeah, but from around 1191, I think I merged at 1199 though
<smoser> well, the one thing we wanat from your branch right now is the fallback_config()
<smoser> so if you want to just quick re-do that, that'd be fine.
<magicalChicken> just pushed, I think it has everything
<magicalChicken> I renamed the function, and it generates the config fully formatted with the 'config' and 'version' keys
<smoser> ok. i'll get it pulled into my branch and then push up. i will drop some of the other changes though for now.
<magicalChicken> Awesome okay
<magicalChicken> yeah, some of the other stuff was mainly for testing
<magicalChicken> it was cool to see it boot and get a dev configured without the timeout, that'll be pretty nice in the future
<smoser> i think you need to push
<smoser> oh. i needed rto reload
<magicalChicken> haha yeah, sometimes launchpad doesn't show new commits for a couple minutes
<magicalChicken> oh, wait, I'm gonna remove the merge lines, I forgot to push that
<magicalChicken> actually nvm, I think that was just launchpad failing to generate a diff
<smoser> http://paste.ubuntu.com/15470624/
<smoser> that is because i have enp0s25
<magicalChicken> Aah, whoops, I thought str.strip would get the ones in the middle too, but I guess it's only at the beginning and end
<magicalChicken> I can fix that real quick
<smoser> k
<magicalChicken> just pushed, it's fixed now, although the fix is kinda ugly, but I couldn't think of another way to do it on one line
<smoser> k
<magicalChicken> I think I'm gonna sign off for the night, that's pretty much everything I was working on done. I'll look into vmtests for this tomorrow
<smoser> magicalChicken, thanks.
<smoser> magicalChicken, it still needs some work, but good job
<smoser> $ python3 -c 'from cloudinit import net; print(net.generate_fallback_config())'
<smoser> {'config': {'routes': [], 'interfaces': {'vethVJLV0G': {'subnets': [{'type': 'dhcp4'}, {'type': 'dhcp6'}], 'type': 'physical', 'mode': 'manual', 'name': 'vethVJLV0G', 'mac_address': 'fe:15:28:38:df:27', 'inet': 'inet'}}, 'dns': {'search': [], 'nameservers': []}}, 'version': 1}
<smoser> pulled yours into my branch
<smoser> man...
<rharper> smoser: if I'm reading trunk.net1 right , now with fallback from magicalChicken , we can use nocloud-net instead of nocloud for ds mode ?
<rharper> smoser: which openstack did you use to get network_data via config drive ?
<smoser> rharper, sorry. in now.
<smoser> openstack that reads config drive is ConfigDrive
<smoser> i had forgotten that yesterday too.
<rharper> smoser: I meant which cloud
<smoser> serverstack or canonistack will provide a config drive.
<rharper> as you said, serverstack doesn't populate the metadata service with 'network_data' link
<smoser> right.
<rharper> yeah, I saw the config drive
<smoser> so that sucks.
<rharper> so I'm using the example json from the link
<smoser> i dont know where i can get one....
<rharper> but Ideally we'd be able to test this parser out
<smoser> yeah
<rharper> we'd have to standup one and enable it
<smoser> let me search a couple public clouds and see if we can get network md anywhere for openstack.
<rharper> or poke around any of the stacks you have access to
<rharper> yeah
<smoser> i'm not sure why we dont get it
<rharper> so, one thing, I think the routes handling in the network stuff needs an adjustment
<smoser> can you bother someone on opentsack team to see if you can get them to make it go on serverstack ?
<rharper> yeah
<rharper> I need to add support to the subnets section to allow one to specify a list of routes (which we already emit the 'up route ' stanzas for)
<rharper> the 'networks' section in the config drive network bits is made just for that (network netmask gateway)
<smoser> rharper, wel...
<smoser> i got half
<smoser> ssh root@162.253.53.94
<smoser> ok. so half way good news.
<smoser> serverstack has the network config in the MD , but not on the config drive
<smoser> which is actually the same as i'm seeing on vexxhost where iw as able to launch an  instance after adding some networks.
<smoser> suro-patz, you had some questions about networking and such, right ?
<suro-patz> smoser: yes regarding https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1549403
<suro-patz> basically I am working on MaaS equivalent in OpenStack, i.e. Ironic - and looking for configuring multiple IP (v4 & v6) on the target node through cloud-init
<smoser> ah. ok.
<smoser> is it config drive datasource ?
<smoser> where will the network info come from
<suro-patz> smoser: yes, config drive data source
<suro-patz> Neutron
<smoser> rharper, ^
<smoser> that is exactly what we were wanting to work on next.
<smoser> suro-patz, so in https://code.launchpad.net/~smoser/cloud-init/trunk.net1/
<smoser> that branch, the code is mostly there to do this.
<suro-patz> smoser: I will look into the diff
<smoser> well at least the infrastructure.
<suro-patz> that would be helpful
<suro-patz> I made some changes to get it working for RHEL for us
<smoser> suro-patz, the thing that is missing is the ConfigDrive datasource needs a 'network_config' method (see the one in DataSourceNoCloud)
<suro-patz> but I would love to adhere to the infrastructure
<suro-patz> smoser: I will check
<smoser> and taht network_config basically needs to return a "network config" dictionary
<smoser> and then, in theory, it'd all "just work"
<smoser> ah.
<smoser> there are 3 bits of work
<smoser> a.) getting DataSourceCondfigDrive.network_config to return the right stuff.  That will require a parser/converter from openstack entworking format into 'network_config'
<smoser> b.) some improvements to cloudinit/net/ for rendering that network config on ubuntu
<smoser> c.) support for rendering that into Centos style networking
<smoser> because i think you want centos, suro-patz
<smoser> so now i have a question for you...
<smoser> do you know how to convince nvoa to put a 'network_cofnig.json' on a libvirt config drive ?
<suro-patz> smoser: I will work on that as I get things working end-to-end for me, I expect to start on that next week
<smoser> --config-drive=1 gets me networking information in the metadata service but no network_data.json on the config drive
<suro-patz> nova-compute uses a template to parse using jinja2
<suro-patz> we can pass new template/conditionally
<suro-patz> or we can modify the template too
<suro-patz> I had to modify some of the nova-compute code too
<suro-patz> once I get things working for me, which I expect by end of this week, I will work with the community to upstream
<smoser> suro-patz, ?
<smoser> template to parse?
<suro-patz> nova-compute uses a template for the content file
<smoser> i dont want interfaces
<smoser> here. let me show.
<suro-patz> smoser:https://github.com/openstack/nova/blob/master/nova/virt/netutils.py#L76
<smoser> http://paste.ubuntu.com/15473498/
<smoser> suro-patz, so i dont want the "/etc/network/intefaces" style file.
<suro-patz> smoser: got your point
<smoser> i want the more structured network_config.json
<smoser> so the above came from an instance.
<smoser> but the config drive does not have that file
<smoser> :-(
<suro-patz> I am not aware of nova-community's argument on why we should not put network_data.json
<smoser> there is an argument ?
<smoser> it really looksl ike its just busted
<suro-patz> I am not aware of
<smoser> i poked through the code, and it seems to be *trying* to do it.
<suro-patz> we can fix it
<suro-patz> even I felt the same
<suro-patz> we do process meta_data.json
<suro-patz> but the network config is in old format
<smoser> right. i want to dump the old code.
<smoser> not support the network_interfaces reading... well, leave it for dead i guess.
<smoser> try to keep it working.
<smoser> but in cloud-dinit want to focus on reading the network_data.json
<suro-patz> smoser: will work in that direction
<smoser> suro-patz, ggreat. one last thing...
<smoser> for now the target is to read and apply network information from a local data source only
<smoser> ie, config drive tells me what networking should look like, and we apply it before networking ever comes up
<smoser> the more complex scenario of where you need *some* networking to get to network provided network configuration is for later.
<smoser> that make sense ?
<suro-patz> are you referring to metadata service sort of scenario
<smoser> that branch does a good job (on ubuntu at least) of enforcing the NICs dont just come up before cloud-init has had a chance to write the system config.
<smoser> suro-patz, yes,
<smoser> metadata service.
<suro-patz> its like bootstrap and download further config to apply
<smoser> ie, for openstack metadata service, wed' have to first dhcp on some device (or bring up local networking) and then read the network config.
<smoser> right.
<suro-patz> only catch is they should not be conflicting
<smoser> right.
<smoser> the first step (config drive) is pretty easy
<smoser> and almost done there on ubuntu. we write all the config before the OS gets a chance to bring it all up.  so we dont have to worry about changing it.
<smoser> rharper, http://paste.ubuntu.com/15473498/ is a real world example.
<rharper> smoser: nice!
<rharper> smoser: nice and scary with 3 interfaces with dhcp =)
<rharper> smoser: is there a definitive markup for what will get populated there?  services is new;  would like to see bridge examples as well
<smoser> rharper, i dont know.
<rharper> looks like the code
<rharper> I looked at the above nova link, I can dig through there
<smoser> i kind of doubt anything in openstack will be declaring bridges and such at this point
<rharper> almost done parsing the json into network state so we can render eni
<smoser> oh. /me remembers
<smoser> how was it... jayf . works at rackspace on bare metal. they might have somethign like that.
<smoser> rharper, JayF is going to get us some baremetal data
<rharper> smoser: cool!
<smoser> and suro-patz https://bugs.launchpad.net/nova/+bug/1513267 <-- that seems to be maybe what is missing
<JayF> Just don't mine any dogecoin while you're on the box :P
 * smoser googles dogecoin and assumes he should probably stop the bitcoin mining that he was doing :)
<JayF> lol
<smoser> rharper, jump on that box and look around.
<smoser> JayF, mentions that the vlan stuff is patched in in baremetal
<smoser> not in upstream.
<suro-patz> smoser: with "sudo rm -rf /var/lib/instances/*; sudo rm -rf /var/log/cloud-init.log; sudo cloud-init init âlocal' did not reconfigure network from content/0000 - I had changed the content of the file, but it did not apply
<smoser> hm..
<suro-patz> any trick, how can I re-run cloud-init to reconfigure network, without reboot
<suro-patz> even reboot also din't help
<smoser> i dont knwo why that wouldnt have done it.
<smatzek> rm -rf stuff under  /var/lib/cloud/data as well?
<suro-patz> smatzek: no, I had not done that
<suro-patz> will give a shot
<JayF> smoser: I can probably even supply the nova patch we use if oyu want
<smatzek> at a minimum you need to take  /var/lib/cloud/data/instance-id or change the UUID in it, or cloud-init sees the UUID hasn't changed and doesn't run
<smatzek> take > take out
<smoser> JayF, your metdata is odd
<smoser> the config drive layout
<smoser> has 'latest/' that has network_data.json in it
<smoser> but the LIBERTY version (2015-10-15) does not
<smoser> is that by design ?
<JayF> probably means we haven't deployed that upstream bugfix you linked me in pm
<smoser> well, JayF the bug was that it did not appear *anywhere*
<JayF> oh. hm
<JayF> I don't know why :)
<JayF> Asking about stuff we did, at this point, 2y ago :)
<smoser> rharper, for reference http://paste.ubuntu.com/15474058/
<rharper> smoser: thanks!
<smoser> rharper, and you might as well grab config-drive.img.gz and cnonfig-drive.tar.gz of that system. just in case.
<rharper> k
<rharper> got it
<suro-patz> smatzek: works, after removing the instance-id! Cool!
<smoser> JayF, we're done if you want to kill that thing.
<randeffects__> nocloud question: if I use nocloud ds on a vmware vm does the disk image have to stay attached after the initial boot?  if I reboot the vm and the disk image is not attached, cloud init runs again with ds none.
<smoser> randeffects__, yes
<smoser> you're right :)
<smoser> randeffects__, https://launchpad.net/bugs/1553815
<smoser> that is fixed in trunk, and should soon be in xenial
<smoser> but actually the fix i added for nocloud was to only check the seed direcdtories or the kernel command line (not mounting the disk)
<randeffects__> smoser thanks
<smoser> so... but you can get the behavior you want. you just have to be explicit
<smoser> and set 'manual_cache_clean' in the image or in user data
<smoser> actually, that wont work from user-data
<smoser> i just put a comment in that bug
<JayF> smoser: cool, hopefully it helped?
<smoser> yeah. thank  you very much JayF .
<JayF> np anytime I can help with something like that feel free to ask
<vox_clamantis> hi all ... anyone know where to find info on configuring cloud-init to use OpenStack metadata service?
<smoser> vox_clamantis, it should "just work".
<vox_clamantis> :) i was afraid of that
<vox_clamantis> it always picks another like configdrive or e2c. if i specify OpenStack as the only option in datasource_list it never finds it. although i have no problem accessing with curl
<smoser> hm.
<smoser> definitely if config drive is enabled it will select it.
<smoser> but openstack shoudl run before ec2
<smoser> check the order
<vox_clamantis> let me try again. thanks for the suggestion
<smoser> magicalChicken, around ?
<magicalChicken> smoser: Yeah
<smoser> hey....
<smoser> shoot. sorry to ping you .
<magicalChicken> it's okay
<magicalChicken> is there anything I should be working on today aside from tests?
<smoser> heres one
<smoser> kernel command line
<smoser> with your chagnes for kenrel command line in theory curtin should not need any patching of the image
<smoser> ie, the ENI should be able to be left in tact
<magicalChicken> Right yeah
<smoser> so you could use your kernel command line branch
<smoser> and integrate with net1
<smoser> and then try vmtest for curtin
<smoser> the other thing there...
<magicalChicken> cool, okay
<smoser> we should no longer need persistent names
<smoser> in the curtin xenial command line
<smoser> that make sense ? kernel cmdline ...
<smoser>  net.ifnames=0
<smoser> i never rememberr that string
<magicalChicken> Okay yeah, that makes sense
<smoser> i have to run *right now*
<smoser> i'll check in later.
<smoser> bye
<magicalChicken> bye
<rharper> smoser: ok, I've got something that converts the 3 different network-data json into network_state, when then we can render_eni etc;  I'm looking at cloudinit/source/DataSourceConfigDrive ; where do we determine which dir of the config drive to use? for example, the network_data.json is only in latest  ;
<rharper> openstack helpers
<rharper> OS_HAVANA is the latest, prolly should update that
 * rharper relocates
<smoser> rharper, did you actually see it in latest ?
<smoser> because i've not seen it anywhere. althought hat file sys it will be.
<smoser> i think we should assume correct working order of the cloud though and use the LIBERTY date
<smoser> 'latest' is not really something you can use. they're explicilty stating there is a ABI. using latest/ is guaranteed to fail at some point.
#cloud-init 2016-03-23
<rharper> smoser: around ?
<smoser> here
<smoser> looking at why i broke your test suite
<smoser> theres two issues i'm seeing
<smoser> a.) a WARN in the log says the fallback datasource got used, but idont know how (and other parts say NoCloud got used)
<smoser> b.) it seems the nics aren't being renamed appoprirately.
<smoser> do you have an  easy way to get in to the test vm ?
<smoser> ok. well, 'b' seems to be because we have a /etc/network/interfaces inside that doesnt' srouce /etc/network/interfaces.d
<rharper> smoser: it's somewhat tricky to get it  but:  I usually drop the poweroff script in generate_payload_scripts();  this will leave the system up
<rharper> the if you ps, you can get the /tmp/launch.XXXX/boot.img ; copy that out before you kill the vm and the qemu command line;  then I replace -serial file.* with -serial telnet:localhost:2446,server,nowait , quote the --append, then add -monitor stdio -S;  when you run it; you'll get monitor prompt on stdio, guest will be paused;  you can setup your telnet to localhost on the port; then type 'cont' on the monitor;
<rharper> smoser: I've got the configdrive openstack helper updated to search for the network_data.json, so we can fetch this,  we use the same convert_to_json() logic and stuff that in configdrive.networkdata_raw;  I see in stages where userdata_raw and vendordata_raw are processed (and written out);  ideally I'd do that with networkdata and write out a networkdata.json ;  I'm trying to find where best to convert the json networ
<rharper> kdata into network config;  it appears that I need to write out network_config.yaml where we write out userdata conf as well, and then the distro apply_net will do it's thing
<smoser> oh. well the ps is easily fixable
<smoser> rharper, CURTIN_VMTEST_KEEP_DATA_PASS=logs,collect
<smoser> run with that, add 'disks' to it and you'll keep the disks too
<rharper> I usually want the interactive nature
<smoser> right. but i was just saying to keep the disk around
<rharper> oh
<smoser> so you dont have to race it.
<rharper> yeah
<rharper> ah, gotcha
<rharper> or modify the power stage
<rharper> yep
<smoser> ok. so i understand most of the failure now
<smoser> and have some changes.
<smoser> but test_etc_network_interfaces
<smoser> will have to be updated
<smoser> as now i'm (by design) not writing to /etc/network/interfaces
<rharper> http://paste.ubuntu.com/15476168/  (my debug script)
<rharper> smoser: yes;  I think we should use the net.parse_eni() which dumps a dictionary
<rharper> from there we can compare/contrast with a network_state
<smoser> ok.
<smoser> i'll getthis as close as i can
<smoser> and then can you do that ?
<rharper> alternatively, we need to write a network_state that can re-render network_config.yaml
<smoser> i'll submit MP for your git in next 5 minutes.
<rharper> prolly
<rharper> but I'd like to get your thoughts on the config drive bits
<rharper> that's what I'm trying to finish up right now
<smoser> k
<rharper> where to convert the network_data into network_config yaml
<smoser> ah. ok
<smoser> so you have the networkdata_raw (although its not *really* raw) as you json loaded it.
<rharper> right
<smoser> you can call it whatever you want.
<smoser> sure keep it around in the datasource datasource.network_json
<rharper> the same as vendordata_raw (which also was json loaded)
<smoser> then, in datasource.network_config()
<smoser> its a property
<smoser> you can convert it there or anywhere really.
<smoser> would probably be nice to have the code easily runnable with no class wrappers or something
<smoser> def network_config(self):
<smoser>    if not network_json:
<smoser> err.
<smoser>    if not self.network_json:
<smoser>      return None
<rharper> I was looking at also doing the _store_networkdata() in stages.py (also done for userdata/vendordata)
<smoser>    return convert_network_json_to_netweork_config(sle.fnetwork_json)
<smoser> dont worry about that i dont hthink
<rharper> ok
<smoser> just look at how nocloud does it.
<smoser> execept you ahve the conversion stage.
<smoser> userdata and vendordata are different
<smoser> because the get parsed
<rharper> ok
<smoser> and that parsing may include otehr things
<smoser> basically we preproces them once
<smoser> and save the result.
<rharper> ok
<rharper> in the convert, don't we want to write out the network_config file?   somewhere in the cloud dir like the other files ?
<smoser> rharper, http://paste.ubuntu.com/15476227/
<smoser> could you apply that ?
<smoser> you can write it in the cloud dir if you want.
<smoser> i guess its more nicely available for ispection then.
<rharper> smoser: I'll apply
<rharper> smoser: ok
<smoser> as long as its store in the object
<smoser> the object gets pickled
<smoser> and then restored for inspection
<smoser> the files like userdata are only stored because things outside read them.
<smoser> or could i guess.
<smoser> oh. and also, fyi
<smoser>  https://code.launchpad.net/~smoser/+recipe/cloud-init-daily
<smoser> the bug i'll need to fix tomorrow is why i see that WARN for fallback
<smoser> and now i turn into a pumpkin
<smoser> rharper, thank you for all your help.
<smoser> this is really comming along.
<rharper> ok
<rharper> smoser: do I need updated trunk.net1 to verify your patch?
<rharper> looks like it
<rharper> smoser: the curtin eni parser does handle source.d now (blake_r added that just a bit ago)
<rharper> smoser: magicalChicken: ok,pushed your fixes to cloud-init-test master
<smoser> rharper, do you have a branch up for config drive ?
<rharper> smoser: not yet; ran out of time last night;
<rharper> I did commit your change to cloud-init-test
<smoser> yeah, saw that.
<smoser> i'm chasing that fallback datasource issue.
<smoser> once i get that, plan to get the kernel command line stuff
<rharper> I'm gonna get a test-case in for configdrive;
<rharper> that'll help verify the changes to configdrive datasource
<smoser> yeah.
<smoser> that will be very good.
<smoser> magicalChicken, aroun d?
<smoser> rharper, ping
<rharper> smoser: here
<smoser> https://code.launchpad.net/~smoser/cloud-init/trunk.net1-cmdline-ip/
<smoser> rharper, so was lookoing at magicalChicken 's kernel command line there.
<smoser> i'd like some tests, but other than that it seems generally reasonable
<smoser> the thing that i want to ask about, is about using the 'network_state'
<smoser> rather than network_config
<smoser> magicalChicken did this on the fallback config also.
<smoser> i wanted / needed *config* rather than network state.
<smoser> so questions are
<smoser> a.) what renders me a config from a network state? i'm guessing there is something
<smoser> b.) do you thikn its preferable to work with one or the other directly
<smoser> rharper, ^
<smoser> i realize that the merge proposal above doesn't really help you look at it.
<smoser> pull my branch and diff it versus trunk.net1 http://paste.ubuntu.com/15480216/
<rharper> smoser: nothing yet renders a network_config from network_state ; I was looking for that myself but it's not written
<rharper> smoser: I've worked with both;  I find the network_config to be more useful since it can be written out and shared
<smoser> well, that and right now its network_config -> network state as one way
<smoser> :)
<rharper> right, but in general, we're employing conversions from *something* (ip, configdrive) to network_config
<rharper> and then allowing distro.apply_net() to take config to state to eni
<rharper> that seems like a reasonable path for consistency across cloud-init
<smoser> 2 way mapping in NetworkState would be good though.
<smoser> its actually confusing, because 'dump' creates a dictionary with a 'config' and a 'version' that then makes you think "oh, taht is network_state -> network_config'
<rharper> smoser: yes;  we should write the state -> network_config to make it 2 way; that's a good way to ensure we don't miss something
<rharper> smoser: question on the ConfigDriveReader,  it has a _fetch_available_version() call which does some os.listdirs and collects dir names;  it returns it in no particular order;  I seem to be suffering from unsorted errors;  sometimes it doesn't select latest first; I'm only populating the network_data.json under latest/  in the test-case;  this passes; but if it chooses something else (2012-08-03) it will fail to find ne
<rharper> twork_data.json
<smoser> so _fetch_available_version() returns one ?
<smoser> and not necessarily the one you want.
<smoser> yeah, i think then just fix it.
<rharper> so I think _fetch_available_versions which refernces OS_VERSIONS , shouldn't that include OS_LATEST ?
<rharper> ah, it had only passed when selected = OS_LATEST, and supported array was empty; then it defaulted to OS_LATEST which is where I had put the network_data.json
<rharper> it seems to me that OS_VERSIONS should include OS_LATEST; but I'm a bit worried about this change since I don't have a lot of history with configdrive
<smoser> rharper, by design OS_VERSIONS does not include OS_LATEST.
<smoser> unless some special circumstance, you really dont want to ever use 'latest'.
<rharper> ok, then I'll need to populate another dir in addition to latest
<rharper> that's easy enough
<smoser> we're given a versioned api essentially , and assuming we can deal with whatever is "latest" is going to fail at some point.
<rharper> smoser: I need to fix the one unittest, but this is working otherwise: lp:~raharper/cloud-init/config-drive-network-data ; I;'ve got to run, will be back in a while
<smoser> k. i'll review. thanks
<vox_clamantis> smoser ... still having trouble using openstack metadata. would you mind looking at a snippet of my log?
<smoser> sure. pastebin the cloud-init.log
<vox_clamantis> thanks very much
<vox_clamantis> btw to test changes ive been removing /var/lib/cloud/* and rebooting. is that a bad idea? takes much longer to reconfigure the template
<vox_clamantis> http://pastebin.com/CLm5c70E
<vox_clamantis> this is with "datasource_list: [ OpenStack ]"
<smoser> vox_clamantis, thats generally clean.
<smoser> i clear /var/log/cloud-init* also
<vox_clamantis> y that too good
<vox_clamantis> and i reset to dhcp so i get the route to metadata
<smoser> you ran cloud-init init --local
<smoser> right ?
<smoser> that will nto fidn a openstack datasource, as it will only search local data
<vox_clamantis> just a reboot
<smoser> hm.
<vox_clamantis> can i initiate a non-local run?
<smoser> so , without reboot, you can just do
<smoser> rm -Rf /var/lib/cloud /var/log/cloud-init-*
<smoser> cloud-init init --local
<smoser> cloud-init init
<vox_clamantis> ok
<smoser> Searching for data source in: []
<smoser> Mar 23 12:59:21 lin-test [CLOUDINIT] util.py[WARNING]: No instance datasource found! Likely bad things to come!
<smoser> er.. the searching for data sourcein: []
<vox_clamantis> yeah
<smoser> that seems wrong.
<vox_clamantis> i was concerned about that
<smoser> but that coudl be a bad message.
<smoser> could you just let me in some wher e?
<smoser> is that possible, i can probalby solve it in a couple minutes if you can
<smoser> rather than asking you 40 questions
<vox_clamantis> depends what you mean by let you in
<vox_clamantis> lol
<vox_clamantis> not to the network unfortunately
<smoser> 0.7.4 is pretty old. i know it sucks for someone to tell you "can you try a newer version"
<smoser> but....
<waldi> and python 2.6. does cloud-init provide support for that?
<vox_clamantis> i dont mind at all
<vox_clamantis> i believe it is the latest from rhel6 epel
<vox_clamantis> yep still there
<vox_clamantis> id be happy to try anohter
<vox_clamantis> 0.7.6?
<vox_clamantis> just need to install some build tools i think
<smoser> thats surely better.
<smoser> cloud-init should work on 2.6. but it could be that.
<vox_clamantis> thanks for the suggestions
<vox_clamantis> and appreciate all the work you do on this project
<vox_clamantis> ugh i cant even build it on rhel 6. maybe i best just go to configdrive
<rharper> smoser: ok, fixed up that last unittest for config-drive network_data
<smoser> ok. i'll read.
<smoser> rharper, i think your KILO is LIBERTY
<rharper> smoser: OK, I wasn't sure;  i"ll fix up
<smoser> https://github.com/openstack/nova/blob/master/nova/api/metadata/base.py#L76
<rharper> k
<rharper> smoser: do we yet have a unittest that combines configdrive + distro.apply_network() ?  I don't think I saw a distro.apply_network() yet;
<smoser> that does apply_network ?
<smoser> or apply_network_config
<rharper> which ever renders the network_config as distro config file (our case eni)
<rharper> hoping to have a unittest that pulls the json data from ConfigDrive, convert to network_config yaml, then run apply_network() which should call net.render_interfaces() which writes out an eni
<rharper> for VM test, we will need to generate a configdrive iso file and feed the VM that (does the cloudds-local util do that? )
<smoser> convert_networkdata_json
<smoser> what does that do ?
<smoser> rharper, cloud-localds does not do that at the moment.
<rharper> converts the json into network_config yaml
<rharper> this shows up as datasource.network_config (attr)
<rharper> like in nocloud ds
<rharper> I put a long description in the function itself
<smoser> no..
<smoser> convert_networkdata_jso
<smoser> convert_networkdata_json
<rharper> oh, in openstack helpers
<smoser> right.
<smoser> not convert_network_data
<rharper> I assumed it was needed to do the json read , like vendor data
<rharper> but maybe it's not needed at all
<rharper> it looked to me like it ensured that we opened the data as list (json.load()) and returned the list that should be in the json data
<smoser> hm.. ok. i think it should be really necessary.
<smoser> we just want to laod whatever is there. if its bad, then its bad.
<smoser> rharper, does network config have a 'hostname' ?
<rharper> hrm
<rharper> I don;'t think so since hostname wasn't set in eni
<rharper> but nothing prevents us from adding it
<rharper> smoser: you mean it should *not* be needed ?
<smoser> right.
<smoser> should not need that i think
<rharper> k
<smoser> right.
<smoser> sorry
<rharper> heh, we don't actually run get_data(); that's sorta mocked out
<rharper> which makes me somewhat uncomfortable with knowing whether or not we needed that or not (though I think using util.load_json() should be fine as you say)
<smoser> rharper, it shoudl be reasonably easy to *acutally* test.
<smoser> kind of painful
<smoser> but easy to add the config drive data as /dev/vdb on a instance
<smoser> ie, serverstack then just dd that stuff that we got yesterday
<smoser> except for with the fixed diredtory
<rharper> smoser: ok
<smoser> i do understand wanting to actually have unit test though
<rharper> I might try my hand at a get_data() call on datasource
<rharper> mock can help for most of it
<rharper> smoser: if I supply a config drive, what should I specify on the kernel cmdline to have cloud-init use that?
<smoser> rharper, it will just use it.
<smoser> so what you can do is boot the system
<smoser> then just dd the content to /dev/vdb
<smoser> the ephemeral device
<smoser> and then run
<rharper> I don't need to specify ds= XXX or antthing like that
<smoser> rm -Rf /var/lib/cloud /var/log/cloud-init=*
<rharper> ah, and reboot
<smoser> no. because its in the normal search
<smoser> dont have to reboot
<rharper> it'll re-run cloud-init
<rharper> oh
<smoser> cloud-init init --local
<rharper> nice
<rharper> ok
<smoser> cloud-init init
<smoser> that shoudl push it through the code your'e after.
<rharper> nice
<rharper> should I update something in cloud-utils to generate config-drive ?  that'd be useful for cloud-init-test
<smoser> yeah, it woudl be useful. definitely we need it for the vmtest
<rharper> ok
<rharper> should I modify cloud-localds, or create something new cloud-config-drive ?
<rharper> I assume a new command
<rharper> bbiab, need to pickup kiddo
<smoser> yeah, i think new command
<smoser> magicalChicken, https://code.launchpad.net/~smoser/cloud-init/trunk.net1-cmdline-ip/+merge/289968
<smoser> if you have time, take that and run with it.
<smoser> that may be functioanl for ip= on cmdline.
<smoser> based on a lot of your work
<smoser> rharper, i have to run now.
<smoser> will be in tomorrow
<smoser> give your code a test. but largely it looked good to me.
<magicalChicken> smoser: Sure, I'll get it merged tonight and make sure it works
<rharper> smoser: cool
<rharper> smoser: urg, so when I re-inited cloud-init, it decided to use DataSourceOpenStack instead of DataSourceConfigDrive since I'm running this in a OS cloud;  need to figure out how to specify order I supose
<rharper> bah, and /dev/sr0 is populated with configdrive as well
 * rharper launches without config-drive=1 
<rharper> smoser: cool, got it parsing a config drive;  I need to generate a clean config-drive that doesn't include meta-data that dumps the content file (like the rackspace one does) and we should see or interfaces.d/50- file rendered
<rharper> ok, now we're not writing out network_config from content;  but I don't see distro.apply_network() ; I wonder if I need to enable a different module
<rharper> sweet, works!  I don't quite understand why it wont' apply when I run cloud-init init ( the only call to init.apply_network_config()) happens in stage5 and only if you pass --local parameter;  however, when I run cloud-init init --local; it's definitely not run either.  so as a hack, I dropped the if args.local check; and it writes it out fine;  so I'll leave the MP as-is and smoser you can help me figure out what I'm m
<rharper> issing here
#cloud-init 2016-03-25
<ahmetalpbalkan> hi folks I'm trying to build rpm for cloud-init on centos: https://bugs.launchpad.net/cloud-init/+bug/1561169 any ideas why I might be getting this?
<suro-patz> ahmetalpbalkan: I was getting the same error
<suro-patz> I can help you on that
<suro-patz> I updated https://bugs.launchpad.net/cloud-init/+bug/1561169/comments/1 as the work-around for this â¦ ahmetalpbalkan:
<ahmetalpbalkan> suro-patz: wow thanks it helped
<ahmetalpbalkan> suro-patz: still getting an error though, https://bugs.launchpad.net/cloud-init/+bug/1561169/comments/2
<ahmetalpbalkan> I started to suspect if this is happening because I'm already on an azure VM
<ahmetalpbalkan> it complains about /lib/udev/rules.d/66-azure-ephemeral.rules that ought to be packaged.
<suro-patz> ahmetalpbalkan: this is a different error, and appears to be associated with Azure
<suro-patz> I have not encountered this
<suro-patz> I am glad that you could proceed past the previous cheetah-related error
<ahmetalpbalkan> suro-patz: makes sense, I'll try on another machine. thanks for help!
<ahmetalpbalkan> yeah I would've never figured out
<ahmetalpbalkan> I spent 2 hours debugging the other day and couldn't find where string 'd' comes from
<suro-patz> ahmetalpbalkan: incidentally I was debugging the same issue yesterday
<suro-patz> that's why I could readily share the insight!
<ahmetalpbalkan> suro-patz: awesome. I think I'll play with it more over the weekend, my goal is to see if we can fully use cloud-init for Azure cloud. It's only debian/ubuntu at this point.
<suro-patz> cool!
#cloud-init 2016-03-26
<sterfield> hey there
<sterfield> I have a question, hope itâs not a stupid one
<sterfield> I can see that cloud-init is grabing metadata and userdata from different sources, (EC2, nocloud, etc..)
<sterfield> but is it possible to use those metadata / userdata in scripts ? I mean, they are pulled by cloud-init at boot time, so are they available ?
<sterfield> or is it only for cloud-init internal configuration, like default hostname etcâ¦
<sterfield> ?
<waldi> take a look into /var/lib/cloud/instance
<sterfield> oh ok
<sterfield_> sorry about that
<sterfield_> I guess I was D/C at the wrong moment :x
<sterfield_> so you were telling me to look in /var/lib/cloud/instance
<sterfield_> which is a link to the instance/ID folder, which contains all the information pulled by cloud-init, right ?
<waldi> smoser: any news on https://bugs.launchpad.net/cloud-init/+bug/1536961 and https://bugs.launchpad.net/cloud-init/+bug/1536964 ?
<sterfield> Out of the 4 steps of cloud-init (init âlocal, init, config and final), I understood the last three and can see the modules called by each phase. However, Iâm not fully understand the âlocal phase. This is a phase called at very early stage in the boot process, where network for example is not available. However, I donât understand whatâs done during this phase, and also how it is configured.
#cloud-init 2017-03-20
<amarao> Hello everyone. I'm trying (continue to try) to make ConfigDrive working on Debian. It just can't use ConfigDrive, as far as I can see. There is ConfigDrive module, but it is ignored (?). Can someone help me? https://snag.gy/ZcfIvd.jpg
<amarao> cloud-init 0.7.7
<amarao> Ðh, I found what's wrong. 0.7.7 works really badly with ConfigDrive. After I installed 0.7.9 (from Sid, it start to work). Still, rather annoying.
<smoser> amarao, i'm  not sure what would be broken, not enough info there to know
<smoser> you could potentially file a bug against debian asking them to patch their 0.7.7 version with a fix for you
<smoser> i'mn not sure what is wrong, we've definitely used config drive successfully for quite some time
<smoser> but if 0.7.9 works, that seems simplist
<smoser> rharper, so... bonds are probably never going to work in a container
<smoser> at least not unprivilidged
<rharper> smoser: we should ask stgaber
<smoser> well, they do kernel stuff
<smoser> in /sys/
<smoser> oh...
<rharper> but... namespaces?
<smoser> wait we talked about this once. he thought they *should* work, if they were implemented differently
<rharper> huh
<smoser> that ifupdown path uses /sys/ which is not namespaced
<rharper> hrm, I can look at the networkd code
<rharper> not that I Want to
<smoser>  
<smoser> http://paste.ubuntu.com/24215499/
<smoser> thats what happens when you try (after you've modprobed 'bonding' on the host)
<rharper> but in any case, the rename should avoid the mac duplication
<smoser> rharper, i'm not arguing otherwise :)
<rharper> ok
<smoser> i just wanted to be able to recreate in a container
<smoser> i looked at examples/tests/bonding_network.yaml though (curtin) for an example
<rharper> yeah; that looks broken;
<smoser> shouldnt the bond have address of one of the nics ?
<rharper> shame they can't namespace mount /sys
<rharper> it will
<rharper> when you enslave a device to a bond
<rharper> it will pick up one of the slaves' mac address as assign to to bondX
<smoser> http://bazaar.launchpad.net/~curtin-dev/curtin/trunk/view/head:/examples/tests/bonding_network.yaml
<smoser> but why do we explicitly declare its mac then
<rharper> the bond itself does not require an ip or anything
<rharper> that's not required but it's OK
<smoser> hm.. ok.
<rharper> but that's not the issue
<rharper> the mac is going to get duplicated when you list nics by mac
<rharper> ls /sys/class/net/*/mac_adress
<smoser> i know. i understand the issue, i'm not arguing that it is not a bug.
<rharper> ok; I'm somewhat confused with the tangents
<smoser> they're not tangents. :). i was trying to recreate it.
<rharper> in a container yes; it recreates just fine in a VM;  I certainly support fixing/filing bugs against the container
<smoser> i was hoping to be able to look around and find a different way to ignore 'bond' devices than "they are virtual"
<smoser> as i have other virtuals that id o not wish to ignore.
<smoser> do you have an example of how to recreate?
<rharper> smoser: lemme pastebin, it's basically launch an image with the attached network-config from the bug and create a VM with the right macaddress;
<rharper> I think bond is a special w.r.t mac cloning; I don't think other netdevs assume the mac of an underlying secondary device
<smoser> (the other virtuals are 'tap', lxc)
<rharper> macvtap, tun, dummy; ppp, etc...
<rharper> there are a lot of them
<rharper> https://www.freedesktop.org/software/systemd/man/systemd.netdev.html
<smoser> rharper, https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/320291
<smoser> rharper, sure there are lots of virtual types. but when renaming, i'd rather just specifically ignore bonds than ignore all virtual types (which breaks renaming in lxc as it's nics are virtual).
<smoser> anyway, i suggested some changes at that MP, if you want to chat about them we can do that. they're small, and then i think we should get this in
<rharper> smoser: replied for the v2;  most are fine; I think I do want to make sure the get_cfg_by_path is sane;
<rharper> as I said, without that change, the unittest doesn't pass; I've not used pdb to walk through the get_cfg_by_path to see why it didn't fetch value from 'system_info' but that's what I needed for it to pass unittest
<smoser> sure. you can and should. but either yoru change or the '_get_arch_package_mirror_info' is wrong.
<smoser> then the unit test is being passed the wrong config is suspect
<rharper> smoser: so if we provide a network-config in a localds seed it's on the iso but we don't copy it off AFAICT;
<rharper> we should do that
<smoser> "the iso"
<smoser> meaning NoCloud ?
<rharper> yeah
<rharper> cloud-localds
<smoser> its probably just stored in the object
<rharper> I can put in a network-config file in there and pass via iso seed
<rharper> yeah
<smoser> that gets pickled
<rharper> but we write out user-data and other .cfg files
<rharper> just something helpful to see
<smoser> well, we write out user-data.
<rharper> so, I'm looking at the util.get_cfg_by_path and I don't see where it should know to pre-pend 'system_info' in  the distro object ... what am I missing?
<smoser> the Distro() object is passed cfg that is just the system_info() portion
<smoser> system_info: portion
<smoser> so your unit test is probably just wrong in that it passes it the whole thing.
<smoser> i suspect
<rharper> well, heh not my unittest; I just added some on top of how the others worked
<rharper> but I understand what you're saying now
<rharper> the unittest it supplying the distro object the whole cfg, not just the system_info portion
<rharper> if that's how it works, I can fix up which cfg the unittest passes into the Distro class
<smoser> but do you agree that *one* of the two things is wrong.
<smoser> rharper, cloudinit/stages.py in Init.distro is where the distro object is created.
<smoser> and there it clearly gets the system_config
<smoser> well, maybe not clearly :)
<rharper> smoser: it works fine if I pass the write system_info key in;
<smoser> powersj, around ?
<smoser> i have some good news and some baad news
<powersj> yes
<powersj> :\
<smoser> which first ?
<powersj> (has enough bad news with the test failures this weekend)
<powersj> "baad" news, as you say, first please
<smoser> bad news is that something needs improvements in order to avoid failures in cloud-init-integration-[xyz]
<smoser> good news is that those are valid failures.
<smoser> similar to what we saw with curtin
<smoser> you added a test that tested a new feature that was not goign to work with old cloud-init
<smoser> and then we ran tests with old cloud-init expecting that new feature to work
<powersj> ahhh! ok
<smoser> and \o/ surprise!
<smoser> it didnt work :)
<powersj> ok so I need to update the proposed tests to run from source of proposed
<smoser> yeah, i guess we need to do somethinng like that.
<powersj> and similar the daily to test the daily
<powersj> ok that I can fix :)
<powersj> currently looking at curtin CI and why jenkins launchpad plugin is passing the wrong arguments
<smoser> what sucks is that twice (curtin and cloud-init) now i've spent a couple hours thinking and debugging
<magicalChicken> powersj: the new version of tests can do '--ppa=cloud-init-dev/daily', which may be useful for the daily tests
<smoser> why is this breaking! ... when the answer is that fixing it in trunk does not fix all versions everywhere :)
<powersj> smoser: right
<powersj> magicalChicken: that will work, I just need to make sure the unit tests that are ran are from that ppa as well. Otherwise we get this test mismatch between what code is ran versus what tests are ran.
<smoser> magicalChicken, thats cool. but what we (re)found out is that we need to test the integration tests that were in the source of the package.
<smoser> or we have to deal with the mix-matched expectations
<smoser> and somehow version every test
<magicalChicken> oh, right i missed the unit tests bit
<smoser> this is the integration test path
<smoser> powersj, added a unit test for a new feature that went in
<smoser> and then we tested that new integration test with old cloud-init
<smoser> and it failed
<smoser> :)
<smoser> which is expected.
<magicalChicken> ok, that makes sense :)
<magicalChicken> at least the test works
<smoser> ie, we ran trunk integration test and old code
<smoser> right
<smoser> we found this out with curtin a while ago too
<smoser> DUH!
<smoser> the solution we came up with for curtin was that we installed the package and then downloaded the source with apt
<smoser> so that they were guaranteed to match expectations
<magicalChicken> simple but inefficient solution here is to clone the branch we want to test and use tree_run inside that branch, so both cloud-init and integration tests are the version in that branch
<magicalChicken> has the disadvantage that a deb is built each time
<smoser> magicalChicken, well, we're interested in testing *exactly* what is in -proposed or in the archive provided.
<smoser> not something we build, which *might* work out to be the same.
<magicalChicken> smoser: fair enough, pulling the deb-src like with curtin should work then
<smoser> in curtin it was easier, because we actually install curtin into a container and use it there.
<rharper> smoser: when you get a chance, I wanted to discuss how we handle v2 as input; this goes to why I had a .config property on the NetworkState class (it enabled passthrough of v2 in the netplan renderer); I updated a comment in the MR
<smoser> ok. http://c.brickies.net/hangout
<rharper> k
<rangerpb> smoser, paulmey Odd_Bloke larsks Hey would you guys mind peeking at https://code.launchpad.net/~bbaude/cloud-init/+git/cloud-init/+merge/320411 ?
<powersj> smoser: I pushed a fix for the proposed tests to avoid mismatches. It is the same as what we do for curtin. As far as the other failures for integration-[x,y,z], those were introduced last Friday see LP: #1674317. I pushed a MP with fix and ran one test locally.
<powersj> https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/320414
<rangerpb> can someone take pity on me and remind how to run the cloud-init test suite locally?
<powersj> rangerpb: if you are trying to run what the CI tests run, then you need the 'tox' command
<rangerpb> just execute tox ?
<powersj> yep it will then go and build the python envs needed to run the checks we have seutp
<rangerpb> thanks man
<powersj> e.g. python2 and python3 unit tests + flake8 + xenial based tests
<powersj> np
<rangerpb> powersj, can the tests be run outside the context of tox or is that hopeless?
<rangerpb> i have quite a few things to fix apparently
<powersj> rangerpb: you can run outside of tox. Tox is nice because all you need is tox and it keeps you from needing to install other things. However, seeing your failures, I would start with just running nose. Familiar with that?
<rangerpb> nope
<rharper> smoser: I've pushed an update to the v2 net branch; I believe I've fixed up all of the issues you raised
<rangerpb> im familiar with running unittests straight up but i think these tests go beyond just that
<powersj> rangerpb: essentially nose runs the unit tests and the command would be something like 'python -m nose tests/unittests assuming you have all the requirements
<rangerpb> ok
<powersj> there are quite a few things to install, so best to do in a VM, LXD, or virtualenv
<powersj> rangerpb: also looking at your failures, TestAzureBounce is trying to read from /sys/firmware/acpi/tables/OEM0, but that is not always going to be on a system. So you would want to mock that or some how deal with it.
<rangerpb> yeah, i did some refactoring there that is biting me in the ass
<rangerpb> powersj, is there a way with tox to be more specific about what to test ?
<powersj> rangerpb: yes you can specify an env. by running 'tox -e <env>' like 'tox -e py27' to run only the python2 tests. Then if you want to go further and only specify a certain test, you can do that by passing more arguments like 'tox -e py27 -- tests/unittests/test_sshutil.py' to run only the tests in test_sshutil.py
<rangerpb> think i got it
<rangerpb> and done to class and method it looks like
<powersj> yes!
<smoser> rharper,
<smoser> 2017-03-20 19:08:56,496 - network_state.py[WARNING]: NetworkState Version2: missing macaddress
<smoser> that is expected path in lxc
<rharper> right, if you don't have a macaddr and no other match criteria it leaves a warning
<smoser> so i dont want the warning.
<smoser> as its expected path.
<rharper> well
<rharper> no
<rharper> but I understand;  LXD knows the mac
<rharper> no reason to put it in the config it generates
<rharper> I find with dropping the warning; but oracles should emit everything they know
<smoser> well, it certainly *could*, but it does not. so this would make a WARN statement in /vr/log/cloud-init.log in all containers
<Odd_Bloke> rangerpb: That doesn't set the hostname back, right?
<rangerpb> Odd_Bloke, that == ?
<Odd_Bloke> rangerpb: Your MP.
<smoser> i understand your position, but since its expected case... we can't generate warnings when its operating as expected.
<rangerpb> Odd_Bloke, no it doesn't... are you talking about the context method I removed?
<rharper> smoser: ack
<Odd_Bloke> rangerpb: Yeah; I believe we set it back previously because cloud-init's hostname handling gets confused later on if we don't.
<rangerpb> Odd_Bloke, in talking with paulmey, he said it isn't needed.
<rangerpb> i suppose there is some weird use case?
<Odd_Bloke> rangerpb: If it isn't needed, why are we setting the hostname at all?
<rangerpb> azure needs it
<Odd_Bloke> That should be left for cc_set_hostname to do later in the cloud-init run.
<rangerpb> azure needs it when the nic is bounced
<Odd_Bloke> rangerpb: OK, so I'm confused about what paulmey said wasn't needed.
<Odd_Bloke> :)
<rangerpb> azure needs it when the nic is bounced so the dns can get set up correctly
<rangerpb> Odd_Bloke, that make sense?
<Odd_Bloke> rangerpb: Right, so paulmey said that we didn't need to set the hostname _back_?
<smoser> rharper, testing http://paste.ubuntu.com/24216898/
<Odd_Bloke> In Azure terms, he's (presumably ;) correct.  In cloud-init terms, however, ISTR that cc_set_hostname has some behaviour that is affected.
<Odd_Bloke> Give me a minute to see if I can remind myself of what it was.
<rharper> smoser: ok
<smoser> 2017-03-20 19:17:57,467 - network_state.py[DEBUG]: NetworkState Version2: missing "macaddress" info in config entry: eth0: {'dhcp4': True, 'dhcp6': True}
<rharper> smoser: I think it's fine to pull it out entirely;  one could submit the same v2 config to a VM
<smoser> hm..
<rharper> smoser: paste me your netconfig and I'll fix
<smoser> ok then.
<smoser> lets just drop it to a debug
<smoser> rharper, http://paste.ubuntu.com/24216919/
<rangerpb> Odd_Bloke, He didn't ... but ill follow up with him for sure
<smoser> created with http://paste.ubuntu.com/24216920/
<smoser> actually, created with http://paste.ubuntu.com/24216923/
<rharper> I think we can relax the macaddr in the parse
<Odd_Bloke> rangerpb: OK, so when you said "in talking with paulmey, he said it isn't needed", what specifically were you referring to?
<rharper> I did on the rendering side
<rangerpb> Odd_Bloke, that context method
<Odd_Bloke> OK, I see.
<smoser> rharper, and those errors on your mp by the c-ii are valid
<smoser> i didnt' seem them , but no 'indent' on py27
<rharper> I don't know what the c-ii is
<Odd_Bloke> rangerpb: So the problem with setting the hostname and not setting it back is that cc_update_hostname (which is distinct from cc_set_hostname) has some logic that compares the previous hostname cloud-init detected to the desired one and behaves accordingly.
<rangerpb> and how is the desired one determined ?
<Odd_Bloke> So in the case that "update_hostname: B" is given, but Azure wants "A", I think the following can happen: (1) Azure DS will set hostname to "A", (2) cc_update_hostname will set hostname to "B", (3) user reboots, (4) Azure DS sets hostname to "A", (5) cc_update_hostname compares previous-hostname and cloud-config, and determines no hostname change is needed.
<Odd_Bloke> Leaving the instance with hostname "A" instead of "B".
<Odd_Bloke> I _think_ that's the failure case.
<rangerpb> Odd_Bloke, so ... you would prefer the use of the context method yes?
<rangerpb> i need to work thsi out because its right where the test cases are failing and I dont want to do this twice
<Odd_Bloke> rangerpb: Well, I'm pretty sure we need to set the hostname back somehow.  I'd done that with the context manager, but I'm not fussy.
<rangerpb> yep ok
<rangerpb> Odd_Bloke, ok, let me see if I can work that back around
<rangerpb> though I do feel a little bit like I am stuck between Mom and Dad here
<rangerpb> would nice to nuke the agent path in DataSourceAzure to simplify things
<smoser> rharper, c-i bot
<smoser>  https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/319259
<smoser> err.. https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/320291
<rharper> smoser: looks like I need a py2 or 3 wrapper for using indent from textwrap
<Odd_Bloke> rangerpb: :( Sorry.
<rharper> I ran into it a while ago, need to look up the wrapper
<smoser> rharper, http://paste.ubuntu.com/24216970/
<rharper> smoser: k
<rharper> do you want me to pull and push?
<rangerpb> Odd_Bloke, what is the purpose of this conditional? -> https://git.launchpad.net/cloud-init/tree/cloudinit/sources/DataSourceAzure.py#n123
<rangerpb> Odd_Bloke, and also why is line 124 in that important? shouldnt this always be run ?
<rharper> smoser: ok, dropped the warning, and picked up your diff; commited and pushed back up
<rharper> bbiab (picking up kiddos)
<smoser> rharper, i can do it here.
<smoser> ok
<Odd_Bloke> rangerpb: I believe that allows people to modify whether or not they want this behaviour.
<Odd_Bloke> I don't really recall why, I'm afraid.
<smoser> i think Odd_Bloke is probably right above.
<smoser> wrt the case that the temporary hostname is protectign against.
<smoser> https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/319259 rharper, wrt your comment there, you could have used the same merge proposal.  just push over the top of the one (--force) and launchpad will figure it out.
<rharper> smoser: it pushed fine (without force) I did push to the same MP
<rharper> at least I think I did
<smoser> you pointed to another one
<smoser> "I lacked the git-fu to force a proper rebase against itself, so I've pushed a new branch for review:"
<rharper> oh, sorry
<rharper> I don't think I repointed to the old one; but hopefully you'll  find the new commits in my rebase-netconfig-v2-passthrough branch
<rharper> smoser: I think I understand; you're saying I could have forced pushed my rebase branch onto the previous MR;  and yes --force would have done that;  I sort of have an irrational fear of --force and so I worked around it by starting afresh and then abandon the other;  I think it does mean split discussions (which may be one of your concerns)
<smoser> i dont really are... i was just letting you know.  as even a rebase would have required --force
<smoser> rharper, anyway. for right now  i merged your branch.
<smoser> i'm running an integration test on  a few modules and then going to upload to ubuntu.
<rharper> ok
<rharper> so on the rebase; I've found that if I pull from remote (and basically re do the rebase a second time) and then push; it goes in, but I find that I need to do the rebase effectively twice annoying;  I suppose I should just use --force there
<rharper> or figure out why I can't fast-forward the remote (merge trunk in)
<smoser> powersj, https://jenkins.ubuntu.com/server/job/cloud-init-integration-x/123/display/redirect
<smoser> that is curious.
<paulmey> Odd_Bloke, rangerpb, sorry I wasn't here earlier...
<smoser> what is cloud-init-integration-x ?
<smoser> is that daily ppa installed into x ?
<paulmey> I didn't know about the cc modules later on needing the original 'localhost' hostname
<paulmey> I was told that the context method was necessary for waagent, which didn't seem to be true
<paulmey> but I guess it's necessary to do the network bounce at that time in the cloud-init workflow
<rangerpb> paulmey, i think i have another patch set up ... just booting the image on azure as we speak
<rangerpb> if it works, ill update you and highlight you
<paulmey> good, thanks
<powersj> smoser: that is run by checking out master, building, and running the integration tests. Those errors are fixed from my merge you pulled shortly ago.
<powersj> long story short, master was broken and our integration tests found it. Ideally our tox tests would have caught it, but flake8 is too lax. pylint would have caught it though.
<magicalChicken> the tests found a bug \o/
<magicalChicken> i think that's the first one
<powersj> we got one last week too I believe :)
<magicalChicken> nice
<smoser> powersj, yeah... iwas confused for a bit. i was thinking it was just testing xenial or xenial-proposed
<smoser> which should not have shown the error.
<powersj> ok, we do have a xenial-proposed test cloud-init-integration-proposed-x
<smoser> rharper, you there ?
<rharper> here
<smoser> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1669860
<smoser> the bond rename thing... you have the same nic there miultiple times in the config
<rharper> oversite
<smoser> http://paste.ubuntu.com/24217410/
<smoser> ok.
<rharper> wait
<rangerpb> how do I get the ci tests to run again ?
<smoser> tox -e citest -- run -v -n zesty --deb=cloud-init_all.deb
<smoser> thats the easiest thing.
<rharper> smoser: sorry, I'll upload a fix, the upper values should have been gone
<powersj> rangerpb: are you trying to run integration tests or just the simple unit tests?
<smoser> right ok.
<rangerpb> integration, the unit tests finish now and I updated my PR
<powersj> rangerpb: awesome! then trying the command smoser printed to run the integration tests might be good too
<rangerpb> thats just local though right ?
<powersj> the cloud-init_all.deb is produced if you build your working tree
<rharper> smoser: it appears that you could read the list of devices out if /sys/class/net/bonding_masters
<rharper> will need to create a second or third bond to see what that looks like with multiple, but that could provide a filter
<smoser> rharper, i was justlooking and thinking maybe just ignore it if /sys/class/net/<devname>/bonding
<smoser> if that existed
<rharper> well, the bond0.108 vlan also has the mac of the bond,
<rharper> so that's a vlan, but it's mac is duplicate because of bond
<rharper> it get's a bit stacked
<rharper> and we can have a bridge over a bonded vlan
<rharper> or vlan'ed bond
<rharper> which was sort of why I was saying no-virt
<rharper> but I know the lxc case exists
<smoser> maybe its simpler.
<smoser> apply_network_config_names() already filters on 'physical'
<smoser> so we do have a bit of that information present.
<smoser> but i guess it doesnt have the macs. hm.
<smoser> nah. i dont think that helps
<rharper> that's on *config*
<rharper> input
<rharper> my point was that when we "get_current_state"; the non-virt filter really does the right thing (except for LXC) which certainly could handle renames outside of cloud-init; frustrating ...
<smoser> i got to run, i think the right hting to do is to just skip the bonds when collecting info for bymac in this case.
<smoser> i wonder if it will even work though
<smoser> i wonder if you can rename a part of a bond
<smoser> seems unlikely
<smoser> yeah, you cant.
<smoser> but the goal i guess is nto that, but rather to not attempt the rename.
<rharper> yeah
<rharper> I'll be running trunk cloud-init through curtin vmtests setup to do networkd/netplan testing
<rharper> will update here if I find anything
<rharper> thanks for helping push this through
#cloud-init 2017-03-21
<smoser> rharper, https://jenkins.ubuntu.com/server/job/cloud-init-tests/nodes=metal-ppc64el/116/display/redirect
<smoser> that sysstem was deployed wiht cloud-init, so it has a file /etc/systemd/network/50-cloud-init-enP2p1s0.link
<smoser> and the glob in _render_systemd_links tried to unlink it
<smoser> there is a FilesystemMockingTestCase that does a lot of what you had done there.
<smoser> (and also does not get glob.glob
<smoser> )
<smoser> but it could and it'd be good if we did do all these things in one place.
<rharper> smoser: hrm; that's really old cloud-init which emitted .link files instead of doing it manually ...
<rharper> huh; I didn't think we still emitted .link files at all it 's present in the eni renderer; that can't be helpful;
<phr34k> question, how do i go about verifying cloud-config file for correctness? i am trying to automate openstack provider with libcloud and running some cloud_init actions..
<phr34k> it's my first time and i can't figure out where it goes wrong in my setup
<smoser> phr34k, /var/log/cloud-init.log look for WARN is the first thing. /run/cloud-init/result.json will show errors.
<smoser> also /var/log/cloud-init-output.log
<smoser> and the easiest thing to do to debug / quickly launch/test/cycle is lxd
<smoser> lxc launch ubuntu-daily:zesty "--config=user.user-data=$(cat /tmp/mycc.cfg)" my-container
<rangerpb> Odd_Bloke, heya, i updated the patch proposal.  Does https://code.launchpad.net/~bbaude/cloud-init/+git/cloud-init/+merge/320411 look any better to you ?
<Odd_Bloke> rangerpb: Yep, looks much better!  My only comment would be to rename the method to something like "bounce_network_with_azure_hostname" as it no longer really performs a "set".
<rangerpb> ok
<rangerpb> Odd_Bloke, done!
<phr34k> @smoser okay had a look at the files which gave me the following
<phr34k> https://gist.github.com/phr34k/d8caf3f28977cc236a08e8a72e7c8210
<Odd_Bloke> rangerpb: Approved!
<phr34k> is there anything suspicious about the bootcmd command, it's pretty much copied over from the documentations..
<Odd_Bloke> smoser: Can I safely land the Azure bouncing branch, or do you have other stuff going on that means you'd prefer not to ATM?
<smoser> phr34k, i suspect there is some stuff in the output of cloud-init-output.log
<smoser> phr34k, secondly i suspect it will tell us that you need a -F on your mkfs /dev/vdb
<smoser> Odd_Bloke, will review
<Odd_Bloke> smoser: Cool, thanks. :)
<smoser> Odd_Bloke, rangerpb i'm just lookiong at the merge proposal, but i think that the comment on  line 31 ("Bouncing" the network, same ...) is not valid as i think that both are using the *same* method there.
<phr34k> @smoser maybe -i've checked some things form the output.log, related to warnings/bootcmd (updated on the gist https://gist.github.com/phr34k/d8caf3f28977cc236a08e8a72e7c8210)
<smoser> als, the commit message/description needs updating
<smoser> s/als/also/
<smoser> Subject\nBlank Line\nDescription
<phr34k> but i'm not seeing actual references to errors / which commands go wrong in particulair / why it does wrong.
<smoser> phr34k, i'm pretty sure that 'output' is not /var/log/cloud-init-output.log
<smoser> that looks like /var/log/cloud-init.log
<smoser> and i'm pretty sure if you just run: sudo mkfs /dev/vdb
<smoser> it will fail
<rangerpb> smoser, ok just fixed that comment ... by commit message, you are talking about the launchpad commit message not the git commit message right?
<rangerpb> what goes in there?
<smoser> rangerpb, well, the MP might have several commits, and then i "squash" it and commit with the commit message taht you provide.
<smoser> so for your 1 commit, you can take that and put it into 'commit message' in the merge proosal
<phr34k> ah yeah you are right, hmm it seems to complain about /dev/vdb that doesn't exists, what do the things in the [] do in specific? are they even needed for the bootcmd?
<smoser> (as i could too).  but please do mention ideally in the subject that this only affects (i think) the "builtin" path.
<rangerpb> ok done!
<rangerpb> oh
<rangerpb> ok
<smoser> phr34k, i dont know what you're trying to do :)
<smoser> if you want to put a filesystem on /dev/vdb, then doing so is pretty important
<smoser> if there is no /dev/vdb, then it would not seem so important
<smoser> :)
<phr34k> ah the only thing i was trying to do was add a host entry to the host file
<phr34k> there is no /dev/vdb no
<rangerpb> smoser, Odd_Bloke ok, see how that suits ya
<smoser> rangerpb, i'm going to change the subject line of that to
<smoser>  Bounce network interface on Azure when using the built-in path.
<smoser> and then merge
<rangerpb> ok
<rangerpb> totally cool with that.
<smoser> rangerpb, is there a lp bug ?
<rangerpb> Not that I am aware of ... this is something paulmey identified and I fixed for the built-in path
<smoser> ok. thanks.
<smoser> https://bugs.launchpad.net/cloud-init/+bug/1674685
<rangerpb> ah cool
<smoser> rharper, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/320495
<smoser> powersj, ^
<smoser> i verified that this was correct by making 'return_value=["/etc/passwd"]' and see it faili trying to remove that.
<rharper> smoser: yes; that fixes the unittest; I'm worried that we're still writing those files in the first place
<smoser> i think we should be writing them.
<rharper> what effect do they have when we use ip to rename interfaces anyhow?
<smoser> they're not harmful. they will be put intot the initramfs on next update, and will make less work for cloud-init to do. it does seem like duplication a bit.
<rharper> now that i see we're still writing them, I'll need to look at how the interaction works with netplan
<rharper> netplan will also write .link files
<smoser> we only write them when rendering eni
<rharper> y
<rharper> btw; I don't think we should unlink the files before writing
<smoser> so its pretty much the same situation as netplan will be in
<smoser> we unlink ones that cloud-init wrote
<rharper> sure
<smoser> otherwise we end up with stale values there.
<smoser> these are "owned" by cloud-init
<smoser> new instance , they get wiped
<smoser> (same as /etc/network/interfaces.d/50-cloud-init.cfg)
<rharper> yeah;  I was thinking we wouldn't touch it if we've nothing to write;
<smoser> rharper, do you want to approve that?
<smoser> i'm willing to entertain investigation into whether ornot those should be written in eni path.
<smoser> since cloud-init will set the names itself. i think we ended up writing them so they'd get into the initramfs
<smoser> other wise, later attempts would fail due to the "only rename once" thing...
<rharper> smoser: I think it's fine to clean those up; as you say, cloud-init owns them;
<smoser> and then, we decided that in order to support renaming of nics in lxc we had to do it ourselves.
<rharper> well, with ip rename in cloud-init itself, those files just don't matter nor the logic of systemd won't rename; we're doing it anyhow
<smoser> (because no matter what, systemd will refuse to rename those devices that are guaranteed to have been renamed once (from tap-XXXXXX to ethX by lxd))
<smoser> right. the value is on update-initramfs and reboot, there is *less* movement of things.
<smoser> but the end result should be the same. so it is arguable that they're confusing duplication.
<rharper> we could look at transitioning
<smoser> at least those are the cases that i remember.
<rharper> we'd stop writing, but continue removing
<smoser> you do remember all the fun we had getting this right the first time :)
<rharper> that is, we would really want them to linger
<rharper> "fun"
<rharper> I'm still having that fun every time I touch a .service .link or .network file
<smoser> yeah. and i'm afraid that there is probably some use case that i'm not thinking of.
<rharper> smoser: what other paths will cloud-init look in for initial system config?  I think we have /etc/cloud/cloud.cfg.d/*   and /run/cloud-init/*.cfg ;  do we have like a /usr/lib/cloud/cloud.cfg ?
<rharper> smoser: I'm poking at the UC16 case where UC16 bind mounts files or dirs on top of /etc/cloud/cloud.cfg.d (it can't do a union mount as I would prefer);  which means things like the default logging config and other things end clobbered by the mount, or we have to duplicate them into the writable partition when we create the image
<rharper> if we had say a /usr/lib/cloud/cloud.cfg.d/*   we could move things like 05_logging.cfg and other system defaults there, leaving /etc/cloud/cloud.cfg.d for clobbering ...   and I suppose we wouldn't even have to change the default location of those files from etc to the system path; as long as cloud-init also read from that location (UC16 could move them during it's core snap build)
<smoser> rharper, it seems to me that maybe uc16 needs to change. i'm really sort of adverse to adding yet another location for config files.
<rharper> smoser: k
<smoser> do you disagree?
<rharper> smoser: not entirely sure;  to some degree it wouldn't be bad to have the same  system, etc, run config paths;  that would let cloud-init set it's defaults in /usr/lib/cloud/* ; and then override those wil files from etc or run;  that said; there are two issues 1) this is a change in default behavior w.r.t location of these files (if we updated the packaging; the core snap could move them there if we didnt)  2) it certa
<rharper> inly introduces yet another path for config (which has positives and negatives)
<smoser> rharper, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/320555
<powersj> smoser: https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/320560 something for us to talk through at somepoint
#cloud-init 2017-03-22
<smoser> rharper, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/320555 updated
<rharper> smoser: +1
<smoser> and if you could quickly...
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/320656
<rharper> well
<rharper> that one reminds me that we don't have unittests for ds-identify yet, do we ?
<rharper> it's a good fix
<smoser> :-(
<smoser> yeah... i had started a unit test branch.
<smoser> thanks for the advertisment in the review :)
<rharper> =)
<rharper> smoser: hrm, cc_apt_configure.py doesn't appear to have a way to disable itself via cloud-config;
<rharper> smoser: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1675185
<rharper> smoser: also, not sure what to do about UC16's existing netplan config;  if cloud-init runs and has network config; we'd remove that like eni's _maybe_remove_default_config()
#cloud-init 2017-03-23
<smoser> :-(
<uppock> does cloud-init support setting the root password from the openstack admin_pass (using config drive)?
<smoser> powersj, lets chat later about pylint
<powersj> smoser: sent you invite for tomorrow, I'm headed to airport in less than an hour
<jgrimm> smoser, any ideas on 1674861 (gce ds-identify bug)?
<smoser> ah
<smoser> i was just looking at that jgrimm
<smoser> and hit 'submit' on a comment there.
<jgrimm> oh, hope i didn't conflict
<jgrimm> whew, i had added comment about the correct path to invoke ds-identify just in case you still wanted that
<smoser> i think its good now.
<jgrimm> just read, interesting theory
<jgrimm> warning ftw
<rharper> smoser: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1675576  # maybe remove netplan embedded config
#cloud-init 2017-03-24
<powersj> smoser: https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/320560
<powersj> smoser: https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/317692
<powersj> https://code.launchpad.net/~wesley-wiedenmeier/cloud-init/+git/cloud-init/+merge/308218
<powersj> https://code.launchpad.net/~wesley-wiedenmeier/cloud-init/+git/cloud-init/+merge/314496
<smoser> powersj, https://code.launchpad.net/~raharper/curtin/trunk.snappy-dd/+merge/317672
<smoser> powersj, you said you filed a pylint issue
<smoser> for
<smoser>  +    _REQ_VER = LooseVersion(_REQ.version)  # pylint: disable=no-member
<powersj> smoser: yes
<smoser> link K?
<powersj> https://github.com/PyCQA/pylint/issues/1390
<smoser> powersj, http://paste.ubuntu.com/24242020/
<smoser> that look sane ?
<smoser> on top of yours
<smoser> err. http://paste.ubuntu.com/24242021/
<powersj> I like that change, did you run pylint to make sure it is still good?
<smoser> yeah, it is.
<smoser> you're welcome to add a pylint-tip
<powersj> ok let me add your changes and -tip
<powersj> thanks for looking
<smoser> powersj, i can just do it here too.
<smoser> and pull it.
<powersj> ok whatever is easier
<smoser> or you can . etierh way. i wont add the -tip right now if i do it.
<powersj> let me do it then
 * powersj is seriously missing his large monitor right now
<powersj> smoser: merge updated
<smoser> oh..
<smoser> theo hter question...
<smoser> would be nice to get Renderer out of .pylintrc
<powersj> https://paste.ubuntu.com/24242207/
<powersj> I did start to mess with that code, but thought better
<smoser> http://paste.ubuntu.com/24242277/
<smoser> powersj, ^ i think that is right.
<powersj> smoser: is there a way to test that?
<powersj> other than building the deb and injecting it in some image :)
<smoser> powersj, well, pylint runs
<powersj> :D
<smoser> and it is looking at all the things that subclass from Renderer
<smoser> well, i thought it was
<smoser> wait. it might
<powersj> mine just passed with that
<powersj> smoser: want me to commit?
<smoser> powersj, how do i make it show an error ?
<smoser> i want to see the error number
<powersj> hmm I have to remember, I don't think it is as simple as turning on the report
<smoser> pylint --disable=C,R --enable=W0223 cloudinit/net/eni.py '--msg-template={path}:{line}: [{msg_id}({symbol}), {obj}] {msg}'
<smoser> so... its annoying though
<smoser> you cant enable just that warning
<smoser> if you disable W (like you did in the .pylintrc) then you can't enable a specific warning
<smoser> powersj, so... i added that @abc.abstract
<powersj> added to?
<smoser> and then i removed an the eni.Renderer
<smoser> http://paste.ubuntu.com/24242414/
<smoser> and saw that pylint *does* complain about that.
<smoser> so i thought "ok, i'd like to see other occurences of W0224"
<smoser> but, thats not so easy
<powersj> :)
<smoser> i would have hoped that explicitly listing a W0224 in the enable would do that.
<smoser> but it seems if they're turned off in the disable with 'W' (all warnings) then you cant turn back on
<smoser> i thik
<powersj> I agree, I tried disable: all and then enable a few things, but that didn't seem to work
<smoser> i think you can disable the things one by one
<smoser> WXXX,WXXX,WXXX
<smoser> so we could get there.
<smoser> i just merged.
<smoser> you can diff trunk versus your branch to see the delta
<powersj> thanks! looks like you fixed up renderer
<smoser> yeah. and removed a duouble declaration of thread._local ?
<smoser> powersj, now i'd love to get to a point where we did not have to disable all warnings.
<powersj> oh wow.. didn't even see that
<powersj> yeah that would be nice; as you said those are helpful
<powersj> a lot of the warnings are about exceptions being too broad
<smoser> powersj, why didnt ci run on https://code.launchpad.net/~multani/cloud-init/+git/cloud-init/+merge/320815
<powersj> ah this is the whole groups permissions thing
<powersj> jenkins-launchpad-plugin only lets you run jobs if people are in certain groups, so I would need to add his name to the list of valid users
<powersj> I've done this for a few other people as well
<smoser> ah. k
<powersj> next run should pick his up
<powersj> smoser: https://paste.ubuntu.com/24242635/ 465 warnings: 129x too general exception 112x using deprecated warn 47x unused argument 33x unused variable 25x TODO
<smoser> so it'd be nice to see the unused variable
<smoser> as honestly we should fix those
<smoser> and the deprecpated warn we could fix too
<powersj> smoser: when you are really bored this weekend you can snap cloud-init ;) https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/320994
#cloud-init 2018-03-19
<rharper> https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/341662
<rharper> smoser: blackboxsw ^
<smoser> blackboxsw: cloudinit/config/cc_puppet.py:143
<smoser> did you l ook at that more closely ?
<smoser>  cloudinit/config/cc_puppet.py:143: [W1505(deprecated-method), handle] Using deprecated method readfp()
<smoser> hm..
<smoser> need to fix one way or another.
<blackboxsw> smoser: nope, the puppet fix is one I had wanted to come back to today/tomorrow
<smoser> that was a dude-on-the-internet-broke-our-ci change.
<blackboxsw> we need to sort it across releases :/
<blackboxsw> I just put a bandaid on it
<blackboxsw> ...and it's that time again
<blackboxsw>  #startmeeting Cloud-init bi-weekly status meeting
<blackboxsw> #startmeeting Cloud-init bi-weekly status meeting
<meetingology> Meeting started Mon Mar 19 16:02:30 2018 UTC.  The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology.
<meetingology> Available commands: action commands idea info link nick
<blackboxsw> ok, let's kick off this cloud-init bi-weekly meeting. welcome all!
<dpb1> o/
<blackboxsw> it's been a busy couple weeks for a few of us w/ planning meetings and vacation, but let's see what progress we've made on cloud-init.
<blackboxsw> #topic Recent Changes
<dpb1> smoser vacation specifically
<blackboxsw> hehe. Generally we're tracking high-points of what  lands in our trello board
<blackboxsw> #link http://trello.com/b/hFtWKUn3/daily-cloud-init-curtin
<blackboxsw> but from changelogs folks have made progress on azure, vmware and FreeBSD deployment targets
<blackboxsw>     - netplan: render bridge port-priority values (LP: #1735821)
<blackboxsw>     - net: recognize iscsi root cases without ip= on kernel command line.
<blackboxsw>       (LP: #1752391)
<blackboxsw>     - util: Fix subp regression. Allow specifying subp command as a string.
<blackboxsw>        (LP: #1755965)
<blackboxsw>     - This commit fixes get_hostname on the AzureDataSource.
<blackboxsw>       [Douglas Jordan] (LP: #1754495)
<ubot5`> Launchpad bug 1735821 in nplan (Ubuntu Artful) "netplan needs bridge port-priority support" [Medium,Fix committed] https://launchpad.net/bugs/1735821
<blackboxsw>     - shellify: raise TypeError on bad input.
<blackboxsw>     - FreeBSD: Set hostname to FQDN. [Dominic Schlegel] (LP: #1753499)
<blackboxsw>     - Make salt minion module work on FreeBSD.
<blackboxsw>       [Dominic Schlegel] (LP: #1721503)
<ubot5`> Launchpad bug 1752391 in cloud-init "cloud-init does not recognize initramfs provided network config in all cases" [Medium,Fix committed] https://launchpad.net/bugs/1752391
<blackboxsw>     - set_hostname: When present in metadata, set it before network bringup.
<blackboxsw>        (LP: #1746455) VMWare
<ubot5`> Launchpad bug 1755965 in cloud-init "util.subp regression: no longer accept commands as string" [High,Fix committed] https://launchpad.net/bugs/1755965
<blackboxsw>     - cc_snap: Add new module to install and configure snapd and snap
<blackboxsw>       packages.
<blackboxsw>     - doc: fix all warnings issued by 'tox -e doc'
<blackboxsw>     - tests: Make pylint happy and fix python2.6 uses of assertRaisesRegex.
<ubot5`> Launchpad bug 1755965 in cloud-init "duplicate for #1754495 util.subp regression: no longer accept commands as string" [High,Fix committed] https://launchpad.net/bugs/1755965
<blackboxsw>     - tests: fix run_tree and bddeb
<blackboxsw>     - tests: Fix some warnings in tests that popped up with newer python.
<ubot5`> Launchpad bug 1753499 in cloud-init "hostname in FreeBSD should prefere FQDN" [Undecided,Fix committed] https://launchpad.net/bugs/1753499
<blackboxsw>     - tests: fix flakes warning for unused variable
<blackboxsw>     - tests: patch leaked stderr messages from snap unit tests
<ubot5`> Launchpad bug 1721503 in cloud-init "salt module not able to be used on FreeBSD" [Medium,Fix committed] https://launchpad.net/bugs/1721503
<blackboxsw>     - tests: Centralize and re-use skipTest based on json schema presense.
<ubot5`> Launchpad bug 1746455 in cloud-init "cloud-init vSphere cloud provider DHCP unique hostname issue" [High,Fix committed] https://launchpad.net/bugs/1746455
<blackboxsw> a big thanks to dojordan (Azure) and Dominic Schlegel (FreeBSD) for patching some gaps in support as cloud-init master progresses
<blackboxsw> On the ubuntu side of the house we got tip of tree published into Bionic thusday & friday, we are awaiting cloud-image builds which look like they are stale at 03-15-2018. once those builds are published, all clouds should be getting latest cloud-init on Bionic
<blackboxsw> I think that's probably it for 'done' work. We have a few things in flight at the moment
<blackboxsw> #topic In-progress Development
<blackboxsw> Ubuntu is getting a number of new cloud-config modules:
<blackboxsw>  - new cc_snap module (deprecated cc_snappy and cc_snap_config modules) the ability to install and manage snap package
<blackboxsw> - new cc_ubuntu_drivers: support to install 3rd party drivers on install
<blackboxsw> - new cc_ubuntu_advantage: manage Ubuntu Advantage subscriptions for services such as Extended Security Mainenance (trusty), canonical livepatch and FIPS PPAs
<blackboxsw> these should be landing in the week(s) to come
<blackboxsw> and cc_snap landed already
<blackboxsw> also there are a couple of branches that we are trying to wrap up for first class chrony support (per rharper, inspired by robjo's work)
<rharper> blackboxsw: smoser: on the lander emails, the subject could include the git hash (or branch name); it's currently only in the body;
<smoser> rharper: i had suggested to blackboxsw that it should acutally change to *not* send a subject. so it threads in your email reader with the other MP mails.
<rharper> heh
<blackboxsw> maybe we can toggle between the two modulus 2 :)
<rharper> sorry, didn't meant to disturb the flow
<rharper> continue
<blackboxsw> yeah, we've also touched a little bit of our code landing automation this last week. powersj also is working on a git lander plugin that we might be able to use to automate landing of approved branches w/ tox test runs
<blackboxsw> anything to free up developer time will give us more time for reviews/code
<ajorg> should vendor-specific modules be shipped in a separate package?
<smoser> vendor specific modules ?
<ajorg> ubuntu_advantage
<blackboxsw> good question/point. I hand't thought about that separation as a lot of the modules cloud-init delivers support a subset of distros
<blackboxsw> each module has a distro attribute defined as to whether or not it will even run
<blackboxsw> so we have spacewalk, zypper_repos etc
<smoser> and cloud.cfg is rendered based on knowledge of the distro
<smoser> so ubuntu_advantage wont even be in the list of config modules
<smoser> having a static config module list is WIN in this case (but pain elsewhere)
<smoser> at some point whe may have a more dynamic config module list.
<smoser> but anyway... at the moment the only cost to non-ubuntu of that module being shipped is bytes on disk.
<blackboxsw> if/when we do define that more dynamic config module list, I'd like us also to look at having configurable/separate plugin directories defined for folks providing vendor-specific content.
<ajorg> agree that having a more dynamic config module list is prerequisite to being able to parcel out modules to other packages
<blackboxsw> so that we don't expect folks to add plugins directly into /usr/lib/python3/dist-packages/cloudinit/config/ for instance
<smoser> yeah. at the point when it is dynamic, the module would still declare its support for a list of distros and would be filtered out.
 * ajorg is satisfied
<blackboxsw> :).    the only other thing I can think of in progress two more datasources  softlayer cloud support by smoser and hetzner cloud
<blackboxsw> so cloud-init is getting it's grubby hands into a couple of more clouds shortly.
<blackboxsw> s/it's/its/
<blackboxsw> it's nice to see the adoption continue to grow
<smoser> looks like someone followed up on hetzner
<smoser> so that hopefully is ready to land
<blackboxsw> #link https://code.launchpad.net/~lp-markusschade/cloud-init/+git/cloud-init
<blackboxsw> rharper: also has a couple of branches to allow cloud-init to work a bit better when rendering netplan configuration
<blackboxsw> I *think* that's all for in-progress development at the moment.
<smoser> man we need to fix that pylint thing.
<smoser> did you mention ?
<rharper> blackboxsw: yeah, I just pushed a fix for v1 global dns entries to get rendered under interfaces without any dns configuration
<blackboxsw> Anything else that should be noted by anyone?
<blackboxsw> #link https://code.launchpad.net/~lp-markusschade/cloud-init/+git/cloud-init/+merge/338439
<blackboxsw> oops
<smoser> cloudinit/config/cc_puppet.py:143: [W1505(deprecated-method), handle] Using deprecated method readfp()
<blackboxsw> #link https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/341662
<smoser> that needs fixing. it has come to us due to a new version of some of our tox environemnts.  we do not fully pin the versions , only the top level packages.  Ie, pylint's dependencies changed, but we only pin pylint version.
<blackboxsw> yeah how much should we freeze our deps?
<blackboxsw> it's kindof annoying to have your branch locally pass ci, and a fresh build of CI deps fail when you try to land
<blackboxsw> but I don't really know whether it's worth us 'pinning' everything
<smoser> i think pinning everything is generally the best practice for this sort of thing now.
<smoser> oh my.
<blackboxsw> so if we pin the world, should we also just make it a habit to occasionally tox -e tip-pylint?
<smoser> blackboxsw: did you know you accidently fixed that in trunk ?
<blackboxsw> smoser: I know I intentionally added a pylint ignore on that to come back and address it today.
<ajorg> i tend to believe it's better to stay current and take your punches a few at a time so you don't have a major upset when you have to upgrade.
<smoser> oh ko. i see.
<smoser> ajorg: well, sor tof. if you have c-i that you want to be green, and consider it bad when it is not, then you dont want dude-on-the-internet to break you
<smoser> there is the "good" break, where new upload to pypi identifies some lingering bug
<blackboxsw> +1 ajorg, but I'm good (on avoiding a avalanche) if we agree to run tip-pylint target fairy regularly to avoid the landslide
<smoser> but also the "bad" break where some upload breaks your c-i for invalid reason.
<smoser> one huge advantage to pinning is the ability to re-create things.
<blackboxsw> it definitely felt like last week was a lot of c-i breaks  for changes unrelated to the code up for review
<smoser> ie, if you were looking to it bisect something...
<smoser> git bisect
<blackboxsw> I should transition to the office hours topic so we can continue discussion
<smoser> you can't really do that if trunk from a point in the past does not pass C-I because an external dependency changed.
<smoser> sure we can transition to office  hours.
<blackboxsw> #topic Office hours (next ~30 mins)
<smoser> but yeah... i want c-i on tip to not just start failing.
<stanguturi> @blackboxsw, I have couple of requests. First, https://bugs.launchpad.net/ubuntu/+source/open-vm-tools/+bug/1724128 discussed this in last meeting as well. Any help is greatly appreciated
<ubot5`> Ubuntu bug 1724128 in open-vm-tools (Ubuntu) "Need a Success / Failure notification mechanism when cloud-init finishes." [Undecided,New]
<blackboxsw> that's fair. smoser can we maybe add a jenkins job to run tip-pylint then weekly. So, we don't have a huge backlog of lint failures against tip?
<smoser> blackboxsw: i'mi fine with that... thats why we added the tip-* targets. so it was easy enough to keep current.
<blackboxsw> +1 smoser I'll put a card up for that
<smoser> stanguturi: your suggestion there is not a bad idea at all.
<blackboxsw> #link https://trello.com/c/vD1em9WP/698-jenkins-job-to-run-tox-tip-pylint-weekly-pin-all-lint-versions-otherwise
<smoser> the desire to have cloud-init tell the platform/datasource that it failed or succeeded is valid.
<smoser> with MAAS, that his done via reporting
<blackboxsw> stanguturi: hiya. I think we talked after that meeting about trying to allow the datasource to subscribe to a callback when cloud-init exists
<blackboxsw> stanguturi: hiya. I think we talked after that meeting about trying to allow the datasource to subscribe to a callback when cloud-init exits
<smoser> cloud-init reports status and results via a Reporter.
<smoser> dojordan (i think) had also put up a request for a reproter module on azure.
<smoser> so we *do* kind of have the function you're after  in place.
<stanguturi> smoser: ok. Any inputs / examples of using it will be really great.
<smoser> stanguturi: well, the reporter interface is pretty simple. you can cloudinit/reporting/handlers.py
<smoser> blackboxsw: http://paste.ubuntu.com/p/6KjDX8WHQH/
<smoser> did you intend boht of those changes ?
<stanguturi> smoser: ok. Then do I need to write a new handler for our DataSource?
<smoser> stanguturi: well you write a reporting Handler, ankd then either system confi or optionally datasource config would turn that reporter on.
<blackboxsw> wow smoser, no
<blackboxsw> wow, ok, I'll put up a branch to fix that
<stanguturi> smoser: ok. Will work on that.
<smoser> ok.
<stanguturi> I have another quick request about https://code.launchpad.net/~sankaraditya/cloud-init/+git/cloud-init/+ref/set_hwclock_module
<smoser> stanguturi: do you find this is actually needed ?
<smoser> to my knowledge the only time anyone would ever set their hardware clock to something other than UTC would be dual booting with windows.
<smoser> which i can't seem to believe is all that a common situation in VMs
<stanguturi> smoser: Yeah. We need this for 'VMware guest customization workflow'.
<dpb1> smoser: oh man, I hope not
<stanguturi> smoser: if you think, this is not worth for the base cloud-init modules, I can modify to do this change in our datasource specific modules.
<smoser> stanguturi: does it actually solve a *current* problem for you ?
<stanguturi> smoser: For 'VMware managed VMs', customers can specify in the specification file if they want UTC or localtime for the hardware clock.
<smoser> or one that originally came in from a decade ago
<stanguturi> smoser: Our existing customization (non cloud-init) engine does it. If we want to move to cloud-init, we want to port all our changes from our engine to our datasource in cloud-init.
<smoser> hm... so my sugestion is really to stop allowing that :)
<stanguturi> smoser: Oh ok. Can you please add a comment to that merge request just for the record.
<smoser> i very well could be wrong
<smoser> but the only time that i ever had to deal with this was when dual booting
<smoser> with windows specifically
<stanguturi> smoser: ok.
<smoser> am i wrong there ?
<smoser> i really *could* be.
<stanguturi> smoser: I can discuss this within our team. But to be on par with our existing engine, want to port the changes.
<smoser> and even if you get it wrong, generally speaking you havhe some sort of ntpdate or ntp that will fix your system clock anyway.
<smoser> stanguturi: yeah. i understand that.
<stanguturi> I have another request. For Ubuntu 18.04, we are planning to set 'disable_vmware_customization' flag to False by default in /etc/cloud/cloud.cfg file.
<stanguturi> Want to know your opinion, shall we set it in cloud-init installation phase or request Ubuntu maintainers to set it in 18.04
<stanguturi> smoser: And when is the cloud-init 18.2 scheduled for release? 3/22?
<blackboxsw> probably a good time for us to bring that up
<smoser> yeah. whoops.
<smoser> :)
<smoser> 18.2 is scheduled for 3/22 (thursday)
<blackboxsw> We cloud-init 18.2 have it scheduled for an arbitrary 3/22 date, we'd like to slip that out to next week Tuesday 3/27
<smoser> but amoung our internall team we decided to push that to 3/27
<stanguturi> smoser: ok. Thanks for the update.
<blackboxsw> there a a few in flight branches, azure, softlayer etc that we'd like to get in and get tested before 18.2
<smoser> we will send an email today or tomrrow witih "pending release" like subject like we've done before.
<blackboxsw> dpb1: others any objections to cutting the 18.2 release on Tuesday 3/27?
<dpb1> none
 * blackboxsw adds the upcoming date to the topic
* blackboxsw changed the topic of #cloud-init to:   Reviews: http://bit.ly/ci-reviews | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting: Monday 4/2 16:00 UTC | cloud-init 18.1 released (02/22/18) | cloud-init 18.2 (03/27/18)
<blackboxsw> ... ok, folks interested in discussing today?
<blackboxsw> #link https://cloud-init.github.io
<blackboxsw> The above link will have our captured logs for this meeting.
<blackboxsw> thanks again for tuning in
<blackboxsw> #endmeeting
<meetingology> Meeting ended Mon Mar 19 17:09:23 2018 UTC.
<meetingology> Minutes:        http://ubottu.com/meetingology/logs/cloud-init/2018/cloud-init.2018-03-19-16.02.moin.txt
<blackboxsw> smoser: good point on ubuntu-advantage common code between cC_snap and cc_ubuntu_advantage. Shall it live in cloudinit.config.helpers? cloudinit.config.common? or somewhere else?
<blackboxsw> or cloudinit.config_util?
<smoser> blackboxsw: hm... wonder where to go with it.
<blackboxsw> rharper: same question for common functions that are reused only in cc modules, where would you expect them to live? I don't want to bloat cloudinit.util any more that it already is
<smoser> not the monolith 'util'
<blackboxsw> yeah
<blackboxsw> yeah we still need to decouple functionality from util into smaller modules if we can
<rharper> for sure
<rharper> I wanted to unbundle the network bits out of util
<blackboxsw> +1
<rharper> so we pull in fewer modules for cloud-init local
<rharper> I sort of like cloudinit.config.util
<smoser> for the functions you are adding, they're subprocess related.
<smoser> it might make sense to have a cloudinit.subprocess
<smoser> tjhat had
<rharper> yeah
<smoser> subp
<smoser> runparts
<smoser> shellify
<blackboxsw> that works for me. so we create a new module cloudinit.subprocess then.
<blackboxsw> smoser: rharper will move runparts shellify into subprocess as a separate branch ok?
<Odd_Bloke> Bikeshedding, but semi-shadowing a builtin module could be confusing.
<Odd_Bloke> (from cloudinit import subprocess -> confusion)
<smoser> yeah. name subprocess isnt great.
<Odd_Bloke> (Also from .subprocess import blah would work and also be confusing to read.)
<blackboxsw> subp_utils || util_subp?
<blackboxsw> we have cloudinit.util and cloudinit.temp_utils currently
<blackboxsw> and ssh_util
<blackboxsw> hrm cs_utils should probably be under cloudinit/source/helpers
<rharper> cloudinit.supb would be decent and avoid Odd_Bloke good point
<Odd_Bloke> SUP B
<blackboxsw> :) all caps on typos works much better
<blackboxsw> sin boldly
<rharper> *whoosh*
<Odd_Bloke> This is a great definition: http://onlineslangdictionary.com/meaning-definition-of/'sup,-b%253f
<rharper> maybe cloudinit.whasssuuuuuuuuuuuuuuuup
<blackboxsw> heh
<Odd_Bloke> cloudinit.sup.b
<Odd_Bloke> cloudinit.sup.what.is
<rharper> lol
<smoser> that reminds me of garrent holmstrom irc/launchpad name
<smoser> https://launchpad.net/~gholms
<smoser> gee homes
<dpb1> smoser: thankfully mutt threads by in-reply-to:, not the subject, but I see what you mean in gmail.
#cloud-init 2018-03-20
<redcavalier> Hi, several months ago I reported https://bugs.launchpad.net/cloud-init/+bug/1722992 . Since then, we've been running an outdated version of cloud-init. Are there plans to fix this soon?
<ubot5`> Ubuntu bug 1722992 in cloud-init "On the latest centos 7 release, we are unable to resize our instances filesystems" [Medium,Confirmed]
<Odd_Bloke> smoser: blackboxsw: Is https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1757176 a known issue?
<ubot5`> Ubuntu bug 1757176 in cloud-init (Ubuntu) "TypeError: get_hostname() got an unexpected keyword argument 'metadata_only' on GCE bionic boot" [Undecided,New]
<blackboxsw> wow Odd_Bloke I'll look at that. thanks. That feels like version mismatch (as we just grew the metadata_only arg)... but will glance after this meeting I'm in.
<Odd_Bloke> blackboxsw: Thanks!
<blackboxsw> Odd_Bloke: have access to cloud-init collect-logs  on the CLI there? it'll dump a *log.tar in the cwd
<Odd_Bloke> blackboxsw: I can't get in to the machine, unfortunately.
<blackboxsw> no prob. will see if I can reproduce the prob
<Odd_Bloke> SSH host keys weren't created, so I can't SSH in.
<blackboxsw> Odd_Bloke: confirmed and I can fix that today
<Odd_Bloke> Thanks!
<blackboxsw> I see what happened
<Odd_Bloke> blackboxsw: Should we expect it to only happen on GCE?
<blackboxsw> working it right now Odd_Bloke . I'm checking other datasources now
<blackboxsw> Odd_Bloke: CloudSigma AliYun, OpenNebula looks like too
<blackboxsw> but fallout is contained to those
<Odd_Bloke> Ack, thanks.
<blackboxsw> easy fix, should be up, reviewed and posted to bionic today
<blackboxsw> smoser: rharper here's the fix the the bug above http://paste.ubuntu.com/p/RVC6FQbwTB/
<blackboxsw> I'm trying to decide how much unit testing we should have to cover it
<smoser> surprised pylint didnt scream on that.
<smoser> well, mayb e not
<blackboxsw> to exercise this, I could have a suite of tests that initializes each fake datasource only to run the base class get_hostname method passing metadata_only to make sure it is accepted... but that seems overkill.
<smoser> different signatures on subclasses.
<smoser> yeah. i agree.
<smoser> i dont know.
<smoser> ii dont want to say "no unit tests are OK".
<blackboxsw> yeah smoser I was suprised about the different named param especiialy in b/cloudinit/sources/DataSourceAliYun.py case which renames resolve_ip to _resolve_ip
<smoser> but agree on the pain.
<smoser> well, that is probably fallout *from* pylint
<blackboxsw> I feel like I should fix aliyin param name too so it matches base class
<smoser> it complains if you dont use a varibable somteims
<smoser> i think its fine to change the aliyun param  name
<blackboxsw> maybe I add a unit test per datasource I've changed to make sure get_hostname accepts the proper param names as defined in base class?
<blackboxsw> at least then the subclass tests that it's implementation provides the same 'api' as the base class
<blackboxsw> s/it's/its/
<blackboxsw> though if parent class implementation moves on, these unit tests would still incorrectly succeed
<blackboxsw> I've got an idea.
<smoser> blackboxsw: did we have a general soltuion for json.dumps({dictionary that miht have some binary keys })
<blackboxsw> hrm, we have process_base64_metadata in cloudinit.sources.__init__
<blackboxsw> and json_dumps uses json_serialize_default  in cloudinit.util
<blackboxsw> which prefixes base64 content with ci-b64
<smoser> nice. thanks
<smoser> blackboxsw: http://paste.ubuntu.com/p/8Hxy67h5X2/
<blackboxsw> smoser: interesting, will fix now.
<blackboxsw> smoser: care if I land that branch in the process of fixing review-mps?
<smoser> blackboxsw: fine wit em
<blackboxsw> my regex was bad
<blackboxsw> didn't allow for digits in the lp_user
<blackboxsw> strangely we hand't hit that type of user until now
<blackboxsw> ok tox is running against it prior to merge. should land in a couple mins
<blackboxsw> review-mps pushed
<blackboxsw> OpenNebula should land shortly
<blackboxsw> ...done
<blackboxsw> smoser: rharper for review and landing today to unblock GCE bionic
<blackboxsw> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/341757
<rharper> looking
<blackboxsw> thanks
<rharper> inspect, nifty
<smoser> blackboxsw: fyi, c-i says go away
<smoser> i suspect due to 'inspect' :)
<smoser> no. py27
 * blackboxsw is testing on py2.6 now too and no go on that unit test approach
<blackboxsw> hrm
<blackboxsw> ahh gotit smoser rharper 30 seconds
<blackboxsw> my last change with lints busted it
<blackboxsw> will push a fix
<blackboxsw> smoser: rharper fd165e92b0e22245d41e7f55a5f89de4b6b522a6 pushed
<blackboxsw> just tested on py2.6
<rharper> k
<blackboxsw> as well
<blackboxsw> giving a GCE deployment of it to confirm sucees
<blackboxsw> giving a GCE deployment of it to confirm sucess
<blackboxsw> validated on GCE by upgrading xenial to my branch and cloud-init clean --logs --reboot
 * blackboxsw validates failure case on GCE now using cloud-init-tip without get_hostname fix
<blackboxsw> sure enough, here's the failure on GCE without my changeset https://pastebin.ubuntu.com/p/NpMNDVMhz6/
<blackboxsw> ok gotta run. I still need to sort that lint error on my branch looks like
<blackboxsw> back in 30
<blackboxsw> back
<dojordan> @blackboxsw, any update on the UrlError handling discussion?
<blackboxsw> dojordan: saw your changes at least to readurl yesterday. hadn't gotten through a followup discussion on URLError handling.
<blackboxsw> let's see if we can hash it out here.
<blackboxsw> so URLError can be raised by readurl in two cases:
<Odd_Bloke> Can someone remind me what the new cloud-init SRU process/cadence is?
<blackboxsw>  1. IMDS service is currently unavailable (during service upgrade or something) This would raise an URLError with the string "[Errno 111] Connection refused"
<blackboxsw> 2. No network is configured on the instance with the e.cause string containing '[Errno 101] Network is unreachable'
<blackboxsw> right now cloudinit's URLError doesn't surface errno on the exception raised
<blackboxsw> should it handle that programmatically when raising an URLError?   I'm not certain we really need to do that or Azure should really care a much about Errno 101
<blackboxsw> Odd_Bloke: cadence is at will currently when we've decided there is enough content to warrant an SRU. Generally I'd like to keep it minimally at quarterly (or at least when shortly we cut a cloud-init upstream release).  The higher frequency the better
<dojordan> hmm, where does timeout fit into the picture
<dojordan> because in reality that is what is used in the happy path
<dpb1> blackboxsw: let's just simpmlify that... "after we release"?
<blackboxsw> Odd_Bloke: for the short term, Since we have a cut of cloud-init v.18.2 next week, I'd like to start an SRU of that to xenial within a week after the release.
<Odd_Bloke> blackboxsw: "the release" == the 18.2 release, or bionic?
<blackboxsw> +1 dpb1 ... /me is rambling a bit too much on that. so, post 18.2 cut next week, we'll kick off an SRU
<Odd_Bloke> OK, cool; I'm interested in https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1752391 ending up in xenial, and it sounds like we're on the path to that being true pretty soon.
<ubot5`> Ubuntu bug 1752391 in cloud-init "cloud-init does not recognize initramfs provided network config in all cases" [Medium,Fix committed]
<blackboxsw> Odd_Bloke: SRU to xenial after cloud-init 18.2 is cut next week. I'd estimate 2 weeks until SRU publish
<Odd_Bloke> Ack, thanks. :)
<blackboxsw> dojordan: I think the timeout you provide to readurl  is on the intial readurl request cap for when it raises a TimeoutException, it's passed into requests directly
<blackboxsw> so readurl will sit 1 second in your use case before raising that timeout exception
<blackboxsw> if IMDS is not functional for some reason
<dojordan> and then does timeout exception get wrapped in a urlerror? (sorry its been a little while since i wrote this code)
<dojordan> I believe it does
<blackboxsw> in testing, that's what it looks like happened. on my end. I setup a SimpleHTTPService  and killed it
<dojordan> I don't think cloud init needs to do too much special casing here - but maybe im wrong
<dojordan> we tried to communicate, and didn't get a response
<dojordan> vs got a response we don't like (5xx)
<blackboxsw> right only difference in the first case was that you are re-EphemeralDHCPv4'ing when you don't really need to.
<blackboxsw> but I think that's probably an unlikely corner case
<dojordan> except with a timeout exception we do need to re dhcp
<dojordan> which was the first case
<blackboxsw> ahh right, much concern about nothing then
<dojordan> but i agree, if we got a 101 we wouldn't necessarily need to re dhcp but it does'
<dojordan> t really harm anything  IMO
<blackboxsw> dojordan: yeah and generally I can't imagine what case would actually get you down the path of Errno 101, because that means you got a temporary dhcp lease and something managed to get there at the moment and ifdown your interface
<blackboxsw> .... not going to happen.
<dojordan> right, since systemd-networkd is not up yet
<blackboxsw> ok I'm good with this approach. will mark approved after I get one test run on EC2
<blackboxsw> thanks for your patience
<dojordan> sounds good, thanks!
<blackboxsw> smoser: rharper finally sorted my CI woes https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/341757
<rharper> dojordan: in other threads about using netlink, there is some mention of reducing as much downtime as possible; if we can avoid DHCP'ing when we don't need to; that will keep the waiting overhead to a min, no ?
<smoser> blackboxsw: rharper https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/341774
<smoser> i have to run now. blackboxsw  i can come back in in ~ 4 hours and upload anything that you've got landed.
<blackboxsw> +1 smoser thanks will have something up
<smoser> just approved your branch there.
<blackboxsw> just approved your Hetzner enable branch too smoser  https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/341679
<blackboxsw> smoser: Odd_Bloke put up https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/341778 for bionic GCE fix
<blackboxsw> that's to publish our fix which is in tip
<blackboxsw> if it lands tonight it should be in bionic cloudimages tomorrow morn
#cloud-init 2018-03-21
<smoser> blackboxsw: merged your branch uploaded to bionic
<blackboxsw> thanks smoser
<smoser> and /me is out.
<march__> Hello #cloud-init
<march__> I'm looking for a way to disable a specific service to start. Basically my service is not provisioned yet and I want to delay the service start after custom script execution
<march__> maybe with bootcmd and systemctl disable
<danmcd> I've a silly question, and I'm happy to take doc pointers as an answer.
<danmcd> I'm in charge of a DataSource.  I have something new:   "routes" to provide.
<danmcd> <prefix, dst, linklocal>
<danmcd> prefix and dst should be straightforward.  "linklocal" is what indicates if this is an "interface route".
<danmcd> If I turned those tuples into route(1M/8) commands, e.g.:
<danmcd> <10.1.2.0/24 10.1.1.1, false> ==> route add 10.1.2.0/24 10.1.1.1
<danmcd> <10.51.50.0/24, 10.1.1.5, true> ==> route add -interface 10.51.50.0/24 10.1.1.5
<danmcd> (And 10.1.1.5 damned well better be your local IP.)
<danmcd> How would a DataSource feed such information up to its consumer in cloud-init?
<rharper> danmcd: which datasource are you using ?
<danmcd> SmartOS.  I'm trying to modify it to turn the tuples I mention into something nice for cloud-init.
<rharper> ah
<danmcd> http://cloudinit.readthedocs.io/en/latest/topics/network-config-format-v1.html#route
<rharper> so smartos has a helper to convert the sdc:nics info
<danmcd> not sdc:nics.
<danmcd> sdc:routes
<danmcd> Which correspond, I think, to the cloud-init concept whose documentation I linked to above.
<rharper> there are two places to set routes in the v1 config;  there is the top-level type: route which has gateway,network;  or each subnet entry under an interface can accept a routes list
<rharper> those end up in /etc/network/interfaces as post-{up/down} route commands
<danmcd> No way to specify interface routes then?
<rharper> there is, if you hang the routes under the interface subnets configuration
<danmcd> Okay.
<danmcd> Wait..
<danmcd> Let me gist something...
<danmcd> https://gist.github.com/danmcd/15a9bdeadbfb1bfbc018fa293a1af26b
<danmcd> It's still not obvious whether or not I can add an interface route.
<danmcd> The pointed examples where I'm shouting.... it's not clear from the documentation that's allowed.
<rharper> we don't explicitly add the -interface, those stanzas result in a route add -net <network> gw  <gateway>
<danmcd> So if I want to add -interface routes, what do I do?
<rharper> we don't have a way to add the -interface
<danmcd> THere's something about outputting scripts... do I just hack that in there?
<rharper> you can certainly have a script run that
<danmcd> Okay.  THank you.
<danmcd> For your time & patience.  :)
<rharper> np, on the -interface vs. the -net gw  ; does that generate a different static route ?
<danmcd> Yes.
<danmcd> -interface is essentially:  "Hey, ARP these destinations".
<danmcd> Or "NDP these" if it's IPv6.
<danmcd> You can't have both with a matching prefix, it's one or the other.
<rharper> danmcd: hrm, ok;  I need to read more;
<danmcd> I hear you...
<rharper> I suspect we can file a bug to track this is missing;
<danmcd> Sure.
<rharper> I think this might be a BSD thing in route?  net-tools on linux route doesn't have a -interface parameter
<danmcd> -interface originates from 4.3BSD yes...
<rharper> danmcd: you might try the syntax you gist'ed and see what route you end up with;  the gw value is used to determine which interface to us, the man page mentions this is a BSDism ;  so I wonder if that's equivalent ?
<danmcd> Maybe.
<danmcd> Finding out locally.  Thank you.
<rharper> yw
<danmcd> Yes, BTW, "route add -net <pfx> -gw <my-local-ip>" is like "route add -interfaces <pfx> <my-local-ip>" on BSD and illumos.
<smoser> blackboxsw: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/341851
<smoser> can you give that a review ?
<blackboxsw> np smoser
<AscII> cloud-init generates a strange/broken netplan config if the ipv6 gateway is a link local address
<AscII> https://paste.ubuntu.com/p/M7Yd4vvSS5/
<AscII> if I remove the routes section, everything works as expected
<AscII> this route section also triggers the following warning
<AscII>  (generate:666): GLib-WARNING **: 18:19:49.916: GError set over the top of a previous GError or uninitialized memory.
<AscII> in case of v1, I get
<AscII>  post-up route add -net :: netmask 0 gw fe80::1%eth0 || true
<AscII> which doesn't hur
<AscII> t
<AscII> Any idea how to disable these extra routes?
<AscII> otherwise the newly added Hetzner datasource works in the official bionic images
<AscII> (which seem to have a broken protective MBR)
<AscII> smoser: thanks for the merge, btw
<smoser> AscII: /me is embarassed though
<smoser> i stole your credit with our merge tool
<smoser> :-(
<smoser> i'm apt to stuff a commit message in with your name proper.
<smoser> the squash took the top-most author
<AscII> oh well
<smoser> AscII: can you give the input ?
<smoser> AscII: the only good thing about someone steeling your credit is that they get blame if theere are bugs :)
<smoser> i'm really sorry that happened though.
<AscII> hehe
<AscII> no worries
<smoser> the input, what was the input config... ie, did cloud-init  mangle the input to break things ? or is netplan just doing the wrong thing .
<AscII> no mangling. The config is just wrong
<AscII> The routes shouldn't be there
<AscII> they are not pushed
<AscII> net/netplan.py
<blackboxsw> fwiw, we got credit right on the debian/changelog for Markus :)
<smoser> yeah, blackboxsw saves the day
<AscII> hehe
<blackboxsw> after breaking it. shame on review-mps tool
<AscII> so L114 in net/netplan.py generates routes for the subnet
<AscII> always
<AscII> no matter what
<blackboxsw> hrm if routes is unspecified on the subnet it should be empty list and remain unset  right?
<AscII> yes, something like that
<AscII> net/eni.py does a the same thing for v1 in L361
<AscII> but that results in the no-op 'post-up route add -net :: netmask 0..'
<AscII> well, actually this command is wrong as well, but the added '|| true' prevents any problems
<AscII> there is a comment above that section in net/eni.py that says:  We may at somepoint not want to emit this additional postfix
<AscII> ah, that relates to the ||true
<AscII> hmm
<blackboxsw> AscII: sry I'm still trying to get the rest of the context on this. Do you have /var/log/cloud-init.log or a tarfile produced by 'cloud-init collect-logs'
<blackboxsw> I'm specifically wondering about the log message Applying network configuration from...
<AscII> sure
<blackboxsw> which should contain the netconfig we base rendering on
<blackboxsw> which may or may not have the 'routes' incorrectly specified
<AscII> https://paste.ubuntu.com/p/RVrBH9YqBb/
<blackboxsw> AscII: ok so ilne 89 in your paste shows that we are feeding in 'incorrect' route information to the netplan generator
<blackboxsw> I'm trying to grok where that comes from
<blackboxsw> AscII: it looks like it comes from HetznerCloud datasource.metadata['network-config']
<blackboxsw> cat /run/cloud-init/instance-data.json will show you the metadata harvested on the cloud
<blackboxsw> under 'ds':'meta-data' keys
<AscII> hmm, that file is not present (anymore). Possibly because I corrected the generated netplan yaml
<blackboxsw> smoser: I posted comment/question on https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/341851
<blackboxsw> smoser: simpoir: I finally got ci to like me on https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/341543  turns out you actually have to 'git add' a new module that you've coded up otherwise those silly import errors. Â¯\_(ã)_/Â¯
<AscII> blackboxsw: ah, crap. My bad. It seems we actually have these routes in the metadata
<blackboxsw> AscII: I like to pass the blame around ;)
<AscII> Need to check with my team why. I guess some other distro needed it
<blackboxsw> good deal.
<simpoir> blackboxsw: I just saw. I having a second look at it.
<blackboxsw> simpoir: I'm on your landscape branch atm
<blackboxsw> it will happen today ;()
<simpoir> ð
<AscII> well, it looks like rhel/fedora is to blame
<AscII> they add the route as gateway
<AscII> IPV6_DEFAULTGW=fe80::1%eth0
<AscII> and I think, we couldn't directly pass the interface in the gateway
<AscII> since the gateway is not in the subnet (L346) it doesn't get rendered there, but later during routes (L392)
<AscII> great
<AscII> so, the only way to work around this, would be to render the meta-data with routes to rhel/centos/fedora and without to debian/ubuntu, I guess
<AscII> At least for now
<papertigers> rharper: are routes not picked up on ever reboot?
<rharper> papertigers: not following, could you provide some more context ?
<rharper> papertigers: are you manually modifying network configuration after cloud-init has generated one for you?
<papertigers> following up on what danmcd was asking about earlier. It seems only certain things are setup again on reboot.  For example I had to run 'cloud-init clean' to get DNS resolvers to change in systemd on this box
<danmcd> But my newly-parsed-by-DataSourceSmartOS.py "route" doesn't seem to work.
<danmcd> Here's some json output I got from the logfile, including teased-out to show things clearly:https://gist.github.com/danmcd/7d1ae639d7877e760a47ad83489a342f
<rharper> network configuration is rendered per-instance;, so a reboot won't modify it; if you're adding additional configuration manually via a script, that would need to be marked per-always rather than per-instance
<rharper> dns resolves should be encoded in /etc/network/interfaces.d/50-cloud-init.cfg
<rharper> in which case they survive reboots on the same instance
<rharper> danmcd: yeah, the nameserver should be under the loopback interface in the eni file I refernced, resolvconf reads eni config and updates /etc/resolv.conf which is symlink to /run/resolvconf/resolcv.conf maintained by the resolvconf program
<danmcd> So nameserver can apply with "cloud-init clean" and a reboot.
<danmcd> I'm wondering how I can tell why my "route" directive isn't being acted upon.
<danmcd> Sorry if papertigers didn't make that clear.
<danmcd> (See the gist above.)
<rharper> can you paste your /etc/network/interfaces.d/50-cloud-init.cfg ? or if cloud-init collect-logs and share that?
<rharper> from the yaml in your gist, it should render this:  https://paste.ubuntu.com/p/RDbq9qsFJ9/
<papertigers> I dont seem to have a at /etc/network/interfaces.d/50-cloud-init.cfg
<rharper> are you booting ubuntu ?
<danmcd> "cloud-init collect-logs" barfs
<rharper> if so , what release? Xenial
<papertigers> Ubuntu 17.10
<rharper> ah, netplan
<rharper> fun
<rharper> let me rephrase the output
<rharper> so, /etc/netplan/50-cloud-init.yaml is what youll have,  the resolvers should show up under systemd-resolve --status;  the static routes appear to be dropped;  =/
<blackboxsw> danmcd: I'd also be curious about the collect-logs barfs comment, as that script doesn't do much except tar files on the system and run journalctl
<rharper> we'll need to refactor the network  conversion to put the sdc:routes data under the network interfaces
<blackboxsw> ahh but it definitely needs a fix for non-deb systems as it leverages dpkg-query
<danmcd> it's missing routes...
<danmcd> So do I need to do any more work in DataSourceSmartOS.py?  Or is this a consumer-of-that's problem?
<danmcd> blackboxsw: hang on...
<danmcd> blackboxsw: https://gist.github.com/danmcd/577325259016909c33fd8e18f01cb1c2
<blackboxsw> ahh sudo cloud-init collect-logs danmcd
<danmcd> DOH!
<blackboxsw> the instance-data.json is protected root readonly
<danmcd> silent.
<blackboxsw> :)
<rharper> danmcd: yes, we need to fix the converter to put the routes under the interface
<danmcd> Phew!
<rharper> danmcd: one sec, lemme paste
<rharper> http://paste.ubuntu.com/p/ScdyqwMq6z/
<danmcd> I'm a newb in this arena, so thanks for your patience.
<rharper> no worries, we'll file a bug and it should be an easy adjustment
<rharper> with the change, then the netplan config file looks like this: http://paste.ubuntu.com/p/htRC3hcK5m/
<rharper> which will include the routes
<danmcd> So what you pasted... that is something I'll have to fix.
<rharper> danmcd: I'll file the bug, and we just need to stuff the sdc:routes data in a different location
<blackboxsw> sorry danmcd on the silent run of collect-logs, smoser asked me to add a print saying 'Wrote cloud-init.tar.gz' in your cwd.
<rharper> the subnet we generate, can accept a 'routes' array
<danmcd> Since now I hang the routes as their own same-level-as-physical.  But according to your pastebin, I'll have to put it inside the physical "object".
<danmcd> right?
<blackboxsw> I forgot to add that output to the collect-logs cmd. But anyway not a problem you care about at the moment
<rharper> yeah, under the subnets for the interface
<danmcd> Okay, so that is my problem.  Since I'm fixing sdc:routes anyway.
<danmcd> BTW, since sdc:routes is independent of interface, I'm going to have to collect them, and then hang them under all interfaces.
<danmcd> Which shouldn't be a problem,. even for multi-homed?
<rharper> yeah, we struggled with the routes
<danmcd> Okay...
<rharper> in a sense a  route has to go out *one* interface
<rharper> so, for the netplan format, the routes are stuffed under interfaces
<danmcd> Oh, this is a netplan thing?
<rharper> yeah, 17.10+ has netplan
<danmcd> (Sorry, I come from the illumos TCP/IP stack...)
<danmcd> Okay, I see.  It'll be more challenging, but at least I understand WTF is going on now.  You've been very helpful.  Thanks for your time & patience.
<rharper> np
<papertigers> if we do this work for netplan, does that break anything below 17.10?
<rharper> papertigers: no, v1 renders the global routes correctly already
<rharper> 16.04 should work as-is with the config that danmcd posted in his gist
<danmcd> I'm trying brute-force... putting it in both places.
<danmcd> Isn't the idea it works in one form and deploys everywhere, though?
<rharper> that may cause issues on 16.04
<rharper> yes, cloud-init converts the v1 into internal network state, then renders to the distro's format (eni, netplan or sysconfig)
<papertigers> rharper: and my question earlier was basically after the yml file or interfaces file gets written out to disk on first boot it doesn't try to set things up again unless you run 'cloud-init clean'?
<danmcd> So putting "routes" as a peer of "physical" should not be done?
<rharper> what we're hitting is that netplan doesn't handle "global" static routes per the design choice that routes should be bound to at least one interface, since they have to egress one anyhow
<rharper> papertigers: so on 17.10, after your first boot, run systemd-resolve --status
<danmcd> That seems to be a problem beyond just DataSourceSmartOS, right?
<rharper> you should see your DNS enatries there
<papertigers> rharper: yeah I know that works.  I am saying if the data in sdc:resolvers changes, cloud-init wont attempt to change that unless you run clean. Just want to confirm behavior
<rharper> papertigers: cloud-init writes out either an eni on 16.04, or netplan on 17.10 before the "networking" layer starts, then ifupdown or networkd brings the config up
<rharper> so you shouldn't need to do anything to have all of your network config applied
<rharper> papertigers: yes, we don't yet have dynamic network config changes
<rharper> that's something we're actively discussing
<papertigers> cool, thanks.  That would be super useful to us if the discussions is a matter of if and not when
<rharper> https://hackmd.io/M1Tae41PQBC7a9qMsurTJw
<rharper> that was one of the discussion we've had with the community
<rharper> we're very interested in feedback , especially from datasource folks
<rharper> papertigers: danmcd: I've got to step our for bit; feel free to leave comments, issues, I'll see them later tonight, and if not then tomorrow
<danmcd> RIght now, I generate multiple "routes" entries.
<danmcd> I'll find out the hard way if that's a real problem or not.
<danmcd> Thanks rharper
#cloud-init 2018-03-22
<smoser> blackboxsw: responded to the open-telekom branch
<blackboxsw> smoser: I'm going to land the open-telekom branch (wanted to test review-mps getting the authorship correct there)
<smoser> blackboxsw: ok. great. thanks.
<blackboxsw> smoser: is this the comment you wanted to see on bugs when commits happen in tip, or something a bit different? https://bugs.launchpad.net/cloud-init/+bug/1756471/comments/6
<ubot5> Ubuntu bug 1756471 in cloud-init "ds-identify does not identify openstack Open Telecom Cloud" [Medium,Fix committed]
<blackboxsw> when we release to bionic I'd expect a comment along the lines of "cloud-init v.X released in Bionic, if this is still an issue, re-open the bug or create a new one"
<blackboxsw> anyway it's late. can chat tomorrow about it
<smoser> blackboxsw: yes, that is great.
<blackboxsw> AscII: if you're around today rharper/smoser was peeking at your metadata network config and the thought was that your metadata service should actually not be presenting the aliases eth0:0 interface name, but it could fold the subnet defined under eth0:0 under a single interface 'eth0'
<blackboxsw> AscII: lines 23-32 should live @ line 17 http://paste.ubuntu.com/p/PDjVZTM9sp/
<blackboxsw> rharper: I'm tweaking your commit message and landing https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/341662 review-mps balked at line length.. sorry about the mp comment noise on that
<rharper> thanks
<danMS_> hey scott, https://git.launchpad.net/cloud-init/tree/cloudinit/sources/DataSourceAzure.py#n606, can we make this âNTFS â one fileâ check failure ignorable, because if the OS doesnât know how to do NTFS, the contents *canât* be important?
<AscII> blackboxsw: thx for the input. If I remember correctly this is again a workaround for rhel/fedora
<rharper> well, in general the v1 config format doesn't regonize an alias interface as a physical device
<smoser> danMS_: you're saying that line 642 could return True, "you probably didnt have anything there because your kernel couldnt mount ntfs"
<smoser> right ?
<smoser> it is probably reasonably safe, but could obviously false positive and destroy data.
<smoser> if you booted into a new kernel and had forgot some module s or something.
<smoser> blackboxsw: https://code.launchpad.net/~kgarloff/cloud-init/+git/cloud-init/+merge/341844
<smoser> that one is good.
<blackboxsw> smoser: good like you wanted it landed or reviewed?
<smoser> please give it some thought.
<smoser> i pointed c-i bot at it
<smoser> but it looked logically good ot me
<blackboxsw> roger, it affects dojordans Azure branch I think
<blackboxsw> as he's now using readurl w/ exception_cb too
<blackboxsw> will take it into account.
<smoser> blackboxsw: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/341543
<smoser> Simon pointed out you seemed to have dropped the TestPrependBaseCommand tests.
<blackboxsw> yes, I finished testing that this morning.
<smoser> oh. ok.
<smoser> you didnt respond ot that
<blackboxsw> checking the dropped basecommand test... I moved it into cloudinit/tests/test_subp.py
<blackboxsw> only question I have smoser is the order in which I'm running ubuntu-advantage module in cloud.cfg.tmpl
<blackboxsw> I put it after the apt pipelining and apt configure  modules as they might tweak apt a bit prior to ubuntu-advantage installing the deb
<smoser>  yeah.
<blackboxsw> if you had any thoughts there it feels a bit heavy to order an {% if variant in ["ubuntu"] %} just after a section that was ubuntu|debian|unknown
<smoser> those couldprobably move earlier... they just set configuration. but you definitely want to be after them if you're going to install packags.
<smoser> you have a merge conflict showing right now there.
<smoser> i think
<smoser> look for <<<<
<smoser> i think its fine as you have it.
<smoser> "heavy" doesnt matter really there... its build time.
<smoser> if thats what you meant
<blackboxsw> smoser: if I git merge cs/feature/ubuntu-advantage-module against tip of master checked out locally I get no conflicts
<smoser> ah. so lp is just out of sorts
<blackboxsw> yeah, it just felt like I'm being ubuntu-explicit in the template just after a more general ubuntu, debian, unknown section felt like it might have been overkill, but as you said, template build happens once, so no biggie
<blackboxsw> and we don't want ubuntu-advantage to show of in cloud.cfg anywhere else
<blackboxsw> and we don't want ubuntu-advantage to show up in cloud.cfg anywhere else
<blackboxsw> smoser: ahh I forgot to add cloudinit/tests/test_subp.py
<blackboxsw> that's why simpoir suggested I dropped it without replace.  pushing now
<blackboxsw> man it's a bit confusing in that wait_for_url's exception_cb is expected to behave differently than readurl's exception_cb
<blackboxsw> but anyway, doesn't need to be aligned with this branch
#cloud-init 2018-03-23
<kalyan> hi all ... i am facing some issue with cloud init on suse.... is anyone available now to help me? thanks in advance !
<kalyan>  i have installed cloud-init version 0.7.6 ... and i am seeing cloud-init-local or config status as unused
<kalyan> could someone please help/correct me if i missed something..thanks
<smoser> kalyan: cloud-init-local does need to run.
<kalyan> smoser: Thanks, but when i tried starting it ..i am seeing the message "failed"
<kalyan> smoser: just curious.. so it will be in "unused" state?
<smoser> what is the init system ?
<kalyan> cloud init 0.7.6 on suse
<kalyan> suse 11
<smoser> kalyan: the init system.. sorry i'm not familiar enough to know
<smoser> upstart ? systemd? sysvinit ?
<smoser> you'll have a file on your system:
<smoser>  /etc/cloud/cloud.cfg.d/05_logging.cfg
<smoser> you proably have a line like
<smoser>  - [ *log_base, *log_syslog ]
<smoser> in it.
<smoser> comment that line out
<smoser> then re-run and pastebin /var/log/cloud-init.log
<kalyan> ok sure
<do3meli> hey smoser. do you have some time to look at some code in https://code.launchpad.net/~d-info-e/cloud-init/+git/cloud-init/+merge/340220 ?
<do3meli> i am struggeling to find a nice way to easy parse a file that can have 2 different layouts.
<do3meli> while writing tests for this branch i realized that the output of "mount" command can be different - so we have to take care of both layouts
<smoser> do3meli: sure
<smoser> do3meli: well i guess the one thing is to have some intermediate object there.
<smoser> def parse_mounts():
<smoser>   """Return a representation of mounts on the system."
<smoser>   if util.isFreeBSD():
<smoser>       return one thing
<smoser>     else
<smoser>        another
<smoser> then use that thing
<smoser> but maybe that doesnt help you
<smoser> unrelated: dont bother passing the LOG thing around . i just didn't really understand python logging when i put that stuff in (handle taking a 'log=' param)
<smoser> just use LOG.debug or LOG.warning
<smoser> oh. i see so we do have 'parse_mount' . and that returns a dev path, fstype and mount point.
<do3meli> ahh did not know that. my pycharm always told me its not so nice to use a variable that has not been passed.
<do3meli> i may can get rid of get_mount_parse_regex() i am just about to put together a better regex that makes it obsolete
<do3meli> i have this regex now that matchs both formats:
<do3meli> https://regex101.com/r/2F6c1k/1 and the other format https://regex101.com/r/T2en7a/1
<do3meli> the problem now is that the group id's are differnet. for the first one the fs_type is group id 4 and for the second one its group id 3.
<smoser> do3meli: you can do names in your regex
<smoser> and reference them by name
<smoser> do3meli: (?P<name>...)
<smoser> https://docs.python.org/3.4/library/re.html
<rharper> groupdict() FTW
<do3meli> let me try that. this would make things much easier ;)
<do3meli> not sure that works in combination with a positive lookahead
<do3meli> or the other way around: having the if and else clause the same group name does not work. it says "name must be uniqe"
<do3meli> see this here: https://regex101.com/r/2F6c1k/2
<smoser> do3meli: thats neat. i'd nto seen that.
<do3meli> i think i found a solution in the meanwhile smoser
<do3meli> refactoring that whole stuff now ;-)
<smoser> great. thanks.
<smoser> do3meli we really apprecate your contributions
<do3meli> thanks :-)
<do3meli> saw the monday protocol btw. was not able to attend unfortunately. and most likely can't attend the next one too
<smoser> blackboxsw: https://code.launchpad.net/~kgarloff/cloud-init/+git/cloud-init/+merge/341844
<blackboxsw> smoser: just saw you merged it
<smoser> merged that with
<blackboxsw> was bringing that up. will respond on dojordan's branch
<smoser>  review-mps -v --project-name=cloud-init --git-user=smoser --merge
<smoser> but the related bugs did not get marked fix-committed
<blackboxsw> ahh, something to tackle today. I'll re-run it and make sure the bugs get marked by review-mpos
<blackboxsw> ahh, something to tackle today. I'll re-run it and make sure the bugs get marked by review-mps
<blackboxsw> got it.
<blackboxsw> running now and pushing the regex improvemnt. needed to match \s* at end of line not \s+
<blackboxsw> dojordan: welcome, sorry for the delay, I'm posting a minor round of  review comment to your azuretimeout branch and testing it today. something landed in trunk today that affects the exception_cb passed to readurl.
<blackboxsw> .. and sorry for IRC pouncing
<dojordan> no worries, its why im here!
<blackboxsw> I'll have a patch suggestion for you shortly to see what you think. unless you beat me to it after you git fetch; git rebase master
<blackboxsw> basically exception_cb to readurl needs to return True to retry
<blackboxsw> if it returns False the current exception gets raised
<dojordan> yup just found that
<blackboxsw> basically this I'm thinking //paste.ubuntu.com/p/qPd7fTp7rz/
<blackboxsw> but wanted to test it
<dojordan> yup, my thoughts exactly
<blackboxsw> ok good deal.
<blackboxsw> I'm running through a deployment of that + tip + your branch now should clear out on your branch then in the next hour
<dojordan> want me to push that patch?
<blackboxsw> +1 dojordan it'd save me time
<blackboxsw> after a git rebase
<dojordan> sure, just running unit tests first
<blackboxsw> just to include Kurt's changes. thanks
<dojordan> hmm, getting a slow test: tests.unittests.test_ec2_util.TestEc2Util.test_userdata_fetch_fail_server_not_found 5.0287s . Not a chance my change of AzureDataSource affects this but just FYI
<smoser> hm... yeah, i was just seeing that oto. will investigate, dojordan
<blackboxsw> ahh good find, that tends to mean a missed requests mock
<smoser> thanks
<blackboxsw> or a readurl/wait_for_url mis-mocked
<dojordan> @blackboxsw pushed
<blackboxsw> +1 I'm testing live now on a xenial instance w/ your patch ++ the minor pastbin
<dojordan> sounds good
<blackboxsw> BTW we are going to change the logic of exception_cb passed to readurl in a subsequent branch (post 18.2 release). Now exception_cb will be expected to raise an exception if warranted and ignore if retries are expected. It's more intuitive than this return True|False stuff (as you have to go re-read readurl implementation to see what expected behavior is)
<blackboxsw> that change will then align more with wait_for_url behavior too
<blackboxsw> and we'll fix this callsite for readurl + openstack and scaleway datasources
<blackboxsw> which also use exception-cb
<dojordan> great, i'll keep my eye out for that change
<dojordan> thanks for the warning
<blackboxsw> dojordan: landing it, both failure modes (404 versus service not present work great) thanks for the bump
<dojordan> great, thanks for the help!
<smoser> blackboxsw: 2 reviews please. quick-ish.
<smoser>  https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/341995
<smoser>  https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/341997
<smoser> second is (i think) not objectionable in any way
<smoser> blackboxsw: ping
<blackboxsw> smoser: +1 on the second and will get through the first. didn't want to report back til finished :/
<powersj> blackboxsw: smoser: thoughts on https://bugs.launchpad.net/cloud-init/+bug/1758409 when you have a sec
<ubot5> Ubuntu bug 1758409 in cloud-init "integration tests: restructure ssh timeout " [Undecided,New]
 * blackboxsw removes python-jsonschema deb from my env as it should have failed for me locally 
<smoser> blackboxsw: i was pinging becauase of https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/341998
<smoser> which dojordan pointed out.
<smoser> if we wanted to touch that like this... or just go for the bigger fix
 * blackboxsw looks over read_file_or_url implemetation
<blackboxsw> dang, ok missed the third caller
<blackboxsw> hrm. I think we should just go for the bigger fix since we have context at the moment. it shouldn't be that big. OpenStack. Scaleway, Ec2 and Azure. Then we don't have to revist risk in 18.3 SRU validation too
<blackboxsw> smoser: well, actually, I have to get through your IBMcloud run too and I'd like us to make a bionic cut today including it if we can
<smoser> blackboxsw: sure.
<smoser> i can work on b igger fix
<blackboxsw> so, I wouldn't be able to write up that branch. :/ so don't want to force you to do it
<smoser> but we can just as well take the small one  now.
<blackboxsw> yeah +1
<blackboxsw> will approve
<smoser> note that https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/341995
<smoser> made jenkins catch the failure
<smoser> blackboxsw: just becausae youre bored...
<smoser>  https://code.launchpad.net/~dojordan/cloud-init/+git/cloud-init/+merge/340546
<smoser> that has a comment on it with a bad hash
<smoser> (ie, click the link)
<dojordan> @blackboxsw, random question. Any reason why the upstream commit https://git.launchpad.net/cloud-init/commit/?id=2d618e27 isn't showing up?
<dojordan> haha
<smoser> :)
<dojordan> i had that typed out but assumed it was a stale cache or something
<blackboxsw> interesting.... hmm, checking how the lander got that hash
<blackboxsw> hrm hold up.
<blackboxsw> dojordan: smoser ok fixed. I had the lander sitting at pdb
<blackboxsw> it did all the launchpad work, just didn't git push
<dojordan> hahaha
<dojordan> nice
<blackboxsw> smoser: minor tweak and then approved https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/341997
<smoser> blackboxsw: maybe it wasnt clear to you why you had jsonschema in your tox env.
<smoser> its in requirements.txt
<smoser> so it gets put in there by setup.py
<smoser> so we could either deal with "optional" things in a generic way, which woudl be fine  or like this.
<blackboxsw> smoser: yeah, I just generally had performed a pip remove of it when testing new schema additions, I forgot about that when I put up test_snap and test_ubuntu_advantage
<smoser> but for the moment thats the only optional
<smoser> this way C-I will catch it for you.
<blackboxsw> hence the tests which passed locally, (because or requirements.txt)
<blackboxsw> yeah
<blackboxsw> I like that
<blackboxsw> smoser: and oauthlib right?
<blackboxsw> as for optional requirements
<smoser> and pyserial :)
<blackboxsw> ohh right yep
<smoser> so we could do something more generic
<smoser> but the schema is the most invasive
<smoser> the others are very optional paths
<blackboxsw> yeah  I think your approach in tox -e xenial. it at least asserts we are exercising optional paths without the dependency
<blackboxsw> we could come up with something a bit more generic if we have more cases I guess.
<blackboxsw> can you rebase and push https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/341995 so we get the pretty ci green
<blackboxsw> now that you've landed the prior branch
<smoser> done
<smoser> blackboxsw: wow. MAAS comopatibiltiy testing takes 11 minutes now
<smoser> i swear it wasnt that long before
<smoser> powersj: ^
<smoser> https://jenkins.ubuntu.com/server/job/cloud-init-ci/
<smoser> something got slower... but of course i have to data to that.
<powersj> that's about right
<powersj> that was half the ci time
<smoser> well, it looks like its more 2/3 now
<smoser> but ohw ell
<powersj> the other times, for reasons unknown to me, seem to have gone down
<powersj> I know blackboxsw and I found a number of test escapes last fall and fixed them
<powersj> The build and the integration tests specifically
<blackboxsw> right, yeah invalid mocks w/ httpretty, but yeah I don't know what else would have been improved
<powersj> I would like to consider moving the maas compat back out and run it only nightly on trunk
<smoser> blackboxsw: c-i happy now
<blackboxsw> then bbsw is happy
<smoser> and i updated the commit message
<blackboxsw> landed
<blackboxsw> IBMcloud next
<blackboxsw> then chrony/ntp
<blackboxsw> minor comments on https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/341774ConfigDriveReader
<blackboxsw> sans the ConfigDriveReader
<smoser> blackboxsw: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/342007
<smoser> thats not all the way done.. but
<smoser> blackboxsw: i  had thought the .json thing too...
<smoser> i dont know.
<blackboxsw> smoser: yeah I just have a hard time looking over 3 item tuples instead of two. take it or leave it not critical by any means
<smoser> blackboxsw: i really have to go now.
<smoser> i'd like to have that in too, bjut just out oftime
<smoser> if you want you can take it and just finish it.
<smoser> i'm really 30  minutes over :-(
<blackboxsw> +1 see ya.
<blackboxsw> have a good weekend
<smoser> i will pop backc in if you have aMP for upload
<blackboxsw> +1 will have one
<smoser> just because that is "easy"
<smoser> later
<rharper> hrm, all of the reword of the url reading makes me a little nervous, post 18.2, right ?
<blackboxsw> rharper: your comment on the ibmcloud branch.... blocker for release or tech-debt we can pickup as a separate branch?
<blackboxsw> Shouldn't we update util.find_devs_with() to make use of this? or some other refactoring
<blackboxsw> since we now have to callers to blkid?
<rharper> not a blocker
<rharper> I understand the two modes
<rharper> but it is tech debt IMO
<blackboxsw> I see your point, lots of commonality with find_devs_with. it seems like a good candidate for consolidation though
<rharper> find_devs should be implemented with util.blkid() however, util.blkid needs to change to accept the parameters that are in-use by find_devs_with()
<rharper> so, that'll be a bit of refactoring
<blackboxsw> agreed
<blackboxsw> your comment about the reword of url reading is related to this one right? https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/342007
<rharper> well, there are like at least two no? the raise exception, and then the follow up ?
<blackboxsw> yeah we've landed the prior already which was to invest the readurl(exception_cb) logic to continue retries when exception_cb returns True. False == raise
<rharper> blackboxsw: the invert and then consistent
<blackboxsw> rharper: right those have both landed, invert and consistent across 4 datasources
<rharper> =/
<rharper> ok
<blackboxsw> yeah, it was inconsistent for OpenstackDatasource for months. resulting in long retries on 404s
<rharper> I should have spoke sooner; those sort of changes seemed post 18.2 release, but I guess we'll just need to poke on them
<rharper> Openstack 404ed in what scenario ?
<blackboxsw> https://bugs.launchpad.net/cloud-init/+bug/1702160
<ubot5> Ubuntu bug 1702160 in cloud-init "OpenStack datasource should not retry user-data on 404" [Medium,Fix committed]
<blackboxsw> when no user-data is provided, it'd retry to exhaustion
<blackboxsw> instead of bailing and continuing
<rharper> lol
<rharper> why wouldn't the file be empty ?
<blackboxsw> and finally smoser's last attempt is to change behavior of exception_cb to actually raise if needed, instead of expecting that readurl will do the raise. only reason I thought about getting that 'in' is because we are already exposed on the exception_cb changed behavior front
<blackboxsw> so might as well only cross that changed-behavior on 4 clouds once.
<rharper> y
<rharper> it all makes sense, just wish it was post 18.2; we have one more release before 18.04 anyhow; but I guess for bionic, at least, sooner is better
<rharper> it does worry me re: SRU though
<blackboxsw> +1, yeah we'll have to manually exercise upgrade path and fresh install on Openstack too (we already handle Azure/ec2). Not sure what to do about Scaleway though
<blackboxsw> I'll give ec2 a run right now with tip before we cut an MP for publish to bionic (just to make sure the system hasn't been killed)
<rharper> y
<blackboxsw> just tested on EC2 with tip of smoser's branch.
<blackboxsw> I'm awaiting a jenkins integration test then I'm merging and proposing a bionic release
<blackboxsw> https://jenkins.ubuntu.com/server/view/cloud-init/job/cloud-init-ci/922/
<blackboxsw> tested on azure earlier today with basically the same content (regarding the exception_cb behavior changes as do-jordan's branch had to account for that)
<blackboxsw> smoser: it's up for review
<blackboxsw> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/342011
<blackboxsw> if you wanted to enable IBMCloud in debian cloud.cfg.tmpl go4it
<blackboxsw> :)
#cloud-init 2018-03-24
<smoser> blackboxsw: missed this https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/342016
<smoser> but since i added it to ubuntu default it wont have any effect
<blackboxsw> smoser: reviewing, ci claims failure on   File "/var/lib/jenkins/slaves/torkoal/workspace/cloud-init-ci/tests/unittests/test_datasource/test_common.py", line 68, in test_expected_default_local_sources_found
<blackboxsw> https://jenkins.ubuntu.com/server/job/cloud-init-ci/925/console
<blackboxsw> smoser: apply the following and we're good http://paste.ubuntu.com/p/vTw6tfcn3d/
<blackboxsw> or I can
<blackboxsw> and land it
<blackboxsw> took your branch over, awaiting CI and I'll propose a bionic mp which will include it
<blackboxsw> https://jenkins.ubuntu.com/server/view/cloud-init/job/cloud-init-ci/926/
<blackboxsw> smoser: though you should be EOW :) https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/342021
<blackboxsw> I'm officially out
#cloud-init 2020-03-17
<ongy> Hi, quick question (Assuming runcmd: thingy in lxd config uses cloud-init): do the runcmds get executed on each boot/container start, or only once in some init phase during firstboot?
<ongy> hm, looking into the FAQ there's some boot-done filemark, so I guess the init-phase
<Odd_Bloke> ongy: runcmd runs once per instance (i.e. first boot), yeah.  If you want to run a script every boot, you can put it in /var/lib/cloud/scripts/per-boot/ (and ensure it's executable).
<Odd_Bloke> (And you could do that with write_files: https://cloudinit.readthedocs.io/en/latest/topics/modules.html#write-files)
<ongy> ok, thanks
<ongy> ahh, the per-boot might help as well at some point
<ongy> My main issue was the pulse lxd profile not working if I attach afterwards, because the sed that patches the config didn't trigger
<blackboxsw> Odd_Bloke: one question for you before I can wrap up ec2 multi-nic/ip support https://github.com/canonical/cloud-init/pull/114#discussion_r393921933
<blackboxsw> think I'll have the rest of the review comments addressed
<blackboxsw> and will push once we resolve the above
<blackboxsw> Thanks again for the review Odd_Bloke I've addressed all comments and force pushed https://github.com/canonical/cloud-init/pull/114. Just ran networking tests on upgrade path, dual nic and single nic configs to show route-metric is only presented on multi-nic VMs. It's ready for review tomorrow
#cloud-init 2020-03-18
<Odd_Bloke> blackboxsw: So I'm going to look through in more detail in a few minutes, but I think the key unresolved issue on #114 is the naming of the config option.
<Odd_Bloke> blackboxsw: I can't decide whether consistency with the other DSes or more accurate naming for this DS is more desirable.
<Odd_Bloke> (Though this is something of a false dichotomy: we could always rename those other config options too, while retaining support for the old spellings.)
<paride> blackboxsw, this is one: https://github.com/canonical/cloud-init/pull/254/files
<blackboxsw> thanks paride
<blackboxsw> +1 Odd_Bloke
<powersj> paride, what failed that required that change?
<paride> powersj, the nocloud-kvm tests: https://paste.ubuntu.com/p/XQ6Q8BHxXD/
<powersj> paride, shouldn't this have been failing for years then?
<powersj> why now?
<paride> powersj, I was expecting this question :) And I don't have a definitive answer. Let me dig a bit more.
<paride> powersj, because the failing assert is new: https://github.com/canonical/cloud-init/commit/71af48df3514ca831c90b77dc71ba0a121dec401#diff-27f8cf430e53c95119b64a768e67e6e4R323
<paride> added a comment to the PR
<blackboxsw> +1. yeah paride it was my bad :/
<blackboxsw> and merged :)
<blackboxsw> yes because our cloud tests were contructing the version from yaml it interprets 19.10 as 19.1 float, so text comparisons in that case would turn out invalid.
<paride> blackboxsw, but it's yaml itself that has the concept of floats and strings
<paride> so I don't think it was a fault in your commit at all
<blackboxsw> paride: right, just at fault was that I only tested on *.04 series before landing :)
<blackboxsw> instead of *.10 :)
<blackboxsw> Odd_Bloke: how does the apply_secondary_network_config sound for ec2 multi-nic-secondary-ip PR #114? really, I'm up for any suggestion you feel is more tractable.
<blackboxsw> or understandable
<Odd_Bloke> blackboxsw: I liked the wording you changed to internally, perhaps `apply_full_imds_network_config`?
<Odd_Bloke> blackboxsw: FYI, I've opened up a few small PRs which should only take a couple of minutes to review: https://github.com/canonical/cloud-init/pull/255 https://github.com/canonical/cloud-init/pull/257 https://github.com/canonical/cloud-init/pull/258
<Odd_Bloke> In case you're looking for another distraction. ;)
<blackboxsw> Odd_Bloke: thanks, renamed ds cfg for  ec2 #114 and pushed. awaiting CI
<blackboxsw> will grab your reviews now
<garga> Hi! I have a question.  I'm using cloud-init to deploy a CentOS 7 based VM at Azure and I need cloud-init to only configure eth0, it must ignore all other NICs present on the machine.  Is it possible?  I couldn't find anything relevant in the docs
<powersj> garga, are you using a custom image or something Azure provides?
<powersj> AnhVoMSFT, ^
<Odd_Bloke> powersj: blackboxsw: I'm now +1 on https://github.com/canonical/cloud-init/pull/114.  Are you +1 on dismissing rharper's review so we can land it?
<powersj> Odd_Bloke, done
<Odd_Bloke> powersj: Thanks!
<garga> powersj: custom image
<blackboxsw> Thanks for the review/land and bug update Odd_Bloke. So should we raise that bug now in ubuntu-release https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1866930
<ubot5> Ubuntu bug 1866930 in cloud-init (Ubuntu) "[FFe] ec2 add support for configuring secondary NICs and secondary ipv4 and ipv6 addresses" [High,Fix committed]
<blackboxsw> since release team hasn't reviewed it yet?
<powersj> blackboxsw, yes please ping vorlon to ack it
<powersj> then upload
<powersj> and profit
<Odd_Bloke> We'll need to cherry-pick rather than new-upstream-snapshot, I think.
<powersj> I'm of the opinion that if we are going to SRU a whole version back right after focal releases, then why not do that now
<Odd_Bloke> powersj: Well, we don't have an FFE for most of the changes that have landed.
<Odd_Bloke> So I think we need to have a conversation with the release team about it, at the very least.
<blackboxsw> Odd_Bloke: put up the cherry pick branch https://github.com/canonical/cloud-init/pull/260
<blackboxsw> for review
<blackboxsw> powersj: if we SRU everything, I think vorlon will likely say go through the whole SRU validation process which I don't know we have time for during this freeze period. I'd like to get funcationality public sooner in case there were a problem. Then we still have runway to fix an unexpected corner case before Focal is released.
<powersj> blackboxsw, that's fine
<blackboxsw> I think it's simpler/faster for the FFe, but we do have a minor wrinkle in our release process in that I'm also including dan's package build-deps change in #260 though I'm not sure if that'll cause some concern
<Odd_Bloke> blackboxsw: Including that is fine, I think, I've reviewed the PR.
<blackboxsw> thanks Odd_Bloke ok I queue to upload for review in the event that the FFe is accepted
#cloud-init 2020-03-19
<Odd_Bloke> powersj: A couple of small (not at all urgent) PRs of mine that you could take a look at once you're around: https://github.com/canonical/cloud-init/pull/257 https://github.com/canonical/cloud-init/pull/263
<powersj> Odd_Bloke, ok will review post-standup
<powersj> Odd_Bloke, leave the merging to you
<Odd_Bloke> powersj: Thanks!
<cloaked1> So if `write_files` is used in `/etc/maas/preseeds/*some*userdata` and then let's say a machine is instantiated using cloud_init which calls `write_files` will both `write_files` properly function? I'm thinking that one (user_data) will supercede that of `curtin` it seems. Can someone confirm or deny that observation?
<cloaked1> https://discourse.maas.io/t/write-files-for-curtin-and-write-files-for-cloud-init-using-both-seems-to-conflict/1403
<powersj> cloaked1, hey thanks for the link - I think the best person to answer that isn't around this week unfortunately
<cloaked1> ah man!
<cloaked1> that sucks.
<cloaked1> I kinda need to know...
<cloaked1> ok, I'll figure something out. Thanks @powersj
<cloaked1> powersj: do you happen to know if there's a recommended way to add lines to /etc/fstab?
<powersj> cloaked1, if you are using cloud-config there is a mounts key
<powersj> https://cloudinit.readthedocs.io/en/latest/topics/examples.html#adjust-mount-points-mounted
<cloaked1> beautiful
<cloaked1> thank you!
<cloaked1> so does that have a curtin equal?
<cloaked1> that's what I'm trying to figure out...sorry, I misspoke in my questin.
<cloaked1> question
<powersj> I believe modifying /etc/fstab is what curthooks are for in curtin
<powersj> but rharper or smoser know much better than me :)
<cloaked1> powers: thank you
<cloaked1> er powersj:)
<smoser> cloaked1: are you snafuxnj ?
<smoser> it doesnt sem likely to me that user-data and curtin_userdata_custom would conflict.
<cloaked1> yup
<cloaked1> I am
<smoser> one would be applied in install time frame
<smoser> and one would be installed during first boot
<cloaked1> hmm. OK. for some reason I seem to be observing that they conflict, but I could be wrong. I'll some code to the discourse question. Maybe someone can validate what we're doing. I may have something wrong in my write_files object.
<smoser> for the most part , curtin is done after install
<smoser> so , anything it was going to do (write_files) is done
<smoser> user-data is like ec2 user data.
<smoser> in that it goes to the system on first boot
<smoser> the one overlap that i'm aware of in curtin and cloud-init is the networking bit.
<smoser> curtin just passes the network config that it is given through to the system rather than rendering it itself.
<smoser> then  cloud-init will render on first boot
<cloaked1> ok cool. I'm running some testing right now with the mount: option you provided. If that works then I'm not gonna press the write_files issue I'm observing for now.
<cloaked1> otherwise, I'll need to press it because it will likely be the only way to achieve what I'm trying to achieve.
#cloud-init 2020-03-20
<Max0815> hey, short question. Is it possible to read a value for a key of a cloud-config file from a different file? Like I want to read ssh_authorized_keys from a file rather than specifying it in the cloud-config file.
<Max0815> would be nice to reuse the code. If I want to put my cloud-config file under version control I obviously don't want to put stuff like passwords under version control.
<Max0815> So far I guess using a template, putting that under version control and generating the cloud-init dynamically will solve my use case.
<tribaal> Max0815: that is one option. Another option depending on your use case could be to execute an aribtrary command during cloud-init that will do the thing you need (read from a remote list of authorized keys or whatever).
<Max0815> tribaal: thx, didn't think about that. However, creating a template looks a bit easier. If I were to retrieve data from a remote I would need to provide a mechanism to access the remote in the cloud init.
<tribaal> oh yes, of course it entirely depends on your use case :)
<ananke> what's the easiest way to tell which cloud-init module is still running? I have cloud-init status showing 'running', but I can't figure out what's not finished
<blackboxsw> ananke: cloud-init status --long. or check for stages that have no start and stop time in /run/cloud-init/status.json
<blackboxsw> I bet it's something to the with something introducing a systemd dependency chain issue. (that's what typically blocks cloud-init from finishing
<ananke> interesting: DataSourceEc2Local
<blackboxsw> https://unix.stackexchange.com/questions/193714/generic-methodology-to-debug-ordering-cycles-in-systemd
<ananke> so long story short, we're in the process of using packer & cloud-init to make new AMIs for our educational platform. we have stuff working on ubuntu 18.04, and took that over to kali. this appears to be the first hurdle to overcome
<ananke> while the source kali has cloud-init, and looks like it sets up stuff as expected, it never finishes. we use '/usr/bin/cloud-init status --wait' as a stage in the build, to make sure things are up and running, but clearly something is waiting for AWS info
<blackboxsw> today I learn about kali :) amanke if you grab /var/log/cloud-init.log it'd likely show you specifically what cloud-init is looping on `cloud-init analyze blame` may also give you some info. (though I can't recall if that doesn't help you if you are still in an unfinished state for cloud-init)
<fredlef1> @rharper: I'm just circling back to PR-229
<fredlef1> After reading your doc, I think a detail is escaping me.
<Odd_Bloke> fredlef1: rharper is out until Monday, FYI, so commenting on the PR might be a better way to be sure to get a response.
<fredlef1> good to know. Thanks
<Odd_Bloke> A very small precursor to my Oracle work to sort the imports: https://github.com/canonical/cloud-init/pull/266
<blackboxsw> Odd_Bloke: approved, needs CI
<Odd_Bloke> Thanks, landed.
<blackboxsw> ok put up Focal to prioritize netplan above ENI rendering https://github.com/canonical/cloud-init/pull/267
<blackboxsw> for Ubuntu
#cloud-init 2020-03-21
<ananke> blackboxsw: thank you for the suggestions, sorry I didn't have a chance to respond (very hectic life/work day). I'll try those out either this weekend or on Monday and let you know what the root cause was
<ananke> I did grab all the various logs and output though
#cloud-init 2020-03-22
<andras-kovacs> Hi! Could you tell me how can I disable cloud-init's resolv-conf module? I've added the manage_resolv_conf: false to my cloud.cfg but seems like cloud-init doesn't care about it. â¹ï¸ Thank you! (I'm using cloud-init-18.5)
