#cloud-init 2014-08-18
<harmw> RaginBajin: ping
<RaginBajin> harmw pong
<harmw> I've got something for you to try :)
<harmw> fbsd 10 packages
<RaginBajin> oh cool
<harmw> http://buildservice.weites.com/freebsd/
<harmw> the 2 json packages are enroute to the official ports tree
<harmw> I'm waiting with my submission of py27-cloud-init until I have some verified results it realy does work :p
<harmw> it based on my branch btw, so smoser will need to merge that + version kick it first before I can submit the port
<RaginBajin> Gotcha. 
<harmw> just let me know if this is ready and usefull - I'm eager to get this thing in the ports repo :)
<RaginBajin> Ok. So I think I have the ConfigDrive all working and not mucked with specific to Fbsd 
<RaginBajin> What do you think the best way to share it with you is 
<harmw> ah ofcourse
<harmw> pastebin is ok..
<harmw> which code is it based on, trunk?
<harmw> in that case just branch it and push to LP
<harmw> you only have configdrive stuff right?
<harmw> (as in, let's get that in as well before merging with trunk)
<RaginBajin> ok. I'm going to put it on my github so you can see it first. Make some suggestions for me and then I can do the LP stuff. 
<harmw> can't we just skip github :p
<harmw> (but np, go ahead)
<RaginBajin> I'm not very familiar with LP. So it was easier that way 
<harmw> ah, Joseph :)
<RaginBajin> So here's the Github of just my changes. - https://github.com/RaginBajin/cloud-init/tree/freebsd-configdrive
<RaginBajin> two files, utils.py and sources/DataSourceConfigDrive.py
<RaginBajin> Let me see if I can figure out this LP stuff
<nvucinic> hi guys, i managed to get NoCloud working
<harmw> perhaps you could explain as to why you introduced the lazy/non-lazy bool RaginBajin 
<harmw> great nvucinic :)
<RaginBajin> harmw: You mean as a comment? 
<harmw> yea, perhaps, I think that would be best
<RaginBajin> gotcha
<harmw> RaginBajin: you've mentioned something last week about a hosts template?
<RaginBajin> yeah. I didn't include it in this one
<harmw> if you have a fix, let's see if we can included it as well
<RaginBajin> but it's a simple fix, just copy a template file to have freebsd instead of say debian 
<RaginBajin> Ok. 
<RaginBajin> Let me  work on that. 
<RaginBajin> Do you have a link to that maybe walks through committing to LP using bzr?
<harmw> no not realy, LP tells you how to branch and push after you've done local commits
<harmw> just need a launchpad login
<harmw> bzr who-am-i
<harmw> bzr launchpad-login
<harmw> bzr <the checkout cmd>
<harmw> <do stuff>
<harmw> bzr diff
<harmw> bzr commit 
<harmw> <do more stuff>
<harmw> bzr diff
<harmw> bzr commit
<harmw> and when you're ready to push, just bzr push :)
<RaginBajin> cool. 
<smoser> harmw, i will review your freebsd this week, i promise.
<harmw> smoser: great :)
<johnca|home> http://cloudinit.readthedocs.org/en/latest/topics/modules.html  was this section always blank?
<johnca|home> Where can I get information on the different things you can do with cloudinit (files command, package, etc)
<RaginBajin> harmw or smoser I got my stuff uploaded to LP but I don't think I did it right.. https://code.launchpad.net/~josephbajin/cloud-init/freebsd-configdrive
<RaginBajin> can one of you guys take a look when you have time
<smoser> RaginBajin, sure.
<RaginBajin> smoser: I did follow the readthedocs info on "hacking on cloud-init". Just seems weird that I have all the other commits. 
<smoser> all other commits ?
<smoser> oh. i see. RaginBajin . bzr really just has a saner merge story than git does.
<smoser> so i'll review your proposal
<smoser> and then merge into trunk
<smoser> and put a merge commit message on the merge.
<smoser> ie, that ocmmit message is the one that you'd submit in a git workflow (as you keep 'squash'ing your commits)
<smoser> RaginBajin, 
<smoser> https://code.launchpad.net/~josephbajin/cloud-init/freebsd-configdrive/+merge/231214
<smoser> put a commit message on that.
<smoser> (and if you can't, i'll delete it and *you* can propose merge and do so)
<RaginBajin> Ok. I think I understand. I was just looking at harmw repo and saw just a few messages vs mine which had mine and yours and everyone elses. 
<RaginBajin> ok let me try
<RaginBajin> Is there any special way that you do commit messages? 
<RaginBajin> nevermind. I added a commit message. I went back and forth in another project on the specifics of a commit message. I think it matches others. 
<smoser> RaginBajin, well, i like the git format
<smoser> < 74 chars of one line summary
<smoser> then newline
<smoser> then longer message
<RaginBajin> smoser If you need me to redo the commit just let me know. 
<smoser> just change the commit message in the merge proposal
<smoser> that make sense ?
<harlowja> smoser any idea who's running https://github.com/number5/cloud-init ?
<smoser> number 5
<harlowja> lol
<harlowja> ya, durn, nm 
<smoser> i dont know though. 
<harlowja> np, guess we should move to github to ;)
<RaginBajin> smoser: Ok I fixed the commit message. Should look closer to git's way 
<harmw> hehe, you guys noticed this: http://svnweb.freebsd.org/ports?view=revision&revision=365329
<harlowja> cool
<harmw> the othr one is 'in' as well
<harlowja> nice
#cloud-init 2014-08-19
<nvucinic> hi guys, one more question 
<nvucinic> when i dismount iso from nocloud 
<nvucinic> all settings get reverted also
<smoser> nvucinic, that doesn't make much sense. 
<smoser> what is "dismount" ?
<smoser> you mean eject cdrom ? 
<smoser> the next boot will look for another datasource, yes.
<smoser> you can change that with 
<smoser>  manual_cache_clean: True
<smoser> see doc/examples/cloud-config.txt
<nvucinic> smoser: it was centos, not cloud-init
<nvucinic> cloud-init works like a charm 
<nvucinic> gave a few bitchslaps to drones 
<nvucinic> everything is working fine now
<devicenull> hmm
<devicenull> I didn't realize I could just provide a user-data script via the ec2 metadata api
<devicenull> rather then implmenting all the different parts of that api
<harmw> RaginBajin: ping
<RaginBajin> pong
<harmw> ah
<harmw> I was wondering if you've had a chance to checkout those packages yet :)
<RaginBajin> I did.. I installed them. I just haven't gone through actually running anything with them 
<harmw> ah ok
<RaginBajin> I did notice that they built the cloudinit python portion differently though
<harmw> what do you mean?
<RaginBajin> In the directory /usr/local/lib/python2.7/site-xxx/
<RaginBajin> if you build it by hand you just get the cloud-init.xxxx.egg directory
<RaginBajin> with yours I seemed to have both the egg and just a directory called cloud-init
<RaginBajin> I can get the exact names if you need them 
<RaginBajin> I'm just trying to go off of memory
<harmw> hmk
<harmw> well, as long as it just works :) the port's Makefile doesn't do that many, and certainly not very special things
<RaginBajin> gotcha. I'll check it out and let you know probably tomorrow. 
<harmw> great
<devicenull> is there a way to pass per-instance scripts in via user-data?
<devicenull> I see talk about executing them from directories, but that's not exactly what I want
<devicenull> so, I'm writing a file to /var/lib/cloud/scripts/per-instance/ , but it never seems to get run
<devicenull> I'm testing with "rm -rf /var/lib/cloud/instance/*; cloud-init --debug --force init"
<devicenull> is there something else I need to be nuking to get this to work?
<devicenull> ahh missing shebang
<devicenull> works perfectly now :)
#cloud-init 2014-08-20
<smoser> devicenull, ir you put it in /var/lib/scripts/per-instance it probably will get run once per instance, but if you're writing after it has done that it might be too late.
<smoser> by default, though, if you pass in a '#!' it will be executed once per instance.
<smoser> and you can pass in multiple #! in multipart
 * smoser has to run. sorry to tease you.
<devicenull> smoser: actually, it works okay if you use "write-file" 
<devicenull> it executes once per instance, it seems
<harmw> RaginBajin: I hope I'm not pushing you to much :p
<RaginBajin> Hey.. Nope. Working on it today. Just going through some issues iwth FreeIPA and then I can start looking at it. 
<harmw> great man
<harmw> btw, which timezone are you in?
<harmw> or just country :p
<RaginBajin> harmw: I'm in the Eastern Time Zone. I'm just outside of Washington,DC
<harmw> ok, this is NL
<harmw> RaginBajin: ping!
<RaginBajin> It's alive and it works!
<RaginBajin> I even added my changes to it and was able to get it to work with ConfigDrive successfully. 
<harmw> no shit :)
<harmw> nice
<harmw> smoser: time to review :>
<smoser> harmw, well, feature freeze is tomorrow. so i'm planning on getting a upload in by then.
<harmw> freeze for juno? what we're talking about :)
<smoser> for utopic
<harmw> ah, and you want ci 0.7.6 in there
<smoser> i was considering that, yah. b
<smoser> but was basically planning on doing a round of bug fixes and pulling stuff in
<smoser> and that included your stuff
<RaginBajin> Would that include my stuff as well 
<harmw> smoser: ok
<smoser> RaginBajin, maybe. .. where is that?
<RaginBajin> https://code.launchpad.net/~josephbajin/cloud-init/freebsd-configdrive/+merge/231214
<devicenull> hmm
<devicenull> is there any way I as a provide can convince cloud-init to only use certain metadata sources?
<devicenull> I've just found that the debian cloud-init package will try to use the CloudStack metadata source.. which adds two minutes to boot time while it repeatedly retries
<devicenull> ah I think there's a preseed option for it
<devicenull> cloud-init cloud-init/datasources multiselect     Ec2, None
<devicenull> if anyone's wondering!
<RaginBajin> devicenull, you can do it in the cloud.cfg file as well.   You can just say Datasources: Openstack 
<devicenull> ya, that's what the preseed option does
<devicenull> I dont suppose you know of any 0.7.x packages for ubutnu precise?
<devicenull> ah found one
<kfox1111> odd problem. we are using a vendor data plugin we wrote for openstack. its outputing a json document, as its based on the example json one. it causes newer cloud init's to crash though.
<kfox1111> know how to make it ignore things if its just a json doc?
#cloud-init 2014-08-21
<harmw> smoser: on schedule for the freeze? which projects do you need to keep on track anyway, besides this one :)
<TTimo> Hello. Is there an easy way to supply user data 'manually'? e.g. I am getting access to a machine that was manually installed, but I have software that is already cloud-init ready .. as a special case I'd like to write out the user data 'manually' on the machine, and kick it off ..
<TTimo> the 'No cloud' data source looks like what I want, but do I really have to create a .img ?
<TTimo> even then .. probably not. the example shows spinning it up with kvm, which isn't what I want to do
<devicenull> hmm, isn't cloud-config supposed to run after could-init?
<devicenull> the ubuntu configs seem to start them at the same time
<nvucinic> TTimo: you can create iso and mount it on boot
<TTimo> the instance is already booted at that point
<TTimo> but it's not 'cloud-init' aware
<nvucinic> reboot it then 
<nvucinic> with iso mounted
<TTimo> so I wanted to manually install cloud init, and pass it the stuff
<TTimo> any option without a reboot? 
<TTimo> it's a VPS so I'm not sure what kind of options I would have on kernel parameters and a reboot
<TTimo> (they are working on cloud-init support .. it's not ready yet)
<devicenull> can't you just run the init scripts?
<devicenull> or like, 'cloud-init init'
<TTimo> yes .. that's what I want to do .. my question is probably very simple .. how/where do I put what I would normally give as 'user data' on EC2 before running cloud-init init ?
<smoser> TTimo, you can feed cloud-init user-data for NoCloud via either 'seed'
<devicenull> hmm, /etc/cloud/cloud.cfg.d/something
<devicenull> I think is what you're after
<smoser> or you can put it right into /etc/cloud/cloud.cfg.d/something.cfg
<smoser> links coming
<TTimo> oh
<smoser> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/files/head:/doc/examples/seed/
<smoser> that is "seed" (you populate the directory with 2 files)
<smoser> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/examples/cloud-config-datasources.txt
<smoser> that shows how to do user-data-blob inside of NoCloud Datasource
<smoser> which means just adding 1 file in /etc/cloud/cloud.cfg.d/my-config.cfg
<TTimo> nice!
<smoser> also, i started 'ci-tool' https://code.launchpad.net/~smoser/cloud-init/ci-tool/
<smoser> that does some of this
<devicenull> oh, maybe I'm hitting https://bugs.launchpad.net/cloud-init/+bug/1236463
<devicenull> hmm
<devicenull> oh, dumb
<devicenull> apt-get update was hanging, so my scripts didn't get run
<smoser> devicenull, /var/log/cloud-init-output.log will help you in such a case.
<smoser> (or console output if you're on something < trusty )
<devicenull> ya
<harmw> smoser: on par for the freeze?
<smoser> no. :O)
<smoser> but ui'm working on cloud-init now.
<harmw> oh cool
<harmw> I was mostly asking out of interest, not because of pending ci merges :)
<smoser> i may or may not make it :)
<smoser> i'd not realized that i'd gotten to ~ 95 bugs open now
<smoser> yuck
<harmw> damn
<harmw> harlowja_: you need to fix 95 c-i bugs so smoser can focus on merging :p
<harlowja_> eck
 * harlowja_ runs for the hills
<harlowja_> lol
<harmw> lol
<harlowja_> i think some of the bugs on https://bugs.launchpad.net/cloud-init can be cleaned up/closed/removed
<smoser> i thikn so too
<harlowja_> i can try to close/change some of those today, will see what i can do
<harlowja_> smoser harmw https://etherpad.openstack.org/p/TaskFlow-0.4 if u want to do some reviews ;)
<smoser> harlowja_, dont really understand how we regressed here:
<smoser>  https://bugs.launchpad.net/cloud-init/+bug/1356855
<smoser> and part of me wants to say : fix your ec2 metadata service to be ... ec2 like!
<harlowja_> def
<harlowja_> thats sorta nuts
<smoser> i think its really boiling down to that they dont like the "list" request to have a / on the end
<harlowja_> most web servers handle that :/
<harlowja_> kk
<harlowja_> weird stuff
<harlowja_> lol
<smoser>  http://paste.ubuntu.com/8108001/
<smoser> that is on oepnstack.
<harlowja_> meta-data/ 
<harlowja_> omg a slash!
<harlowja_> ^ ////
<harlowja_> anyways
<harlowja_> can be fixed pretty easily i think
<harlowja_> but would be useful to see that persons logs first
<TTimo> damn, no luck with that manual cloud-init run so far. it's like it's completely ignoring my NoCloud datasource
<TTimo> https://gist.github.com/TTimo/1c9c7ce9686aa45c63b5 - my /etc/cloud/cloud.cfg
<TTimo> should .. just work? when I execute cloud-init -d init, I see it parsed up .. however nothing happens
<harmw> smoser: merge! :p
<smoser> TTimo, that looks reasonable.
<smoser> you may have to run 'cloud-init local'
<smoser> as that will remove the cached data in /var/lib/cloud/instance/
<smoser> otherwise if thats there, 'cloud-init init'
<smoser> will just use the cache
<smoser> ie, first run:
<smoser>  cloud-init init --local
<smoser> then
<smoser>  cloud-init init
<TTimo> checking ..
<TTimo> or you mean the other way, first run doesn't need --local, and runs after that will
<TTimo> should I be concerned about: No 'init' modules to run under section 'cloud_init_modules'
<TTimo> __init__.py[WARNING]: Unhandled non-multipart (text/x-not-multipart) userdata: '#/bin/sh\necho "Test"\n...'
<TTimo> followed by No 'init' modules to run under section 'cloud_init_modules'
<TTimo> that's how the run ends atm
<TTimo> I don'
<TTimo> I don't understand what those modules are about in cloud.cfg though
<TTimo> is there a module I need to list to make sure user data gets handled ?
<TTimo> what are they useful for otherwise .. do they just add supported functionality basically
<smoser> oh.
<smoser> you replaced /etc/cloud/cloud.cfg ?
<smoser> dont do that.
<smoser> :)
<TTimo> yeah :)
<smoser> just add stuff to it or to /etc/cloud/cloud.cfg.d/local-nocloud-ds.cfg
<smoser> or something.
<TTimo> right .. getting a fresh vps, this one has seen enough
<smoser> so the warning aobut no 'init' modules is reasonable
<smoser> unhandled... that seems odd. not sure why that wouldget get picked up
<smoser> where are you playing ?
<smoser> just curious
<TTimo> digitalocean
<smoser> in theory user-data is coming there.
<smoser> https://digitalocean.uservoice.com/forums/136585-digitalocean/suggestions/3736274-support-cloudinit-instance-metadata
<TTimo> yep
<smoser> TTimo, fwiw, you can do a *lot* of this with lxc containers
<smoser> and clone is very fast.
<TTimo> docker you mean?
<smoser> no. here.
<smoser> http://ubuntu-smoser.blogspot.com/2013/08/lxc-with-fast-cloning-via-overlayfs-and.html
<TTimo> oh right. 'here' meaning ubuntu
<smoser> well, no 'here' meant: i'm looking for a url, just a minute 
<smoser> :)
<smoser> so you can do an lxc-clone, and then could just write the file into /var/lib/lxc/<container>/ ...
<smoser> if you us clone without overlayfs, then you'll see the whole filesystem in /var/lib/lxc/<container-name>/rootfs/
<smoser> make sense ?
<TTimo> yeah I think I see what you mean
<TTimo> the vps is fast and cheap, locally I'm on OSX so no native lxc etc.
<TTimo> cloudinit/handlers/__init__.py:206
<TTimo> def walker_callback
<TTimo> so looks like the content_type is not in handlers
<TTimo> so there's no run_part
<TTimo> Unhandled non-multipart (text/x-not-multipart) userdata: '#/bin/sh\necho "Test"\n...'
<smoser> harlowja, when you have nothing to do
<smoser> https://bugs.launchpad.net/cloud-init/+bug/1316323
<smoser> dont' knwo why i couldnt get that jsonpatch to work
<harlowja> kk
<harlowja> lol, marco
<harlowja> maybe i just assign marco to fix it (he's my manager)
<smoser> alright.
<smoser> i'm out.
<smoser> later all
<harlowja> lata
<TTimo> DataSourceNoCloud appends a nocloud to the seedfrom: parameter passed in .cfg
<TTimo> it also requires a trailing / on the seedfrom
<TTimo> but then .. it actually looks and loads user-data etc. from /var/lib/cloud/seed, not /var/lib/cloud/seed/noclip/
<TTimo> err /var/lib/cloud/seed/nocloud/
<TTimo> :)
<TTimo> man it's all messed up
<harlowja> smoser i think the reason jsonpatch stuff wasn't getting activated is that running a single module doesn't trigger the whole handler stuff
<smoser> harlowja, yeah, i thoguht that too, but i tried putting the file in /etc/cloud/cloud.cfg.d/ also
<smoser> and had hoped it would there.
<smoser> its quite possible you're right though.
<harlowja> possible, the ohther josh-merging-algo approach works though
<harlowja> smoser btw, cloud-init will be used for all deployments of all machines at yahoo pretty sooon
<harlowja> *will be included in and used
<harlowja> it was used for openstack vms and machines & stuff but that scope is increasing to all the things
<harlowja> so good work, ha
<TTimo> all the things
<harlowja> :-P
#cloud-init 2014-08-22
<harmw> smoser: don't forget your promise :P
<smoser> harmw, i'll look at that right now.
<smoser> are you going to paris ods ?
<harmw> no, afraid not
<smoser> booo
<harmw> :)
<harmw> smoser: let me know if you need feedback on reviewing that merger, I haven't got much time though
<smoser> k
<harmw> 30minutes left :P
<harmw> smoser: I think LP is broken
<harmw> it still didn't send me any notification about an accepted merge request :>
<smoser> :)
<smoser> it should have sent you some info though
<smoser> i've been going through bugs in cloud-init all day.
<smoser> i do sitll paln to get on that mp thoug h today.
<harmw> haha
<harmw> well, I don't think I've ever submitted bugs to c-i
<harmw> hence I didn't get any mail today :p
<harmw> but nice, fixing bugs is also needed :)
<TTimo> https://code.launchpad.net/~cloud-init-dev/cloud-init/trunk <- th
<TTimo> this the main source, right?
<harmw> yep
<TTimo> I might try to submit my problem with not being able to use NoCloud and a seed as a bug
<TTimo> bring some closure to the issue, well at least for me :)
<harmw> go ahead :)
<smoser> harmw, you still there?
<harlowja> smoser harmw if u guys want, https://code.launchpad.net/~harlowja/cloud-init/schema-validate/+merge/231950 , probably useful for documentation purposes and validation and all that...
<harlowja> can be progressively expanded to include other modules
<smoser> i'm ok with that. i like it even :)
<smoser> harlowja, if we can make jsonschema usage optional that would be goodl
<smoser> ie, wherever it loaded it it would just try/except importerror
<smoser> and if not there, then say "looks good to me"
<harlowja> hmmmm, and then no do validation?
<harlowja> k
<harlowja> thats easy with that way of decorating stuff
<harlowja> if done correctly we can suck these schemas into the docs somehow
<smoser> yeah. i really like that harlowja 
<smoser> yeah, if jsonschema isn't available, then dont' do validation. 
<smoser> i'm only thinking that i still care some about precise.
<smoser> and probably other linuxes dont have jsonschema
<smoser> RaginBajin, you there?
<RaginBajin> yep
<smoser> good
<harlowja> smoser k, updated with optionalness
<smoser> so lookoing at your mp
<smoser> RaginBajin, i dont like lazy unmount
<smoser> it just seems prone to error.
<RaginBajin> Which part do you mean 
<smoser> cloud-init admittedly should not do this... but during search-for-datasource it often mounts and unmounts devices
<harlowja> smoser i think i might have a new cloud-init contributor guy that might want to/be ok with flushing out all the rest of the modules there
<smoser> and if a lazy unmount happened, and then another came in and mounted...
<smoser> then i think unexpected behavior
<smoser> RaginBajin, why did you add "lazy mode" for umount
<RaginBajin> because in FreeBSD there is no such thing as the -l option
<smoser> but we didn't use -l before
<smoser> so you didnt' have to do anything
<RaginBajin> it's in the code that I pulled. 
<RaginBajin> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/util.py#L1302
<RaginBajin>  umount_cmd = ["umount", '-l', umount]
<smoser> stop showing me that i'm wrong!
<RaginBajin> ha sorry
<RaginBajin> If there is a better way, I'm all ears. That's the only way I could think that would make that option work 
<RaginBajin> I figured my crappy code would be critiqued and shown a better way. 
<smoser> bah. that code blames to harlowja 
<smoser> harlowja, why did we 'umount -l'
<harlowja> hmmm
<harlowja> errr
<harlowja> not sure, lol
<RaginBajin> Well probably because in linux it's makes systems less angry a bit.. You say just get rid of it so I can't see it, and when I'm no longer in the directory or something is using it do the work in the background
<RaginBajin> instead FreeBSD wants to choke you out till you umount it cleaning
<RaginBajin> cleanly* 
<smoser> RaginBajin, yeah, i understand why you want lazy
<smoser> but i explicitly think its a bad idea here.
<smoser> i want it unmounted now!
<harlowja> maybe we should have that contextmanager have a lazy=True or lazy=False option?
<harlowja> smoser didn't u add http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/550/cloudinit/util.py?start_revid=550#L697 ;)
<harlowja> before JH arrived, lol
<harlowja> ;)
<harlowja> back-in-them-olden-days
<smoser> i'm not making this stuff up, harlowja 
<smoser> http://paste.ubuntu.com/8116527/
<smoser> i mak ea lot of stuff up. but i dont htink i'm making htis up
<harlowja> ;)
<smoser> you even snuck it in
<harlowja> lol
<smoser> anyway. i'm gonna take it out.
<harlowja> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/517 ?
<harlowja> lol
<smoser> RaginBajin, so there is no 'mount -t auto' ?
<RaginBajin> correct
<smoser> hm.. yeah, that looks bad, harlowja 
<smoser> why would i have done that.
<harlowja> :-P
<RaginBajin> You have to be explicit in everything. But they don't even name their cdrom like everyone else. iso9660 vs cd9660
<harlowja> smoser u were young back in 2011
<harlowja> times were differnent
<RaginBajin> had more hair, etc right?
<smoser> that is true.
<smoser> i hate you all. 
<smoser> next time i say that one of you is wrong, just accept it.
<smoser> rather than proving that it is me that is wrong
<harlowja> lol
<smoser> ;)
<smoser> RaginBajin, what does platform.system() show ?
<smoser> can you let me into a system RaginBajin ? i guess i can just luanch one on ec2...
<RaginBajin> 'FreeBSD'
<smoser> thanks.
<smoser> seems like a better long term thinking idea to be able to specify a list of types
<smoser> RaginBajin, so if its not a cdrom
<smoser> if its not a cd device
<smoser> then we'd not try to mount with -t
<RaginBajin> It's a CDROM device
<RaginBajin> but it's called something else. 
<smoser> not always.
<RaginBajin> atleast in freebsd
<smoser> the config drive does not have to be a cdrom 
<smoser> and it can be vfat formated
<RaginBajin> right. 
<smoser> so it could just be /dev/sdb with either iso9660 format or vfat format
<smoser> RaginBajin, ok. lets say i booted this "freebsd" thing..
<smoser> how would i a.) do anything
<smoser> b.) install python
<RaginBajin> python should come already installed. 
<RaginBajin> It shoudl work like your linux box 
<RaginBajin> just a little bit more explicitness is needed most of the time. 
<smoser> $ python -c 'import platform; print(platform.system())'
<smoser> python: not found
<smoser> http://www.daemonology.net/freebsd-on-ec2/
<RaginBajin> check /usr/local/bin/python
<RaginBajin> actually. 
<RaginBajin>    /usr/local/bin/python2.7
<RaginBajin> so you may have to create a symlink to /usr/local/bin/python 
<smoser> good user experience, that :)
<smoser> RaginBajin, are you wliling to fix things up a bit ?
<RaginBajin> absolutely Just let me know what you need. I'm a little tight for time today, but over the weekend I can 
<smoser> k
<smoser> i'll put comments in there.
<smoser> RaginBajin, do we have a prayer at mounting windows 
<smoser> er... vfat on freebsd
<smoser> RaginBajin, thank you
<RaginBajin> smoser I'm not sure about windows or vfat.  I don't know if it has support. I assume it does
<smoser> i dont know.
<smoser> do my comments make sense?
<RaginBajin> I haven't looked at them just yet 
<smoser> k
<smoser> harmw, finally, smoser looked at your MP.
#cloud-init 2014-08-23
<harmw> smoser: on it, thanks!
<harmw> about the missing python, thats due to a missing meta package in your instance :)
<harmw> god I hate bikeshedding :>
<harmw> especially whn it concerns a temporary script
#cloud-init 2015-08-17
<natorious> anyone know if the stackforge/cloud-init master branch is functional under windows currently?
<natorious> got my answers from the subject line :)
#cloud-init 2015-08-18
<minfrin> Hi all. I have a quick question about ordering of sections in cloud-init. I have a "groups" section and a "write-files" section, and in the "write-files" section I need to assign group names that were created in the "groups" section. This doesn't work, because "write-files" runs before "groups", and "write-files" fails with the error group-not-found. Is there a way around this?
<smatzek> I don't see one short of modifying /etc/cloud.cfg to change the order but I don't know what other effects that might have.  What you could do is use write files to write the files and groups to create the groups, and then add a runcmd section to do the chown since runcmd runs after both of them.
#cloud-init 2015-08-19
<murphyslawbbs> Hi guys. i'm trying to boot coreos in a rather braindead enviroment that doesn't allow me to add a config-drive. I *do* have access to the filesystem image that I deploy, so I guess I could specify a URL or something and make cloud-init use that. But I don't know how to do that
<murphyslawbbs> So my question is how to manipulate the image so that it goes and looks for a cloud-init someplace else
<claudiupopa> smoser, Odd_Bloke I guess there's no meeting today?
<natorious> murphyslawbbs: CoreOS uses their own homebrew version of cloud-init.  Perhaps try and hit up their support channels?
<murphyslawbbs> ok ty
<Odd_Bloke> claudiupopa: I won't be at it, for sure.
<Odd_Bloke> At LinuxCon. :)
<claudiupopa> Oh, nice, enjoy. :-)
<lars-gregor> Hello, I am trying to add a repo to my cloud init file and it never actually adds it. I cant find anything helpful in the cloud init logs on the instance either. Can anyone see an issue? http://pastebin.com/jSpjGMkQ
<lars-gregor> The repo file just never shows up in /etc/yum.repos.d
#cloud-init 2015-08-20
<williamg> hi, I have conflicts while installing python-libs / cloud-init on CentOS
<williamg> does anyone already meet the problem ?
<williamg> (file /usr/lib64/python2.6/ctypes/__init__.py from install of python-libs-2.6.6-64.el6.x86_64 conflicts with file from package python-2.6.6-29.el6_2.2.x86_64)
<williamg> solved.. (yum update python)
#cloud-init 2016-08-22
<prometheanfire> smoser: what's the diference between _write_network, _write_network_config, _bring_up_interfaces and _bring_up_interface?
<prometheanfire> smoser: I have gentoo networking working, at least for dhcp, probably for static as well
<prometheanfire> smoser: last two commits here https://github.com/prometheanfire/cloud-init
<prometheanfire> only 'error' that happens doesn't seem to hurt anything
<prometheanfire> 2016-08-22 03:44:21,495 - __init__.py[WARNING]: apply_network_config is not currently implemented for distribution '<class 'cloudinit.distros.gentoo.Distro'>'.  Attempting to use apply_network
<smoser> prometheanfire, here now. reading your comments.
<smoser> you gave the systsem bonding config ?
<prometheanfire> smoser: hi
<prometheanfire> smoser: no, the kernel just supports it
<prometheanfire> and this breaks things
<smoser> hm..
<prometheanfire> because bonding_masters exists as a file within /sys/class/net
<smoser> well, there is another fix for that, that i do need to get in. but i didn't believe that it was as simple as you say.
<smoser> the ubuntu kernels *do* support bonding
<prometheanfire> well, for us the module is loaded static
<smoser> probably its as a module though, and likely it isnt loaded when that runs.
<prometheanfire> maybe that's why we hit it
<smoser> "module loaded static"
<smoser> ?
<smoser> you m ean builtin
<prometheanfire> ya
<smoser> :)
<prometheanfire> static kernel :D
<prometheanfire> it's how I (personally) compile my kernels
<smoser> ok.
<prometheanfire> easy to ship
<prometheanfire> this is the same code used in glean if it helps
<smoser> glean ?
<prometheanfire> simple-init?
<prometheanfire> same thing
<prometheanfire> https://github.com/openstack-infra/glean/
<prometheanfire> https://github.com/openstack-infra/glean/blob/master/glean/cmd.py#L686
<smoser> prometheanfire, so https://git.launchpad.net/~smoser/cloud-init/log/?h=bond_name has the fix for bond_master stuff
<smoser> and wrt glean, i can't just take that, as license is not compatible.  cloud-init requires signing cla for contribution.
<prometheanfire> right
<smoser> :-(
<prometheanfire> about the code
<prometheanfire> I wrote it
<prometheanfire> so I should be able to contribute it to both projects
<prometheanfire> I'd think at least
<smoser> oh. yeah, then you can.
<smoser> yes.
<prometheanfire> :D
<smoser> the other coment is that '_write_network' is kind of the "legacy" mechanism
<prometheanfire> in any case we should probably ignore the other interfaces, but if your boding update code works then that solves the immediate problem
<prometheanfire> so, should I convert to _write_networks?
<smoser> we want "_write_network_config", and using a Renderer like ubuntu/debian and rhel does
<smoser> unfortunately i think you're going to want more doc / info on the "network state" :)
<prometheanfire> is netconfig the same thing as settings?
<smoser> no. its more like what is described at http://people.canonical.com/~rharper/curtin/topics/networking.html
<smoser> and ubuntu and rhel basically load that thing into a 'network_state' and then render from mit.
<prometheanfire> ya, that's doable
<prometheanfire> a useable datastructure
<smoser> the one thing that i can tell you is that there are unit tests that you can easily run to poke around at what is happening
<smoser> (and please do contribute unit tests for your new code)
<prometheanfire> shouldn't need unit tests for that, but ok :P
<smoser> prometheanfire, thank you. i'm really happy to have your contributions
<smoser> well, for rendering... you  can have quite complex state
<prometheanfire> yarp
<prometheanfire> at the moment I'll be using this patch on 0.7.7
<prometheanfire> or will soon
<prometheanfire> but I've tested it at least and it fufills the simple use case
<prometheanfire> and even better, it actually works
<prometheanfire> current code is broken for networking on gentoo, it lays down debain style configs
<prometheanfire> right now the code to use the new method looks over complicated
<smoser> prometheanfire, yeah. i'm not really happy with it either. note though that it supports much more complex config
<smoser> multiple ips per nic, ipv4 ipv6, bond, vlan..
<prometheanfire> it does
<smoser> so yes, it is more complicated
<prometheanfire> I just don't even know where to start
<prometheanfire> it's that hairy
<smoser> yeah.
<smoser> i'd suggest looking at rhel, and how that works.
<smoser> and then just working from unit tests to get somethign sane.
<smoser> her... i'll try to get some scaffolding in place for you
<smoser> s/her/here/
<prometheanfire> current code actually started from arch
<smoser> waht wyould you suggest as the name for the ntwork config style ?
<prometheanfire> we have a name, sec
<smoser> ie, 'eni', 'sysconfig' ...
<prometheanfire> https://wiki.gentoo.org/wiki/Netifrc
<prometheanfire> netifrc
<prometheanfire> looking at rhel's stuff in net/sysconfig
<smoser> k. thanks..
<prometheanfire> ok, this is an example of the netconfig
<prometheanfire> {'version': 1, 'config': [{'name': 'eth0', 'subnets': [{'type': 'dhcp'}], 'mac_address': 'fa:16:3e:00:10:ee', 'type': 'physical'}]}                                                                                                                                        [ ok ]
<smoser> right. thats netconfig, then it gets loaded into 'NetworkState', and the renderers actually take the NetworkState.. a middle ground
<prometheanfire> right
<smoser> give me 10 more minutes, and i'll try to hand you a unit test thing to fill in
<prometheanfire> cool
<prometheanfire> have been looking at sysconfig.py, looks like it's just a longer version of my if  stuff
<prometheanfire> does netconfig allow you to have vlans in your bonds or bonds in your vlans, etc?
<prometheanfire> oh, it's listed explicitly
<prometheanfire> neat
<smoser>  http://paste.ubuntu.com/23078466/
<smoser> prometheanfire, ^ that should at least allow you to easily poke through the code and see your results easily.
<smoser> i did not add the _write_network_config to gentoo distro but i'm guessing you can figure that out.
<smoser> one thing to point out, the renderer should be idempotent. whatever network config is there, the one provided is what the sysstem *should* do
<prometheanfire> ya, that's the easy part
<prometheanfire> I've already started on it
<smoser> mgagne_, i'd appreciate your thoughts on https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/303563
<smoser> ideally i'd liek to have that merged today, you mentioned it had some issues still i think
<smoser> i think you still say 3.2 is busted.
<mgagne_> smoser: I will test the whole patch set. Is it complete yet?
<mgagne_> smoser: 3.2 got fixed with the bonding_slaves detection
<smoser> mgagne_, then i think it is complete ...
<smoser> pending your test :)
<smoser> and my fuzzy memory
<mgagne_> smoser: for some reasons, cloud-init configures network twice. At this point, I don't care about the reason, it's beyond my expertise. So detecting bonding should do the job.
<smoser> it configures networking once.
<mgagne_> ok, will launch a build with your patches against 0.7.7~bzr1256-0ubuntu1~16.04.1
<smoser> it renames devices twice.
<mgagne_> smoser: It goes through the network json config parsing twice at least
<smoser> i'mi pretty sure thats the case, and i have to think again about why we rename network devices after we've configured.
<mgagne_> so I will be applying this patch http://paste.ubuntu.com/23078862/ verbatim to cloud-init 0.7.7~bzr1256-0ubuntu1~16.04.1 and will boot on a baremetal with bonding+vlans to validate.
<Tim_> hi
<prometheanfire> thanks
<smoser> prometheanfire, merged your branch
<prometheanfire> yarp
<prometheanfire> will rebase my integration branch when your stuff merges
<prometheanfire> I'll see if I can test your patch
<mgagne_> smoser: so I tested the patches. Somehow I managed to reproduce an intermittent bug my coworker had. Default gateway fails to configure and server doesn't ping.
<smoser> hmm... i think maybe rharper might know something.
<smoser> mgagne_, xenial, right ?
<mgagne_> yes
<smoser> ifupdown is very "fun".
<mgagne_> smoser: is there anything I can look at? default gw is configured in post-up with route add || true
<mgagne_> so I'm not sure how I'm supposed to debug that
<smoser> and i know that rharper has been doing some hair pulling over bonds recently.
<smoser> so you can get into it, mgagne ?
<mgagne_> so we never supported ubuntu 16.04, even without bonding yet so I'm not sure if it's a known issue with bonding or cloud-init.
<mgagne_> any logs I can pull to help debug?
<prometheanfire> smoser: reviewed your patch, worksforme
<smoser> mgagne_, grab /var/log/cloud-init.log
<prometheanfire> also, closed the other merge request
<smoser> mgagne_, can you get to it while its up?
<smoser> or are you ust able to shut down and collect files
<mgagne_> yea, will get the logs after my meeting =)
<mgagne_> will get the whole /var/log if needed
<smoser> if you can get while its up, please get output of
<smoser> ifconfig -a
<smoser> and any thing else you might find useful
<smoser> systemct status
<smoser> woiuld be good
<mgagne_> if post-up fails, will the output be logged?
<rharper> mgagne_: any stderr from the ifup will be capture in the networking.service log, so you should see something in systemctl status -l networking
<mgagne_> ok, I ran that command and didn't see anything
<rharper> mgagne_: from your gist though, the devices and routes all came up as configured
<mgagne_> will rerun
<mgagne_> to make sure
<mgagne_> check cloud-init-output.log, you won't see the default gw
<mgagne_> only link local routes
<rharper> in your gist, do you have the etc/network/interfaces.d/50-cloud-init.cfg file ?
<smoser> rharper, interfaces-cloud-init.txt
<mgagne_> yes
<smoser> https://gist.github.com/mgagne/fbc1b05412f41426f2e248acd5efad14#file-interfaces-cloud-init-txt
<rharper> smoser: ah right
<mgagne_> added systemctl status -l networking output
<mgagne_> so I suspect that *maybe* the default route is added but something removes it later?
<mgagne_> or will || true hide the failure?
<rharper> mgagne_: in your gist, the bond0.602 default gw is the one you expect ?  (the launchpad bug had other config)
<mgagne_> yes
<mgagne_> I thought it would be configured with the gateway stanza but Â¯\_(ã)_/Â¯
<smoser> Odd_Bloke, around ?
<smoser>  test_exception_fetching_fabric_data_doesnt_propagate
<smoser> why would i not want that to propogate?
<mgagne_> rharper: is there anything I can do to help debug? I don't mind rebuilding an image with debug config or whatever.
<rharper> mgagne_: a plain route -n  would be nice
<rharper> and the original network_data.json;
<mgagne_> rharper: I added the default route already so I can SSH and pull logs
<rharper> it's applying some routes; I just can't see why it wouldn't do the post up
<mgagne_> rharper: added to gist
<mgagne_> rharper: I will try to reboot a 2nd server which didn't have the issue and see if I can reproduce after multiple reboots
<rharper> ok
<mgagne_> rharper: let me know if you would prefer to get SSH access for further debug, this can be done
<rharper> ok
<smoser> rangerpbzzzz, around ?
<smoser> wonder if its ok if i open a bug and assign it toyou.
<rharper> mgagne_: so I can recreate the case where the cloud-init-output does not contain the default route; but post-up on bond0.602 does run and work;  maybe we could add a cloud-init final command to run route -n so we can see that?  in xenial, cloud-init writes the files and networking.service is doing an ifup -a (which will bring up any non physical devices ;  the physical devices with bond-master will create the bond0 and
<rharper> enslave them) and then the ifup -a will trigger an ifup on bond0.602 and bond0.612;  they'll run and run the post-up which should add the default gw you need;
<mgagne_> rharper: so you think that: cloud-init runs route -n, doesn't see default gw at this point in time but later route should be configured by ifup?
<rharper> I'm not entirely certain but in my recreate;  the output info doesn't show the bond.vlan route; but when I login and run route; it's fully up
<mgagne_> because it's true that the /32 route doesn't show in route output. this means something is running later to add routes.
<rharper> I don't know when it runs to collect the network status
<rharper> but possibly too soon or some other reason
<rharper> not sure if smoser has more details
<mgagne_> could it be the slaves link aren't fully up and therefore routes aren't applied yet since it's in post-up?
<rharper> yes
<rharper> slaves can take some time
<rharper> bonding scripts will wait up to 600 seconds for a bond to join
<rharper> err , 60 seconds
<smoser> hm.
<rharper> (60 * 0.1)
<mgagne_> 6 seconds? :D
<smoser> cloud-init writes network status during 'init' stage
<rharper> *sigh*  600 * 0.1
<smoser> which shoudl be after static networking is up
<smoser> so if that runs before all if the 'ifup' stuff is finished, then that is a bug
<smoser> # systemctl cat cloud-init.service | grep networking
<smoser> After=cloud-init-local.service networking.service
<rharper> smoser: no, I can see the route info now;  I was looking at the top of the -output file before I added the config;  I definitely see the default routes running but; this is a VM versus baremetal;
<smoser> Requires=networking.service
<mgagne_> but Net device info shows the interface as up
<mgagne_> I don't know where it takes the status but it means slaves are up too?
<rharper> http://paste.ubuntu.com/23079501/
<rharper> it should look like that
<rharper> I only added the one bond with default route (and used active-backup on a second nic in a VM);  but the output should look similar in number of routes
<rharper> but it is odd that during the dump of the route in mgagne_ case, there is nic message from kernel about being up;
<rharper> the info table runs at Up 48.87 , but the nic up message isn't until 57
<mgagne_> so I'm rebooting in loop to try to reproduce the problem and so far, no luck
<mgagne_> coworker says that if you reboot a node with the problem, gw is configured properly and your issue is fixed.
<rharper> yeah, the switch delay
<mgagne_> so I'm wondering if it's something cloud-init does at that time
<mgagne_> which isn't done in next boot
<mgagne_> like renaming an interface
<rharper> you can force cloud-init to re-run by nuking /var/lib/cloud/*  in the instance befere rebooting
<rharper> renaming happens on each boot
<rharper> or an attempt
<mgagne_> by cloud-init?
<rharper> yes
<mgagne_> ok, will nuke /var/lib/cloud/* on an other node I have
<mgagne_> and reboot forever
<prometheanfire> that's how I test as well
 * rharper steps away for  a bit
<mgagne_> so I reboot and rebuilt 10+ times and I can't reproduce
<mgagne_> it looks to be a very unlucky race condition
#cloud-init 2016-08-23
<Odd_Bloke> smoser: I haven't looked at it, but my recollection of test_exception_fetching_fabric_data_doesnt_propagate is that data sources shouldn't raise exceptions.
<Odd_Bloke> Reading cloudinit.sources.find_source, that isn't true.
<Odd_Bloke> But I believe that was the motivation behind it.
<smoser> prometheanfire, there is one question for you at https://code.launchpad.net/~prometheanfire/cloud-init/+git/cloud-init/+merge/303339
<smoser> i'd pull that, but would ike to know why we need that change
<smoser> harlowja, around ?
<smoser> https://code.launchpad.net/~harlowja/cloud-init/+git/cloud-init/+merge/301731
<smoser> try:
<smoser>   from unittest import mock
<smoser> except ImportError:
<smoser>  import mock
<smoser> why do we need that ?
<smoser> (we have it lots of palc).
<nacc> smoser: python3 and python2 compat?
<nacc> (possibly)
<smoser> nacc, well, yes, but we do not do it in other places
<smoser> and it works happily
<smoser> or i'm oblivious to the failure :)
<smoser> mgagne_, i'm going to pull bond_name
<mgagne_> smoser: cool, it fixes all known issues described in the bug report
<nacc> smoser: heh could be -- it looks like maybe unittest.mock has been backported, unclear
<mgagne_> smoser: I couldn't find the root cause of the race condition we talked about yesterday, rebooted 20x times and no luck
<smoser> rharper, around ?
<smoser> given http://paste.ubuntu.com/23082413/
<smoser> should enp0s1 have 'auto' on line 253 ?
<smoser> i think it probably should
<rharper> smoser: here
<rharper> smoser: let me compare to the curtin vlan tests
<rharper> I don't think so as the vlan config will raise the underlying iface
<rharper> smoser: so, if you put auto, and manual, networking-online check fails
<rharper> ifquery --list --allow-auto will show the iface
<rharper> but since it has a manual config, there's nothing to ifup
<rharper> this will break the wait on all ifaces to be online
<smoser> rharper, ok. i actually didnt' see 'inet manual'
<smoser> so you think that is ok ?
<rharper> it's required
<rharper> vlan hooks do an up on the raw interface
<rharper> before applying the vlan config
<rharper> we don't want manual interfaces in the auto list or networking service fails to come up
<rharper> (ie, nothing is going to "up" a manual interface)
<rharper> hence the manual part
<rharper> explicitly,  we do not auto $IFACE on underlying devices unless they also have a network config (subnet)
<rharper> the hooks for bond, vlan and bridge handle bringing up the lower interface, if needed
<harlowja> i don't think unittest.mock got backported
<smoser> harlowja, so how does tests/unittests/test_datasource/test_configdrive.py work ?
<smoser> it just does
<smoser>  from ..helpers import TestCase, ExitStack, mock
<smoser> and 'mock' from helpers is just 'import mock'
<harlowja> hmmm, whats in our requirements for that
<harlowja> https://git.launchpad.net/cloud-init/tree/test-requirements.txt#n3
<harlowja> i think we can just do import mock
<harlowja> it should be fine
<harlowja> thats more or less what the openstack folks did
<harlowja> because mock is maintained such that its equivalent to the py3.x one
<harlowja> equivalent or better
<harlowja> but overall yes smoser we should do some test cleanup
<harlowja> it needs it :-P
<harlowja> *its needed
<smoser> harlowja, ok.
<smoser> so can you
<smoser> a.) just rebase that thing oncurrent tip
<smoser> and then push
<smoser> (no more changes necessary)
<smoser> then i can merge and push and the MP will know it got pulled
<smoser> b.) go ahead and fix those import mock things everywhere.
<harlowja> hmmmm, k
<harlowja> why rebase needed :-P
<harlowja> just wondering
<harlowja> usually if there aren't conflicts then rebase shouldn't be needed?
<smoser> harlowja, you either have to rebase and push or i have to merge and accept the 'merge commit'
<harlowja> k
<harlowja> hmmm, intersting
<smoser> its same as anywhere.
<smoser> if you're full fast foward, then i can just pull and still use your commit hashes
<harlowja> kk
<smoser> and thus the merge proposal can just magically "know" that your merge got pulled
<harlowja> gotcha
<harlowja> ok, good to know
<smoser> if do 'merge' and its not a FF then i get a "merge commit"
<smoser>   Merge remote-tracking branch 'harlowja/space'
<smoser> and i hate looking at those things :)
<harlowja> :-P
<harlowja> fair nuff
<harlowja> anything to make the scott happy
<harlowja> lol
<harlowja> i'll get around to that soonish
<harlowja> btw
<smoser> harlowja, i'll take care of it.
<smoser> you do 'b' instead.
<smoser> thanks
<harlowja> lol
<harlowja> k
<harlowja> but i wanted to do a
<harlowja> lol
<prometheanfire> :D
<dccunha> Hi
<dccunha> I'm having some trouble to use vendor-data for setting up instances. There is any config at cloud-init that must be done for this to work?
<dccunha> I getting vendor-data from an OpenStack's datasource and it's been saved like this:
<dccunha> cat /var/lib/cloud/instances/d1a0a861-e386-444c-aea8-196e2540cfc4/vendor-data.txt
<dccunha> #cloud-config
<dccunha> package_update: true
<dccunha> packages:
<dccunha>  - htop
<cn28h> how can I enable logging for cloud-init? smoser mentioned being interested to see the contents of /var/log/cloud-init.log if I am able to reproduce an error (and we do seem to be reproducing it) but the log file is always empty.
<cn28h> haven't been able to track down any docs saying how to set log levels or anything
<smoser> cn28h, change logging to only go to a file.
<smoser>  https://code.launchpad.net/~harlowja/cloud-init/+git/cloud-init/+merge/301729
<cn28h> aha
#cloud-init 2016-08-24
<prometheanfire> smoser: in your example, was there a reason you did your symlink as relitive?
<prometheanfire> smoser: also, I added comments in my gentoo net fix that shouldn't be there (but aren't harming anything)
<ffledgling> hello, I'm trying to launch a fedora.qcow2 cloud image with qemu-kvm locally, by mounting the meta-data and user-data files on the cd-rom, but sometimes it doesn't respect those settings, any idea why this might be? Can someone point me to how should begin debugging this?
<smoser> prometheanfire, it is at least generally prerred in debian for symlinks to be relative. they're easier to work with for sure.
<smoser> even in the test, the goal is to "render" stuff all into a target root, and easier to read content from that directory if it doesnt leak outside.
<smoser> ffledgling, it fails sometimes ?
<ffledgling> smoser: yes, it fails sometimes. When I clean everything up and start it from scratch, it works fine, but if I restart the instance it fails
<ffledgling> I'm using the no-cloud plugin I think with the user-data and meta-data file
<ffledgling> I suspect it might be because of "Note: that the instance-id provided (iid-local01 above) is what is used to determine if this is âfirst bootâ. So if you are making updates to user-data you will also have to change that, or start the disk fresh."
<ffledgling> But I don't quite get what that means
<smoser> ffledgling, 2 things coudl be happening
<smoser> a.) if you want the system to re-run initialization "first boot", then you will have to provide a new instance-id
<smoser> cloud-init only runs '#!' scripts and much other user-data bits on first instance boot
<smoser> so it determines 'first instance boot' by seeing the instance id and comparing it to one it stored from the last time.
<smoser> so if you boot once with instance-id of 'abc' and then change the user-data and boot again with 'abc' it wont pick it up.
<ffledgling> Okay, I'm not doing that, and I don't think I want/need to. The default behaviour seems to be what I want
<smoser> b.) it could be a race condition where the nocloud disk isn't there yet at the point in which cloud-init runs.
<smoser> i've not seen this in practice, but it is plausible.
<ffledgling> oh
<ffledgling> That would make sense
<smoser> if you think this is happening, please
<ffledgling> I don't know if it is, how would I confirm?
<smoser>  - change logging to go to a file only
<smoser>   see https://code.launchpad.net/~harlowja/cloud-init/+git/cloud-init/+merge/301729 for how to do that.
<smoser>   (basically change /etc/cloud/cloud.cfg.d/05_logging.cfg as show)
<smoser>   - boot the system
<smoser>   - shut the system down
<smoser>   - collect /var/log/cloud-init.log
<smoser> you dont have to shut down if you dont want, if you can log in feel free to get it that way.
<smoser> i have to go afk for a bit.
<smoser> back later.
<ffledgling> Okay, I can't actually login to the box at all, so I'm not sure I can modify any of the cloud.cfg file?
<ffledgling> I'm using cloud-init to provide a password to the instance.
<smoser> ffledgling, mount the image and change it before booting.
<smoser> you can use mount-image-callback or guestfish or something. http://ubuntu-smoser.blogspot.com/2014/08/mount-image-callback-easily-modify.html
<prometheanfire> smoser: do I have to use the returned object? 'ns = parse_net_config_data(netconfig)' or can I use the json directly?
<smoser> prometheanfire, well, i'd like a Renderer and the Renderer takes network state.
<smoser> i'd like it to be have like the others, so that as we modify / improve things, we wont have snowflakes
<prometheanfire> smoser: it just seems like a lot of work for rendering something so simple
<cn28h> smoser: http://dpaste.com/2V9JTX3 here's the full log which I obtained after adding the change you suggested yesterday to the logging config.  Looks like ssh-authkey-fingerprints is running before cloud-user gets created (even though the logs seem to suggest cloud-user was created first?)
<smoser> prometheanfire, fyi my comment https://code.launchpad.net/~utlemming/cloud-init/+git/cloud-init/+merge/303471/comments/784454 applies to you also
<smoser> i assume you probably followed the HACKING ;)
<smoser> actually, no. it i think says it right.
<prometheanfire> smoser: smoser this is an example of doing it all (bridge bond phys vlan) https://gist.github.com/prometheanfire/d1eb9d60fecf25ac7418c8f4160201b3
<prometheanfire> it allows access to a vlan trunk
<prometheanfire> smoser: ?
<prometheanfire> what about that comment?
<smoser> you ust that you can use a shorter remote
<prometheanfire> I have three origins for my cloud-init repo
<smoser> you have git.launchpad.net/~prometheanfire/cloud-init/+git/cloud-init
<smoser> you can do
<smoser> you have git.launchpad.net/~prometheanfire/cloud-init
<prometheanfire> well, the remote's already set up
<prometheanfire> prometheanfire@git.launchpad.net:~prometheanfire/cloud-init
<prometheanfire> that's how it's already set up
<smoser> prometheanfire, right. the only value in it is then i can more easily do things like:
<smoser>  for user in prometheanfire utlemming; do git remote add $user git.launchpad.net/~prometheanfire/cloud-init; done
<smoser> well, probalby should use '$user' in the url, but i thjink you get the idea
<smoser> i did that yesterday, and couldn't figure out why getting your remote was failing
<prometheanfire> ya
<prometheanfire> not sure why it was failing
<prometheanfire> given it seems it's set up right
<smoser> shoot.
<smoser> i typoed in that comment
<ffledgling> smoser: do you mean the iso or the actual fedora image?
<smoser> that is what is confusable
<smoser> ffledgling, the image
<prometheanfire> smoser: anyway, soooooooooooooooooo much boilerplate
<smoser> prometheanfire, i know... let me see.
<smoser> cleaned up hopefully more obvious comment now
<prometheanfire> sorry
<smoser> https://code.launchpad.net/~utlemming/cloud-init/+git/cloud-init/+merge/303471/comments/784467
<prometheanfire> I'm just not used to dealing with this much indirection and flow and functions that don't really do aything useful
<prometheanfire> smoser: so you need me to do something?
<smoser> prometheanfire, if you can push to the shorter version, it'd be nice. its just more consistent. but if not , oh well.
<smoser> wrt the indirection, let me see if there is some sane way to use that for you.
<prometheanfire> thanks
<prometheanfire> this is java like python from what I can see :P
<prometheanfire> smoser: guess that's what you wanted (push wise)
<smoser> prometheanfire, yeah, its all there to try to make things easier. :-(
<smoser> take a look at eni.py and see if that is any easier for you to wor off of than sysconfig.py.
<smoser> the network state thing doesn't really provide a lot.
<prometheanfire> ya
<smoser> the interfaces are just like the network-config object that went in... you can iter_interfaces()
<prometheanfire> I'll take another look tomorrow
<prometheanfire> today is really full
<smoser> and iter_routes()
<smoser> it shouldnt' be really too much more difficult to use that than to walk through the network_config object
<prometheanfire> ya, it's more how they construct each network stansa that is hard
<prometheanfire> no string formating :P
<prometheanfire> I'll see what I can do I guess
<prometheanfire> tomorrow
<smoser> prometheanfire, thank you
<harlowja> prometheanfire yes its not optimal ;)
<harlowja> i did a first stab at making it better, prometheanfire any more u want to do is welcome :)
<harlowja> (and yes its confusing as all-hell, lol)
<prometheanfire> ya
<harlowja> 3 different layers of translation and processing
<harlowja> for <?> not really sure why, ha
<prometheanfire> yep
#cloud-init 2016-08-25
<harlowja> looks like python2.6 is busted again
<harlowja> lol
<harlowja> https://gist.github.com/harlowja/7e1e7106e81f5aa809645d76dd74bddd
<prometheanfire> meh, no one really uses that :P
<prometheanfire> is that still maintained even for sec bugs?
<harlowja> python 2.6, its prob not
<prometheanfire> :P
<harlowja> sadly cent6 is still maintained
<harlowja> and cloud-init 0.7.x still says 2.6 supported (i'll have to support it no matter, ha)
<harlowja> simple stuff like data = {v: "UNAVAILABLE" for v in fmap.values()} though isn't hard to fix
<harlowja> just a PITA that its not getting detected earlier
<harlowja> oddly though even 2.7 seems currently busted @ https://gist.github.com/harlowja/3ea5fd7ad3739704e3298ab35ceefd1c
<harlowja> thats on rhel7
<ffledgling> smoser: I figured out the problem, it was in my yaml :|
<ffledgling> Sorry for all the trouble
<ffledgling> I also did change the qemu-kvm boot options and added a `-boot dc` to elimnate the race condition
<smoser> ffledgling, well, fwiw, -boot dc wont really fix any race.
<smoser> the race happens because cloud-init doesn't attempt to wait for disks that aren't there to become attached (as they may never become attached)
<smoser> and by not waiting, linux might not have enumerated them yet.
<ffledgling> smoser: I see
<smoser> as i said though, i've never seen it in practice.
<ffledgling> smoser: got it
<ffledgling> I really ended up linking cloud init once I got it working correctly. It was just strange when I started with it
<smoser> ffledgling, thanks.
<smoser> yaml is indeed easy to mis-type
<smoser> one very simple thing you can do is just:
<smoser> python -c 'import yaml, sys, pprint; pprint.pprint(yaml.load(open(sys.argv[1])))'
<smoser> and if it stack traces, thats bad yaml
<smoser> it doesn't do any further checking obviously, and we want to better lint or schema data, but do not at the moment
<ffledgling> smoser: Yeah, I think my problem was I was using a makefile to generate the yaml on the fly and I accidently did the thing where I converted tabs to spaces, which resulted in incorrect indentation.
<ffledgling> I think python on the whole doesn't have great support for YAML
<smoser> ?
<smoser> it has really good support for yaml
<ffledgling> in terms of identifying and point out syntax errors I mean
<smoser> tabs and spaces are by yaml specification not intermixable
<smoser> yeah, the loader doesn't give you a ton of good info.
<smoser> one thing you *can* do is just use json
<smoser> yaml is a strict superset of json
<ffledgling> the builtin json module is pretty bad as well, and very unforgiving iirc (simplejson is better I think?)
<ffledgling> Does cloud init support json?
<ffledgling> Is automatic package installation a new feature (post 0.7.6) ? For whatever reason the packages directive doesn't work for me
<Odd_Bloke> ffledgling: JSON is valid YAML, I believe, so yes. :)
<ffledgling> Ignore my old complaint about installation not working, it was just taking time and running in the background
<smoser> Odd_Bloke, yes.  YAML is 100% strict superset of JSON
<smoser> its basically json with some nice human features like:
<smoser>  a.) "anchors" which are like references and really nice
<smoser>  b.) easily mis-typable in non-obvious ways
<smoser> magicalChicken, if you're lookin for things... on cloud-init cleanup
<smoser> one thing that would be nice.. in nplaces we stack trace and put a failure in the log (at warn level)
<smoser> but the strings are all one line and very hard to read
<smoser> it'd be nice if put well formatted human readable stack traces somewhere on failure.
<magicalChicken> smoser: right yeah that makes sense
<magicalChicken> similar to the issue with curtin util.ProcessExecutionError
<magicalChicken> I'll figure out how to fix the formatting there for cloud-init, shouldn't be too difficult
<harlowja> @smoser if u get a chance https://code.launchpad.net/~harlowja/cloud-init/+git/cloud-init/+merge/304001
<harlowja> that i think should fix the 2.6 occurence that i think is busted
<harlowja> or at least one of them, lol
<harlowja> though i think there are more, lol
<harlowja> ie https://gist.github.com/harlowja/4ee5f7da70e422d918f3bfa8a5f2bf33
#cloud-init 2016-08-26
<smoser> harlowja, how do i stop this from happening
<smoser>  aka, please show me how to run your tox-runner thing.
<smoser> i can probably also manage a python2.6 tox runner... actually, powersj might be able to help with something
<Tim_> hi
<Tim_> could someone provide the steps to create a cloud-init ready windows image?
<cpaelzer> Tim_: do you mean a "windows image" like Windows Server 2012/2016 or "windows image" like in Ubuntu or such running on Azure (which isn't a windows image IMHO)?
<Tim_> oh, sorry. :-) I'm using xenserver with xen-orchestra, and I'd like to create a cloud ready windows 2012r2 template
<cpaelzer> Tim_: this is kind of out of my scope, but AFAIK the cloud init this channel is for has no native windows function
<cpaelzer> smoser: is this correct or did I miss something ^^?
<Tim_> I found win images on cloudbase, but I'm not sure if it's the same thing
<cpaelzer> Tim_: it is a different thing
<cpaelzer> Tim_: what you refer to is based on https://github.com/openstack/cloudbase-init I think
<powersj> smoser, I could investigate a 2.6 tox - how urgent?
<smoser> powersj, well... you woudlnt be able to run it in c-i until c-i supports git :)
<smoser> so, less urgent than that i think.
<powersj> ah true :) I could at least see what it takes to add it to tox.ini
<smoser> powersj, so, the tircky thing in ubuntu is *getting* py26
<smoser> https://launchpad.net/ubuntu/+source/python2.6
<smoser> no python2.6 in any supported ubuntu
<smoser> that is how old it is
<smoser> there is https://launchpad.net/~fkrull/+archive/ubuntu/deadsnakes?field.series_filter=vivid
<powersj> lol
<smoser> i've used the dead-snakes before
<cpaelzer> really - dead snakes
<cpaelzer> ... on a plane I guess
<cpaelzer> powersj: if you want to do the triaging session now please let me know
<powersj> cpaelzer, give me 5 to finish breakfast
<cpaelzer> powersj: so the world collides with breakfast vs dinner constraints :-) no hurry at all - just ping me when you are really ready
<powersj> haha ok :)
<powersj> cpaelzer, shall we :)
<cpaelzer> powersj: yep, powering up HO
<cpaelzer> powersj: somewhere a bell should ring :-)
<cpaelzer> powersj: but to easen this https://hangouts.google.com/hangouts/_/canonical.com/triagethepainuhbugs
<harlowja> smoser compile it in a jenkins job then test against that :-P
<harlowja> powersj smoser feel free to take https://gist.github.com/harlowja/6329b413a7572417148d9a292330f9bb :-P
<harlowja> its the jenkins pipeline groovy stuff that i'm using
<harlowja> +- some other functions u aren't seeing there, lol
<harlowja> but if u just using jenkins
<harlowja> then add in to that groovy there a step to "build-python26"
<harlowja> but sure smoser if u want to use the remote-tox stuff
<harlowja> i can show u :-P
<harlowja> few ways to skin this cat
<smoser> didnt you show me something that you had? a remote tox runner with py26 ?
<harlowja> ya, but u still need a machien somewhere with 26, ha
<harlowja> https://github.com/harlowja/remote_tox#remotetox
<harlowja> not super complicated stuff ^
<smoser> harlowja, ah. i thought you were giving us said machine
<harlowja> lol
<smoser> aren't you running a free cloud platform ?
<harlowja> errrr
<smoser> doesnt it cost like $1 year  for my own domain ?
<harlowja> something like that, ha
<smoser> just run me a domain with py26 and bill your employer.
<harlowja> hmmmm
<harlowja> tempted to try that, lol
<smoser> tox-for-dead-snakes.com is still open
<harlowja> neat, i think travis still does 26 also
<harlowja> so maybe travis can be plugged in
<harlowja> at least to run it and (report back here?)
<smoser> y
<harlowja> let me see about that
<harlowja> or if others have time, they are welcome to also :-P
<harlowja> may just involve either a .travis.yml file in cloud-init
<harlowja> or telling travis what to run
<harlowja> something like https://github.com/jd/tenacity/blob/00fe1f1cd026a3b2a8c641c12552bf5cfe945990/.travis.yml
<harlowja> (for example)
<smoser> harlowja, random python mock question
<smoser> class Foo(SuperClass):
<smoser>    pass
<smoser> how can i replace Superclass with something else when testing Foo
<smoser> i used @mock.patch("path.of.import.SuperClass", new=something.else)
<smoser> that worked for testing SuperClass
<smoser> but then i tried to test Foo similarly, and its reference to its SuperClass is already made so i couldnt replace it
<smoser> (i think)
<harlowja> eck
<harlowja> not sure
<harlowja> never tried that
<harlowja> why u doing that, lol
<harlowja> don't do that, hahaha
<smoser> ok
<harlowja> annnnd smoser
<harlowja> https://github.com/kubernetes/kubernetes/pull/30757#issuecomment-242795731
<harlowja> u may be interested in that
<harlowja> something about openstack + k8s and they are reading /var/lib/cloud
<harlowja> and https://github.com/kubernetes/kubernetes/pull/30757#issuecomment-242770372
<smoser> what we want there is the cloud-inti query
<smoser> thats what i want to reply to that with
<harlowja> right
<harlowja> reply with that ;)
<smoser> or well formed json in that dir
<harlowja> right
<harlowja> but yes, u should reply, ha
<harlowja> @smoser any idea bout https://gist.github.com/harlowja/5c67d774a65c3f5c2b8f342ef78c58a0 ?
<harlowja> lol
<harlowja> hmmmm, looks like travis is to much connected into github to be useable outside very easily
<harlowja> so i mean i could sync into github i guess :-/
<harlowja> and then use travis
<harlowja> puke
<harlowja> @smoser are u ever syncing https://github.com/cloud-init/cloud-init ?
<harlowja> perhaps can plug that into travis if so
<harlowja> i think a mirror from the offical to there would let travis then run
#cloud-init 2016-08-27
<smoser> harlowja, i'm fine to try to mirror to github
<smoser> i'm not here really. but that is fine with me.
<harlowja> who are u
<harlowja> :-P
#cloud-init 2017-08-21
<Prerak> Hello
<Prerak> Do we have any guide to use cloud-init on freebsd?
<smoser> blackboxsw, http://paste.ubuntu.com/25363324/
<smoser> that'd be the yaml like syntax that i thought might throw you fits
<smoser> and then second, like this: http://paste.ubuntu.com/25363330/
<smoser> those are yaml "anchors"
<blackboxsw> smoser: that might actually throw jsonschema fits (self-referential stuff & json) hmm
<blackboxsw> good examples. I'll play with them smoser
<smoser> well, they're valid to json schema
<smoser> because yaml.load() returns you a dictionary with all things "copied out"
<smoser> ie, the result of cloudinit.safeyaml.load() is always somethign that jsonschema can look at. json schema doesn't actually look at the "S" in JSON ("serialized"), but rather it looks at the "O" ("object").
<smoser> and yaml.load() returns a object
<blackboxsw> ok smoser rharper all changes pushed to the analyze branch https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/328819 CI works
<rharper> k
<rharper> smoser: if the sbuild of the new release package is good, do you want us to push the upstream tag ?
<sabari> Folks, cloud-init with config drive on openstack instance incorrectly sets the dns-nameservers on the loopback interface. Rest of the configuration is correctly set on eth0. Any ideas to troubleshoot this ?
<sabari> I am running Ubuntu 16.04
<rharper> sabari: it's not an incorrect setting, are you not seeing the values show up in /etc/resolv.conf ?
<rharper> the resolvconf package will read any of the dns-* settings across any interface, including lo;
<smoser> rharper, i guess you could propose a merge
<smoser> and i could then pull, build, push
<rharper> smoser: ok, let me see if I can get that working
<rharper> I suspect I'll need to push my tags to my remote branch
<smoser> right. i guess the only tag that matters is ubuntu/<...>
<smoser> and i can just as well tag that and push
<sabari> rharper: /etc/resolv.conf is empty.  When I use cloud-init with metadata service, the dns-nameservers are set on eth0 and hence I expected the same with configdrive as datasrouce.
<smoser> sabari, that is odd.
<smoser> well, or maybe not.
<smoser> so the difference between info from the openstack api and the config drive is that cloud-init will read all the networking config from the config drive and apply it form config drive.
<smoser> but it does not currently do that for the metadata service.
<rharper> smoser: configdrive *may* include an eni  vs. network_data.json
<rharper> and IIRC< it prefers the eni vs network_data.json ; where as meta-data service, only has network_data.json ?  do I have that right?
<rharper> ISTR an issue w.r.t an 'eni' in the config and a 'network_data.json'
<rharper> and cloud-init had to make a choice w.r.t which one has precendence
<rharper> possibly the eni included does not match what's in network_data.json
<sabari> rharper: I only see network_data.json in the mounted iso. I don't see an ENI. Let me check again.
<rharper> if you can, could you share the network_data.json ?  I can render that locally and see what eni cloud-init should generate
<sabari> rhaper: sure
<sabari> rharper: http://paste.openstack.org/raw/618941/
<smoser> well...
<smoser> so cloud-init is putting those on 'lo' as they're "global" dns servers.
<smoser> theres no specific data that says these are attached to eth0.
<smoser> obviously you could come up with rules to figure that out but cloud-init does not do that at the moment.
<sabari> smoser: I see. So the global dns-servers are set only on the lo interface. rharper mentioned that resolconf is supposed to read it and set to /etc/resolv.conf, let me try to check that.
<smoser> well, yeah. resolvconf will apply dns nameservers
<smoser> on each interface as it comes up
<smoser> and as 'lo' came up, and had dns-nameservers, it should have made its way into /etc/resolv.conf
<rharper> smoser: hrm, the openstack helper makes local renedering unhappy because of that known_macs feature
<rharper> net-convert used to work on those files; will need to fix that
<smoser> ?
<smoser> it will work, you just have to pass in the macs
<rharper> right
<rharper> that's work for net-convert
<rharper> to pull them out of the json and pass it to the method for rendering
<rharper> bbiab
<smoser> right. you could do that.
<sabari> smoser, rharper: thanks for all the info. Let me debug what's happening with resolconf and get back.
<powersj> rharper: looks like there is support in cloud-init for setting a sortlist in resolv.conf however LP: #1711967 says there is an issue with that
<ubot5> Launchpad bug 1711967 in cloud-init "dns-sortlist" [Undecided,New] https://launchpad.net/bugs/1711967
<smoser> rharper, sabari note, this is upstream related
<smoser>  https://review.openstack.org/#/c/467699/
<smoser> ie, them fixing the dns info to be per-interface
<smoser> i had started cloud-init chagnes at http://paste.ubuntu.com/25220242/
<sabari> smoser: Thanks a lot for the upstream patch! It will try it out and hopefully that will fix the problem :)
<smoser> sabari, well, cloud-init does not support it yet.. eventually it would.
<sabari> smoser: Ah, I see. So this also needs the corresponding cloud-init fix that you referenced.
<sabari> smoser: Little confused here..If I patch nova with the upstream change, the existing cloud-init should be able to set the nameserver on eth0 as well right ? Or does I need the changes you are working on.
<sabari> *do I
<smoser> you'd also need cloud-init changes to take advantage of the upstream change.
<smoser> the upstream change was basically upstream realizing "hey, dns servers generally available per-interface."
<smoser> blackboxsw, https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/328241
<blackboxsw> checking
<smoser> did you see ajogen's comments ?
<blackboxsw> hrm saw all of his initial comments, didn't see that last one
<rharper> smoser: https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/329324   see if I did that right;  I pushed with --tags which pushed *all* my tags; not sure if that's correct
<nacc> rharper: fwiw, your tags are relatively irrelevant to a git merge
<nacc> rharper: not sure entirely on the context, just an fyi
<rharper> nacc: yeah; I figured;  we're documenting upstream release into ubuntu/devel process, so it includes a few tag steps that are meant for :master but if I'm preparing the changes in a branch ,  hoping to bring the needed single tag along but we'll have to see what works here
<nacc> rharper: so you can push a single tag with `git push <remote> <tag>` and git should figure it out (tags are a special ref case). You can also push with a refspec `git push <remote> refs/tags/<tag>:refs/tags/<remotetagname>'
<nacc> rharper: yeah, what you  wrote makes sense, though -- if you need any help, let me know
<rharper> nacc: ah, that make sense (for single tag push0
<rharper> )
<smoser> rharper, you picked the wrong target
<smoser> for the merge
<smoser>  https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/329324
<smoser> you dont want to 'Merge into:' target
<smoser> you want into ubuntu/artful
<nacc> or ubuntu/devel, probably?
<smoser> right
<nacc> they presumably point to the same thing
<nacc> our general hope is for 'ubuntu -> ubuntu merges', ubuntu/devel can always be the target
<nacc> and for ubuntu -> debian merges, debian/sid can always the target
<rharper> smoser: ah, right
<rharper> do I just kill the MP and resubmit ?
<nacc> rharper: i think you can just resubmit it from the web UI and pick a new target
<smoser> rharper, i rejected it :). you can't (or i couldnt) change the target. maybe you can, but resubmit with different target is as easy
<nacc> smoser: yeah, user can resubmit and change it, iirc, but not with the existing MP
<rharper> smoser: k
<rharper> hrm, LP doesn't like ubuntu/artful
<nacc> rharper: did you get an error? i see it existing: https://code.launchpad.net/~usd-import-team/ubuntu/+source/cloud-init/+git/cloud-init/+ref/ubuntu/artful
<rharper> nacc: I did, complained that it didn't recognize ubuntu/arful ;   I used ubuntu/devel  instead
<rharper> artful
<rharper> even
<nacc> rharper: hrm, strange
<rharper> smoser: nacc: hrm, so it doesn't like ubuntu/* whatever the first time;  it reports an error, but if you just resubmit, then it works. ..
<rharper> maybe a git/launchpad integration error ?
<rharper> smoser: https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/329332
<smoser> rharper, you dont have a ubuntu/0.7.9-243-ge74d775-0ubuntu1 tag pushed.
<smoser> https://git.launchpad.net/~raharper/cloud-init/refs/tags?h=ubuntu-devel-new-artful-release-v2
<smoser> not a big deal, i can just tag and push there.
<smoser> i think what you wanted would have been:
<smoser>  git tag ubuntu/0.7.9-243-ge74d775-0ubuntu1
<smoser>  git push <your-repo-name> HEAD ubuntu/0.7.9-243-ge74d775-0ubuntu1
<rharper> smoser: well, I wanted to push it, but I couldn't quite figure out how to specify my branch remote;
<rharper> smoser: I've done the tag
<rharper> but not sure how to push with -u raharper (which expands to the lp remote)
<rharper> smoser: that last one worked ( had to look up my remote name )
<rharper> maybe I was just missing the HEAD , git push -u raharper HEAD <tag>
<smoser> head would have been same as 'ubuntu/devel'
<smoser> git push raharper ubuntu/devel ubuntu/0.7.9-243-ge74d775-0ubuntu1
<smoser> or
<smoser> git checkout ubuntu/devel && git push raharper HEAD ubuntu/0.7.9-243-ge74d775-0ubuntu1
<smoser> and i just uploaded that and pushed.
<rharper> cool
<nacc> rharper: the syntax you listed `git push remote HEAD <tag>` doesn't do what you think most likely :)
<nacc> well, it depends on what you're trying to do, i suppose
<nacc> you very rarely push HEAD explicitly like that
<rharper> nacc: I think you want to ping smoser; I was blindly trying what smoser suggested;  I think smoser has resolved to something more repeatable with the git push <lpuser> ubuntu/devel (or whatever my local branch with the change sare on) <tag name I just tagged>
<rharper> which should work regardless of what HEAD is pointing at (which I think is your point)
<nacc> rharper: right, you want to push a local branch and tag?
<rharper> yeah
<nacc> rharper: and use the same naming on the remote, it seems like?
<rharper> yes
<nacc> rharper: yep `git push <remote> <branch name> <tag name>` is perfect then. And by default, with `git ubuntu`, the remote name is your lp username
<smoser> rharper, https://lists.ubuntu.com/archives/artful-changes/2017-August/008659.html
<rharper> smoser: thanks!
<smoser> rharper, it is easy to think that
<smoser>  git push <remote> <local-hashish> <remote-name>
<smoser> but that is not the syntax
<smoser> its
<smoser>  git push <remote> <local-hashish>[:<remote-name>] [...]
<rharper> I know of the colon; I used that often for branches
<rharper> it's the tag that was confusing
<nacc> yeah
<rharper> I guess tag goes in [...] ?
<nacc> just keep in mind that everything after remote is a refspec
<rharper> git push raharper my-local-branch:my-remote-branch-name <Tag> ?
<nacc> so any refspec is allowed and those without colons are degenerate cases
#cloud-init 2017-08-22
<niluje> hi, any idea when cloud-init will be released? :p
<smoser> niluje, :-(.
<niluje> don't be sad :(
<niluje> what did I tell wrong :(
<smoser> you just pointed out how delinquent i am
<smoser> i really do feel like we're rpobably acceptably in a release-able state.
<smoser> it just always gets put on the back burner.
<smoser> niluje, is there something in particular you're wanting from a release ? other than just a stamp ?
<niluje> a stamp and a hope to see it eventually included in distribs
<smoser> thats quite reasonable.
<smoser> rharper, done with our gce instance for the moment.
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/329387
<rharper> smoser: k
<smoser> well... done. you can kill it.
<rharper> fwiw, a 17.04 instance (and I've not launched another xenial) it was like 3 seconds
<smoser> it seems to me that their metadata service caches. and first hits to it are in fact slow.
<rharper> so *bad* luck
<rharper> y
<smoser> you can see it. cd /home/ubuntu/cloud-init
<smoser>  python3 -m cloudinit.sources.DataSourceGCE
<rharper> smoser: if you're touching, we should switch HTTP header to the current preferred
<rharper> Metadata-Flavor: Google
<rharper> https://cloud.google.com/compute/docs/storing-retrieving-metadata#querying
<blackboxsw> smoser: think I've addressed all schema subcommand branch comments: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/329233
<blackboxsw> updated the branch description to capture all changes
<smoser> "This header indicates that the request was sent with the intention of retrieving metadata values, rather than unintentionally from an insecure source,"
<smoser> yeah... that provides security. only rootkits that can change their headers can hit that md.
<smoser> rharper, can you re-submit your comment to https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/329397
<smoser> (just copy from https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/329152)
<smoser> err..
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/329387
<smoser> blackboxsw, i approved https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/329233
<smoser> but c-i needs to
 * smoser out
<blackboxsw> smoser: thanks. see ya
<rharper> smoser: ok
<rharper> smoser: done
<sabari> smoser: rharper: I was finally able to fix the missing dns-namservers on eth0 in the ENI file (nova instance with config drive on Ubuntu 16.04) using a custom nova patch.
<sabari>  I did this in nova http://paste.openstack.org/raw/619103/ which results in the following network_data.json http://paste.openstack.org/raw/619101/
<rharper> sabari: \o/
<sabari> I am not sure if this is right but it fixed my problem for now. Since the upstream patch that smoser referred https://review.openstack.org/#/c/467699/ needs a corresponding cloud-init fix that is yet to be merged.
<sabari> I will file a LP bug just to track this issue. Feel free to close it if you feel it's already being worked upon.
<blackboxsw> sabari: that'd be super
<blackboxsw> it helps us track/prioritize
<sabari> Filed https://bugs.launchpad.net/cloud-init/+bug/1712440
<ubot5> Ubuntu bug 1712440 in cloud-init "Missing dns-nameservers in eth0 with config drive on openstack instance" [Undecided,New]
<sabari> blackboxsw: ^:)
<blackboxsw> +1 sabari
<blackboxsw> thanks
#cloud-init 2017-08-23
<sde> I need to override the search domain set through DHCP, but of course resolv.conf is generated by NetworkManager from /etc/sysconfig/network-scripts/ifcfg-eth0, which in turn is generated by cloud-init. Been digging, but can't get cloud-init to alter ifcfg-eth0.
<sde> This on a Redhat image in AWS
<rharper> I don't think the RedHat AWS image has cloud-init new enough to generate networking config; it's likely the ifcfg-eth0 is baked into the image as-is (dhcp on eth0 was a common default in cloud-images).    It may be possible to have a dhcp hook script which can append/prepend DNS settings
<sde> The image I'm working on is using cloud-init 0.7.9, which seems to tie with the docs I was looking at that suggested this was possible, and at the top of the ifcfg-eth0 it states it was generated by cloud-init.
<sde> I've tried putting in cloud.cfg -
<sde> manage_resolv_conf: true  resolv_conf:   searchdomains:     - vpc01.grn.poc.tbaws.com   domain: vpc01.grn.poc.tbaws.com
<sde> (mangled format in cut-n-paste, sorry)
<rharper> Ah, 0.7.9 is new enough
<rharper> it maybe that the fallback network configuration doesn't look at those cloud-configs;  would you be able to file a bug against cloud-init on launchpad with details? https://bugs.launchpad.net/cloud-init/+filebug
<sde> Sure, I can do that. Have I got the settings in the right location and format through? I've never looked at cloud-init before today, and I got confused reading the docs about exactly where I should be putting things
<rharper> sde: w.r.t format, not quite sure since your paste didn't have newlines.  but I suspect you want it to look something like: http://paste.ubuntu.com/25376871/
<sde> Yes, that's exactly what I've got. Directly in cloud.cfg is the correct/acceptable location for that, rather than as a separate file under cloud.cfg.d/ ?
<sde> Need to step away, but thanks for your assist, appreciate the time. I'll get that bug raised later. Thanks
<smoser> rharper, wrt "what image for development release on ec2"
<smoser>  http://paste.ubuntu.com/25376930/
<rharper> ah, search by ami number
<smoser> image status is https://github.com/smoser/talk-simplestreams/blob/master/bin/image-status
<smoser> rharper, blackboxsw fyi, the instance-data roadmap entry
<smoser>  https://trello.com/c/AYaCdQyT
<rharper> k
<blackboxsw> rharper: /smoser trial run  ipv6 from AWS metadata :) https://us-east-2.console.aws.amazon.com/ec2/v2/home?region=us-east-2#Instances:sort=desc:statusChecks
<blackboxsw> oops
<blackboxsw> http://paste.ubuntu.com/25378444/
<blackboxsw> looks like generating the network V1 worked
<blackboxsw> I'm throwing together some unit test coverage
<blackboxsw> ehem, I mean I'm all TDD and must already have the unit tests around some place
#cloud-init 2017-08-24
<pester> hi all, I`m trying to use cloud-init
<pester> with configdrive
<pester> I can see partition with data is present in machine, but not mounted and in logs "no module "DataSourceConfigDrive"
<rharper> pester: do you have your /var/log/cloud-init.log ?    no module sounds like a packaging issue , are you using official cloud-images or rolling your own?
<pester> I use ubuntu cloud image for Ironic
<pester> I can see DataSourceConfigDrive.pyc in sources
<pester> can you tell what module is doung mount configdrive to /media ?
<rharper> cloud-init doesn't mount the config drive to /media;  it will search for filesytems with the 'ci-data' label and mount them to a tmp directory to read and consume the metadata
<rharper> then umount the drive
<pester> hm, ironic generates fs with label 'config-2'
<rharper> I think that's also valid
<rharper> yes
<rharper> cloudinit/sources/DataSourceConfigDrive.py:28:LABEL_TYPES = ('config-2',)
<rharper> cloud-init won't leave the config drive mounted, that's expected.  Is something else wrong other than the no module message?
<pester> yes, cloud-init trying to mount it with mount -o ro,sync - but it fails, correct mount comand -o loop,ro,sync
<rharper> it's a file?
<rharper> I don't think ConfigDrive expects to mount a file; it's a block device with a filesystem label; but let me read the code
<pester> it is block device
<rharper> loop means it's a file
<rharper> open the file with the loop driver (which takes files and makes them block devices)
<pester> it is partition with iso9660 fs
<rharper> right, so no need for loop
<pester> without loop  - mount fails with 'already mounted or /mnt is busy'
<rharper> maybe it's already mounted?  ?
<rharper> maybe something is mounting it before cloud-init can ?
<rharper> if you can attach your boot journal and cloud-init.log to a bug ; that'd be helpful and we can figure out what's going wrong.  https://bugs.launchpad.net/cloud-init/+filebug ,  I've got to run for now.
<pester> I can`t see any mount
<rharper> be back on in about an hour
<pester> https://bugs.launchpad.net/cloud-init/+bug/1712851
<ubot5> Ubuntu bug 1712851 in ironic-python-agent "cloudinit can`t mount configdrive partition" [Undecided,New]
<smoser_> blackboxsw, http://paste.ubuntu.com/25384304/
<smoser_> whoops
<smoser_> :-(
<smoser_>  python3 -m cloudinit.analyze blame
<smoser_> ^ that does work, but it seems like just 'cloud-init analyze' is busted.
<blackboxsw> smoser: I have a branch that's up for review to fix that
<blackboxsw> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/329493
<smoser> \o/
#cloud-init 2017-08-25
<pester> smoser: hi, are you here ?
<pester> regarding this - https://bugs.launchpad.net/cloud-init/+bug/1712851
<ubot5> Ubuntu bug 1712851 in ironic-python-agent "cloudinit can`t mount configdrive partition" [Undecided,New]
<pester> I`m currently have this environment and can do tests if anyone interested in debugging this
<smoser> blackboxsw, _maybe_remove_legacy_eth0
<blackboxsw> larsks: tests/unittests/test_handler/test_schema.py
<smoser> powersj, https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/329616
<smoser> why didn't ci bot run on that ?
<powersj> I do not know, looking atm
<powersj> smoser: ^
<powersj> "DEBUG: Users "rjschwei" not allowed to trigger jobs"
<powersj> I need to add him to the white list, but I thought anyone who has signed the CLA was on it.
<powersj> I can't get onto the VPN right now, keeps timing out, so I can launch it manually
<smoser> powersj, he's not signed the cla himself. suse has
<smoser> so that would be it.
<powersj> ahh! ok at least we can explain it
<smoser> rharper, https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/329122
<smoser> Currently the python logging module will default to a local time which may contain an TZ offset in the values it produces. Switching to UTC time for logging produces consistent values in the cloud-init.log file and avoids issues when the timezone is changed during boot.
<smoser> nevermind. i'll update that to mention that logging did not get a timezone.
<smoser> blackboxsw, https://code.launchpad.net/~larsks/cloud-init/+git/cloud-init/+merge/329620
<smoser> can you review that ? did that look fine to you ?
<blackboxsw> smoser: yeah this is the branch larsks is working currently to add a unit test around this behavior (but he hit the httppretty issue before pushing the commit)
#cloud-init 2017-08-26
 * larsks figured out dependent merge requests in launchpad
<larsks> Yay!
<larsks> I don't know what to do with the error from launchpad:
<larsks> Launchpad encountered an internal error during the following operation: emailing a reviewer requesting a review.  It was logged with id OOPS-fa154f4726bb5e94f22e09b5ea7fd2e0.  Sorry for the inconvenience.
#cloud-init 2018-08-20
<blackboxsw> hey folks, I think it's about time for our bi-weekly cloud-init  status meeting.
<blackboxsw> #startmeeting
<meetingology> Meeting started Mon Aug 20 16:08:48 2018 UTC.  The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology.
<meetingology> Available commands: action commands idea info link nick
<blackboxsw> #endmeeting
<meetingology> Meeting ended Mon Aug 20 16:08:52 2018 UTC.
<meetingology> Minutes:        http://ubottu.com/meetingology/logs/cloud-init/2018/cloud-init.2018-08-20-16.08.moin.txt
<blackboxsw> #startmeeting Cloud-init bi-weekly status meeting
<meetingology> Meeting started Mon Aug 20 16:09:08 2018 UTC.  The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology.
<meetingology> Available commands: action commands idea info link nick
<blackboxsw> For those around, let's kickoff another cloud-init status meeting.  Feel free to interrupt as we go through the agenda for today.
<blackboxsw> agenda: Previous Actions, Recent Changes, In-progress development and office hours (~30 minutes)
<blackboxsw> #topic Previous Actions
<blackboxsw> nothing to speak of here as far as I recall.
<blackboxsw> #topic Recent Changes
<blackboxsw> We have recently landed the following content in tip of cloud-init over the last 2 weeks
<blackboxsw>   - Add datasource Oracle Compute Infrastructure (OCI).
<blackboxsw>   - azure: allow azure to generate network configuration from IMDS per boot.
<blackboxsw>   - Scaleway: Add network configuration to the DataSource [Louis Bouchard]
<blackboxsw>   - docs: Fix example cloud-init analyze command to match output.
<blackboxsw>     [Wesley Gao]
<blackboxsw>   - netplan: Correctly render macaddress on a bonds and bridges when
<blackboxsw>     provided. (LP: #1784699)
<blackboxsw>   - tools: Add 'net-convert' subcommand command to 'cloud-init devel'.
<blackboxsw>   - redhat: remove ssh keys on new instance. (LP: #1781094)
<ubot5> Launchpad bug 1784699 in cloud-init "cloud-init not setting mac address for bond or bridge in bionic" [Medium,Fix committed] https://launchpad.net/bugs/1784699
<blackboxsw>   - Use typeset or local in profile.d scripts. (LP: #1784713)
<blackboxsw>   - OpenNebula: Fix null gateway6 [Akihiko Ota] (LP: #1768547)
<ubot5> Launchpad bug 1781094 in cloud-init "cloud.cfg.tmpl should not include "ssh_deletekeys: 0"" [Medium,Fix committed] https://launchpad.net/bugs/1781094
<ubot5> Launchpad bug 1784713 in cloud-init (Ubuntu) "cloud-init profile.d files use bash-specific builtin "local"" [Low,Confirmed] https://launchpad.net/bugs/1784713
<ubot5> Launchpad bug 1768547 in cloud-init (Ubuntu) "OpenNebula DataSource adds null gateway6 to netplan config" [Medium,Confirmed] https://launchpad.net/bugs/1768547
<blackboxsw> most notable is the new datasource for Oracle and Azure datasource now emitting network configuration per boot.
<blackboxsw> new Ubuntu cosmic images should contain 18.3-24-gf6249277-0ubuntu1 with the lastest patches
<blackboxsw> #topic In-progress Development
<blackboxsw> The team in general is perparing for the cloud-init summit conference which will be the second half of this week with cloud and distro vendors so we'll probably be landing a couple more branches in preparation for talks/demos there.
<blackboxsw> *preparing* rather
<blackboxsw> Our trello board is a good representation of any existing work we are "Doing"
<blackboxsw> #link https://trello.com/b/hFtWKUn3/daily-cloud-init-curtin
<blackboxsw> recently powersj has also moved our jenkins server around to a new network so there has been a good bit of work getting all things jenkins back up and running behind the new proxies/firewalls etc.
<blackboxsw> thanks for the heavy lift powersj
<blackboxsw> I think that wraps up all things cloud-init....
<blackboxsw> #topic Office Hource (next ~30 mins)
<blackboxsw> We'll have eyes on this channel for any quips, complaints, bug, feature or branch discussions for the next half hour
<blackboxsw> Otherwise, we'll be excited to see a few of you again in this year's cloud-init summit.\
<paulmey> Hi all, I'd like to request a review for this MP: https://code.launchpad.net/~andyliuliming/cloud-init/+git/cloud-init/+merge/351742 (attached to LP: #1722959)
<ubot5> Launchpad bug 1722959 in cloud-init "Implement Key-Value Pair Telemetry for Azure" [Undecided,In progress] https://launchpad.net/bugs/1722959
<paulmey> Hoping we can get this merged sometime soon. :-)
<blackboxsw> Hi paulmey thanks for the ping on this.
<blackboxsw> #link https://code.launchpad.net/~andyliuliming/cloud-init/+git/cloud-init/+merge/351742
<blackboxsw> #action rharper/blackboxsw close out on this review before cloud-init summit https://code.launchpad.net/~andyliuliming/cloud-init/+git/cloud-init/+merge/351742
<meetingology> ACTION: rharper/blackboxsw close out on this review before cloud-init summit https://code.launchpad.net/~andyliuliming/cloud-init/+git/cloud-init/+merge/351742
<blackboxsw> that has gotten dusty, thank you
<blackboxsw> I'll start a review in earnest now
<paulmey> Thanks. The dust is mostly mine... Andy has been working on this, but he's in a different time zone, so I'm still pushing it...
<paulmey> ð
<blackboxsw> paulmey: I'll try spinning up an azure vm to test this out
<paulmey> let me know if you need anything
<blackboxsw> ok thanks folks for tuning in. See you next time
<blackboxsw> #endmeeting
<meetingology> Meeting ended Mon Aug 20 17:04:48 2018 UTC.
<meetingology> Minutes:        http://ubottu.com/meetingology/logs/cloud-init/2018/cloud-init.2018-08-20-16.09.moin.txt
<blackboxsw> just published minutes to https://cloud-init.github.io/
<blackboxsw> smoser: rharper do you guys recall what platforms we expect don't have jinja2 support?
<rharper> blackboxsw: hrm, I feel we discussed but I don't recall the details
<blackboxsw> it's an optional dependency from the looks of things in cloudinit/templater.py
<rharper> there was a reason we had the simple renderer
 * blackboxsw was thinkings centos 6... but python-jinja2-2.2.1-3.el6.x86_64 exists... hrm
<blackboxsw> many non-epel customer or something
<smoser> it was probably just added as optional because we didn't want to break existng installations (or supported/sttable OS... like trusty or something)
<rharper> related to cheetah being dropped too
<rharper> which was pre jinja IIRC
<smoser> blackboxsw: why do you care ?
<blackboxsw> in jinja template parthandler, I'm trying to decide if I can rely on jinja2 module to exist or do the try/except ImportError dance
<smoser> in this scenario, when rendering the thing, the user has declared the template to be jinja
<smoser> right?
<blackboxsw> yeah, they have
<blackboxsw> so that shouldn't run unless ##template: jinja is in user-data
<smoser> well, its going to fail either way.
<smoser> user is not going to get what they wanted.
<smoser> we could chooes to hold off because vmtest seems failing. as a result of the journal colleciton change that rharper memtioned today
<rharper> smoser: I've a fix for that now
<rharper> I'll put up an MP
<apollo13> let's assume I have set some data wrong in cloud-init userdata (namely the hostname). Can I edit that somehow after the instance booted? modifying set-hostname didn't seem to work
<apollo13> the metadata service (cloud-init) did update
<rharper> apollo13: the default is for metadata (and userdata) to be only applied once per instance (first boot on a platform);   If you do make manual modifications (and don;t mind re-running things, which will for example, change the instance host ssh keys) then you could do something like 'cloud-init clean --logs --reboot' which will remove the current instance information and boot like it was a new instance
<apollo13> lets see what that does :D
<rharper> apollo13: if you wanted to do a more targetted run of a specific module you can make use of cloud-init single,   something like
<apollo13> to late
<apollo13> worst case I nuke the machine and redo it
<rharper> hehe
<rharper> https://paste.ubuntu.com/p/dRfsfX8gzt/
<rharper> apollo13: there's a forced update of hostname via a local cloud-config file
<apollo13> but that will still keep the old data around?
<apollo13> anyways /etc/hosts etc is correct now
<apollo13> all that did change seems to be ssh key
<apollo13> ansible is now refixing the rest
<rharper> cool
<rharper> yes, it would leave the old data around in the metadata/user-data  but the hostname files would have the values you wanted
<blackboxsw> ok just pushed up the cloud-init devel render CLI subcommand to https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/335290   think I'll just await ci builds on that.
<apollo13> rharper: ok thanks, cloud-init is still magic; not exactly sure what it all touches and when :D
<apollo13> rharper: shouldn't the reinit process recreate the default user? (I mean I am happy it did not, butâ¦)
<rharper> apollo13: well, I didn't think users got nuked with a clean; but if it did, then yes the default user should have gotten recreated via a clean --logs --reboot
<apollo13> ah no, digitalocean overrids it to root via vendor-data
<rharper> yes, that makes sense after looking at source
<rharper> clean nukes stuff down /var/lib/cloud and logs
<apollo13> anyways, thanks for the help. all is well now :)
<rharper> apollo13: sure thing; glad you're good to go
<blackboxsw> paulmey: I did a  pass on  https://code.launchpad.net/~andyliuliming/cloud-init/+git/cloud-init/+merge/351742   I still need a bit more time to think a bit about how we want to handle threading/queues in cloud-init for this feature. but initial comments are up
<paulmey> blackboxsw: Thanks, I'll take a look (and ask Andy to take a look)
#cloud-init 2018-08-21
<blackboxsw> FWIW, seems to "work fine"= on the cosmic azure instance I deployed. I could see the encoded content in the proper kvp_pool file
<otubo> smoser: stupid question: that's my first time contributing on Launchpad (I'm from the mailing-list era). Shall I create a new branch a new pull-request? Or just push on top of that branch you already reviewed?
<smoser> otubo: welcome to the cloud era ;)
<smoser> you can push --force over whatever branch you have
<smoser> launchpad will do the right thing
<smoser> or you can 'git commit -a -m "add stuff smoser requested"'
<smoser> and then git push (without --force)
<smoser> generally i prefer the latter as it then is obvious what you've done to address that.
<otubo> smoser: anything else after that? Like, create a new pull-request? Just push -f and we're good?
<smoser> otubo: sorry, looked away. yeah, if you push over then launchpad upates the existing merge proposal
<smoser> blackboxsw: in your jinja branch... should i be able to reference 'v1.cloud_name' ?
<smoser> or is that not yet done.
<blackboxsw> smoser: I think I found a bug in render, I'm trying to fix it now.
<blackboxsw> rendered instance-data.json isn't being converted, I need to add a unit test for this
<blackboxsw> smoser: and yes  you should be able to
<blackboxsw> I was just testing it trying to draw up slides
<blackboxsw> ohh I forgot to back out our persisting underscore values in initial instance-data.json and move it back into convert_jinja_instance_data
<smoser> yeah, that seems to be it. if i changed my  instance_data to have underscores then it works.
<blackboxsw> yeah, and I think we decided we didn't want to persist underscores in instance-data.json as snap relies on v1 keys that don't have underscores
<smoser> yeah.
<smoser> thats unfortunate though
<blackboxsw> yeah costly to do that processing on each part render
<blackboxsw> smoser: alternative: maybe at instance-data.json write time (in ds.get_data()) we could write the jinja processed content
<blackboxsw> then we do it just once
<blackboxsw> and souce that processed jinja_vars file when rendering jinja content?
<blackboxsw> *source* rather
<smoser> we can figure something out there.
<blackboxsw> ok pushing a fix, but I need to sort a unit test failure still
<smoser> blackboxsw: for your consideration http://paste.ubuntu.com/p/QqGC8SdkgG/
<blackboxsw> smoser: for your consideration https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/353530
<blackboxsw> thanks smoser will play with that and pull it in.
<smoser> blackboxsw: imds is https://docs.microsoft.com/en-us/azure/virtual-machines/windows/instance-metadata-service ?
<smoser> shouldnt we have a version in our data then
<blackboxsw> correct. smoser. after our discussion yesterday I started a branch to better represent the raw azure data (which has already landed)
<blackboxsw> I'm not done w/ that branch yet to restucture instance-data.json on azure
 * blackboxsw thinks it was yesterday, or maybe it was friday
<smoser> :)
<smoser> well, i hit a comment there. to that effect.
<smoser> i have to run now.
<blackboxsw> in any event, I think openstack and azure ds's both need a little tweak
<blackboxsw> see you
<smoser> blackboxsw: fyi, there is a 0821 cosmic build now \o/
<blackboxsw> woot!
<blackboxsw> getting easier
<blackboxsw> each time
 * smoser heads out.
<socket-> Can anyone help me understand if/how cloudinit sets the hostname to ip-xxx-xxx-xxx-xxx? We want to change the default hostname as we spin up images to be 15 random hex characters instead of ip based.
<blackboxsw> socket-: cloud-init generally gets hostname from the cloud's metadata service if present and applies it. For setting or generating hostname a user can specify a static hostname with a #cloud-config yaml file at instance creation time.
<blackboxsw> https://pastebin.ubuntu.com/p/DZ43ptvtPT/
<blackboxsw> example ^
<socket-> blackboxsw: I'm not sure how familiar you are with AWS, but for example, we are using the ubuntu AMI (which I believe Ubuntu created) and it's default cloud init. Are you saying I would just modify the cloud-config to say something like hostname: $rand(15,[A-Z,a-z,0-9], save it as a new image, and each time someone launches one, they will get a random hostname?
<socket-> looking at the default cloud.cfg, the only hostname option i see is "- set_hostname" under cloud_init_modules
<blackboxsw> socket-: when the following branch is published you'll be able to specify something like the following:    https://pastebin.ubuntu.com/p/jcVvmgZSPh/
<blackboxsw> socket-: correct, via aws CLI or web UI   you could currently a user-data file containing your "#cloud-config\nhostname: <random-value>"  statically generated content
<blackboxsw> socket-: sorry branch in question is https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/335290 with some examples of rendering jinja templates.
<blackboxsw> hrm. socket- I so you want a custom AMI that will by default set the hostname to a random string right... you aren't in control at the time the instance is created is that right?
<blackboxsw> hrm, gotta bail for dinner
<socket-> sorry man, got called away from my desk, yeah your right I want a custom ami that wukk set the hostname to a random string, and I am not in control of the time its created
#cloud-init 2018-08-22
<otubo> smoser: not sure what kind of test I should write for the nm-support. I didn't write a new renderer or how the renderer works, I just hack how it gets available.
<smoser> otubo: well, we wan tto make a test that shows that sysconfig gets rendered if networkmaager config has that setting.
<otubo> smoser: ok, you convinced me, I'm gonna work on that and will push soon.
<otubo> smoser: btw, I might as well push my fixes now and the test later this week
<smoser> otubo: yeah. thats perrfectly fine
<otubo> thanks :)
<smoser> otubo: i'm sorry to say i dont think we have a test of sysconfig.available
<smoser> so you wont be able to just add one
<smoser> but rather have to create a class
<smoser> but i can help you
<otubo> smoser: oh that would be really nice. I need to fix some other things right now. I can ping you tomorrow if you have some free time on your day.
<smoser> otubo: we actually have cloud-init-summit tommorrow in US/Pacific times
<smoser> so.. i wont be around much, but i'll see if i can get something together to give you pointers.
<otubo> smoser: ohh that would be really nice to attend, to bad I live in the other side of the world.
<otubo> smoser: that's ok. Perhaps rharper can help? Or Ryan Mccabe?
<otubo> But I really do appreciate your help on that :)
<rharper> otubo: I'll be at the summit as well, but we'll do our best to respond to pings here and on your merge proposal
<otubo> duh, of course you would :)
<otubo> smoser: rharper enjoy the summit guys, we can talk next week. No worries.
#cloud-init 2019-08-19
<otubo> rharper, just a quick question from last week's issue: I can't define which datasource cloud-init SHOULD NOT look for, only a data source it SHOULD look for, right? My question basically is: If I don't want cloud-init to spend so much time trying yo find network-based datasources, I should have different ds-identify configuration files for every cloud provider we (as in Red Hat) support.
<otubo> rharper, Am I correct? If I deploy a vm with only openstack it shouldn't spend time looking for EC2 configuration and taking a long time to eventually timeout in the end
<Odd_Bloke> otubo: Do you know about ds-identify?  It's used early in boot to determine what data source applies to the current instance (without using the network), and writes out configuration so cloud-init will only run that data source.
<otubo> Odd_Bloke, I know ds-identify and that's exactly what I mean by one configuration per cloud provider. For OpenStack I should have it set to search for OpenStack and nothing else, if the VM is on EC2 I should set it to search for EC2 and nothing else, right?
<Odd_Bloke> otubo: Oh, I'm sorry, I skipped over the word "ds-identify" in your question. >.<
<otubo> :-D
<Odd_Bloke> something something monday morning something ;)
<Odd_Bloke> ds-identify doesn't use the network, so you shouldn't need to configure it differently per provider.
<otubo> Odd_Bloke, So basically I should just list all RHEL supported cloud providers on ds-identify.cfg and we're good? Neat :-)
<otubo> Odd_Bloke, hard question now: We're on cloud-init-18.5 shall that be ok, or should I update to a newer version?
<Odd_Bloke> So if you're talking _configuration_ then you might be configuring an _override_.
<otubo> No specific reason to stay on cloud-init 18.5, we just do a downstream rebase on demand.
<Odd_Bloke> Ah, no, you presumably mean configuring datasource_list, not the other option I was thinking of.
<Odd_Bloke> otubo: I'd recommend taking the latest if you're putting in effort to move to a newer version, but obviously that's your call!
<otubo> Odd_Bloke, I mean defining the 'datasource' attribute on the file /etc/cloud/ds-identify.cfg
<otubo> Odd_Bloke, ok cool, I'll run some tests and check if we get it going. :-)
<Odd_Bloke> otubo: OK, "datasource" _will_ override, I think.
<otubo> Odd_Bloke, ok, I'll give it a try. Thanks!
<Odd_Bloke> You probably want to use datasource_list.
<Odd_Bloke> That way you can ship the same list on every platform, and ds-identify will work it out at first boot.
<Odd_Bloke> (OTOH, if you already ship per-platform images, it doesn't necessarily hurt to hardcode the specific platform you know they're used for.)
<otubo> Odd_Bloke, we don't have specific packages for specific environments, it's a single rpm for all environments.
<otubo> Odd_Bloke, but 'datasource' on ds-identify.cfg also accepts a list, right? At lease that's my understanding from ds-identify
<Odd_Bloke> otubo: I think it can, yes, but that will set the list of datasources that cloud-init iterates through.
<Odd_Bloke> So if you put a network data source in there, cloud-init will attempt to hit the network for that data source.
<otubo> oh I see
<Odd_Bloke> datasource_list defines the list of datasources that _ds-identify_ iterates through.
<otubo> Yes, right, right, I remember now.
<Odd_Bloke> And then when it finds one, it sets that as the one DS that cloud-init will "iterate through".
<otubo> Odd_Bloke, then I'm not really sure where to set datasource_list.
<Odd_Bloke> otubo: It's configurable in the same location, I believe.
<otubo> Odd_Bloke, ok I'll give it a try.
<otubo> thanks a lot :-)
<Odd_Bloke> Happy to help!
* blackboxsw changed the topic of #cloud-init to: Reviews: http://bit.ly/ci-reviews | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting Aug 19 16:15 UTC | cloud-init v 19.2 (07/17) | https://bugs.launchpad.net/cloud-init/+filebug
<blackboxsw> rharper: Odd_Bloke if either of you get a chance to re-review the ubuntu-drivers branch, here is the debconf based approach to latelink setup. https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/371369
<blackboxsw> #startmeeting Cloud-init bi-weekly status
<meetingology> Meeting started Mon Aug 19 16:18:48 2019 UTC.  The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology.
<meetingology> Available commands: action commands idea info link nick
<blackboxsw> Hi guys and girls, welcome to cloud-init biweekly status meeting
<blackboxsw> #chair rharper
<meetingology> Current chairs: blackboxsw rharper
<blackboxsw> #chair Odd_Bloke
<meetingology> Current chairs: Odd_Bloke blackboxsw rharper
<blackboxsw> cloud-init upstream uses this meeting as a platform for community updates, feature/bug discussions, and an opportunity to get some extra input on current development.
<blackboxsw> All discussions and interjections are welcome
<blackboxsw> our format is the following topics: Previous Actions, Recent Changes, In-progress Development, Office Hours
<blackboxsw> last meeting's minutes are herer
<blackboxsw> #link https://cloud-init.github.io/status-2019-08-05.html#status-2019-08-05
<rharper> o/
<blackboxsw> we host the meeting every two weeks at the date and time indicated in the IRC channel topic ^
 * blackboxsw changes that topic now, since we(I) forgot last time 
<blackboxsw> #topic cloud-init Reviews: http://bit.ly/ci-reviews | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting Sept 2 16:15 UTC | cloud-init v 19.2 (07/17) | https://bugs.launchpad.net/cloud-init/+filebug
<blackboxsw> next meeting in two weeks
<blackboxsw> #topic Previous actions
<blackboxsw> I see no previous actions raised during last meeting. Woo hoo!
<blackboxsw> #topic Recent Changes
<blackboxsw> the following are commits that
<blackboxsw> have landed in tip of master for cloud-init since the last meeting: git log --since 2019-08-04
<blackboxsw>     - cloudinit/distros/parsers/sys_conf: add docstring to SysConf
<blackboxsw>       [Daniel Watkins]
<blackboxsw>     - pyflakes: remove unused variable [Joshua Powers]
<blackboxsw>     - Azure: Record boot timestamps, system information, and diagnostic events
<blackboxsw>       [Anh Vo]
<blackboxsw>     - DataSourceOracle: configure secondary NICs on Virtual Machines
<blackboxsw>       [Daniel Watkins]
<blackboxsw>     - distros: fix confusing variable names [Daniel Watkins]
<blackboxsw>     - azure/net: generate_fallback_nic emits network v2 config instead of v1
<blackboxsw>       [Chad Smith]
<blackboxsw>     - Add support for publishing host keys to GCE guest attributes
<blackboxsw>       [Rick Wright]
<blackboxsw>     - New data source for the Exoscale.com cloud platform [Chris Glass]
<blackboxsw>     - doc: remove intersphinx extension [Daniel Watkins]
<blackboxsw>     - cc_set_passwords: rewrite documentation [Daniel Watkins] (LP: #1838794)
<ubot5> Launchpad bug 1838794 in cloud-init "Set Passwords documentation describes incorrect behaviour for `password` config key" [Low,Fix committed] https://launchpad.net/bugs/1838794
<blackboxsw> We have also published commits though "    - Azure: Record boot timestamps, system information, and diagnostic events"   to Ubuntu Eoan (19.10) (cloud-init v.19.2-13) if folks want a glimpse of those features
<blackboxsw> Many thanks to Azure and GCE folks for their commits and a hi five to tribaal for adding Exoscale
<blackboxsw> #topic In-progress Development
<blackboxsw> As always, we try to keep most of our work up to date in trello
<blackboxsw> #link https://trello.com/b/hFtWKUn3/daily-cloud-init-curtin
<blackboxsw> cards in the "Reviewing" column should represent the work we expect to have up for review in the short term.
<blackboxsw> rharper is mid-stream on some investigations that will likely lead to significant speed improvements for cloud-init
<blackboxsw> Odd_Bloke: is working on some significant improvements for Oracle's datasource rendering network config
<blackboxsw> and I'm working on getting OpenStack and Ec2 datasources to talk network config v2.
<blackboxsw> beyond that work we have a pretty healther active review queue
<blackboxsw> #link https://code.launchpad.net/cloud-init/+activereviews
<blackboxsw> of note, some freebsd work is in flight, gce dns improvements, udev triggers and OVF handling user-defined scripts.
<blackboxsw> We
<blackboxsw> will spend the latter part of this meeting looking over the review queue to see that open branches are in the proper state
<blackboxsw> Also, our plan for this week is to cut a Cloud-init SRU (Stable release update) for upload into xenial, bionic and disco.
<blackboxsw> expectation is that those Ubuntu series will see an update for cloud-init after our ~7 days of testing and verification
<blackboxsw> rharper: Odd_Bloke anything else in flight that we should note here?
<rharper> that looks like everything
<blackboxsw> without further ado, we can transition to office hours
<blackboxsw>  #topic Office Hours (next ~30 mins)
<blackboxsw> We're here for any questions, bugs, discussions people would like to have around cloud-init. This block of time is available for any discussions or requests people may have.
<blackboxsw> We will also spend this time grooming the active review queue to make sure developers get any needed feedback on their active branches.
<blackboxsw> If there are any branches that need more eyes, please bring them up here or make sure they are in the 'Needs review' state in Launchpad
<tribaal> blackboxsw: thanks!
<blackboxsw> tribaal: good work. I think Odd_Bloke landed the followup work to enable exoscale datasource config to cloud-init.templates to 'enable' it. And looks like that has landed
<tribaal> blackboxsw: I'm happy to help verify SRU bugs when the process is kicked - just let me know
<blackboxsw> so it's 'on' in Eoan, once SRU is kicked off, it'll be in there
<tribaal> yep, I need to push an Eoan template to our preprod environment tomorrow to kick the tires, but I don't expect anything funny
<blackboxsw> tribaal: will do. I think the only thing we are waiting on before SRU is landing this ubuntu-drivers branch  https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/371369
<blackboxsw> any feedback on behavior your in Eoan from you tribaal would be helpful.
<blackboxsw> let's try that in English this time:  any feedback on the behavior in your Eoan environment would be helpful tribaal.
<tribaal> blackboxsw: haha that's what I inferred :)
<blackboxsw> :) /me hits the review queue
<blackboxsw> rharper: if you get a chance: you've landed https://git.launchpad.net/cloud-init/commit/?id=b3a87fc0.   Do we also still need the following branch? https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/363571
<blackboxsw> If so, I'll spend this time trying to write up unit tests for this if possible
<rharper> blackboxsw: I don't think so; my branch should include all the needs of that branch
<rharper> and the branch tests the wait_for_physdev as well as updates the opensuse net render paths to account for the udev rule number change
<blackboxsw> that's kindof what I was thinking/hoping.  I'll mark it rejected in favor of your commit, and we'll see what robjo thinks on that. We can reopen and try to address the unit test aspect of his branch if still needed.
<rharper> I think we can mark that branch closed
<rharper> robjo: had already looked at the branch before landing
<blackboxsw> ok done
<blackboxsw> thanks Florian for your first commit! https://code.launchpad.net/~florian-mueller-v/cloud-init/+git/cloud-init/+merge/371298  ... doc update approved
<robjo> I don't recall having looked at https://git.launchpad.net/cloud-init/commit/?id=b3a87fc0 and there was no entry in the bug to remind me that I did. Anyway, I've done so now and yes, this obsoletes https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/363571
<blackboxsw> thank you robjo for that
<rharper> robjo: https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/366667  ;  yes, I should have linked to the bug in my MP
<rharper> you did take a look a while back though
<robjo> rharper: I believe you, just cannot remember....
<rharper> heh, it was a while back
<blackboxsw> I think we should probably wrap up the meeting for this week.  I've got one more review to clear.
<blackboxsw> Thanks again all for joining. minutes will be posted to github
<blackboxsw> #link https://cloud-init.github.io/
<blackboxsw> #endmeeting
<meetingology> Meeting ended Mon Aug 19 17:25:18 2019 UTC.
<meetingology> Minutes:        http://ubottu.com/meetingology/logs/cloud-init/2019/cloud-init.2019-08-19-16.18.moin.txt
<Odd_Bloke> rharper: https://code.launchpad.net/~daniel-thewatkins/cloud-init/+git/cloud-init/+merge/371350 <-- happy with this now?
<rharper> Odd_Bloke: yes
<Odd_Bloke> smoser: Do you want to change your review on https://code.launchpad.net/~daniel-thewatkins/cloud-init/+git/cloud-init/+merge/371403 or shall I just Approved it?  (Does the lander check the reviews?)
<Odd_Bloke> blackboxsw: There's a typo, I'm afraid.
<smoser> that wasnt the right link i think
<smoser> was it ?
<smoser> https://code.launchpad.net/~daniel-thewatkins/cloud-init/+git/cloud-init/+merge/371403
<rharper> yeah
<smoser> i approved therel
<rharper> thx
<powersj> \o/
<powersj> rharper, blackboxsw Odd_Bloke did someone take notes?
<rharper> I can do a mental brain dump; and Odd_Bloke summed up the next steps at the end
<blackboxsw> powersj: rharper admittedly brief notes https://pastebin.canonical.com/p/Tm3QBsvwSx/
<blackboxsw> Odd_Bloke: ^  could use second set of eyes to decode what that means in terms of followup branches
<Odd_Bloke> powersj: So there are a couple of sets of decisions to be made.  The
<rharper> blackboxsw: thanks
<Odd_Bloke> *nods sagely* I meant to press Return then.
<blackboxsw> that didn't capture what pre-flight network configuration/interrogation needs to be
<Odd_Bloke> powersj: So there were a couple of sets of decisions to be made.  We decided that we _do_ need to parse the network config that the initramfs produces, because anything else is likely to break assumptions that other tools make about system configuration being accurate (or it will just flat out break!).  We also decided that it probably makes sense for the initramfs network configuration determination to
<Odd_Bloke> move to be part of the fallback network config source.
<Odd_Bloke> Because the correct network configuration to fall back to on, e.g., an iSCSI root system is _not_ DHCP-on-first-interface, it's to use the network configuration the initramfs generated.
<Odd_Bloke> The fact that the initramfs network config source is separate now means that we can continue along our current implementation path (i.e. land my change, and put dracut parsing into the "initramfs" network config source) and once that's done, we can address moving the initramfs network config source into the fallback network config source.
<Odd_Bloke> (Some Oracle DS changes will be required at that point, but they should be minimal so we won't be throwing any significant work away.)
<Odd_Bloke> rharper: blackboxsw: smoser: Does ^ represent what you think we agreed on?
<blackboxsw> that is much clearer than my abbreviated notes Odd_Bloke +1.  1. your branch lands 2. dracut parsing 3. fallback config handling initramfs config on appropriate platforms
<blackboxsw> I *think*
<smoser> +1
<powersj> excellent :)
<rharper> Odd_Bloke: +1
<blackboxsw> gah Odd_Bloke sorry about that. endswidth :)
 * blackboxsw runs tox this time
<Odd_Bloke> blackboxsw: CI was +1, so we're probably missing a test.
<blackboxsw> yeah I'll add that test now
<blackboxsw> it's all mocked in tests
<blackboxsw> Odd_Bloke: pushed a fix
<blackboxsw> 8b8401cd5f99e907a944b05e2a1caf1e889ecbbf
<smoser> once i saw a program (or possibly imagined) that allowed you to do:
<smoser>  some-long-running-command | that-program
<smoser> and it would hand back a url that you could share with someone else
<smoser> so they could see status or "watch" it
<smoser> something like 'pastebinit' but with progress
<smoser> my google is failing now
<smoser> anyone's brain do better?
<Odd_Bloke> blackboxsw: LGTM, Approved.
<Odd_Bloke> That sounds familiar.
<Odd_Bloke> smoser: https://seashells.io/ <-- this?
<Odd_Bloke> (I feel like that isn't what I was remembering, but looks like it would do the trick.)
<smoser> i think so
<smoser> or maybe
<smoser> https://github.com/anishathalye/seashells/issues/2
<smoser> provides some alternatives too.
<smoser> thanks Odd_Bloke
<blackboxsw> rharper: do you recall how we describe global dns in network v2?
<rharper> there is no global dns
 * blackboxsw thought there was a bug on this
<rharper> dns is always per interface
<rharper> there is a v1 bug where we will decorate v2 with v1 's "global" values if the target interface isn't configured already
<blackboxsw>  rharper: network_data.json describes a global 'services' key (outside of specific network/interfaces) which can contain dns type items
<rharper> yes, the easiest thing to do is repeat that on each interface
<blackboxsw> what shall we do with such 'global' service definitions
<blackboxsw> ok will do
<rharper> https://bugs.launchpad.net/cloud-init/+bug/1750884
<ubot5> Ubuntu bug 1750884 in cloud-init "[2.4, bionic] /etc/resolv.conf not configured correctly in Bionic, leads to no DNS resolution" [Medium,Fix released]
<rharper> https://git.launchpad.net/cloud-init/commit/?id=d29eeccd
<blackboxsw> ah thanks!
<rharper> we may want to follow that same logic
<rharper> that is, devices getting DHCP are likely to get DNS from that response;
<rharper> if you've got static IPs, then dns away
#cloud-init 2019-08-20
<cpaelzer> just stumbled over https://github.com/virt-manager/virt-manager/commits/cloudinit
<cpaelzer> is that known or something you want to help with?
<tribaal> blackboxsw: for the record: I built and booted an Eoan template with 19.2-13-g2f3bb764-0ubuntu1 on our preproduction environment, and all seems well :)
<Odd_Bloke> tribaal: \o/
<Odd_Bloke> cpaelzer__: I'm not aware of it, but it looks cool!  What help could we offer?
<Odd_Bloke> (If you know!)
<tribaal> I guess we're ready to have Eoan available on day one :P
<cpaelzer> Odd_Bloke: I don't know what they work on or woudl need right now
<cpaelzer> I just have seen the branch while cloning virt-manager for a fix
<cpaelzer> and wondered if that is part of a community activity that you'd know of
<cpaelzer> interesting ...
<cpaelzer> waiting if rharper doesn't know it either ...
<rharper> it's new to me as well; looks like virt-install adding support for installing their cloud images by prepping some in-image cloud-init ;
<blackboxsw> Odd_Bloke: rharper Steve's suggestion to use debconf-loadtemplate adds a dependency on debconf-utils that cloud-init doesn't currently have. That adds about 57K of packages if we made this strict package dependency.  Would we want to avoid adding this package unless someone specified ubuntu-drivers cloud-config?
<rharper> blackboxsw: let's check in #ubuntu-devel with vorlon for alternatives/thoughts
<blackboxsw> sounds good
<mator> hello
<mator> how do i install a cloud-init server, not instance? for example, i have kvm server for vm deployment and want to use cloud init for cloud images VMs
<mator> besides of .iso images with cloud file metadata settings, is there network way to init vm ?
<rharper> mator: hi,  I'm not sure that I understand your questions; but it sounds like you'd be interested in Ubuntu cloud-images[1], which are Ubuntu server images with cloud-init already installed.  And if you aren't running these images on a cloud, you'll want to learn to use cloud-init's NoCloud datasource[2], which allows cloud-init to find configuration on secondary devices (like a cdrom);   1. http://cloud-images.ubuntu.c
<rharper> om/daily/server/bionic/current/ 2. https://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html
<Odd_Bloke> rharper: blackboxsw: https://write.wrestle.town/oddbloke/dracut-findings-for-cloud-init <-- your thoughts would be appreciated
<blackboxsw> Odd_Bloke: if we were to go down the route of extending dracut to emit a more friendly/standardized network configuration instead of cloud-init growing the ability to parse dracut's config, what is the  possibility that we have issues upstreaming dracut fixes for cloud-init if needed? If we have issues with dracut upstream, I'm not certain what *we* would do to host our own official yum repo, other than our dev
<blackboxsw> copr repos
<blackboxsw> with the option of  cloud-init proper growing dracut-parsing logic, we control the publishing of that feature, so if we have bugs, we can be certain that we fix them quickly and fast-track public releases of cloud-init that can cope
<Odd_Bloke> blackboxsw: We won't be able to push any dracut changes back into the releases that people actually care about, I don't think.
<Odd_Bloke> Getting dracut to emit v2 network YAML would be very nice, but we'd only be able to glean the benefits in several years, really.
<Odd_Bloke> So we need something to bridge that gap.
<blackboxsw> ohh /me misread "2. implement parsing in the dracut initramfs code"   as implementing it in dracut package.
<Odd_Bloke> Ah, yes, that's confusing wording.
<Odd_Bloke> Let me fix that up.
<Odd_Bloke> blackboxsw: Is that clearer?
<blackboxsw> +1 Odd_Bloke
<Odd_Bloke> Thanks for the catch!
<blackboxsw> I'd agree with path 2. Odd_Bloke until we have more use cases
<blackboxsw> s/catch/getting confused myself/
<Odd_Bloke> :)
<blackboxsw> Odd_Bloke: rharper for tomorrow: just put up ubuntu-drivers v2 https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/371546
<rharper> blackboxsw: cool
#cloud-init 2019-08-21
<seven-eleven> hi
<seven-eleven> why is the hashed passwd not picked up by cloud init? login to user fails http://dpaste.com/3FQ5MDD
<chillysurfer> rharper: good morning! you and i had a chat a couple of weeks ago about vendordata and userdata, and merging them. from my tests (also sent a mail on the mailing list yesterday) it looks like even with merge_how for both userdata and vendordata that the latter is still overwritten. i got a response exlpaining this behavior. but wanted to close the loop on it and see if you had any other ideas of ways to po
<chillysurfer> ssibly not have userdata overwrite vendordata. thanks!
<otubo> Odd_Bloke, any chance to have an official ics calendar entry file for the cloud-init meeting? I consistently missed *all* meetings so far :-) I think it could be useful for mor people as well.
<Odd_Bloke> powersj: ^ You might be able to help out with getting some sort of public calendar entry for the cloud-init status meeting?
<Odd_Bloke> seven-eleven: How did you create the hashed password?
<rharper> chillysurfer: hey, thanks for following up;   I don't have another idea with existing code;  it's something that we want to support and I have some ideas on how we can do it
<chillysurfer> rharper: awesome makes total sense! thanks again for all the help!
<blackboxsw> hrm rharper Odd_Bloke, I'm struggling a bit with trying to pre-enabled debconf selections in cloud-init during early boot by running the shell
<blackboxsw> db_x_loadtemplatefile   versus just using debconf-loadtemplate
<blackboxsw> using shell, I run into Debconf.pm /tmp text file is busy issues, even if I set env['TMPDIR'] /var/tmp/cloud-init/* (as cloud-init does)
<blackboxsw> errors like this: https://paste.ubuntu.com/p/hvfggHKFFF/
<blackboxsw> yet if cloud-init just installs debconf-utils and runs [debconf-loadtemplate cloud-init mydebconf.template] it succeeds
<blackboxsw> I'll try explicitly writing the cloud-init debconf script to /home/ubuntu to confirm that it's not cloud-init's script that is the problem, but the Debconf perl modules which are choking on file creation
<blackboxsw> if I don't find an easy workaround for this today, I'd like to conditionally install debconf-utils package if #cloud-config has a 'drivers' configuration
<Odd_Bloke> "Can't exec" doesn't suggest the issue is file creation to me.
<blackboxsw> Odd_Bloke: yeah, I'm trying to sift through the perl code to find out how to rememdy the file creation logic in perl's open
<blackboxsw> as it doesn't seem to be paying attention to my TMPDIR in environ
<blackboxsw> I'd really like to avoid installing another package for htis
<Odd_Bloke> "The Linux kernel will generate a bad interpreter: Text file busy error if your Perl script (or any other kind of script) is open for writing when you try to execute it." <-- https://stackoverflow.com/a/1384594
<blackboxsw> Odd_Bloke: maybe I'm not closing/flushing the tmpf I've created within that ExtendedTemporaryFile context manager
<Odd_Bloke> I don't really understand the moving parts here, but is it possible you're trying to execute inside a `with open(..., "w"):` block?
 * blackboxsw tries to unwind that 
<rharper> it certainly looks like an open file handle to me
<rharper> blackboxsw: I would suggest using the temp_files mkdtmp to create a dir, then plain util.write_files() which should generate files with no handles left open to them
<blackboxsw> rharper: yeah I'm unwinding/not using the context manager anymore along those lines to see if that resolves my issue
<blackboxsw> thanks
<blackboxsw> gents
<blackboxsw> Odd_Bloke: rharper yeah thanks guys, I was bungling the ExtentedTempFile context manager
<blackboxsw> using temp_utils.mkdtemp  and cleaning up afterward works fine
 * blackboxsw fixes unit tests for this and tests on a clean gpu instance
<rharper> \o/
<chillysurfer> is there already a helper in cloudinit that retrieves the distro?
<rharper> chillysurfer: cloudinit/distros/__init_.py is the base distro class, then there are distro specific implements of the distro class in the same directory
<rharper> chillysurfer: the distro variant is set at build-time, see setup.py for details;  this is extracted from the util.get_linux_distro/system_info;  which reads python info + host related files (like os-release/redhat-release/lsb_release);
<chillysurfer> nice thanks!
<chillysurfer> i'm trying to use packages/bddeb to build a source pkg but also signed. with debuild i'd use -k to specify which gpg key id i want to sign it with, but i can't seem to pass through with bddeb
<chillysurfer> am i missing something obvious?
<chillysurfer> nvm i can debsign the file then
<Odd_Bloke> Yeah, that's easiest.
<seven-eleven> Odd_Bloke, i created the password hash for cloud-init like so http://dpaste.com/1RAP7VH
<seven-eleven> i tried also setting the password hash in chpasswd key, not successful either :-)
<Odd_Bloke> seven-eleven: The output I get from running that is more like `$6$zTJtXk035OHCYOBf$/GX3r335pk5z9Pa2TpDgG1nj/ES6kqQOU6QbnIVAMYBDFCjIGfj9yMkrmR8MQdeCEWCzFZz60xb8ySOQAkLJK1`, are you sure your password hash is correct?
<seven-eleven> Odd_Bloke, hm, maybe I should try another one
<seven-eleven> Odd_Bloke, i saw that multiple people recommended python to generate the hash
<seven-eleven> my hash looked similar to yours
<seven-eleven> quiete long
<seven-eleven> im trying right now with `openssl passwd -1 -salt somesalt`
<blackboxsw> rharper: I validated as far as I can on eoan-proposed ubuntu-drivers debcon-selections setup. The wiring still needs to be put into ubuntu-drivers-common to install linux-modules-nvidia-* on our behalf. https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/371546 is ready
<rharper> blackboxsw: cool, I'll review
<blackboxsw> our CI is unhappy at the moment, but tests all pass locally
<seven-eleven> Odd_Bloke, didn't work with the openssl hash. the problem looks like that the hash is never added to /etc/shadow. the question is why
<seven-eleven> Odd_Bloke, I use `kvm-install-vm` script to create my cloud-init image
<seven-eleven> see: https://github.com/giovtorres/kvm-install-vm/blob/master/kvm-install-vm#L437-L537
<Odd_Bloke> seven-eleven: OK, in that case I would suggest filing a bug with the output of `cloud-init collect-logs` attached and pointing to it here.
<blackboxsw> Odd_Bloke: seven-eleven hashed pw issues reminds me of a bug that we had fixed
 * blackboxsw digs it up... I'm thinking if we are on older cloud-init that'd be a known problem
<seven-eleven> blackboxsw, i'm on version: /usr/bin/cloud-init 19.1-1-gbaa47854-0ubuntu1~18.04.1
<blackboxsw> bummer, me was thinking https://bugs.launchpad.net/bugs/1811446
<ubot5`> Ubuntu bug 1811446 in cloud-init (Ubuntu) "chpasswd: is mangling certain password hashes" [Undecided,Fix released]
<blackboxsw> specifically that fix corrected hashes that started with $6
<seven-eleven> "This bug is believed to be fixed in cloud-init in version 19.1"
<blackboxsw> yeah no dice :/
<seven-eleven> ok going to file a bug as Odd_Bloke suggested
<Odd_Bloke> If it's not showing up at all, then it probably isn't a subtle password handling bug.
<seven-eleven> i will verify by setting NOPASSWD in sudoers then I can read the shadow file
<blackboxsw> seven-eleven: in your cloud-init log, I would also expect a    grep 'setting hashed password' /var/log/cloud-init.log to return a line for your user.   In our testing of the fix for 1811446 we provide chpasswd: with a 'list' value: https://pastebin.ubuntu.com/p/Mr77D6t9KG/
<seven-eleven> oh shadow has a hash set for both root and my user, but the bash is totally different
<seven-eleven> blackboxsw, i tried chpasswd too: http://dpaste.com/3986T25
<blackboxsw> example of our chpasswd use for setting root user's hashed password: https://github.com/cloud-init/ubuntu-sru/blob/master/bugs/lp-1811446.txt
<blackboxsw> hrm seven-eleven looks sane
<seven-eleven> let me try list with quotes, didn't try that before
<blackboxsw> seven-eleven: that forward slash at the end may end up funky or shell escaped which could affect behavior.
<blackboxsw> you could also try leading hyphen before each chpasswd list item and then quotes;   list:\n  - 'root:<hash>'\n  - 'user2:<hash2>'\n
<seven-eleven> ehm, it's actually setting a hash to shadow
<seven-eleven> because this time I omitted the user, and now only the root hash is visible in shadow
<seven-eleven> but the hash is different than the supplied one
<seven-eleven> look likes $6$ELfTiASdsab$Zt6RUzzoio2lkwfhsldhfasdfl9rGKKH5tN9k/qYlca8/OoIBWbLmR0HYLVWLPo3s8YkZZJvv2j99426k/.:18419:0:99999:7:::   (slightly modified)
<seven-eleven> not sure if cloud init converts the supplied hash
<seven-eleven> or simply uses a newly random generated passwd
<blackboxsw> one thing to also check from the system if you can get in:    sudo cloud-init query userdata to see if your userdata is being mangled
<seven-eleven> let me try
<seven-eleven> oh yeah
<seven-eleven> see: http://dpaste.com/29MGBYY
<seven-eleven> the hash is eaten up
<blackboxsw> seven-eleven: yep, looks like that leading $ is actually being variable substituted
<seven-eleven> oh
<blackboxsw> single quote should prevent that I think.
<seven-eleven> I have to probably escape it with \
<seven-eleven> i try
<blackboxsw> or escape it
<seven-eleven> i used single quotes list: 'root:$1$someafk$taAISFjdAzgAb/'    (modified hash)
<seven-eleven> i think this is a #bash question, I will google how to properly escape EOF text
<rharper> for hashes we use the | inline content, right ?
<rharper> that's quite strange
<seven-eleven> blackboxsw, hehe with backslash escaping it worked :D
<seven-eleven> interesting that single quotes were not sufficient
<rharper> so, it's making it into subproces and we pass the the user:hash + newline into chpasswd -e
<rharper> seems quite odd to have that get mangled;  looking at sub, we encode the data (into bytes) ;  what OS are we talking about?   python3 ? python2 ?
<smoser> i think its probably a problem before it gets to cloud-init
<seven-eleven> rharper, are you refering to https://bugs.launchpad.net/cloud-init/+bug/1811446 or my hash being eaten up? in my case my problem was that single quotes are not sufficient for EOF things, I need to use backlash. in case of blackboxsw's link to the bug it looks like ubuntu bionic
<ubot5`> Ubuntu bug 1811446 in cloud-init (Ubuntu) "chpasswd: is mangling certain password hashes" [Undecided,Fix released]
<rharper> seven-eleven: no,  just wondering what cloud-init actually got, did you file the bug with the cloud-init collect-logs tarball ?   your paste earlier showing that your list http://dpaste.com/29MGBYY  was truncated; that suggests to me (what smoser said) that the user-data isn't making cleaning into cloud-init;
<rharper> if this is reproducible (sounds like it is) then filing a bug with your steps and the collect logs from within the instance would be most helpful to sort things out
<seven-eleven> rharper, hm, cloud-init received an empty variable due to the missing escape in my bash script, that's why http://dpaste.com/29MGBYY is truncated, shouldn't it be truncated if cloud-init receives an empty hash?
<rharper> seven-eleven: I'm at a loss with how you;re feeding cloud-init data;  do you have a /var/log/cloud-init.log that I can see?  depending on the datasource, you may want to base64 encode the user-data
<seven-eleven> rharper, i'm using this script which feeds the cloud init with EOF to genisoimage, see line 529: https://github.com/giovtorres/kvm-install-vm/blob/master/kvm-install-vm#L435-L537
<rharper> shell esacping
<seven-eleven> interestingly `/var/log/cloud-init.log` is empty
<seven-eleven> rharper, yep, single quote was not enough with EOF, I have to use \$, not '$'
<seven-eleven> ah, the log is on the guest
<seven-eleven> rharper, I found the log on the guest but we don't need it anymore. shell escaping :)
<rharper> right =)
#cloud-init 2019-08-22
<chillysurfer> i'm probably reading this wrong, but if we have an exit hook for dhclient (https://git.launchpad.net/cloud-init/tree/tools/hook-dhclient) unless `cloud-init dhclient-hook` returns a non-zero return code then dhclient will never fail, right?
<Odd_Bloke> chillysurfer: That feels like a precursor to a real question. ;)  What's prompting you to ask?
<chillysurfer> Odd_Bloke: haha the reason i'm asking is that we're seeing empty dhcp lease files and i'm following the trail back to this. i see in cloudinit.net.dhcp.dhcp_discovery that we would fail for a non-zero return code, and we are passing in the -1 opt to dhclient which would indicate it _would_ be a non-zero return code if dhclient fails, which had me scratching my head
<chillysurfer> and then i saw that we have this dhclient exit hook
<chillysurfer> and visually consuming it, it seems like this would hide a non-zero return code from dhclient main lease mechanism
<chillysurfer> unless ::is azure and cloud-init dhclient-hook fails:: in which case i think we would get nonzero rc
<chillysurfer> well... add the condition there that $reason is one of [BOUND, DOWN, RELEASE, REBOOT, STOP, EXPIRE]
<Odd_Bloke> OK, I don't really know enough about dhclient to answer this off-hand, and I'm about to head into a meeting.
<Odd_Bloke> chillysurfer: A bug report capturing the information you have might be the best next step, so that we aren't having to refer people to IRC backlog.
<Odd_Bloke> But I agree with your assessment, it looks like we would mask a failure.
<Odd_Bloke> Well, I'm not sure it's "masking", per se.
<Odd_Bloke> Because it daemonises, it's not clear that we have easy access to an indication that there was an error.
<chillysurfer> Odd_Bloke: totally agreed, and good point. i'll create a bug report!
<Odd_Bloke> Thanks!
<Odd_Bloke> rharper: Is there some specific reason we need a cloud-init FFe?  Or is it just because we know we're going to want to continue snapshotting?
<rharper> Odd_Bloke: my understanding was that while we have an SRU exception for bringing things back, we're still supposed to land features by freeze date and if aren't going to make freeze to do FFe
<rharper> historically we've not been great on doing that; so attempting to rectify that;  (assuming my understand is correct w.r.t the need for FFe)
<blackboxsw> oops
<Odd_Bloke> blackboxsw: I just commented on some weird indentation in https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/371546 and it looks like the autolander failed, so maybe we could fix that before it lands?
<blackboxsw> Odd_Bloke: will do
<Odd_Bloke> Thanks!
<blackboxsw> Odd_Bloke: rharper looks like auto-lander is failing due to slow git.launchpad response time
<blackboxsw> "ERROR: Timeout after 10 minutes" on git fetc
<blackboxsw> "ERROR: Timeout after 10 minutes" on git fetch
<chillysurfer> is there any official guide/doc on creating a custom cloud-init module?
<chillysurfer> i see cc_foo.py https://git.launchpad.net/cloud-init/tree/cloudinit/config/cc_foo.py but was wondering if there was anything beyond this sample module
<Odd_Bloke> I'm not aware of any.
<chillysurfer> seems straightforward to construct a module, but i'm more curious to the "install" process. i guess it's just cp'ing a module file into the python cloudinit dist in the cloudinit/config/ dir
<Odd_Bloke> chillysurfer: You'd also need to modify the /etc/cloud/... configuration to actually get the module to run.
<Odd_Bloke> (I think.)
<chillysurfer> Odd_Bloke: yeah that makes sense. need to add the module name to the respective config section
<Odd_Bloke> chillysurfer: You can also pass config modules in as user-data.
<Odd_Bloke> But that's about as much as I know about that. :p
<chillysurfer> Odd_Bloke: sounds like a good start to me :)
<blackboxsw> ok internal fixes on git.launchpad in progress. CI builds should come back out
<blackboxsw> Odd_Bloke: chillysurfer correct. drop-in custom config modules can just be added to the installed cloudinit/config directory and an addition in one of the modules sections in /etc/cloud/cloud.cfg
<chillysurfer> blackboxsw: awesome thanks!
<blackboxsw> cloudinit/stages does a find_modules lookup for any module that contains a 'handle' function
<blackboxsw> chillysurfer: if your config module solves a public early-config need, we'd also be interested helping you  push that confent upstream
<blackboxsw> *public* or *more general*
<chillysurfer> blackboxsw: i don't think it'll be so helpful. i'm tossing around ideas to prevent userdata from overwriting vendordata. if the same config sections are present in both, vendordata is overwritten (never merged). i was thinking a possible solution to that is to create an "internal" module and use that to specify vendordata actions
<Odd_Bloke> blackboxsw: rharper: https://code.launchpad.net/~daniel-thewatkins/cloud-init/+git/cloud-init/+merge/371673 <-- this is the refactoring in preparation for the dracut stuff that I mentioned at stand-up
<Odd_Bloke> We don't necessarily need to land it as-is (though that will make the review of the dracut-specific stuff easier), but I would appreciate some review so that I can course-correct sooner rather than later.
<Odd_Bloke> (Ideally, I'd have this all done and landed by EOW, because I'm on vacation for the next 6 working days, hence pushing the review sooner than I might otherwise.)
<Odd_Bloke> But, to be clear, I think this is appropriate for inclusion in master, it shouldn't change any behaviour.
<rharper> Odd_Bloke: cool
<blackboxsw> ok https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/371546 merged. I'm kicking off an eoan upload and then SRU
<blackboxsw> Odd_Bloke: will review that branch after eoan-proposal is syncd
<blackboxsw> eoan-upload rather. can always do another upload today if needed
<Odd_Bloke> rharper: Responded to your comments there, BTW.
<rharper> thx
<blackboxsw> rharper: I sent triage question in email regarding whether there is something to do with IPv6MTUBytes now that Eoan has support
<rharper> looking at it now
<blackboxsw> maybe it's a curtin action item, dunno
<rharper> curtin handles this for xenial, it;s always been broken on networkd;  we can re-run with those tests enabled to see what's been fixed.
<blackboxsw> btw: [ubuntu/eoan-proposed] cloud-init 19.2-21-ge6383719-0ubuntu1 (Accepted)
<blackboxsw> I'm queuing an SRU now
<rharper> blackboxsw: yes, we'll need to update our renderer; the key name is now 'ipv6-mtu'
<rharper> we're due for an update I think;  in e, I think netplan there now will dump it's features and keys;  so we'll want to start using that to control how we render things.
<Odd_Bloke> rharper: Responded again and pushed that doc change I mentioned.
<rharper> great
 * rharper waits for git to update
<blackboxsw> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1841099 filed for cloud-init SRU
<Odd_Bloke> blackboxsw: Ryan is +1 on my dracut branch; are you happy for me to mark Approved, or do you want to review too?
<Nick_A> What is the correct way to do dhcp4 and dhcp6 on a single interface in network config format 1? Do I need two subnet sections or do I list dhcp,dhcp6 on the same type line?
<Nick_A> Or perhaps two type lines, one with dhcp and one with dhcp6?
<rharper> Nick_A: you need to subnets,  subnets: [{type: dhcp}, {type: dhcp6}]
<Nick_A> thanks
<metsuke> What is the best way to insert a multi-line (3) cron file using cloud-init?  I can use 3 runcmd lines but I'm guessing there is a better way?
<rharper> write_files
<metsuke> thank you
<rharper> you can use the | marker to emit multi-line content
<metsuke> if only one line is needed, is it better to use write_files or one runcmd echo?
<rharper> write_files runs sooner ; but neither is better than the other
<metsuke> oh interesting.  Is there a way to make write_files run later than run_cmd?
<rharper> I guess it depends on what you need;  write_files you can also set the mode/permissions/owners on the file
<rharper> not easily, you have to specify all of the modules for a stage and update the order;
<rharper> in /etc/cloud/cloud.cfg, the init_modules: list and config_modules: list
<metsuke> got it, thank you!
<rharper> yw
<blackboxsw> SRU is queued for xenial. bionic disco https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/371685 https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/371686 https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/371687 Odd_Bloke rharper  if you get a chance to review, we can push it up for SRU
<blackboxsw> wanted to ask if there was anything to disable in this SRU
<rharper> blackboxsw: thanks
<rharper> I'll check the changelog
<rharper> blackboxsw: Odd_Bloke: is there a way to "shell" into one of our tox environments?  I'm trying to debug a unittest failure under py3  only
<Odd_Bloke> rharper: Yep, you can just use it as a virtualenv; . .tox/py3/bin/activate
<rharper> Odd_Bloke: excellent thanks
<blackboxsw> Odd_Bloke: rharper looks like we still need to enable exoscale in cloud-init.templates in each series
<blackboxsw> shall I add that to the SRU proposal now?
<rharper> also, I'm on xenial, and my nosetest3 on the outside passes fine, but inside tox -e py3 it fails ... I dont think there are any package differences in that ;
<blackboxsw> I was expecting to see something like abe6bcdfacdf2f6745e3acf5c76b332380b9c493
<blackboxsw> rharper: Odd_Bloke I think I'll add that to my current SRU proposal MPs now
<blackboxsw> just need to enable Exoscale
<rharper> yes, I think that's right
<blackboxsw> rharper: did we intentionally not add Oracle to Xenial/bionic as a datasource option (in favor of retaining Openstack detection behavior?)
<blackboxsw> because I don't see it in debian/cloud-init.templates
<rharper> blackboxsw: yes; we've work yet to do
<rharper> Odd_Bloke: has to finish up the dracut parsing so the cmdline parameter to disable netconfig can be removed
<blackboxsw> ok, good that it wasn't omission
<blackboxsw> ok rharper xenial, bionic disco I just added Exoscale datasource and repushed my MP
<blackboxsw> letting CI get through it
<blackboxsw> Odd_Bloke: sorry, reviewing your branch nowhttps://code.launchpad.net/~daniel-thewatkins/cloud-init/+git/cloud-init/+merge/371673
<Odd_Bloke> rharper: As I'm starting to write this config generation code, I'm wondering if we should actually move to v2 config for initramfs stuff first.
<Odd_Bloke> Otherwise I'm just writing techdebt. :p
<rharper> hehe
<rharper> well
<rharper> control-manual
<Odd_Bloke> Ah, yeah.
<Odd_Bloke> Conscience salved, I will continue on. :p
<rharper> oh, I think we can use critical: true on the iscsi connection
<rharper> so while we don;t have a contro-manaual for leaving stuff _down_
<rharper> we do have a way to say, don't bring the dhcp lease on this connection down
<Odd_Bloke> Ah, right, yeah.
<rharper> so restarts of networking won't break root
<Odd_Bloke> But changing the initramfs stuff means also changing the Oracle stuff, for which we _do_ want control-manual.
<Odd_Bloke> So I think I'm better off writing (not very much) code to produce v1, and then we can move everything up to v2 together once we're sure we're ready to do that.
<Odd_Bloke> rharper: Does that sound reasonable?
<blackboxsw> thanks Odd_Bloke approved https://code.launchpad.net/~daniel-thewatkins/cloud-init/+git/cloud-init/+merge/371673
<blackboxsw> we can land that right ^?
<Odd_Bloke> Yep, just marked as Approved.
<rharper> well, I don't think we did want control manual, but I think we agreed to start with that?  ISTR that we could bring everything up given the "proper" parsing of the metadata and sorting out the first nic to provide a subnet would get a gateway
<rharper> so I'm not sure if we're staging things;  if we did all v2, and sorted the gateway logic on the secondary nics in one go, then we'd not need to emit v1 as there's no control manaul requirement if we know we're not breaking primary interface routing
<Odd_Bloke> rharper: Yeah, we did agree to start with that, but the criteria for _stopping_ doing that are not necessarily clear.
<blackboxsw> ok xenial, bionic and disco SRU MPs all passed CI
<blackboxsw> and have Exoscale enabled in debian/cloud-init.templates
#cloud-init 2019-08-23
<paride> Odd_Bloke, I've been able collect the logs of an i3.metal instance failing to reboot following your suggestion of attaching the volume to another instance
<paride> Odd_Bloke, https://bugs.launchpad.net/cloud-init/+bug/1841182
<ubot5> Launchpad bug 1841182 in cloud-init "cloud-init fails when rebooting EC2 i3.metal instances" [Undecided,New]
<paride> there is my bug report
<Odd_Bloke> Thanks!
<blackboxsw> Odd_Bloke: or rharper if you get a chance to review my SRU branches into Xenial, Bionic and Disco, landing those are the only thing blocking us from moving forward with SRU Xenial: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/371685 Bionic: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/371686   Disco:
<blackboxsw> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/371687.   The only thing unique there is my enablement of Exoscale in debian/cloud-init.templates
<rharper> right
<rharper> blackboxsw: I looked at xenial/bionic, both have changelog entries mentioning enabling Aliyun, which I think you meant exoscale, right ?
<blackboxsw> rharper: grr right fixing
<blackboxsw> git.lp is super slow again today
<blackboxsw> rharper: all fixed bionic xenial and disco debian/changelog s/AliYun/Exoscale
<blackboxsw> and retagged
<rharper> ok, I'll re review
<rharper> blackboxsw: one more fix up on the sru branches in the cloud-init.templates
<blackboxsw> rharper: per the cloud-init.templates, there is a mix-n-match set of descriptions, some datasources say "Read from X metadata service", others say "X metadata services".
<rharper> *shrug* ok
<blackboxsw> shall we update them all to be the same verbiage?
<rharper> I wouldn't risk changing them
<rharper> I'm not sure what happens if we do; isn't this debconf input or something ?
<blackboxsw> s/X metadata services/X metadata service/
<blackboxsw> rharper: that particular line item is just the human readable description that you can see in the 'ncurses' TUI if running dpkg-reconfigure
<blackboxsw> maybe it's not ncurses anymore, but whatever the TUI is
<rharper> so, we don't have to now; but I would like a tools/add-datasource script which manipulates/renders that file
<rharper> that way, we don't have to manually edit that file each time
<rharper> blackboxsw: so I'll rereview and approve
<blackboxsw> powersj: Odd_Bloke: just pushed sru templates to https://github.com/cloud-init/ubuntu-sru/tree/master/20190823
<powersj> blackboxsw, thx!
<blackboxsw> Odd_Bloke: it's t-minus 0 mins until end of week your time, but if you end up grabbing one of the sru-related bugs listed in https://github.com/cloud-init/ubuntu-sru/tree/master/20190823 just let me know
 * blackboxsw was going to try looking at one of the cloud manual scripts to see if it needs to be extended to check for a specific bug in this release.  (I imagine azure will have a bit)
<Odd_Bloke> blackboxsw: Ack, will check in a few minutes if I think there's something I can take care of before EO[DWM].
 * blackboxsw queues  xenial/bionic and disco sru 
<blackboxsw> Okie dokey SRU into xenial, bionic disco approved by vorlon. so it should be built shortly
<powersj> woohoo
<blackboxsw> tribaal: for next week there should be cloud images with updated cloud-init 19.2.21 with exoscale enabled. If you do get a chance to peek at it on your platform, feel free to comment on the SRU bug: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1841099
<ubot5> Launchpad bug 1841099 in cloud-init (Ubuntu) "SRU cloud-init (19.1.1 to 19.2.21): Xenial, Bionic, and Disco" [Undecided,New]
<blackboxsw> ... it better be  well past tribaal's EOD
<blackboxsw> you'll need to:    echo deb $mirror $(lsb_release -sc)-proposed main |
<blackboxsw>        tee /etc/apt/sources.list.d/proposed.list; apt-get update
<blackboxsw>   mirror=http://archive.ubuntu.com/ubuntu
<Odd_Bloke> blackboxsw: Cloud images won't include it until after the SRU is complete, right?  Or am I misunderstanding what you're asking tribaal to do?
<blackboxsw> Odd_Bloke: agreed. but <series>-proposed will have it. per my comment: echo deb $mirror $(lsb_release -sc)-proposed main |
<blackboxsw> 14:02        tee /etc/apt/sources.list.d/proposed.list; apt-get update
<Odd_Bloke> blackboxsw: Right, but "there should be cloud images with updated cloud-init 19.2.21 with exoscale enabled" is misleading, I think?
<blackboxsw> tribaal: will need to add a <series-name> proposed apt-source to his instances to see and install updated cloud-init-19.2.21 and then run "sudo cloud-init clean --reboot  --logs" to exercise the updated cloud-init
<blackboxsw> Odd_Bloke: ahh right sorry tribaal. That is misleading. there are a couple of steps to enable that new -proposed cloud-init in the image; 1. add <series>-proposed apt source;  2. apt-get update; 3. sudo apt-get install cloud-init; 4: sudo cloud-init clean --reboot --logs;
<blackboxsw> we run the following script on a new vm https://github.com/cloud-init/ubuntu-sru/blob/master/bugs/lp-1797480.txt#L17-L23
<blackboxsw> followed by cloud-init clean --logs --reboot.   given that a new datasource is enabled, you might have to add  Exoscale to /etc/cloud/cloud.cfg.d/90_dpkg.cfg
<blackboxsw> or dpkg-reconfigure cloud-init
<Odd_Bloke> Do we have docs on how to do this?
<blackboxsw> Odd_Bloke: I can add something to our SRU docs if needed, but we do have the following: https://cloudinit.readthedocs.io/en/latest/topics/datasources.html?highlight=datasource_list#adding-a-new-datasource
<blackboxsw> "Add your datasource name to the builtin list of datasources:" and "Enable datasource by default in ubuntu packaging branches:"
<Odd_Bloke> blackboxsw: I meant specifically on how people can test cloud-init from -proposed for us.
<blackboxsw> Odd_Bloke: I'll add a trello card, yeah we should add new section on the testing/debugging page simple on https://cloudinit.readthedocs.io/en/latest/topics/debugging.html
<blackboxsw> rharper: yeah for openstack network v2 I'll have to address network_state passthrough for v2 IBMCloud, ConfigDrive or OpenStack all use the adapted cloudinit.sources.helpers.openstack.py:convert_net_json helper to create NetworkState  instances. So, it looks like this branch will be a bit bigger :/
#cloud-init 2019-08-24
<rharper> blackboxsw: right;
#cloud-init 2019-08-25
<jilocasin> evening evceryone.
<jilocasin> Does anyone know where cloud-init stores the hostname?
<jilocasin> Anyone?
<jilocasin> Does no one here know where cloud-init stores the initial hostname given at install?
<jilocasin> this is some seriously brain damaged stuff
<jilocasin> apparently it stores the initial hostname (and the one it will revert to unless you set preserve_hostname to false) in multiple locations:
<jilocasin> cloud.config.txt, user-data.txt.i, user-data.txt, and the *binary_file* obj.pk1.  Not sure at the moment which if all require a change, but that's apparently where it's stored.
<jilocasin> Any idea why any of this *isn't* mentioned in _any_ of the docs?
<jilocasin> then of course there's always /var/lib/cloud/data/set-hostname
<jilocasin> Uggg.
<factor> Does cloud-init know the IP address that would be assigned to the instance? So I could modify configs based on it?
#cloud-init 2020-08-17
<johnsonshi> What is the upstream team's planned cutoff date for the 20.3 release of cloud-init? I looked at the bi-weekly meetings, the cloud-init docs, and ubuntu wiki, but I couldn't find a projected release timeline.
<johnsonshi> I'm assuming the following historical precedent, the cutoff date for 20.3 will be sometime soon (next 2-3 weeks)?
<meena> "when it's ready" ?
<paride> smoser, hey, I'm reviewing https://github.com/canonical/uss-tableflip/pull/57 and wonder where get-orig-tarball comes from
<smoser> https://gist.github.com/smoser/fa77f61a2b650d115b5fc2a8ab028ede
<smoser> i'm fine if you want to pull that in somewhere else.
<smoser> paride: and if you know of an easier way to do that, i'm all for it.
<smoser> its suprisingly difficult in my experience to find an orig tarball.
<smoser> paride: do you know what the correct behavior is if upstream has a debian/ in the tarball ?
<smoser> does build-package extract the debian.tar.gz over the top? or remove the existing and then extract
<smoser> as we should make build-package do the right thing.
<chillysurfer> hello, all! could i get a review for merge on this pr? thanks in advance! smoser reviewed some time ago but not sure if another maintainer needs to check it out as well
<chillysurfer> https://github.com/canonical/cloud-init/pull/509 forgot the pr link
<johnsonshi> Ooops I got logged off so I would not have been able to see whether there was a response regarding the SRU cutoff date. :)
<johnsonshi> When is the projected SRU/release cut off date for cloud-init by the way?
<johnsonshi> (for 20.3).
<rick_h> johnsonshi:  soon, we're going to write up the announcement to prepare it tomorrow I think
<rick_h> johnsonshi:  so we're looking this week at WIP that needs to land and try to get a sha that's ready before EOW
<johnsonshi> rick_h: Thanks! So the likely timeline for the cutoff date is very soon?
<rick_h> johnsonshi:  this week for sure
<rick_h> johnsonshi:  what are you wanting to get in?
<johnsonshi> rick_h: Thanks! Just asking because I've got a couple of internal teammates asking about the cutoff date to align priorities.
<rick_h> johnsonshi:  ah ok. Let me know if there's anything we want to make sure and we'll try to help review/etc to get it in.
<AnhVoMSFT> rick_h: when will the next SRU be after this one?
<rick_h> AnhVoMSFT:  so after that we've got a project to dev the hotplug support through and the SRU will be after that. Maybe by end of cycle Oct?
<AnhVoMSFT> thanks rick_h
<AnhVoMSFT> If it's at all feasible it would be nice to land this one for this SRU https://github.com/canonical/cloud-init/pull/529
<rick_h> AnhVoMSFT:  ty, will do. Dan was trying to get to it before his holiday but we spoke today about getting it picked up by others. We'll get eyes on it.
<johnsonshi> Would the upstream team be open to allowing the Azure apply_network_config option to be set through Azure instance metadata? Referring to https://bugs.launchpad.net/cloud-init/+bug/1889112
<ubot5> Ubuntu bug 1889112 in cloud-init "cloud-init should support setting apply_network_config through Azure instance metadata" [Undecided,Triaged]
<johnsonshi> The Azure datasource's apply_network_config flag determines whether cloud-init is supposed to set up the instance's networking stack by querying IMDS.
<johnsonshi> As of now, this flag is only read by cloud-init from the disk configs at /etc/cloud/cloud.cfg.d/*.cfg.
<johnsonshi> By default, Azure apply_network_config flag is set to true for the Ubuntu 18.04 LTS images published on Azure.
<johnsonshi> A customer that wants to "opt out" of this flag cannot do it during deployment time, as this config cannot be set using the Azure datasource.
#cloud-init 2020-08-18
<otubo> smoser, can you confirm if this bug has been addressed? https://bugs.launchpad.net/ubuntu/+bug/1225922 I can't track anything on the commit log that could relate to this
<ubot5> Ubuntu bug 1225922 in linux-gcp (Ubuntu) "Support static network configuration even on already configured devices" [Undecided,Confirmed]
<rick_h> johnsonshi:  I don't think we've got anything against it. We haven't had a chance to look at the change yet but did discuss it and don't have any reason not to.
<smoser> otubo: i sure would think that is fixed.
<smoser> replied in the bug
<otubo> smoser, thanks a lot! Also there was another issue that - if you have some time to spare - I'd like you to check: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+ref/fix/1788915-vlan-sysconfig-rendering
<otubo> smoser, I also couldn't track on the commit log if this was merged or not. We might have an issue in RHEL that might benefit from this patch
<paride> smoser, I think `origtargz --download-only --tar-only` could be used instead of get-orig-tarball in build-package
<paride> it's included in devscripts
<paride> but it's doesn't do some tricks implemented by get-orig-tarball
<paride> like `get-orig-tarball for ...`
<smoser> thats the first i'd seen of that.
<smoser> a few things i like better from get-orig-tarball:
<smoser>  doesnt need apt source lines
<smoser> things i like in origtargz
<smoser>  uses pristine-tar (that'd be really cool if it worked with git-ubuntu pristine-tar)
<smoser>   that was something i wanted to add to get-orig-tarball
<smoser>  - uses uscan (but --download-current-version isnt sufficient)
<paride> and uscan it not guaranteed to fetch the .orig that was used for the packaging
<paride> for example in some packages I maintain I use uscan to get notified about new upstream releases
<paride> but then I import the upstream git tag and generate the orig tarball from my local git tree
<paride> with gbp-buildpackage or `git deborig`
<smoser> yeah.
<smoser> i find it surprising/terribly-annoying that there isn't a "right way" to do this.
<smoser> and a slew of wrong ways that will get your upload rejected
<rharper> otubo: smoser: on that vlan-sysconfig bug;  I had someone else bring it up but I've not been able to get a reliable reproducer (I don't have a real host with bonded vlan and switches (where the bond may takes some time to come up);  in particular from a sysconfig perspective, there was some mention of use of ONPARENT=yes ;   so in the case of RHEL7.latest images which have NM only (with the sysconfig helpers)  is there
<rharper> documentation on what should be present in the sysconfig file to do bonded vlans?    https://bugs.launchpad.net/bugs/1888315
<ubot5> Ubuntu bug 1888315 in MAAS "MAAS does not set type=VLAN for CentOS & RHEL" [Undecided,Triaged]
<smoser> otubo: wrt https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/366602 ... it just got dropped.
<smoser> i dont have anything local more than what is there.
<smoser> imo 'a' really isn't OK.
<smoser> that is just slogging on an pretending stuff is happy, hiding original failures.
<smoser> and doing something seemingly randomly different.
<smoser> but... doing it right (b) is harder, and failing (c) probably breaks someone's idea of "working"
<otubo> rharper, I'm not aware if there's a direction if we should use this option or not, but on the BZ[0] I have the reporter that might be willing to try the original patch from smoser. If that fixes the issue I might send a new PR with that code.
<otubo> [0] https://bugzilla.redhat.com/show_bug.cgi?id=1861871
<ubot5> bugzilla.redhat.com bug 1861871 in cloud-init "[rhel7][cloud-init] ifup bond0.504 Error: Connection activation failed: No suitable device found for this connection" [Unspecified,New]
<otubo> smoser, right, I'll give it a try, see if it really fixes the issue first. We'll see from there.
<otubo> *might be willing to try, meaning, the reporter has the hardware and the environment necessary :-)
<rharper> otubo: well, I believe it does resolve an issue; but I don't understand why;
<rharper> The other alternative w.r.t sysconfig and cloud-init  is to emit NM output;  for example in RHEL8, right, NM is the default now;  is the sysconfig compat layer still around?  If the sysconfig files are still supported; then we should figure out why what we currently write doesn't work for bonded vlans;  it's certainly a regression between sysconfig-only setups (where this code works perfectly fine) and NM-only setups which import
<rharper> sysconfig files.
<smoser> well, the right behavior is to render what the system is configured for.
<smoser> if it is NM, then render NM. if it is sysconfig, then sysconfig.
<rharper> we don't render NM;  AFACIT NM supports reading sysconfig files;  so yes, someone can write a NM renderer for cloud-init;  even without cloud-init though, either the sysconfig file we're writing is valid or invalid; and I'd like to understand why it works without NM; but when NM imports it; it doesn't;
<momousta> blackboxsw_, We would like to land this PR https://github.com/canonical/cloud-init/pull/529 as part of the next SRU, Can you take a look?
<blackboxsw_> momousta: yes thanks for the piing
<blackboxsw_> will look today
<blackboxsw_> just got back from "vacation": )
<blackboxsw_> momousta: I was wondering whether we want to make that filtering a bit more generic. I'll think a bit this morning and get you a concise response and/or review today
<blackboxsw_> momousta: could you run rebase your current PR  and resolve merge conflicts please? `git fetch -i origin;  git rebase -i origin/master; git push <your_remote>`  might do it.
<momousta> Sure
<blackboxsw_> thanks
<blackboxsw_> sorry about that git fetch -i origin; just git fetch origin;
<blackboxsw_> thx
<cut> is there any way to silence cloud-init so it doesn't print the "Created by.." message in various files in /etc/? "# Created by cloud-init on instance boot automatically, do not edit."
<blackboxsw_> cut, not by any configuration setting. you'd have to do post-processing on those files, something like " #cloud-config\nruncmd:\n - "sed -i 's/Created by cloud-init on instance boot automatically, do not edit./Silence safety message/' /etc/<your_file_path>.
<cut> thanks
<minimal> Hi folks. I saw the email about the planned release of 20.3 on Thursday. I wanted to check if there's a chance to get my PR #535 into this release. I'm currently working on changes based on the last set of review comments and hope to get them committed in the next couple of hours time.
<rharper> minimal: I'll review again
<minimal> rharper: Thanks
<blackboxsw_> momousta: only one significant note so far on your branch it looks like with every boot you'll end up dumping the full compressed cloud-init.log for every boot across kvp, that's going to get more and more costly the more times an azure instance is rebooted. https://github.com/canonical/cloud-init/pull/529/files#r472392050
<momousta> blackboxsw_. I'm looking into that now. Thanks for the quick feedback.
<blackboxsw_> I'm not sure, but you may want to think about persisting something with that byte-offset on disk somewhere in /var/lib/cloud/data/   like "azure-kvp-offset" if you ultimately want to continue to always log updated compressed cloud-init.logs per-boot
<blackboxsw_> no prob momousta
<rharper> minimal: I did review yesterday, did you see the comments about the ntp test-cases?
<minimal> rharper: yes I saw your review yesterday - I'm working on the changes for it currently. Hope to have them committed shortly.
<minimal> rharper: re: the cc_power_state_change.py re.match change I did add a note about 30mins ago to explain why I'd done that. I've since added "\s*" (zero or more whitespace) to the start and end of the regex pattern, that'll be committed shortly.
<rharper> minimal: great;  yeah; it looks like int() eats leading +,- and any number of whitespace (or trailing);  I just commented; my preference is to not change the code even if the docs don't match since if there's a cloud-config using something that passes the current regex; I want that config to still work rather than break because we tighted it;  OTOH, I don't think there's much else that int() won't catch besides + - and spaces
<rharper> (which it ignores anyhow;
<minimal> rharper: well the main thing I wanted to trap was someone trying to use '2:30' to shutdown in 2.5 hours time - this is valid for shutdown command if passed as a param in that format but doesn't match the docs for cc_power_state_change
<rharper> minimal: yeah, the exception message is nicer than the int() failure to convert
<rharper> but we could also print the same message in the exception for int()
<rharper> ValueError: invalid literal for int() with base 10: '+30:30'
<rharper> nm, they'll throw the error  before
<rharper> that logic likely needs a refactor there;   it seems to me that we should handle now, and then try int() and except ValueError on the conversion and reraise as TypeError like it currently does if if fails;
<rharper> minimal: https://paste.ubuntu.com/p/hQZtDR97Ft/
<rharper> then with your branch changes on top of that (we get to drop the regex since int() already handles what we expect
<minimal> rharper: ok, could go with that
<minimal> rharper: actually I'm confused :-) The code you pasted don't reflect the last commit I made a couple of days ago when I added the convert_delay function as you suggested and the if-alpine-else-others block to call convert_delay. Should I simply change the regex pattern back to the way I was and left things at that?
<rharper> minimal: sorry, I was just demonstrating the refactor around using the int() and exception so we could drop the regex;  I meant to have that change I posted integrated on top of your change;
<blackboxsw_> thanks again momousta I just added a few more notes, minor concerns to this branch. I'll peek at this again when you say it's ready for re-review
<rharper> minimal: so I think you can remove the try/except in convert_delay() and instead in the main function try: convert_delay() except: raise TypeError()
<minimal> rharper: from memory the try/except in convert_delay() is needed as for non-Alpine distros a value 'now' can be passed in and therefore returned to be used in the shutdown command
<rharper> minimal: yes, but also only calling convert_delay() if delay != 'now'
<blackboxsw_> merged https://github.com/canonical/cloud-init/pull/428 thanks otubo !
<blackboxsw_> That'll be in our next upstream release 20.3
* blackboxsw_ changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting Aug 25 16:15 UTC | 20.2 (Apr 26) | 20.3 (estimated Aug 19th) https://bugs.launchpad.net/cloud-init/+filebug
<blackboxsw_> community notice: As falcojr announced on the mailing list we are planning on cutting upstream release 20.3 thursday (and shortly thereafter starting an Ubuntu SRU process to release to Xenial, bionic and focal).
<blackboxsw_> community notice: if there are existing branches that folks really would like in this time-based release. please raise them here or on the mailing list
<blackboxsw_> on our list currently are the following PRs https://github.com/canonical/cloud-init/pull/529 ~https://github.com/canonical/cloud-init/pull/428~ https://github.com/canonical/cloud-init/pull/487
<blackboxsw_> we would like to land these branches in tip before cutting an upstream  release 20.3
<rharper> blackboxsw_: PR #535 is hopeful for it
<rharper> minimal and I are currently working to land this for Thursday
<blackboxsw_> #529 has a couple of review comments to discuss/address and I think we are good there on Azure.   #428 just landed   and #487 needs a little review love
<blackboxsw_> rharper: checking that out thanks. and will add it to the list
<minimal> blackboxsw_: yes I'm working on the remaining issues in PR #535 as we speak
<blackboxsw_> excellent minimal/rharper. just holler if we are stalling out on that. I think we have (side-channel) about 3 prs to land for grub fixes in debian/postinst for ubuntu/* related branches.
<blackboxsw_> I'm off on PTO for tomorrow (OddBloke is out all week), but falcojr and I will be trying to get through any outstanding reviews that are not already being handled by rharper and or smoser too.
<rharper> blackboxsw_: ok
<minimal> rharper: Hitting a problem with test_handler for cc_apk_configure - multiple test fails and all I suspect is the add_patch for write_file that you suggested I add
<minimal> I'm seeing tox errors: E       FileNotFoundError: [Errno 2] No such file or directory: '/tmp/ci-TestConfig.gc5tegx6/tmp/template_name-povkwjvo.tmpl'
<minimal> which is the temporary template file write by cc_apk_configure which it is then unable to load to do the config file generation from this template
<rharper> minimal: yeah, we write a temparary template file and then write the content sof that
<minimal> actually multiple unrelated tests across various modules are failing with the same error
<minimal> so somehow the changes I made in test_handler_apk_configure are failing large numbers of tests across the board
<minimal> 137 failed
<rharper> minimal: if you dump what you have (git diff origin/master) and paste; I can help refactor some of the unittests;
<minimal> its a big diff for the test_handler, approx 300-odd lines
<rharper> or push what you ahve to a branch and I'll pull from github;
<rharper> I've got what you've pushed locally; I can work on a refactor of what I suggested on the apk stuff;
<rharper> bbiab
<minimal> rharper: https://gist.github.com/dermotbradley/42e6f4cb7298498f9ce9a87d44800ed3
<rharper> minimal: ok
#cloud-init 2020-08-19
<rharper> minimal: https://paste.ubuntu.com/p/xszQq4Vc48/
<minimal> rharper: Thanks. Should the "import mock" be "from unittest import mock" or "from cloudinit.tests.helpers import mock"?
<rharper> yes
<rharper> I'm still working on getting tox to be happy
<rharper> I've a few more fixes since that post
<rharper> https://paste.ubuntu.com/p/tr6SRWKqjR/
<rharper> there, tox is happy with that
<rharper> minimal: either works
<minimal> yeah, went with  cloudinit.tests.helpers
<minimal> tests pass fine now. Thanks! Don't really follow the changes you provided, need to read up on Python mock some time
<minimal> Getting 2 flake2 errors about line > 79 which is 2 of the text.dedent lines for the expected content. Is there any way I can split/wrap those lines without it breaking the assert comparison?
<rharper> minimal: so, I changed to mocking out your _write_repo_contents method (which called util.write_files)  for the no-write case, it was easier to check the call count on that
<rharper> minimal: the second issue was the path to the template files, so I looked at test_handler_cc_ntp.py  and it sets up the helpers.Path object to the same reRooted path (self.new_root) this allowed the module to find the temporary template files it writes
<rharper> and the final bit was just extra new line on the dedents
<rharper> minimal: see my latest paste, I just left shifted the few tests that were too long
<minimal> I should have spotted the missing "\"s on the dedents myself.....
<rharper> it's not easy (IMO) to read the assertEqual output expectially for multi-line output
<rharper> so I basically do a print(expected); print(); print(actual)
<rharper> so I can see what the two things look like
<rharper> then it was obvious
<minimal> wasn't sure if left-shift the extra long sections would cause flake8 to complain about lack of indent. I'll do that now.
<rharper> I think since it's wrapped in a function call it's happy; plain tox came out clean after I did the left shift
<GI_Jack> hi
<GI_Jack> what resources exist for writing your own cloud-init modules
<minimal> Decided to shorten that repositories comment line in both cc module and test_handler to solve to line length problem
<rharper> GI_Jack: https://github.com/canonical/cloud-init/blob/master/cloudinit/config/cc_foo.py   is an example module;
<rharper> minimal: =)
<GI_Jack> most excellent, thank you kindly
<GI_Jack> new interest is writing custom cloud images... :)
<minimal> rharper: So just have to look at your ntp test_handler comments now.
<minimal> thanks for all your helper. Not sure what timezone you're in but I'm sure its late there
<rharper> https://cloudinit.readthedocs.io/en/latest/topics/hacking.html
<rharper> minimal: US Central;    np;
<rharper> for the ntp one;  that'll take more work;  it really should be a pytest.mark.parametrized ... but that may be a bit steep refactor;
<rharper> so I'd be happy with changes to not duplicate code paths with large bodies of if alpine else default ... and try to make the if else as *small* as possible, or use distro_name and/or client lookups into a mapping to adjust the separate results;
<rharper> bbiab
<minimal> rharper: over to you for review. Hopefully this can make it into 20.3 tomorrow.
<otubo> blackboxsw_, thanks a lot :)
<otubo> blackboxsw_, any chance to also take a look at https://github.com/canonical/cloud-init/pull/521
<b0urne> I launched an windows-server-2016 instance in my cloud environment, When I tried to log on via vnc, the server rebooted. After checked the system log, the server was rebooted by cloud-init. Is this a normal behavior of cloud-init?
<crandon> Hi! Is there a way to completely disable IPv6 via cloud-init? While I found the question on some forum I haven't seen any replies nor such option in the official docs
<rick_h> otubo:  he's out today. Did you want him to look at it when he gets back or I can ask someone else to peek at it later today
<otubo> rick_h, Hi! Yeah, doesn't matter actually, I'd be glad is anyone could review it :) thanks!
<otubo> *if anyone could review it
<rharper> crandon: it's not a directy cloud-init option, but you could use a bootscript to  run a script which uses sysctl to disable ipv6
<rharper> *directly*
<crandon> rharper: thx, I cam to the same conclusion.
<rharper> crandon: +1
<crandon> rharper: +1 = ?
<rharper> crandon: sorry just acknowledging your response;  +1 is sometimes used in code review to mark agreement
<falcojr> I'm looking at https://github.com/canonical/cloud-init/pull/521 (thanks otubo) and it's working around our DHCP sandboxing, but I'm wondering if we could find a better directory to run dhclient from
<falcojr> it's not uncommon to mount /var/tmp as noexec, so does it make sense to copy dhclient somewhere else before running it?
<minimal> rharper: working on your feedback now
<rharper> minimal: great
<AnhVoMSFT> @rharper @blackboxsw_ we have seen a few deployments that failed with these kind of messages "UnicodeEncodeError: 'latin-1' codec can't encode character '\u1ecb' in position 373: Body ('á»') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8."
<AnhVoMSFT> https://paste.ubuntu.com/p/fqjmzZ38fJ/
<AnhVoMSFT> I tracked that down to basically the hostname that the customer specified had some unicode character in it, which results in the instance_id returned from wireserver having the unicode character
<AnhVoMSFT> is latin-1 the default encoding that python will use, if I were to encode the body of the report ready request to utf-8 would that help?
<rick_h> AnhVoMSFT:  looking at the python code there a comment states it has to be laten-1 per RFC :/ so yea have to encode it first though then the question is what is going to read/decode it after the fact. What is it POST'ing to?
<rick_h> AnhVoMSFT:  https://paste.ubuntu.com/p/Pqf5QxvvCp/
<AnhVoMSFT> it's posting to the wireserver, a component in the Azure platform. I'm verifying whether it can handle UTF-8. It probably should, since it has been supporting unicode hostname on the Windows side
<rick_h> looks like the code around there has changed a bit. send_ready_signal() vs _report_ready() but yea I would expect just needing to encode the data in there somewhere to be wire-safe would be on par
<rick_h> interesting on the latin-1 requirement for the RFC in the http library.
<minimal> rharper: cc_ntp.cc, the names.append you suggested (line 393) hitting pylint/flake errors.
<rharper> ok, I;ll try to clean up in a sec
<rharper> minimal: what you've pushed to tip ?  or some additional delta?
<minimal> rharper: what I pushed 1 hour ago
<rharper> and a change I suggested
<rharper> minimal: looks like just formatting; https://paste.ubuntu.com/p/YgqF4RvjJq/
<minimal> rharper: I had a missing "]". Tox passing locally, just pushed again so buildjob should pass fully this time
<rharper> yeah, and your suggestion works with fewer list joins, so +1
<minimal> rharper: Saw you re-wording comment. Happy with this: "By default, cloud-init will generate a new repositories file ``/etc/apk/repositories`` based on any valid configuration settings specified within a apk_repos section of cloud config."?
<rharper> yes
<rharper> minimal: yeah, that looks great
<minimal> rharper: So what's left to get this merged?
<rharper> minimal: nothing, I've approved;
<rharper> I was waiting for CI to check out as well
<rharper> I'll merge
<minimal> rharper: Great stuff. Thanks for all your help
<rharper> minimal: Thank you for the contribution and active refactoring!
<minimal> rharper: No problem. Now I can prep to update the Alpine package tomorrow or Friday
<macjunkie> provisioning some AWS instances and referencing a cloud-init template in terraform the template (appears to get end up on the instance) but nothing ever gets executed. I found user-data.txt in /var/lib/cloud/instance on the instance (ubuntu 16) but no activity in /var/log/cloud-init.log etc..
<macjunkie> anyone have any ideas?
<minimal> macjunkie: what about /var/lib/cloud/data/status.json?
<macjunkie> nope no mention of user-data
<minimal> that'll show if any error occurred during each of the cloud-init stages
<macjunkie> i don't see any errors listed
<minimal> Ubuntu's systemd based, anything in the journal?
<macjunkie> woohoo found it
<macjunkie> "util.py[WARNING]: Failed loading yaml blob. Invalid format at line 24 column 6: "while scanning a plain scalar"
<macjunkie> it is processing it
<macjunkie> now to fix my f'ing yaml
<macjunkie> thx
#cloud-init 2020-08-20
<momousta> blackboxsw_, This PR https://github.com/canonical/cloud-init/pull/529 is now ready for rereview.
<blackboxsw_> momousta: excellent. I'll peek at it tomorrow
<blackboxsw_> BTW on first glance. things look good. will peek in depth tomrrow morn
<momousta> Great, thanks in advance.
<blackboxsw_> momousta: if around. review complete on your PR with the expection of one question https://github.com/canonical/cloud-init/pull/529#pullrequestreview-471784769
<blackboxsw_> and 2 minor inline comments left on your branch
<momousta> Sure, I'll take a look. Thank
<momousta> blackboxsw_, answered the question and incorporated the suggested changes
<blackboxsw_> momousta: thanks. I'm giving this one more run in azure and will land it
<blackboxsw_> merged https://github.com/canonical/cloud-init/pull/529 thanks momousta
<blackboxsw_> falcojr: I just merged Odd_Bloke's xenial grub fix. looking over yours now
<blackboxsw_> https://github.com/canonical/cloud-init/pull/514
<blackboxsw_> falcojr: https://github.com/canonical/cloud-init/pull/537 landed
<blackboxsw_> https://github.com/canonical/cloud-init/pull/538 landed
<blackboxsw_> so xenial bionic and focal in good shape
<blackboxsw_> for grub-related issues
<blackboxsw_> falcojr: I'm wrapping up review comments on compressed ud via cloud-init query branch https://github.com/canonical/cloud-init/pull/516
<blackboxsw_> when that's landed, I think we are a "go" for cutting upstream release 20.3
<blackboxsw_> rharper and minimal thanks for the work, review and landing Alpine linux support via https://github.com/canonical/cloud-init/pull/535
<blackboxsw_> I don't *think* there were any other branches that we were waiting on for upstream release of cloud-init 20.3
<blackboxsw_> if there are other in flight PRs that we expect to land before SRU, someone can bonk me over the head with a keyboard.. otherwise, by EOD today, I think we are ready to cut that release
<rharper> blackboxsw_:  sure;  I don't know of anything else
<blackboxsw_> thanks. ok awaiting CI on https://github.com/canonical/cloud-init/pull/516 falcojr, then we can cut a new upstream 20.3 I think
<blackboxsw_> that should contain the handling of compressed userdata from cloud-init query (which breaks juju deployed cloudinit vms)
<blackboxsw_> ... well, which breaks ubuntu-advantage-tools `ua attach` of juju deployed vms on clouds
<blackboxsw_> to be more specific
<blackboxsw_> ok one more branch that we are hoping to close out on for this upstream release.... if possible: https://github.com/canonical/cloud-init/pull/516 needs an upstream reviewer +1 rharper or smoser if you guys get a chance. OddBloke is out for two weeks
<blackboxsw_> I addressed all review comments
<blackboxsw_> failure case is testable by juju deploy ubuntu on ec2 and trying to run 'cloud-init query --all'
<blackboxsw_> github won't let me merge it as I haven't gotten an official upstream dev +1
<rharper> blackboxsw_: lemme look
<blackboxsw_> thanks ryan
<rharper> blackboxsw_: reviewed; just a few suggestions on the docstrings;  and I don't have strong opinions on the pytest stuff so I hope suggestions/changes to that can be done separately if we land the PR today
<blackboxsw_> thanks rharper I've addressed those docstring concerns, I think I was trying to document too much in the load_userdata function. That base64-encoding is what cloud-init utility function load_json does when it can't serialize binary data (before we ultimately output the JSON instance-data merged dictionary)
<blackboxsw_> so that docstring didn't really belong in that load_userdata function
<blackboxsw_> and I figured the unittest rework I'd have to talk to Odd_Bloke about in 2 weeks just to see if generally we can make the pytest fixture a bit more flexible for the use-case I have
<rharper> ok
<blackboxsw_> thanks again. I think we are awaiting feedback on one more Azure PR for boot timeouts, but that should wrap it for this upstream release
<rharper> nice
#cloud-init 2020-08-21
<smoser> blackboxsw_: well, i reviewed there. (https://github.com/canonical/cloud-init/pull/516)
<paride> blackboxsw_, lucasmoura, falcojr, https://bugs.launchpad.net/cloud-init/+bug/1776958
<ubot5> Ubuntu bug 1776958 in cloud-init "error creating lxdbr0." [Medium,New]
<paride> reason for the CI failure I mentioned before
<blackboxsw_> thanks paride
<blackboxsw_> smoser thanks for the review. I'll address what I can on that today
<blackboxsw_> falcojr: rick_h also just merged the ssh key type support PR as well https://github.com/canonical/cloud-init/pull/487
<rick_h> cool, falcojr are we going ahead then with the dhcp pr and that's the last one for release lock down?
<falcojr> yep
<falcojr> wait, rick_h what about the sr_iov driver issue?
#cloud-init 2020-08-23
<minimal> Hi folks. Is there a new ETA for the 20.3 release?
<meena> minimal: last time i heard an ETA, it was last Thursdayâ¦ given that there's still no release, maybe that was just a rumour
<minimal> meena: yeah that was the plan. There were 2 PRs that were merged on Thu and 1 on Friday and I see 2 new MRs were opened recently so wasn't sure if they might hold up the release further or if its likely to come out some time on Monday
<meena> is this going to be another SRU release?
<minimal> not familiar with the term SRU
<nargit> hello, I am a completly noob to cloud-init but I would like to install a ubuntu server for raspberry pi. I just try to understand how to passe the config.yaml at boot time ?
<nargit> pass*
<nargit> because I need to flash the img to an sd card, but cannot find any doc on how to include my config.yaml file to the img
