#cloud-init 2014-04-21
<sauce> hey smoser 
<smoser> sauce, here now .
<smoser> whats up?
<sauce> just saying hey
<sauce> cloud-init is such a joy
<smoser> :) thanks.
<smoser> feel free to say "but it could be better if .... "
<sauce> i've been using ubuntu 12.04 LTS AMI, which comes with cloud-init pre installed
<sauce> well, all i use it for is 1 thing: configuring puppet.  puppet handles the rest
<sauce> i probably don't need the complication of puppet in my setup but i enjoy using it
#cloud-init 2014-04-23
<harlowja> any bugs are smoser fault sauce_ 
<harlowja> haha
<harlowja> *not mine
<yjiang5> smoser: hi
<smoser> hi
<yjiang5> smoser: I'm talking with alexpilotti on the config drive format support.  alexpilotti is considering to support only iso format, but version 1 of cloud-init has only vfat format. Do you know how is the usage of cloud-init version 1?
<alexpilotti> smoser: hey there!
<smoser> yjiang5, i'm on a call now. can read respond in 20 minutes
<yjiang5> smoser: sure.
<smoser> ok. 
<smoser> yjiang5, alexpilotti here now.
<smoser> sorry.
<smoser> whats the question ?
<smoser> what is "version 1" ? 
<smoser> the config-drive-v1 ?
<smoser> i was initially more strict on what i would take in as config drive.  i would only reaad a "full disk" and only if it was the last disk on the system.
<smoser> and people wanted it to be more forgiving.
<smoser> yjiang5, alexpilotti i would'nt concern myself with config-drive-v1.
<alexpilotti> smoser: my point is that multiple configdrive fs and disk formats = more support hassles
<alexpilotti> smoser: Iâd like to keep only iso9660 + CD/DVD
<alexpilotti> smoser: thus deprecating vfat and raw hdd
<smoser> i dont think it makes sense to say CD/DVD
<smoser> its only your broken OS that cares about such things
<smoser> the requirement for attaching a CD in order to pass data is annoying.
<alexpilotti> smoser: so you think itâs feasible or there are still lots of images using versions of cloud-init with vfat support only or no cdrom?
<smoser> i'd have to dig up data on what versions of cloud-init are where.
<smoser> i really dont knwo ff the top of my head.
<smoser> but i really would suggest that we not require hardware toimplement a virtual CD-rom in order to implement config drive.
<alexpilotti> afaik you added cdrom in 0.7 or 0.7.1 max
<alexpilotti> no idea for iso9660, that was wayy before I got started with OpenStack :-D
<smoser> think about that, you're saying new hypervisors will be required to have support for shiney circles with data on them.
<smoser> how does openstack create vfat at the moment?
<alexpilotti> depends on a pragmatic point IMO: if the images out there w/o iso / cd are like 5%, Iâd say letâs move on
<smoser> you'r suggesting lets move off of a ISO standard
<alexpilotti> as they move their shiney butts and update cloud-init :-)
<smoser> on to a possibly patent encumbered filesystem
<alexpilotti> ISO + cd/dvd drive
<smoser> oh. is see.
<smoser> i say ISO + hard-disk
<smoser> is the saner path.
<alexpilotti> hard disk is a no go on Woindows before 2012
<smoser> and you have to fix your silly operating system that thinks iso9660 filesystems can only be found on cd-roms.
<smoser> because if you go the route of "iso + cd/drive"
<alexpilotti> smoser: thatâs not the point: itâs the raw hdd the issue
<smoser> then you force all future hypervisors to implement CD roms
<alexpilotti> windows expects to have partitions on disks
<smoser> right.
<smoser> so you're saying either:
<alexpilotti> smoser: arenât they all supporting cdroms already?
<smoser> smoser-hypervisor-2015 doesn't.
<smoser> why would i implmeent a cdrom ? ide ? whats iDE ?
<smoser> this is 2015!
<alexpilotti> well smoser-nypervisor has plenty of time to look at the specs before release ;-)
<smoser> i'm suggesting that CDroms are more arcane than filessystems on unpartitioned disks.
<alexpilotti> Iâm suggesting that plenty of people use guests OSs that donât support unpartitioned disks
<alexpilotti> and Iâm surely not gonna tell them that openstack is not the tool for them
<smoser> i reallydont undstand it.
<smoser> i admit to not understanding windows
<smoser> but somehow, somewhere, you can open a device
<smoser> and read data from it
<alexpilotti> what we do now on 2012 is:
<smoser> and then realize 'oh look, these bytes look like a filesystem!'
<alexpilotti> open the device, dd out of it on a file
<alexpilotti> mount the file (iso) as a loopback device
<alexpilotti> read from it
<smoser> why mount it as a looback ?
<smoser> why "mount" at all
<alexpilotti> because I need to access the content
<smoser> there are many user space iso9660 filesystem implementations.
<alexpilotti> as I have no native tool to extract it
<alexpilotti> I need an Apache 2 friendly tool
<alexpilotti> lib I mean
<smoser> see.
<alexpilotti> or whatever OS tool that I can bundle with cloudbase-init
<smoser> now you're talking about personal problems
<smoser> :)
<alexpilotti> well not really ;-)
<alexpilotti> cloudbase-init is Apache 2 and itâs surely not getting GPLed due to a lib
<alexpilotti> anyway I didnât really find one
<alexpilotti> at some point I started thinking on writing an ISO lib to extract the content
<smoser> ah.
<smoser> apache2 ?
<smoser> how about bsdtar
<smoser> that good enough ?
<alexpilotti> btw 7-zip  fails
<alexpilotti> for some reason teh ISO images taht come out from the the iso tool used on teh server are not valid for it
<smoser> libarchive13
<alexpilotti> IMO due to the very small size
<alexpilotti> bsdtar?
<smoser> bsdtar depends on libarchive
<smoser> libarchive supports reading iso9660
<smoser> http://www.libarchive.org/
<smoser> BSD license.
<alexpilotti> I tested this one, no luck
<alexpilotti> but Iâm gonna do another round, they might have fixed the issues I had
<alexpilotti> consider that almost 2 years passed
<smoser> i'd be surprised if it didnt work.
<smoser> http://gnuwin32.sourceforge.net/packages/libarchive.htm
<alexpilotti> consider that 7-zip, the most popular OSS archiver on WIndows, didnât
<alexpilotti> doing a build now, if this works, well, problem solved
<alexpilotti> cmake is still crying for mising posix stuff, will take a bit to finish
<alexpilotti> in the meaning: did you do anything with MaaSâs fast boot for WIndows thing since we talked last time?
<alexpilotti> s/meaning/meantime/
<yjiang5> alexpilotti: smoser sorry, just back for lunch.
<smoser> alexpilotti, i've not done anything on maas fastboot for windows, no.
<yjiang5> alexpilotti: http://openstack.10931.n7.nabble.com/HDD-type-config-drive-and-Windows-VM-td32968.html give some interesting point that they want vfat so that the cloud-init can delete all contents for security reason .
<alexpilotti> yjiang5_away: well, you could wipe an ISO raw hdd as well
<harlowja> i've used winrar to extract iso's on windows before
<harlowja> guess thats not apache2 though
#cloud-init 2014-04-24
<smoser> alexpilotti, is correct. 'dd if=/dev/zero' is better than 'rm $MOUNTPOINT/*'
<alexpilotti> smoser: not sure what you meant :-)
<alexpilotti> smoser: in the meantime: libarchive compiled, looks promising
<smoser> alexpilotti, in your response to yjiang5
<smoser> most certainly, vfat is better from a data security perspecitve
<smoser> than iso.
<alexpilotti> smoser: why? whatâs the difference in wiping with dd a raw hdd containing iso vs vfat?
<smoser> as why  would you 'rm *' when you can 'dd if=/dev/zero of=/dev/by-label/CONFIG_DRIVE'
<smoser> bah
<smoser> vfat is *no* better
<smoser> (that was an important word to miss :)
<smoser> ie, i agree with " well, you could wipe an ISO raw hdd as well" completely
<smoser> which is actually an argument against CDROM
<smoser> :)
<smoser> unless you were going to attach a cdrw
<alexpilotti> smoser: on that point, I agree
<alexpilotti> smoser: but, metadata are no secure mean
<alexpilotti> so giving the message that you can secure a clear-text config drive just becuase your attacker is not fast enough is IMo absolutely wrong
<alexpilotti> especially if somebody pretends to put passwords in there
<alexpilotti> my 2c are that natural selection should take itâs toll in such cases ;-)
<alexpilotti> anyway, if your suggestion with libarchive works fine, my concerns for not using a raw hdd disappear
<alexpilotti> and we already agree on ISO, from what I got so far
<smoser> alexpilotti, your attacker does not have access to your system before you have config drive wiped.
<smoser> if he does, then all bets are off.
<smoser> ie, if he's rooted you before rc.local is run, you are compltely SOL
<smoser> i think we can manage to secure things to thath poitn :)
<alexpilotti> smoser: what about faulty heat templates for example?
<alexpilotti> beside that, we also donât have that advantage on Windows
<smoser> you're suggesting that my system is rooted before its booted.
<alexpilotti> while it boots
<alexpilotti> I guess youâre going to do something with those metadata :-)
<alexpilotti> true that you can wipe them off before starting any activity 
<alexpilotti> this mitigates a bit more
<alexpilotti> this precludes anyway execting specific plugins at each boot
<alexpilotti> unless you plan to store the metadata somewhere, but then weâre at the starting point
<alexpilotti> you could use some symmetric encryption
<alexpilotti> not safe anyway
<smoser> alexpilotti, yeah, actually you're right.
<smoser> the attacker can't read /dev/sdb without root
<smoser> and once they have root, well, you lose
<alexpilotti> yep
<smoser> and if they had exploited you before you 'mount /dev/sdb /mnt'
<smoser> then you lose anyway
<smoser> so, yeah, you're right.
<alexpilotti> this thing of how to handle sensitive info in the metadata is quite hot
<alexpilotti> what we did for the passwords in Nova is IMO cool and could work in other scenarios
<alexpilotti> where the plugins generate some secret, encrypt it with the userâs SSH pub key and POST them to some metadata service
<alexpilotti> configdrive wonât be the case I guess
<alexpilotti> smoser pushed the code with bsdtar support, I owe you a beer :-)
<smoser> whoohoo. 
<smoser> bsdtar as in actually that binary ?
<smoser> and not just using the library ?
<smoser> i figured you'd have to use the library
<smoser> but that s great
<smoser> hm..
<praneshp> smoser: yt?
<praneshp> How can I run files in cloudinit/config individually?
<praneshp> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/config/cc_apt_configure.py
<praneshp> for eg
<praneshp> is there a nice handler tool available or do I just have to work backwards and find the righ targs to pass??
<harlowja> praneshp u all under control now i hope :)
<praneshp> yup
<alexpilotti> smoser: yep, itâs part of http://www.libarchive.org/
<alexpilotti> by building it you get both the lib and the exe
<alexpilotti> so I just wen with a simple âbsdtar -xf xxx -C xxxâ 
<alexpilotti> for once I avoided some crazy ctypes work
#cloud-init 2014-04-25
<sauce_> i notice that when using cloud-init (with ec2), there is quite a delay in the machine booting up and cloud-init doing its thing. i judge "doing its thing" by tailing syslog.  any way to reduce the delay?
<harlowja> sauce_ reduce the number of services in your operating system that startup before cloudinit
<harlowja> mininize packages, services, ...
<sauce_> harlowja its the bare ubuntu 12.04 AMI.  not much is going on
<sauce_> i watch it just sit there for about 60 seconds, maybe more.  i'm patient so i don't mind :)
<sauce_> but i was just wondering why
<harlowja> where is this running?
<harlowja> amazon?
<sauce_> yes
<harlowja> so then u might have hit a hypervisor in amazon that is heavily loaded
<harlowja> and thats why its taking longer
<harlowja> http://allthingsd.com/20130225/the-problem-with-noisy-neighbors-in-the-cloud/
<sauce_> is this common though?
<harlowja> i'm unsure, the general problem is known
<harlowja> but i don't use amazon, so not sure
<sauce_> ok cool so i guess that answers my question: cloud-init does not have any sort of delay programmed in
<harlowja> sauce_ ya, it doesn't
#cloud-init 2015-04-20
<Odd_Bloke> smoser: utlemming: I've attached a debdiff to https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1398997; is it worth trying to get that into vivid?
<Odd_Bloke> (It also affects trusty and utopic, but not precise)
<smoser> Odd_Bloke, its by design
<smoser> well... maybe
<smoser> let me find bug
<Odd_Bloke> smoser: Ah, is this because SmartOS uses the serial console which can make other things sad?
<Odd_Bloke> Looking at precise, it's included in the choices, but _not_ in defaults.
<smoser> bug 1316475 is the bug that disabled it.
<Odd_Bloke> smoser: That's (mostly about) CloudSigma, not SmartOS.
<Odd_Bloke> Or is it also SmartOS because SmartOS falls in to the 'negative side effects' category?
<Odd_Bloke> Oh, but CloudSigma is in the default list still.
<smoser> Odd_Bloke, yeah, cloudsigma had negative side effects so it got taken out of the image.
<smoser> and just never put back in
<smoser> but the fix that wawas added should be sufifcient
<Odd_Bloke> smoser: Looking at the latest packaging, CloudSigma is in the default list...
<smoser> i'm confused.
<smoser> yeah, you're right. its in vivid/debian/cloud-init.templates as defaults.
<smoser> cloud biuld process seedxs that list maybe ?
<Odd_Bloke> For cloud-specific images, we generally drop a file in to cloud.cfg.d that overrides the list completely.
<Odd_Bloke> So I don't think this is a problem for us, necessarily, just people who want to install cloud-init themselves.
<smoser> ok. so, we *should* re-enable it. as the fix that was put in should be sufficient to avoid the negative side affect that caused its removal
<smoser> but i wouldn't do that for vivid "right now". just to avoid dealing with regression 3 days before release. escpecially as you said, because it has a work around
<smoser> which makes it "medium" by definition i think
<smoser> and not "kitten killer"
<Odd_Bloke> Ack.
<smoser> but i woudl support it for SRU
<smoser> Odd_Bloke, why did you say lcoudsigma above ?
<smoser> oh. i see. i did that.
<smoser> i'm osrry . confusing. i was just mis-remembering
<smoser> i dont know why SmartOS is not i nthat list. probalby just missed it.
<smoser> it would appear it SmartOS just never got in to that list.
<smoser> but it seems that smartos does the same look (dmi info) so it should be fine.
<Odd_Bloke> Yeah.
<Odd_Bloke> I'll revisit it early next cycle.
<smoser> can you confirm i nominated approrrately ?
<smoser> and mark 'confirmed' and importance on Trusty anhd W ?
<smoser> (i just ignored utopic. i'd not bother with SRU for this to utopic)
<Odd_Bloke> smoser: Confirmed; I can't do importance.
<smoser> k. there ya go
<Odd_Bloke> smoser: Thanks!
<smoser> sorry for confusing cloudsigma and smartos
<smoser> :)
<Odd_Bloke> No worries; the reasons for them being off this list are very similar. :p
<smoser> interestingly, they are the same in this regard (they both expect something ont he othe rside of /dev/ttySX, and now both look for dmi data before polling around there)
<smoser> Odd_Bloke, i put a comment in https://code.launchpad.net/~daniel-thewatkins/cloud-init/lp1403617/+merge/256812
<smoser> but i'm willing to be wrong on it
<smoser> since we're already able to test read_url . 
<Odd_Bloke> smoser: No, it makes sense; I was initially relying on more state in that function, but refactored it to avoid that need.
<Odd_Bloke> smoser: I've made the change locally, will push it up some time this afternoon after doing a quick smoke test.
<Odd_Bloke> (About to head in to a meeting)
<Odd_Bloke> smoser: Updated that MP.
#cloud-init 2015-04-21
<suro-patz> smoser: I would appreciate, if you can review https://code.launchpad.net/~suro-patz/cloud-init/vm-clone-ip-reusage-dep-fix, which I came up with as per your review comment on https://code.launchpad.net/~suro-patz/cloud-init/vm-clone-ip-reusage-issue
<smoser> suro-patz, that sure seems reasonable
<suro-patz> smoser: cool, thanks for having a look
<smoser> thank you for your patience
<smoser> suro-patz, do you consider any of the bugs you mentioned fixed by your commit 
<suro-patz> yes, I had verified with Rhel, and the bug mentioned for image creation from snapshot looked to be fixed
<smoser> which bug though. 
<smoser> both ?
<suro-patz> the bug description for 1275098 definitely is addressed 
<suro-patz> I am checking the desc of the other
<suro-patz> brb
<smoser> suro-patz, thanks. pushed
<suro-patz> smoser: appreciate it!
<harlowja> smoser alexpilotti should we have like a sprint or something (on the phone or whatever) about cloud-init v2.0
<harlowja> get something to happen there...
<harlowja> *aka dedicated time...
<harlowja> *happy fun time
#cloud-init 2015-04-22
<smoser> harlowja, well, that might be happening, with claudiopoppa next week
<smoser> but yeah
<Odd_Bloke> harlowja_away: Thanks for the merge!
<harlowja> smoser cool
<harlowja> let's get er done
#cloud-init 2015-04-23
<innosam> Hi, i was looking to build cloud-init from source as debian packages. Anybody have idea on, how to do that?
<innosam> Hello
<alexpilotti> smoser harlowja: just got the plane tickets for Malta
<harlowja> malta, i don't think i be at that one :-P
<alexpilotti> smoser: cpopa will be there as well
<harlowja> unless smoser  pays for me to go, lol
<alexpilotti> aka next week :-)
<harlowja> cool
<harlowja> sounds nice 
<alexpilotti> harlowja: Iâll feed him some beers and try to convince him to buy your ticket :-)
<harlowja> ha
<harlowja> thx
<alexpilotti> so what about we do some hangouts?
<alexpilotti> I mean next week?
<smoser> alexpilotti, hey. awesome. i was just thkning 'why didn't allesandro respond to me'
<alexpilotti> smoser: we like to live dangerously and we handle event logistics at the last possible moment :-D
<alexpilotti> smoser: whene did you write me? I might have missed it!
<devicenull> I dont suppose anyone here knows how to convince dpkg to actually reconfigure things in ubuntu?
<devicenull> it seem that whenever I run 'dpkg-reconfigured -f noninteractive cloud-init', it just updates the debconf database with whatever I have in 90_dpkg.conf
<smoser> devicenull, what did you want it to do ?
<devicenull> https://gist.githubusercontent.com/devicenull/40c80e71bf8ce21bff1f/raw/3e443af8e83848c4a9bc8990c764cf257b79255f/gistfile1.txt
<devicenull> well, that's what it does.. I wanted it to set the metadata config to Ec2, None
<devicenull> I just resorted to overwriting 90_dpkg, which actually works
<smoser> devicenull, i think thats as intended.
<smoser> ie, debconf-is-not-a-registry
<smoser> you can seed package installation
<smoser> but after that, i think its supposed to behave like it is
<devicenull> well, I'm not sure what the point of the 'noninteractive' frotnend is then
<devicenull> I dont use ubuntu much, so I could be doing something wrong here
<smoser> devicenull, i'm sorry, this has always been confusing for me too.
<smoser> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=709773
<devicenull> ahh, so debconf values are generated based on what's actually configured on the machine
<devicenull> rather then debconf being the source of truth, and updating files based on what debconf says
<harmw> god I hate openstack
<harmw> should a vxlan deploymnt contain any vxlan specific iptable rules?
<smoser> devicenull, that is my understanding, yes.
<smoser> on first install, cloud-init wil read preseed values
<smoser> but after that, debconf-is-not-a-database. (but it is still confusing to me)
#cloud-init 2015-04-24
<smoser> JayF, harlowja thanks for input.
<harlowja> np
<JayF> smoser: Nathan from Rackspace who you've been tlaking to is now a direct co-worker of mine
<JayF> smoser: just fyi
<harlowja> suro-patz1 & mmorais are direct co-workers of mine (so u don't feel left out smoser ) 
<harlowja> :-P
#cloud-init 2015-04-26
<smoser> harmw, where would you get a freebsd cloud image ?
<harmw> I'd build one myself
<harmw> though they do offer vm images
<smoser> link ?
<smoser> all i see is http://images.openstack.nctu.edu.tw/bsd-cloudinit/
<harmw> nono
<harmw> thats some other project
<harmw> ftp://ftp.freebsd.org/pub/FreeBSD/releases/VM-IMAGES/
<harmw> but those lack cloud, eg. no cloud-init
<smoser> do they boot ?
<harmw> in a vm, yes
<smoser> are they usable ?
<smoser> can i ssh in ?
<harmw> idk :)
<harmw> i've only build my own, using oz
<harmw> and installed ci later on
<smoser> k. thanks.
<smoser> harmw, your mission is to get cloud-init installed in one of those off the shelf images.
<smoser> and submit to http://thornelabs.net/2014/06/01/where-to-find-openstack-cloud-images.html on how you did it :)
<smoser> harmw, i was just trying to get a freebsd image.
<smoser> to add to http://smoser.brickies.net/image-streams/
<smoser> that is 'streams' format of data. and you can then sync it into your openstack cloud using http://bazaar.launchpad.net/~smoser/simplestreams/example-sync/files
<harmw> yea, been wanting to do get ci in an official image
<smoser> http://paste.ubuntu.com/10902086/
<harmw> and I'm watching the Hobbit, so quest seems more appropriate
<smoser> thats the script that we have syncing smoser-images into our cloud.
<smoser> harmw, well, for cloud-init 2, we're going to want c-i testing cluod-init trunk on freebsd.
<smoser> so some automated mechanism for doing that wooill be needed.
<harmw> ah yes, you sent an e-mail regarding 2.0
<harmw> damn
<harmw> forgot all about that
<harmw> but yes, getting fbsd on par some more is on the short list
<harmw> (but that list is already filled beyond everyting :P)
<nvucinic> q
#cloud-init 2016-04-26
<frickler> did cloud-init change the handling of ssh-keys recently(ish)? I'm having trouble getting multiple keys into my xenial instance. with cirros-0.3.4 this is working fine
<smoser> frickler, hm.. i dont think so
<frickler> smoser: right, so this is a semi-bug, but seems to have existed for quite some time. .ssh/authorized_keys works quite fine with entries containing only keytype and keyhash, but no description after that
<smoser> yeah. i think that was fixed actually to ugh
<frickler> cloud-init seems to choke on that, though, and pastes the second key in the same line after the first one
<smoser> frickler, you're right. i must have mis-remembered. maybe there is an open bug on this.
<frickler> smoser: I'll do some more testing and create a new one if necessary, thx
#cloud-init 2016-04-27
<jogo> is there any way to install cloud-init from pip?
#cloud-init 2016-04-28
<Odd_Bloke> jogo: I don't believe so; it's very much designed to operate as a system package (rather than a Python library).
<Odd_Bloke> jogo: (And you aren't installing pip packages globally, right? ;)
<jogo> Odd_Bloke: asking in regards to testing code via unit tests
<jogo> we have some code that imports some cloud-init stuff (although that may  be the part we shouldn't be doing ...)
<smoser> jogo, it should pip install right from bzr
<smoser> to a virtual env.
<smoser> trunk runs unit tests in tox which uses virtual env
<jogo> smoser: oh right doh, I forgot pip supports that
<jogo> nice
<jogo> smoser: thanks, don't know how I forgot about pip's ability to install from bzr and git
#cloud-init 2016-04-29
<clungemonkey> hello
<clungemonkey> anyone here
<clungemonkey> so
<clungemonkey> I'm still getting this error
<clungemonkey> util.py[WARNING]: Failed to set the hostname to salt-test7.novalocal (salt-test7)
<clungemonkey> bugzilla says its fixed
<clungemonkey> but no
<clungemonkey> https://bugzilla.redhat.com/show_bug.cgi?id=964006
<clungemonkey> hello who's here
<nacc> clungemonkey: i would expect this to be a place to discuss upstream cloud-init, seems like you should contact RH for support if you're on RHEL7?
#cloud-init 2017-04-24
<smoser> rharper, so i dont think this behavior has changed.
<smoser> but in order to make ds-identify be strict on ec2, you have to set the cloud-init setting in either /etc/cloud/cloud.cfg or /etc/cloud/cloud.cfg.d/[Ee][Cc]2*.conf
<smoser> datasource:
<smoser>  Ec2:
<smoser>   strict_id: true
<smoser> basically, its not a ds-identify setting, its a cloud-init setting that ds-identify respects, (and attempts to read more quickly by being specific on which files are read)
<rharper> so, I know it's changed; I built the image and tested explicitly in the core image that if you don't supply a datasource, it skips starting cloud-init;
<rharper> when I rebuilt the core snap on Friday, it picked up cloud-init from xenial-updates and it ran cloud-init and tried to hit ec2
<rharper> how would we have gotten to ec2 datasource anyhow if I have found=first,maybe=all,notfound=disabled ?  maybe can't have network datasources, and if we didn't find ec2, then we should have disabled cloud-init
<rharper> smoser: ^
<smoser> ec2 returns maybe unless strict is set
<smoser> it has to. as "maybe there is an ec2 metadata service here"
<rharper> on my local system ?
<smoser> possibly you ran previously on non-intel ?
<rharper> xenial laptop
<smoser> how am i to knwo that there is not a ec2 metadata service there?
<rharper> lol
<rharper> I though if we don't see ec2 in UUID, then no network hit unless you have notfound=enabled ?
<rharper> s/though/thought
<rharper> maybe says that no network datasource can return a  maybe
<rharper> so, found=first would have had to have had a uuid of startswith('ec2')
<smoser> that is what strict is
<smoser> you're defining the behavior in strict
<rharper> so, let me ask a different way;  if I want to prevent cloud-init from running unless it's found a ds;  I don't want maybes at all (even though they are supposedly non-network);  what do I set ds-identify to ?
<smoser> maybe isn't "no  network", its just "there is no information available that says this is not a potential source".
<smoser> "no network sources are allowed to return 'maybe'."
<smoser>  ^-- except ec2
<smoser> :-(
<rharper> what ?
<rharper> something changed
<rharper> so maybe you fixed it to allow ec2 to retrun
<rharper> I'm switching to maybe=none
<rharper> which is also fine
<rharper> however
<rharper> I'm still seeing the generator run when I provide no user-data locally
<rharper> so something is broken;  I'm going to diff cloud-init from my image a few weeks ago vs what's in xenial-updates
<rharper> to see if there are any ds-identify changes
<smoser> i know what changed for you.
<smoser> hm... or maybe i dont
<smoser> but it is packaging that is only available in the ubuntu/xenial branch that changes the default behavior of strict_id on ec2
<smoser> of course the generator runs
<blackboxsw> smoser, do we need to worry about support of unit tests in dev environment for cloud-init on a fresh xenial system?
<rharper> http://paste.ubuntu.com/24448558/
<blackboxsw> or rharper
<rharper> smoser: that's the change to ds-identify from 3/31 to friday
<rharper> (what's in xenial-updates)
<smoser> blackboxsw, unit tests as in running 'make' ? or as in running 'tox'
<blackboxsw> so s/search/report from ds-identify updates. /me is lurking/learning about ds-identify behavior
<smoser> they wont "just work", you need python dependencies. you have to get python dependencies in some way
<smoser> either
<smoser> a.) install the runtime (and build-time) dependencies listed in the ubuntu/ branches debian/control
<smoser>   (or the trunk packages/debian/control.in)
<smoser> b.) use tox to install the dependencies with virtual-env or pip or whatever.
<blackboxsw> smoser, both fall over on my xenial system missing various dependencies out of the box.  tox runs on fresh yakkety no prob, but make test needs some updated system pkgs as you said
<smoser> tox does not run on xenial "fresh" ?
<smoser> how do you define "fresh".
<blackboxsw>  a new xenial lxc without additional dev packages installed.
<blackboxsw>  a new xenial lxc without additional dev packages installed == "fresh"
<blackboxsw> :)
<smoser> rharper, where did you get your package from in the 'from' portion of that
<rharper> probably master before the release, let me find it
<smoser> blackboxsw, good. so i'm surprised that yakkety would work.
<rharper> I would have done a package build from master and pushed to ppa
<smoser> right.
<smoser> which meant you didnt pick up xenial packaging modifications
<smoser> the daily ppa will have xenial packaging modifictions
 * blackboxsw isn't blocked. I'll  just poke at a couple things to firm up this assessment and make sure I didn't try tox on an lxc I had already installed ome deps on.
<smoser> as it builds from trunk + ubuntu/xenial branch
<rharper> smoser: possible not
<rharper> https://launchpad.net/~raharper/+archive/ubuntu/snapbuilds/+files/cloud-init_0.7.9-87-gd23543e.orig.tar.gz
<smoser> rharper, i suspect that is what it is. that you built with ./tools/bddeb there, right ?
<rharper> y
<smoser> yeah, that is trunk's behavior. which is what you wanted.
<rharper> I really hate tripping over this stuff though
<smoser> xenial changes that behavior
<smoser> for stable release reasons
<rharper> surely I can configure it from ds-identify.cfg
<smoser> no. because its not a ds-identify setting. its a cloud-init setting. that ds-identify respects.
<rharper> =(
<rharper> I don't want to have to hotpatch cloud-init in core but it looks like I have to
<smoser> yeah, which sucks for you as you have no input into the cloud-inti settings.
<smoser> rharper, https://code.launchpad.net/~cloud-init-dev/+archive/ubuntu/daily
<smoser> that *does* have trunk + ubuntu packaging for each release.
<rharper> sure, I know of it
<smoser> so thats the recomended way of grabbing it. i knwo that doesnt really help.
<rharper> I can't always rely on that since we've been testing before fixes/features have landed
<rharper> for example the network-v2 stuff
<rharper> I needed to stack the solution together;  wait till it landed in truck and then use it
<rharper> turns out the "patch when built on xenial" bit me (again)
<smoser> yeah, then you need to do something else (or you could set up a recipe build from your branch). or do a recipe build itself.
<rharper> I don't want to have a recipel I just want zesty cloud-init behavior in xenial
<rharper> so, whatever we need to do to get it, I need to learn how ASAP
<rharper> if it's hotpatching cloud-init, I'll do that in the core recipe
<smoser> well, you can't do that on xenial right now without a config file change or a change to ds-identify
<rharper> but ideally we control this with cfg files that core can lay down
<rharper> I'm ok with the config file change
<smoser> ?
<rharper> we write out a ds-identify.cfg file in core recipe
<smoser> not ds-identify config
<smoser> clodu-init config.
<rharper> sure
<rharper> I'll take that too
<smoser> then as i said above
<rharper> the Datasource:
<rharper> Ec2:
<smoser> yes
<rharper> hrm; so I'm still seeing the generator *start* cloud-init
<rharper> that's not Datasource related (ec2)
<rharper> so maybe I need that ds-identify change too (s,report,search) in addition to the ec2 datasource change too
<blackboxsw> smoser, you mentioned you were working on ds-identify unit tests. Where will that unit test live? Under lp:cloud-init/tests subdir or down in ./tools? I'm asking as I'm looking to add some simple unit tests for ./tools/read-dependencies
<blackboxsw> and I want to follow the same convention for other ./tools/ modules
<smoser> blackboxsw, probalby tests/unittests/ let me see what i have
<blackboxsw> although, maybe read-dependencies doesn't need unit test coverage as it's just a facility to setup your environment that's not used in cloud-init proper
<smoser> i think i agree with you.
<rharper> smoser: I think my policy is getting ignored;  http://paste.ubuntu.com/24448820/
<rharper> also OVF is always returning DS_MAYBE (it's local) ; but I suspect that's new
<smoser> http://paste.ubuntu.com/24448826/
<rharper> so then we always think we might be OVF if you have a cdrom in the instance at all (which seems broken) but not harmful on xenial, but broken on zesty (AFAICT)
<smoser> that is not new though
<smoser>     # FIXME: currently just return maybe if there is a cdrom
<smoser>     # ovf iso9660 transport does not specify an fs label.
<smoser>     # better would be to check if
<smoser>     return ${DS_MAYBE}
<rharper> we're setting cloud-init to run if we have a maybe
<rharper> I have in the policy file maybe=none
<smoser> blackboxsw, that was agree that i owuldnt bother writing unit tests for read-dependencies.
<rharper> and we still run cloud-init
<smoser> hm.
<smoser> can i see ?
<rharper> see the paste, right ?
<blackboxsw> +1 dropping that test idea for tools/read-dependencies idea smoser
<rharper> smoser: the most recent change was the ds-identify change to config drive path
<rharper>  git show 169a7105
<rharper> smoser: so, need policy: in the start of the file to get thing set;
<rharper> smoser: I'll work up a patch to log that ds-identify looked at a ds-identify file but didn't file a policy;  it would have helped poke me in the eye that I didn't format the policy config properly ;  that sound OK ?
<smoser> rharper, i'lm looking at this. you're right its busted.
<smoser> rharper, i was wrong.
<smoser> its your input that is busted. probably bcause there is no doc.
<rharper> smoser: ack
<rharper> hence the patch to log that it read a policy file but found none
<rharper> that'd be a nice fat warning in the log file that you didn't format it properly
<rharper> and I'll likely include a line like 'If you'd like strict behavior as in zesty, use the following line: XXX'
<smoser> but you actually dont need that.
<smoser> right ?
<smoser> you dont need to set that there.
<rharper> sure;  but since I'm attempting to maintain "run xenial more strictly than default, use this config"
<rharper> I wanted it documented (and ideally unittested)
<rharper> so when I screw it up again (or change how I build the core snap, or whatever) we have a very clear doc on what it *should* look like
<smoser> better doc of the config file format is fine.
<smoser> unit tests are fine
<rharper> logging that it found a policy file (ds-identify.cfg) but found no settings (missing ^policy: )
<rharper> fine ?
<smoser> i'm not interested so much in documenting
<smoser>  ci.datasource.ec2.strict_id: true
<smoser> if you want to warn that no policy was set, thats fine. but you can't warn in the following cases:
<smoser> a.) empty ds-identify.cfg (thats perfectly fine)
<smoser> b.) contains only : ci.datasource.ec2.strict_id: true
<smoser> c.) contains only comments '# this is wehre i would put a setting'
<rharper> I didn't want a warning
<rharper> I just want to log it in ds-identify.log
<rharper> ie, nothing printed outwardly
<smoser> warning/logging, thats fine. but still same behavior.
<rharper> but ack to a-c
<rharper> I think (a) seems odd;  b) is something we mention to do  which is fine  3) ack to ignoring comments
<rharper> what's OK about a) ?
<smoser> thats pretty standard behavior.
<rharper> to touch a file; yes; but what do we do differently with an empty policy file ?
<rharper> the defaults?
<smoser> yeah, that woudl take defaults.
<rharper> seems like one would get that without the file though;
<smoser> and thats perfectly fine.
<smoser> and its fine to have an empty file.
<rharper> well, depending on the core
<rharper> code
<rharper> in some cases the mere presence changes behavior
<rharper> 'etc/cloud/cloud-init.disabled '
<blackboxsw> strange, so we "want" to avoid duplication of dependencies outside of the control file in cloud-init  by using a makefile target or dependency install script, but we basically have that duplication already in tools/bddeb which defines a package name lookup for any deb package dependency in STD_NAMED_PACKAGES / NONSTD_NAMED_PACKAGES
<smoser> well, we declare python (pypi named) pythonn dependencies once
<smoser> we have to translate those to ubuntu package names . thats unavoidable.
<blackboxsw> it seems it's only unavodable because our control file has parameterized test_requires
<smoser> and the fact that names are not consistently mapped from pypi -> ubuntu is simply fact
<blackboxsw> right
<blackboxsw> if it were hardcoded in debian/control we wouldn't have to lookup/translate though
<blackboxsw> just write it in debian/control instead of a pkg name lookup in tools/bdeb.
<blackboxsw> I guess the lookup us something for pip names which conform to python-<packagename>
<smoser> well,, i guess we could just drop the STD_NAMED_PACKAGES, adn then use the standard naming confvention if not presnt in NONSTD_NAMED_PACKAGES
<smoser> but if we write it in debian/control, then new adds need 2 touchees.
<blackboxsw> smoser, it kinda feels like that STD_NAMED_PACKAGES is superfluous. Do we run integration/install tests with the built deb?
<smoser> yes.
<smoser> and yeah, i sgree with std_named_packages being superflous.
<blackboxsw> ok, yeah so we could drop STD_NAMED_PACKAGES I guess as our install test should cover that aspect
<blackboxsw> +1  will work on that
<blackboxsw> as part of this small PR
<blackboxsw> (pull request0
<smoser> i guess the only value it has right now...
<smoser> is that you get a nicer error message
<smoser> in bddeb says
<smoser> Do not know how to translate pypi "
<smoser>                                     "dependency %r to a known package"
<smoser> where if you didnt have that , you wouldjust get a fail later . but maybe that is obvious enough
<smoser>  https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/323059
<smoser> rharper, ^
<blackboxsw> Just posted a near trivial up for "make deb".  Some cleanups in packages/bddeb.
#cloud-init 2017-04-25
<smoser> rharper, so you're good right now with core, right ?
<fekle> hi!
<fekle> anyone here who can help me with a little cloudinit issue? :)
<fekle> I want to setup a server with Chef, and added the respective configs to my cloud-init file.
<fekle> I tested it on ubuntu and centos (in openstack), however it fails on both oss
<fekle> with the following error:
<fekle> Running module chef (<module 'cloudinit.config.cc_chef' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_chef.py'>) failed
<fekle> I even ran cloud-init in debug mode, but I always only get this one line of log
<smoser> fekle, can you paste /var/log/cloud-init.log ?
<smoser> ubuntu preferably
<fekle> https://pastebin.com/GDRAEygu
<fekle> this is my clou dinit
<smoser> fekle, fyi, 'pastebinit' command is available on ubuntu.  'pastebinit file'
<fekle> oh thanks :) also available on my arch i see
<smoser> fekle, if you can get me /var/log/cloud-init.log  and /var/log/cloud-init-output.log it'd help.
<smoser> i realize you're probably worried about sensitive info.
<fekle> im currently running again, gonna see what i find in the logs
<smoser> look for WARN, and there will probably be a stack trace
<fekle> well that's the thing, there is not stacktrace, neither in cloud-init.log nor in cloud-init-output.log
<fekle> https://pastebin.com/EUqSqyYH
<smoser> thats all that is in /var/log/cloud-init.log ?
<fekle> that's all I get below the ip/route/interface table thing in the log
<fekle> before net device info and route info i only get
<fekle> Cloud-init v. 0.7.5 running 'init-local' at Tue, 25 Apr 2017 13:42:23 +0000. Up 3.23 seconds. Cloud-init v. 0.7.5 running 'init' at Tue, 25 Apr 2017 13:42:25 +0000. Up 6.07 seconds.
<smoser> this is 14.04 ?
<fekle> nope, centos 7.3
<fekle> on ubuntu 16.04 we get the exact same error
<fekle> haven't tried 14.04
<fekle> no stacktrace on 16.04 either ://
<fekle> it just fails
<fekle> without doing anything
<fekle> even better,
<fekle> If i go in the vm, delete /var/lib/cloud, and re-run cloud-init -d init && cloud-init -d moudles
<fekle> *modules
<fekle> it fails again, no stacktrace
<fekle> but this time it always creates the correct chef config in /etc/chef
<smoser> well, lets focus on 16.04 for now
<smoser> you're using the ubuntu images ?
<fekle> yep
<fekle> official image
<smoser> ok. launch one of those. lets focus on that.
<fekle> qcow2 from here https://cloud-images.ubuntu.com/xenial/current/
<smoser> then, if you can let me in it would be easier. i  understand your hesitancy. i really supect there is *something* in a 16.04 /var/log/cloud-init.log or /var/log/cloud-init-output.log
<fekle> well the VMs are in a private network ;) but I can send you the full contents of both log files
<smoser> sure.
<fekle> wow, you're right
<fekle> on ubuntu I get waaaay more logs in /var/log than on centos
<fekle> didn't show a difference in openstack console tho
<fekle> https://pastebin.com/H8JT2pwC
<smoser>  initial_attributes:
<smoser> that needs to be a dictionary or not present
<fekle> yeah, they're empty, but I had the same erorr with some set
<smoser> well, this is the error you're seeing now
<fekle> I'll try again
<smoser> it should be fine to just remove it
<fekle> weird, normally an empty dictionary in yaml should be fine, not?
<smoser> $ python -c 'import yaml; print(yaml.load("foo:\n"))'
<smoser> {'foo': None}
<fekle> here's the full log https://pastebin.com/uaLjiRMC
<smoser> ie, you gave it a None type, not a dictionary
<smoser> its expecting a dictionary
<fekle> and seing a None Type
<fekle> got it
<smoser> (i'm not suggesting that the fail path there should not be more informative, but thats what happened)
<smoser> once it statck traced it wasn't going to go further, so launch another with cleaned up entry there.
<fekle> yeah, it's an understandable error, but it could be handled better, ie print a message warning that it's empty, or just ignoring the nulltype
<fekle> I'm spawning a new VM with attributes now
<rharper> smoser: yes
<fekle> now i get a new error: https://pastebin.com/ZQqird0D
<fekle> but nice that cloud-init shows stacktraces on ubuntu, the missing stacktraces on centos really make fixing the issue very hard
<fekle> @smoser I tried again, but AttributeError: 'UrlResponse' object has no attribute 'encode' is still the error - it's now creating the chef config correctly on the first run tho
<smoser> fekle, https://git.launchpad.net/cloud-init/commit/?id=482b2746b59
<smoser> thats the bug. it should be fixed in a 16.04 image newer than 4-20.
<smoser> sorry i ditn just recognize this right away
<fekle> aaah, awesome, thank you :)
<fekle> no problem
<smoser> can you verify what cloud-init you have in the image ?
<smoser> dpkg-query --show cloud-init
<smoser> if it is 0.7.9-90-g61eb03fe-0ubuntu1~16.04.1
<smoser> then it is not fixed correctly
<fekle> cloud-init	0.7.9-48-g1c795b9-0ubuntu1~16.04.1
<smoser> yeah. so if you get taht current image http://cloud-images.ubuntu.com/daily/server/xenial/current/ (which is actually http://cloud-images.ubuntu.com/daily/server/xenial/20170424/)
<smoser> then it shoudl be good
<smoser> ie, 0424 has that fix
<fekle> however, we are using ubuntu only for this testing here, everything else is CentOS
<fekle> ok, I'll download the newwest one
<smoser> well, *thats* your problem :)
<fekle> yeah :D
<smoser> kidding.
<smoser> but yeah, you'll need to somehow get that fix into your centos then
<fekle> it is .... often enough, believe me
<smoser> i expect a cloud-init release of 0.7.10 very shortly
<fekle> yeah, I'll try downloading the newest version of the centos image
<smoser> so , that should be released and larsks does a really good job of picking stuff up.
<fekle> awesome
<fekle> so, just tried with the new image
<fekle> now i get another crash when instaling chef
<fekle> but this time it seems to be a chef thing, not an cloud-init issue
<fekle> https://pastebin.com/7Kd6gHCC
<fekle> on Centos it fails too, but no stacktrace, although i think it's the same error
<smoser> hm..
<smoser> fekle, so it downloads the 'omnibus_url'
<smoser> and puts that in a file with execute permission and then tries to execute it
<smoser> the default url is
<smoser>  https://www.getchef.com/chef/install.sh
<smoser> you had '<censored>' that you were using something else.
<smoser> so this issue should be recreatable with just:
<smoser>  wget <thaturl> -O out
<smoser>  chmod 755 out
<smoser>  ./out
<blackboxsw> smoser, powersj do we have and jenkins set up for building centos/rhel or suse packages?
<powersj> no, but something I wanted to go through with you today.
<powersj> I've been using an old smoser script to launch a lxd
<powersj> for centos7 and wanted to see best way to get that in jenkins to run unit tests and build rpms
<powersj> from there we can decide what we do with them or if we just test that things work
<blackboxsw> powersj, yeah I feel like it'd be nice to make sure packaging/dependency changes don't break things. I set up an ec2 instance last night to to poke around at things.
<fekle> yeah, we are using our own script
<powersj> lxd ftw
<fekle> for version pinning
<fekle> colleague forgot to add a shebang, so it didn't run if you don't directly pipe it into bash :p
<blackboxsw> heh
<smoser> fekle, right. its more functional to just let the kernel decide
<smoser> that way, if you put a binary there, it just works
<smoser> or a python script
<smoser> i dont expect it to be shell
<smoser> blackboxsw, the scripts he speaks of are https://gist.github.com/smoser/19e65095b342e98fd4d6466e4d4fa1ac
<blackboxsw> smoser, ahh I missed that in the url collection effort of 2017 ;) thanks for the pointer
<smoser> i think we should look into getting a yum / rpm repo of trunk somehow
<blackboxsw> smoser, I completely agree w/ that
<blackboxsw> simpler for ci on the final product
<smoser> blackboxsw, if i just give you my firefox awesomebar history, you'llk have just about the collected bunch of knowledge that i have
<smoser> the rest lives in irc logs
<rharper> logs4life
<rharper> % du -sh ~/.xchat2/xchatlogs/
<rharper> 784M	/home/rharper/.xchat2/xchatlogs/
<rharper> that's getting beefy
<fekle> I had openstack keystone logs with 20+ million lines of logs, due to forgetting to turn of debug messages :p
<fekle> loading those logs with lnav is fun
<smoser> $ du -hs ~/data/bip/
<smoser> 2.7G	/home/smoser/data/bip/
<rharper> smoser: nice!
<rharper> smoser: is you're gzipped ?
<smoser> no.
<smoser> the logs on my local system are a measley 490M or something
<smoser> and its those that i most often read, i only go to the bip logs if i need to
#cloud-init 2017-04-26
<fekle> hi! does anyone know a way to install a newer version than cloud-init 0.7.5 on CentOS 7 ? any repos/rpms?
<fekle> this old version in centos7 unfortunately doesn't work with openstack and chef, as discussed here yesterday.
<fekle> i tried installing the fedora 25 package, however it doesn't install due to python abi differences (centos is 3.4, cloud init wants 3.5)
<powersj> https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/323246
<powersj> Enables artful
<rharper> powersj: I'm trying to build the centos7 rpm in lxd container with smoser's gist, and there looks to be some unittest failures for non-ubuntu; known issue? http://paste.ubuntu.com/24461774/
<powersj> rharper: I saw those as well
<powersj> haven't put in a fix yet or looked into what to do
<rharper> ok, I can push a MR with fixes
<rharper> it's just not mocking out distro properly
<smoser> that isnt specific to centos
<rharper> non debian will fail
<rharper> if that's what you mean
<smoser> it should not be looking at that path, ever. (/etc/cloud/)
<smoser> how would that pass on my desktop ?
<rharper> I dunno, it works today, right ? nosetests ?
<rharper> pretty strange
<rharper> I don't have those files but trunk passes locally
<smoser> yeah, you're right. its not mocking distro properly i think
<smoser> those tests *have* passed before in centos and even python 2.6
<smoser> see e8730078df8c99696b1b684e09c803eef7c4926c
<rharper> hrm
<rharper> smoser: I think it's fallout from us not running apt unless there is config
<rharper> yeah
<rharper> we pass an empty config 'apt: {}'  which then prevents the default apt templating from firing
<rharper> if you pass a non-empty apt config, then it passes;  what I can't understand yet is why it passes on ubuntu but fails under centos
<rharper> e80dbb80987ba44be2899e34fbbbf7d48389b6b5
<rharper> so, on ubuntu we have the apt command
<rharper> but on centos we dont
<smoser> right
<rharper> shall I send a PR or do you want to fix ?
<smoser> if you have it, please do
<smoser> realisitcly we shoudl be mockking the which calls
<smoser> rharper, http://paste.ubuntu.com/24461986/ <-- is another piece of it
<rharper> smoser: only a simple workaround (adding non-zero config to the dict passed to the apt_configure module)
<smoser> actually, iguess ew dont need the importlib portions . those fell out from trying to run tox on the tests/vmtests/
<rharper> vmtest?
<smoser> er... integration tests
<smoser> the default 'tox -e centos6' ran on tests/
<rharper> gotcha
<smoser> which failed, and i saw it needed importlib
<smoser> but didnt' realize that was from the tests/cloudtests
<smoser> rharper, i'll look
<smoser> i think i have areasonable way
<rharper> smoser: ok
 * rharper built a centos7 rpm anyhow to test passthrough network
<blackboxsw> ok smoser powersj rharper i have an cut at github travis integration for unit tests ++ a project README https://github.com/blackboxsw/cloud-init/pull/1
<blackboxsw> had to sort out hithub's default clone depth of 50 not being enough to "see" our latest official tag so tools/read-version wasn't working in the travis env.
<blackboxsw> you'll notice the above PR has travis PR checks voting on the merge request. I am still working on jenkins integration which would also give us integration test votes on the project.
<powersj> blackboxsw: this is awesome
<smoser> rharper, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/323265
<rharper> smoser: reading
<rharper> smoser: http://paste.ubuntu.com/24462666/  ;  where would we get python-oauthlib for centos 7 ?
<smoser> its there on cent6
<rharper> I'm on 7
<rharper> maybe bad mirror ?
<rharper> I see it in my cent7 lxd container
<smoser> rharper, i dont love that set of tests as it is
<smoser> the and i dont love one more level of indentation for every patch
<smoser> actually, the docroartor on that fuction would have owrked i think.
<rharper> I don't either
<smoser> oh. but patch.object doesnt decorate
<rharper> you don't need to patch object
<smoser> or does it
<smoser> your'e right, but thats what it does now
<rharper> you can even just patch it outright before calling the function via an add_patch()
<rharper> or decorator
<smoser> add_patch () ?
<rharper> one of the base unittest classes added an add_patch method
<rharper> which does an auto start and stop on remove
<rharper> hrm
<rharper> guess not, we have that one in curtin
<rharper> it's handy
<rharper> http://paste.ubuntu.com/24462767/
<rharper> looks like the maas centos image doesn't have epel-release package installed by default
<rharper> so that's why it didn't find python-oauthlib
<rharper> rather annoying that the lxd image and the centos image in maas aren't the same despite being the same release (7.3.116  or something)
<rharper> smoser: harlowja: just updated https://bugs.launchpad.net/cloud-init/+bug/1603533/ ;  I built cent7 rpm , installed it but it complains at runtime;
<ubot5> Ubuntu bug 1603533 in cloud-init "Can't build correctly using brpm on cent7" [Medium,Fix released]
<rharper> thanks ubot5
<harlowja> rharper kk, need to build that more often to expose those things
#cloud-init 2017-04-27
<niluje> hello, I'm trying to write a connector for Scaleway (https://www.scaleway.com/)
<niluje> By running `cloud-init --file ~/cloudinit.yaml init` with "datasource_list: [Scaleway]" in ~/cloudinit.yaml, I have an exception I don't understand:
<niluje> https://pastebin.com/98PwbVBR
<niluje> any chance someone can help me understand what is going on?
<smoser> rharper,  you do not need python-argparse on centos7
<smoser> python-argparse is really just a 2.6 thing. as its builtin to 2.7+
<smoser> $ python3 -c 'import argparse; print(argparse)'
<smoser> <module 'argparse' from '/usr/lib/python3.5/argparse.py'>
<smoser> $ python2 -c 'import argparse; print(argparse)'
<smoser> <module 'argparse' from '/usr/lib/python2.7/argparse.pyc'>
<smoser> i'm not saying that the rpm isn't broken, but the answer is to probably not depend on it. not sure how really to do that in rpm an dsuch.
<smoser> niluje, hm.. not sure what woudl have ahppened there.
<smoser> i suspect you must have a network_config thing that is returning invalid data
<rharper> smoser: I know we don't need it; but somehow it's in the rquires.txt file which python checks when loading
<smoser> rharper, that gets used for the rpm i guess ?
<rharper> smoser: harlowja: running with this which works but not sure that's the right thing to do: http://paste.ubuntu.com/24466885/
<rharper> smoser: it's used by python setuptools or whatever, entry point thingy that reads egginfo
<niluje> about my issue, here is the code of the connector: https://github.com/brmzkw/cloud-init-scaleway/commit/110bb3e02608cb29ec67fe180cac4a48e3bc548b  and my configuration + the errors I have: https://pastebin.com/qjBW8TGg
<rharper> https://github.com/certbot/certbot/commit/8f101034960ffc1e47879314585898efda234e60
<rharper> this is sorta the issue as well;
<smoser> niluje, can you get a cloud-init.log ?
<rharper> IIUC, special case python2.6 to add that dep (in packaging) and from > 2.6 it's standard library, so shouldn't be in requirements (or at least not show up in egginfo) ?
<niluje> smoser: where is it stored?
<niluje>  /var/lib/cloud?
<smoser> i suspect that you've gone down the path of reading command line parameters that cloud-init has... and that it went wrong.
<smoser> /var/log/cloud-init.log
<smoser> if it sees ip= on the command line, it takes *that* as the fallback networking, and then tries to read files that ubuntu's initramfs tools writes
<rharper> so, on ubuntu, we package the requires.txt file, but it's empty
<niluje> smoser: cloud-init.log is empty
<smoser> ubuntu ?
<niluje> yes, xenial
<niluje> I git-clone'd the cloud-init repository and pip install -e .
<smoser> hm..
<smoser> i suspect that there is no /etc/cloud/cloud.cfg.d/loggin* ?
<smoser> (i'd install from package and just make your updates that way)
<smoser> or build a package from trunk and install it.
<smoser>  ./tools/bddeb
<smoser> installing with pip is not something i've really ever tested.
<niluje> ewww
<niluje> ok
<niluje> let me try installing the package with apt then
<smoser> you'll still fail for sure
<smoser> but hopefully get a log :)
<niluje> how to you work on cloud-init then?
<niluje> apt-get install cloud-init, then git clone, pip install -e ., $> cloud-init init?
<niluje> I just apt-get install'ed the package, updated /etc/cloud/cloud.cfg to add "datasource_list: [Scaleway]" and call "cloud-init init" but cloud-init is trying to contact the EC2 APIs :/
<smoser> niluje, generally i install the package. taht is what a distro is going to do anyway.
<smoser> hm..
<niluje> and then?
<niluje> how do you edit the code?
<smoser> when hacking i just change files in /usr/share///
<smoser> and re-try and such
<smoser> then i take those chagnes and re-build / dpkg -i
<smoser> yeah, i know its not the best. but dpkg -i isn't actually *all* that different from pip install.
<niluje> hm
<niluje> how do you build the package?
<smoser> ./tools/bddeb
<smoser> soryr
<smoser> ./packages/bddeb
<smoser> https://bugzilla.redhat.com/show_bug.cgi?id=1194451
<ubot5> bugzilla.redhat.com bug 1194451 in cloud-init "[PATCH] Add dnf support to cloud-init" [Low,Closed: rawhide]
<smoser> niluje, so... what was happenging for you was that
<smoser> a.) you boot with 'ip=' on the kernel cmdline (which makes good sense)
<smoser> b.) cloud-init recognizes that as saying taht the initramfs configured networking and that it should read its networkign from that.
<smoser> rather than generating "fallback" config, which is "dhcp on the first nic you think looks good"
<smoser> because obviously bouncing the nic would be bad on a network moutned root device
<smoser> the problem is that your initramfs does not generate /run/net-<NAME>.cfg files that cloud-init expected to read some information from.
<smoser> and thus it ends up generating a completely empty config that fails
<niluje> ok
<niluje> but should our initramfs generate the /run/net-<NAME>.cfg file?
<smoser> well, mkinitramfs-tools does (ubuntu/debian)
<smoser> but there is no official format of that, its just kidn of happensatnce
<smoser> the other option would be to have cloud-init realize there are no net-* and do some other sort of searching to figure out what the network config is.
<niluje> ok so what you did (I haven't checked yet) is remove the read from /run/net-xxx.cfg and it works, right?
<smoser> no. those files do *not* exist for you
<smoser> and cloud-init is expecting them
<niluje> I know
<niluje> I understand that
<smoser> oh. right. yeah.
<smoser> http://paste.ubuntu.com/24467477/
<smoser> i just made it not pay any attention to the kernel cmdline
<smoser> but that probably fails eslwehere
<niluje> ok
<smoser> as cloud-int will write a /etc/network/interfaces that says it should dhcp on eth0
<niluje> ok
<smoser> which best case fails on 'ifup' (because that is already up)
<smoser> and worst case brings down the link that you have to the root device
<niluje> so what I need to do is 1/ first, create manually the file un /run, 2/ make cloud-init work 3/ in //, update our initrd to create the files in /run/net-xxx.cfg
<niluje> and everything should work
<niluje> correct?
<smoser> if your initramfs writes those files, it should work yeah.
<niluje> ok
<smoser> theres no standard way to do this.
<niluje> I don't even know what is the format of those files :p will check
<niluje> thanks a lot for your help smoser
<smoser> niluje, see cloudinit/net/__init__.py _klibc_to_config_entry
<smoser> blackboxsw, http://paste.ubuntu.com/24467573/
<smoser> thats what i'll just upload with
<blackboxsw> smoser, ok so that's our bug list for the SRU?
<smoser> yeah
<smoser> hm.. and maybe i'd rip out the  yum one
<smoser> (not reference it)
<smoser> yeah, i think i iwill
<niluje> there's no reference to _klibc_to_config_entry in cloudinit/net/__init__.py
<niluje> k in cloudinit/net/cmdline.py
<smoser> sorry
<smoser> yeah
<smoser> niluje, and then in the tests therea re examples
<niluje> smoser: this is the function reading /run/net-xx.cfg and convert it to a cloud-init config object, right?
<niluje> sorry for the stupid questions I'm not familiar with cloud-init so I struggle a bit to understand what needs to be done
<niluje> ok yes, that's it
<niluje> I will look into it tomorrow
<niluje> thanks a lot again for your help smoser, I'll post a message here when the connector will be working
<smoser>  _klibc_to_config_entry is what converts
<smoser> and the test functions show examples of the net-*
<blackboxsw> smoser, so SRU info, last time you created https://public.etherpad-mozilla.org/p/cloud-init-sru-info   will we do the same type of thing for this sru?
<smoser> blackboxsw, i think its useful, yeah.
<rharper> https://docs.fedoraproject.org/en-US/Fedora/8/html/SELinux_FAQ/index.html#id503239 ; that's troublesome;  ifconfig output is thrown away due to selinux, you can get it by reading it from the pipe (instead of the terminal) and writing to file (ifconfig -a | cat >out)
<smoser> blackboxsw, https://public.etherpad-mozilla.org/p/cloud-init-sru-info
<blackboxsw> ... clunky  rharper
<rharper> yeah, I'm more worried about cloudinit.util.subp
<rharper> not sure that's working as expected
 * rharper is testing that now
<smoser> rharper, see, i told you enforcing=off
<smoser> :)
<rharper> lol
<rharper> not the *right* answer
<rharper> but yes
<rharper> you were right
<smoser> if everyone in the world says that 2+2 = 5, then that is 'right' for some definition of right ;)
<smoser> i wonder
<smoser> rharper, if we close the standard input
<rharper> playing with subp now
<smoser> hm..
<rharper> I think it's already closed though
<smoser> well, standard output is probably indirectly attached to a temrinal
<smoser> i don tknow
<rharper> serial console
<smoser> well, /dev/console
<rharper> but yeah; this gets into the magic of tty
<rharper> and other things
<smoser> we should be ablet to just close enough file handles and such though so that , before that subp (or all with 'capture') no open file handles are a termianl
<smoser> id' think that would do it
<rharper> so, subp defauts to data=None (stdin is fd of /dev/null) and capture=True, sets up a pipe
<rharper> I just need to confirm we get output; there's a separate netstat -rn returning 1; I don't yet know why
<rharper> https://bugs.launchpad.net/ubuntu/+source/net-tools/+bug/1251563
<ubot5> Ubuntu bug 1251563 in net-tools (Ubuntu) "netstat command returns nozero even if successively executing" [High,Fix released]
<rharper> well, tha;s just strange
<rharper> bbiab
<powersj> https://paste.ubuntu.com/24468378/
<powersj> ^ unit test failures on centos 6 + 7
<powersj> I'll go play with https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/323265 shortly
<smoser> powersj, great.
<rharper> https://bugzilla.redhat.com/show_bug.cgi?id=1406520
<ubot5> bugzilla.redhat.com bug 1406520 in libselinux "calling libselinux python restorecon fails on /var/lib/nfs/rpc_pipefs" [High,Verified]
<rharper> it appears that's hitting in centos7 image
 * rharper is confirming 
<rharper> bummer, I don't see 2.5-7 released yet
<powersj> https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/323351
<powersj> All I got rid of was the use of the array for the patches and got rid of one that was useless
<powersj> oh and cleaned up pylint errors
<powersj> rharper: if you could help me better understand what else you would want changed to smoser original proposal that would be good. You mentioned moving the asserts
<rharper> powersj: yeah, lemme get a pointer
<rharper> powersj: so, each of the patch_XXX should be @mock.patch.object(util, 'write_file') on the helper function; then pass that in as a mock_xxx to the _apt_source_list method;  the decorator does the start/stop automatically for you;  then in them ethod, the mocks that have return values or side-effects, you just need to apply those based on the input,
<powersj> oh you wanted decorators
<rharper> powersj: I think all of them are constant values, except the mock_shouldcfg, in which case you just mockshouldcfg.return_value = (cfg_on_empty, "this is for test")  in the method
<rharper> first
<rharper> and then, I really want the caller who knows if they sent "debian", "ubuntu" to run the asserts;
<powersj> so end that function after running cc_apt_configure.handle
<rharper> so, test_apt_v3_source_list_debian knows what config it sent and which distro is being used
<rharper> knowing those things, it can test that it expects for should_config to return true
<rharper> the other variable is "is_system_snappy"  which I think I'd like to assume its no , and then have a single test_apt_v3_source_list_snappy; where we return True on that and then can confirm that apt_configure wasn't run
<rharper> that'll break up much of the if block of asserts based on the config_on_empty list
<rharper> the test_ method calling the helper should construct the path and test it explicitly;  if there is something that we know *always* gets called then I think the helper could assert it; but I'd rather the caller do the asserts since it's setting up the test scenario
<rharper> if that makes sense
<powersj> rharper: ok I think I follow all that. I'll give it a shot and come back.
<powersj> it does
<rharper> cool
<rharper> would you like me to paste this log into the MR  comment ?
<powersj> sure :)
<rharper> k
<rharper> smoser: http://paste.ubuntu.com/24468995/  ; looks like we get to have "fun" with centos7 network/cloud-init.local ordering  in systemd
<rharper> or I need to figure out how in brpm to tell it we require systemd;  noticed that difference in the 0.7.5 vs. a local build
 * rharper learns of -b systemd in brpm 
<rharper> now for more fun
<rharper> ha, systemd units worked!
#cloud-init 2017-04-28
<niluje> smoser: I reverted all the modifications you did yesterday, and created manually the file /run/net-eth0.conf as it seems to be expected according to your unittests and various searches on google (resulting to https://pastebin.com/uwu7e1aJ)
<niluje> however, running cloud-init init raises because:
<niluje>   File "/root/cloud-init-scaleway/cloudinit/net/network_state.py", line 296, in handle_physical
<niluje>     if ':' in subnet['address']:
<niluje> KeyError: 'address'
<niluje> shoudln't we add 'ADDRESS' here? https://github.com/brmzkw/cloud-init-scaleway/blob/wip/cloudinit/net/cmdline.py#L104
<smoser> niluje, it would appear so.
<smoser> and the unit test is just busted.
<smoser> you can fix the STATIC_EXPECTED_1
<smoser> as it does not have an 'address' which it clearly should
<smoser> (this path we've not tested much as everyhthing maas used was dhcp)
<smoser> you are doing dhcp though, right?
<niluje> yes
<smoser> yeah, so that is another path (although we should fix this)
<niluje> but I configured the interface as static for testing purpose
<smoser> and actually, yeah, its busted in dhcp also
<smoser> well, in dhcp, there isn't really a sense in adding the static address
<niluje> yes
<niluje> but since nothing is done if the value doesn't exist in the net-xx.conf file, I still can add "ADDRESS" in the list and fix the unittests accordingly. Correct?
<niluje> 'ADDR'* not 'ADDRESS'
<smoser> yeah, please fix the unit tests.
<smoser> niluje, and add ADDRESS as you suggested. you diagnosed this correctly. thanks.
<blackboxsw> next sru in progress https://public.etherpad-mozilla.org/p/cloud-init-sru-info-20170427
<blackboxsw> ok, I'm heading down the list to associate cloud-init sru bugs w/ the proper series
<blackboxsw> smoser, why is this bug in the list again https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1676460   just because we need it in zesty now too?
<ubot5`> Ubuntu bug 1676460 in cloud-init (Ubuntu Yakkety) "dpkg-reconfigure does not list Bigstep" [Medium,Fix released]
<dpb1> blackboxsw, smoser: #ubuntu-devel says its fine to leave them as just nominated
<blackboxsw> sounds good thx dpb1
<dpb1> when the sru is requested, they will do the accepting.
<smoser> blackboxsw, hm..
<smoser> that shoudlnt be there.
<smoser> removing
<blackboxsw> ok can you kill my nominations too smoser
<smoser> (that was tehre just because i left it there to make the new sections look like it)
<smoser> as an example
<smoser> then forgot to remove it
<blackboxsw> ahh makes sense
<blackboxsw> smoser, dpb1 how do I switch my lp view to allow nominating for series on the distro project instead of the main project? https://bugs.launchpad.net/cloud-init/+bug/1684869
<ubot5`> Ubuntu bug 1684869 in cloud-init "growing root partition does not always work with root=PARTUUID=" [Medium,Confirmed]
<blackboxsw> right now on th above bug I have a highlighted cloud-init project so I can't select  nominate for series on the distro project
<blackboxsw> the selections I get when nominating for series are trunk/2.0 instead of xenial/yak/zesty..etc
<nacc> blackboxsw: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1684869
<ubot5`> Ubuntu bug 1684869 in cloud-init "growing root partition does not always work with root=PARTUUID=" [Medium,Confirmed]
<nacc> blackboxsw: is that what you mean?
<smoser> blackboxsw, edit the url :)
<smoser> not kidding.
<blackboxsw> yeah nacc that is highlighting the appropriate project for me
<blackboxsw> ahhh
<nacc> blackboxsw: yeah, so i hovered over the cloud-init (ubuntu) task
<smoser> thats why my template has 2 urls
<nacc> and that showed the other task's direct url base
<blackboxsw> heh I hadn't noticed the url path difference
<blackboxsw> ok
<blackboxsw> thanks
<nacc> blackboxsw: yeah, only ubuntu/ urls can have ubuntu release tasks
<blackboxsw>  smoser, do I also add needs-verification tag now  or do we wait until the SRU template is written
<nacc> blackboxsw: the sru team adds those tags once they upload
<nacc> *approve
<blackboxsw> ahh thanks nacc
<blackboxsw> smoser,  how's this sru template for #1682160   http://paste.ubuntu.com/24474635/?
<blackboxsw> it took me a while to just validate the breakage/script
 * blackboxsw has to head out for a late lunch back within an hour
<smoser> blackboxsw, install would p[robably trigger it to
<smoser> but sure. that looks sane.
<smoser> if you just launch an image on an openstack (or otherwise utilze the cloud image) then grub-legacy-ec2 will be there already.
<blackboxsw> back
#cloud-init 2018-04-23
<blackboxsw> smoser: pushed my functional code to https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+ref/feature/os-local  I had to back out some wip stuff. tested on serverstack. I'm thinking of moving the fallback_nic property up to base Datasource and adding some additional error checking on metadata type
* blackboxsw changed the topic of #cloud-init to: Reviews: http://bit.ly/ci-reviews | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting: Monday 4/23 16:00 UTC | cloud-init 18.2 released (03/28/2018)
<blackboxsw> aaaahhhhh :)
<blackboxsw> analyze.local:Total Time: 7.48900 seconds
<blackboxsw> analyze.net:Total Time: 17.32100 seconds
<blackboxsw> 10 second saved  on openstack boots :)
<rharper> \o/
<mgerdts> it seems as though the most important of my fixes landed a day too late to be included in the latest xenial and artful packages, at least according to https://launchpad.net/ubuntu/+source/cloud-init.  When is the next chance for them to be picked up?
<blackboxsw> mgerdts: sorry that was a 2 week process for the SRU xenial/artful. we'll probably be handling another sru within a month as we'll be publishing more content into Bionic too I imagine
<blackboxsw> and we'll have to SRU into bionic too at that point
<mgerdts> ok.  Will artful get another update?  Or will it be sufficiently aged by then that it's no longer supported?
<smoser> mgerdts: https://wiki.ubuntu.com/Releases
<smoser> artful will go EOL in July.
<blackboxsw> smoser: rharper can you confirm. I thought there would be a window for a while where xenial, artful and bionic are all SRU'd to.
<blackboxsw> thanks smoser
<blackboxsw> yep
<smoser> per /usr/share/distro-info/ubuntu.csv 2018-07-19
<dpb1> once bionic is out, artful is not very interesting.
<dpb1> (that's our general experience)
<smoser> dpb1: but stable release update process requires you to do the srus there also
<dpb1> yes, I understand
<smoser> blackboxsw: http://hangouts.google.com/hangouts/_/canonical.com/cloud-init have a minute ?
<rharper> % distro-info -y --days=eol --all -r  | grep -v -- -
<rharper> 14.04 LTS 359
<rharper> 16.04 LTS 1094
<rharper> 17.10 87
<rharper> 18.04 LTS 1829
<rharper>  
<rharper> fancy
<mgerdts> Oh, once I finally mapped out the release names in my head you show me something that has release numbers.  :)
<rharper> hehe
<rharper> % distro-info -y --days=eol --all -f  | grep -v -- -
<rharper> Ubuntu 14.04 LTS "Trusty Tahr" 359
<rharper> Ubuntu 16.04 LTS "Xenial Xerus" 1094
<rharper> Ubuntu 17.10 "Artful Aardvark" 87
<rharper> Ubuntu 18.04 LTS "Bionic Beaver" 1829
<mgerdts> :)
 * rharper adds a new tool to the chest 
<smoser> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1766302
<ubot5`> Ubuntu bug 1766302 in cloud-init (Ubuntu) "netdev_pformat does not show unconfigured devices" [Undecided,New]
<mgerdts> Is this one expected to be in the GA build of bionic? https://launchpad.net/ubuntu/+source/cloud-init/18.2-14-g6d48d265-0ubuntu1
<smoser> http://paste.ubuntu.com/p/BmkH37g9qf/
<blackboxsw> mgerdts: yep. that's in :   rmadison cloud-init tells you the released versions of a pkg
<blackboxsw> http://paste.ubuntu.com/p/35zrnQPZCY/
<mgerdts> thanks
<blackboxsw> no problem, it's in devscripts deb if you a frequent need for rmadison find you need
<blackboxsw> sorry, typo-central.
<blackboxsw> n
<dpb1> o.O
<jocha> Hi @bloackboxsw, I'm Josh a new dev from Azure. I was wondering if you could take a look at my merge request : https://code.launchpad.net/~jocha/cloud-init/+git/cloud-init/+merge/343563 :)
<blackboxsw> welcome jocha. sure can look it over today
<blackboxsw> was in a bit of an Ubuntu-release mode last week, will get through some reviews
<jocha> awesome! Thanks!
<philroche> Is `cloud-init collect-logs` expected to work when run inside a chroot? I get the following errors https://pastebin.ubuntu.com/p/wRvgGZ6HnR/
<blackboxsw> interesting philroche, hadn't tried that use-case. I presume it doesn't given that your chroot's rundir would be different.
<blackboxsw> I think it's worth a bug...
<blackboxsw> here's what it tries to grab for you
<philroche> blackboxsw: ack
<blackboxsw> http://cloudinit.readthedocs.io/en/latest/topics/capabilities.html#cloud-init-collect-logs
<philroche> blackboxsw: That's the exact link I was looking for :)
<blackboxsw> I'll take off my mind-reading glasses now to save them for a better time
<philroche> :)
<philroche> blackboxsw: Bug filed - https://bugs.launchpad.net/cloud-init/+bug/1766335
<ubot5`> Ubuntu bug 1766335 in cloud-init "Running cloud-init collect-logs inside a chroot is not possible" [Undecided,New]
<blackboxsw> thank you sir
<blackboxsw> philroche: frequently I like to see /run/cloud-init/instance-data.json as it tells us what the datasource gives us as far as metadata and userdata etc. but judging from your chroot, that no longer exists.
<philroche> blackboxsw: Yes, that no longer exists
<smoser> blackboxsw: well thats on /run which is memory only.
<blackboxsw> bummer, ok. I'll work on making collect-logs more resilient in a chroot today/tomorrow. So at least we can collect some remnant of the files around
<blackboxsw> smoser right, but we also have symlinks to files that should still exist, which we could pull, like status.json -> ../../var/lib/cloud/data/status.json  and result.json. But, cloud-init.log and journalctl would give us the content of those files at least.
<mgerdts> Is there any special way that I should poke redhat or others to get my fixes into rhel 7 and/or centos 7?
<blackboxsw> mgerdts: not sure, we might have representatives in next week's #cloud-init status meeting on Monday per topic in this IRC channel. larsks (IRC nick) might have info on where to start if he joins in .
<mgerdts> ok, thanks.  What time is that meeting at?
<blackboxsw>  Monday 4/23 16:00 UTC
<mgerdts> thanks
<blackboxsw> oops
<blackboxsw> 4/30 :0
<mgerdts> yeah. :)
<blackboxsw> I mis-set the date this morning :/
<blackboxsw> #topic #cloud-init  Reviews: http://bit.ly/ci-reviews | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting: Monday 4/30 16:00 UTC | cloud-init 18.2 released (03/28/2018)
<blackboxsw> ok fixed
<blackboxsw> rharper: looks like we call it fallback_nic twice : http://paste.ubuntu.com/p/zMkBd8hWbW
 * blackboxsw creates a bootable image from this
<rharper> interesting
#cloud-init 2018-04-24
<blackboxsw> jocha: initial review posted for tomorrow https://code.launchpad.net/~jocha/cloud-init/+git/cloud-init/+merge/343563
<blackboxsw> and I'm outta here
<smoser> blackboxsw: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/343127 could use a review if you were bored
<blackboxsw> looking
<blackboxsw> was just trying to spawn a failed instance on gce
 * blackboxsw was working through capturing an image and booting a new instance with it
<smoser> with serial console ?
<blackboxsw> as we'll need to do that inevitably if we can find a reproducer
<blackboxsw> yeah trying with serial console + root password backdoor and custom cloud-init
<smoser> cool.
<blackboxsw> I'll wrap up the gce instance from image spawn on cmdline and add a placeholder script to  qa-scripts/scripts/launch-gce. then I'll grab that review, sound good as far as priority?
<rharper> smoser: hrm, we actually have a run-time depends on iproute2 ;  for the minumal cloud-images, there's no iproute2 in that even though it has cloud-init, the fallback nic code runs ip route commands; which it doesn't find;   in ubuntu/devel branch the control file does not include iproute2;
<smoser> http://paste.ubuntu.com/p/87ZhVjWjMW/
<smoser> rharper: not a huge deal
<smoser> its part of ubuntu-minimal
<smoser> which .... should be part of ubuntu "minimal" image :)
<smoser> so yes, we could add it and possibly should add it. but its goign to be there.
<smoser> and that is the case for xenial also
<rharper> well, http://cloud-images.ubuntu.com/minimal/daily/bionic/ those are all broken w.r.t not having iproute2 in it
<blackboxsw> smoser: rharper by add it, you mean an 'or' depends on iproute2 or net-tools?
<blackboxsw> or an explicit iproute2 dep in bionic & later
<rharper> explicit, depite the net banner, fallback nic configuration uses the 'ip' command
<smoser> blackboxsw: well, at this point i'd chagne upstream packaging to just say iproute2
<smoser> but would leave x, a, b as they are
<blackboxsw> ok fair 'nuf
<smoser> rharper: why would we create something with the name 'ubuntu' and 'minimal' that did not have 'ubuntu-minimal'
<smoser> :-(
<rharper> (â¯Â°. Â°ï¼â¯ï¸µ â»ââ»)
<smoser> even stating:
<smoser>  It is also used to help ensure proper upgrades, so it is recommended that
<smoser>  it not be removed.
<dpb1> smoser: you have a few hours to listen to the story about names?
<dpb1> smoser: if you want a fun time, click https://askubuntu.com/questions/tagged/ubuntu-minimal, which is a combination of questions about the mini iso and the minimal desktop.
<blackboxsw> just FYI, I'm able to spawn gcloud instances based on an image w/ updated cloud-init ( which logs gcloud compute instances create image-test2 --zone=us-central1-b --image upgraded-cloud-init-disk --image-project cloud-init-testing --metadata-from-file user-data=sethostname.yaml --metadata=serial-port-enable=1).  I'm dropping this stuff into qa-scripts/scripts/launch-gce now
<blackboxsw> still no reproducer on my end
<dpb1> I'm both happy and sad by that news
<rharper> blackboxsw: thanks
<rharper> I'll use that
<smoser> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1766711
<ubot5`> Ubuntu bug 1766711 in cloud-init (Ubuntu) "cloud-init missing dependency on iproute2" [Undecided,New]
<smoser> bug https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1766714
<ubot5`> Ubuntu bug 1766714 in cloud-init (Ubuntu) "cloud-init missing dependency on isc-dhcp-client" [Undecided,New]
<blackboxsw> ugh, though I placed it in Package: cloud-init
<blackboxsw> Architecture: all
<blackboxsw> Depends: ${misc:Depends},
<blackboxsw>          ${${python}:Depends},
<blackboxsw>          isc-dhcp-client
<blackboxsw> in packages/debian/control.in
<blackboxsw> s/I/we/
<smoser> thats for trunk packaging . i probably blame to the foobar on the ubuntu/devel branch
<smoser> blackboxsw:
<smoser>  https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/344186
<smoser>  https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/344186
<smoser>  https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/344190
<rharper> smoser: you have a few minutes to sync on the rename stuff?
<smoser> rharper: sure.
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/344181
<blackboxsw> smoser: approved and landed  https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/344190
<blackboxsw> approved https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/344186... will land it
<mgerdts> smoser any more thoughts on https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/343712 .  Was just chatting on #smartos where one of our early adopters was surprised that a NIC that he added after the first boot did not get configured on a subsequent boot.
<blackboxsw> smoser: I'm not merging https://code.launchpad.net/~vkuznets/cloud-init/+git/cloud-init/+merge/343601 at the moment. as I didn't want unnecessary churn on the bionic branch we want to cut
<rharper> blackboxsw: you're still working on the recreate?
<rharper> on gce ?
 * smoser out
<blackboxsw> rharper: I haven't recreated, but I can continue to spin instances up using and updated cloud-init that tracks name_assign_type
<blackboxsw> rogjt mpw O
<blackboxsw> right now I'm merging in smoser's bionic fixes
<rharper> blackboxsw: ok; I think we have a more general solution here w.r.t the the renaming;
<blackboxsw> and generating a bionic upload mp which includes isc-dhcp-client dependencies
<rharper> y
<jocha> blackboxsw: I've resolved the comments you made, feel free to review whenever you have time :) https://code.launchpad.net/~jocha/cloud-init/+git/cloud-init/+merge/344192
<rharper> did you push the launch-gce script to qa-scripts ?
<blackboxsw> rharper: noper. it's a big lift
<rharper> I'd like to see if I can fire up a loop to trigger the failure
<blackboxsw> I'm re-writing a lot of launch-ec2 at the moment
<rharper> can you pastebin it to me (no without creds) ?
<rharper> I guess I can just use your gcloud command from earlier
<blackboxsw> rharper: as in, not even functional. I'll give you a pastebin of the commands I run on gcloud
<rharper> yeah, thanks
<rharper> will work on a reproducer loop there; with root login on serial console;
<blackboxsw> rharper: here's the basics https://pastebin.ubuntu.com/p/xHgHRY3vbt/
<rharper> thx
<blackboxsw> it involved me having installed google-cloud-sdk
<blackboxsw> I think there was some setup there
<rharper> yeah
<rharper> I have that installed
<blackboxsw> I see your ssh pubkeys are part of our project so that should be good
<blackboxsw> in that pastebin I forgot the addition commands to dpkg -i cloud-init*deb; cloud-init clean --logs; and sudo passwd ubuntu setup before stopping the instance and creating an image
<rharper> yeah; mostly wanted the instance image and type details and that serial port bit
<rharper> is there a gcloud command for getting the console ?
<blackboxsw> rharper: with that metadata serial-port-enable=1 stuff you can then cloud compute connect-to-serial-port bionic-test
<blackboxsw> rharper: with that metadata serial-port-enable=1 stuff you can then "gcloud compute connect-to-serial-port bionic-test"
<rharper> nice
<rharper> blackboxsw: smoser: ok; I've recreated, updated the bug with details for possible fix and how to test it; https://bugs.launchpad.net/cloud-init/+bug/1766287
<ubot5`> Ubuntu bug 1766287 in cloud-init "18.04 minimal images on GCE intermittently fail to set up networking " [High,New]
<blackboxsw> rharper: nice, so maybe it was region related then ? I see you switched to europe-west1
<rharper> yes
<rharper> and back to the 420 image
<blackboxsw> man.
<rharper> I've not tried on the 421
<rharper> but I suspect that europe-west1 is just less crowded
<rharper> also, it's night time now
<blackboxsw> I'll give that a spin tonight if you are about EOD.
<rharper> now that I have the recreate, I'll try again in other zones
<blackboxsw> nice
<blackboxsw> k  back in an hour
<rharper> k
#cloud-init 2018-04-25
<blackboxsw> back
<rharper> blackboxsw: ok; I've got 55 consecutive reboots with no issues
<blackboxsw> ok rharper I misread your comment, I thought you did see the hang in europe
<rharper> oh, I do
<rharper> this is with my fix applied
<rharper> I wanted to make sure it wasn't a fluke;
<blackboxsw> ahh good deal. so we may not need to write network names
 * blackboxsw is trying again is us-central1
<rharper> smoser and I were chatting, and that is still proobably the right thing to do anyhow
<rharper> but we can decide when to land that;
<rharper> this also needs maas qa before we push it too
<rharper> I'm worried about all of the other places were we don;'t see this now; and we're not using the systemd-udev-settle.service; it's just not running anywhere except if you've got zfs installed or lvm enabled
 * rharper probes some vmtest runs on bionic with zfs and lvm to see what their journal says 
<blackboxsw> again rharper you saw this just w/ cloud-init clean --logs and reboots right?
<rharper> oh yeah
<blackboxsw> man, us-central1 still not reproducing it for me
<rharper> first boot in europe-west1 with the 420 bionic image works fine
<rharper> then I rebooted
<rharper> saw it
<rharper> rebooted, it came up
<blackboxsw> will try switching to europewest again
<rharper> so not always, since it's racy
<blackboxsw> s/again/for once/
<blackboxsw> of course this could be as well that my recent attempts had debug message checking and printing name_assign_type
<blackboxsw> heh, didn't realize our account instance view was shared
<blackboxsw> I see rharper-b1 now
<blackboxsw> sure enough first boot in europe-west1-b bricked
<blackboxsw> geez man region-related for sure
<rharper> yeah
<rharper> blackboxsw: that's super interesting
<rharper> w.r.t the region, I suspect it's load
<blackboxsw> could be. trying to see if I can a success boot so I can add my debug deb on  followup reboots
<rharper> ah, yeah, you have to get one good boot to set the root password
<blackboxsw> yeah or boot from my previous image snapshot. I'll try that
<smoser> so .. we can reproduce fairly easily ?
<blackboxsw> looks like europe-west1-b or europe-west1-d regions
<rharper> smoser: yeah
<smoser> rharper: if nothing was running trigger....
<smoser> er.. if nothing was runnign the trigger service , then what was doing the cold plug?
<rharper> not the trigger service
<rharper> that;s always running
<smoser> *something* has to be doing it. or we wouldn't have .link files respected at all
<rharper> it's the settle service
<smoser> so any possible 'udevadm settle' from anywhere would make it work
<rharper> yeah
<rharper> in xenial, the networking.service unit runs a pre command with udevadm settle in it
<rharper> artful could show it, if things were fast enough; and even in bionic, it has to be in this one region where things run slightly odd
<rharper> ok, I need to step a way
<smoser> blackboxsw: logs of launch-softlayer needs combining with launch-ec2 too.
<smoser> blackboxsw: this got missed.
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/342334
<smoser> not terribly important
<smoser> and then this one needs landing too
<smoser>  https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/344189
<blackboxsw> smoser: landed https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/344189
<smoser> blackboxsw: thanks
<rharper> blackboxsw: smoser:  if we wanted to be more targetted with the settle, we could for example, trigger it within cloud-init-local if we detech non-renamed interfaces with knames (and ifnames=0 no in cmdline); that would then only impact systems which happen to have that early race between cloud-init-local and udev-trigger
<smoser> right. and that would be in some ways safer
<smoser> from the perspective of not changing boot
<smoser> i'd like to have slangasek or xnox thoughts
<rharper> yes
<smoser> as i am apt to agree with you, that not having the settle service active in boot is ... well just wrong.
<rharper> there is a swarm of "why is my boot slow/ systemd-analyze blame shows udev-settle.service"
<smoser> oh?
<rharper> yes
<smoser> so we did it as an optimization :)
<rharper> but it's because they have things like usb nics or other storage devices that take *time* to come up
<rharper> no
<rharper> I don't think so
<smoser> (it was a joke)
<rharper> it's not clear to me why it's not enabled by default
<rharper> yet
<smoser> i can make a system boot REALLY REALLY FAST
<smoser> and sometimes even do what you want!
<rharper> but, lvm2 has a generator which forces it on, if lvm2 is needed in some sitations
<rharper> and zfs of course, Requires it
<rharper> since they need all of their devices up before they can mount or build a raid, etc
<rharper> so, it *really* seems like it should just always be on
<rharper> one ends up "Waiting" for rootfs anyhow
<smoser> i agree. we should request slangasek and xnox review of your MP ?
<rharper> we've seen those "waiting for device ... foo to appear"
<rharper> smoser: or possible add a systemd task and ask in the GCE bug
<smoser> rharper: well, in my fast boot, sometimes / isnt' there, so but it boots really fast.
<rharper> but I would like foundation review
<rharper> smoser: lol!
<rharper> I get (initramfs) prompt *so* fast those times
<smoser> exactly.
<smoser> and systemd-analyze does not blame udev!
<rharper> I usually take the extra savings and then compile my own kernel, kexec into it to find my root
<rharper> blackboxsw: interesting observation w.r.t zone and image;
<rharper> I wonder if we can further disect what's special about the 420 image in europe-west1 vs. current stuff
<rharper> none-the-less; it does make sense to do something to detect if we've raced and try to fix that in the case we do
<rharper> I'm going to see if I can target the settle within cloud-init-local on the reproducer
<smoser> i compile my kernels with -O4 and funroll-loops . its the best.
<smoser> "it does make sense to do something to detect"
<smoser> maybe
<smoser> it only makes so much sense to determine when a system is broken... why didn't we just fix the system ?
<rharper> that's fair; for now I'm mostly intereted in if we can detect it;
<rharper> whether we target a more narrow fix so as to not "udevadm settle" the world ; aka smoser's favorite alias to 'sleep 1'
<rharper> needs more discussion
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/344189
<smoser> bah. bad link
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/344255
<smoser> that one.
<smoser> that is softlayer doc improvement.
<smoser> rharper, blackboxsw, dpb1 ^
<rharper> yeah, saw that
<rharper> 6 ways
<smoser> from sunday
<rharper> hehe
<rharper> 2018-04-25 15:44:49,632 - __init__.py[DEBUG]: WARK: found unstable device names: ['eth0']; calling udevadm settle
<rharper> 2018-04-25 15:44:49,968 - util.py[DEBUG]: WARK: Waiting for udev events to settle took 0.336 seconds
<rharper> smoser: we can detect, and "resolve" it more narrowly
<rharper> if we want
<rharper> I'll put up an alternative patch with this change
<smoser> link ?
<smoser> philroche: https://launchpad.net/~smoser/+archive/ubuntu/ibmcloud-test
<smoser> should  be populated shortly with a test.
<blackboxsw> smoser: reviewed https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/344255
<rharper> smoser: blackboxsw: this is an alternative, more targetted settle, https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/344339
<jocha> blackboxsw: I think I pinged you about my updated merge request, but I don't think I received a response, I might've restarted the chat, anyways here it is again : https://code.launchpad.net/~jocha/cloud-init/+git/cloud-init/+merge/344192 :)
<blackboxsw> ahh thanks jocha I'll give it a looksie today
<jocha> awesome thanks!
<smoser> blackboxsw: i responded to your https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/344255 .
<smoser> really just wanting to know if you think i cleared things up
<blackboxsw> smoser: yes cleared. land at will, or I can
<smoser> ok. ill land
<blackboxsw> I'm camping in cloud-init hangout trying to get my IBMcloud setup up
<blackboxsw> now that I'm approved
<blackboxsw> but can't seem the find/create my API creds
<blackboxsw> got it. and updating launch-softlayer script
#cloud-init 2018-04-26
<smoser> rharper: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/344546 is ready
<smoser> i'll look at yours now
<rharper> smoser: reviewing
<rharper> smoser: reviewed, and i've updated my detect branch with your changes
<smoser> http://paste.ubuntu.com/p/FWkpmcPRpB/
<mgerdts> What amount of begging do I need to do to get my fixes into trusty?  rmadison tells me that trusty is stuck at 0.7.5.
<smoser> rharper: ^
<rharper> whoa
<smoser> my ^ was for the pastebin there
<rharper> heh
<rharper> ok
<smoser> mgerdts: trusty though..
<rharper> mgerdts: drops a bomb on release day no less =)
<mgerdts> sorry. :-)
<rharper> nah
<smoser> its going to remain stuck at 0.7.5... we'd have to do a SRU and cherry pick your changes.
<rharper> it's a fair question
<dpb1> at least he didn't ask that in #ubuntu-release
<smoser> i'd support helping you out there if you wanted to
<dpb1> :)
<rharper> dpb1: lol
<rharper> smoser: re paste, can I just take your test_util. change?  I think I've just pushed the others
<smoser> ie, if you want to get your datasource into a state your happy with and propose a merge to trusty, i'd be willing to help.
<mgerdts> If you can shine a light down the right path, I'm happy to wander down it.
<mgerdts> ok.  I have a couple more changes that are pending.
<smoser> rharper: probably yeah. as in i dont need you to take exactly my changes in the test cases if you have the same.
<rharper> ok, I do
<mgerdts> We need to figure out what to do about the post-first-boot network configuration.  That bit one of our early adopters this week.
 * rharper notices we have tests/unittests/test_util.py and cloudinit/tests/test_utils.py 
<mgerdts> Deployed a VM, booted once, then added a network.  Was surprised it didn't work.
<smoser> yeah, cloudinit/tests/ is new
<smoser> and the preferred path to add things too
<rharper> I thought that was for new files, but I guess we'll never migrate if we don't start putting stuff there
<smoser> mgerdts: so... going down the trusty path
<smoser> the first thing to do would be to checkout ubuntu/trusty
<smoser> then in order to  make changes, you have to do the debian/patches route.
<mgerdts> ok, that path seems to be lit well enough for me to get started.
<smoser> mgerdts: get code into a state your happy with, and then bug me, and i can help with the sort of busywork that it will entail
<mgerdts> sounds good.  I'll pick that up after my last two changes are are settled.
<mgerdts> Happy release day, btw.
<smoser> rharper: you'll push the udevadm_settle tests on top ther e?
<rharper> did I not ?
<rharper> ah, tox is done
<rharper> now I'll push
<rharper> smoser: it's there now was waiting on tox to finish
<smoser> k
<smoser> rharper: you've done some actual test with the code path ?
<rharper> I've not revalidated with the refactor of calling util.udevadm_settle() vs subp([udevadm, settle])
<rharper> I can fire that up if you like
<rharper> smoser: ^
<smoser> well before we upload it to bionci lets have *some* run :)
<rharper> ok
<rharper> do you want to land and we push a trunk package into some instances ?
<rharper> or would you prefer a positive test on gce before landing with the current branch ?
<rharper> it won't take long either way
<dpb1> some run or some fun?
<smoser> well, just make sure not obviously broken. build a deb, install it, run some reboots.
<rharper> k
<rharper> ok, I've got two instances in reboot loop on europe-west1;  hoping to catch a positive detection and a handful of continuously successful reboots with networking.
<paulmey> Hi all, I have a bug and MP for the NTFS mounting issues we've seen on RHEL when using cloud init on Azure: https://bugs.launchpad.net/cloud-init/+bug/1767002 and https://code.launchpad.net/~paul-meyer/cloud-init/+git/cloud-init/+merge/344538
<ubot5`> Ubuntu bug 1767002 in cloud-init "When stop(deallocate) and then start a RHEL 7.5 VM, cloud-init cannot mount /dev/sdb1 to /mnt." [Undecided,New]
<paulmey> When you have time, PTAL and let me know what you need...
<paulmey> this is blocking RH from supporting cloud-init on Azure...
<rharper> paulmey: isn't that kernel missing the ntfs module or possible the mount tools?
<rharper> we have ntfs-3g package installed on ubuntu
<paulmey> probably, but neither RH or SUSE want anything to do with NTFS
<rharper> and of course ntfs kernel module loaded
<paulmey> (for good reasons)
<rharper> so, what's the alternative? isn't that what you format the filesystem with on azure ?
<paulmey> way back when animals could still talk, Microsoft thought that they would only ever host Windows
<paulmey> so they 'prepare' the ephemeral drive by partitioning and formatting it NTFS and putting some files on it to warn for data loss
<rharper> but this is part of how one can detect whether the disk is "new" or not, right ?
<paulmey> cloud-init uses that to detect a 'freshly prepared' ephemeral drive before nuking it and putting ext4 on there, because ntfs is silly
<rharper> so I don't think it's so easy to ignore
<rharper> for sure, but users could have put something on it
<rharper> and we don't want to lightly nuke data
<paulmey> not on RHEL or SUSE they can't
<paulmey> because they have no way of mounting  it
<rharper> ok, then we'll need some distro related override here
<paulmey> alternatively, I was thinking of making this a datasource config flag? (whether or not to ignore mount errors)
<paulmey> but as you say, we could also have this as a flag on the distro class
<paulmey> or even move the method to the distro class?
<paulmey> if its datasource config, RHEL and SUSE can bake the config in their images ...
<rharper> the datasource get's a distro object
<rharper> so you can just ask distro.name
<rharper> s/ask/use
<paulmey> :-) ok, so then if RHEL or SUSE, ignore errors?
<paulmey> errors being this specific 'NTFS not supported' error
<smoser> paulmey: can you look in /proc/filesystesm ?
<smoser> rather than going by distro?
<paulmey> not really
<smoser> or did i tell you not to do that :)
<paulmey> because if the module is not loaded, it doesn't show up there yet
<paulmey> also, I think some distros now mount ntfs via fuse
<smoser> oh. hm.
<paulmey> When I get time, I'll go and fix my time machine, go back to when animals could still speak and then tell MS that they are also going to run Linux on their cloud (duh) and that they should just attach a raw disk... let the OS do the preparation
<rharper> lol
<paulmey> *sigh*
<rharper> smoser: I've not had the gce instances trip down the udevadm settle path yet; but I've 20 reboots on the branch just fine
<smoser> :)
<smoser> ok. lets land taht.
<paulmey> do you need some time to think on this bug/MP?
<rharper> smoser: ok
<smoser> i'm landing it.
<smoser> i just updated the sl bug with some info (and the ticket)
<rharper> k
<rharper> smoser: should I make a bionic upload branch then? is everything in master ?
<paulmey> rharper: smoser: are you happy with switching the 'ignore' behavior on the distro name? If so, I'll make that change in the MP
<rharper> paulmey: my preference would be to also check if it wasn't possible; like no kernel module and no tools;
<rharper> that woudl be slightly better in case someone rolls their own
<rharper> smoser: I've landed your ibmcloud branch, and I've put up a MP for master into ubuntu/devel, https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/344560
<paulmey> rharper: is there a good way to check that then?
<paulmey> rharper: plus, isn't 'mount' the ultimate authority on whether or not it is possible? Available modules should be loaded automatically, correct?
<paulmey> the current MP tries to mount and only ignores the error when mount says it can't because ntfs is not supported
<paulmey> what else should I check on top of mount's fstype=auto logic?
#cloud-init 2018-04-27
<guest____> hello there
<dpb1> o/
<rharper> paulmey: so os.path.exists('/sys/module/ntfs')  is one; and typically there should be a mount.$fstype binary in $PATH; so we have /sbin/mount.ntfs ;  mount -t $sf is going to invoke a mount.$fs binary to handle it;
<smoser> rharper: i just uploaded .. slight change to your MP.
<smoser> we had not added chagnelog entries for the commits on the ubuntu/devel branch
<smoser> so i added them.
<smoser> its https://launchpad.net/ubuntu/bionic/+queue?queue_state=1&queue_text= now
<rharper> smoser: ok; is that change that should be in qa-scripts new-upstream-snapshot ?
<smoser> rharper: i'm not sure. that might be possible.
<smoser> but what I had always been doing is when i commit to the ubuntu/devel branch (or any ubuntu/* branch)
<smoser> to "queue" the commits for next
<smoser> i would 'dch -i'
<smoser> and make a commit with 'git commit -m "update changelog"' debian/changelog
<smoser> i guess we could traverse the ubuntu branch since merge also and grab those commits to write the entry.
<smoser> kind of weird to have new-upstream-snapshot do it though as those commits/entries-in-changelog are not from the new-upstream-snapshot. they were from commits to the ubuntu/ branch
<smoser> woudl be nice though to have something doing that / envforcing it.
<rharper> I didnt mean for it to do it, but just printing of the steps
<rharper> I just need a reminder
<smoser> oh. well, they're kind of supposed to be done *befor* you run new-upstream-snapshot
<smoser> ie, when we commit to that branch
<rharper> heh
<smoser> we should do it
<rharper> hrm
<rharper> nothing new is committed ot ubuntu devel until we do the snapshot, right ?
<rharper> I'm confused (as you already knew)
<smoser> rharper: see 85ff391b29ec8ea2168371498ffe083c51258ef3
<smoser> we have packaging only on the ubuntu branches
<smoser> so those commits have to go to that branch
<smoser> i prefer commits like that to be in one commit that makes the change, and one commit that updates the changelog ("git commit -m 'update changelog'")
<smoser> that way they can be easily cherry-picked to other ubuntu/ branches
<smoser> as debian/changelog changes are going to always conflict
<rharper> oh, I see
<rharper> that hit master, but should ahve been ubuntu/devel only
<smoser> rharper: well we fixed in on master too
<smoser> in master its just for the bddeb work flow
<rharper> ok
<rharper> right
<smoser> i really only noticed that because I knew wer were explicitly looking for those 2 fixes
<smoser> (isc-dhcp-client and iproute2)
<blackboxsw> smoser: you setting up the Bionic update ?
<smoser> blackboxsw: yeah, i'll file a bug now
<blackboxsw> ok then I'll poke at IBM xenial a bit
<smoser> blackboxsw: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1767412
<ubot5> Launchpad bug 1767412 in cloud-init (Ubuntu Bionic) "SRU cloud-init 18.2-27-g6ef92c98-0ubuntu1" [Medium,Confirmed]
<rharper> smoser: blackboxsw and I were discussing if there was a way to keep the ConfigDrive -> softlayer detection in the xenial/artful only branches so bionic didn;t have to have that logic; but there is that gnarly hard-coded datasource list in /etc/cloud/cloud.cfg.d/ that's present which restricts what ds-identify looks for;
<smoser> um... what detection is that ?
<rharper> I don't think we have a way to tell ds_check on a particular datasource that it's really a different one;
<rharper> there's a datasource_list: [NoCloud, ConfigDrive, None] list in those images, that's what blackboxsw found on the xenial instance
<smoser> right
<rharper> which means that after upgrade, ds-identify still won't "See"  softlayer
<rharper> hoping to somehow transition to the softlayer datasource but that sort of means either nuke or re-writing , or somehow overriding it
<rharper> which I'm not sure how we can do that reasonably
<smoser> it should nto "see" softlayer
<smoser> thats the plan
<smoser> it shoudl go on using the datasource on disk.
<smoser> separately we can/should discuss with CPC for them to start making new images on IBM in the same way they make bionic images.
<smoser> so as to not have the slop that is there.
<danMS> @rharper thanks for taking a look at the ntfs bug (1767002), i saw your comments from yesterday, you asked if there are further checks that can be done, but Iâm not sure what further checks can be done? Many of the distroâs on the platform do not support ntfs in the images.
<rharper> danMS: generally, is there a kernel module loaded (ntfs); and are there tools (mount.ntfs, fsck.ntfs, mkfs.ntfs); would be two other indicators (specically mount.ntfs)
<rharper> that would allow modified centos/rhel images which have those tools to work rather than banning the distro entirely
<rharper> if that makes sense
<smoser> danMS: maybe another way out of this would be to just allow this configurable
<smoser> and have RH and SuSE build images with "support_ntfs: false"
<smoser> or what not.
<smoser> hten dont worry at all about blowing away data on an ntfs partition.
<smoser> rharper: http://paste.ubuntu.com/p/GYsxMyTR2F/
<smoser> blackboxsw: ^ that is what i was thinking is the fix
<rharper> smoser: but how would the softlayer ds get added to the datasource_list ?
<smoser> it wont
<smoser> and thats fine.
<rharper> smoser: I was hoping to transition into the softlayer datasource
<rharper> oh
<rharper> so, once launched on a ConfigDrive, it would stay that way through upgrade and dist-upgrade  to bionic ?
<smoser> yeah.
<rharper> that has some asymmetry w.r.t things checking what ds was used;  not sure if that's a big issue or not
<smoser> if the user removed the entry from the 99networklayer.cfg and dpkg-reconfigure, then they'd transition just as a new instance had been created.
<rharper> right
<blackboxsw> sure on pastebin smoser though maybe we can fold in the is_ds_enabled check inside is_ibm_cloud in ds-identify
<blackboxsw> because we also have that check in dscheck_NoCloud
<blackboxsw> I also just noted that our DataSourceNoCloud doesn't have a comparable filter for ibm too
<blackboxsw> like ds-identify does
<smoser> blackboxsw: yeah, you could potentially do that. but the fact that something is or is_not ibm cloud is not specifically related to whether or not hte datasource is in that list
<smoser> so.. i just wrapped. but yeah, i dont have a real good reason to not do what you did.
<smoser> uyeah..that was kidn of why i dropped the check in NoCloud
<smoser> (in my diff)
<smoser> if you have a Nocloud, then thats your issue. (and ours since we have aofficial images with it... )
<danMS> thanks guys, will chat with paulmey
<smoser> rharper: i'm going to force push the ubuntu/devel branch back to before we merged you.
<rharper> smoser: ok
<smoser> and then i'm going to make a bionic branch
<rharper> sure
<smoser> rharper: you want to do this or wnat me to ?
<rharper> which ?
<rharper> or what
<smoser> ubuntu/bionic is now in a state that you can run new-upstream-snapshot
<smoser> and it will all "just work"
<smoser> one change to new-upstream-snapshot that i just pushed
<rharper> I can run it again, but likely faster for you to do it
<smoser> and your bug nuber is dch --release --distribution=bionic
<smoser> oops
<smoser> bug number is 1767412
<smoser> ok. i can just do it.
 * rharper looks at git log to see what's different
<rharper> smoser: nice change to new-upstream-snapshot
<smoser> rharper: it should differ only in the changelog
 * smoser verifies that is true :)
<rharper> from what to what ?
<smoser>  7609065fd30c3a02122bded6ab213ebc3912584b
<smoser> is raharper/ubuntu/devel/newupstream-20180426
<smoser> what you had proposed
<smoser> and diff current ubuntu/bionic -> that
<smoser> you get
<dpb1> smoser: upload good now?
<smoser> http://paste.ubuntu.com/p/ScfhfhXvBP/
<smoser> yeah, i just uploaded
<rharper> yes
<smoser> i shoudl have reversed the order there
<smoser> but good.
<rharper> smoser: that looks right (minus the order)
<rharper> http://paste.ubuntu.com/p/DbqJ945jMh/
<rharper> I put that here: raharper/ubuntu/devel/newupstream-20180427
<smoser> just uploaded
<rharper> \o/
<smoser> https://launchpad.net/ubuntu/bionic/+queue?queue_state=1&queue_text= shows it.
<jocha> blackboxsw: Hey, just following up on merge request: https://code.launchpad.net/~jocha/cloud-init/+git/cloud-init/+merge/344192 :)
<blackboxsw> jocha: thanks sorry for the delay, if I can't get to it today, it will be tomorrow. (was on vacation yesterday)
<jocha> no worries! I understand that you must have a lot on your plate
<smoser> blackboxsw: did you get sorted on launch ?
<blackboxsw> smoser: I'm currently trying to use the UI for it. I'll try again cli and really look at the traceback to see what gives
<blackboxsw> smoser: this is what I hit https://pastebin.ubuntu.com/p/Vq7fw5S2tf/
<blackboxsw> ahh got it smoser
<blackboxsw> it's because of how I set up my account
<blackboxsw> to automatically inject my ssh key.
<blackboxsw> adding a check in launch-softlayer now
<blackboxsw> ok have a fix. pushing now
<blackboxsw> pushed, testing a potential SL fix now
<smoser> blackboxsw: i just pushed another on top there.
<smoser> make sure it still works for you
<smoser> i just verified emptying my keys and having it add one for me and then matching too.
<smoser> blackboxsw: you make progress ?
<smoser> i booted xenial on IBMCloud, upgraded, put this chagne in place
<smoser>  http://paste.ubuntu.com/p/cRPzBt3Qg7/
<smoser> (by hand)
<smoser> and then rebooted and came up ok.
<blackboxsw> smoser: I used the original changeset and tried a ds-identify run and saw an error on line 616
<blackboxsw> DS_LIST param not set
<smoser> yeah. i fixed taht
<smoser> DI_DSLIST
<smoser> is the name
<blackboxsw> yeah
<blackboxsw> smoser: the issue I think I'm seeing with that fixed is that xenial was originally ds-id'd as ConfigDrive but now returns NoCloiud
<blackboxsw> hrm, wait a sec
<blackboxsw> need to recheck that
<blackboxsw> :q
<blackboxsw> n/m strike that, it was originally detected NoCloud, not ConfigDrive. ok so logic remains intact
<smoser> well,. remember 4 scenarios
<smoser> with user-data and without, template and os_code
 * blackboxsw is waiting on reboot and reboot clean
<blackboxsw> even though the clean reboot isn't really an intended upgrade path.
<smoser> their apt  mirror is nice
<smoser> Fetched 83.1 MB in 2s (28.2 MB/s)
<rharper> suggests IO is decent as well
<smoser> well, eatmydata helps with the io
<smoser> and i dont run apt-get any other way :)
<rharper> lol
<paulmey> rharper: re "is there an ntfs kernel module", I propose to run 'modinfo ntfs" (rc=0) to check that is true and I'll run a util.which on 'mount.ntfs' to see if that exists in $PATH
<paulmey> does that sound like the right path?
<rharper> paulmey: yes;
<paulmey> alright, I'll make it so...
<smoser> blackboxsw: I did get the datasource list rioght in ConfigDrive
<smoser> 2018-04-27 20:52:41,219 - DataSourceConfigDrive.py[DEBUG]: devices=['/dev/xvdh'] dslist=['ConfigDrive', 'NoCloud']
<smoser> yeah, and then as i think....
<smoser> your case enabling IBMcloud on a template image without user-data on xenial
<smoser> hm.. maybe not
<smoser> i htought i had a thoguht on it.
<smoser> buti have to go now.
<smoser> later.
<blackboxsw> later
<blackboxsw> yeah and going through the motions here
#cloud-init 2020-04-20
<punkgeek> how can i import a shell script to the vm by cloud-init?
<andras-kovacs> Hi! Do you have any official images in Azure with cloud-init + LVM? The meta data collector part and runcmd don't work for me with my custom image.
<andras-kovacs> It was almost the same with OpenStack + LVM, but cloud-init could collect the meta data there and runcmd didn't work at all.
<andras-kovacs> I've tried to comment all the mount, resizefs and growpart related modules
<andras-kovacs> but no luck :(
<andras-kovacs> I really don't want to rely only on that stupid waagent
<andras-kovacs> I like cloud-init
<meena> 11:27 <andras-kovacs> Hi! Do you have any official images in Azure with cloud-init + LVM? The meta data collector part and runcmd don't work for me with my custom image. â¬ï¸ is there any other way to growfs in Linux than LVM?
<andras-kovacs> meena: sorry is it a question? By default cloud-init will increase your root or other pre-defined partition early in the boot time. It never worked for me with LVM but I wrote a script for that. My problem now is other modules don't work also when I haven LVM on my disk.
<andras-kovacs> Now I'm struggling with https://cloudinit.readthedocs.io/en/18.5/topics/datasources/azure.html
<andras-kovacs> And now I had to remove cloud-init from my images with LVM because even if cloud-init services are disabled, waagent will start whining because of cloud-init's dhclient scripts.
<Odd_Bloke> andras-kovacs: I'm not aware of any images using LVM, no.  What's the use case?
<Odd_Bloke> meena: You don't need LVM to grow partitions on Linux, generally only for more complex use cases.
<andras-kovacs> The use case is some legacy colleagues and strange apps -> not in the good way. LVM is a must... :(
<andras-kovacs> I still want to enjoy/utilize cloud-init + make my VM templates auto-grow to the selected flavor, use the datasources, etc.
<andras-kovacs> With OpenStack, the runcmd oart was ignored + I had to make a custom script to increase the LVM.
<andras-kovacs> With Azure, not the the datasource/meta-data part, neither runcmd works with LVM. :(
<andras-kovacs> I think I'll open an issue/bug report for you with all the details soon.
<Odd_Bloke> Yep, sounds like a bug would be useful.
<andras-kovacs> (I'll do the same for the dhclient vs NetworkManager lease files also just like you've recommended Odd_Bloke )
<Odd_Bloke> FWIW, I would suggest not using the root disk of your instances for whatever complicated layout you want, but I understand that may not be practical. :)
<andras-kovacs> And sorry for me whining all the time... cloud-init is just great and I really like it
<Odd_Bloke> :)
<andras-kovacs> this is how silly me made the root volume grow automatically: https://pastebin.com/FbggLAXJ
<andras-kovacs> (xfs is also a must)
<meena> "legacy colleagues" is a new oneâ¦
<smoser> andras-kovacs: https://bugs.launchpad.net/cloud-init/+bug/1799953
<ubot5> Ubuntu bug 1799953 in cloud-utils "growpart does not work for lvm" [Medium,Triaged]
<Odd_Bloke> rharper: blackboxsw: https://github.com/canonical/cloud-init/pull/326 <-- this should fix Jenkins failures we've been seeing
<rharper> Odd_Bloke: nice ... I'm wondering if we need to have a general mock of net.read_sys_path like we do for util.subp
 * rharper just recently ported the util.subp replacement from cloud-init to curtin to catch those leaking execs; 
<Odd_Bloke> Yeah, that's a good thought.
#cloud-init 2020-04-21
<caribou> Hello everyone, before filing a bug on this, I'd like your opinion : when setting *disable_root: false*, shouldn't *disable_root_opts
<caribou> be disabled as well ?
<caribou> as if one wants root login enabled, then disable_root_opts should not fill /root/.ssh/authorized_keys with blocking options
<caribou> I'd be happy to submit a patch for this
<meena> caribou: generally, disable_root disables password login for root, not the entire account. that means that you deffo want ssh keys
<caribou> ah, right fine then
<caribou> thanks for the precision
<Odd_Bloke> caribou: There's actually a recently-filed bug for that, let me see if I can find it.
<Odd_Bloke> caribou: Does https://bugs.launchpad.net/cloud-init/+bug/1871879 look like what you're asking for?
<ubot5> Ubuntu bug 1871879 in cloud-init "Configuring a user should not configure root's authorized_keys" [Wishlist,Triaged]
<Odd_Bloke> Oh, no, I think that's actually the opposite of what you're asking, my mistake.
<Odd_Bloke> (It's too early for me to process the double negative of "disable_root: false". ;)
<Odd_Bloke> So https://cloudinit.readthedocs.io/en/latest/topics/modules.html#authorized-keys is the documentation we have for the option, and it's not particularly clear IMO.
<Odd_Bloke> caribou: And, actually, reading the code, I believe that `disable_root: false` should lead to `disable_root_opts` being ignored.
<caribou> @Odd_Bloke actually, with meena's precision, I don't even need to use disable_root
<caribou> yeah, that's what I meant, just move the opt inside the 'if'
<caribou> but if disable_root is only for disabling pwd login, then it may still be usefull to have that entry in authorized_keys
<Odd_Bloke> caribou: I've just set up a cloud-config with an `authorized_keys:` section and `disable_root: false` and it works as I would expect (disable_root_opts is not applied).  If that isn't what you're seeing, could you file a bug please?
<caribou> sure, will check htat
<caribou> thanks for looking it up
<punkgeek> how can i import a shell script to vm by cloud init?
<Odd_Bloke> punkgeek: Could you explain what you mean by that?  If you want cloud-init to execute a shell script, you can either specify the script as your entire user-data (with an appropriate shebang) or use the runcmd cloud-config option: https://cloudinit.readthedocs.io/en/latest/topics/modules.html#runcmd.  If you want to write a shell script, then you probably want to take a look at
<Odd_Bloke> https://cloudinit.readthedocs.io/en/latest/topics/modules.html#write-files.
<Odd_Bloke> (If you want something else, let us know. :)
<punkgeek> Thank you that was what i want
<Odd_Bloke> Great! :)
<blackboxsw> at long last paride https://github.com/canonical/cloud-init/pull/231#pullrequestreview-397636926
<blackboxsw> Goneri thanks for the review updates on your PRs, grabbing that today and hopefully we can close out there
<Goneri> Yeah!
#cloud-init 2020-04-22
<caribou> Hello everyone, did someone report issues with cloud-init cloud-config & cloud-final services being blocked by snapd in recent ubuntu cloud images ?
<caribou> All I can find is a recent discussion in linuxcontainers.org reporting similar issues
<caribou> well looks like it's more of a snapd problem than cloud-init
<caribou> ok, it IS snapd's fault; removing the package release the cloud-config & cloud-final jobs
<caribou> any snapd IRC channel around ?
<caribou> FYI : https://bugs.launchpad.net/snapd/+bug/1874249
<ubot5> Ubuntu bug 1874249 in snapd "snapd service never completes on boot off focal cloudimg" [Undecided,New]
<Odd_Bloke> caribou: I haven't seen that particular failure.  BTW, it looks like you have truncated lines in your journal output there, which might make it harder for the snapd folks to debug.
<caribou> just got it today with the new cloudimg
<caribou> ok, I'll check the logs & add a new set
<blackboxsw> Odd_Bloke: I see the same issue I'm seeing over on ua-client repo. no travis links to jobs that are in progress. https://github.com/canonical/cloud-init/pull/323  have you  noticed this before?
<blackboxsw> I'm finding as well on ua-client that even completed travis jobs are not firing a status response back to the source PR, so it remains unmergeable
<Odd_Bloke> It happens from time-to-time, yeah.
<Odd_Bloke> Migrating to travis-ci.com should also fix this, I believe.
<Odd_Bloke> (Because AIUI .org uses an older, now-deprecated GitHub API.)
<blackboxsw> similar to intermittent probs like this I think https://github.com/travis-ci/travis-ci/issues/7363
<blackboxsw> gotcha
<blackboxsw> yeah I can see your travis run has completed with success https://travis-ci.org/github/canonical/cloud-init/builds/678274876 but no status update on your PR yet https://github.com/canonical/cloud-init/pull/323
<Odd_Bloke> Yep.
<blackboxsw> https://www.githubstatus.com/ github issue:  Update - We have implemented a fix and are processing a backlog of notifications.
<blackboxsw> Apr 22, 01:26 UTC
<Odd_Bloke> Travis reported they were operational after that.
<Odd_Bloke> Oh, not after that, but I think notifications are probably user-facing notifications?
<Odd_Bloke> https://www.traviscistatus.com/incidents/bj882gcyxh9v corresponded to https://www.githubstatus.com/incidents/dsf2qtzh4jpz
<AnhVoMSFT> if I want cloudinit to write netplan yml file into /run/netplan, where is the right place to change the netplan_path?
<AnhVoMSFT> I changed it in the datasource's __init__ (distro.renderer_configs['netplan']['netplan_path']) but I don't think it's being picked up
<Odd_Bloke> AnhVoMSFT: I don't know off the top of my head.  What are you trying to achieve by doing this?
<AnhVoMSFT> cloud-init, once disabled/removed leaves behind the /etc/netplan/50-...yml configuration file that has a mac address hardcoded in it. This causes problem for customers who snapshots the VHD and wants to boot them up as a separate VM.
<AnhVoMSFT> since network configuration is re-generated upon every boot on Azure anyway, it makes more sense to write the netplan configuration file in /run where it does not persist across boot
<AnhVoMSFT> I'm trying to change the path of the netplan config from within the datasource so that it writes to /run/netplan instead
<Odd_Bloke> Shouldn't cloud-init run on those VMs and regenerate the correct configuration (with the appropriate MACs for that VM)?
<AnhVoMSFT> A couple scenarios where that does not work: 1) Customers already disabled/removed cloud-init, 2) In some scenarios, the metadata source isn't available when booting these VHDs
<AnhVoMSFT> So I changed the distro's renderer_config that was passed to the datasource, but when I print it out from distros/__init__.py's _supported_write_network_config, it does not seem like the change was picked up
<Odd_Bloke> And what would happen to an instance that was rebooted and had cloud-init fail for some reason?  I think it would fall off the network if its networking config was all in /run?
<AnhVoMSFT> That is a good point. I think it would depend on when/where cloud-init fails. Let me think about it a bit
<Odd_Bloke> In fact, (1) is a case where storing the network config in /run would fail too, isn't it?  `apt remove cloud-init; reboot` -> no network config
<AnhVoMSFT> indeed. Would writing a netplan file into /etc/netplan without a mac address, then write one with mac-address into /run work? thinking out loud
<AnhVoMSFT> although that is probably as good as not writing mac address into /etc in the first place
<Odd_Bloke> AnhVoMSFT: This feels quite complex, and I'm worried that we will miss/forget stuff if we discuss it in IRC.  Do you think you could file a bug for it so that we can make sure we all understand the requirements/problem statement?
<AnhVoMSFT> let me see if we had an existing bug on it
<AnhVoMSFT> we did talk about this with Ryan and Josh in one of our sync meetings and at the time the /run approach seemed reasonable, but you pointed out a pretty big gap
<AnhVoMSFT> I guess the main problem is cloud-init is leaving behind the netplan file with a hardcoded mac address in it
<Odd_Bloke> Well, it's "leaving it behind" so that it can apply network configuration correctly on the next boot, so it's not entirely a "problem". :)
<AnhVoMSFT> I think what I meant was when it gets removed/uninstalled, etc...
<AnhVoMSFT> but the problem isn't so much of leaving it behind, the problem is it hardcodes the mac address in it, which potentially can become stale and if there isn't an entity that updates it
<Odd_Bloke> Right, but I think hardcoding the MAC address is the correct thing to do in the general case.  Because if we don't do it then, potentially, on future boots, interfaces can be presented to userspace with different names (this can happen due to races in the kernel, so it's not platform-specific, or it can be the platform presenting them at different PCI addresses), and we'll apply incorrect configuration.
<Odd_Bloke> (Do you already have a deprovisioning process that these customers are expected to follow?  Could that be expanded to include a step which calls cloud-init somehow?)
<AnhVoMSFT> if there is only one nic there's no need for hardcoding mac. Or do we still need to hardcode it?
<AnhVoMSFT> the trouble is the backup/restore scenario where the customer takes snapshot or backups the OSDisk, then later restore it (as a different VM)
<AnhVoMSFT> although backup/restore might not be as big of a problem if they provision it again as a normal VM, because cloud-init will run and perform network config
<Odd_Bloke> And boots of those restored VMs don't run cloud-init?
<Odd_Bloke> Aha, we raced on the question and answer there. :)
<AnhVoMSFT> only if they attach OS Disk as specialize VM (which is the only way to boot up from a vhd today)
<AnhVoMSFT> so there're some limitation of the platform there - when attaching disk as specialize vhd there isn't provisioning information being made available and cloud-init fails at some point earlier on and doesn't really do network config if I remember correctly
<AnhVoMSFT> (it would fail to find Azure datasource, because there's no provisioning ISO attached)
<Odd_Bloke> To answer a slightly earlier question: we wouldn't need to hardcode the MAC if we were sure there would only _ever_ be one NIC.  But instances could have NICs attached, or disk images could be restored to systems with multiple NICs, so we can't assume that.
<Odd_Bloke> (Obviously the restore case would break with a hardcoded MAC, so perhaps that wasn't the best example.  Still, the attach case is valid.)
<AnhVoMSFT> yeah, customers can add new NIC, reboot, and probably lose network :-)
<AnhVoMSFT> actually in that case no, because when they reboot they will get new config with 2 NICs and we'll be writing network config correctly (hopefully)
<Odd_Bloke> Right, this would be the case where cloud-init had been disabled, I guess.
<Odd_Bloke> Instance booted with a single NIC, cloud-init persists MAC address, cloud-init is removed, NIC added, reboot -> the cloud-init generated config will still reliably apply to the original NIC
<Odd_Bloke> (Right?)
<AnhVoMSFT> right
<AnhVoMSFT> this is tricky...
<Odd_Bloke> Agreed.
<Odd_Bloke> :p
<AnhVoMSFT> let me look into the scenario where we boot up vhd and no provisioning ISO attached
<Odd_Bloke> Yeah, this definitely feels like we need to understand the exact requirements driving the change, because that could make a substantial difference to the solution.
<AnhVoMSFT> perhaps we can do something there
<AnhVoMSFT> yeah, we have these support cases from backup/restore customers who now fail to boot up VM due to mac address in netplan. I will take a closer look and perhaps file a bug with better details so we can discuss
<Odd_Bloke> OK, cool, thank you!
<AnhVoMSFT> thanks Odd_Bloke
#cloud-init 2020-04-23
<mydog2> hi
<Odd_Bloke> o/
<ananke> I'm trying to build some failsafe checks in our packer pipeline. when 'cloud-init status' is 'running', what's the exit code?
<ananke> error results in expected non-zero, but I want to make sure my check doesn't fail if it's still running
<ananke> actually, nevermind, I'll just rely on --wait
<ananke>                 "/usr/bin/cloud-init status --wait || (cat /var/log/cloud-init.log /var/log/cloud-init-output.log; exit 1)"
<Odd_Bloke> Yep, --wait is your best bet, I think.
<ananke> my problem was that we already used --wait, but if it resulted in errors we never knew what happened: packer would terminate the provisioner and kill everything
<ananke> so by adding the second part I will now have logs emitted to stdout if something goes wrong
<Odd_Bloke> You could also consider using --long, if you already capture the logs some other way.
<Odd_Bloke> That gives you some indication of the outcome, without emitting 100s of (mostly) irrelevant lines.
<potoftea> Hi, I'm looking for way to load larger cloud-init.yaml (AWS 16kb limit) to instance. Is there anyway I can download the file during startup and then pass to cloud-init? maybe somebody can point me into right direction. Thank you in advance
<rharper> potoftea: your user-data can use #include http://path/to/your/cloud-config
<potoftea> rharper sadly that is not an option, download requires auth access/secret. But thank you for suggestion. My plan was to somehow download it from S3 if that's possible at all
<Odd_Bloke> Could you configure S3 to allow "unauthed" access to a particular role, which you could then grant to your instances before launch?  I'm not sure if that would affect HTTP(S) traffic though, or only S3 API traffic.
<potoftea> Not really file contains keys (certs), which requires limited access.
<Odd_Bloke> Right, you would limit access using IAM roles.
<potoftea> I do have that now, but as far as I know that works only with S3 API, not thought HTTP
<Odd_Bloke> Hmm, bummer, OK.
<potoftea> I know that ignition before it was deprecated , solved is this in a nice way, where I could load data from S3 remote storage.
<Odd_Bloke> I was going to suggest a part-handler, but that doesn't give you a way of feeding into the rest of cloud-init's operation: https://cloudinit.readthedocs.io/en/latest/topics/format.html?highlight=%23include#part-handler
<potoftea> Can I call cloud-init from cloud-init ? Going trough doc didn't gave clear picture, how can I from cli execute cloud-init and pass config as arg
<Odd_Bloke> potoftea: Oh, are you compressing your user-data?
<Odd_Bloke> cloud-init should detect it as gzip compressed and uncompress it before processing it.
<potoftea> yeah I'm with compresion 24kb and without around 30kb
<Odd_Bloke> Damn.
<Odd_Bloke> This is a chunky user-data file! :)
<potoftea> I've bunch kubernetes certs '=D
<Odd_Bloke> Aha, that would explain why compression doesn't help much
<rharper> potoftea: cloud-init --file /path/to/config ... so, you can run cloud-init from cloud-init; is there a particular config module you want to run ?  if so, then cloud-init --file config single --name cc_module --frequency=always will make it happen
<potoftea> I have module: "apt", "write_files"
<rharper> you'd need to run single twice, once with different module names ...  ;  how does this help your payload size issue ?
<rharper> or unrelated question
<rharper> blackboxsw: not sure if you saw, but daily-ppa focal image failed to build again with patches not applying ...
<Odd_Bloke> rharper: It would allow for a runcmd which fetches the large config from elsewhere but still uses cloud-init to apply it.
<meena> oh, nice, "kubernetes"
 * meena jumps ship
<rharper> Odd_Bloke: I see
<Odd_Bloke> (And that runcmd could use s3cmd or whatever to fetch from S3, which we can't/don't do generically for #include.)
<rharper> Odd_Bloke: that'd be neat to do
<potoftea> rharper I would have 2 cloud init config. 1. bare bone that contains awscli, download config from s3, and then call cloud-init with 2. config.
<Odd_Bloke> Agreed, though we would need to think about how to handle the required dependencies for each object store.
<rharper> potoftea: yeah, got it
<Odd_Bloke> (And whether they're even in main, from an Ubuntu perspective.)
<rharper> is it in their tools snap ?
<Odd_Bloke> I do wonder if we're also missing a generic way of letting people fetch from $wherever.
<rharper> other than the #include URL ?
<Odd_Bloke> Like a part-handler, but which returns cloud-config instead of just doing some stuff.
<rharper> but delegating the "acquire" to tool specified ?
<rharper> like #include <cmd> <input> ?
<Odd_Bloke> I was thinking even more like a part-handler: user specifies a Python script with a "def fetch(whatever, common, params, make, sense):" and it returns a YAML string (or a parsed dict, perhaps).
<potoftea> rharper yeah it worked.
<rharper> nice!
<Odd_Bloke> rharper: Thanks for all these reviews!
<Odd_Bloke> Now to clog up Travis for the rest of the day landing them. :p
<potoftea> Odd_Bloke I would be nice to have this kind of functionality of the box, as to me it seems 16kb is only useful for simple configurations
<potoftea> Thank you for helping me with this issue.
<Odd_Bloke> So I do think a lot of people will graduate to config management if they would be producing 16kb of user-data (and then their user-data just needs to configure Chef/Puppet/..., so is much smaller), but if you have a lot of big files (big in the context of 16kB, at least :p) to write then I agree that it's a low limit.
<Odd_Bloke> And particularly if those are certs, because then compression isn't going to be very effective.
<rharper> Odd_Bloke: =)
<rharper> Odd_Bloke: I definitely s3 seems like the right place to put blobs and it would nice for cloud-init to security get them from within the instance;
<potoftea> Do you guys have features request? I can create one, where I ask for S3 support, if that makes any sense
<Odd_Bloke> Yeah, S3 is right for EC2, I'd definitely want us to think about how to make it cloud-agnostic (even if we leave implementation of others to later).
<rharper> potoftea: yeah, https://bugs.launchpad.net/cloud-init/+filebug ; we'll triage it to Wishlist (that's the bucket we have for feature requests)
<rharper> Odd_Bloke: Yeah, object-store fetching ... I suspect most clouds have something like that
<rharper> though I think many of them have s3-like apis
<Odd_Bloke> And we also shouldn't tie S3 to EC2 tightly; plenty of places will deploy across multiple platforms but want to store objects in a single place.
<rharper> right, I think s3 is fairly generic object-store api ... though I've not dealt with it in detail to know if that's accurate ; just what I've seen mentioned in various places
<Odd_Bloke> Yeah, S3 support would buy us a decent chunk.
<Odd_Bloke> Ideally, I think we'd implement this as a generic API that people could implement for other data stores, and ship a concrete implementation of S3 as part of cloud-init.
<Odd_Bloke> So then we could allow people who want to fetch userdata from a SVN repo that's stored on NFS to write a Python script to that themselves, but support 90% of people out-of-the-box.
<potoftea> I've created ticket, thank you
<rharper> Odd_Bloke: github actions busy often?  I got an error report mentioning There was a failure in sending the provision message: Unexpected response code from remote provider InternalServerError
<Odd_Bloke> Yeah, I haven't seen that before, I'm seeing it too.
<Odd_Bloke> https://www.githubstatus.com/incidents/zdxk6xq21405
<Odd_Bloke> GitHub are having quite the week.
<rharper> heh
<rharper> maybe it's release week for them as well ?
<Odd_Bloke> Haha, I was about to say, perhaps someone there is having a worse week than you. ;)
<rharper> hehe
<Odd_Bloke> Oh, actually, we don't require the CLA check to pass, so I can still land my branches. \o/
<Odd_Bloke> (Maybe we should fix that, though let's do that after I've landed my branches. ;)
<Odd_Bloke> rharper: I had to push another commit to fix CI, and it's a bit of an odd one (albeit small) so I'm asking for a re-review of it: https://github.com/canonical/cloud-init/pull/322/commits/315478ba587ef0165d846d45ac0d6407a3e948b9
<Odd_Bloke> (As we squash-merge, we'll also need to make sure the merge commit has that info.)
<rharper> Odd_Bloke: ok
#cloud-init 2020-04-24
<blackboxsw> Goneri: https://github.com/canonical/cloud-init/pull/298#pullrequestreview-400211001 is good if you can resolve merge conflicts
<Goneri> blackboxsw, yup, give me 2 minutes
<blackboxsw> excellent thanks
<blackboxsw> I'm on your PR 305 now
<Goneri> blackboxsw, done
<blackboxsw> merged Goneri thx again
<Goneri> :-)
<Goneri> has anyone already considered a MacOS port?
<Odd_Bloke> What's the use case for cloud-init on MacOS?
<blackboxsw> hrm. are you thinking xcloud.me type stuff?
<blackboxsw> Goneri: review posted on https://github.com/canonical/cloud-init/pull/305/files minor unit test suggestion based on the new pytest style we have at https://cloudinit.readthedocs.io/en/latest/topics/hacking.html#unit-testing
<blackboxsw> otherwise good to go on that PR
<Goneri> Odd_Bloke, yeah, to be able to start a MacOS VM and create the user on the fly
<blackboxsw> Woot thx for the review rharper write_files schema is merged
<rharper> blackboxsw: \o/
#cloud-init 2020-04-25
<vegardx> Using NoCloud, what part of meta-data or keys take presence of values provided in SMBIOS or values defined in meta-data and user-data? I was hoping to create a somewhat generic user-data, but be able to provide the instance with a unique hostname using the parameters in SMBIOS.
<vegardx> Unless I've misunderstood something I have to basically specify the same thing twice. Which means I have to generate a new disk image for every machine.
<vegardx> (using KVM with libvirt and virt-install locally)
<blackboxsw> vegardx: could you provide a bit more information on that. NoCloud can obtain metadata from a seedfrom url so you can provide dynamic metadata from custom urls if we wish.
<blackboxsw> if *you* with I mean https://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html
<blackboxsw> e.g. you can pass this option to QEMU: ...    -smbios type=1,serial=ds=nocloud-net;s=http://10.10.0.1:8000/
<blackboxsw> 's' a short option for  seedfrom= which can be customized with whatever meta-data and user-data you want to provide
<blackboxsw> vegardx: and for more generic userdata you could use jinja templates in your cloud-config that react to the underlying environment of a system (dhcp hostnames, ip address range, distro etc) https://cloudinit.readthedocs.io/en/latest/topics/instancedata.html#using-instance-data
 * blackboxsw has to fall off into the end-of-week abyss. see folks later
<LongLiveCHIEF> ð new to the room! I'm the maintainer of the official octoprint docker image. Came here as a result of some testing I'm doing to prepare new user docs for the octoprint image
<LongLiveCHIEF> what I'm running into atm, is user-data being ignored at boot on raspberry-pi devices when it's in the root of the `system-boot` partition, and i'm wondering if I should be placing user-data in the writable partition instead
<LongLiveCHIEF> i've spent a few days getting up to speed with cloud-init, so i'm familiar with the boot stages, user-data and meta-data formats, etc..  don't be afraid to ask low-level questions if you need more info from me
#cloud-init 2020-04-26
<goodarzi_ah> Hi all. I need some help with cloud-init.
<goodarzi_ah> I want
<goodarzi_ah> I want to create a custom image disk (vmdk) which is imported to a openstack. this image will install and configure nextcloud and other stuffs.
<goodarzi_ah> the problem is when i create iso from my own config.yaml file, i have to attach it to cloud images (qcow2).
<goodarzi_ah> Is it possible to create a vmdk which is include the iso file and the cloud image?
