#cloud-init 2013-09-23
<smoser> harlowja, are you awake yet?
<harlowja> sup
<harlowja> smoser poke
<smoser> hey
<smoser> harlowja, i was gonna ask you to look at gholms mp for selinux
<harlowja> whats that
<harlowja> u took golhoms ring
<harlowja> why did u do that
<harlowja> *gollum
<harlowja> ah, a review :-P
#cloud-init 2013-09-25
<daveops> I'm not sure how well cloud-init is supposed to support RHEL-based OSes, but I'm having issues with it not setting the hostname permanently.
<daveops> Related (old) bug: https://bugzilla.redhat.com/show_bug.cgi?id=750979
<daveops> tl;dr the correct hostname gets set on the first boot, but is unset on subsequent boots because RHEL uses /etc/sysconfig/network instead of /etc/hostname
<daveops> CentOS 6.4 guest, cloud-init-0.6.3-0.12.bzr532.el6.noarch
<smoser> daveops, 0.6.3 == old
<smoser> should be fixed in newer
<daveops> smoser: thanks... i figured it was pretty antiquated. i'll update my stuff.
<harlowja> if u get stuck daveops let me know, y! runs 6.3, 6.2, 6.4 internally using cloud-init
<harlowja> and even 5.8 (yes i know)
<daveops> harlowja: thanks. i got it working on my version by doing some hackery in my cloud.cfg. i'm just removing /etc/hostname with a bootcmd command. it's not pretty, but we need to pull our packages from EPEL and that's the latest version they have atm.
<harlowja> ah
<harlowja> ya, in about 10 years RH will have the newer version
<harlowja> lol
<harlowja> +-
<daveops> haha... right?
<harlowja> ya, we fight with that stuff daily
<daveops> harlowja: how are you guys addressing the issue?
<harlowja> old depenedencies
<harlowja> we build it :-P
<harlowja> its not so hard, its hopefully pretty much 'make rpm'
<harlowja> we have a few patches for cloud-init modules that do some y! stuff that also gets sucked in
<daveops> oh.. nice. i'll test it out and maybe we'll just end up doing that
<harlowja> but thats about it
<harlowja> ya, it shouldn't be to bad
<daveops> appreciate the advice
<harlowja> np
<harlowja> if its not just 'make rpm' let me know daveops and then something has to be fixed
<harlowja> of course u might want to have a patch for cloud.cfg if u build it yourself
<harlowja> to remove various things
<harlowja> *debian/ubuntu things
<harlowja> those nasty things smoser put in there, ha
<daveops> for sure... i'll just steal the patches the fedora guys are using
<smoser> everything should "do the right thing", no?
<harlowja> i think so
<harlowja> but just incase, ha
<harlowja> i typically just reconfigure cloud.cfg to what y! wants, and remove all the other stuff
<harlowja> but people don't have to do that if they don't want
<harlowja> its not that many changes really
<smoser> https://code.launchpad.net/~gholms/cloud-init/rsyslog-programname/+merge/186906
<smoser> i'm not sure what i htink about that one
<smoser> it just seem like possibly something could go wrong ...
<harlowja> hmmm, whats the issue, i'm not really a syslog expert :-/
<smoser> yeah. thats why i dont want to just take it.
<smoser> i dont understand what si wrong.
<harlowja> :)
<harlowja> i'm with u there, maybe we can get an explanation as to whats broke
<smoser> harlowja, it does work for you, right?
<harlowja> not using syslog :-P
<harlowja> ha
<smoser> bah.
<harlowja> lol
<gholms> smoser: Yo
<smoser> hey.
<smoser> i commented in that mp.
 * gholms looks
<gholms> Oh, the logging one.
<smoser> basically "it doesnt seem to be working, but it used to, so we should change somethign ..."
 * gholms nods
<gholms> Yeah, that is a terrible description.
<smoser> :)
<gholms> At that point I was terribly fatigued from working on my bug backlog all day.  ;)
<smoser> gholms, well, i'm expecting 0.7.3 like end of this week.
<gholms> Oh!  That's good to know.
<gholms> I'm going to see if I can get you a better description of what it was doing, but the short version is that cloud-init logs were going to the default log file instead of cloud-init.log, and that patch was how I fixed it.
<smoser> theres one change i'm expecting from utlemming, and i'd like to hvae the fix for https://bugs.launchpad.net/cloud-init/+bug/1225922
<smoser> but its tricky
<gholms> That one looks fun.
<smoser> gholms, yeah... i'm just weary of a config file change for something that used to work and should work.
#cloud-init 2013-09-26
<gholms> smoser: My first name has two 't's in it.  ;)
<smoser> gholms, fixed in revno 877. Sorry about that.
<gholms> smoser: No worries
<harlowja> smoser do u know if there is a good email list for questions about launchpad, basic stuff like can a blueprint span 2 projects
<kwadronaut> harlowja: what about https://answers.launchpad.net/launchpad or #launchpad here on freenode?
<harlowja> ah
<harlowja> let me try that irc channel
<harlowja> thx
<kwadronaut> yw
<kwadronaut> there's also launchpad-users@lists.launchpad.net
<harlowja> cool
<kwadronaut> https://xkcd.com/1254/
<harlowja> ha
<harlowja> once of these will work :-P
#cloud-init 2013-09-27
<pedroalvarez> Hello
<pedroalvarez> I need some help configuring cloud-init. Is not for use it in a common distro as you already know
<pedroalvarez> For example how can I avoid that cloudinit thinks I'am in ubuntu? Should I create a new distro configuration file? or is enough adding something to cloud.cfg?
<pedroalvarez> I don't have any log as well, I only can see what is happening through `journalctl`
<smoser> you will need ot have a distro. it will think yuou are ubuntu unless configured otherwise. you can tell it to think you are something else, but that might have similar fallout.
<smoser> cloud.cfg says (default):
<smoser> system_info:
<smoser>    # This will affect which distro class gets used
<smoser>    distro: ubuntu
<pedroalvarez> I already deleted the "system_info"
<smoser> so that will just use its builtin
<smoser> which is probably distro: ubuntu too
<pedroalvarez> I see
<pedroalvarez> And, another important question is
<pedroalvarez> If I want cloud init reading scripts from OpenStack, I have to configure...
<pedroalvarez> a) datasource
<pedroalvarez> b) datasource and something else
<pedroalvarez> c) another file instead cloud.cfg
<pedroalvarez> :)
<smoser> the built in datasource list should contain both config-drive and ec2
<smoser> (which are the datasources openstack will provide)
<smoser> you probalby have to maintain your own cloud.cfg file. or, you can overwrite things in /etc/cloud/cloud.cfg.d/MY_OVERRIDES_.cfg
<pedroalvarez> smoser: Thanks, I'll continue researching around there :)
#cloud-init 2013-09-28
<samppah> good evening
<samppah> i'm testing cloud-init with oVirt 3.3 and centos 6.4 (cloud-init installed from epel).. it seems that oVirt is able to creat config-drive image and attach it to VM but for reason or another the VM seems to bypass it completely
<samppah> any ideas what might be wrong? :)
<samppah> mount LABEL=config-2 /media works fine inside VM
#cloud-init 2014-09-22
<hiren_> harmw: around? 
<hiren_> harmw: did you get a chance to make freebsd port out of your changes?
<smoser> hiren_, he is waiting on a release i tihnk
<smoser> ie, an upstream tarball that has it al lin
<smoser> i think the necessary pieces are there.
<smoser> would like to get the https://code.launchpad.net/~josephbajin/cloud-init/freebsd-configdrive/+merge/231214 in though
<hiren_> smoser: ah okay. so, looking at that review, its being worked on, I guess.
<hiren_> smoser: the change looks okay to me.
<smoser> there is one change i asked for there that he didnt addres
<smoser> the 'b'
<smoser> i can probably just do that. but i'd need someoen to be able to test it
<smoser> can you test tha t?
<hiren_> smoser: thats what I am trying to setup at $work right now. So that I can test things. 
<harlowja> hiren_ u can do it !
<harlowja> lol
<hiren_> hah. :D
<harlowja> did u get the SE guys to get u a working mini-cluster?
<smoser> hiren_, if you wan that MP to get accepted. 
<smoser> and you dont want to wait on me.
<smoser> sometimes you can trick harlowja into doing things for you
<harlowja> lol
<harlowja> FYI, don't tell the person that u want to trick that u might try to trick him
<harlowja> :-P
<hiren_> heh harlowja thats the new trick. None expects that.
<smoser> i guess that could be a problem if the trickee was of suitable intelligence.
<smoser> :)
<harlowja> :-P
<harlowja> what u saying smoser , lol
<smoser> fool me once, shame on you.
<smoser> fool me twice, shame on me.
<harlowja> fool me thrice, shame on smoser 
<harlowja> lol
<hiren_> haha
<smoser> touche
<harmw> hiren_: I guess you hve your answer :)
<hiren_> harmw: yep :-)
<hiren_> harlowja: about testing, yes, I do have my own little clustr now, fighting with other little things. 
<harlowja> k
<harlowja> openstack is a fight with many little things lots of the time :-P
<hiren_> heh yeah.
<smoser> what is platform.system.lower() on freebsd ?
<smoser> and is it the same for *bsd ?
<smoser> hiren_, ^ ? or harlowja ?
<hiren_> smoser: I believe you'll get "freebsd" 
<hiren_> not sure how is is for other bsds
<hiren_> smoser: on my laptop
<hiren_> flymockour-l7% uname -a
<hiren_> FreeBSD flymockour-l7.corp.yahoo.com 11.0-CURRENT FreeBSD 11.0-CURRENT #29 r271744M: Thu Sep 18 10:14:21 PDT 2014     hirenp@flymockour-l7.corp.yahoo.com:/var/tmp/waste/usr/home/hirenp/head/sys/GENERIC  amd64
<hiren_> flymockour-l7% python3.3
<hiren_> Python 3.3.5 (default, Sep 18 2014, 19:50:32) 
<hiren_> [GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd11
<hiren_> Type "help", "copyright", "credits" or "license" for more information.
<hiren_> >>> import platform
<hiren_> >>> platform.system()
<hiren_> 'FreeBSD'
<hiren_> >>> platform.release()
<hiren_> '11.0-CURRENT'
<hiren_> >>> 
<smoser> thanks
<hiren_> np and I am very positive this will hold true for other bsdx.
<hiren_> bsds*
<smoser> hiren_, ?
<smoser> you think that netbsd will show platform.system() as 'FreeBSD' ?
<smoser> or openbsd ?
<smoser> harmw, or hiren_ or harlowja_ if you freebsd like folks get a chance, try
<smoser>  https://code.launchpad.net/~smoser/cloud-init/freebsd-configdrive/+merge/235513
<smoser> lp:~smoser/cloud-init/freebsd-configdrive
<harlowja_> hiren_ hmmm, 3.3.5 u say, thats gonna get interesting, lol
<harlowja_> they don't have a 2.7?
<harlowja_> afaik, without further things being merged 3.3.5 won't be working
<smoser> thats just on his laptop
<smoser> they do have a 2.7
<hiren_> smoser: I meant, netbad would return "netbsd" :-)
<hiren_> harlowja_: I do have 2.7
<harlowja_> k
<hiren_> flymockour-l7% python2.7
<hiren_> Python 2.7.8 (default, Sep 18 2014, 18:38:38) 
<hiren_> [GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd11
<hiren_> Type "help", "copyright", "credits" or "license" for more information.
<hiren_> >>> import platform
<hiren_> >>> platform.system()
<hiren_> 'FreeBSD'
<hiren_> >>> 
<hiren_> there you go :D
<harlowja_> cool
<smoser> ok. well, give that branch a try please.
<hiren_> smoser: yes, when I can resolve this local issue, I will.
<hiren_> with the help of harmw, of course :-)
<harmw> I can't help
<harmw> I suck at that
<hiren_> harmw: yes you can. you have helped me. :-)
<harmw> nah, thats just because I'm good at guessing which routes to take :p
<hiren_> :-)
#cloud-init 2014-09-23
<hiren_> harlowja: now that we have something running, do you think I can test fbsd realted things to move smoser's patch forward?
<harlowja> hiren_ sure, the next qustion is how u want to do that ;)
<hiren_> heh, yeah.
<hiren_> so, thats why I was hoping for a port
<harlowja> since u got libvirt to boot something, u can attach/create a config-drive without requiring a larger openstack cluster
<harlowja> and u can add another disk in the libvirt xml file
<harlowja> that is the config-drive that we would also use at yahoo
<hiren_> okay
<hiren_> and that config drive has "things" that I'd have to parse as part of bringup
<hiren_> ?
<harlowja> well cloud-init parses them, but yes
<hiren_> yes, okay.
<hiren_> so for that cloud-init part, I'd need a port or soething I can apply
<harlowja> get an openstack vm for rhel, look at /dev/vdb (mount it)
<harlowja> yes, a port, source install, unsure whatever else it comes packaged as :-P
<hiren_> yeah. okay.
 * hiren_ -> afk
<smoser> hiren_, you should fairly easliy be able to build for freebsd. there is something in tools/ that i suspect should do that.
<smoser> at least that is what i understood harmw's intent with that.
<harmw> indeed smoser, tools/build-freebsd or something
<harmw> though hiren_'s default env doesn't have dhcp nor vnc, complicating things a little
<JayF> So you guys probably prefer using the built-in build system, but I made some docker builders I'm using for cloud-init that others might find interesting: https://github.com/jayofdoom/cloud-init-docker-build (I believe the cloud-init-(debian|fedora)-pkg repos we're using are modified versions of the various pieces pulled out of the source rpm/deb)
<smoser> JayF, thats nice.
<JayF> smoser: I do good work :P
<harlowja> harmw hiren got it working, some weird thing with his image i think
 * harlowja got it all under control (ha)
<harlowja> or somewhat under control :-P
<harlowja> afaik he's trying to get it all going under libvirt + kvm for now, they will try with openstack
<harlowja> get the basic working outside of the bigger system first
<hiren_> howdy! 
<hiren_> yes what harlowja says.
<hiren_> :-)
<harlowja> jay of doom, nice JayF  :-P
<JayF> :)
<harmw> oh lol, ok hiren_ 
<harmw> using kvm is indeed way easier :p
<hiren_> hah. I dont know shit. 
<harmw> harlowja: didn't you know someone from the Neutron team?
<harlowja> ya, the prior PTL works for yahoo, so i guess thats someone :-P
<harmw> hehe
<harlowja> *PTL before the current PTL
<harmw> yea, I got that :p
<harlowja> kk
<harmw> well I'm just cirous on any ipv6 status, like support for running dualstack L3 routers
<harlowja> hmmm, have u tried the #openstack-neutron room?
<harlowja> they might know
<harmw> they're not very responsive
<harlowja> :-/
<harmw> atleast not last 2 times 
<JayF> I know there were blueprints about v6 in neutron
<JayF> I think they support it for almost everything now
<JayF> but imbw
<harmw> yea, v6 works
 * JayF sadly doesn't have IPv6 yet
<harlowja> when mark mcclain comes in to that IRC we can tag team him
<harlowja> *not tag team him like that u sickos
<harlowja> lol
<harmw> ah, well, knowing a name atleast might help :)
<harmw> haha
 * harlowja should now be quiet
<harmw> JayF: my problm is I want to run dualstack, which requires multiple external adresses and such
<harmw> and that currently wont work
<JayF> ah, idk neutron things :)
<harmw> :p
<JayF> I just know they were talking about that they could spin down the v6 working group b/c they finished support
<harmw> hmk
<harmw> well with a flt network there isn't anything special about it, so theyre probably right:p
<JayF> yeah I hate that OnMetal doesn't do v6 right now
<JayF> the only thing stopping us is our vendor supporting some of the security things we do in v6
<harmw> I hate my ISP for not running v6 :p
<harlowja> so much hate
<harmw> hate,yea!
<smoser> fwiw, i'd try to get it working with just kvm. before i touched libvirt.
<smoser> pretty much i stay away from interacting with libvirt and xml by hand.
<harmw> qemu-kvm -bla -anotherbla disk.image
<harmw> BAM
<harmw> just like that
<pquerna> 'm happy to do $1.6k contracting for doing now oork :P
<pquerna> no work.
#cloud-init 2014-09-26
<smoser> harmw, or hiren_ were you able to test lp:~smoser/cloud-init/freebsd-configdrive 
<greenmoon55> hi, anyone here? why do we have update-hostname plugin in addition to set-hostname?
<harmw> smoser: ofcourse, but tomorrow
<hiren_> smoser: busy with bash :/ Will try to get back to his soon.
#cloud-init 2014-09-27
<harmw> smoser: I've tried that branch, doesn't work just yet
<harmw> Ive got some preliminary debug results attached to the LP mergerequest
#cloud-init 2014-09-28
<harmw> smoser: more in-depth debug notes 
<harmw> https://code.launchpad.net/~smoser/cloud-init/freebsd-configdrive/+merge/235512
#cloud-init 2015-09-21
<ashishjain> hello
<ashishjain>  a newbie here .. trying to use cloud-init with libvirt
<ashishjain> however data injection is not happening :(
<ashishjain> cloud-init.log gives some errors like "Failed at attempted import of DataSourceNone"
<ashishjain> No local datasource found etc
<ashishjain> I am using cloud-init 0.7.5
<ashishjain> Kindly help.
<ashishjain> hello...
<ashishjain> any samaritan here  who can help me
<arkin> ashishjain: what with?
<arkin> Ah no, thats outside of my expertise
<ashishjain> arkin: cloud-init with libvirt
<ashishjain> cloud-init with kvm
<smoser> ashishjain, what is it you're wanting ?
<smoser> explain how yu're try;ing to do it.
<ashishjain> @smoser: Data injection does not work for me
<smoser> and look in cloud-init.log for WARN
<ashishjain> sure
<ashishjain> Here is what I do
<ashishjain> I have got a base ubuntu image say base.img and a iso say init.iso.
<ashishjain> this init.iso has been created using genisoimage using user-data and meta-data files
<ashishjain> Now I have tried using virt-manager as well as virt-install to create a vm, while my VM is created fine but it does not pick any data from init.iso
<ashishjain> I am not sure if thei is some field which is not being correctly passed
<ashishjain> I will paste the command which I use
<ashishjain> sudo virt-install -r 1024   -n test   --vcpus=1  --import --disk vol=test-pool/coe-lab-002-Backup.img,format=vmdk,bus=virtio --boot hd --disk vol=test-pool/init.iso,bus=virtio
<ashishjain> here test-pool is the custom pool created by me
<ashishjain> .img file is of type vmdk
<smoser> ashishjain, its probably the type of data you're putting on it.
<smoser> i suggest you use cloud-localds
<smoser> (or look at what it does)
<smoser> the type of the disk should not matter, but the label on the volume must be 'cidata'
<smoser> http://bazaar.launchpad.net/~cloud-utils-dev/cloud-utils/trunk/view/head:/bin/cloud-localds
<ashishjain> @smoser: This is the command I use to generate the iso which has got cidata
<ashishjain> genisoimage -output init.iso -volid cidata -joliet -rock user-data meta-data
<ashishjain> @smoser: How to use cloud-localds?
<ashishjain> @smoser: Is their a binary which can be invked
<smoser> apt-get install cloud-image-utils
<smoser> cloud-localds (see the help there)
 * smoser has to reboto
<ashishjain> ${0##*/} my-seed.img my-user-data my-meta-data
<ashishjain> ya I could see here
<ashishjain> what is "${0##*/}"?
<smoser> ashishjain, basename $0
<smoser> if you have ubuntu, just apt-get install it and use it.
<ashishjain> @smoser: Yes I got ubunutu I will  try it out and let you know.
<ashishjain> @smoser: It does not help I still have the same issue
<ashishjain> kvm -net nic -net user,hostfwd=tcp::2222-:22 -drive file=coe-lab-002-Backup.img,if=virtio -drive file=init.iso,if=virtio
<ashishjain> @smoser: I used the following command
<ashishjain> cloud-localds init.iso user-data meta-data
<smoser> hm.
<smoser> and you're booting a ubuntu image ?
<ashishjain> yes ubuntu trusty
<ashishjain> can it be a cloud-init bug
<ashishjain> http://paste.openstack.org/show/472810/
<ashishjain> this is my user-data and meta-data
<ashishjain> some messages in my cloud-init.log are as follows:
<ashishjain> Failed at attempted import of "DataSourceNone" due to: No module named DataSourceNone
<ashishjain> No local datasource found
<ashishjain> one bug which I found just now https://bugs.launchpad.net/cloud-init/+bug/1356855
<smoser> ashishjain, can you log in and get /var/log/cloud-init.log ?
<smoser> you can "backdoor" the image with 'backdoor-iamge' easily so you'll be able to log in
<smoser> https://code.launchpad.net/~smoser/+junk/backdoor-image
<ashishjain> https://code.launchpad.net/~smoser/+junk/backdoor-image
<ashishjain> aah google search helps a lot
<smoser> right. basically it just ads a user and a password , makeing sure you can log in
<ashishjain> I am able to log-in
<ashishjain> because the base image has got a username and password
<ashishjain> @smoser: I am able to login but no idea how can copy the log file :(
<ashishjain> the login is possible only through virt-manager console or kvm UI
<ashishjain> but none of these interfaces are allowing me to copy the log file
<smoser> sudo apt-get install pastebinit
<smoser> pastebinit /var/log/cloud-init.log
<smoser> if you have networking then you can scp it out too.
<smoser> or, you can use 'mount-image-callback'' and mount the image and get it that way.
<smoser> sudo mount-image-callback /path/to/that/image.img -- sh -c 'cat $MOUNTPOINT/var/log/cloud-init.log'
<ashishjain> @smoser: here is the complete log http://paste.openstack.org/show/472903/
<smoser> ashishjain, i think you have some config that tells it only to look for datasource none
<ashishjain> @smoser, here is one  bug https://bugs.launchpad.net/cloud-init/+bug/1356855
<ashishjain> which says cloud-init always looks for some particular datasource
<ashishjain> @smoser: Any ideas where  this configuration can be?
<smoser> $ cat /etc/cloud/cloud.cfg.d/90_dpkg.cfg
<smoser> # to update this file, run dpkg-reconfigure cloud-init
<smoser> datasource_list: [ NoCloud, ConfigDrive, OpenNebula, Azure, AltCloud, OVF, MAAS, GCE, OpenStack, CloudSigma, SmartOS, Ec2, CloudStack, None ]
<smoser> that is what you'd normally see, and you can configure it by (as it says) running dpkg-reconfigure
<smoser> i suspect somewhere in /etc/cloud/cloud.cfg.d or /etc/cloud/cloud.cfg you have 'datasource_list' defined
<ashishjain> In 90_dpkg.cfg I have only one entry " datasource_list: [ None ]"
<ashishjain> @smoser: I configure it to have all the entries I will try it now
<ashishjain> thanks for this pointer
<smoser> for this purpose you only need NoCloud . fwiw. and you shoudl always enable 'None' as fallback.
<ashishjain> @smoser: voila !!
<ashishjain> @smoser: It worked.
<ashishjain> @smoser: thanks a lot for your time and help
<smoser> no problem.
<ashishjain> @smoser: One more question is their a python api to automate the creation of disk for cloud-init...same as what is done by genisoimage or cloud-localds
<smoser> there is not. i'd just use cloud-localds and subprocess.
<arkin> Hi all, I've just started getting this error on my providerâ __init__.py[WARNING]: Unhandled non-multipart (text/x-not-multipart) userdata: 'hostname: skasu2\nfqdn: s...'
<arkin> its preventing the server from being setup, can someone help me get past it ??
<ashishjain> @smoser: Thanks, are their any other alternatives to inject data into a VM..something whcih is done by openstack by hosting on a 169.254.169.254
<smoser> arkin, '#cloud-config' at top.
<smoser> its ignoring your data by design
<smoser> you need to tell it it is for cloud-init.
<smoser> ashishjain, https://gist.github.com/smoser/1278651/
<arkin> smoser: I have thatâ https://www.dropbox.com/s/08nduxrxsh0dwia/Screenshot%202015-09-21%2017.23.22.png?dl=0
<smoser> arkin, that would seem like it shoudl be ok
<arkin> It was working, I thought it was something about my host updating cloud-init :/
<arkin> maybe it had to be changed, I guess not
<ashishjain> @smoser: Thanks for that link. I will look into it.
<arkin> smoser: I'll debug whether #cloud-config gets passed as I do replacements on the {{ alias }} etc
<smoser> arkin, yeah, i wonder if your templter is stripping that as a comment
<arkin> smoser: You are spot on actually, I recently added that
<arkin> doh
<arkin> smoser: Trimming the config files before uploading them ;)
<smoser> arkin, you can view what cloud-init saw inside by:
<smoser>  at /var/lib/cloud/instance/user-data.txt
<arkin> smoser: perfect, thanks
<arkin> smoser: don't suppose you are any good at regex ? :D
<smoser> i can try quickly
<smoser> but only quickly :)
<arkin> smoser: I have no issues coming up with ^\s*#.+$ to strip comments
<arkin> I want to add an exception for cloud-config using a negative lookahead
<smoser> do you have a build-your-own template engine ?
<smoser> rather than cheetah or jina or something
<smoser> jinja
<arkin> smoser: Yeah only needs to be super simple
<smoser> i dont know.
<smoser> if you know that you're always producing cloud-config,
<smoser> one thing that you coudl do is just let the renderer render
<smoser> and then add the cluod-config
<smoser> and then you could also verify that the renderer produced valid yaml (which is oftne non-obvious0
<smoser> by just verifying: yaml.load(that_rendered_text)
<arkin> smoser: good suggestion actually, thanks. I'm new to yaml
<arkin> smoser: and thanks for your help/advice
<sputnik13> hah, sweet
<sputnik13> just thought I'd try and see if cloud-init had a channel, of course it does :-D
<harlowja> sputnik13 whats uppp
<harlowja> lol
<harlowja> i know u!
<harlowja> ha
<harlowja> harlowja is everywhere!
<sputnik13> yes, yes you are
<sputnik13> when do you start the harlowja for president of the universe campaign
<sputnik13> :)
<harlowja> oh, i already won that
<sputnik13> cloud-init doesn't support lvm resizing right?
<harlowja> i don't think so,
<sputnik13> disk-image-builder has a -l option to set up the root volumes on lvm
<harlowja> https://github.com/stackforge/cloud-init/blob/0.7.x/cloudinit/config/cc_resizefs.py#L78 resize stuffs/module
<harlowja> i may or may not have touched that code at some point
<sputnik13> but such images fail to resize at boot for the obvious reason that cloud-init don't handle it :)
<harlowja> damn cloud-init
<harlowja> lol
<harlowja> smoser might know more about all this though, if he's around
<smoser> i know all about all
<smoser> cloud-init does not due lvm resizing
<smoser> why do you build images with lvm ?
<harlowja> sputnik13 why did u do such a thing????!!!?
<smoser> (im not opposed to supporting that)
<harlowja> Odd_Bloke smoser sputnik13 is a taskflow-core, so can also be bugged about taskflow
<harlowja> :-P
<harlowja> we are all about cross-pollination, ha
<sputnik13> uh oh
 * sputnik13 goes in to hiding
<sputnik13> smoser: mine is not to ask why, mine is to do or die...  in other words I was told make lvm work :)
<sputnik13> for the time being we're probably going to use some custom scripts or whatever at boot time via userdata, but I was holding out hope that there's some hidden feature in cloud-init that solves all my problems
<sputnik13> and makes me coffee and a corned beef hash with poached eggs for breakfast
<sputnik13> looks like I'm not getting any of those :(
<smoser> sputnik13, i think have to add that support to growpart
<smoser> cloud-initm ight be trying to run resize2fs correctly
<smoser> but if nothing made the root volume any larger, then that doesn't do anything.
<smoser> and the thing that does that on non-lvm is growpart
<sputnik13> ic
<harlowja> corned beef module isn't merged yet, lol
<sputnik13> can it make me corned beef hash too?
<harlowja> sure
<sputnik13> doh, beat me to the punch :)
<sputnik13> so growpart normally does parted or something similar yes?
<sputnik13> for lvm support, the partition probably needs to be resized, then pvresize, vgresize, and lvresize
<sputnik13> or some combination thereof
<smoser> growpart uses sfdisk
<smoser> so yeah, growpart would have to recognize that this was an lvm volume and use those tools
<sputnik13> hmm, so I would expect that at least the partition resize to still go OK, but growpart fails with the following... http://paste.openstack.org/show/473343/
<sputnik13> ahhh, need to run partprobe to re-read partition table but growpart is failing and reverting because sfdisk is returning an error
<sputnik13> orrrr, not
 * sputnik13 is confused
<sputnik13> errr orrr yes...  yes, it is failing and reverting
<harlowja> hmmm, that seeems bad
<harlowja> lol
<harlowja> don't do that
<sputnik13> I think what I want is for it to do a partprobe, re-read the part table, and if it's not what it should be then attempt a revert
<sputnik13> maybe?
<sputnik13> :)
<sputnik13> it's not that the update actually failed, sfdisk isn't doing a partprobe for you, so it returns an error saying you need to do it
<sputnik13> I think
<sputnik13> too many "i think"s here for my comfort
<harlowja> ya
<harlowja> exactly
<harlowja> lol
<harlowja> :-/
#cloud-init 2015-09-22
 * sputnik13 is confused
<sputnik13> is cloud-init maintained in bzr or git?
<smoser> 0.7 is in bzr, but harlowja wants to move it to git.
<smoser> 2.0 is git
<sputnik13> yay git :)
 * sputnik13 is lazy and doesn't want to learn another tool if he can help it
<sputnik13> :)
<sputnik13> woot, taskflow in cloud-init?
<smoser> thats proposed
<sputnik13> taskflow all the things :)
<sputnik13> hmm, no growpart in 2.0
<harlowja> sputnik13 2.0 is a WIP :-P
<harlowja> smoser is 0.7 in git yet :-P
<smoser> sputnik13, no much in 2.0
<harlowja> but sputnik13  u are gonna make 2.0 more super!
<harlowja> i just know it
<harlowja> i believe
<harlowja> lol
<harlowja> sputnik13 talking to a coworker internally, i heard that IPA may be doing/getting LVM support
<harlowja> *ironic IPA
<harlowja> although i guess u are thinking of LVM outside of ironic
<sputnik13> ipa?
<sputnik13> indian pale ale?
<sputnik13> pale ales are not my thing
<sputnik13> weird, I'm getting errors about unable to find resizer for 'growpart'
<sputnik13> when specifying 'growpart' as the 'mode'
<harlowja> lol
<harlowja> https://wiki.openstack.org/wiki/Ironic-python-agent
<harlowja> that thing i think
<sputnik13> using cloud-init 0.7.6~bzr976 on debian
<sputnik13> http://paste.openstack.org/show/473637/
<sputnik13> ^ is in /etc/cloud/cloud.cfg.d/00_growpart
<sputnik13> and getting this in cloud-init.log
<sputnik13> http://paste.openstack.org/show/473638/
<harlowja> sputnik13 i suppose https://github.com/stackforge/cloud-init/blob/0.7.x/cloudinit/config/cc_growpart.py#L49 ?
<harlowja> i guess https://github.com/stackforge/cloud-init/blob/0.7.x/cloudinit/config/cc_growpart.py#L84 broken?
<harlowja> sputnik13 ^
<harlowja> seems to be searching for r"--update\s+"
<harlowja> in the output of --help
<harlowja> not sure why
<sputnik13> that is most odd
<sputnik13> smoser any clues?
<sputnik13> re ^
<sputnik13> harlowja smoser what do you think about extending growpart to handle resizing lvm logical volumes as well?
<sputnik13> or should it actually be a separate module that handles lvm?
<harlowja> unsure, either is fine with me, not sure what is better
<sputnik13> well it definitely shouldn't be resizefs, that part I'm pretty sure of
<sputnik13> :)
<sputnik13> I kind of like having it as a separate thing so that conceptually you have growpart that deals with "physical" partitions
<sputnik13> physical partitions in virtual disks, heh
<sputnik13> :)
<sputnik13> and some lvm handler thing that handles logical volumes
<sputnik13> and resizefs is still concerned only with filesystems
<sputnik13> keeps the three independent
<harlowja> sounds super to me
<harlowja> :-P
<sputnik13> hah
<sputnik13> ugh
 * sputnik13 is confuzd
<sputnik13> I swear it was loading my /etc/cloud/cloud.cfg.d/00_growpart before when growpart had no --upgrade in the help output
<sputnik13> but now that growpart has --upgrade in the help output (I hacked it in) it's acting like 00_growpart isn't there
<harlowja> :-/
<harlowja> not running the module?
#cloud-init 2015-09-23
<smoser> sputnik13, i'm not opposed to having it extended to support lvm
<sputnik13> smoser do you think it's better to extend growpart vs having a separate module for lvm?
<smoser> i think it is most  useful if a signle executable can "do the right thing" personally.
<smoser> with minimal depends.
<harlowja> would it just be called 'part' instead of 'growpart'?
<smoser> right now going forward, growpart works for gpt or mbr with just a dependency on sfdisk
<harlowja> or better name
<smoser> which is like everyone
<smoser> and i'd like "growpart --any"
<smoser> or something
<harlowja> miraclegrowpart
<smoser> it is kind of silly that you ahve to tell it which partition it needs to grow.
<smoser> it looks and could quite easily know which partitions have space between them
<harlowja> growpart --miracle
<harlowja> lol
<smoser> i have to run. sputnik13 i can look tomorrow at what is going wrong ther.
<sputnik13> k, ttyl
<sputnik13> growpart --magic
<harlowja> growpart --ifeellucky
<natorious> I'd noticed some issues w/ growpart and raid recently.  Wheres the issue tracker for that project?
<smoser> natorious, cloud-utils
<smoser> https://bugs.launchpad.net/cloud-utils/+filebug
<natorious> aw sweet, thnx
<natorious> https://bugs.launchpad.net/cloud-utils/+bug/1498930
<natorious> who uses raid in the cloud lol
<smoser> hm.
<smoser> i'd think you  might have this same problem without growpart
<smoser> if the md device was on an un-partitioned disk
<smoser> and then you grew that disk
<smoser> you'd still have "lost" that metadata.
<smoser> right?
<natorious> so if your to grow the raid volume say md126 or whatnot, it doesn't corrupt it
<natorious> but if you grow say sda1 which is in raid 1 or 0 ex on md126, it would
<smoser> right.
<smoser> but if sda was not partitioned
<smoser> and you "grew" that "physical volume"
<smoser> in the host.
<smoser> so there were zeros at the end
<smoser> you'd then have busted that md device
<smoser> right? ie, you cant just add zeros to a disk that has been used as a md
<natorious> the physical volume or the raid volume
<smoser> "physical"
<smoser> /dev/vda
<smoser> ie, its common in cloud/virt to just make a disk bigger.
<smoser> shutdown. add zeros at the end. start up.
<smoser> that would bust a md that was on an un-partitioned disk.
<smoser> we're supposed to be having the cloud-inti 2.0 dev interlock right now.
<natorious> if the disk isn't part of a raid volume w/ a filesystem, it shouldn't be an issue
<smoser> so lets go do that right now.
<natorious> if there is a filesystem and its raid 1, that same fs would show on sda
<natorious> not raid 0 though
<natorious> or most the others
<natorious> k
<smoser> k.  i think i'm right, but just not explaining well,. but maybe im' wrong.
<natorious> np
<smoser> http://bit.ly/cloudinit-reviews-public
<smoser> cloud-inti 2.0 meeting time.
<smoser> lets run through those really quick, and then smoser will actually look at them.
<smoser> and maybe have some other work today on cloud-init.
<smoser> ok. i just started reading claudiopopa's https://review.openstack.org/#/c/220095/
<smoser> i will post some thoughts on it after this meeting.
<smoser> then we have the task flow thing (which is much larger a python2.6 discussion really as Odd_Bloke is now a taskflow fanboy)
<smoser> natorious, i'll take a quick look at your mp and have anyone else here interested do the sam.e
<natorious> thnx
<Odd_Bloke> o/
<Odd_Bloke> I don't have anything new at the moment, I'm afraid.
<Odd_Bloke> I'm heads down on some stuff that needs to happen for the next Ubuntu release.
<Odd_Bloke> smoser: The Debian maintainer for taskflow has it building for Python 3 now, so that's another blocker gone.
<Odd_Bloke> Well, a sync/merge away.
<smoser> are we delta'd on it ?
 * smoser swears at sillyness of <python>-<libraryname> as the source package name.
<smoser> https://launchpad.net/ubuntu/+source/python-taskflow
<smoser> we do have a delta.
<Odd_Bloke> That looks like a fairly trivial delta.
<smoser> ok. i'm goign to read over claudiopopa's review above, and will be around.
<smoser> right, easy enough sync, i was just wondering if it was *no* sync
<smoser> if its just a sync, i'm not oppsed to filing a bug and doing that in wily
<smoser> above, the sillyness of python-libraryname is made abundently clear when <libraryname> drops python support and only works on python3.  meaning that the source packagek "python-libraryname" then only has binaries of "python3-libraryname".
<smoser> anwyay... </rant>
<natorious> the pip2dist stuff looks cool.  Would you need to follow cross platform dep changes though?
<smoser> natorious, cross platform dep changes ?
<smoser> oh. i think by nto allowing them :)
<smoser> ie, if we're supporting a given OS, then stuff has to run in that. or it cant be accepted.
<natorious> ah, nice catch on the misspelling btw.  I should remember to double check everythings when working late on weekends
<smoser> i never make typos
<smoser> (do not read 2 lines above where i spelled 'not' wrong)
 * natorious chuckles
<natorious> there was another bit with that buffer regarding behavior change that was glanced over in review
<natorious> so as previously written the base's meta_data.json returned a buffer obj
<natorious> and w/ the refactor bc it always has json suffix, it would be processed and returned as dict
<natorious> and so http ds wouldn't be able to return the buffer attr from that
<natorious> not sure if the dict or buffer would be preferred w/ something explicitly json
<smoser> i think we want to rid of '.buffer' early rather than late.
<natorious> k, cool.  Was trying not to change behavior for existing and had noticed that after the fact
<smoser> so this is sort of wheere 0.7.x has userdata_raw and userdata (and vendordata_raw too)
<smoser> its handled differently there, with  user-data and vendorddata , but maybe we want these unified.
<natorious> vendordata and meta_data makes sense.  Userdata can be anything though right?
<smoser> the difference being that cloud-init 0.7.x datasource puts in the vendordata the stuff that is meant for it.
<smoser> versus *everything*
<smoser> ie, in that json dict, it will only take out 'cloud-init' stuff  if there is such a thing there.
<smoser> you are correct, yeah.
<smoser> userdata is a binary blob
<smoser> but user-data does have a similar behavior to vendor-data
<natorious> so buffering for http would still be sufficient or were you saying axe it?
<smoser> in that cloud-init might find some things intended for it
<smoser> and some things not.
<smoser> and kind of this is inconsistent in 0.7 . the way it handles.
<smoser> i know i'm being confusing . sorry.
<natorious> more so than usual but ok
<smoser> :)
<smoser> let me think.
<natorious> the buffer was really the part that needed to go for the configdrive module to inherit from base
<natorious> that was the primary reason for this
<natorious> the refactoring and adding network json support in base would be needed for my test cases
<natorious> to get it working with my openstack envs etc
<smoser> so i dont think it makes sense for the user of userdata() to do .buffer
<smoser> that seems silly.
<natorious> I was thinking that was a req for http ds
<natorious> but maybe no
<sputnik13> alo alo alo
<harlowja> smoser did i hear https://launchpad.net/ubuntu/+source/python-taskflow
<harlowja> lol
<harlowja> ding ding
<harlowja> lol
<harlowja> smoser hey, soooo about turning off bzr,,,,,,
<harlowja> sputnik13 i think was wondering where to submit a 0.7.x patch
<harlowja> annnnnddddd
<smoser> harlowja, can i have 45 minutes and ping me back?
<harlowja> 44.9 minutes
<harlowja> lol
<smoser> 44.6
<smoser> ahh.. will that clock ever stop ticking!
<harlowja> :-P
<sputnik13> :)
<sputnik13> ohhhh, taskflow packaged in to ubuntu...  harlowja's baby is moving up in the world :-D
<harlowja> lol
<harlowja> sputnik13 our baby
<harlowja> haha
<harlowja> errr
<sputnik13> I'm just the step parent that's helping out and trying to win its affection, it was already birthed by the time I came around
<sputnik13> :-D
<sputnik13> this is spinning off in a weird direction, I'm going to stop now
<sputnik13> :)
<harlowja> lol
<harlowja> def
<harlowja> ubuntulog2 stop logging all this
<harlowja> tiems up
<harlowja> lol
<gster> hi all, I am hitting this old bug here https://bugzilla.redhat.com/show_bug.cgi?id=836269 on cloud-init v 0.7.5 release 10.el7.centos
<gster> do you know any workaround e.g set systemd timeout maybe?
<harlowja> smoser come back
<harlowja> lol
<smoser> almost back.
<smoser> stupid security
<smoser> harlowja, ok.
<smoser> focus-on-git time.
<harlowja> lol
<smoser> what do you have
<harlowja> i have whatever u need man
<harlowja> lol
<harlowja> https://code.launchpad.net/~harlowja/cloud-init/cloud-init-git-tarball-make might help
<harlowja> but i guess we can further fix things that pop up
<smoser> ok. let me take alook
<harlowja> and i thinks https://github.com/harlowja/cloud-init/tree/0.7.x-fixed-final is the latest resync from bzr
<harlowja> layered ontop of https://github.com/stackforge/cloud-init/tree/0.7.x
<gster> harlowja: is this about the systemd timeout problem I  metionned earlier?
<harlowja> gster nope, for that one not sure who is the best to respond (not me)
<harlowja> some redhat person?
<smoser> harlowja,
<smoser> $ python setup.py --version
<smoser> 0.7.7
<smoser> (sorry to ignore you gster, but dont have time right now to look)
<harlowja> smoser ?
<smoser> it always shows that.
<smoser> but i need a REVPOSTFIX
<harlowja> no postfix for u
<harlowja> lol
<harlowja> smoser so integration of pbr will let u have a REVPOSTFIX because it makes one
<harlowja> but for 0.7.x maybe in git-mode REVPOSTFIX can just be an env variable ?
<gster> No problem smoser. I also found it in here https://github.com/sdake/heat-jeos/issues/1 on Fedora.. Quite a old bug though (2012)...
<smoser> harlowja, ok. so some symantic differences between what you're suggesting and what i've done before.
<harlowja> possibly
<harlowja> from my understanding alot of openstack stuff has the following
<smoser> right. reading from http://docs.openstack.org/developer/pbr/semver.html
<smoser> which is what pbr says it does
<harlowja> https://github.com/openstack/nova/blob/master/nova/version.py#L21
<harlowja> others also have a 'NOVA_PACKAGE = None # OS distro package version suffix'
<harlowja> or equivalent
<harlowja> right, so semver + a prefix that is plugged into that (or equivalent) file
<harlowja> https://github.com/openstack/taskflow/blob/master/taskflow/version.py
<harlowja> and others
<smoser> so i think theres 2 things i'm struggling with
<smoser> a.) cloudinit, i would behvae like:
<smoser>  release 0.7.X
<smoser>  h... call taht : release 0.7.2
<smoser>  then "open 0.7.3"
<harlowja> right
<harlowja> we can keep more of the status quo without pbr for that 0.7.x branch and tags (0.7.2)
<smoser>  so at this point, python setup.py --version shows '0.7.3'
<harlowja> right
<smoser>  but then i'd do a REVPOSTFIX with a '~bzrXXX'' where XXX was the trunk
<smoser> so that '~' means to dpkg "this is *less* than 0.7.3"
<smoser> but on 2.0 branch right now:
<smoser> $ python setup.py --version
<smoser> 1.9.0.dev158
<smoser> where:
<harlowja> ya, the 2.0 branch is all weird right now with respect to version
<harlowja> pbr is trying to calculate that, not sure why it picked 1.9
<smoser> $ dpkg --compare-versions 1.0.0.dev1 gt 1.0.0 && echo dev is greater than tag || echo dev is less than tag
<smoser> dev is greater than tag
<smoser> well, it picked 1.9 because its in setup.cfg
<harlowja> ah
<harlowja> k
<smoser> so the issue rigth now is that if i were to use the pbr symantics exactly as the are, my released versions would be versioned smaller than my development versions leading up to the release.
<harlowja> right
<harlowja> i think we just need to make sure pbr has its deb version command exposed
<harlowja> like https://review.openstack.org/#/c/196191/
<harlowja> pretty sure it knows how to generate a deb version correctly
<harlowja> just might need to be exposed as --deb-version command or something
<harlowja> if not, we need to bug pbr authors
<harlowja> https://github.com/openstack-dev/pbr/blob/master/pbr/version.py#L231 exists
<harlowja> likely just needs to be expoed
<harlowja> *just like the rpm one that i exposed
<harlowja> lifeless is also trying/working on getting pbr adopted by pip, so it will probably be everywhere soon
<harlowja> * https://review.openstack.org/#/c/219468/
<harlowja> so with 'debian_string(self):' exposed, i think that will do the right sorting for u
<harlowja> i can propose that smoser
<harlowja> *pretty much same as 196191
<smoser> all it does is ~ right ?
<smoser> in my reading ?
<harlowja> seems like it
<harlowja> https://github.com/openstack-dev/pbr/blob/master/pbr/version.py#L311
<harlowja> if its busted and/or not working, can just tweak pbr
<smoser> k. i can work somethign together.
<harlowja> okie
<smoser> so then i wonder..
<smoser> how does it count "commits since last tag" or whatevre it does to get 'devX'
<harlowja> ya, that logic i'm not sure of
<smoser> i just dont know how to figure which tag to go from
<harlowja> https://github.com/openstack-dev/pbr/blob/master/pbr/packaging.py#L636
<harlowja> https://github.com/openstack-dev/pbr/blob/master/pbr/packaging.py#L516
<harlowja> but some pbr folks can probably better describe that (vs reading code)
<smoser> well thats magic
<smoser> git describe
<harlowja> magic might need better docs, not sure
<harlowja> other option, not use PBR, just increment the numbers ourself
<smoser> i'm ok to use pbr on 2.0 and maybe use 'git-describe' which seems mostly functional on 0.7.X
<harlowja> k
<smoser> from git-describe:
<harlowja> git describe
<harlowja> 0.7.6-443-g23d4282
<harlowja> interesting
<smoser> Otherwise, it suffixes the tag name with the number of additional commits on top of the tagged object and he abbreviated object name of the most recent commit.
<smoser> right
<harlowja> thats https://github.com/harlowja/cloud-init/tree/0.7.x-fixed-final
<smoser> but what is that g23d ?
<harlowja> commit 23d4282b1fd8aee9ecf372435b10185dfb59731d
<smoser> as 'git show' doesnt show it. how can i turn that into something useful if i was curious
<harlowja> last commit
<harlowja> g(commit)
<smoser> a. g
<smoser> right.
<smoser> duh.
<harlowja> maybe leave off g23d4282
<harlowja> :-/
<smoser> it doesnt hurt.
<harlowja> k
<smoser> her. i'll get something .thanks for patience. let me poke
<harlowja> cool beans
<smoser> we're even lucky
<harlowja> ??
<harlowja> we're always lucky
<harlowja> lol
<smoser> oh shoot. its not.
<smoser> i thought the 443 was lucky enough to be > trunk's revno
<harlowja> its ssl port
<harlowja> lol
<smoser> right
<smoser> as a tribute to sabdfl
<harlowja> distance to 0.7.6?
<harlowja> but 443 changes seems like alot
<smoser> well, my thought was that i could just grab that (0.7.6-443) and turn it into 0.7.7~dev443
<smoser> $ git log --format=oneline 0.7.6.. | wc -l
<smoser> 443
<harlowja> ya
<harlowja> so i guess thats how it got that
<harlowja> but u want it since commits for all time right?
<smoser> well, that would work . but this is easy enough.
<harlowja> ya
<harlowja> tell it to use first commit
<harlowja> *tell git describe
<smoser> so here is plan
<smoser> i'll git a shell snippet
<harlowja> after/before https://github.com/harlowja/cloud-init/tree/0.7.x-fixed-final is resynced -> stackforge?
<harlowja> *or other repo that openstack-infra needs to sync
<harlowja> to replace https://github.com/stackforge/cloud-init/tree/0.7.x
<harlowja> ^ which is missing commits...
<smoser> let me work out what iw ant to do
<smoser> and then you can help me. k ?
<harlowja> k
<harlowja> sure boss
<harlowja> sputnik13 get er all done
<harlowja> thx
<harlowja> lol
<smoser> harlowja, http://paste.ubuntu.com/12534794/
<smoser> i think i'm happy with that for those tools.
<smoser> and seems to retain bzr support also.
<smoser> harlowja, http://paste.ubuntu.com/12535144/
<smoser> i like that better.
<smoser> that is on top of your lp:~harlowja/cloud-init/cloud-init-git-tarball-make
<smoser> at http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/wily/cloud-init/wily/view/head:/debian/README.source#L43
<smoser> the 'improt a new snapshot' i'll have to adjust to work with git.
<smoser> but the tools/read-version --dev will be useful.
<harlowja> smoser cool, works for me
<harlowja> i'm all good with whatever u are good with :-P
<bdx> smoser: how's it goin?
<bdx> smoser: does cloud-init using nocloud provider preform any /etc/ssh/sshd_config modifications by default in trusty cloudimg?
<sputnik13> sooooo...  where do I submit patches bzr or git? :)
<sputnik13> smoser harlowja ^
<harlowja> ummmm
<sputnik13> I know your answer already
<sputnik13> :)
<harlowja> :-P
#cloud-init 2015-09-24
<smoser> bdx, no.
<smoser> SpamapS, we're getting there.
<SpamapS> :)
<SpamapS> smoser: o/
<smoser> bah
<smoser> stupid tab complete
<smoser> sorry SpamapS
<smoser> ok... it wasn't tab complete's fault
<smoser> stupid smoser
<Odd_Bloke> smoser: If you could give https://code.launchpad.net/~daniel-thewatkins/cloud-init/shim_fixes/+merge/269199 a quick review, it'd be much appreciated.
<smoser> Odd_Bloke, but then harlowja will get all fussy about it being bzr
<smoser> he whines a lot
<smoser> Odd_Bloke, it seems maybe the part find_endpoint after 'No endpoint found in DHCP config.' could/should be its own method . and then some tests on it.
<smoser> i'll put that in review. and one other comment.
<smoser> anyone know how to disable the stupid gist editor and get me a nice plain text window in a gist ?
<smoser> so i can use view-source-with or the like to use vi ?
<Odd_Bloke> smoser: gist's can be git clone'd, which might be the easiest way to do what you want.
<smoser> yeah, and i do that sometimes.
<smoser> but sometimes i like to use their editor
<smoser> as its just easy
<smoser> and i'm particularly lazy
<smoser> but now that stupid editor
<harlowja> smoser Odd_Bloke what is that bzr thing, lol
<sputnik13> bzr... is it like a flea market?
<harlowja> sputnik13 i think so
#cloud-init 2015-09-25
<arkin> It isn't immediately clear from my google searches, but how do I add a public key to my "root" user
<arkin> Any suggestions on the command failingâ   - [mysql, -e, "GRANT ALL PRIVILEGES ON `{{ mysql_db }}`.`*` TO '{{ mysql_user }}'@'%'"]
<arkin> I think its the * causing problems
<arkin> I've tried escaping it with \* and the problem persists...
<crobertsrh> I'm looking at adding swap to a vm (either fedora or ubuntu).  I'm currently trying to do it by using fallocate and mkswap in bootcmd, but that seems to be failing.  It leaves me with a 2GB disk (before launching, I used qemu-img resize to make a much larger disk) and no active swap (it did create a file, but it was much smaller than I tried in fallocate, probably because there wasn't enough disk).  Anyone have any tips?
<smoser> crobertsrh, you could have cloud-init to it for you
<crobertsrh> smoser:  How would I do that?  (Sorry, slightly new to cloud-init)
<crobertsrh> The docs regarding this didn't make much sense to me.
<smoser> crobertsrh, that is because it is not documented *at all*. :-(
<smoser> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/1022
<smoser> other than that commit.
<smoser> theres an example config there.
<smoser> if growpart is groing your disk, then when bootcmd runs, it wont have run yet.
<smoser> so fallocate might correctly be telling you no (becase at that point there *is* not any space)
<smoser> but there might be .2 seconds later
<crobertsrh> smoser:  Thanks.  I think I got kicked toward the end of your messages.  I got things up to "but there might be .2 seconds later"
<crobertsrh> Will the "swap:...." bit work with 0.7.5?  Or should I try to make images with a newer cloud-init version?
<smoser> crobertsrh, probably not. :-(
<smoser> it wont.
<crobertsrh> That's what I was gathering.  I did try that at one point...which lead me to believe that I was just unable to understand the documentation, but then I got looking into versions.  Thanks for your help.
<smoser> ok. so i think the reason it is not working for you is this:
<smoser>  * you use qemu-img to resize a disk from 2G to 10G
<smoser>  b.) you boot system.
<smoser>  c.) cloud-init runs your bootcommand with does 'fallocate 2G'.
<smoser>    that fails, because there is no space
<smoser>  d.) cloud-init (.2 seconds later or thereabouts, in the 'resizefs' module) grows the partition and runs resize2fs
<smoser>  e.) crobertsrh goes in and says "why does the same thing i ran in 'c' work now, but it didnt then!"
<smoser> s/*/a./
<crobertsrh> Makes sense.  I think my workaround will do me fine.  I'm doing fallocate, etc in runcmd, which seems to get me what I need for now.
<crobertsrh> I realize it's not ideal, but I think it will be "good enough for now".  Thanks again for the help and explanation.
<smoser> it should be good enough
<jtheuer> Can I add a custom cloud-init module? (I want it to run after cc_ssh.py)
<smoser> you can. you have to add it to the list though
<jtheuer> so I would have to add to /etc/cloud/cloud.cfg, correct?
<smoser> right.
<smoser> and add the 'cc_xxx.py' in the right place
<jtheuer> yeah, that would be my next question ;-)
<smoser> dpkg-query --show cloud-init | grep cc_
<smoser> put it in that directory. where you see the .py
<arkin> smoser: tried your recommendation [mysql, -e, "GRANT ALL PRIVILEGES ON `{{ mysql_db }}`.`*` TO '{{ mysql_user }}'@'%'"] but im still getting warnings about the * even when I put \* any recommendations?
<jtheuer> btw: I want to sign the generated ssh host keys (obviously between key generation and sshd start) -- ideas or hints are more than welcome!
<smoser> arkin, i'm not sure what you mean. its possible you have other yaml rendering un-friendly characters there.
<smoser> and *yaml* is screwing you before shell gets the chance (although you've saved yourself from shell by the array)
<smoser> i'd yaml.load() your config and look at what the result is
<smoser> to make sure the array looks like what you think it should
<arkin> ok cheers
<arkin> with the array do I need to esacpe * ?
<smatzek> if the sql syntax and parsing becomes unbareable, could you try writing the command to a file using the write-files module where you could base64 encode the command / file contents, and then use runcmd to run "mysql < /tmp/commandfile" ?  Though who knows how yaml and the shell will work with that stdin redirect.
<smoser> arkin, you'll only have to escape if its a problem to yaml
<smoser> arkin, run http://paste.ubuntu.com/12557719/
<smoser> and see if the arry there is what you think you would want
<smoser> if you your self correctly escaped all that on  shell command line.
<crobertsrh> Is there a way I can re-run the cloud-init bits from the command line?  I've tried sudo /usr/bin/cloud-init -d modules, but I'm not seeing the things I did in write_files get re-done (they did get done at boot time, but the ownership of a file was not correct.  I'm trying to correct that).
#cloud-init 2015-09-26
<arkin> ok smoser thanks
#cloud-init 2015-09-27
<natorious> are we grouping the different distro's in 2.0?  As in osys.redhat would cover centos and fedora and osys.debian would cover ubuntu?
#cloud-init 2016-09-26
<smoser> harlowja_at_home, pitti (ubuntu systemd maintainer) is complaining about your Before=NetworkManager.service
<pitti> well, not "complaining", but it should be a no-op
<pitti> all network-y things like NM, networkd, ifupdown etc. already run After=network-pre.target -- that's the raison d'Ãªtre for that target
<pitti> so I was wondering if there is some race condition somewhere and that just hides a real bug
<harlowja_at_home> nrezinor1, ^
<harlowja_at_home> x58, ^
<harlowja_at_home> apparently thats not how it works in RHEL
<harlowja_at_home> lol
<harlowja_at_home> those 2 folks are the experts here
<harlowja_at_home> i'm just a poor small enginner man
<harlowja_at_home> lol
<harlowja_at_home> bbiab
<harlowja> ok backs
<nrezinor1> i see my name was pinged lol , wat now
<harlowja> nrezinorn something about the network manager stuff
<harlowja> nrezinorn  https://irclogs.ubuntu.com/2016/09/26/%23cloud-init.html#t16:28
<nrezinorn> yeah i dont remember most of that haha
<nrezinorn> that was so last week
<harlowja> :-P
<nrezinorn> the tl;dr is "MOAR TESTING"
<nrezinorn> i dont have time to test things today lol   stupid "real work"
<x58> smoser: RHEL requires Before=NetworkManager.service because otherwise NetworkManager starts at the same time as cloud-init-local, and takes over managing /etc/resolv.conf for instance.
<x58> smoser: I filed bugs for these issues, that describe what is going on.
<x58> RHEL doesn't have NetworkManger.service require network-pre or something that allows us to order cloud-init-local first.
<x58> pitti: Yeah, it would be nice if network-pre.target would actually be defined that way on RHEL/CentOS too... but alas it is not.
<x58> pitti: Also, networking.service is network.service on RHEL, so there are just difference between Ubuntu/RHEL.
<x58> Either split out the systemd per system, or template it someway, but those are things that need to be accounted for.
<harlowja> agreed
<harlowja> pitti if we need to split out thats fine
<harlowja> if there is a RHEL bug somewhere, that'd be nice to know also
<smoser> x58, so pitt was saying (i think) that upstream NetworkManager's service file does have network-pre.target
<harlowja> in RHEL?
<harlowja> or ubuntu
<harlowja> ?
<smoser> upstream
<harlowja> upstream!!!
<harlowja> :-P
<harlowja> which stream
<smoser> https://github.com/NetworkManager/NetworkManager/blob/master/data/NetworkManager.service.in
<smoser> that one
<harlowja> x58 ^ any idea, maybe RH just doesn't have that one yet
<pitti> x58: right, forget the particular .service names; the point is that <stuff that fiddles with network interfaces> is After=network-pre.target, and <stuff that sets up firewall or config> runs before that
<pitti> x58: and neither network-pre.target nor NetworkManager.service are an Ubuntu invention -- these both come from their upstream projects
<pitti> x58, harlowja: so, it's plausible that RHEL's NetworkManager.service does not (yet) have After=network-pre.target, and then that workaround would actually DTRT; I just wanted to confirm that this is actually the case
<pitti> because, if NM.service already does have After=network-pre.target, then that change is a no-op, and something is still not understood
<apollo13> are you guessing? cause I can look into centos7 if you want
<pitti> I am just guessing, yes (Debian/Ubuntu guy here)
<apollo13> https://dpaste.de/UKgv/raw that is the current file on centos/rhel 7
<pitti> ah, so that indeed still misses it
<apollo13> anything else I should look at?
<pitti> so that explains it (even though fixing the actual NM.service would be preferrable, so this ought to be a downstream patch)
<pitti> but nevermind, I mostly just wanted to know if that was maybe just a misunderstanding
<apollo13> great
<apollo13> anyways, your After came in with nm 1.2, centos has 1.0.6
<apollo13> https://bugzilla.gnome.org/show_bug.cgi?id=761001 was the original issue
<apollo13> although according to that bugreport it would be on 1-0 too? seems that redhat didn't cherrypick that (yet)
<x58> Yes it should probably be fixed by RHEL/CentOS, but that will probably take till version 193943834342112 ...
<x58> (being overly generous with that version number, knowing how slow RHEL moves)
#cloud-init 2016-09-27
<smoser> x58, thanks!
#cloud-init 2016-09-28
<dynek_> hey
<dynek> anyone using cloud-init to setup redhat atomic host? I'm don't understand where DNS settings are stored on disk when I mention them in network-interfaces
<dynek> and on first boot /etc/resolv.conf is not populated
<dynek> and finally, it's not possible to use power_state to reboot the machine :/
<smoser> rharper, around ?
<smoser> looking at https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+ref/nocloud-boottime-improvements seems like a general caching mechanism might be best
<rharper> here
<rharper> yeah, I started looking at a memoize decorator
<rharper> but those typically require arguments as it uses a dictionary to cache previously seen invocations
<rharper> the blkid output parsing  is different than typicall caching of the result since it just opens and reads instead of execing blkid again;
<smoser> well, blkid unless told otherwise does the same, does it not?
<rharper> no
<rharper> it refreshes data and re-writes tab each run AFAICT
<rharper> even if it does read it; the exec of blkid vs. open/read  is non-trivial
<smoser> maybe. but i think that might be over optimization.
<rharper> *shrug*; those items were tracked as the highest cost
<rharper> during nocloud image boots, and subsequent boots
<rharper> I didn't pick them out of the hat
<smoser> blkid as non-root here takes 2 thousandths of a second as (0.002)
<smoser> and as root in a container 0m0.003s
<smoser> as root not in container it does take almost 0.03 and i can see that is expensive.
<rharper> I'm almost done with preparing a post with the tools
<smoser> but its kind of non-avoidable.
<rharper> so, we can reproduce the data
<smoser> well, i suggest that generally just being able to cache a 'subp' call
<smoser> would make many things easier. and would lose performance-wise to your blkid specific solution by a very small margin
<rharper> the memoize  decorator could do that
<rharper> I suspect though that there may be some commands where we do expect different answers given the same call
<rharper> in which case a general caching of subp is not going to be correct
<rharper> I think I'd rather continue to drive changes based on profiling
<smoser> right.
<smoser> so a 'cachable' option on subp defaulting to False
<smoser> and then call subp(['dpkg', '--print-architecture'], cachable=True)
<smoser> means "use the cache or fill it"
<rharper> as you say for the blkid, it won't help as much;  we make 4 different calls
<rharper> I think a cachable subp could produce additional improvements; but I would posit that we keep the blkid tab parsing in-addition
<smoser> there is a blkid method in curtin
<smoser> which i originall did some thing s like this.
<smoser> but the idea woudl easily have a cache added.
<rharper> lsblk ?
<smoser> no. block.blkid()
<smoser> it basically ujust ensures that no cache is read adn the retuns result as dict
<rharper> right;  so the other challenge with blkid specifically is that the parameters are mostly just formatting/filtering output;  but for the use of blkid in cloud-init now; they really just want the filtering part (TYPE=X , path=Y) and to return the device name (if it's found)
<rharper> I had a stab at adding 'read from cache' to the current find_devs_with() function but it seemed like lots of extra work for no gain since no part of cloud-init was using things like -sUUID to return the UUID value of the match, vs. just the devname
<smoser> right. i'd just change all callers to just use a 'blkid()' method and operate on thee dict response.
<rharper> yeah
<rharper> I forgot the dict part;
<rharper> and a wrapper could exist to just match and return the device name like users of find_devs_with expect
<smoser> yeah.
<smoser> just noticed that 'devs' in blkid in curtin does ntohing.
<rharper> yah pylint
<rharper> we did *something* with it
<rharper> I expect that was a devs filter
<smoser> rharper, revno 221 made it do nothing
<smoser> i swear udev is supposd to update some blkid cache.
<smoser> or run blkid so that it updates the cache
<rharper> it doesn;t look like it was used even with change; ie, I don't see any callers passing devs in
<nacc> there are some blkid rules in /lib/udev, but nothing is sticking out to me
<nacc> it seems like mostly for optical?
<nacc> oh for md devices too
<smoser> rharper, http://paste.ubuntu.com/23247253/
<smoser> that is a cachable_subp that seems like it shoudl generally work
<rharper> so, my first though is that my current patches are more focused and have fewer possible side-effects;  it would be nice to see them land and then have more time with cachable subp and blkid (with cache) after? or hold off until each of those can land?
<smoser> well, i dont  like specifically caching _IS_CONTAINER when a much more generic solution works.
<rharper> surely using generalized subp cache has a wider impact;  it's not clear to me that it's safe to cache all callers of subp at this time
<rharper> where as a specific caching of _is_container is straightforward and safe
<smoser> of course its not safe to cache all callers of subp
<smoser> change the callers that *are* safe to use cachable_subp
<rharper> ok
<smoser> i'm also kind of worried about the the blkid /dev/sr0 and blkid /dev/sr1 specifically
<smoser> as you can blame them... they are basically there to kick that device
<smoser>  812f82e7b3bad3f8127face552c76ef974b54661
<rharper> that's also for 2.6
<rharper> at least the comment says
<smoser> right
<smoser> rharper, comments on bug 1628337
<rharper> just the module order should be needed?
<rharper> your pastebin had another change in util
<smoser> oh. yeah, just the module order
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/307055
<harlowja> ok rharper smoser i updated https://code.launchpad.net/~harlowja/cloud-init/+git/cloud-init/+merge/301729
<harlowja> with the tiny change there
<harlowja> i still would just like 1 simple logging location :-P
<smoser> harlowja, i agree with that.
 * harlowja just likes the simple things
<harlowja> lol
 * smoser doesn't even know how messages get to rsyslog anymore
<smoser> $ ls -l /dev/log
<smoser> lrwxrwxrwx 1 root root 28 Aug 31 12:12 /dev/log -> /run/systemd/journal/dev-log
<smoser> this is odd
<harlowja> lol
<rharper> harlowja: thanks;  I agree with smoser;  however, I do find the change in format between logging and rsyslog in the *same* file cloud-init.log to be jarring;
<rharper> if cloud-init wanted to join with syslog, I'd at least like it to have a confirming message format
<rharper> I don't yet know how to ensure we get consistent formatting between the two;  if we fallback to syslog, then it's a per-image/distro config mechanism; ie, what the default is for rsyslog in the image; unless we can have cloud-init emit specific format that matches the syslog format
<harlowja> meh, i like files
<harlowja> files good
<harlowja> lol
<smoser> harlowja, https://code.launchpad.net/~harlowja/cloud-init/+git/cloud-init/+merge/305882
<smoser> ...
<smoser> would be nice
<harlowja> oh
<harlowja> hey
<harlowja> i know u
<harlowja> yes, one sec
<harlowja> reworking cloud.cfg -> cloud.cfg.ubuntu and cloud.cfg.fedora
<smoser> :)
 * smoser runs now
<harlowja> lol
<harlowja> smoser nrezinorn https://code.launchpad.net/~harlowja/cloud-init/+git/cloud-init/+merge/307100
<harlowja> that also fixes why brpm wasn't working
<harlowja> (or part of it)
<harlowja> part uno
<harlowja> of part 300
<harlowja> lol
<harlowja> nrezinorn i'm going to take a stab at destroying brpm
<harlowja> though i know u love it
<harlowja> lol
<harlowja> smoser https://code.launchpad.net/~harlowja/cloud-init/+git/cloud-init/+ref/sys-io-errors updated
<harlowja> in return can u fix the following
<harlowja> https://gist.github.com/harlowja/4f996d19b633862eb3a647d0c6053ac9
<harlowja> i shall include 1 goat
<harlowja> ^ things that aren't working on rhel
#cloud-init 2016-09-29
<smoser> harlowja, the errors... you really have no 'ip' command ?
<smoser> you need to fix that nonsense.
<smoser> silly paths on redhat
<smoser> harlowja, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/307200
<smoser> that fixes some of those gist failures i think
<smoser> harlowja, ping
<smoser> magicalChicken, around ?
<harlowja> smoser sorry, got up late and just got in :-P
<harlowja> us cali people, lol
<harlowja> smoser so there is an ip command
<harlowja> but something maybe just isn't mocked right
<smoser> hm.
<smoser> can you help me get a cent6 and cent7 environment ?
<smoser> would like to know how to do that.
<magicalChicken> smoser: i have a work in progress of my branch that adds that
<smoser> magicalChicken, embarrasing question
<smoser> i was looking at your doc branch
<smoser> how do i build doc
<smoser> https://public.etherpad-mozilla.org/p/cloud-init-centos-unittest
<magicalChicken> Oh haha, its 'sphinx-build docs/rtd /tmp/docs'
<magicalChicken> Oh, I don't have a setup for lxd based centos environment yet, just got curtin vmtests. I can look into that though tonight
<smoser> magicalChicken, you want to add : http://paste.ubuntu.com/23252743/
<magicalChicken> Sure, I'll get that added in
<magicalChicken> Right now we're using the default template too, not the one for read the docs, I can add that into the repo too, but I think when we build the docs there it will use it by default
<magicalChicken> I've gotta relocate to go get my laptop, I'm on a lab machine right now. I'll get that merged in in 15
<harlowja> virtualbox :-P
<harlowja> smoser let me run your branch on cent7
<harlowja> see what happens
<smoser> harlowja, https://public.etherpad-mozilla.org/p/cloud-init-centos-unittest
<smoser> teach me
<smoser> ssh ubuntu@45.55.168.77
<harlowja> $ git clone -b open-nebula-no-ip https://git.launchpad.net/~smoser/cloud-init cloud-test
<harlowja> Cloning into 'cloud-test'...
<harlowja> remote: Counting objects: 22313, done.
<harlowja> remote: Compressing objects: 100% (7809/7809), done.
<harlowja> Receiving objects:  28% (6292/22313), 2.47 MiB | 12.00 KiB/s
<harlowja> smoser  are the tubes broken over there, lol
<harlowja> down to 8kb/s
<harlowja> lol
<harlowja> restarted, seems better
<harlowja> maybe a tube was broken somewhere
<magicalChicken> smoser: I just got the tox env for docs added in and pushed to the repo
<smoser> magicalChicken, ok. thanks.
<magicalChicken> I noticed though, in tox its trying to highlight everything as python
<magicalChicken> it wasn't doing that outside of tox
<magicalChicken> I need to figure out why real quick
<magicalChicken> i think its py2 sphinx vs py3 sphinx
<smoser> hm.
<smoser> ok.
<smoser> got to run now
<magicalChicken> I'll get that fixed and pushed tonight
<harlowja> smoser  https://gist.github.com/harlowja/62315edad713304076f35ff40ac9c4f1
<harlowja> so 1 fail i think still
<harlowja> *that's using https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/307200
<harlowja> using/running
<harlowja> one more mocking fix probably addresses that
#cloud-init 2016-09-30
<smoser> harlowja, well, i fixed the ones i fixed
<harlowja> lol
<smoser> the 2 i fixed with ip
<harlowja> thx :)
<smoser> they are i'm pretty sure either
<smoser> a.) you have no output of 'ip'
<smoser> or (i think)
<smoser> b.) you have not 'ip' in your path
<smoser> because redhat typically does noth ave /sbin in path
<smoser> but either way
<smoser> the next one i think is just that have dns hijacking going on on that host
<harlowja> could be
<smoser> see is_resolvable
<smoser> it tries to work around that stuff
<smoser> but apparently not enough there.
<smoser> so...
<smoser> help me get a cent6 / cent7 container
<harlowja> i only know VMs
<harlowja> lol
<smoser> well, i have a container
<harlowja> VM
<smoser> you just get it so things work
<harlowja> :)
<harlowja> agreed
<harlowja> will get that in a few, templating cloud.cfg
<smoser> well https://public.etherpad-mozilla.org/p/cloud-init-centos-unittest
<smoser> right now a bunch of stuff fails for me
<harlowja> kk
<harlowja> paage still loading
<harlowja> lol
<smoser> i have a node up 45.55.168.77
<smoser> that you can go to ubuntu@
<harlowja> k
<smoser> and lxc is there...
<harlowja> cubswin?
<harlowja> lol
<smoser> ssh keys.
<harlowja> k
<smoser> 101
<smoser> is cubs win total
<smoser> kind of like lol
<harlowja> haha
<harlowja> close enough
<smoser> do you use python-mock from the distro ?
<smoser> i think its too old.
<harlowja> i usually run tox which then installs all the things
<harlowja> so nothing from distro
<smoser> hm.
<smoser> well, i couldnt run tox
<smoser> it didn't work either.
<harlowja> hmmm, did u install tox ?
<harlowja> how much did work?
<smoser> from rpm ?
<smoser> er.. yum
<harlowja> or from pip
<smoser> http://paste.ubuntu.com/23253475/
<harlowja> ah, yes, pip install setuptools --upgrade also
<harlowja> stupid old busted
<harlowja> likely also pip install virtualenv --upgrade to
<harlowja> i believe virtualenv has the bundled version of setuptools that tox then uses
<harlowja> but i forget
<harlowja>  pip install setuptools tox virtualenv --upgrade
<smoser> still issues
<smoser> checking that.
<harlowja> k
<harlowja> and keep on saying to yourself 'old stuff is more stable'
<harlowja> lol
<smoser> so essentially if this works, i'm basically running in tox
<harlowja> ya
<smoser> which...
<smoser> protects me from most things
<harlowja> *for testing
<smoser> i'd like to run nosetests with the versions of things that are in the distro
<harlowja> probably need to cap mock version then
<harlowja> which is prob 1.0.1 on epel
<harlowja> *epel 7
<harlowja> and 0.8 on epel 6 :-/
<harlowja> don't forget keep on saying to yourself 'old stuff is more stable'
<harlowja> lol
<smoser> well, i guess i dont care too much about the mock version
<harlowja> ya, as long as it works, which it should
<smoser> right. i'm fine to have newer of that, but if we're using a library we want to ouse the version in the distro.
<smoser> i like that:
<smoser>  yum install valid-package invalid-package valid-package2
<harlowja> yum is sorta retarded...
<smoser> goes and installs everything and then somewhere in the log it says 'invalid-package not available'
<smoser> rather than jsut saying "um... can't do that"
<harlowja> ya, i think the exit code is also 0 for that case
<harlowja> from what i remember
<harlowja> lol
<harlowja> so u can't detect it in bash scripts
<harlowja> (at least not via exit codes)
<harlowja> hopefully dnf fixes that
<smoser> FAILED (SKIP=28, errors=30, failures=1)
<harlowja> nice, more than with tox it seems, lol
<smoser> so thats in my cent6 after pip install and then tox
<smoser> some recently regressed. :-(
<smoser> TypeError: decode() takes no keyword arguments
<harlowja> ah, ya, that one
<harlowja> didn't i fix that
<harlowja> i forget
<harlowja> lol
<smoser> how do you fix that ?
<harlowja> cent6 and py26 are awesome
<harlowja> i think u can just not give a keyword argument :-P
<harlowja> and just do it positional argument only
<harlowja> ya, just positional should be fine
<harlowja> S.decode([encoding[,errors]]) -> object
<harlowja> prob could do encoding=encoding in py27 only or something
<smoser> oh. right.
<harlowja> 'old stuff is more stable'
<smoser> ok. with taht change, now
<harlowja> lol
<smoser> FAILED (SKIP=28, errors=3)
<harlowja> cool
 * powersj thinks we should probably get CI unit tests going again as they fail right now
<harlowja> whats the last 3
<harlowja> 'old stuff is more stable' 'old stuff is more stable' 'old stuff is more stable'
<harlowja> lol
<smoser> http://paste.ubuntu.com/23253529/
<smoser> k. first is easy enough (content)
<harlowja> ya, the rest are dict comphrenhsions
<smoser> set comprehension on one of them :)
<harlowja> oh ya
<harlowja> ya, just turn those into set(iterable) and dict(iterable)
<harlowja> and that's all those become
<harlowja> so not so bad
<smoser> http://paste.ubuntu.com/23253540/
<harlowja> cool u can even do
<harlowja> set(m['uri'] for m in f['apt']['primary'])
<harlowja> if u really care
<harlowja> either will be fine
<harlowja> no need for intermediary list
<smoser> oh.
<smoser> how is that
<smoser> oh. ididnt know you coudl
<smoser> i like that
<harlowja> :-p
<harlowja> m['uri'] for m in f['apt']['primary'] is a generator
<smoser> yeah, but only if wrapped in parens
<smoser> $ python -c 'm for m in (1,2,3)'
<smoser>   File "<string>", line 1
<smoser>     m for m in (1,2,3)
<smoser>         ^
<smoser> SyntaxError: invalid syntax
<smoser> $ python -c '(m for m in (1,2,3))'
<smoser> happy
<harlowja> right
<harlowja> so u put it in parans
<harlowja> ha
<harlowja> set( )
<harlowja> lol
<smoser> it is kinda wierd like that
<smoser> that the parens to the function call suffice
<smoser> that kind of hurts my brain
<harlowja> ya, its probably a weirdness in the python syntax that allows for it
<smoser> how should i say "what version of centos am i on"
<smoser> ie, i want to know 6 or 6
<smoser> or 7
<smoser> alright. good.
<harlowja>  in /etc/redhat-release i think
<harlowja> or lsb_release i think has something
<harlowja> or python has some stuff u can use
<smoser> https://public.etherpad-mozilla.org/p/cloud-init-centos-unittest
<smoser> so... given that patch above, i can run tox in cent6 or cent7
<smoser> thats good, but ideally we'd be able to run 'nosetests' against distro-installed versions (as would be found in a runtime)
<harlowja> ya, that may require a little more version tweaking
<smoser> and the 'build' thing mostly worked... at least used to.
<smoser> so, tahts good. thanks harlowja
<harlowja> ya, there is a file or 2 that is missing for brpm
<harlowja> but i'm hoping with nrezinorn that brpm goes away
<smoser> you seem my comments https://code.launchpad.net/~harlowja/cloud-init/+git/cloud-init/+merge/305882
<smoser> i have to run
<harlowja> ya
<smoser> thanks for your help jxharlow
<harlowja> whos that
<harlowja> lol
<smoser> you changed your middle name when you moved to godaddy
<harlowja> JXMenHarlow
<smoser> is that because harlowja.coolguy was taken?
<smoser> but you could still get harlowjx
<harlowja> nah, i asked and they just said, meh that's what u got
<harlowja> i asked about harlowja, and it seemed like alot of work
<harlowja> so i just gave up
<harlowja> lol
<harlowja> https://code.launchpad.net/~harlowja/cloud-init/+git/cloud-init/+ref/kill-brpm (where brpm goes away)
<harlowja> it seems to work on cent7 at least, ha
<harlowja> nrezinorn just wants a spec file, lol
<smoser> but i like brpm
 * harlowja runs away
<harlowja> lol
<smoser> as long as some way we can make an rpm
<harlowja> ya
 * smoser out
<smoser> later
<smoser> harlowja, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/307333
<smoser> what do you think about that ?
<avsh> Hi, I am using vmware with openstack
<avsh> Virtaul CD-Rom which used for config drive is not removed after Machine is provisioned
<avsh> Leaving admintrative password and other data as clear text in ( DVD-DRIVE config-2 )
<avsh> what configuration is need for cloud init to remove the drive when vm is provisioned?
<avsh> can anyone please help me with my query?
<smoser> avsh, do you have non-root users who can mount a cdrom ?
<avsh> yes
<smoser> to my knowledge, cloud-init couldn't on its own rid the system of that drive. you could 'eject /dev/cdrom'
<smoser> and then it would not be there, but possibly a 'eject -t /dev/cdrom' would pull back in the tray and have it again
<smoser> and if it is Read-only media, then cloud-init can't write it to blank it
<smoser> can you give the file that has the password in it ? it'll help me diagnose where to look to see what esle could be done.
<avsh> ec2/
<avsh> openstack/2012-08-10/
<avsh> openstack/2013-04-04/
<avsh> openstack/2013-10-17/
<avsh> openstack/content
<avsh> openstack/latest/meta_data.json
<avsh> openstack/latest/user_data
<avsh> openstack/latest/vendor_data.json
<avsh> This is the Folder Structure
<avsh> and openstack/latest/meta_data.json has contents like { "admin_pass": password, "random_seed": "*******" }
<smoser> oh. well, good for you that cloud-init doesn't care what is in there
<avsh> smoser, let me know if you need any info
<smoser> it ignores it
<smoser> as on linux, ssh keys are preferred.
<avsh> but users can see other sensitive information which is a security concern
<avsh> like if i install a software on the vm, that software password is an example
<smoser> avsh, do you have a reason to let users mount that disk ?
<smoser> why not just remove users from the 'cdrom' group
<avsh> we can do that. we don't see this issue with Openstack + KVM
<avsh> only with Vmware + Openstack
<avsh> trying to understand, it is configuration with cloud-init or vmare nova driver
<smoser> just because you're not looking in the right place :)
<smoser> i suspect the same data is availble in http://169.254.169.254/openstack/
<smoser> and any malicious user already knew that.
<avsh> ok
<smoser> cloud-init can route off the that particular address so that only root would be able to get at it
<smoser> with
<smoser>  disable_ec2_metadata: true
<avsh> ok, let me check on kvm instance with the url you provided
<avsh> smoser, you made my day
<avsh> you are correct, I am not seeing at right place with kvm + openstack
<avsh> It can access all the data with the above url
<avsh> I can rule out vmware opensstack nova drive
<smoser> avsh, alternatively you can do things in a different way.
<smoser> you can use '#include-once' to include other cloud-config things... and make those expiring or one-time-read urls
<smoser> the metadata services are not intended to be secure
<avsh> smoser, I will check on the #include-once, thanks
<smoser> rharper, around ?
<smoser> i've 2 things for you... one. do you have a readthedocs.org account (or can you get one).
<smoser> 2.  http://paste.ubuntu.com/23256482/
<rharper> smoser: here
<rharper> I have a rtd account
<rharper> lemme get on (2)
<rharper> what am I looking at with (2) ?
<smoser> what is rtd account ?
<smoser> i will share access to cloud-init project
<rharper> ah, I see , a slow ish boot; total time is 11 seconds though
<rharper> read-the-docs
<rharper> ah, right
<smoser> rharper, yeah....
<smoser> um more interesting than that.
<rharper> raharper
<smoser> that was a 2+ minute boot
<smoser> :)
<rharper> 17:32:37 to 17:32:56
<rharper> it's not cloud-init log
<smoser> yeah.
<rharper> that's like 20 seconds wallclock
<smoser> thats what is fun
<rharper> so something else (look at systemd-analyze blame
<rharper> can I haz ssh to ami ?
<smoser> i think clock is moving backwards
<rharper> oh, ntp!
<smoser> you can... yeah
<smoser> let me set access up for you through my bastion
<smoser> i assume its reproducible on serverstack
<smoser> but
<rharper> ok
<rharper> dmi data /sys/class/dmi/id/product_name returned OpenStack Nova  -- that log is not from AMI on EC2 ..
<smoser> right
<rharper> if it's kernel related, it's possible that it could be reproduced in sstack; however, given that the virt layer is going to handle memory differently (booting xen on ec2, vs kvm on openstack) that may mean we won't reproduce the same amount of slowdown;  the kernel but that's reference has to do with SLAB/SLUB config and other changes
<smoser> rharper, ssh-via proxy-user@10.245.162.60 ubuntu@10.5.0.185
<smoser> first ip is my bastion. i set you up to jump through there.
<smoser> second is the system.
<smoser> ssh-via is http://smoser.brickies.net/git/?p=tildabin.git;a=blob;f=ssh-via;
<rharper> blob_plain is what I want
<rharper> smoser: in
<rharper> Sep 30 17:31:33 ubuntu systemd[1]: Time has been changed
<rharper> something *did* reset the clock
<rharper> Sep 30 17:31:33 ubuntu systemd-timesyncd[556]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).
<rharper> Sep 30 17:30:42 - prior event
<rharper> Sep 30 17:31:33 - time has changed
<rharper> so, slow clock moved forward by just under a minute
<rharper> and then, ntp syncs it and moves it another minute forward, Sep 30 17:32:23
<rharper> brb
<smoser> rharper, its wierd though that cloud-init's logging didn't see that.
<smoser> rsyslog must be doing it ? and it keeping its own clock or something?
<rharper> smoser: it's in journctl
<rharper> the time change happened async from cloud-init execution
<rharper> not quite sure how journctl keeps track of time vs. python logging/rsyslog
<rharper> smoser: note the odd delta between the entry timestamp (Sep 30 17:32:37, vs the timestamp collected for the welcome message: at Fri, 30 Sep 2016 17:30:39 +0000)
<smoser> yeah. its wierd.
<smoser> and systemd is confused by this
<smoser> as it *does* say cloud-init took 2 minutes to run
<smoser> when i'm pretty sure watching a wall clock that is not the case.
<rharper> correct
<rharper> I was going to do the relative time between events in ci and I'm positive it didn't take all that time (rather we've got a clock jump)
<rharper> I suspect if you uncloud-init data , reboot
<rharper> it won't be as a long
<rharper> I wonder why the VM clock is so far off though
<rharper> 2 minute adjustment is pretty large
<rharper> you launched me at: Fri, 30 Sep 2016 17:29:59 +0000
<rharper> kernel booted     : Fri, 30 Sep 2016 17:30:30 +0000
<harlowja> smoser interesting if u haven't seen it
<harlowja> https://cloud.google.com/compute/docs/containers/vm-image/#using_cloud-init
<smoser> rharper, yeah, i agree.
<rharper> so, I don't know what to do unless timectl could tell us how much time was adjusted
<rharper> it seems like it's a systemd bug too
<smoser> really sucks
<rharper> since analyze didn't update (acknowledge that ntp changed time)
<rharper> if it knows the timedelta, it could apply that
<smoser> in cloud-init we could read uptime
<rharper> to provide true time
<rharper> despite clock shift
<smoser> theres a way to get that i think rather thatn /proc/uptime
<smoser> (althoguh /proc/uptime is mocked in a container for us, and a kernel interface probably isnt)
<rharper> right
<smoser> https://github.com/xmonader/linuxsysinfo/blob/master/sysinfo.py
<rharper> yeah, the procfs is slightly slow in container
<rharper> the cloud-final message is always high on blame due to reading sysinfo
<rharper> it's relatively slow to other modules
<smoser> rharper, well, we could read via sysinfo.
<rharper> syscalls, yeah; but that'd be host data, right ?
<rharper> sorta like dmesg
<smoser> probalby could even read once from /proc/uptime
<rharper> it's just not right
<smoser> and then count that offset
<smoser> :)
<rharper> right, proc/uptime once and caching that would be useful
<smoser> and then form then on out ask the sysinfo
<rharper> but the issue is the 4 invocations of cloud-init
<smoser> well, 4 reads of /proc/uptime is probably "not that bad" in the grand scheme of all the bad things.
<rharper> but from exec to exit, we could do sysinfo for time;  but honestly; I think rdtsc is likely faster for absolute cycles
<rharper> smoser: it's the hottest thing left on lxd reboots
<smoser> well then.
<rharper> it's not bad at all
<smoser> :)
<rharper> we're at 0.25 second reboot
<rharper> but it's still roughly 30% of that in cloud-final message
<rharper> so, if we could reduce that , then we'd see sub .2 second reboots
<rharper> rdtsc would be faster than call to proc due to fuse
<rharper> hrm, the monotonic clock jumps too when time is set; I suppose that's expected... but that sounds wrong
<rharper> smoser: so, can we see if /var/lib/systemd/clock exists in the cloud-image ?
<rharper> ah, it doesn't (at least the lxd rootfs doesn;'t have it)
<smoser> as in is it created at runtime you mean ?
<smoser> versus already present?
<rharper> right
<rharper> timesyncd uses that as a restore clock to this as soon as it starts (if it exists)
<smoser> oh.
<rharper> that could be a jump but it would have been backwards quite a bit
<smoser> restore from what ?
<rharper> each sync with ntp, that file is updated with the last good stamp from ntp
<rharper> https://lists.freedesktop.org/archives/systemd-devel/2015-May/031988.html
<rharper> It
<rharper>   implements sNTP and will sync the last known time to disk every time
<rharper>   it gets an sNTP sync or the system is shut down.
<rharper> At boot it uses
<rharper>   that time to reinitialize the clock, as early as possible, before
<rharper>   NTP is done. THis will give you monotonic time which should solve
<rharper>   your probelm.
<rharper> so, the journal, and other loggers on the system use CLOCK_MONOTONIC, which is susceptable to NTP changes in clock (always forward)
<rharper> so, AFAICT, this is just a really out of sync clock on the host
<smoser> which *is* susceptable ?
<smoser> or is not succeptable
<rharper> is
<rharper> CLOCK_MONOTONIC_RAW is not
<smoser> hm.
<smoser> harlowja, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/307363
<smoser> powersj, that gives us a way (via the gist linked there) to run uni tests in centos6 pretty easily
<smoser> rharper, raharper is now a maintainer of CloudInit in readthedocs
<smoser> and fyi, magicalChicken's doc fixes are there now!
<smoser> (it had stopped building when we moved from bzr)
<rharper> smoser: cool, i see it
 * smoser has to run
<smoser> harlowja, if you could look at that... you can merge it if you want
<smoser> it seems nice.
<harlowja> kk
<harlowja> cools
<smoser> later.
<powersj> smoser, very cool
<harlowja> woah
<harlowja> nice nice
<harlowja> all the modules got filled in???
<harlowja> sweet
<harlowja> that only took a couple years to finish, lol
<harlowja> thx guys! :)
<jgrimm> magicalChicken, ^^
<jgrimm> harlowja, smoser: so in the gce link, it mentions that they've implemented setting UID. Any particular reason that hasn't been implemented in base cloud-init up to now besides priorities?
<harlowja> i forget
<harlowja> i thought i remember a patch for that
<harlowja> lol
<jgrimm> bug for it. https://bugs.launchpad.net/cloud-init/+bug/1396362
<jgrimm> yeah, i remembered seeing it in the backlog
<harlowja> unsure about that one
<harlowja> why is it a diff file
<harlowja> lol
<harlowja> did they not sign the CLA
<harlowja> weird
<jgrimm> when i saw it in the gce doc, reminded me that i'd seen it requested before
<harlowja> ya
<powersj> smoser, I setup cloud-init to run unit tests 2x a day now across the architectures. It will git clone master, figure it is better than noting.
<powersj> I can figure something out for the new centos one too when I get back home
<powersj> and since it does not respond to merge requests, but just does master, it will email us on failures, hence the mail you probably just got
#cloud-init 2017-09-25
<blackboxsw> ejb1123: a couple of our unit tests mock metadata services, (tests/unittests/test_datasource/test_ec2.py) but I don't think that's what your asking. you probably want a simple way to setup your own rest service in python? flask might present some simple rest servers you could set up.  https://blog.miguelgrinberg.com/post/designing-a-restful-api-with-python-and-flask
<blackboxsw> then you'd provide the backend that would serve up your metadata
<blackboxsw> ... maybe other folks have ideas there.
<smoser> blackboxsw, i think he was asking for metadata servier
<smoser> https://gist.github.com/smoser/1278651/ is one.
<smoser> that mimics ec2 (incompletely)
<niluje> release 17.1 4 days ago
<niluje> !!! <3 smoser
<smoser> niluje, yeah, horay!
<smoser> https://lists.launchpad.net/cloud-init/msg00106.html
<niluje> can't wait to see it into zesty
<smoser> niluje, you could/should join that low volume mailing list if you're interested in such messages.
<niluje> (into xenial too?)
<smoser> zesty and xenial will get SRU at some point.
<niluje> SRU?
<rharper> stable-release-update
<rharper> it will get pushed into the xenial-updates after some time and testing
<niluje> ok cool :)
<hrybacki> hey all, trying to figure out why some custom networking isn't working via cloud-init. Following the debugging docs: http://cloudinit.readthedocs.io/en/latest/topics/debugging.html I'm seeing an 'cloud-init error: invalid choice: analyze' upon `$ cloud-init analyze show -i <my init file>` -- not sure if the docs are out of date or if my version of cloud-init doesn't support it
<hrybacki> fwiw: cloud-init 0.7.9 on my system
<smoser> hrybacki, no 'analyyze' on .7.9
<hrybacki> smoser: okay. can I trouble you with a networking question? If I want to disable the default cloud-init behavior of setting eth0 to pick up an IP via dhcp, is setting `network: \n config: disabled` enough to do so? Looking at http://cloudinit.readthedocs.io/en/latest/topics/network-config.html?highlight=network%3A%20disabled specifically
<hrybacki> I've created a custom-networking.cfg file and copied it into the image in the /etc/cloud/cloud.cfg.d/ dir via `virt-customize --copy-in ...` prior to launching the VM
<smoser> and the file 'custom-networking.cfg' contains the disable ?
<hrybacki> smoser: aye
<smoser> hrybacki, then that should work.
<hrybacki> smoser: that is what I thought. I must be doing something wrong or have overlooked something else
<smoser> hrybacki, should work with content:
<smoser>  network: {config: disabled
<smoser> er..
<smoser>  network: {config: disabled}
<hrybacki> smoser: okay, networking /is/ coming up as expected but cloud-init is holding up the boot process doing it's lookups. Not sure why this broke but -- the behavior changed in a bump from 7.6 to 7.9
<hrybacki> https://paste.fedoraproject.org/paste/6RXoPjFoxWEGbs63lJp3iw
<hrybacki> 192.168.23.1 is the host for the respective guest in this instance
<smoser> hrybacki, well, a full paste would be good
<smoser> but it sure seems like it is failing to find anything, which ends up falling back to the ec2 datasource which is a real pita
<smoser> and sits there and polls some endpoint that you do not have.
<hrybacki> that is precisely what is happening smoser . I'm just curious why this started happening now and not prior to the version bump
<smoser> hrybacki, need more info. not sure what you were hoping would work. what is the cloud / platform that you're running this on ?
<smoser> hrybacki, ^ ?
<hrybacki> smoser: sorry, in meetings. it's an atypical environment. I'm deploying a supplementary libvirt node on which FreeIPA will be running and interacting with an OpenStack deployment laid down by TripleO
<smoser> so openstack is the platform ?
<hrybacki> the node is being deployed manually prior to OpenStack being deployed
<smoser> where did you want this node to get its data from?
<hrybacki> nowhere on first boot
<smoser> i dont know how that ever would have worked.
<hrybacki> smoser: well I'm not sure how it was working before. I was probably using this wrong (out of ignorance) but now with the new version breaking things I'm not really sure what I'm trying to fix. Does that make any sense?
 * hrybacki slides back into mtg
<smoser> hrybacki, i understand , and yeah, nothing was meant to revert. but the "there is no datasource" path never really was supported. i'm not sure how it worked.
<smoser> powersj, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/331284
<smoser> https://git.launchpad.net/cloud-init/tree/debian/patches/ds-identify-behavior-xenial.patch?h=ubuntu/xenial
<sedition> hi team. stupid question. are all parts of an apt config required? just trying to add a single additional ppa under sources is giving me trouble.
<smoser> sedition, example ?
<rharper> smoser: https://bugs.launchpad.net/cloud-init/+bug/1716773
<ubot5> Ubuntu bug 1716773 in cloud-init "cloud-init doesn't cache network_config property in cache" [Undecided,New]
<sedition> smoser: sure, here's the apt block with the packages above it for good measure. the rest of my script is all about user setup (which works great) https://pastebin.com/e1kyw7Uz
<sedition> forgive my ignorance, i'm super new to cloud-init
<smoser> sedition, needs to have '#cloud-config' at the top
<smoser> cloud-init doesnt assume that all data is for it
<smoser> you have to tell it the type of data.
<sedition> Sorry, I do have #cloud-config at the time. I was just posting a snippet.
<sedition> s/time/top/
<sedition> i'll redact the sensitive data and post the full script
<smoser> sedition, http://paste.ubuntu.com/25616466/
<smoser> try that.
<sedition> heres the whole thing https://pastebin.com/i9wQ0zDW
<sedition> smoser: will do. one moment while i destroy and rebuild
<sedition> smoser: No such luck. No entries under /etc/apt/sources.list.d
<sedition> I've gotta be doing something silly.
<sedition> checking the output logs, im hitting errors adding the repository and hitting ubuntu mirrors (which is weird) so it might not be the script.
<sedition> i'll keep debuggin
<sedition> thanks!
<smoser> sedition, http://paste.ubuntu.com/25616651/
<smoser> that "worked for me"
<sedition> smoser: totally an error with my egress configs. i'm a dummy.
<sedition> Sorry to bother, and thanks so much for the patience and help
<smoser> sedition, No worries.
<smoser> we're well aware that config is hard to "validate"
<smoser> and have been workignon a schema for some of the config modules
<sedition> That will be slick
<powersj> smoser: https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/331167
<powersj> smoser: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1718681
<ubot5> Ubuntu bug 1718681 in cloud-init (Ubuntu Artful) "Package copyright file omits Apache 2 license" [Undecided,In progress]
<sedition> Welp. Everything works. I am really impressed. I just started learning Terraform and figured I might as well do user data scripts correctly with cloud-init.
<sedition> really excellent stuff.
#cloud-init 2017-09-26
<smoser> blackboxsw, https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1717983
<ubot5> Ubuntu bug 1717983 in ubuntu-meta (Ubuntu) "replacement of isc-dhcp-client with with systemd-networkd for dhclient needs integration" [Undecided,Triaged]
<smoser> blackboxsw, https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1718029
<ubot5> Ubuntu bug 1718029 in cloud-init (Ubuntu) "cloudstack and azure datasources broken when using netplan/systemd-networkd" [Critical,Confirmed]
#cloud-init 2017-09-27
<htaccess> hello i am getting some very consistent errors when using cloud init on xenial boxes
<htaccess> i built 15 xenial boxes  that use #cloud-config to set the timezone, local and install one package
<htaccess> 4 have failed to install the package
<htaccess> becuase: E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
<htaccess> relevant /var/log/cloud-init-output.log output : https://paste.ubuntu.com/25624581/
<htaccess> as it say its pretty easy to trigger, i am building 5 servers each time, every time i have done it at least one box has the issue
<htaccess> s/local/locale/
<dpb1> htaccess: we had a race with automatic updates a while back, could be back.  Could you please file a bug?
<kszarlej> How in NoCloud datasource in network-config set the default route?
<kszarlej> It is only possible to set the route as cidr, and I need 'default via'...
<rharper> kszarlej: you want to add a gateway: value to your subnet configuration, http://cloudinit.readthedocs.io/en/latest/topics/network-config-format-v1.html#network-config-v1
<kszarlej> rharper: but I have two subnets.
<kszarlej> So the routes are created like: 10.10.1.0/24 dev eth0 proto kernel scope link src 10.10.1.120
<kszarlej> for each subnet (that one was 10.10.1.0/24). But what about a default(default via 192.168.1.10 dev eth0)
<kszarlej> the gateway adds default route for configured subnet. but when my instance wants to access WAN not LAN it needs default gateway
<kszarlej> ok so the proper way should be adding a static route with destination of '0.0.0.0/0'
<kszarlej> ye but it ofcourse doesnt work :D eh.. cloud-init
<kszarlej> ok - adding route directly to subnet routes: works. Working example: http://paste.ubuntu.com/25628125/
#cloud-init 2017-09-28
<redcavalier> Hi, since the latest update of cloud-init on centos 7, cloud-init doesn't resize my openstack instances anymore
<redcavalier>  /var/log/messages shows me errors regarding text.encode(encoding) showing that "'dict' object has no attribute encode"
<redcavalier> seems to be related to vendor-data, which is empty as I don'T use it
<nacc> redcavalier: presumig cloud-init does come from centos, it seems like you should also report a bug there (and link to it here)
<redcavalier> ok, I'll do so right now.
<blackboxsw> redcavalier: thanks.  if you are using centos 7.4 and our daily cloudinit (17.1) released at https://copr.fedorainfracloud.org/coprs/g/cloud-init/cloud-init-dev/packages/ there is a 'cloud-init collect-logs [--user-data]' command that will tar/gzip a file for your that'd ease analysis
<blackboxsw> you could attach that to the bug (or just follow the bug filing suggestions which basically grabs the same content
<redcavalier> blackboxsw, it's not the daily cloudinit but the centos package though (0.7.9)
<blackboxsw> ahh, roger
<nacc> blackboxsw: yeah i was basing my guess off redcavalier sayig it was from a recent cent update
<nacc> which *might* be a distro issue, i'm not sure how y'all handle the cent releases
<redcavalier> well, I should specify the recent centos cloud-init update. Strangely enough, it doesn't affect our cloudlinux instances
<redcavalier> so it seems really specific
<blackboxsw> yeah, redcavalier I'd be curious  what /var/log/cloud-init.log and  /run/cloud-init/reslts.json  has in it. on the systems just to rule out a datasource ordering issue with some changes that went into the Ec2 datasource which is a fixed bug that affected  openstack specifically in 0.7.9
<blackboxsw> but that'll come with the bug I presume
<redcavalier> blackboxsw, let me make you a pastebin real quick. IT appears to be related to cloud-init trying to write the cache for vendor-data but failing because it can't find an encoding property on the object to write (but I might be wrong, I'm not a developper).
<redcavalier> blackboxsw, https://pastebin.com/t8yyghGM is the log
<blackboxsw> thanks redcavalier gotta head to bed for the evening. Well looks like it still properly identified your Openstack datasource, so not this issue I wondered about w/ ec2
<redcavalier> Alright, thanks
<blackboxsw> it's possible that the vendordata_raw format changed across this update.... umm yeah please file a bug and we'll track it and see if we can get to the bottom of the failure redcavalier
<blackboxsw>  cheers.g'night all.
<sedition> catch you later
<redcavalier> FTR, I've filed a bug with centos. Find it at https://bugs.centos.org/view.php?id=13931
<orf__> hey, is it possible to mount any other files in a system via the vsphere 'user-data.txt' ISO data source?
<orf__> as in, set up some network configuration?
<smoser> orf__, i do not believe you can pass networking configuration in via OVF datasource
<cesarjorge> Hi, I'm need help with cloud-init
<cesarjorge> The cloud-init when start, changes (in a linux), the network definition interfaces: /etc/sysconfig/network-scripts/ifcfg-enp0s3
<cesarjorge> Howto do, to say to cloud-init not change the interfaces?
<larsks> cesarjorge: https://cloudinit.readthedocs.io/en/latest/topics/network-config.html#disabling-network-configuration might help
<cesarjorge> many thanks!!! I will try
#cloud-init 2017-09-29
<nakul> Hi all
<nakul> does cloud-init supports Debian 7 (wheezy) ?
<nakul> specially 32 bit arch
<nakul> i have created an deb 7 (32bit) kvm based image and installed cloud-init cloud-utils and cloud-initramfs-growroot packages from "http://ftp.debian.org/debian wheezy-backports main" repo
<nakul> and uploaded in glance to work with userdata and config drive but its not accepting any credentials either
<nakul> but i can curl to metadata url http://169.254.169.254 and can see the config drive as well
<nakul> i am not sure whats wrong here and i dont have such issues with debian 8 and debian 9 release
<powersj> nakul: and you please share the logs in /var/log/cloud-init*?
<nakul> [....] Starting Cloud service: cloud-init[....] Starting NFS common utilities: statd idmapd[?25l[?1c7[1G[[32m ok [39;49m8[?25h[?0c. [....] Starting rpcbind daemon...[....] Already running.[?25l[?1c7[1G[[32m ok [39;49m8[?25h[?0c. [....] Starting enhanced syslogd: rsyslogd[?25l[?1c7[1G[[32m ok [39;49m8[?25h[?0c. [....] Starting ACPI services...[?25l[?1c7[1G[[32m ok [39;49m8[?25h[?0c. [....] Starting deferred execution scheduler: atd
<nakul> hi powersj
<nakul> i am able to share the full logs here
<nakul> not*
<nakul> Traceback (most recent call last):   File "/usr/bin/cloud-init", line 509, in <module>     sys.exit(main())   File "/usr/bin/cloud-init", line 505, in main     return functor(name, args)   File "/usr/bin/cloud-init", line 256, in main_init     iid = init.instancify()   File "/usr/lib/python2.7/dist-packages/cloudinit/stages.py", line 311, in instancify     return self._reflect_cur_instance()   File "/usr/lib/python2.7/dist-package
<nakul> @powerj : do you have any mail id to share the logs ?
<nakul> i followed this url for image creation : https://thornelabs.net/2014/04/07/create-a-kvm-based-debian-7-openstack-cloud-image.html
<blackboxsw> nakul: if possible to redact the /var/log/cloud-init.log  to something that could be shared it would be helpful to file a bug at https://bugs.launchpad.net/cloud-init/+filebug with any information you can provide
<blackboxsw> so others could benefit from the fix too. we can contact you with more requests if the bug filed doesn't have enough information to understand the problem
<nakul> blackboxsw: ok but i am not sure weather its a bug or some misconfiguration  or something else
<blackboxsw> nakul: that howto is really dated (2014)... maybe this would help https://asciinema.org/a/132009
<blackboxsw> rewriting that cloud.cfg file is potentially dated/broken in that howto you referenced
<nakul> dont think so..still let me try once more else i'll post the issue in +filebug site
<nakul> thanks blackboxsw
<nakul> ohh..thanks Chad Smith
<blackboxsw> no problemo nakul  that'd be smoser to thank
<blackboxsw> :0
<blackboxsw> :)
<utlemming> smoser: around?
<utlemming> https://gist.github.com/utlemming/288607e974f597d89d940474874551df
<utlemming> smoser: looks like SysConfig rendering for network configuration is broken for IPv6 default routs
<utlemming> I can't tell whether its Fedora/RH or Debian/Ubuntu that is not following the spec
<utlemming> since network_data.json is not well documented
<robjo> smoser: Could you please give https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/331149 another look? Would like to get this finished so I can put the patch into the 17.1 SUSE package.
<blackboxsw> will give it a pass robjo
<robjo> blackboxsw: thanks, I just fixed the alpha order issue I hope, as I couldn't reproduce that locally, so the bot will need to do another run
<benray> Hope it's OK to ask a general question here. How to populate ifcfg file with 'DHCP_HOSTNAME=<hostname>' prior to network start? If I use bootcmd it gets overwritten
<rharper> benray: we don't currently have a way to do that;  ifupdown doesn't support DHCP_HOSTNAME, but it doe support hostname <hostname value> which should get passed to the client
<benray> Thanks! I might be able to live with rc.local
<rharper> *ifupdown in ubuntu at least
<benray> RHEL in this case
<benray> I believe ubuntu is a little more elegant in this regard
<rharper> ok; so for the sysconfig backend
<rharper> but generally, it appears there are some client dhcp options we should support (hostname, lease, client, vendor) strings that dhcp clients would submit in their request IIUC
<benray> That would be cool. Thought I would ask in case it was just hiding somewhere
<benray> :)
<rharper> benray: ok, I filed a bug https://bugs.launchpad.net/cloud-init/+bug/1720426
<ubot5> Ubuntu bug 1720426 in cloud-init "network-config should allow dhcp client option values" [Undecided,New]
<benray> Nice! Thanks for looking into it
<rharper> benray: sure, thanks for pointing it out
<benray> Keep up the great work
<rispa> hello. I'm trying to figure out how to disable different cloud-init datasources. Not sure why, but I can't seem to figure out from the docs.
<rispa> Basically, we're using cloud-init on internal vms with a cloud-config drive. When the cloud-config drive isn't there or is misconfigured, cloud-config tries to hit a bunch of metadata urls and eventually times out
<rispa> this really slows down boot time -- takes around 10 minutes
<blackboxsw> rispa:  on your vms you can either dpkg-reconfigure cloud-init  to enable/disable
<blackboxsw> rispa:  on your vms you can either dpkg-reconfigure cloud-init  to enable/disable datasources....
<blackboxsw> or you can edit your /etc/cloud/cloud.cfg.d/90_dpkg.cfg datasource_list
<blackboxsw> and remove any datasources in your images you don't want to try
<blackboxsw> rispa: I'll add a work item for additional docs on this as I can't find them either
<rispa> blackboxsw: datasource_list is working great -- thanks very much!
<smoser> rispa, just fwi, there really should not be a reason to do that.
<smoser> if it is timing out, its due to the ec2 datasource, which should be done last.
<smoser> wonder if you could post a cloud-init.log
<smoser> rispa, oh. yes. if you remove the disk it will do that.
<smoser> you can set in the image "manual_cache_clean: true"
<smoser> and that means that if there is a cached datasource cloud-init will not go looking again
<smoser> so you can boot once with the disk, and then take it away
<smoser> but if you do that then you have to clean yourself, or cloud-init will never go looking again.
<rispa> smoser: thanks! it's definitely only a problem when the cloud config drive is missing or broken. more of annoyance than anything else
#cloud-init 2019-09-24
<otubo> rharper: powersj Good morning guys. I just arrived in Seattle yesterday. I would be free for a coffee or any social event of the kind :)
<smoser> otubo: i arrive tonight
<rharper> otubo: great!  I'm flying in tonight
 * rharper glares at pylint tests/unittests/test_handler/test_handler_growpart.py:134: [E1101(no-member), TestConfig.test_handle_with_no_growpart_entry] Instance of 'AbstractContextManager' has no 'enter_context' member 
<blackboxsw> rharper: I've pushed softlayer SRU for review https://github.com/cloud-init/ubuntu-sru/pull/55
<rharper> ij
<rharper> ok
<rharper> I put up the OpenStack one
<rharper> let me put up GCE too
<blackboxsw> reviewing thx
<rharper> https://github.com/cloud-init/ubuntu-sru/pull/54
<blackboxsw> I've attached ec2, lxd/kvm ,azure results to sru bug
<rharper> https://github.com/cloud-init/ubuntu-sru/pull/56
<rharper> blackboxsw: I've sorted the MAAS QA jobs; bionic running, almost done, will queue the rest today as well
<blackboxsw> thx rharper, I'm only trying to re-activate my oracle account so I can test the upgrade path there too. CDO QA if giving a run on their infrastructure
<blackboxsw> so should be able to check the boxes
<blackboxsw> ok gce and openstack SRU validation merged. I'm attaching them to the SRU bug
<otubo> I got "FAILED: Unit & Style Tests" on my pull reuquest[0], but the jenkins link is timing out [1]. I'm unable to see the problem on the style of code to fix it without the jenkins report
<otubo> [0] https://code.launchpad.net/~otubo/cloud-init/+git/cloud-init/+merge/372981
<otubo> [1] https://jenkins.ubuntu.com/server/job/cloud-init-ci/1166/
<blackboxsw> otubo: checking that now. it's stuck behind the firewall
<blackboxsw> sorry about that. we're trying to move to something public
<otubo> blackboxsw: no problem, thanks for checking that :)
<blackboxsw> otubo: looks like
<blackboxsw> cloudinit/ssh_util.py:226:80: E501 line too long (80 > 79 characters)
<blackboxsw> cloudinit/ssh_util.py:235:80: E501 line too long (82 > 79 characters)
<blackboxsw> tox -e pycodestyle should show that set of errors to you
<blackboxsw> or just generally run 'tox' and it should cycle through most of the tests run by ci
<otubo> blackboxsw: ah! good to know
<blackboxsw> rharper: ok softlayer, gce, openstack all attached the the SRU bug. so we'll work on the remaining MAAS, oracle and cdo qa verification.
<otubo> blackboxsw: still regarding my pull request, on line 70, it means all the keys will be stores on auth_key_fns[0] which should be always %h/.ssh/authorized_keys, but if the configuration is switched like "AuthorizedKeysFile /etc/ssh/userkeys/%u .ssh/authorized_keys" then it breaks.
<otubo> I guess the keys should be always be stored on %h/.ssh/authorized_keys, right?
<otubo> nah, never mind, just ignore it
<mgerdts> I'm trying to suss out how /etc/cloud/cloud.cfg.d/80-smartos.cfg is generated in the Ubuntu certified image.  Per a SmartOS user, it would be very handy if cloud_final_modules included package-update-upgrade-install.
<mgerdts> https://smartos.topicbox.com/groups/smartos-discuss/T7d92768361787248-Mec61af626a11795ac0831d34
<mgerdts> FWIW, the complete file is: https://paste.ec/paste/poJLHkJD#hWAYwiZsNpbVT76bDWN9CsfHxYyPPx6EVC9RII4vR10
<rharper> mgerdts: if you want to file a bug against cloud-images with the details, I think that should get it to the right folks who can look at that  https://launchpad.net/cloud-images/+filebug
<mgerdts> thanks rharper
<lungaro_> Does cloud-init provide any value in a bare metal pxe boot world?
#cloud-init 2019-09-25
<rharper> https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/373177
<rharper> blackboxsw: smoser:  ^^ fixes CI failures
<nils_> is there a linter for cloud config?
<smoser> nils_: sort of.
<smoser> python3 -m cloudinit.cmd.main devel schema
<smoser> thats the best we can do
<mgerdts> I rebased an old branch, made some changes, force pushed, then resubmitted a merge request.  The diff shows conflicts that don't really exist, seemingly related to the 13 month old version that was previously there.  Would it be easier if I just submitted a fresh merge request rather than trying to make this one happy?
<mgerdts> https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/373171
<smoser> rharper: so the upload failed because it tried to re upload
<smoser>   cloud-init_19.2-2142-g571f7c3-0ubuntu1+1669~trunk~ubuntu18.04.1.tar.xz
<rharper> yeah; I clicked it manually to request the build since nothing was built for 6 days
<rharper> but I don't understand the failure
<smoser> the recipe build builds the tarball
<smoser> and its tar and timestamps so it will happen to differ
<smoser> basically it already built this exact thing
<smoser> so re-tries will fail
<smoser> you could i guess maybe delete the old one from https://code.launchpad.net/~cloud-init-dev/+archive/ubuntu/daily
<smoser> and it might re-build
<smoser> or just merge something
<smoser> or you could change the recipe text to have '0ubuntu2'
<smoser> at https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily-bionic
<rharper> oh, no code changes
<rharper> but the build would fail for next commit
<rharper> without a fix for the asteroid version / ignoring generated modules on the context manager
<smoser> maybe it would. it probably wouldnt fail though
<smoser> as the ast dep comes from pip, not archive i suspect. unless they updated it in archive.
<smoser> other option looks to b add {time} to the recipe
<smoser>  https://help.launchpad.net/Packaging/SourceBuilds/Recipes
<smoser> that would mean you could always hit 'build now' (well... once per second)
<smoser> err, it looks like per-minute
<rharper> interesting
<rharper> we must not run tox on builds, I dont think the builder would allow pip install bits, right ?
<smoser> right. it runs make test i think. tests still run, but with the package installed dependencies.
<smoser> which is what you really want... because you want to test the distro
<garrett__> Hi. I'm trying to debug an issue, and I'm wondering if someone might have some ideas. I have some machines with one physical interface and a bunch of virtual interfaces. eth0 and eth0:0, eth0:1, etc. After updating from cloud-init 18.2-1 to 18.5-3, I see that LOCAL_IPV4 in /etc/environment is getting set to the IP bound to eth0:0 instead of eth0. I see some calls to udev settle in the cloud-init logs, and it looks like those changes might have bee
<garrett__> Sorry, this is centos. I'm trying to figure out exactly what patchsets are in there as we speak. I'm just wondering if this is an issue someone might have seen before, or if I'm going entirely down the wrong path
<smoser> garrett__: cloud-init doesn't do anything with LOCAL_IPV4  in /etc/environment.
<garrett__> @smoser: aha! okay
<garrett__> smoser: much thanks. you just saved me a bunch of time.
<Odd_Bloke> rharper: blackboxsw: smoser: If you have a minute, this fixes the build so we can continue landing stuff: https://code.launchpad.net/~daniel-thewatkins/cloud-init/+git/cloud-init/+merge/373221
<Odd_Bloke> Oh, ha, I see I'm late to this.
<Odd_Bloke> Should have fully caught up on my inbox before jumping on it, I guess.  Have Approved Ryan's MP.
<Odd_Bloke> I've filed https://github.com/PyCQA/pylint/issues/3137 for the issue we're seeing.
<smoser> Odd_Bloke: thank you for doing that.
<smoser> (we think it is probably astroid related though... or at least that was the change htat caused it)
<Odd_Bloke> Yeah, I mentioned in the issue that we only started seeing it on astroid upgrade, but it looks like pylint is the place to report such things.
<Odd_Bloke> (It's the same development team anyway, AFAIK.)
<powersj> Summit Agenda: https://docs.google.com/presentation/d/1Ch4QDfK8pd968mfBgnXQK7_0JWAatrkix069sxsVgOs/edit?usp=sharing
<powersj> Project Update: https://docs.google.com/presentation/d/1Rfyr7zhWWhijPD1TbF85Z540OXhTLpKeZSSomiYrUlo/edit?usp=sharing
<smoser> python3 -m cloudinit.cmd.main devel schema
<smoser> oops
<Odd_Bloke> blackboxsw: https://github.com/cloud-init/ubuntu-sru/pull/57 <-- the Oracle SRU verification
<rharper> Odd_Bloke: thanks
<powersj> rharper, blackboxsw does this mean the SRU is done?
<blackboxsw> powersj: checking
<rharper> no
<rharper> waiting on MAAS QA and CDO QA
<rharper> I've Xenial, but bionic/disco fail on MAAS, pinged in their channel
<blackboxsw> powersj: rharper; right  CDO QA hasn't gotten back yet with SRU results
<rharper> https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/372727
<blackboxsw> powersj: so per your email back to jog, I'm not sure if they're indefinitely waiting on defining 'what they do' w.r.t. SRU validation as they are radio silent after your followup.
<powersj> blackboxsw, I would respond to his last email and ask for an eta. Say that their validation is all we are waiting for
<rharper> powersj: a gentle poke to MAAS would be nice as well;
<blackboxsw> We just added the community: low-hanging fruit lane to our trello board  where anyone should be able to grab cards  https://trello.com/b/hFtWKUn3/daily-cloud-init-curtin.
<blackboxsw> If folks have interest in working any cards in the "Community: low-hanging fruit" lane, we can assign individuals to cards as they are being worked. They generally should represent bite-sized work that is aligned with upstream cloud-init goals for the next development cycle
<blackboxsw> we'll touch more on this in next bi-weekly meeting Oct 8th as well to give more context
<rharper> smoser: https://code.launchpad.net/~sdhd/curtin/+git/curtin/+merge/371401
<rharper> https://code.launchpad.net/~sdhd/curtin/+git/curtin/+merge/371401/comments/972028
<blackboxsw> azure manual validation for last sru https://github.com/cloud-init/ubuntu-sru/blob/master/manual/azure-sru-19.2.36.txt
<blackboxsw> for cloud-init SRUs, we update this PPA with upcoming cloud-init releases that are under tests https://launchpad.net/~cloud-init-dev/+archive/ubuntu/proposed
<blackboxsw> We only update this PPA with a cloud-init version under test during an SRU (stable release update) for Ubuntu. at maximum it may be updated once a month. So, it should be pretty stable
#cloud-init 2019-09-26
<paride2> Hi smoser
<paride> https://bugs.launchpad.net/cloud-init/+bug/1845489 <- this is basically a question for you :)
<ubot5> Launchpad bug 1845489 in cloud-init "Variable self-assignments causing pylint W0127 (self-assigning-variable)" [Undecided,New]
<paride> Odd_Bloke, while your general point on ^^ is completely correct (module variable != class variable), in the specific case it seem to me that no reference to the class variable exists (self.schema)
<paride> OK, found it!
<amansi26> I am trying to install cloud-init 19.1 on SLES15. But cloud-init init command throws an error : stages.py[ERROR]: Unable to render networking. Network config is likely broken: No available network renderers found. Searched through list: ['eni', 'sysconfig', 'netplan']   and the deployed VM on top of image captured from Base SLES15 VM is not assigned with the IP nor the cloud-init is active on the deployed VM
<amansi26> Is there any workaround for this?
<Odd_Bloke> amansi26: How are you installing it?
<amansi26> Installing cloud-init rpm, that I made adding some custom modules
<smoser> Odd_Bloke: htanks for filing pylint issue
<smoser> amansi26: i think that rharper mentioned this yesterday.
<rharper> blackboxsw: Odd_Bloke: https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/372727
<rharper> https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/373277
<rharper> that looks almost the same, crazy number selection
<rharper> https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/368845
<rharper> otubo: ^
<ahosmanmsft> Hello anyone here?
<ahosmanmsft> Getting this error running cloud_tests, on query streams.
<ahosmanmsft> tests.cloud_tests - ERROR - stage: collect for platform: azurecloud encountered error: no images found with filter: ['arch=amd64', 'endpoint=https://management.core.windows.net/', 'region=westus2', 'release=bionic']
<ahosmanmsft> I might be off of IRC, message me at ahosman@microsoft.com
<ahosmanmsft> Thanks
#cloud-init 2019-09-27
<blackboxsw> smoser: I think I have it
<blackboxsw> will send out a MR shortly
<powersj> ahosmanmsft region needs to be "West US 2"
<powersj> annnd he isn't here
<smoser> blackboxsw: ... ?
<blackboxsw> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/373291
<blackboxsw> smoser:
<blackboxsw> ^
<blackboxsw> ok gotta head out
<blackboxsw> Good points smoser. Will address in flight.
<blackboxsw> Have good flights folks. Thanks for a great cloud init summit.
<blackboxsw> Flights/travel
<amansi26> smoser: Can you please point me the location where I can find the discussion on this error: stages.py[ERROR]: Unable to render networking. Network config is likely broken: No available network renderers found. Searched through list: ['eni', 'sysconfig', 'netplan']
