#cloud-init 2013-09-09
<harlowja> smoser hopefully the commit should be ok now :-p
<harlowja> can do schema stuff later
<harlowja> u think schema stuff way to go?
<smoser> harlowja, i like the 'validate'
<smoser> and i'm fine with 'json-schema'
<harlowja> cool
<smoser> a module wouldn't have to use json-schema
<smoser> but probaly would
<smoser> (and could run the schema and then further test some stuff in its validate)
<smoser> you tihnk it seems sane ? i hope?
<smoser> and did you look at the opennebula stuff ?
<smoser> i'm really leaning towards just pulling their stuff in.
<harlowja> haven't check opennebula stuff recently
<harlowja> :-/
<harlowja> ya, no need for module to use json-schema, just provide its own validate() method
<harlowja> that the cloud-init stages can call before running
<smoser> why's it called 'json schema' ?
<smoser> really its just 'dict, list, string, int' schema
<smoser> right?
<harlowja> ya
<harlowja> :-p
<harlowja> got me, i guess the RFC is for json , but not really limited to
<harlowja> http://json-schema.org/
<harlowja> sorry, not RFC, ' IETF published draft '
<harlowja> *whatever that is, ha
<smoser> harlowja, 
<smoser> http://paste.ubuntu.com/6085177/
<smoser> adding that
<smoser> harlowja, is there a better way to read /sys ? 
<smoser> should i use read_file_or_url there ?
<smoser> for mocking / wrapping / test purposes
<smoser> harlowja, please think, respond, and i'll update and merge your code and then we'll have "random seed" support on azure too 
<smoser> wee!
<harlowja> haha
<harlowja> weee
<harlowja> smoser why not just load_file ?
<harlowja> u can still mock load_file ot
<harlowja> *out
<smoser> duh. load_file. yeah.
#cloud-init 2013-09-10
<harlowja> :)
<smoser> merged. thanks.
<harlowja> np
<harlowja> smoser so when u gonna help out with taskflow??
<PedroAlvarez> Hello
<PedroAlvarez> There is any log of what happening with the user-data commands?
<smoser> PedroAlvarez, /var/log/cloud-init.log has cloud-init log
<smoser> by default output of anything done goes to /dev/console
<smoser> which on openstack can be read with 'nova console'
<smoser> if you want to chagne that default, you can set output via cloud-config 'output'
<smoser> see example / description at http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/examples/cloud-config.txt
<smoser> i use
<smoser>  output: {all: '| tee -a /var/log/cloud-init-output.log'}
<smoser> by default.
#cloud-init 2013-09-11
<PedroAlvarez> smoser: Doing this change: http://paste.ubuntu.com/6070331/ I have this error:
<PedroAlvarez> RuntimeError: Failed running ['tools/read-version'] [rc=1] (, sed: can't find label for jump to 'a}')
<smoser> PedroAlvarez, that really should be "standard sed".
<smoser> workson gnu sed and busybox sed.
<PedroAlvarez> smoser: This is the change I did to get it running: http://paste.ubuntu.com/6093190/
<smoser> try adding either a ';' or a ' ' after 'ba'
<smoser> or both
<PedroAlvarez> smoser: but I have it already running.
<smoser> but i'm asking you to od something that doesn't require 3 forks and actually checks for failure correctly.
<smoser> (all yours does is check that 'head -n 1' succeeded.
<PedroAlvarez> smoser: ok :)
<smoser> and will silently echo nothing and not fail)
<PedroAlvarez> smoser: it works http://paste.ubuntu.com/6093408/
<smoser> PedroAlvarez, ok. i'll commit.
<smoser> thanks.
<PedroAlvarez> smoser: are you going to merge it with the trunk branch? 
<PedroAlvarez> smoser: tvm
<smoser> pushed revno 868.
<PedroAlvarez> :)
#cloud-init 2014-09-08
<bracki> Can I specify multiple runcmds? E.g. runcmd, write_files, runcmd?
<smoser> bracki, inside multiple cloud-config parts you can specify multiple pieces of config.
<smoser> so you could have one part do 'runcmd' , one do 'write_files'
<smoser> each is merged over the current config.
<smoser> arrays (annoyingly to you) overwrite by default.
<smoser> while dicts merge.
<smoser> you can change this behavior in recent versions of cloud-init.
<smoser> by specifying 'merge_how'
<smoser> or by using jsonp
<smoser> see some examples in comment 4 https://bugs.launchpad.net/cloud-init/+bug/1316323
<harmw> smoser: I opened an ipv6 support bug for cirros
<harmw> just to make sure I don't forget to do that once my cloud goes ipv6 :p
<smoser> harmw, yeah.i'm good with that.
<smoser> cirros-0.3.3 is just published.
<harmw> ah nice
<smoser> yeah.
<smoser> i really , really need to get a jenkins up and have daily builds and such.
<harmw> tell me..
<JayF> smoser: travis-ci is fairly easy to use, but I don't know if there's a way to use it 'detached' from github
<smoser> JayF, i suspect i can't use it.
<smoser> cirros needs root to build images.
<smoser> hm..
<smoser> does it launch vms for everything ?
<smoser> i guess it must.
<JayF> I don'
<JayF> *I don't really know, sorry :/
<smoser> it does. http://docs.travis-ci.com/user/ci-environment/
<smoser> its definitely itneresting
<mwhudson> smoser: travis is openvz isn't it?
#cloud-init 2014-09-09
<harlowja> smoser i've plugged into travis before, worked pretty good for my use-case
<harlowja> using it for https://github.com/yahoo/Zake
<harlowja> pretty easy to hookup https://github.com/yahoo/Zake/blob/master/.travis.yml into it
<JayF> yeah we used it for github.com/rackerlabs/teeth-agent and github.com/rackerlabs/teeth-overlord before we moved to working on Ironic
<harlowja> smoser hmmmm, https://review.openstack.org/#/c/119909/
<harlowja> hmmmm
<harlowja> alessandro why :(
<JayF> Competetition is good for your heart
<JayF> or was that exercise? 
<harlowja> lol
<harlowja> JayF ya ya
<harlowja> heart, competietion...
<harlowja> sleep
<harlowja> lol
<harlowja> food
<harlowja> all good for something, ha
<JayF> harlowja: I updated https://code.launchpad.net/~jason-oldos/cloud-init/upgrade-configdrive/+merge/232312 in response to your comments; I still have to fix the vendor_data handling to match what smoser wants but I'
<JayF> *I'll do that first thing in the mornign
<harlowja> :)
<harlowja> cool
<harlowja> JayF how does smoser want to treat the vd?
<smoser> JayF, when you get a chance, could you point me at whatever dock you have of the json netork definition format?
<smoser> the only thing i have is at https://review.openstack.org/#/c/85673/13/specs/juno/metadata-service-network-info.rst
<smoser> https://gist.github.com/jayofdoom/b035067523defec7fb53 is all i have right now.
<JayF> smoser: There's a blueprint up in Openstack Nova right now, but it's not approved and won't be until K so there is no "official" JSON network format... there's just basically what our automation emits
<JayF> for better or worse :/
<JayF> All our devices follow a similar network config: two phys interfaces bonded, and vlans on top of the bonded interface
<JayF> so I don't think we ever documented how it'd look for anything else
<smoser> right.
<smoser> thats fine
<smoser> its 'v1'
<JayF> yeah so tl;dr: no documentation :)
<JayF> as I understand it, this is more or less a dump of the information neutron sends to nova 
<smoser> ok. 
<smoser> did pquerna sign cla ?
<JayF> He has patches in already
<JayF> so I believe so
<JayF> I have to leave in a few minutes to get on a bus and head into the city for work :)
<JayF> this is what I intend on working on today ... I want to get the vendor_data reading stuff in sans network json today
<JayF> then all I'll have is that patch to get network json support in
<JayF> seem reasonable?
<smoser> JayF, yeah.
<smoser> it does
<smoser> i want to get this stuff in too
<pquerna> smoser: i've signed the cla previously
<smoser> pquerna, thanks.
<smoser> aren't you supposed to be being married right now?
<JayF> I think he's supposed to be honeymooning right now
<JayF> :)
<pquerna> i board a flight to FRA in a few hours
<JayF> Nice. 
<JayF> You two have fun :) 
<harmw> can I use Travis on premises?
<harmw> JayF: or is it payware if I want to use it for private projects?
<JayF> on premises is $$$$$
<harmw> bigtime
<JayF> they don't even publish the code iirc, because they basically say the magic is the infra
<ndonegan_> They seem to publish a lot of code: https://github.com/travis-ci/
<ndonegan_> Including chef cookbooks for deploying...
<JayF> That's apparently changed since I last looked then, nice :)
<JayF> Although afaict using travis-ci without using github is a nonstarter
<harmw> hmk
<harmw> ah ok, well thats to bad then 
<ndonegan_> heh, linked from wikipedia from a while back: https://github.com/travis-ci/travis-ci/blob/2ea7620f4be51a345632e355260b22511198ea64/README.textile#we-are-not-done-yet
<JayF> it really is
<JayF> I like the github workflow a lot better than any other tool I've used, but I really dislike the number of things that are becoming github only
<harmw> yea well, I'm using svn and bzr realy
<JayF> ndonegan_: that sounds like what I remember :)
<ndonegan_> Started using Gerrit internally at work. Can be annoying at time, but the workflow it enforces is handy.
<harmw> ever heard of the OpenSuse buildservice?
<JayF> Gerrit is what I use for Openstack, obviously
<harmw> though tht might be restricted to purely building (rpm/deb) packages
<JayF> never heard of it
<harmw> gerrit is nice, yes
<harmw> anyway, perhaps Jenkins is any good for building all sorts of things
<JayF> Jenkins is what I've used
<JayF> generally speaking managing build systems is terrible
<harmw> why :)
<JayF> which is why my knee-jerk was to suggest the hosted (travisci) service I've used before
<JayF> heh
<harmw> hhe
<harmw> lazy :P
<JayF> eh, I just come from an ops background and put a very high value on "just works"
<harmw> ah ok
<harmw> you know if jenkins can build rpm packages as well?
<harmw> next to cloud-init, or even cirros 
<JayF> Of course :)
<JayF> Jenkins is a fancy bash script runner
<JayF> among a shitton of other things
<harmw> hehe ok
<harmw> * firing up a centos7 instance
<harmw> arg, when will the multi prefix stuff land in Neutron
<harmw> I want dualstack :p
<smoser> "<harmw> yea well, I'm using svn and bzr realy"
<smoser> harmw, youre not supposed to say thinhgs like that out loud :)
<harmw> lol wut
<harmw> I'm not allowed to use both? :P
<smoser> right jenkins can do just about anything
<smoser> its really just script runner.
<smoser> and really that is all travis-ci is too
<harmw> probably, yea
<harmw> don't you have some c-i stuff to merge or something? :P
<smoser> cirros doesn't seem like it would fit on travis-ci because somewhere i saw a "50 minute time limit"
<harmw> isn't it fair use?
<smoser> ?
<smoser> i think its perfectly valid for them to do that.
<harmw> I thought the plans had fair use on a number of opensource stuff
<harmw> oh ofcourse
<smoser> but cirros will take longer than that to build
<harmw> right
<harmw> well, if jenkins can build cirros
<smoser> my experience is it takes ~30 seconds per arch.
<smoser> the goal i would have would be to have a jenkins that luanched slaves on ec2 or digital ocean
<smoser> or somewhere 
<smoser> and then built the things there
<smoser> sucked in the results
<smoser> and then published the output somewhere
<harmw> yea
<harmw> wicked :)
<harmw> wtf, jenkins comes with svn support and.... cvs
<harmw> like, wtf
<harmw> time to look into the plugin repository :p
<harmw> ah and it wants me to add bash stuff, per step
<JayF> yeah lots of plugins
<JayF> You can put a whole bash script in any of the steps
<harmw> yea I'm just looking at that
<harmw> nice
<JayF> if it's prefixed with a #!/bin/bash it'll be run with bash, otherwise it runs as /bin/sh
<smoser> travis-ci is basically the same.
<smoser> lets you run arbitrary commands
<smoser> so you could probably abuse their build service by putting *something* on github
<smoser> and pushing to it every time you needed it to build. and then that something pull from somewhere else inside the build system.
<harmw> hehe
<ndonegan_> harmw: We have Jenkins building RPMs, Gems, the hated omnibus pacakges, and even full blown image for use in OpenStakc.
<harmw> cool
<harmw> well that was easy, having Jenkins do a bzr checkout
<harmw> smoser: do we have a build script for cirros that I can have Jenkins call?
<smoser> harmw, ./bin/build-release
<smoser> mostly does it.
<harmw> ok, lets take a look
<smoser> read doc/create-release.txt
<smoser> hat is honestly all i do to create something
<harmw> ok
<harmw> if I make jenkins install stuff through either apt-get or yum, does that intrfere with the os itself or does jenkins use chroots?
<smoser> jenkins doesn't by itself use chroots.
<ndonegan_> harmw: It depend on how you set it up.
<smoser> you'd have to have something that would do that.
<smoser> oh. maybe ndonegan_ knows more. maybe there is explicit support for that.
<harmw> I'm seeing a plugin that does something with regards to chroot
<smoser> my goal would be to allow me to build anybody's branch
<smoser> without security concerns
<harmw> ndonegan_: please enlighten me :)
<ndonegan_> For example, I tend to use venv for certain Python projects. For using apt, you'll probably need to look at chroot.
<smoser> which would mean doing it in a throw away VM or instance on a cloud
<harmw> smoser: thats a cool idea as well
<harmw> just one system that runs the Jenkins master and have that interact with a cloudsystem to spawn additonal worker vm's
<ndonegan_> harmw: The image building I mentioned is using oz, which just starts up local VMs using libvirtd.
<harmw> (instead of what I'm currenly after, which is doing it all in just 1 vm)
<harmw> ah, that can be roughly compared to using some decent cloud then :)
<ndonegan_> Also, it's spectacularly easy to tell jenkins to use alternative build hosts for certain builds.
<ndonegan_> It's just an ssh account and key, and it sorts the rest itself.
<harmw> well that is very nice
<ndonegan_> However, when it comes down to it, all it's doing is running a few scripts for you.
<ndonegan_> https://wiki.jenkins-ci.org/display/JENKINS/Distributed+builds#Distributedbuilds-Havemasterlaunchslaveagentviassh
<ndonegan_> So, if you want to do something like use a chroot, you have to script it the same as if you were doing it on your own shell.
<harmw> great, well thanks :)
<harlowja> smoser i just learned how much heat is doing with cloud-init, lol
<harlowja> scary :-P
<harlowja> btw, since i know all u guys care
<harlowja> https://etherpad.openstack.org/p/TaskFlow-0.4 (new release)
#cloud-init 2014-09-10
<ndonegan_> harlowja: Is Heat making use of cloud-init? Haven't looked into Heat all that much yet
<harlowja> ndonegan_ seems like a very heavy user
<harlowja> https://github.com/openstack/heat-templates/blob/master/hot/software-config/example-templates/example-cloud-init.yaml
<harlowja> and others...
<harlowja> most of that template is just formed into cloudinit blobs, lol
<harlowja> even the crappy merging format
<harlowja> https://github.com/openstack/heat-templates/blob/master/hot/software-config/example-templates/example-cloud-init.yaml#L40
<smoser> harlowja, oh. wow. someone using the crappy merging format.
<smoser> thats neat i ithink :)
<harlowja> smoser its something, hahah
<harlowja> *scary*
<harlowja> lol
<smoser> well, to be fair, anyone using anything that i wrote is scary
<harlowja> ;)
<harlowja> +3
<harlowja> haha
<smoser> and then, when you throw in that josh harlow contributed
<harlowja> +10
<harlowja> lol
<smoser> it just gets down right horrific
<smoser> :)
<harlowja> def
<smoser> have a nice night man. and thanks for all your help always.
<harlowja> np
<harmw> so, people are actually *using* cloud-init! 
<harmw> woohoo :p
<harmw> smoser: thanks!
<harmw> reading the cirros announcement
<smoser> since i suck in being responsive and in so many other ways, the least i can do is publicly thank you
<harmw> lol
<harmw> well, thanks :)
<harmw> looks nice :p
<harmw> btw, when is a new version of c-i due?
<smoser> harmw, a few things hav eto be fixed.
<smoser> openstack things
<smoser> vendor-data
<smoser> and logg woarn. that is currently there.
<smoser> going to look more of that today
<JayF> smoser: Is there any vendor_data handling in any of the current data sources?
<JayF> smoser: hmm... I think I may have thought what I had to do for the first step was harder than I thought
<hoban> Hello everyone
<smoser> JayF, i'm gonna look at that now.
<JayF> I'm working on it as we speak
<hoban> I'm working towards migrating from CentOS 6.x to 7.0. Previously, we've done post-installation scripting via /etc/rc.d/init.d/firstboot, which no longer exists. I presume I could use /etc/rc.local. but I only want this code to run on the first time the system boots. Would cloud-init be a good fit for this? I also see a fairly immediate need to use something similar for OpenStack instances we'll be exploring soon. Any reason cloud-init couldn't handle my nee
<hoban> Mainly, I'm just manually applying some puppet manifests & running some scripts.
<JayF> smoser: what keeps things from blwoing up if vendor_data.json isn't actually json?
<JayF> smoser: that's what I've been trying to figure out w/r/t https://etherpad.openstack.org/p/cloud-init-vendor-data
<JayF> it looks like the current parsing would raise an exception if anyone put anything but JSON in that file anyway
<smoser> "vendor_data.json must be valid json, or cloud-init will warn and ignore"
<JayF> OH
<JayF> so that behavior is based on what's under the 'cloud-init' key
<JayF> so if I have DataSourceConfigDrive loading in vd similarly to how the other sources do it, that should be one chunk of work that can merge separately
<smoser> ok.
<smoser> so that pad is what i thikn it should work like
<smoser> not what it does (we're both aware its buggy)
<JayF> Yeah, I'm trying to break work up into steps
<smoser> but basically, if 'json.loads(open("vendor-data.json"))' returns a 
<smoser>  string: then cloud_init_vendor_data = vendor_data
<JayF> https://code.launchpad.net/~jason-oldos/cloud-init/upgrade-configdrive/+merge/232312 that should get us to the point of ConfigDrive feeding in vd the same as the OpenStack data source (md service) does
<smoser>    (ie, this is  all for me)
<JayF> gotcha
<smoser> you said that ruby would have a hard time getting a string, but python-json does allow that
<smoser> but it'd be odd
<JayF> If I were to do it in python, I'd almost certainly just do a try: except:
<smoser> then if its a list... we assume its clond-config-archive, and inspect each of the elements of the list.
<JayF> bceause afiact json.loads() in python throws a valueerror if it's not valid json
<smoser> right
<smoser> it will.
<smoser> i showed you this before
<smoser> here... just aminute
<JayF> yeah I remember :)
<smoser> $ python -c 'import json; print(json.dumps(""))'
<smoser> ""
<smoser> er.. maybe better.
<smoser> $ python -c 'import json; print(json.dumps("this is just a string"))'
<smoser> "this is just a string"
<JayF> What in the blue hell did I do earlier to make it throw an exception?!
<smoser> $ python -c 'import json; print(json.loads(json.dumps("this is just a string")))'
<smoser> this is just a string
<smoser> $ python -c 'import json; json.loads("this is just a string")'
<JayF> https://gist.github.com/jayofdoom/efd85d68df5bc9c74217
<smoser> ^^ that will show value error
<JayF> smoser: ^
<smoser> but
<JayF> OH
<JayF> dumps vs loads
<JayF> that's the piece I was missing
<smoser> $ python -c 'import json; json.loads("\"this is just a string\"")'
<smoser> a quoted string is valid json
<smoser> a non-quoted string is not.
<JayF> aha
<JayF> That makes a lot of sense, thank you very much for walking me through it
<smoser> :)
<JayF> OK
<JayF> so assuming my existing patch is fine
<JayF> What's the next step for getting the network configuration support in
<JayF> am I dependant on the overall vd fixes?
<smoser> ok. give me a bit . and i'll look at this more. have to finish something up.
<JayF> OK, that works. I need to commute into the office anyway. When you look at it comments on the merge req would be awesome too :)
<JayF> ty
<smoser> harmw, 
<smoser> cloudinit/distros/freebsd.py:267: undefined name 'searchservers'
<harmw> eerrrrr
<harmw> how come 
<smoser> i'm virtually certain it should be searchdomains
<harmw> sounds familiar, since it should probably should be searchdomains
<harmw> yep
<smoser> as searchdomains is assigned to []
<harmw> indeed
<smoser> and never set otherwise
<smoser> k.
<smoser> i'll make that change
<harmw> this does ring a bell though, I'm sure i've fixed that once already
<harmw> perhaps never got merge, or I'm just way off :p
<harmw> thanks though smoser 
<smoser> well, trunk is up to date wrt to lp:~harmw/cloud-init/freebsd-fixes
<smoser> but who cares. fixed.
<harmw> I haven't seen it recently, so probably never got merged somewhere in the past few months - if I'm even correct
<harmw> which ubuntu version should I use to build cirros btw?
<harmw> 12.04 or somthing?
<harmw> ndonegan_: jenkins and jclouds plugin, is that fun?
<harmw> from the looks of it that'll spawn a new vm on any supported cloud to just complete the buildtask
<smoser> harmw, 0.3 will only build on 12.04
<smoser> there is a buildroot bug that causes that.
<harmw> great thanks
<smoser> i will plan to build 0.4 on trusty though
<smoser> so as soon as we get a up to date buildroot we can move there.
<harmw> cool
<smoser> JayF, where is your work for vendor data stuff ?
<JayF> smoser: um I linked it earlier
<JayF> https://code.launchpad.net/~jason-oldos/cloud-init/upgrade-configdrive/+merge/232312
<JayF> like I said it's not a lot, because I don't think a lot was needed to get to the point where DataSourceConfigDrive does the same stuff for vd that DataSourceOpenStack does
<smoser> ok. will look.
<smoser> JayF, the one issue there 
<smoser> hm.. maybe not.
<smoser> never mind. 
<smoser> JayF, i like how you did the search.
<smoser> we should go backwards on OS_VERSIONS until we find on.
<smoser> i think "chronological order by time" is redundant :)
<JayF> smoser: I write comments in JayF -vvvvvv
<JayF> heh
<JayF> smoser: you want me to reword that or any other comments?
<smoser> no. i 'm llooking 
<smoser> and i think i can simplify it a bit
<smoser> JayF, http://paste.ubuntu.com/8312141/
<smoser> that is diff versus trunk at 1009
<JayF> When you said you could simplify, I suspected that'd be what you'd do
<JayF> it lgtm
<smoser> it makes the BaseReader just select the latest supported version
<smoser> ok. can you give it a quick test ? i will too.
<smoser> the other test i have to do is test ec2 right nwo
<smoser> and make sure we didn't break it or make annoying warnings if openstack looks like ec2 and that ds claims it
<JayF> I don't have a great test environment for this right now
<JayF> do you have a suggestion?
<JayF> We tested our local changes by using cloud-init to install a newer cloud-init rpm/deb via user_data then rebooting to see if it worked right, heh
<JayF> but I obviously can't do that until my changes get far enough that trunk will work on my servers
<smoser> ah. ok. i was just going to build from trunk
<smoser> and boot an instanance
<smoser> then install that
<smoser> and run cloud-init --local an dcloud-init
<smoser> and see that it seemed to have made sense.
<JayF> smoser: boot and instance where? for ec2 obviously aws, but what cloud uses ConfigDrive all the way around except OnMetal?
<smoser> updated http://paste.ubuntu.com/8312203/
<smoser> JayF, well, if you boot with --configdrive=1
<smoser> then you will get a config drive
<smoser> you'll still have the metadata service, but cloud-init should choose the configdrive
<JayF> smoser: I think harlowja and I both wanted to get the version= kwarg out of the function signature for read_config_drive() since it's not used
<JayF> smoser: you didn't appear to keep that change in your diff?
<smoser> well, i kept it.
<smoser> so you could explicitly state "i want this one"
<JayF> nowhere in existing code is that used at all though
<smoser> right.
<smoser> ok. lets drop that then
<JayF> cool
<smoser> i do think theres good reason to have it
<JayF> I can absolutely see situations where it /could/ be valuable
<smoser> i'd like to be able to configure it in the dsconfig
<smoser> yeah.
<JayF> but I was always told to not include args for stuff like that until it's used
<smoser> :)
<JayF> but I don't know or have strong opinions in this case
<JayF> although being able to tell cloud-init which configdrive version to read via config could be interesting
<smoser> biggest use case would be able to reproduce a bug on OS_HAVANA cloud
<smoser> when all i had access to was an OS_JUNO
<smoser> or something
<JayF> I don't /think/ ConfigDrive is version bumping in Juno?
<JayF> is it?
<smoser> just a bad example
<JayF> I mean, it's OK if it's a bad example, just wanted to make sure I was up on what was actually happening :)
<JayF> although there will certainly be one for KILO
<smoser> JayF, so it looks good to me so far.
<JayF> Is there any reason not to land the changes with your modifications?
<smoser> the thing annoying right now is that 
<smoser> on amazon:
<JayF> Land those then we can talk about next steps?
<smoser> 2014-09-10 20:10:30,588 - DataSourceOpenStack.py[DEBUG]: Giving up on OpenStack md from ['http://169.254.169.254/openstack/2012-08-10/meta_data.json', 'http://169.254.169.254/openstack/2013-04-04/meta_data.json', 'http://169.254.169.254/openstack/2013-10-17/meta_data.json', 'http://169.254.169.254/openstack/latest/meta_data.json'] after 0 seconds
<smoser> i just landed. 
<JayF> Woo
<JayF> I don't think it's that bad
<JayF> that's one log line, in debug
<JayF> most normal people won't run in debug ... 
<smoser> well, it annoying that it does 4 GETs though.
<JayF> ... we do
<JayF> Ah, I see that
<JayF> also ssomething kinda strange
<smoser> we could have it just first poke '/latest/meta_data.json' or even just '/latest/'
<JayF> is you're either logging in reverse
<smoser> and assume that would always b there.
<JayF> or going through them in reverse
<smoser> that was just from the initial search
<smoser> after it finds one
<smoser> then it goes in reversed order like you did
<smoser> its just looking for presense of a MD server at all
<JayF> Hmm
<JayF> smoser: if a cloud is running md service
<JayF> smoser: does a GET on /openstack/ succeed?
<JayF> If so that's a real quick short circuit for that data source
<smoser> right
<smoser> i'm gonna change that now to just check for /opentsack/
<JayF> Sweet
<smoser> we've made some good progress today.
<JayF> smoser: I'd still like to chat at some point about what the best next step is for me to get the bigger pieces I want to merge in
<JayF> smoser: also, RFE (and something I might try and implement if it's easy), an option to point cloud-init at a config drive dir
<JayF> smoser: for running it against "fake" config drives in a chroot
<JayF> smoser: CoreOS's cloud-init thinger does this, and it's been very useful in troubleshooting
<smoser> JayF, you can do that.
<smoser> almost certain it takes a 'seed' dir.
<JayF> ah, nice. I'll look at that, if so that'll help testing
<smoser> yeah, it works.
<smoser> or should
<smoser> see DataSourceConfigDrive
<smoser>         if os.path.isdir(self.seed_dir):
#cloud-init 2014-09-11
<smoser> harlowja, JayF lp:~smoser/cloud-init/os-vd-cleanup
<smoser> https://code.launchpad.net/~smoser/cloud-init/os-vd-cleanup
<harlowja> cool
<harlowja> just one problem ;)
<harlowja> smoser i think u want to use the raw json loads here then
<harlowja> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/sources/helpers/openstack.py#L224
<harlowja> util.load_json doesn't allow non-dicts as the root type
<harlowja> so u might want to switch that to just json.loads as the function
<smoser> harlowja_away, i think i just fixed that http://bazaar.launchpad.net/~smoser/cloud-init/os-vd-cleanup/revision/1013
<JayF> looking at that now smoser 
<JayF> the vd merge req from late yesterday afternoon
<harlowja> smoser checking
<smoser> JayF, the idea would be to keep the _pure available, and we'd read the networking information from there as it would not be under cloud-init. 
<harlowja> smoser looks good
<JayF> you know how I said I was looking at it
<harlowja> those the root types u wanted, no numbers and such i guess 
<JayF> what I meant was I opened it in a tab and got distracted for 40m
<JayF> he
<JayF> *heh
<JayF> Is there a way in LP to view a diff of a branch vs the master?
<JayF> or do I have to pull that revision down to see a diff
<smoser> JayF, you have to propose it for mergint. 
<smoser> JayF, http://paste.ubuntu.com/8320474/
<JayF> smoser: lgtm -- so the idea is that I can add a hook, that checks to see if self.vendordata_pure is a dict, and if so, if it contains vd['network_magic_string'] and if so, take that data and do things with it
<smoser> JayF, yeah. the "hook" is already there really
<smoser> you'll just improve the code that does it for 'interaces'
<smoser> results['network_config'] = self._read_content_path(net_item)
<smoser> that stuff
<smoser> in cloudinit/sources/helpers/openstack.py
<JayF> makes sense
#cloud-init 2014-09-12
<harmw> smoser: you have a link to 12.04 cloud image for me?
<harmw> found it: https://cloud-images.ubuntu.com/releases/precise/release/ubuntu-12.04-server-cloudimg-amd64-disk1.img
<smoser> :)
<harmw> hm, will glance convert that to raw on the fly in icehouse?
<smoser> "on the fly"
<harmw> (raw, because of my ceph backend)
<smoser> before running, yes.
<smoser> hm.. dont know about that. but i'd hope so.
<harmw> hm hm, oh well I could just download and convert myself :p
<JayF> I can tell you that ironic-python-agent deploys images using qemu-convert to convert them to raw, so anything qemu-convert supports i supported in ipa
<JayF> I have no idea how it works for real clouds :P
<harmw> :p
<harmw> ill just qemu-img <bla> myself
<harmw> hm, 128M is apparently not enough for ubuntu :p
<harlowja_> JayF u have some time to answer some questions i'm wondering about ironic, since i think u do stuff there
<JayF> harlowja_: fwiw I have less insight into the local state stuff b/c we run IPA driver (patched extensively) with our own dhcp.. so our conductors don't keep much local state at all :)
<harlowja_> ya, any local state not gonna surive a conductor crap-the-bucket
<harlowja_> *survive
<harlowja_> and knowing ops people they will probably resort to killing crap sometimes when shit hits the fan, lol
<harlowja_> JayF trying to see if taskflow can help, at least in some part :-P
<harlowja_> since it can persist the local state and transition between 'tasks' (not in the ironic terms) while doing this persistence and stuff
<JayF> I don't know what taskflow is right off
<JayF> and you should tell russell that :P
<harlowja_> sure sure
<harlowja_> JayF so u know http://docs.openstack.org/developer/taskflow/ :-P
<harlowja_> haha
<harlowja_> lots of read if u care/have time, ha
<JayF> the problem is
<JayF> I love reading things
<JayF> and care
<harlowja_> guess u just need the time part, ha
<harlowja_> https://wiki.openstack.org/wiki/TaskFlow#Summary (the basic idea i guess)
<JayF> well when I get distracted by shiny things
<JayF> usually the shiny thing is an interesting and long wiki page
<harlowja_> ^ long wiki page provided, ha
#cloud-init 2015-09-07
<Seth_Karlo> Odd_Bloke: Good morning, are you here? I seem to have found more chef bugs
<openstackgerrit> Claudiu Popa proposed stackforge/cloud-init: Add a draft spec for the parallel discovery of data sources  https://review.openstack.org/220095
<minfrin> Hi all, another quick question - does cloud-init have any kind of http_proxy support? I have a machine on a private subnet that has no internet access, and I'm trying to get the basics to work with a proxy in place and I am struggling. Creating environment variables using cloud-init seems to be too late as cloud-init has already started. Does anyone have a solution to this?
#cloud-init 2015-09-08
<harlowja> smoser Odd_Bloke claudiupopa http://lists.openstack.org/pipermail/openstack-dev/2015-September/thread.html#73571 fyi
<harlowja> if u intersted in reading :-P
<smoser> fun
<harlowja> :)
<harlowja> Odd_Bloke any more investigation into taskflow that u need help with??
 * harlowja here to serve
#cloud-init 2015-09-09
<Odd_Bloke> harlowja: I've been OOO since the weekend, I'm planning on spending some time this morning to get a proof of concept that we can discuss at the meeting this afternoon.
<Odd_Bloke> harlowja: So none as yet, but thanks for the offer!
<openstackgerrit> Daniel Watkins proposed stackforge/cloud-init: [WIP] TaskFlow for running shell commands  https://review.openstack.org/220593
<openstackgerrit> Daniel Watkins proposed stackforge/cloud-init: [WIP] TaskFlow for running shell commands  https://review.openstack.org/220593
<Odd_Bloke> harlowja: ARGH, so networkx have dropped Python 2.6 support in their latest release.
<Odd_Bloke> harlowja: Which the latest two versions of taskflow depend on.
<Odd_Bloke> Total sadface.
<Odd_Bloke> Also the installation failure doesn't happen in tox. :/
<eofs> is't possible to put cloud-config data into file and re-run it using cloud-init cli?
<eofs> currently I've to stop EC2 instance, update UserData and restart it which I find it to be very inconvenient method
<natorious> Morning!
<natorious> eofs: should be doable.  Just rid your instance data from /var/lib/cloud
<eofs> natorious: I tried to put my config into /etc/cloud/cloud.conf.d/10_custom.cfg, removed everything from /var/lib/cloud and rebooted
<eofs> http://pastebin.com/HK99AfNc
<eofs> "final_message" was printed into logs, but htop wasn't installed and no packages were upgraded
<natorious> eofs: what was the cli command you used?
<smoser> rebooting as you suggeted should work . after removing /var/lib/cloud
<smoser> pastebin (hint, wonderful command called 'pastebinit' is installable from ubuntu) your /var/log/cloud-init.log
<natorious> eofs: reason I ask is that the update upgrade install pkg module defaults to loading in the config mode so you would trigger it with something like 'cloud-init modules --mode config' or something along those lines
<eofs> odd, it seems to work now
<eofs> launched new, clean instance, added my settings into /etc/cloud/cloud.conf.d/10_custom.cfg, removed /var/lib/cloud and rebooted
<eofs> maybe I managed to "brick" by previous instance somehow
<natorious> eofs: removing /var/lib/cloud could be problematic on some distros init systems too
<smoser> natorious, it shoudl be ok to do taht.
<smoser> if not, i'd consider it a bug. things should be generally idempotent.
<natorious> ok, cool.  I did see some odd behavior with 0.7.7 when troubleshooting nic config on systemd by blowing away the /var/lib/cloud dir
<natorious> iir I had asked that same q here a year or so ago and blowing away /var/lib/cloud wasn't the rec way of resetting at the time
<Odd_Bloke> smoser: harlowja: claudiupopa: 10 minute warning for meeting. :)
<eofs> coreos-cloudinit seems to have easy to use cli: "coreos-cloudinit --from-file /path/to/file"
<eofs> easier than removing some random folder and rebooting and waiting n-seconds for instance to come online again
<smoser> eofs, well, you dont have to.
<smoser> you can run a single mode again
<smoser> cloud-init single --name=foo --frequency=always
<eofs> aah... where's the documentation!? ;)
<jimbobhickville> did something break in the latest release for CentOS 6?
<jimbobhickville> https://gist.github.com/jimbobhickville/4114f51b55b04a2f4d4b
<jimbobhickville> first bit is the exception that is thrown when trying to run cloud-init, the 2nd is the yum update output where it updated to the latest version.  This was a fresh image built yesterday
<smoser> jimbobhickville, it would seem that your 'requests' package is busted somehow
<Odd_Bloke> jimbobhickville: That _looks_ like a packaging problem with python-requests.
<Odd_Bloke> Hah.
<Odd_Bloke> smoser: harlowja: claudiupopa: Meeting?
<jimbobhickville> this is all just from the official centos repos
<claudiupopa> Yes
<Odd_Bloke> jimbobhickville: https://bugs.centos.org/view.php?id=9139 looks similar.
<smoser> ok. lets start a meeting.
<jimbobhickville> thx Odd_Bloke
<smoser> we'll just walk through open reviews quickly and then a open discussion
<jimbobhickville> Odd_Bloke: looks like the suggested fix of cleaning up the leftover folder worked, thanks
<Odd_Bloke> jimbobhickville: \o/
<smoser> reviews at http://bit.ly/cloudinit-reviews-public
<claudiupopa> So no hangouts meeting?
<smoser> oh. hm.. do you want hangout? i'd gotten to enjoy these in irc.
<claudiupopa> Not necessarily, no.
<smoser> and some others had been showing up for them.
<smoser> and are not on our hangout.
<smoser> lets put that into open discussion :)
<smoser> first review in my list is
<smoser>  https://review.openstack.org/#/c/220593/
<smoser> which is marked WIP
<smoser> i know that Odd_Bloke and harlowja have talked some about taskflow
<claudiupopa> Yeah, I talked with harlowja as well regarding https://review.openstack.org/#/c/220095/
<smoser> this does add a requriement on taskflow, which generally speaking we'd like to avoid. so maybe in the commit message or somewhere, a justification for picking up that requirement woudl be nice.
<smoser> especially since python3 version is not terribly easily available
<smoser> i'll put such coments into review here
<claudiupopa> The question if we really need taskflow for this.
<claudiupopa> For instance, regarding the parallel discovery and execution, it might not be so easy to use it.
<Odd_Bloke> So I really like taskflow for the overall stuff.
<Odd_Bloke> It allows us to declare a graph of tasks and dependencies, and execute as we want (persisting between runs).
<claudiupopa> It supports the whole range of supported Python versions?
<Odd_Bloke> Well, that's something I want to bring up in general discussion.
<Odd_Bloke> It does not support Python 2.6.
<Odd_Bloke> But I'd like to have a discussion about why we support Python 2.6.
<natorious> would /tmp be the best place for a taskflow db though?
<natorious> wouldn't some distros blow away /tmp on reboot
<smoser> all probably do
<Odd_Bloke> natorious: I would expect that to move to /var/lib/cloud.
<claudiupopa> Except windows maybe.
<Odd_Bloke> But that's a PITA for testing.
<Odd_Bloke> s/testing/manual testing/
<Odd_Bloke> Because /var/lib/cloud either does not exist or is owned by someone inconvenient (i.e. root). :p
<smatzek> I believe RHEL 6.x still has Python 2.6 by default.  I'm not sure about SLES 11 / 12.
<Odd_Bloke> Yeah, RHEL 6 is definitely still on 2.6.
<smoser> its worth discussing.
<smoser> we'll have to investigate the plausibility of RHEL 6 support with a newer python if we want to drop 2.6
<tpeoples> iirc sles 11 is using 2.6 as well
<smoser> ie we really do need to have *some* supported path there.
<Odd_Bloke> Do we think that a new version of cloud-init is ever going to hit RHEL 6?
<smatzek> tpeoples: do you know about SLES 12?
<smoser> its quite possible that it would not be in official archives of rhel6
<larsks> Odd_Bloke: if there were a critical new feature + lots of customer demand, the answer is "maybe".
<natorious> Odd_Bloke: I would hope that to be the case too
<smoser> but we have people who would want to use it there in their images (possibly on their official images for their users on public clouds)
<jimbobhickville> that ^^
<jimbobhickville> did taskflow drop 2.6 support recently?  It used to support it
<smatzek> smoser:  I agree, and some companies may build cloud-init 2.0 RPMs for use on RHEL 6 to maintain the same level of cloud-init functionality across their private clouds regardless of the operating system in the image.
<smoser> jimbobhickville, harlowja is sleeping
<smoser> but he can answer that, and i'll follow up with him today
<smoser> that coudl be an option also
<Odd_Bloke> jimbobhickville: Taskflow recently bumped its networkx requirement to a version of networkx that has dropped 2.6 support.
<smoser> there ya go.
<natorious> though, redhat doesn't have specific requirements going into base images and so if needed, they could add in their py 3 deps
<smoser> explain ?
<smoser> natorious, " redhat doesn't have specific requirements going into base images" ?
<jimbobhickville> you're not gonna convince public cloud providers to install py3k in the cent6 images by default
<natorious> smoser: I'm not aware of specific guidelines for a red hat base image per se.  The python 3 dependency packages could also be pre-installed in the base images
<smoser> ok.
<smoser> so we do need to look more into this.
<smoser> and see if there is some path to python versions from this decade in rhel6
<Odd_Bloke> This sounds, to me, like there are a loooong list of people who could club together to backport cloud-init and its dependencies to Python 2.6, if they insist on supporting a version of Python that is EOL.
<smoser> and sles
<jimbobhickville> I imagine that networkx dependency issue was a mistake. openstack requires python 2.6 support last I checked and taskflow is trying to become an official library
<smoser> ok. lets move along. this has been good discussion though. but i need to move.. to make another meeting :)
<Odd_Bloke> As an upstream, I'm not sure why we would take on the burden of supporting a Python version that is, I stress, END OF LIFE. :p
<Odd_Bloke> Anyway, to be continued. :)
<smoser> Odd_Bloke, because we want to support an OS that is *not* END OF LIFE (for another 5 years ;)
<smoser> next: https://review.openstack.org/#/c/220095/
<smoser> for that... anyone reading along here.
<smoser> pelase read that spec that claudiupopa has written and comment on the review
<smoser> make sense ?
 * smoser will read today also
<smoser> https://review.openstack.org/#/c/220536/
<smoser> that seems to me to be straight orward enough
<smoser> the one thing that i would suggest is that '--log-to-console' is long to type.
<smoser> --log=- ?
<smoser> woudl that be possible , Odd_Bloke ?
<smoser> ok. i'll comment on review. lets go on
<smoser> https://review.openstack.org/#/c/220543/
<Odd_Bloke> smoser: It's non-trivial so, yeah, let's continue that conversation in the review.
<smoser> i'll just ack that now.
<smoser> ok. and thats it. the only other one is my pip2distro... which i thin is pie in the sky, but i wish someone had.
<smoser> so that i could do some thing like: pip-install-deps-as-seen-on-ubuntu-14.04
<smoser> or rhel6. or whatnot. to easily test in a similar environment to that you'd get with package installed dependencies.
<smoser> so. open discussion..
<smoser> 2 things here.
<smoser> a.) do people find this medium appropriate, or do we want to have some audio/voice enabled medium ?
<smoser> i've been happy with this one... i want tog et a channel logger in here though at very least.
<Odd_Bloke> I prefer IRC, because it's more transparent.
<smatzek> I like this medium.
<smatzek> isn't ubuntulog2 logging the channel?
<claudiupopa> Yeah, and the fact that more people can chime in is also better.
<Odd_Bloke> And will scale better once we hit the $video_provider channel limit. :p
<smoser> b.) ... shoot . what was this one ?
<Odd_Bloke> +1 on shoot.
<natorious> Im fond of irc as well
<smoser> well, i'll dig a bit on our py26 discussion.
<smoser> please feel fre to ping me at any time here.
<Odd_Bloke> I think it might be worth blocking out an hour for this meeting.
<smoser> smatzek, was right.
<smoser>  http://irclogs.ubuntu.com/2015/09/09/%23cloud-init.html
<Odd_Bloke> (He said, setting up Hangouts for his meeting)
<smoser> that log is sufficient for me.
<Odd_Bloke> It would also be good to have a way of proposing topics etc.
<natorious> So, I've read thru the hacking and spec docs.  Not sure where to get started though.  The spec mentions distro namespaces and datasources that don't exist.
<smoser> agreed.
<smoser> you have thoguths ?
<Odd_Bloke> So, for example, people can't forget what they were going to bring up. ;)
<Odd_Bloke> I don't, but I'm happy to take an action to go and find something.
<smoser> natorious, yeah.. a getting started and list of things to do would be good/necessary.
<smoser> we do have http://bit.ly/cloudinit-roadmap
<natorious> bc the distro classes are already spec'd? Should I start hacking on some distro basics and datasources from scratch?
<smoser> which is intended to be that.
<natorious> like current master branch has an osys namespace which looks like it should be distro related?
<smoser> right.
<smoser> i do have to jump away for a minute.
<smoser> for a while. so natorious you can continue with Odd_Bloke (possibly) or i can return to you in a while.
<smoser> i think you're US time zone-ish, right?
<natorious> smoser: yeah, CST
<claudiupopa> natorious: there are is a data source implemented already.
<claudiupopa> https://github.com/stackforge/cloud-init/tree/master/cloudinit
<natorious> k, so all our current configs use ConfigDrive as a datasource and am looking forward to persistency whatnot.  Would need to get that going to be able to test
<natorious> think I saw a trello task for that
<natorious> claudiupopa: is that something your actively working on or mind if I hack at it a bit?
<claudiupopa> No, I'm not doing any work with that right now, please go ahead and hack at it. :-)
<natorious> cool deal.  I might have a trello login somewhere
<Odd_Bloke> Apologies, was on a call.
<Odd_Bloke> Back now.
<natorious> yeah, I can login and subscribe to trello tasks but, can't assign
<claudiupopa> What's your trello id or email? I'll add you on the board.
<natorious> nathan.house@rackspace.com
<natorious> ty
<jimbobhickville> hrmâ¦ cloud-init should still be writing /var/lib/cloud/data/result.json on completion right?
<smoser> jimbobhickville, /run/cloud-init/result.json
<jimbobhickville> ah, the path changed recently?
<jimbobhickville> that file doesn't exist either
<smoser> jimbobhickville, /run/cloud-init/ shoudl be there.
<jimbobhickville> the folder does not exist.  maybe rackspace overrides that?
<jimbobhickville> result.json used to be in the path I pasted up until I just updated our image and now it no longer seems to be generated at all.  /var/log/cloud-init.log and /var/log/cloud-init-output.log both indicate that cloud-init completed
<jimbobhickville> ooh, missed this error: http://mirror.rackspace.com/centos/6/updates/x86_64/Packages/libXfont-1.4.5-5.el6_7.x86_64.rpm: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"
<harlowja> jimbobhickville what u doing here
<harlowja> ha
<harlowja> lol
<harlowja> jimbo
<jimbobhickville> haha, wrong room on that last paste
<harlowja> lol
<jimbobhickville> so, nevermind, cloud-init is still the default installed version on this image because the yum update failed when I built it.
<harlowja> sooo hmmm, good point on 2.6 and taskflow, arg, didn't think about that
<jimbobhickville> thus no output of the results.json
<harlowja> jimbobhickville 'taskflow is trying to become an official library'
<harlowja> ehmmmm
<harlowja> 'trying''''
<harlowja> lol
<harlowja> :-P
<jimbobhickville> is it official now? I haven't paid as much attention lately
<harlowja> i think its official as u can get
<harlowja> obamba didn't sign it into law though
<harlowja> but i'm working on that
<Odd_Bloke> harlowja: Well, I'm trying to convince everyone to drop Python 2.6 as a requirement.
<Odd_Bloke> Because this problem is only going to get worse and worse.
<Odd_Bloke> And also anyone still shipping Python 2.6 needs to go and sit in a corner and think about what they've done.
<Odd_Bloke> (Whilst paying people to backport modern Python software for them)
<harlowja> Odd_Bloke oh, i'm probably the person asking for 2.6 support :-P
<harlowja> but maybe, it can be done, ha
<Odd_Bloke> harlowja: WHYYYYYYY
<harlowja> and just say use 0.7.x for rhel6
<harlowja> mainly rhel6 + yahoo will probably be around for a while
<harlowja> but i think it might be ok to just say 0.7.x is the last rhel6/python2.6 stuff,
<harlowja> cern i think also even wanted support for rhel5 (which uses 2.4 python)
<Odd_Bloke> I don't understand the person who needs to run RHEL6 but is also happy to run something from outside of the RHEL6 repos as root. :p
<harlowja> some cern guy asked me at the summit about that
<harlowja> i think i told that cern person to go talk to smoser ha
 * harlowja didn't want to touch that with a stick, lol
<Odd_Bloke> (Furthermore, I don't understand why that person wouldn't also be happy to install a modern Python also from outside of the RHEL6 repos :p)
<harlowja> Odd_Bloke ya, its possible, and probably we should just do 2.7 and i'll figure out what to do about the yahoo stuff (which i don't think will be a big issue on my side)
<harlowja> vs making all your lifes painful :-P
<harlowja> i'll take one for the team, lol
<Odd_Bloke> Well, there were other people at the meeting earlier (thanks for showing up BTW ;) who also didn't like the sound of dropping 2.6 support.
<harlowja> hmmm, meeting
<harlowja> u guys still have that to early, lol
<harlowja> Odd_Bloke but jimbobhickville knows some taskflow, so he can represent a little :-P
<harlowja> good ole jimbo
<harlowja> ha
<smoser> Odd_Bloke, i agree with above.
<smoser> er... sort of
<harlowja> i'll let u smart people decide, i think i'd be ok with dropping 2.6 and i'll deal with it here
<harlowja> 0.7.x for rhel6/python2.6 and thats fine i think, for the time being at least from my side
<smoser> i'd expect someone who really wanted new software on rhel6 to be fine to build their own python (or get it from some source they trusted)
<harlowja> smoser agreed, yahoo can/will do that if needed (and already does some of that anyway)
<harlowja> or just https://wiki.centos.org/AdditionalResources/Repositories/SCL
<harlowja> which provides Python 2.7 (python27)
<harlowja> *although its a PITA to use imho, but whateva
<smoser> the use case which woudl warrent 2.6 support is public cloud use
<harlowja> whats this public thing u talk about
<harlowja> lol
<smoser> if someone makes an image available on a pbulic cloud of something called either "centos" or "rhel", then i'd expect that they'd need very-close-to-fficial python
<Odd_Bloke> Would someone want a new version of cloud-init in their "assumed to be normal" RHEL6/CentOS 11 image?
<harlowja> CentOS 11?
<harlowja> ;)
<harlowja> that may arrive in year 2100
<Odd_Bloke> Um, *SLES.
<Odd_Bloke> $enterprise_linux_distro $old_version_number
<harlowja> :)
<harlowja> ya, so i'm torn here, i get both of the views on this
<harlowja> and taskflow also dropped 2.6, because pretty much all of openstack dropped 2.6 (including libraries that taskflow uses, which means that taskflow also has to drop 2.6 due to the transitive of this...)
<harlowja> *transitive nature of this
<harlowja> now one could make a minature taskflow, if they so wanted
<harlowja> but that sorta sucks as a solution imho
<smoser> openstack dropped 2.6 ?
<harlowja> ya
<smoser> what is the rhel 6 support case then ?
<smoser> for openstack, surely redhat supports openstack kilo or liberty on rhel6
<smoser> right?
<harlowja> http://lists.openstack.org/pipermail/openstack-dev/2014-October/048999.html
<harlowja> 'The application projects are dropping python 2.6 support during Kilo' ...
<harlowja> libraries dropped it this cycle
<harlowja> sooo smoser  afaik, there is no kilo + rhel6 support
<harlowja> basically from what i know its 'go use rhel7'
<smatzek> harlowja:  yep
<smoser> wow
<harlowja> which afaik, rhel7 is/was always the recommended RDO 'solution'
<harlowja> never rhel6
<smoser> yeah.
<smoser> https://www.redhat.com/archives/rdo-list/2015-May/msg00209.html
<harlowja> ya
<harlowja> my team (and other companies, cern, godaddy) gets to have the fun time of actually moving rhel6 -> rhel7
<harlowja> its super-fun :-/
<harlowja> especially on hypervisors :-/
<harlowja> it helps that i also maintain http://anvil.readthedocs.org/ but its still a pita
<harlowja> but afaik, no kilo and beyond for rhel6 (from redhat at least)
<harlowja> those companies that used openstack on rhel6 are on there own...
 * Odd_Bloke --> EOD
<smoser> ok. well, then for rhel i'm kind of convinced.
<Odd_Bloke> Final thought: we should consider having this sort of conversation on some sort of ML so that people don't have to be temporally present to have input. :p
<harlowja> Odd_Bloke agreed
<smoser> agreed
<harlowja> isn't there one that is setup for launchpad?
<Odd_Bloke> But that is a discussion for another time when we N meet again. :p
<harlowja> *for/by launchpad
<harlowja> https://launchpad.net/cloud-init/+topcontributors crap, no longer #2
<harlowja> lol
<harlowja> damn u Odd_Bloke damn u !!!
<harlowja> lol
 * harlowja guess i've been slacking, ha
<jimbobhickville> interesting, didn't realize openstack was dropping 2.6 support.  we have to support centos6 because all the hortonworks stuff only just now started offering centos7 support and we support the most recent two versions.  Until they release another version and we can drop the last centos6-only version
<jimbobhickville> which is like 6 months out most likely
<harlowja> jimbobhickville cool, good to know
<harlowja> ya, openstack dropped it like a year ago (for the projects, libraries dropped in last 6 months)
<jimbobhickville> we do install py27 as well, though, so we could probably work around it if cloud-init can be configured to not use the system python install
<harlowja> it should be able to be configured that way
<harlowja> depending on how u guys build py27
<harlowja> and how u want to install cloudinit (rpm?)
<jimbobhickville> both are rpm-installed
<harlowja> k
<jimbobhickville> no cloud-init-env.sh we can force the PYTHONPATH in or something?
<harlowja> not sure, probably a few ways to pull it off
<smoser> jimbobhickville, the goal would be to be virtualenv installable.
<jimbobhickville> anyway, we can probably just stick with the current version anyway, and rhel6 won't ship a version that requires py27
<smoser> so yeah, fairly easily relocatable.
<harlowja> do we have any offical RH person involved here?
<smoser> the one thing that is most annoying for that at the moment is python-yaml. i *think* its the only thing that isnt pure python
<harlowja> (aka, not me) ha
<harlowja> smoser i think the python-yaml library works in pure python (but slower)
<harlowja> it just has 'speedups' that it can compile in
<harlowja> probably can tell it to stop doing that
<harlowja> somehow
<jimbobhickville> you can do compiled modules in a virtualenv anyway
<jimbobhickville> (at least I thought you could, pretty sure we do)
<harlowja> from http://pyyaml.org/wiki/PyYAML 'both pure-Python and fast LibYAML-based parsers and emitters. '
<harlowja> u can probably tell it to just use the pure-python one (or it should be smart and fallback)
<smoser> harlowja, yeah, i just tried that and verified.
<harlowja> cool
<smoser> http://paste.ubuntu.com/12322352/
<harlowja> cool
<smoser> seems like it does a reasonable job of figuring it out
<harlowja> http://svn.pyyaml.org/pyyaml/trunk/setup.py has a interesting 'LIBYAML_CHECK' in it (some c code, ha)
<harlowja> so i guess thats whats being used
<harlowja> anywayssss, so i'll let u guys ponder over the py26 question
<harlowja> i'll be ok with it, and will deal with whatever the consequences are here at y! (nothing to much expected that i can think of right now, ha)
<jimbobhickville> we'll be fine here as long as the new version doesn't show up on the rhel yum repos
<jimbobhickville> (for rhel6 I mean)
<harlowja> smoser somehow i got emailed about this, https://github.com/number5/cloud-init/commit/7bde163d344737b99c594ae1d2cdcb3e31d45197#commitcomment-13125937
<harlowja> lol
<harlowja> not sure whats up with that :-P
<harlowja> someone complaining...
<jimbobhickville> he lost me at "nobody writing code in 2015"
<harlowja> :)
<jimbobhickville> so many logical fallacies in one small statement
<harlowja> good think i wrote it in 2012
<smoser> :)
<jimbobhickville> that's what you should respond.  just that
<harlowja> lol
<harlowja> jimbobhickville ok i responded
<harlowja> and i left a hamburger
<harlowja> ha
<harlowja> hopefully meets your expectations as far as response, lol
<harlowja> bb
<jimbobhickville> a++ would troll again
<harlowja> lol
<natorious> should the configdrive source live in the openstack namespace?
<natorious> it looks like alot of the base can be shared between the two etc
<natorious> or would having them in their own be better bc of configdrivenet?
<smoser> natorious you can look 0.7 to see how that was done
<smoser> harlowja used a lot of common code there.
<natorious> yeah, via helpers
<natorious> nothing other than the openstack helper mod ever made it in there though
<smoser> "in there" ?
<natorious> almost seems like configdrive should live in the openstack ns.  But, if other clouds use it outside of openstack, not sure it would make sense
<natorious> yeah so like 0.7.7 has sources.helpers.openstack
<natorious> but nothing else in sources.helpers
<smoser> i'd consider configdrive to be in openstack namespace.
<smoser> even if someone else used it. they'd be mimicking openstack.
<smoser> if i understand what you're asking
<natorious> k, right.  Like as in should configdrive and httpopenstack share a base
<smoser> sure
<harlowja> doesn't configdrive and httpopenstack share a base?
 * harlowja thought they did
#cloud-init 2015-09-10
<arkin> http://pastie.org/private/aedriwyhrntvtylnf0txgg
<arkin> can anyone help me clean this up? I'm getting some errors atm
<smoser> arkin, you're probably facing shell quote issues
<smoser> avoid shell quote
<smoser> instead of 'runcmd' having a bunch of strings, populate it with a bunch of arrays
<smoser> not this
<smoser> runcmd:
<smoser>   - mysql -e 'CREATE DATABASE \`{{ mysql_db }}\`;'
<smoser> but this
<smoser> runcmd:
<smoser>   - [mysql, -e 'CREATE DATABASE "{{ mysql_db }}";'
<smoser> or whatever would be the right syntax there.
<smoser> but basically then you can avoid shell getting in the way.
<arkin> yea I am smoser
<arkin> I fixed it messing around with " ' ` variations
<arkin> thanks for your reply
<smoser> arkin, really, though. get rid of the shell interpretation
<smoser> its much more sane.
#cloud-init 2015-09-11
<arkin> smoser: what does that mean ?
<Odd_Bloke> arkin: If you pass an array of shell arguments, you don't have to escape things as much.
<Odd_Bloke> arkin: Which makes reading and updating your cloud config easier in future.
<arkin> true ok
<smoser> arkin, http://paste.ubuntu.com/12338745/
<smoser> that is basically how cloud-init interprets runcmd
<smoser> theres a trick there with yaml "anchors" which can be very helpful
<smoser> it allows you to avoid yaml interpretation easily. so you can do more complex things like: http://paste.ubuntu.com/12338768/ and keep your sanity
<smoser> Odd_Bloke, did jerff get some package cleaning done recently ?
<Odd_Bloke> smoser: Not that I know of, but it is an IS-managed machine so... Â¯\_(ã)_/Â¯
<smoser> i dotn think it did. wethe maas merge added python3-yaml as a dep too
<smoser> Odd_Bloke, ok. how do i get python3-yaml installed on jerff ?
<Odd_Bloke> smoser: Yeah, I think that's the way.
<smoser> i thought i was in another channel :)
<elro> Hi, Iâm using cloud-init to configure Ubuntu 14.04 ec2 instances. Is there a way to avoid leaving .ucf-dist files when installing packages? For instance, I have write_files configure /etc/apt/apt.conf.d/20auto-upgrades but that results in a 20auto-upgrades.ucf-dist being left around.
#cloud-init 2016-09-12
<smoser> this is pretty awesome.
<smoser> import base64
<smoser> base64.encodestring("FOO")
<smoser> TypeError: expected bytes-like object, not str
<smoser> so... encodestring takes bytes and not a string. that makes sense.
<rharper> smoser: I guess why I thought the timestamp in the log file was a syslog issue , is on xenial and yakkety, even the entries in syslog itself are missing resolution (journctrl --unit cloud-config)   but maybe that's how we configure the logger ?
<rharper> it appears that when we don't use syslog, we include a timestamp field in the log format;
<smoser> rharper, so... recap
<rharper> that implies to me that cloud-init using log_syslog relies on syslog to insert a timestamp (it does)
<smoser> when cloud-init "comes up", it sets up python logging per 05_logging.cfg
<rharper> but the resolution doesn't have subsecond in xenial and newer
<smoser> i suspect that in xenial+ we're always going through rsyslog
<rharper> I agree
<smoser> and yes, when you log to syslog, its kind of pointless to add a timestamp
<rharper> which is why I think rsyslog must have changed how it records timestamps
<smoser> it would just be duplicate
<rharper> right
<smoser> (outside of the time delta between posting the message and rsyslog getting it)
<rharper> right
<rharper> but the resolution isn't sub-second; that seems wrong (sub optimal)
<smoser> so it is possible that the default rsyslog configuration changed in xenial
<smoser> and no longer logs subsecond when it used to
<rharper> I can change rsyslog conf (there's a comment for high res )
<rharper> in /etc/rsyslog.conf
<rharper> right, I'm hunting for that sort of change
<rharper> I've not found it yet
<smoser> can you verify that it was set the other way in previous releases ?
<rharper> but I think I understand why we see the change, as you say, in trusty, rsyslog wasn't up yet and cloud-init uses the "Write to file" formatter with the timestamp
<rharper> there's no "setting"
<rharper> it's just the default
<rharper> the rsyslog.conf is not different from trusty, xenial or yakkety (just versions of rsyslog)
<smoser> but in trusty we almost certainly should have used syslog for later messages
<smoser> (only cloud-init init should have gone "direct")
<rharper> lemme look
<rharper> yeah
<rharper> I see it
<rharper> it transistions to syslog
<rharper> and drops the subsecond output =(
<rharper> 2016-09-12 14:08:38,438 - util.py[DEBUG]: cloud-init mode 'init' took 0.209 seconds (0.00)
<rharper> Sep 12 14:08:38 t1 [CLOUDINIT] util.py[DEBUG]: Cloud-init v. 0.7.5 running 'modules:config' at Mon, 12 Sep 2016 14:08:38 +0000. Up 3.0 seconds.
<smoser> so...
<smoser> i'm really leaning towards harlow's suggestion
<smoser> of dropping rsyslog unless configured.
<rharper> "unless configured" meaning cloud-config specifically enabling it ?
<rharper> our images clearly have a syslog configured
<rharper> smoser: nice!
<rharper> now you need new $topic
* smoser changed the topic of #cloud-init to: cloud-init 0.7.8 released 2016-09-12. 0.7.9 open. reviews: https://code.launchpad.net/~cloud-init-dev/cloud-init/+git/cloud-init/+ref/master/+activereviews
<jgrimm> \o/
<roaksoax> jgrimm: re: EA, would cloud-init w/ ntp support work on CentOS 6 (python2?)
<jgrimm> roaksoax, not tested for sure, but as far as python version i think should be ok (or fixable).  rharper would talk more definitively to impl or other challenges ^^
<rharper> we need 2.7+
<rharper> this was capture in the NRE doc
<rharper> w.r.t cloud-init requirements
<rharper> centos has 2.6
<rharper> 2.6 *barely* works IIUC
<jgrimm> rharper, +1
<jgrimm> that's a statement about general, not the ntp specifically i'm assuming
<rharper> oh
<rharper> sorry
<rharper> right;  there's not much to the ntp config module
<rharper> I'd like smoser to speak up; but I really want to avoid having to support 2.6 in centos
<jgrimm> yep, though your answer may be ultimately the right answer to the question. :)
<rharper> yeah
#cloud-init 2016-09-13
<smoser> cloud-init does work with 2.6, harlowja uses it.
<harlowja> usually works, ha
<smoser> on centos, and keeps fixing it when we break it.
<smoser> its not really terrible to keep support working jgrimm rharper roaksoax
<robjo> Are nosetests3 and unittest3  Python 3 versions of these or version 3 of those utilities?
<rharper> robjo: should be python3 versions of the package
<robjo> thanks
<rharper> for example, nosetests3 on my system shows a shebang of /usr/bin/python3 and in comments nose==1.3.7
<smoser> for reasonably current versions of nose, you can just run with: $PYTHON -m nose
<smoser> where PYTHON is the interpreter.  if you use python3 it will be python3, python2...
<robjo> Well, the makefile prefers python3 over python2 thus a simple make test goes down the python3 path, of course I could have looked at this before I asked ;)
<smoser> right. thats because python2 is EOL for several years :)
<robjo> but the bigger problem is probably oauthlib for testing, anyway will have to sort that out if I have bigger changes ;)
<robjo> well as distro people we are doing a good job of refusing to accept that fact ;)
<robjo> python3 is only slowly becoming the default
<robjo> UNrelated question, whatever happened to https://launchpad.net/cloud-util years ago I used it to pull the growpart script, where did it move?
<smoser> add an s
<smoser> https://launchpad.net/cloud-utils
<ePh0x> Hi guys, having a little issue with extra_opts in disk_setup when trying to format an aws EBS with xfs -m crc=1 on a centos 7.2 machine running cloud-init 0.7.5. Has anyone come across a similar issue?
#cloud-init 2016-09-14
<Odd_Bloke> smoser: I'm seeing problems upgrading cloud-init on yakkety: http://paste.ubuntu.com/23177787/
<Odd_Bloke> smoser: Known issue?
<Odd_Bloke> smoser: (It looks like the two versions in question there are the xenial-updates and xenial-proposed versions, so we might see this in xenial as well)
<smoser> Odd_Bloke, it causes error ?
<Odd_Bloke> smoser: Ah, no, not now I look at it properly.
<Odd_Bloke> smoser: It was just the last package to be installed before other failures were reported.
<Odd_Bloke> And I can't read, apparently. :p
<smoser> ok. it is definitely a result of me having dropped the list of 'dirs' from cloud-init that were in the packages.
<smoser> because i was advised that it wasnt needed.
<harlowja> @smoser qq
<harlowja> for package building, whats the recommended way to not have `X.Y.Z+xxx.gHASH` show up
<harlowja> and just force X+Y+Z (even if there are additions)
<smoser> you dont like that ?
<smoser> hm..
<harlowja> its ok for dev
<harlowja> but we have some add-on changes (cc_godaddy)
<smoser> hm.
<harlowja> pbr has a way to force a explicit version
<harlowja> maybe we can do the same
<smoser> i dont think i considered that ata ll.
<harlowja> seems like it partially exists in read_version
<smoser> right
<harlowja> read_version has
<harlowja> use_long = '--long' in sys.argv or os.environ.get('CI_RV_LONG')
<harlowja> use_tags = '--tags' in sys.argv or os.environ.get('CI_RV_TAGS')
<harlowja> which are sorta close, ha
<harlowja> just no turn-off-the-thing
<smoser> but it *is* useful to have the hash avialable
<smoser> even from where you started i think.
<harlowja> ya, maybe i should just tell the guy that doesn't like the dirty name to be quiet, lol
<harlowja> :-
<harlowja> :-p
<harlowja> perhaps still have the json write
<harlowja> but have the version reported still be something that can be set?
<harlowja> i think the json still is useful
<smoser> mgagne, can you verify https://bugs.launchpad.net/cloud-init/+bug/1605749 in xenial ?
<mgagne> smoser: yes, will rebuild image with latest package and see what happens
<harlowja> @smoser do u know if https://git.launchpad.net/cloud-init/tree/setup.py?id=0.7.7#n196 is really needed anymore
<harlowja> sucky part for that one is ./brpm doesn't know about it being added on :-P
<harlowja> but i can't remember why we are still adding that in, lol
 * nrezinorn waves at harlowja 
<harlowja> sup
<nrezinorn> lol its jim :)
<harlowja> hi jim
<harlowja> smoser this jim is saying brpm not working cause of cheetah being dynamically included ;)
<harlowja> which makes sense :-P
<nrezinorn> what i said was, and i paraphrase "fix it josh"  lol
<harlowja> lol
<harlowja> then i was like, man those cloud-init people like patches
<harlowja> (especially from new people)
<harlowja> lol
<nrezinorn> lol
<harlowja> (and then we ended up here)
<harlowja> in happy land
<harlowja> also argparse is weird
<harlowja> cause apparently we are seeing that if its listed in requirements, and cent7 says yup its installed (it lies since its built-in) that python later complains (saying its not there)
<harlowja> which is a lie
<harlowja> nrezinorn found that one, i guess i never hit cause i always used a machine that had a older cloudinit on it before
#cloud-init 2016-09-15
<smoser> harlowja, argparse is really wierd on centos i know that ot be true.
<harlowja> def
<smoser> i'm not sure if we really depend on cheetah any where else or not.
<smoser> i really ahve to run... sorry.
<harlowja> np
<smoser> i'm wliling to listen tomorrow
<cpaelzer> smoser: hi, today I have had a few uvt based kvms that stalled infinitely not getting /var/lib/cloud/instance/boot-finished created
<cpaelzer> smoser: known issue / not an issue / worth to report / worth to debug ?
<ffledgling> Hello, quick question about creating users and groups with cloud-init, is it possible to specify the GID and UID numbers as well?
<ffledgling> I don't see it in the syntax/examples
<ffledgling> https://bugs.launchpad.net/cloud-init/+bug/1396362
<ffledgling> Looks like there was a patch but it wasn't merged
<smoser> cpaelzer, hm.. no. not to my knowledge.
<cpaelzer> smoser: ok, then I'll at least check how reproducible that is ... just a sec should be fast
<smoser> cpaelzer, it seems likely it is https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1623868
<cpaelzer> reproducible
<cpaelzer> smoser: no mine has that service properly starting http://paste.ubuntu.com/23181982/
<cpaelzer> I seem to match the second comment with the inactive cloud-init.target
<smoser> cpaelzer, can you get journalctl ?
<cpaelzer> smoser: yes
<cpaelzer> I can log in and check all I need
<cpaelzer> currently writing bug update
<cpaelzer> but the more you help me to debug the more I can help
<smoser> cpaelzer, see in ubuntu-devel
<harlowja> nrezinorn smoser ok its tommorow!
<harlowja> lol
 * harlowja thinks nrezinorn wants to destroy brpm and replace it with the raw spec files
<smoser> suck
<harlowja> lol
 * smoser shakes fist at time
<harlowja> nrezinorn is making a list of naughty things
<smoser> harlowja, what do you need ?
<harlowja> lol
<harlowja> let's see
<harlowja> i'd like a milk shake
<harlowja> let's see when nrezinorn gets around
<mgagne> smoser: cloud-init 0.7.8-1-g3705bb5-0ubuntu1~16.04.1 works fine for me
<harlowja> smoser  if u get some time https://code.launchpad.net/~harlowja/cloud-init/+git/cloud-init/+merge/305882
<harlowja> nrezinorn saw init-cloud-local blowup with a invalid argument on reading from /sys
<harlowja> i'm thinking we need to catch those, due to weird reading issues at that early of boot
<harlowja> [    4.405675] cloud-init[271]: File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 1242, in pipe_in_out
<harlowja> [    4.406060] cloud-init[271]: data = in_fh.read(chunk_size)
<harlowja> [    4.407056] cloud-init[271]: IOError: [Errno 22] Invalid argument
<harlowja> [    4.407377] cloud-init[271]: --------------------------------------
<harlowja> thats weird ...
<harlowja> because its always 1024, lol
<harlowja> so pretty sure its masking some other issue
<harlowja> catching IOErrors should be fine though
<harlowja> the rest of traceback is around https://git.launchpad.net/cloud-init/tree/cloudinit/net/__init__.py?id=0.7.7#n145
<harlowja> does that remind u of anything?
#cloud-init 2016-09-16
<smoser> harlowja, an example wouldbe good.
<smoser> hm..
<smoser>  /sys is wierd. sometimes reading stuff (any stuff) can give invalid argument
 * smoser out
<harlowja> smoser https://gist.github.com/harlowja/189cec2d4a2587edb9664733cff339aa that was from nrezinorn
<harlowja> which is odd, cause chunksize never changes :-P
<jgrimm> smoser, wrt 161968 latest comments.. wouldn't they need an updated xenial image to test?
<jgrimm> the stack trace posted matches cloud-init-0.7.7~bzr1256 (error raised on line 599), so something is suspect
#cloud-init 2017-09-11
<rharper> smoser: blackboxsw: hrm, so openstack metadata service does include mode network configuration information ('networkdata') top level metadata key  (this is on octal IIUC)
<smoser> ?
<smoser> "mode network configuration" ?
<smoser> rharper, ?
<rharper> the openstack metadata service includes a top-level 'networkdata' dict, which is what we see in network_data.json (in configdrive)
<rharper> I'd not seen that before
<rharper> http://paste.ubuntu.com/25515784/
<smoser> that is from network_data.json ?
<smoser> as in
<smoser>  http://paste.ubuntu.com/25515791/
<smoser> ?
<smoser> basically config drive contains what metadata service contains.
<rharper> that's from using DataSourceOpenStack.py:read_metadata_service()
<rharper> then that's expected
<smoser> so yes, we knew taht was there.
<rharper> I didn't
 * rharper catches up
<rharper> why do we do fallback config on metadata service instead of reading 'networkdata' like we do with network_data.json in ConfigDrive ?
<rharper> just haven't gotten around to implementing?
<smoser> we do not yet have a OpenStack datasource (other than ConfigDrive) that runs at local.
<rharper> ah, that's right
<rharper> it'd need to do the same ec2 style bring up of v4 link local
<smoser> we could do that. but handt done it. it'd work the same as ec2, basically guess a nic and dhcp on that.
<smoser> i dont know if ipv4 link local works or not
<smoser> i kind of assume not. but maybe. i will see if ic an try that quick
<rharper> some recent scanning of openstack docs mentioned link local v4 but yeah, let's check
 * rharper shakes fist at nova 
<rharper> interface-attach takes --port-id <instance_id>   , interface-detach takes <port_id> <instance_id>
 * rharper SMDH
<rharper> wait, <instance_id> <port_id>
<rharper> wee
<smoser> why would you have to tell detach which instance you want to detach it from
<smoser> wierd.
<rharper> good news is the mds is updated
<rharper> I can see one nic at first, nova interface-attach; curl shows two nics, remove; cur shows one again
<smoser> yeah. it is sufficeint to implement hotplug
<rharper> yeah
<rharper> default is DHCP though
<smoser> default ?
<rharper> no params on attach port
<rharper> I guess maybe I need to configure the port or something else
<rharper> still playing
<rharper> http://paste.ubuntu.com/25515784/  in there you can see both have 'ipv4_dhcp' as the type
 * powersj finishes new bug triage for the day
<powersj> down from 101 -> 91 bugs
#cloud-init 2017-09-12
<goten> script placed in /etc/cloud/cloud.cfg.d/my_script.cfg is not executed during boot up. Did not find any useful info in /var/log/cloud-init.log, is other places I can look for more info? Or what is the best way to troubleshoot this?
* smoser changed the topic of #cloud-init to:  reviews: http://bit.ly/ci-reviews | mail: http://bit.ly/ci-mail-archive | docs: http://cloudinit.readthedocs.io/en/latest/ | next meeting Tue, 12 Sep 2017 16:30:00 +0000
<powersj> just to have them in here: these are my two MPs for KVM
<powersj> commands as strings: https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/330535
<powersj> add xkvm: https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/330536
<dpb1> xkvm?
<powersj> wrapper around qemu command
<powersj> rharper/smoser created for curtin's use
<nacc> since qemu's parameters make little/no sense to most people, dpb1 :)
<dpb1> people who aren't cpaelzer
<dpb1> powersj: TIL, thx
<smoser> rharper, https://blueprints.launchpad.net/nova/+spec/virt-device-tagged-attach-detach
<smoser> that is what i think i remember.
<rharper> \o/
* smoser changed the topic of #cloud-init to:  reviews: http://bit.ly/ci-reviews | mail: http://bit.ly/ci-mail-archive | docs: http://cloudinit.readthedocs.io/en/latest/ | next meeting Mon, 18 Sep 2017 16:00:00 +0000
<rharper> smoser: in that spec, it mentions the qemu firmware exposing metadata path;  we have support for that now, right?  the nocloud from smbios?  do you think that's the same?
<rharper> "Outside the scope of the Nova work, a simple tool will be created that can parse this metadata file and set tags against devices in the udev database. It is anticipated that cloud-init would trigger this tool. "
<smoser> rharper, ?
<smoser> i dont follow
<rharper> I know we added some dmi path to getting datasource, or user-data, right ?
<rharper> the DMI NoCloud mode ?
<rharper> I was wondering if that was what the spec was referring to w.r.t qemu exposing metadata via firmware to instances;    the spec has a general concern about providing network-based metadata to instances which may not have network configuration (the writers were unaware of how cloud-init handles this in other network-based datasources)
<rharper> and the second quote was just an interesting tidbit;  the use-case of 'tagging' devices for service discovery (sdc is for 'oracledb') and wanted to provide a more "General" way for applications to discover the tagging without forcing the app to learn "openstack device tagging json schema"; and suggested udev tags/attrs as such a method
<smoser> well, the dmi nocloud stuff is not related to the openstack stuff for sure.
<rharper> but it
<rharper> but it was provided by qemu (aka firmware)
<smoser> we just now support reading nocloud info from dmi
<smoser> yeah
<rharper> I wonder if it could be a ConfigDrive style support if they could stuff enough info in there
<rharper> I guess it's only useful during boot;
<rharper> hence NoCloud;  I don't suppose qemu has an interface to update values in the dmi table dynamically
<smoser> powersj, you just replaced my mp ?
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/330459 -> https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/330535
<powersj> Based on your comment it seemed like you wanted a seperate MP. So we can merge yours and then put mine on top to change the other strings.
<smoser> oh i see you added some of the string changes too
<powersj> yeah
<smoser> powersj, i'm sorry.
<powersj> ?
<powersj> smoser: I assume the sorry is for the review?
<smoser> yeah
<powersj> smoser: for KVM when we execute SSH expects a string, and mount-image-callback expect a string
<powersj> should I then do the opposite as LXD, in that if it detects it is a of type array, convert it to a string?
<smoser> powersj, mount-image-callback doesn't expect a string. it has a sane interface.
<smoser> ssh interface sucks
<smoser> it shoves you through sh (or rather the user's shell) more for security reasons than anything else.
<smoser> that way the platform can manage what your user can do by setting the shell differently.
<powersj> smoser: the way I was calling MIC https://paste.ubuntu.com/25522742/
<smoser> to fix ssh, you can do this:
<smoser>  https://gist.github.com/smoser/88a5a77ab0debf268b945d46314ea447
<smoser> you have to be creative.
<smoser> (that does insert a dependency on 'base64', which i dont have a good way to work around)
<powersj> sigh
<powersj> smoser: I guess I'm not sure I understand why the usage of the python library doesn't work
<smoser> that is basically what we use in tools/run-centos : inside_as
<smoser> what python library ? i'ms orry.
<powersj> the integration tests are in python
<powersj> I am using paramiko to call ssh
<powersj> from what you are saying you want me to get rid of that to call your ussh shell script
<smoser> powersj, no. using paramiko is fine
<smoser> we just wrap it sanely.
<powersj> smoser: define wrapping sanely :) outside of your base64 encoded strings this is a new to me
<smoser> :)
<smoser> i want to provide an interface '.execute()' that takes a list. not a string.
<smoser> (just like subprocess.call() does... its not a crazy idea)
<dpb1> powersj: you are familiar with pep8, right?
<dpb1> the pep, not the tool
<powersj> dpb1: of course
<dpb1> ok
<powersj> smoser: you are choosing to pass a list to replicate the behavior or because you feel passing the list is a more design decision?
<powersj> is a better design decision*
<smoser> i think it is a better design decision. because in the end for doing anything complex you fight the shell more than it helps you.
<smoser> and... as shown if you want to invoke a shell, that is easy.
<powersj> smoser: ok that is fair and makes sense when using subprocess calls, what I am trying to grawk is how to take that decision and apply it to my use cases.
<powersj> 1) the use of the python ssh library which expects a string
<powersj> 2) how I call MIC: https://paste.ubuntu.com/25522742/
<smoser> well you can just call mic with a list
<smoser> mount-image-callback image -- command
<smoser> mount-image-callback image -- command arg1 arg2
<smoser> so 2 is pretty easy
<powersj> well, that's not so clear to me
<powersj> I can't append a list to it, so I have to process the list more than simply ' '.join(list) right?
<powersj> or append the list to the whole command, meaning something like: ['mic', 'image', '--'] + command
<smoser> powersj, right.
<powersj> smoser: https://paste.ubuntu.com/25523087/ that is where I get stuck
<smoser> your /bin/sh is the problem there
<smoser> chroot _MOUNTPOINT_ /bin/sh dpkg-query
<smoser> you're telling /bin/sh to execute something named 'dpkg-query'
<smoser> which is not a file
<powersj> and sure enough that works
<smoser> so you just have to say:
<smoser>  if command_is_a_string:
<smoser>      command = ["sh", "-c", command]
<smoser> where 'command_is_a_string' is syntax to do that idea
<powersj> ok, do you even want to allow strings at this point then?
<smoser> we can allow them.
<powersj> you mentioned in only certain places in the last review
<smoser> its just that they're scary to me
<powersj> "lets only fix strings where we were already using sh (via sh -c)."
<smoser> right
<powersj> so if using sh -c --> convert to string and remove sh -c from it
<smoser> so the places where we *were* doing 'sh', '-c'
<smoser> it makes sense to just drop the sh -c
<smoser> and pass the thing in as a string
<smoser> and let the "do the right thing for strings" logic take over
<powersj> if we were not doing that, then use as a list
<smoser> yeah.
<smoser> the one you have theere in that paste has a bug.
<smoser> --showformat='${Version}'
<smoser> we really wanted
<smoser>  --showformat=${Version}
<powersj> I'm not so sure those quotes are a bug ;)
<powersj> that is getting the version to print it in the logs
<smoser> its getting the version with '' around it
<powersj> ok so yes not part of the version
<powersj> but made the output look nice ;)
<powersj> ok one more question to let me get going on this
<powersj> you said this command is easily fixed into sh
<powersj> cmd = ('for ((i=0;i<{time};i++)); do {test} && exit 0; sleep 1; done; exit 1;'
<smoser> the for() is bash syntax.
<smoser> cmd = ('i=0; while [ $i -lt {time} ] && i=$(($i+1)); do {test} && exit 0; sleep 1; done')
<powersj> ah
<smoser> exit 1 at the end.
<powersj> ok I'll get these into a new merge for you to look at in the morning
<powersj> then we need to talk about SSH
<smoser> yeah.
<smoser> it is doable
<powersj> smoser: thx
<smoser> powersj, http://paste.ubuntu.com/25523267/
<powersj> smoser: interesting
<smoser> http://paste.ubuntu.com/25523299/
<smoser> powersj, so whatever you were going to pass to paramiko to execute
<smoser> just pass paramiko.execute(shell_pack(cmd))
<powersj> smoser: but I can't pass a list
<powersj> ;) so convert list to string?
<powersj> or add another isinstance(cmd, list) and run that
<smoser> you can pass shell_pack() a list or a string. or you can change it to only take a list
<smoser> and then do the isinstance business elsewhere.
<smoser> to be nice, we can wrap paramiko.execute() with a try/except and then lie about the commadn that we executed
<smoser> so that the command that is printed looks like what was executed, rather than
<smoser> eval set -- "$(echo J2VjaG8nICcoIGhpIHBvd2Vyc2ogKSc= | base64 --decode)" && exec "$@"
<powersj> heh yeah...
<dpb1> bad memories there
<powersj> one of my biggest complaints is debugging failed commands, especially when they look like lists
<dpb1> in my last last job, we had a windows service that did something like this on every incoming ssh attempt.
<powersj> so to get a base64 encoding string is even worse
<powersj> haha
<dpb1> in our case, it was an improvment
<dpb1> at least then you could cut and paste a string out
<dpb1> instead of a random mix of ASCII and non-ascii characters that windows programs would decide to show you
<dpb1> especially quotes
<smoser> http://docs.paramiko.org/en/2.2/api/client.html
<smoser> that is odd
<smoser> it returns a file-like object representing stdin, stdout, stderr
<smoser> how does it return stdin ?
<rharper> smoser:  https://bugs.launchpad.net/cloud-init/+bug/1716773
<ubot5> Ubuntu bug 1716773 in cloud-init "cloud-init doesn't cache network_config property in cache" [Undecided,New]
 * powersj loves the faster unit tests
<powersj> thx blackboxsw
<blackboxsw> hopefully as we get more that continues to be the case
<powersj> smoser: should the shell pack be a separate merge?
<powersj> working quite nice so far
<powersj> smoser: string merge is ready https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/330535
<blackboxsw> grr smoser https://bugs.launchpad.net/cloud-init/+bug/1691489 I just saw this on zesty on my nth redeployment iteration with our SRU candidate
<ubot5> Ubuntu bug 1691489 in cloud-init (Ubuntu Yakkety) "fstab entries written by cloud-config may not be mounted" [Medium,Confirmed]
<blackboxsw> ubuntu@zesty1:~$ grep reformat /var/log/cloud-init.log 2017-09-12 22:23:52,316 - DataSourceAzure.py[DEBUG]: reformattable=True: partition 1 (/dev/sdb1) on device /dev/disk/cloud/azure_resource was ntfs formatted and had no important files. Safe for reformatting.
<blackboxsw> this message came up with a reformated disk on an instance that I had already created
<blackboxsw> cloud-init	0.7.9-233-ge586fe35-0ubuntu1~17.04.1
<blackboxsw> hrm reproduced multiple times on zesty. not sure what we do w/ that bug then. the "fix" didn't fully resolve  it, but it didn't break cloud-init either.
<blackboxsw> ahh nevermind. even though reformatting happens /mnt is still formatted an no errors in cloud-init.log (per the original bug)
<blackboxsw> I was taking the ephemeral reformat log as a symptom of failure, but it's an ephemeral disk, reformatting it is okay. Not mounting it is not okay
<blackboxsw> ok gotta run
#cloud-init 2017-09-13
<akik> i installed cloud-init on centos 7.3. i provide it the cloud-config through cloud-config-url kernel parameter. the cloud-config ends up in /etc/cloud/cloud.cfg.d/91_kernel_cmdline_url.cfg but it's not executed
<akik> /usr/bin/cloud-init init seems to get it resolved, after strange timeouts
<akik> what does this error message mean? url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [6/120s]: unexpected error ['NoneType' object has not attribute 'status_code']
<akik> there seems to be a timeout of 120s
<akik> DataSourceEc2.py[CRITICAL]: Giving up on md from ['http://169.254.169.254/2009-04-04/meta-data/instance-id'] after 120 seconds
<akik> my cloud-config contains just one user creation and two small local file creations
<akik> ok the problem seems to be connected to those two 120s tasks that just timeout
<akik> one waiting for http://169.254.169.254/2009-04-04/meta-data/instance-id and the other waiting for http://192.168.137.1//latest/meta-data/instance-id
<nacc> akik: fwiw, most of the devs are probably afk right now
<akik> nacc: looks like i need to configure libcloud's datasources for the correct environment where the vm is running?
<akik> at this time, my home :)
<akik> it's a centos vm running in hyper-v manager in win10
<akik> how do i tell cloud-init on which iaas infra it's running on?
<akik> is it the datasource: part in /etc/cloud/cloud.cfg ?
<akik> ds= kernel parameter?
<akik> bc -l
<akik> oops
<smoser> blackboxsw, your diagnosis above is correct. but you did scare me :)
<smoser> akik, the error there is due to falling back to the ec2 metadata service.
<smoser> there is unfortunately really bad swallowing of the error for the nocloud url you provided.
<smoser> ah. akik what was in the url that you put ?
<smoser> but in order for this to work, the rm will have to be configured already to dhcp or whatever on the correct network interface.
<smoser> ie, be already configured to "dhcp on eth0" or the equivalent
<akik> smoser: i used cloud-config-url= to provide the cloud-config, but the provisioning timed out trying to access those two urls
<akik> so i waited 4 minutes, and then cloud-init was able to apply the config to the vm
<smoser> ah.
<smoser> ok. so it sounds like you're just missing one thing.
<smoser> you provided some cloud-config, but it did not find a datasource.  that is just "config".
<smoser> in the config that you provide, you can also data that will be seen as a datasource.
<smoser> let me find
<akik> if i understood right, the cloud-config-url provides the user-data?
<smoser> akik, from doc/examples/cloud-config-datasources.txt ... and ammended.
<smoser>  http://paste.ubuntu.com/25527448/
<akik> i will be moving this image to azure eventually, but i'm now testing it on my local machine
<akik> smoser: is that for cloud.cfg?
<smoser> the thing to be aware of is the networkign configureation.
<smoser> in order to get that url.... networking has to be configured :)
<akik> or is it for the cloud-config-url to process?
<smoser> so cloud-config-url does not provide a datasource. it just provides cloud-config from the command line.
<smoser> so as you saw, it gets dumped into that file
<smoser> and cloud-config can define a datasource
<smoser> as i did in the paste there.
<akik> smoser: but do i need to start that file with #cloud-config ?
<smoser> basically cloud-config-url is just grabbed and dumped to the file, and then cloud-config goes on its merry way.
<smoser> i dont think it does... but it wouldnt hurt :)
<akik> thank you
<smoser> akik, no problem. this is not a very well ridden path, but i do think it should work.
<akik> yes what i'm doing is quite odd too
<akik> azure vms have two provisioning "things" that can be installed simultaneously, waagent and cloud-init
<smoser> yeah.
<smoser> i'm well aware :)
<smoser> the goal is to replace the agent entirely
<smoser> in newaest ubuntu releases, it is gone
<smoser> (waagent that is)
<akik> how well does cloud-init support centos?
<smoser> well, recently  much better. but there are probably still some warts.
<smoser> help is always welcome too :)
<akik> i'll try that nocloud next
<smoser> where'd you get your image ?
<akik> i create it myself
<akik> just finished a shell script that builds the vhd in centos
<dpb1> SRU is through!
<akik> oh btw i had another question about cloud-init. in azure the vm gets by default the sudo configuration NOPASSWD:. is that intentional?
<akik> and now i mean the ubuntu server 16.04 image from azure marketplace
<smoser> akik, literally NOPASSWD:. ?
<smoser> the configured user should have passwordless sudo, yes.
<smoser> as they have no password
<akik> yes it's just weird how it's different from other ubuntu setups
<akik> tell that on #ubuntu and they'll crucify you :)
<smoser> akik, well, its a design decision.
<smoser> you have no password for that user by default
<smoser> and thus, you can't really configure sudo access with a password
<smoser> so you get ssh access in and sudo as that user.
<smoser> so access to the ssh private key essentially provides your sudo auth.
<akik> but cloud-init still supports having a user password
<smoser> sure. and it will let you configure the sudo stanza too if you want.
<akik> was there some discussion about the NOPASSWD: on the mailing lists?
<smoser> akik, well, no. this was proabbly 7 years ago. when ubuntu first got onto a cloud
<smoser> akik, what would you *expect* to happen?
<smoser> if you gave it a password, i could somewhat reasonably expect that it might configure the sudo for password auth
<smoser> but if you dont give it a password, it has no way to convey one to you, so the only acl it possibly has is ssh key.
<akik> i'd give the user a password
<smoser> in config?
<akik> yes in cloud-config
<smoser> then i can see an argument for saying that sudo should have a password prompt also for that user.
<smoser> its not somethign easily changed though due to being backwards compatible
<akik> smoser: i'd give the user also the ssh pubkey
<akik> that way using ubuntu in the cloud would feel the same as using it on the desktop
<powersj> smoser: rharper blackboxsw: this is the merge for xkvm into cloud-init
<powersj> https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+ref/add-xkvm
<powersj> that should be an easy one (?)
<akik> i used the following data for cloud-config-url= kernel parameter. this time the data didn't end up in /etc/cloud/cloud.cfg.d. is there something obvious i need to fix in the data?
<akik> i still get the timeouts for url_helper.py
<akik> had a bad paste. fixed paste here: https://pastebin.com/UVZtUwDB
<akik> there's a slightly different syntax for the datasource here http://cloudinit.readthedocs.io/en/latest/topics/examples.html
<akik> should i edit the default cloud.cfg that gets installed on centos from yum? the version is 0.7.5
<smoser> powersj, ill pull
<powersj> smoser: do you want the base64 functions you wrote as a separate merge? and do you want them in the integration test util file or cloud-init's util file?
<smoser> part of the integration is fine. cloud-init doesn't really have any  use for them.
<akik> i'm giving this on vm boot for the kernel: "ds=nocloud-net;s=URL". i have user-data and meta-data at URL. after booting up, i see in the log "DataSourceNoCloudNet [seed=cmdline][dsmode=net]". it looks like cloud-init didn't fetch those files
<blackboxsw> smoser: pushed apport changes, because I couldn't help myself
<blackboxsw> wrapping up chef
<smoser> \o/
<smoser> ok. lookoing.
<dpb1> blackboxsw: that sounds fantastic
<rharper> smoser: blackboxsw: anything need eyes/reviews right now
<smoser> blackboxsw,
<smoser> can i drop "The schema definition for each cloud-config module is a strict contract for"
<smoser> ... ?
<smoser> http://paste.ubuntu.com/25529138/
<smoser> (per schema-resizefs-bootcmd)
<blackboxsw> +1 smoser
<blackboxsw> https://trello.com/c/1xfreXAe/380-branches-to-land-for-cloud-init-release.  rharper, any of the unchecked in there except chef omnibus I think as I'm not done yet
 * blackboxsw is lunching 
<rharper> k
<smoser> blackboxsw, ok. i'm going to do that
<blackboxsw> Ok thanks smoser
<smoser> and also fix a vertical space in some tets
<dustymabe> hey team - i just submitted a PR for xfs issue on cloud-init
<dustymabe> https://code.launchpad.net/~dustymabe/cloud-init/+git/cloud-init/+merge/330701
<dustymabe> this is my first PR, let me know if there is anything else i need to do
<powersj> dustymabe: thanks for the PR! Do you recall if you signed the contrib agreement?
<dustymabe> powersj: i don't remember signing anything
<dustymabe> i signed in using ubuntuone account (which i created a few years ago to try out the ubuntu phone stuff)
<dustymabe> so... the answer is maybe, but not recently
<dustymabe> :)
<dustymabe> there should be some sort of blockchain that lists any terms you ever sign
<dustymabe> billion dollar idea, there you go world
<dustymabe> powersj: any tips on how to move forward?
<powersj> dustymabe: yeah sorry was looking for the link
<powersj> Take a look at: https://www.ubuntu.com/legal/contributors
<rharper> powersj: dustymabe: this is the boilerplate we'll put into the MP; http://paste.ubuntu.com/25529398/
<dustymabe> rharper: ok. what is the MP?
<rharper> sorry, Merge Proposal (what you submitted)
<dustymabe> rharper: ahh. MP == PR
<dustymabe> cool
<rharper> similar to Pull Request;  we suffer from launchpad  bzr language
<dustymabe> i'll read through the agreement and get back with you guys.
<rharper> I'm added that;
<rharper> in general, the fix looks good, commit msg looks sane; solid fix
<rharper> thanks for submitting
<dustymabe> rharper: thanks :)
<dustymabe> i'll get back to you guys in the ticket about the agreement. i have to step away for a bit right now and i'll be back later
<rharper> dustymabe: great, thanks
<smoser> blackboxsw, if you're bored. . i was looking at that branch. i tried to take the duplicated FakeExtendedTempFile
<smoser> take it out.
<powersj> smoser: if you can get to the string merge today, I can rebase the KVM merge and get another round of testing on it. https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/330535
<smoser> powersj, ok.
<akik> here's an improvement suggestion to the documentation. in http://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html it should be mentioned that ds=nocloud-net and seedfrom= should be separated with "\;". if you just use ";" the kernel parameter loses everything starting with ";"
<smoser> akik, i think that proably not true
<smoser> probably your shell is eating it
<akik> smoser: i just tested it
<akik> on centos 7.3
<smoser> akik, can you show how you did that ?
<akik> first i entered ds=nocloud-net;seedfrom=URL as the kernel parameter. ;seedfrom=URL was cut
<smoser> "entered"
<smoser> where
<akik> then i entered ds=nocloud-net\;seedfrom=URL and it stayed. in grub
<smoser> well, its probably grub that is eating it then. or if its grub-2 , possibly some of the magic make-menu stuff
<akik> it's grub2
<smoser> but i'm pretty sure the kernel doesnt care about a ';'
<akik> on centos it's split there without \
<akik> the rest goes *poof*
<rharper> it feels worth a Note or something in case it happens to someone else; ie, you may need to escape the semi-colon if your distro uses scripts to update the grub command line ?
<akik> rharper: no i mean for the *current* boot
<akik> press e to edit, ctrl-x to boot
<rharper> oh, that's grub itself
<rharper> the note may still apply (live editing or distro config tools)
<rharper> but as smoser said, the *kernel* doesn't care about semicolons
<rharper> the kerenl isn't running at the time you're editing the grub command line
<smoser> so yeah. in grub2, you're typing
<smoser> linux /something root=foo key=val ds=nocloud-net;other stuff
<akik> smoser: yes
<smoser> and grub p robably prints an error quickly and goes on as it read your ; as end of a command
<smoser> so it probably tried to execute 'other stuff'
<smoser> and failed
<smoser> i'd be surprised if you can't do:
<smoser>  linux /boot/kernel 'root=.... ds=;other stuff'
<akik> i checked /proc/cmdline after boot and it just had "ds=nocloud-net"
<smoser> yeah. grub ate it.
<smoser> same as if you type in a bash prompt
<smoser> $ echo foo;other stuff
<smoser> foo
<smoser> other: command not found
<akik> i'm not talking about shell. it's not the same
<rharper> https://www.gnu.org/software/grub/manual/grub/grub.html#Shell_002dlike-scripting
<nacc> akik: grub's shell is shell-like :)
<rharper> actually it is
<smoser> it is the same :)
<smoser> grub is parsing that and tokenizing on the ;
 * smoser has to step away for a bit
<cliffw> Trying to use cloud-init to setup aliases to eth0 using network configuration, details are being passed using EC2 user-data but nothing seems to work. The docs state user-data cannot be used to setup networking, but also say networking can be setup using data sources. EC2 is a valid data source. Any ideas?
<cliffw> ubuntu 16.04 by the way.
 * blackboxsw lobs cliffw the examples at http://cloudinit.readthedocs.io/en/latest/topics/network-config-format-v1.html wondering if you are referring to these docs which mention aliases?
<rharper> cliffw: for network-based datasources (like EC2); the datasource has to do the work to generate a network-config based on cloud metadata;  AWS/EC2 has this info but the DataSource is just becoming network aware; blackboxsw has been working on that;  we don't currently parse all of the EC2 metadata for generating a complete nework configuration.
 * rharper has to relocate
<cliffw> If I need to create a /etc/cloud/cloud.cfg.d/custom-networking.cfg can I do that in #cloud-config using write_files:
<cliffw> Just tried that, the config gets written out, but no network changes occur.
<akik> does cloud-init touch /etc/udev/rules.d/70-persistent-net.rules? i've set it to be immutable but the log says "failed stage init" after that. not sure if they're connected
<akik> i see that cloud-init version changed to 0.7.9 on centos
<smoser> akik, it does touch it yes.
<akik> smoser: how do i make it not touch it? i don't want the device names to change, ever
<smoser> cliffw, by the time that is read, its too late. cluod-init has already made the decision and rendered thenetworking configuraiton.
<smoser> akik, well, what cluod-init is doing is ensuring that they dont
<smoser> but you can disable cloud-init networking and it wont do such things
<smoser> # To disable cloud-init's network configuration capabilities, write a file
<smoser> # /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
<smoser> # network: {config: disabled}
<akik> so cloud-init does other stuff than what i direct it to do in user-data?
<akik> thanks for the config
<smoser> it has some default behaviors.  generally speaking it does the right thing for configuring networking
<smoser> thats the goal
<akik> smoser: if i disable cloud-init networking, is it still able to download if i use ds=nocloud-net?
<akik> download the user-data from that datasource
<blackboxsw> ok finally fixed up chef module unit tests and addressed review comments will await CI and then land it
<blackboxsw> grr, smoser, so we've moved tempdir out  into cloudinit.temp_util to avoid loading a ton of util for tempfile work. Now cc_chef adds util.subp_blob_in_tempfile  which depends on temp_utils.tempdir. Where should subp_blob_in_tempfile live?
<blackboxsw> s/grr/question/
<blackboxsw> :)
 * blackboxsw thinks it should remain definted in util and just locally import temp_utils.tempdir inside subp_blob_in_tempfile
<blackboxsw> ahh n/m we already import temp_utils in util anyway
<blackboxsw> ok /me runs away. please disregard
<cliffw> thanks smoser
<cliffw> Launching the EC2 instances in question from Cloudformation, so trying to have the network setup which Cloudformation handles at the EC2 level be handled also by cloud-init, so I don't have to hardcode config files or wait for a subsequent puppet run for eth0 aliases to be established.
<cliffw> have an EC2 with 5 IP addresses associated with it, as a default eth0 booting with DHCP only finds the primary in Ubuntu.
#cloud-init 2017-09-14
<smoser> cliffw, that's on our roadmap to accomplish
<smoser> the goal would be that you'd not have to do anything and it'd all just come up like magic.
<powersj> smoser: thanks for merge!
<powersj> kvm is looking better, I'll go over it one more time tomorrow and do some more testing: https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/327646
<blackboxsw> smoser: rharper, for tomorrow morning, I've queued up all changes in https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/330384. just looking for an official approve before I push/merge.
<powersj> tot doesn't build...
<powersj> blackboxsw: something is up with https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/330712
<powersj> tox passes fine
<powersj> but when I try building now I get: https://paste.ubuntu.com/25534973/
<blackboxsw> biulding deb or rpm?
<blackboxsw> deb
<blackboxsw> checking
<powersj> deb
<powersj> what I don't understand is why our CI didn't find these
<blackboxsw> agreed, same here
<powersj> are you able to reproduce locally?
<blackboxsw> powersj: nope. not with make deb
<blackboxsw> lemme try a direct call to packages/bddeb
<powersj> I use ./packages/bddeb
<powersj> well make deb failed for me as well :\
<blackboxsw> hrm. lemme look at that stack trace again
<blackboxsw> I'm thinking stale pyc files maybe, not sure
<blackboxsw> I have a fresh checkout though. hmm
<powersj> two unmocked errors
<powersj> same...
<blackboxsw> what's your version of httppretty
<blackboxsw> I'll check mine
<powersj>  git rev-parse HEAD
<powersj> f761f2b5f58c8cf13cfee63619f32046216cf66a
<powersj> ?
<blackboxsw> accd83c97fa3b3b928af2a7955febad410471fc2 for me
<blackboxsw> hold up... lemme check master
<powersj> accd isn't master
<blackboxsw> I think I was in the chef branch specificall
<blackboxsw> yeah
<blackboxsw> sorry was on a branch. checking now
<blackboxsw> hmm make deb  and ./packages/bddeb works for me there too
<blackboxsw> csmith@uptown:~/src/cloud-init (master)$ git rev-parse HEAD
<blackboxsw> f761f2b5f58c8cf13cfee63619f32046216cf66a
<powersj> hmm
<blackboxsw> Removing python3-httpretty (0.8.6-2)    # I'm dropping my xenial system packages to see if something changes
<blackboxsw> ohh wait unmet build dependency if I don't have that system package
<blackboxsw> powersj: dpkg-query --show python3-httpretty
<blackboxsw> python3-httpretty	0.8.6-2
<blackboxsw> I'm wondering if we are hitting an incompatibility in our build envs
<powersj> dpkg-query --show python3-httpretty
<powersj> python3-httpretty	0.8.14-1
<blackboxsw> I'm on xenial here, checking zesty
<blackboxsw> hmm zesty fails for a different reason
<blackboxsw> http://pastebin.ubuntu.com/25535133/
<powersj> xenial lxc fails with resizefs test issues
<powersj> yeah that ^
<powersj> I run artful, so was trying xenial in a lxc
<blackboxsw> ok I'll work master for those two branches on zesty and artful and see what we can sort
<blackboxsw> I think we are dealing w/ either differences in nose or differences in httpretty
<blackboxsw> for the system packages
<smoser> blackboxsw, you have a handle on that reesizefs test issues ?
<smoser> (sorry for pulling that :-(
<blackboxsw> smoser I'm looking at that specifically right now from within an lxc.
<blackboxsw> it's good, it passed CI too
<blackboxsw> but a system package in the lxc on zesty++ is probably behaving differently. I can reproduce w/ osetests3 /root/cloud-init/tests/unittests/test_handler/test_handler_resizefs.py cloudinit/config/cc_resizefs.py on a fresh zesty container
<powersj> smoser: KVM branch ready for another review https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/327646
<powersj> tests completed in description and updated commit msg
<blackboxsw> whoa smoser powersj  am I crazy?
<blackboxsw> .. don't answer that yet:    http://pastebin.ubuntu.com/25535264/
<dustymabe> hey smoser: https://code.launchpad.net/~dustymabe/cloud-init/+git/cloud-init/+merge/330701/comments/866238
<blackboxsw> os.access in a zesty container is returning True for W_OK on a read-only file
<blackboxsw> I need my eyes/work checked on that pastebin
<nacc> blackboxsw: i've seen odd behaviors with containers, what's the backing store?
<powersj> dustymabe: thanks for following up! Now smoser can get that one in
<blackboxsw> nacc: ext4
<nacc> blackboxsw: sorry, i meant the lxd storage pool
<nacc> oh it's on the ext4 disk?
<dustymabe> powersj: np
<dustymabe> :)
<dustymabe> btw do you guys have any idea when the next release of cloud-init will be?
<powersj> Next week :)
<blackboxsw> dustymabe: we are trying to cut branches to get them in by tomorrow for 0.7.10. yeah as powers says ;)
<blackboxsw> need a week of testing/validation
<dustymabe> sweet
<blackboxsw> and fixing the bugs I introduced :/
<dustymabe> :)
<powersj> https://lists.launchpad.net/cloud-init/msg00100.html
<powersj> There is the mailing list announcement
<blackboxsw> nacc: I'm sorry I'm dense on the lxc front. lxc info shows storage type dir for me
<blackboxsw>   storage: dir
<blackboxsw>   storageversion: ""
<smoser> dustymabe, powersj thanks. i'll pull.
<blackboxsw> nacc: but I don't think that's what you are asking. I'm trying to recall if I provided any unique storage pool setup on this install
<nacc> blackboxsw: ok
<smoser> dustymabe, and thanks again.  you did a really good job with that.  6 character change... but loads of good sluething and doc. thanks!
<nacc> blackboxsw: and on a xenial container, does the read/write work?
<nacc> blackboxsw: ah i wonder if the uid mappings mess with it too?
<nacc> blackboxsw: i guess it shouldn't, from within the container
<smoser> dustymabe, you would consider that this "fixes" https://bugzilla.redhat.com/show_bug.cgi?id=1490505
<ubot5> bugzilla.redhat.com bug 1490505 in cloud-init "cloud-init fails when calling `xfs_growfs /dev/mapper/atomicos-root` on Fedora Atomic Host" [Unspecified,New]
<smoser> right ?
<smoser> i'm going to c hange the commit message to include
<smoser> rhbz: https://bugzilla.redhat.com/show_bug.cgi?id=1490505
<smoser> as larsks had done in a commit of his
<blackboxsw> +1 smoser good to know on that. I was wondering whether we duplicate bugzilla bugs  in LP or how that works
<dpb1> blackboxsw: you can add a bugwatch in launchpad if you ever need to
<dustymabe> smoser: thanks!
<dustymabe> smoser: yes once we pull the patches in and rebuild i'll confirm
<dustymabe> but I tested it out in a locally built version and it worked fine
<smoser> dustymabe, right. i was just clarifying that you consider it to *fix* the bug rather than just be related to the bug.
<smoser> thanks. its 'tox'ing here and pushing
<dustymabe> +1
<smoser> blackboxsw, i dont really have a strong feeling on what to do
<smoser> i just knew that larsks had committed some things with a reference, and wanted to keep that going in case he was using anything to read such messages.
<powersj> smoser: "default to /srv/citest/kvm-nocloud or something." what about "/srv/images"?
<powersj> because that's what I already updated it to ;)
<smoser> powersj, well, then what happens when you run on the same system as curtin ?
<powersj> well I am pushing to /srv/images/server
<blackboxsw> ok so powersj smoser the 'make deb' 'packages/bddeb'  error I ran into with resizefs was because I was running as root.
<powersj> that used to work though
<blackboxsw> so our os.access(<path>, W_OK) always returns True regardless of file perms
<powersj> ah
<blackboxsw> I'll tweak the test to mock os.access return val
<blackboxsw> so we don't care if root runs it versus unpriv user
 * blackboxsw will also fix the OMNIBUS/chef bug if I can  in this branch
<powersj> smoser: ok so I'll go with mirror_dir: '/srv/citest/kvm-nocloud'
<blackboxsw> test bug
<powersj> smoser: as far as renaming the class you suggested class KVMNoCloudInstance right? 1) do this to the images, platform, and snapshot files as well? 2) do I need to rename the files as well?
<smoser> blackboxsw, so why did it used to work ?
<smoser> powersj, i think for now rename them all.
<smoser> one could argue that the images that you have could be generic for all kvm
<smoser> but then one could argue that they were generic for even more than kvm.
<powersj> right
<powersj> file name stay kvm.py?
<smoser> i say we rename to KVMNoCloud<Stuff> and then at a later date we can genericize them to just kvm.
<smoser> what do you think about the file ?
<smoser> i think at some point we're going to say "this should get renamed"
<powersj> part of me says be verbose now, and make filename as well kvmnocloud
<smoser> i think that makes sense.
<powersj> ok I'll update the image dir, filenames and class names and resubmit
<powersj> smoser: one more, on the CLI when we invoke which backend, should it also be kvmnocloud or remain kvm?
<smoser> bikesheds are fun
<smoser> so, now i'm thinking... sorry.
<powersj> lol
<smoser> NoCloudKVM
<smoser> as i think it makes sense to have the platform first
<smoser> as the backend is somewhat arbitrary
 * powersj will do pretty much whatever smoser says at this point 
<powersj> do whatever*
<smoser> ie, we could potentially have an AWSInstanceStore
<smoser> and AWSEbs
<powersj> smoser: ok and the platform name for the cli? nocloudkvm as well?
<powersj> or just kvm
<powersj> I'm thinking nocloudkvm
<smoser> i think 'nocloud' there
<smoser> yeah
<smoser> or nocloud-kvm
<powersj> nocloud-kvm it is (I like the best)
<powersj> smoser: creating a dir in /srv requires root?
<powersj> I guess my past runs have worked using /srv/images since it was already there and setup.
<powersj> but if someone runs this the first time it will fail
<smoser> powersj yeah.
<smoser> we can make it ~/citest
<smoser> default to that.
<smoser> curtin defaulted to the /srv path
<nacc> yeah i did somethingn similar on my home machinne
<smoser> as we'd like them generally shared
<nacc> for the purpose of testing as my user
<smoser> but make it configurable
<smoser> (environment is "configurable" enough for me)
<smoser> that is what curtin uses
<powersj> smoser: it is configurable in the yaml file, do you want more than that?
<smoser> ah. thats fine enough.
<powersj> ok
<smoser> but /tmp not ok.
<powersj> pushed renames and that fix to https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/327646
<smoser> powersj, ok. i'm reviwing 'collect-logs' from blackboxsw now. then to you
<powersj> ok thanks
<blackboxsw> powersj: man, I'm on an artful container ./packages/bddeb still succeeds for me
<blackboxsw> no chef unit test failures
<blackboxsw> I had run 'make ci-ubuntu-deps' first though
<blackboxsw> https://www.irccloud.com/pastebin/1tJ4ifK7/
<powersj> blackboxsw: trying a artufl container now
<powersj> blackboxsw: agreed, so something on my local system is causing it
<smoser> how's it fail on your system ?
<blackboxsw> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/330774 here's the fix for leaking os.access as root user
<powersj> smoser: https://paste.ubuntu.com/25535666/
<powersj> Using python3-httpretty 0.8.14-1
<blackboxsw> hrm
<smoser> blackboxsw, question at https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/330774
<blackboxsw> eww. right-o
<blackboxsw> smoser: I think you are right. we'd always see it as writeable in cloud-init
<blackboxsw> so, I need to fix that logic
<blackboxsw> fix or drop... hmm /me attempts to specify resize user-data to lxc to watch it break
<smoser> blackboxsw, well, it will say it can't tehre
<smoser> in a container
<smoser> hm..
<smoser> well, it doesnt seem to do that well (looking at /var/log/cloud-init from a container)
<blackboxsw> smoser: I'm not seeing the check that would trigger that rejection
<smoser> ?
<blackboxsw>     if not os.access(devpath, os.W_OK):   # will never be true right?
<blackboxsw> because we are root
<blackboxsw> Device '/dev/mapper/ubuntu--vg-root' did not exist in container. cannot resize: dev=/dev/mapper/ubuntu--vg-root mnt_point=/ path=/
<blackboxsw> ahh
<smoser> http://paste.ubuntu.com/25535747/
<smoser> yeah. ends up getting there somewhere.
<blackboxsw> so we don't get down that far
<blackboxsw> we bail in the ENOENT handling
<blackboxsw> ok so that said, container logic below line 197 is probably unused paths right?
<blackboxsw> only thing non-container  read-only devpath, which likely can't be triggered as root user
<smoser> well, it'd mostly be bogus path.
<smoser> blackboxsw, in an iscsi root
<smoser> with overlayroot
<smoser> (like maas)
<smoser> we see
<smoser> 2017-09-14 19:33:42,855 - cc_resizefs.py[WARNING]: Device 'overlayroot' did not exist. cannot resize: dev=overlayroot mnt_point=/ path=/
<smoser> http://paste.ubuntu.com/25535835/
<smoser> oh. hmm.. but that is 'overlayroot' that it finds as its dev.
<smoser> oh. hmm.. but that is 'overlayroot' that it finds as its dev.
<blackboxsw> strange, I'm not quite sure what we should do here https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/330774/comments/866258
<smoser> blackboxsw, ok. i verified that the check for writable is pointless
<smoser> as root at least
<smoser> in an overlay root on read-only iscsi
<smoser> i just hacked and changed devpath to be /dev/disk/by-label/cloudimg-rootfs
<blackboxsw> in that case what was the dev path perm?
<blackboxsw> 400?
<blackboxsw> s/perm/mode
<smoser> http://paste.ubuntu.com/25535925/
<smoser> so yeah, that check is pointless
<smoser> in the overlayroot path, it norlaly gets 'overlayroot' as its root device
<smoser> from /proc/1/mounts
<smoser> and that doesnt exist
<smoser> so it warns
<smoser> hack that so it used /dev/sda1 (the actual device on the filesystem)
<smoser> and it sees its a overlayroot filesystem type
<smoser> which it doesnt know how to resize
<smoser> hack *that* so it gets ext4
<smoser> and in this particular case, resize2fs realizes there wasnt a resize to be done :)
<smoser> so it just exited
<blackboxsw> gotcha ok, so we can drop the os.access check then right?
<blackboxsw> that whole conditional
<blackboxsw> pushed the update dropping that check
<blackboxsw> --force pushed
<blackboxsw> smoser: is that description valid? https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/330774
<smoser> blackboxsw, i have to run.
<smoser> but i had a ahah
<blackboxsw> see ya smoser have a good one
<smoser> i remembered one other time i tried to test if something was writable
<smoser> and the filesystem actually was lying to me
<smoser> if you want to test if something is writable , you can say:
<smoser>  hey filesystem can i write to that file?
<smoser> (thats what we're doing now)
<smoser> or you can open it for writing
<smoser> and not write anything
<smoser> the second path gives you more reliable data :)
<smoser> i think probably safe if you do that notrunc and then dont write anything.
<blackboxsw> sorry 1:1 just ended
<blackboxsw> "i think probably safe if you do that notrunc and then dont write anything."  I parsed that safe, if you do 'that' (perform the os.access call?) on trunk and don't write anything
#cloud-init 2017-09-15
<smoser> blackboxsw, well... i was saying instead of doing os.access, you just do
<smoser> open(devpath, "wb", ... some flags that mean dont truncate)
<smoser> but now that i think about it that likely causes issues.
<smoser> as udev or systemd watch open/close on devices.
<smoser> so bad idea
<smoser> ugh https://bugs.launchpad.net/cloud-init/+bug/1717477
<ubot5> Ubuntu bug 1717477 in cloud-init "cloud-init generates ordering cycle via After=cloud-init in systemd-fsck" [High,New]
<dpb1> :/
<blackboxsw> gah
<larsks> Yay ordering cycles!
<blackboxsw> yeah, larsks scott's pushing  a fix for this release && an SRU update on that today.
<smoser> blackboxsw, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/330846
<smoser> ordering cycles are like a game of russian roulette
<blackboxsw> smoser: checking, just pushed https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/330626
<blackboxsw> we'd just need to carry the following for series we want to deliver apport functionality to http://paste.ubuntu.com/25541681/
<blackboxsw> adding it as a comment on the branch
<smoser> blackboxsw, does it know that it is python somehow ?
<smoser> that is good though
<blackboxsw> Oops missed the response yeah just tested apport knows what python to use. So I dropped the bang py3 I'll push another unit test for optional --include-userdata to collect logs
<blackboxsw> Smoser
<smoser> ok. reading. please read my branch
<smoser> blackboxsw,
<smoser> a quick read of https://code.launchpad.net/~rbalint/cloud-init/+git/cloud-init/+merge/330842
<smoser> woudl be appreciated too
<smoser> blackboxsw,
<smoser> http://paste.ubuntu.com/25542078/
<blackboxsw> checking
<blackboxsw> +1
<smoser> on a overlayroot boot (maas), we see this:
<smoser> 14 19:45:28,372 - cc_resizefs.py[WARNING]: Device 'overlayroot' did not exist. c
<smoser> annot resize: dev=overlayroot mnt_point=/ path=/
<blackboxsw> want me to fold that into the drop patch?
<smoser> yeah.
<smoser> i'd like to do that so that we dont WARN there
<smoser> but just debug
<blackboxsw> makes sense
<blackboxsw> adding a unit test on that aspect
<blackboxsw> unit test added and pushed
<blackboxsw> tests-fix-root-os-access-leak 1367b2b911d61ae95c2c635259b2f2b85c8eb541
<blackboxsw> back on your branch
<blackboxsw> flake errors https://jenkins.ubuntu.com/server/job/cloud-init-ci/309/console
<smoser>  fixing flake errors
<smoser> done
<smoser> blackboxsw,
<smoser> $ git show 1367b2b911d61ae95c2c635259b2f2b85c8eb541:cloudinit/config/cc_resizefs.py | grep overlay || echo no overlay mention there
<smoser> no overlay mention there
<smoser> maybe you didn't push that ?
<blackboxsw> smoser: wrong log. was looking at the os.access branch git log
<blackboxsw> hmm
<blackboxsw> wait a sec
<blackboxsw> hah didn't commit
<blackboxsw> e19de87fdae9fb58d39ac415b73fa37e6b6e1e83 pushed sorry about that smoser
<blackboxsw> almost done w/ your dhclient branch
<blackboxsw> review comment posted https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/330846
<smoser> blackboxsw, ok. i addressed that and pushed
<smoser> and i commented on https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/330626
<blackboxsw> smoser: I saw you landed resizefs (overlayroot) thx
<blackboxsw> TIL: open a trello card and ctrl-V a link and it auto attaches
<smoser> ?
<dpb1> oh cool tip
<blackboxsw> I had always clicked the paper clip icon, then ctrl-v in the textbox labeled link
<blackboxsw> and then clicked attach. But ctrl-v just does the right thing
<smoser> blackboxsw, i linked a mp to
<smoser> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/330626
<blackboxsw> smoser: you mean as prerequisite?
<smoser> well post req
<smoser> i guess.
<smoser> mine depends on yours
<blackboxsw> powersj: a couple branches ago your nose-timer additions saved me. thanks again. it alerted me to a problem w/ a unit test that I didn't know I had
<powersj> Woohoo!! Thanks for the feedback
<blackboxsw> retreis on url_helper.read_url are costly (waiting a full second between retries) so my unit test was taking 20 seconds on retries (so I mocked it) :)
<blackboxsw> well looking at the code, I guess I could have set sec_between parameter to 0 :)
<blackboxsw> ahh well, I'll drop the mock next time
<blackboxsw> but in either case I would have unknowingly introduced 20 seconds into our run if tox -e py3 hadn't shown me the "red" tests
<blackboxsw> less mocks == a better world
<blackboxsw> landed https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/330846
<blackboxsw> waiting on CI for https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/330626 then I'll land the collect-logs/apport stuff
<smoser> blackboxsw, shoot.
<smoser> i have that locally about to go
<smoser> but i can cancel
<smoser> here.
<smoser> blackboxsw, i'm landing it.
<smoser> tox && git push upstream HEAD
<smoser> i re-worked the commit message a bit
<smoser> http://paste.ubuntu.com/25542524/
<blackboxsw> smoser: sorry per your branch
<blackboxsw> ohh you meant my collect-logs
<blackboxsw> +1 on commit message
<blackboxsw> smoser:
<smoser> blackboxsw, can you give my https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/330858 a sanity check ?
<smoser> or a test ?
<smoser> an then i'd like to shove a new upload to artful
<smoser> hm..
<smoser> why isnt that packaging my apport-launcher.py
<blackboxsw> smoser: added this https://trello.com/c/2FXn8S9x/385-centos-7-fix-cloudstack-datasource-to-regression-to-handle-dhclient-lease-filenames for hilight
<blackboxsw> checking smoser
<blackboxsw> powersj: this afternoon, I'm working on that trello-board  script for processing cards
<blackboxsw> once these branches land
<smoser> ugh.
<smoser> i need to run shortly.
<smoser> i uploaded to xenial and to zesty the regression revert cherry-pick
<smoser> dpb1, ^
<smoser> but we still need an upload to artful
<powersj> blackboxsw: sweet
<blackboxsw> smoser: content looks good on https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/330858 don't know how I missed that
<blackboxsw> powersj: https://trello.com/c/zGxaIodz/372-tests-support-command-as-string is merged right? it's in doing lane
<smoser> blackboxsw, ok. i'm adding
<dpb1> smoser: xenial and zesty are indeed prio
<dpb1> thx
<dpb1> smoser: whats stopping the artful upload?
<smoser> landing those other things
<smoser> i'md oing build && upload && push on it right now
<dpb1> great
<smoser> ok. thats pushed.
<smoser> i have to run
<smoser> the one thing left for right now is to update the ubuntu/xenial and ubuntu/zesty branches to drop (remove) the cherry pick patch
<smoser> as right now our recipe builds of those branches will fail because they have a patch in them that wont apply to trunk.
<smoser> if someone wanted to try to thikn of a way to make us not have to do that... that'd be good.
<smoser> if we could someohow get the recipe builds to drop all 'cherry-pick-*' patches.
<smoser> i'll check back in later
<powersj> blackboxsw: yes landed
<powersj> as is KVM!
<powersj> \o/
<blackboxsw> good push smoser !
<blackboxsw> powersj: https://trello.com/c/XGGri1K2/315-sru-get-oil-1   and https://trello.com/c/MMRiKIL9/316-sru-get-maas-1   are these ongoing cards?
<blackboxsw> if you can't tell I'm scrubbing the board :)
<powersj> blackboxsw: not on going. working to get sign off on the SRU exception
<powersj> see updates to: https://wiki.ubuntu.com/CloudinitUpdates
<Odd_Bloke> Is https://pastebin.canonical.com/198539/ a known issue?
<Odd_Bloke> Oh, hmph, internal pastebin.
<Odd_Bloke> http://paste.ubuntu.com/25542802/
<blackboxsw> hrm
<blackboxsw> checking
<blackboxsw> Odd_Bloke: don't see any outstanding bugs related to that
<Odd_Bloke> Filed https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1717598
<ubot5> Ubuntu bug 1717598 in cloud-init (Ubuntu) "Traceback when passing user-data on GCE" [Undecided,New]
<blackboxsw> what version of cloudinit? I know we toucheds cloudinit/stages.py today, and a couple of config modules, but nothing specifically in the handlers code.
<blackboxsw> ahh checking the bug thanks
<blackboxsw> I have touched bootcmd module recently
<blackboxsw> Sept 2nd in fact
<blackboxsw> ok. digging
<blackboxsw> thx Odd_Bloke
<Odd_Bloke> blackboxsw: An artful image built today.
<Odd_Bloke> So 0.7.9-267-g922c3c5c-0ubuntu1, I believe.
<blackboxsw> thanks again will work on that
<smoser> :-(
<blackboxsw> strange, can't reproduce w/ rev 267  on ec2 with basically the same user-data http://paste.ubuntu.com/25543349/
<blackboxsw> cloud-init	0.7.9-267-g922c3c5c-0ubuntu1
<blackboxsw> though just found another bug in Ec2Local
<blackboxsw> cloudinit.util.ProcessExecutionError: Unexpected error while running command.
<blackboxsw> Command: ['/run/cloud-init/tmp/cloud-init-dhcp-jlm8h6x2/dhclient', '-1', '-v', '-lf', '/run/cloud-init/tmp/cloud-init-dhcp-jlm8h6x2/dhcp.leases', '-pf', '/run/cloud-init/tmp/cloud-init-dhcp-jlm8h6x2/dhclient.pid', 'eth0', '-sf', '/bin/true']
<blackboxsw> Exit code: -
<blackboxsw> Reason: [Errno 13] Permission denied
<blackboxsw> so we are now getting perm errors trying to run dhclient. I'll debug before fiiling this one
<blackboxsw> Open a socket for LPF: Operation not permitted.... even with apparmor disabled.  hmm
#cloud-init 2017-09-16
<blackboxsw> hmm looks like I can't run dhclient from /run/cloud-init/tmp/something/dhclient
<blackboxsw> http://pastebin.ubuntu.com/25544142/
<smoser> tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=378640k,mode=755 0 0
<smoser> noexec
<smoser> :-(
<smoser> its fallout of https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1707222
<blackboxsw> yeap
<ubot5> Ubuntu bug 1707222 in cloud-init "usage of /tmp during boot is not safe due to systemd-tmpfiles-clean" [High,Confirmed]
<blackboxsw> sooo where to store our tmp files
<smoser> wouldn't it be neat if there was a directory named "/tmp" or something
<blackboxsw> temp_utils dumps them under /run/cloud-init which breaks our dhcp ec2local
<blackboxsw> heh
<blackboxsw> :)
<blackboxsw> yeah I'm testing passing dir='/tmp' right now
<blackboxsw> to temp_utils.tempdir
<smoser> well, that'll work, but possibly have your directory deleted
<blackboxsw> right might have a race still on the cleanup
<blackboxsw> could drop it in /etc :)
<blackboxsw> nasty nasty
<smoser> i think /var/tmp might be an option.
<blackboxsw> not a bad idea
<smoser> not sure. have to look at systemd-tmpfiles-clean
<blackboxsw> : SUCCESS: found local data from DataSourceEc2Local
<blackboxsw> ok so using dir=/tmp works, I'll try a reboot w/ /var/tmp
<blackboxsw> smoser: per unit tests in systemd "        # files in /var/tmp/ older than 30d should get cleaned up
<blackboxsw> "
<blackboxsw> checking the code now
<blackboxsw> looks to be commented out in /usr/lib/tmpfiles.d/tmp.conf in artful #q /var/tmp 1777 root root 30d
<blackboxsw> ok dhclient runs Ec2Local work from /var/tmp
<blackboxsw> smoser: for monday https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/330875
<blackboxsw> gotta run
<smoser> k. i'll ask xnox for review too
<smoser> need to file a bug
<smoser> i will
<smoser> you go
<smoser> later
<smoser> bug https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1717627
<ubot5> Ubuntu bug 1717627 in cloud-init (Ubuntu) "permission denied when executing dhclient in Ec2 datasource" [High,Confirmed]
<blackboxsw> thanks smoser, so maybe my patch should really just be http://paste.ubuntu.com/25544275/
<blackboxsw> shouldn't we change where our temp_utils default tmp dir is?
<smoser> i dont know.
<smoser> its obnoxious
<blackboxsw> yeah it is.
<smoser> i can reproduce the GCE one
<smoser> its failure of some of my changes for GCE metadata servvice
<blackboxsw> thanks, what'd I break?
<smoser> the 'add a main'
<blackboxsw> ahh
<blackboxsw> Added https://trello.com/c/qDqAXygV/387-bug-gce-datasource-doesnt-run-bootcmd-config
<blackboxsw> Ok outta here
<blackboxsw> Have a good one
<smoser> blackboxsw, https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1717598 has a branch for you at your leisure
<ubot5> Ubuntu bug 1717598 in cloud-init (Ubuntu) "Traceback when passing user-data on GCE" [High,In progress]
<dknight87> hi guys, how can I usee masterless puppet and cloudinit to provision aws servers?
<blackboxsw> dknight87.  Puppet-apply is how most folks run puppet locally. Any config keys and. Values provided under the puppet: conf: key get written into /etc/puppet/puppet.conf so you could specify where your local manifests etc live and have a cron job run that runs puppet-apply for you.  I thought this tutorial was fair
<blackboxsw> https://www.digitalocean.com/community/tutorials/how-to-set-up-a-masterless-puppet-environment-on-ubuntu-14-04
<blackboxsw> The puppet: conf: keys I mentioned would be in your #cloud-config user-data file you provide at instance launch time. http://cloudinit.readthedocs.io/en/latest/topics/modules.html#puppet
#cloud-init 2017-09-17
<SSD> Hi all!
<SSD> new to cloud init and trying to figure out if it suits my use case
<SSD> the use case is I need to create a base qemu image for other scientists to use
<SSD> therefore I need to install all the necessary packages into an image and give it to them
<SSD> If I am to use cloud init, are those changes persistent?
<SSD> basically, do I have to run the cloud-config on the image everytime or can I get a base image with all the packages installed after the first run
<SSD> FYI: I'm running on the no cloud mode (providing my own meta-data)
<SSD> Thanks in advance
#cloud-init 2019-09-09
<Odd_Bloke> I'm looking at https://bugs.launchpad.net/cloud-init/+bug/1843221 for triage, and I'm not sure what to recommend the user do.  They're using NoCloud, so I think they probably could specify network config, but am I missing a way for a user without that option to configure DNS in a system?
<ubot5> Launchpad bug 1843221 in cloud-init "manage_resolv_conf not working in ubuntu18.04 cloud images" [Undecided,New]
<smoser> Odd_Bloke: i think that was teh right answer
<Goneri> hey, could you please take a look at https://code.launchpad.net/~goneri/cloud-init/+git/cloud-init/+merge/368507 and https://code.launchpad.net/~goneri/cloud-init/+git/cloud-init/+merge/365641
<Odd_Bloke> rharper: Looks like you've reviewed those two branches previously.
<blackboxsw> annnd it's that time for real: cloud-init status meeting.
<blackboxsw> #startmeeting Cloud-init bi-weekly status
<meetingology> Meeting started Mon Sep  9 16:26:10 2019 UTC.  The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology.
<meetingology> Available commands: action commands idea info link nick
<blackboxsw> Hey folks, welcome to the ~biweekly cloud-init status meeting.
<blackboxsw> #chair rharper Odd_Bloke
<meetingology> Current chairs: Odd_Bloke blackboxsw rharper
<Odd_Bloke> o/
<blackboxsw> cloud-init upstream uses this meeting as a platform for community updates, feature/bug discussions, and an opportunity to get some extra input on current development.
<blackboxsw> Feel free to interject at any time. Our typical format is the following:  Previous Actions, Recent Changes, In-progress Development, Office Hours (~30 mins)
<blackboxsw> Last meeting's minutes live here:
<blackboxsw> #link https://cloud-init.github.io/status-2019-08-19.html#status-2019-08-19
<blackboxsw> #topic Previous actions
<blackboxsw> no actions from last meeting so we'll plow right through to Recent Changes
<blackboxsw> #topic Recent Changes
<blackboxsw> The following branches have landed in tip since last meeting: via    git log --since 2019-08-19
<blackboxsw>     - doc: document doc, create makefile and tox target [Joshua Powers]
<blackboxsw>     - .gitignore: ignore files produced by package builds [Daniel Watkins]
<blackboxsw>     - docs: fix whitespace, spelling, and line length [Joshua Powers]
<blackboxsw>     - docs: remove unnecessary file in doc directory [Joshua Powers]
<blackboxsw>     - Oracle: Render secondary vnic IP and MTU values only [Ryan Harper]
<blackboxsw>     - exoscale: fix sysconfig cloud_config_modules overrides
<blackboxsw>       [Chad Smith] (LP: #1841454)
<ubot5> Launchpad bug 1841454 in cloud-init "Exoscale datasource overwrites *all* cloud_config_modules" [Undecided,Fix committed] https://launchpad.net/bugs/1841454
<blackboxsw>     - net/cmdline: refactor to allow multiple initramfs network config sources
<blackboxsw>       [Daniel Watkins]
<blackboxsw>     - ubuntu-drivers: call db_x_loadtemplatefile to accept NVIDIA EULA
<blackboxsw>       [Chad Smith] (LP: #1840080)
<ubot5> Launchpad bug 1840080 in cloud-init (Ubuntu) "cloud-init cc_ubuntu_drivers does not set up /etc/default/linux-modules-nvidia" [High,Fix released] https://launchpad.net/bugs/1840080
<blackboxsw>     - Add missing #cloud-config comment on first example in documentation.
<blackboxsw>       [Florian MÃ¼ller]
<blackboxsw>     - ubuntu-drivers: emit latelink=true debconf to accept nvidia eula
<blackboxsw>       [Chad Smith] (LP: #1840080)
<blackboxsw>     - DataSourceOracle: prefer DS network config over initramfs
<blackboxsw>       [Daniel Watkins]
<blackboxsw>     - format.rst: add text/jinja2 to list of content types (+ cleanups)
<blackboxsw>       [Daniel Watkins]
<blackboxsw>     - Add GitHub pull request template to point people at hacking doc
<blackboxsw>       [Daniel Watkins]
<blackboxsw> Additionally: we have also cut a stable-18.4 branch from the 18.4 tag as our last supported python2.6 branch. There will be an email sent out to the mailing list about the intent of this branch.  It requires a couple of minor fixes to make sure py2.6 support is functional, but this will be reference branch for any distribution that does not have access to py.27 or later.  No additional feature development is
<blackboxsw> planned on stable-18.4
<blackboxsw> a reminder again that python2.6 support was 'dropped' in cloud-init upstream as of the 18.4 release, so expectations for py2.6 support stopped  in 18.4 and there is a deprecation plan for py 2.7 as well
<blackboxsw> #link https://lists.launchpad.net/cloud-init/msg00170.html
<blackboxsw> Again, see the mailinglist for details and updates
<blackboxsw> #link https://lists.launchpad.net/cloud-init/
<blackboxsw> #topic In-progress Development
<blackboxsw> Last week or so the team has been working on SRU validation for cloud-init 19.2.24 into Xenial, Bionic and Disco.
<blackboxsw> We have passed all SRU validation tests and our expected pubish date for 19.2.24 is tomorrow for those Ubuntu series
<blackboxsw> good work on validation folks
<blackboxsw> and thanks for extra cloud-init community verification from exoscale, azure and VMware for validation efforts
<blackboxsw> There is additional Oracle, FreeBSD and Azure work in flight at the moment as well as some boot speed improvements and analysis from rharper
<blackboxsw> The following link represents any carded work upstream is  tracking. The Doing lane is content or features we expect to land shortly
<blackboxsw> #link https://trello.com/b/hFtWKUn3/daily-cloud-init-curtin
<blackboxsw> Now is probably a good time to also mention that our entire ubuntu server team also reflects our weekly accomplishements over in the ubuntu-server discourse. If there are deeper discussions or questions on various topics or features please join us there as well
<blackboxsw> #link https://discourse.ubuntu.com/c/server
<blackboxsw> I think that about wraps it for in-progress development
<blackboxsw> #topic Office Hours (next ~30 mins)
<blackboxsw> upstream cloud-init devs will have eyes on this channel for any discussions, questions, bugs or feature work the greater community would like to discuss.
<blackboxsw> During this time, we'll also groom our activereview queue to make sure we try to get review comments out to devs who have active branches.
<blackboxsw> Again, thanks for tuning in
<blackboxsw> Ok just addressed review comments on https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/372432 .   I'm reviewing https://code.launchpad.net/~goneri/cloud-init/+git/cloud-init/+merge/368507
 * blackboxsw also sets the next meeting topic so we don't forget.
<blackboxsw> Odd_Bloke: rharper powersj, I *think* we decided to shift from Mondays to Tuesdays for status meetings to avoid collisions with holidays, vacation work travel etc. Are we doing that for next status meeting, or maybe waiting to discuss that more broadly?
<Odd_Bloke> Tuesday in two weeks is likely to be a travel day for anyone heading to the cloud-init summit.
<Odd_Bloke> But Monday is likely to be a swap day for Canonical folks because we're all travelling next week too.
<blackboxsw> hrm right, maybe we wait then and discuss at the summit
<Odd_Bloke> So I would perhaps suggest skipping the next meeting, and then we can resume on Tuesdays two weeks after the summit?
<blackboxsw> discuss scheduling changes that is
<blackboxsw> sure, let's push/postpone until summit +2 weeks
* blackboxsw changed the topic of #cloud-init to: Reviews: http://bit.ly/ci-reviews | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting Oct 8 16:15 UTC | cloud-init v 19.2 (07/17) | https://bugs.launchpad.net/cloud-init/+filebug
<blackboxsw> #action blackboxsw send email to the list notifying of status meeting day change.
<meetingology> ACTION: blackboxsw send email to the list notifying of status meeting day change.
<rharper> +1 Odd_Bloke
<blackboxsw> Also note that the version of cloud-init that has undergone SRU verification is also published to our copr el-testing repo. We only update that repo during upstream cloud-init releases XX.YY and any Ubuntu SRUs so it is much more stable than our daily copr repo.
<blackboxsw> #link https://copr.fedorainfracloud.org/coprs/g/cloud-init/el-testing/
<blackboxsw> I think that about wraps our cloud-init status meeting for today. I'm wrapping up my review here and will post it to the set_passwords branch.
<blackboxsw> #endmeeting
<meetingology> Meeting ended Mon Sep  9 17:33:31 2019 UTC.
<meetingology> Minutes:        http://ubottu.com/meetingology/logs/cloud-init/2019/cloud-init.2019-09-09-16.26.moin.txt
<blackboxsw> minutes published to https://cloud-init.github.io/status-2019-09-09.html#status-2019-09-09
<blackboxsw> thanks folks
<Odd_Bloke> blackboxsw: Thanks for running the meeting!
<blackboxsw> Odd_Bloke: no prob
<blackboxsw> rharper: Goneri I finished my review of https://code.launchpad.net/~goneri/cloud-init/+git/cloud-init/+merge/368507 a couple comments/suggestion inline  for you both to weigh in on. thanks again
<smoser> Odd_Bloke: sorry, do we use 'Fixes LP: #XXXXX' now ?
<blackboxsw> I'm +1 on that approach but had a couple of suggestions
<smoser> https://code.launchpad.net/~daniel-thewatkins/cloud-init/+git/cloud-init/+merge/372491
<smoser> in your commit message. versus just ^LP: #XXXXX
<blackboxsw> smoser: Odd_Bloke haha that is because Dan's mind is doing github branches for ubuntu-advantage-tools
<blackboxsw> cloud-init should be just LP: #XXX
<Goneri> blackboxsw, great, I will address that ASAP.
<Odd_Bloke> Yeah, that's GitHub leaking.
<ozzzo> When I install cloud-init on RHEL 5.7 it throws URL errors and then fails:
<ozzzo> 2019-09-06 14:42:28,584 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [3/120s]: url error [[Errno 113] No route to host]
<ozzzo> In /etc/rc.d/init.d/cloud-init I see it trying to read /etc/sysconfig/cloud-init which doesn't exist
<ozzzo> does that mean I screwed up the install?
<Odd_Bloke> ozzzo: Where did you install it from?
<ozzzo> epel
<ozzzo> rpm -ivh /u/yesuv/Downloads/epel-release-5-4.noarch.rpm
<ozzzo> yum install -y cloud-init
<Odd_Bloke> ozzzo: What does `cloud-init --version` give you?
<ozzzo> [root@image-qscl-rh57 sysconfig]# cloud-init --version
<ozzzo> bad command --version. use one of ('start', 'start-local')
<Odd_Bloke> OK, I think that means that's an ooold version of cloud-init (which probably isn't a surprise in RHEL 5! :).
<blackboxsw> yeah even 0.7.9 has a cloud-init --version command :/
<ozzzo> is 5.7 too old to run cloud-init?
<ozzzo> we're being asked to provide 5.7 VMs to save developers from having to upgrade their code
<Odd_Bloke> The honest answer from me is that I'm not sure.
<ozzzo> What problem is it reporting? Does cloud-init require some kind of support service that works across the 169 network?
<ozzzo> can I fix it by setting up something to answer the metadata request?
<Odd_Bloke> Well, it's trying to talk to the Amazon EC2 metadata servers, and not able to route to them.
<Odd_Bloke> But, honestly, the version you're running is so old that I don't know what else is supported.
<blackboxsw> ozzzo: if you are trying to buildimages might want to look at building or installing the latest cloud-init RPMs in https://copr.fedorainfracloud.org/coprs/g/cloud-init/el-testing/   but yeah per Odd_Bloke's comment, your image isn't contacting Ec2 urls for some reason
<rharper> I suspect it's missing networking config in the image;  you could try our cloud-init in the el-stable copr repo,
<rharper> blackboxsw: I think for RHEL5.x they might need the older release
<Odd_Bloke> Yeah, I think RHEL5 is Py 2.4 or Py 2.5.
<blackboxsw> as in pre-py26?
<blackboxsw> wow
<ozzzo> I'm not in AWS; is this early version of cloud-init designed to only work in AWS?
<rharper> that's known to work on Centos6;  might be old enough, but have the networking features needed to bring up network for an ec2 crawl
<ozzzo> we're setting up openstack VM images
<rharper> ozzzo: no, there are many datasources, openstack also uses the same metadata URL
<rharper> I suspect that the image doesn't have any build-in network config (like a /etc/sysconfig/network-scripts/ifcfg-eth0 which is configured to dhcp);
<ozzzo> I don't think I can reach a 169 IP across the internet; that network doesn't route
<ozzzo> it seems like I would need a server on my local network to answer that request
<rharper> and so cloud-init will try to talk (over network) to different endpoints to determine if it's running on a particular cloud type
<rharper> there are other datasources, like NoCloud which can use a localally attached disk with configuration in it
<ozzzo> so I can setup nocloud and then configure cloud-init to use that instead of 169?
<rharper> yes, https://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html
<Odd_Bloke> Well, in an OpenStack cloud there should still be something on 169.254.169.254, right?
<Odd_Bloke> Or it should be using ConfigDrive.
<rharper> I suspect ozzzo is booting outside of the cloud
<Odd_Bloke> ozzzo: Are you testing your images in the environment you expect them to run, or just locally?
<rharper> to test the image , is that right ?
<ozzzo> This is a VM image; I'm just building it
<ozzzo> yes, in the VM environment, there may be something listening on 169
<ozzzo> we're building the images on VMWare
<rharper> you may find that the image will work fine inside the openstack environment;  testing your local imge with NoCloud is a reasonable first step to see if cloud-init you put in the image is working.
<ozzzo> ok I'll try it, thanks for the advice
<rharper> Odd_Bloke: blackboxsw: I can't sort out why the autoland is failing on pycodestyle ... thoughts?  https://jenkins.ubuntu.com/server/job/cloud-init-autoland-test/315/console
<rharper> I've run tox and tox -e tip-pycodestyle  with no issues;  it's complaining about 3 new lines where there should be two, but the file in the repo is good to go
<Odd_Bloke> Yeah, I can't see why either.
<Odd_Bloke> rharper: Are you on bionic?
<Odd_Bloke> (I'm wondering if it's a Py 3.6 issue?)
<rharper> no, xenial
<rharper> but I can try on bionic
<ozzzo> we're running cloud-init 0.6.3-0.12.bzr532.el5
<ozzzo> I'm going to try ignoring the cloud-init error on the image, finish building it, and then build OS VMs to see if cloud-init works there
<rharper> Odd_Bloke: I needed to rebase, now I see it and removing the extra line, force push should fix here shortly
<Odd_Bloke> \o/
<Odd_Bloke> So the extra line was showing up in the merge commit?
<rharper> yes
<rharper> I didn't realize I was not rebased to tip
<Odd_Bloke> Wow, good job on finding that.
<Odd_Bloke> I think I'd just be unconscious from banging my head on things.
<rharper> I was for a bit in a bionic container
<rharper> but now I've got a bitter headache in #curtin
<rharper> *sigh*
<rharper> bigger (and bitter)
<rharper> blackboxsw: ok, I re-approved my netfailover branch after the force push, hoping that the autolander will be happy now
#cloud-init 2019-09-10
<rharper> https://code.launchpad.net/~xiaofengw/cloud-init/+git/cloud-init/+merge/372469
<rharper> Odd_Bloke: blackboxsw: what's your thought on the last test-case, it's one of those #XXX: parameterize me scenarios;  worth asking for separate test_ for each input scenario or OK as it is ?
<Goneri> blackboxsw, It should be ok now: https://code.launchpad.net/~goneri/cloud-init/+git/cloud-init/+merge/368507
<Odd_Bloke> rharper: That looks like it would be pretty easy to convert to three separate tests, but I wouldn't do another review round-trip just to address that.
<rharper> Odd_Bloke: thanks, that's what I was thinking
<ozzzo> I'm trying to follow these instructions, to build openstack images with nocloud in VMWare
<ozzzo> https://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html
<ozzzo> genisoimage doesn't exist in RHEL 5.7 so I used mkisofs instead; that seemed to work:
<ozzzo> http://paste.openstack.org/show/774856/
<ozzzo> The next step is to create a qcow from img, but my image is in vmdk format
<ozzzo> can I use the same command with vmdk input?
<ozzzo> qemu-img create -f qcow2 -b disk.vmdk boot-disk.qcow2  ?
<rharper> ozzzo: yeah, https://docs.openstack.org/image-guide/convert-images.html
<ozzzo> ok so I want to use "convert" instead of "create" ?
<rharper> yes, create is empty
<rharper> convert will unpack the source format and write data out in the target format
<rharper> ozzzo: ^
<ozzzo> ok trying that, ty!
<ozzzo> I successfully created boot-disk.qcow2. The next step is a kvm command, but I need to run the modified VM under VMWare. How can I tell VMWare to run it with the extra drive file?
<rharper> not sure about the vmware bits
<ozzzo> OK I'll google around. I think there's a way to import VM files into VMWare
#cloud-init 2019-09-11
<rharper> robjo: https://bugs.launchpad.net/suse/+source/cloud-init/+bug/1843502  ;   curious bug here around azure + suse image and network-config parsing
<ubot5> Launchpad bug 1843502 in cloud-init "Network config is incorrectly parsed when nameservers are specified" [Undecided,Incomplete]
<robjo> rharper: I worked with Moustafa a little bit on this, as in, help to find one place in the code why resolv.conf doesn't have the proper value. Looks like he made much more progress since
<robjo> based on the description this appears to be a generic issue with the "v1" syntax and a typo, "address" vs. "addresses"
<rharper> azure renders v2, so I mentioned the 'nameservers' bits need to be indented underneath the interface name (eth0)
<robjo> Also added a comment
<rharper> robjo: the other confusing  part (to me) was where the nameserver values came from;  it's been a while since I've looked at the contents of azure IMDS
<rharper> robjo: thanks,  I suspect if there's no downstream changes, then it's got to be local image modifications
<rharper> there are some odd debug messages in the log as well,
<rharper> 2019-09-09 16:21:29,416 - handlers.py[DEBUG]: start: azure-ds/parse_network_config:
<rharper> 2019-09-09 16:21:29,416 - DataSourceAzure.py[DEBUG]: Azure: generating network configuration from IMDS
<rharper> 2019-09-09 16:21:29,416 - DataSourceAzure.py[DEBUG]: netconfig &s:
<rharper> in which cloudinit has never emitted the string "netconfig &s"
<rharper> so I'm thinking to close this out as invalid since someone is hacking their own changes into cloud-init
<robjo> I just don't know where the nameserver information is supposed to be coming from
<robjo> if Azure puts the info into the metadata then the data source should pick it up and make it availabe
<rharper> so, Azure, AFAIK, only provides dns info via dhcp lease
<robjo> if that is the case then how could network configuration via cloud-init ever work?
<rharper> so we never explicitly generate network config with 'nameservers' values; azure is a dhcp on eth0 shop, you can dhcp on additional interfaces, and add static ips; but beyond that, I've never seen any dns info in IMDS data
<rharper> we generate dhcp on eth0
<rharper> and the OS's dhcp client does the rest
<robjo> Well something triggers that resolv.conf gets written by cloud-init which prevents netconfig from writing the dns info from the dhcp server
<robjo> There was a bug where an empty resolv.conf was written tha triggered that condition but that has since been fixed
<robjo> OK, so now I am even more confused
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/372622
<smoser> if anyone intereseted
<robjo> I guess I have to fiddle with this on my own and see what happens
<rharper> robjo: do you have a recent boot of an unmodified sles image on azure that we can compare cloud-init.logs ?  I really suspect there's some local modifications due to that strange log output
<robjo> Currently I am not building images with cloud-init, it would be a matter of staring a SLES instance in Azure, then adding cloud init and running cloud-init local to see what happens with network configuration
<rharper> oh interesting, so it's almost certainly a  custom image then
<robjo> Given the state a reboot to get a full log may not be advisable as one may not get back into the system
<robjo> It's certainly customized w.r.t. the initialization code, I am not certain Moustafa made any changes to the cloud-init package delivered in the SUSE repos
<rharper> I'll ask explicitly,  would rpm -verify cloud-init tell us if anything has changed?
<robjo> yes that would be expected
<rharper> https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/372624
<smoser> https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/372624
<smoser> rharper: ^
<rharper> smoser: replied
<smoser> thanks
<smoser> rharper: you can approve https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/372622 if you approve my dropping of the bug url
<smoser> that was your only question) and Odd_Bloke already approved.
<rharper> smoser: sure;
<rharper> smoser: hrm, I was hoping for a new url to a bug indicating that we needed to narrow the definition , ie, the reason for the bug ;  was that unclear ? or do you feel it's unnecessary ?
<rharper> that is, if someone else were to add another look-a-like like zstack and brightbox, the comment could indicate that we want to make sure it doesn't widely match other domain names
<smoser> i think its unnecessary.
<smoser> git log / blame will indicate the bug
<smoser> and (for zstack) there will be a doc/datasources/zstack
<smoser> code does not need references like that.
<smoser> when i went looking, i did a git log, didn't even see the in-file comment.
<rharper> ok
<blackboxsw> rharper: Goneri I've updated my review comments on https://code.launchpad.net/~goneri/cloud-init/+git/cloud-init/+merge/368507
<blackboxsw> one minor behavioral thing I'd like to have fixed. but otherwise looks good to me
<blackboxsw> btw I'm also going to add Goneri to our CI group given the number of branches proposed (it'll automate CI runs on any new branch pushes)
<blackboxsw> welcome to the CI team Goneri :)
<blackboxsw> you'll get CI votes on your branches from here on out
<rharper> blackboxsw: +1
<Goneri> blackboxsw, thanks for the review and the group inclusion :-)
#cloud-init 2019-09-12
<smoser> Odd_Bloke: can you point c-i at https://code.launchpad.net/~ruansx/cloud-init/+git/cloud-init/+merge/372445
<smoser> or paride ^
<paride> smoser, looking
<paride> smoser, PASSED
<bit48> Hi everyone.
<bit48> I have what seems to a cloud-init related problem. I have an ubuntu server image I've created for the raspberry pi. It consists of the base ubuntu-server with a few additional scripts and modifications. Now I'd like to make copies of the original image to put in several raspberry pi devices... the problem is that the other PIs don't bring up their
<bit48> NICs on boot since the mac configured in netplan is the mac of the first raspberry pi I created the image on.
<bit48> where can I find an explanation on how cloud-init creates the netplan yaml?
<bit48> and what to do to make it regenerate it?
<rharper> bit48: hi
<rharper> do you know what datasource you're providing to cloud-init ?  for Pi I suspect it's a no-cloud datasource ?
<bit48> I think it is. There's a file /etc/cloud/cloud.cfg.d/99-fake_cloud.cfg with a single line "datasouce_list: [ NoCloud, None ]"
<bit48> This is my first time dealing with netplan and cloud-init. Last time I did something similar, ubuntu still had /etc/network/interfaces
<rharper> bit48: ok, so I think if you use cloud-init clean --logs on your "master" image before capturing then things will work the way you want
<rharper> this will remove the /var/lib/cloud/instance and other files that indicate cloud-init has booted before;
<rharper> cloud-init will generate a /etc/netplan/50-cloud-init.yaml at boot time
<bit48> That's great.
<bit48> No, wait. That's not great. I took a look at /var/lib/cloud. It looks like it might revert some of the changes I've made to the master image - like the removal of the default "ubuntu" user
<rharper> bit48: clean won't revert any changes
<rharper> it just removes files that cloud-init uses to track what it did
<rharper> in some cases like the ubuntu user, cloud-init checks if it's added a user and won't fail if it's already added
<rharper> you can disable the default user creation if you want
<bit48> Yeah, I'm going to have to remove the creation of the ubuntu user. It's a user I've deleted from the master image. Is the user creation "script" the user-data file at /var/lib/cloud/seed/ ?
<rharper> no, the ubuntu user is created by default, you would update the user-data file in /var/lib/cloud/seed/nocloud/user-data
<bit48> So if I remove that file, then it won't be created, right?
<bit48> (my master image already comes with a different user)
<rharper> bit48: let me look a bit;   I do know you can modify /etc/cloud/cloud.cfg to replace the default user (from ubuntu) to whatever you wanted to setup;  that might be easier than disabling the user-creation;
<rharper> bit48: alternatively,  in you master image, you could modify the netplan config to change from a match by mac address to a match by name;   https://netplan.io/examples#configuring-a-loopback-interface  ;    and it accepts wildcards for glob matching, so you could have match: name: eth*  or match: name: en* etc.
<bit48> thanks
<bit48> Why wouldn't deleting the file or commenting out the user creation work?
<smoser> ~
<rharper> bit48: I'm not sure of the contents of your seed, so I can't be sure; however, cloud-init creates a distro user by default, without any supplied cloud-config/user-data.  So there's nothing to delete/comment out.
<bit48> Yeah, deleting the user-data doesn't help. Changing the ubuntu username to the user I want caused cloudinit to reset my password to "ubuntu" :-\
<bit48> the user-data is trivial passowrd:ubuntu chpasswd: ubuntu ssh_pwauth:True
<bit48> So the way I see it, I have a few options:
<bit48> 1. change the mac manually on every image. 2. change the match clause to something other than the mac and hope it works on all devices. 3. change the cloud init configuration to it'll create the user I want instead of the default one, then run cloud-init clean.
<rharper> bit48: yeah, I think including user-data that creates the user you want instead of the default user is likely best, with a clean;  cloud-init handles other things like ensuring you have unique ssh host keys;
<bit48> I see. Where can I find some documentation about user creation? The best source I found was in the cloud config examples (https://cloudinit.readthedocs.io/en/latest/topics/examples.html)
<powersj> https://cloudinit.readthedocs.io/en/latest/topics/modules.html#users-and-groups
<bit48> Okay, thanks
<bit48> One last question (I hope) - how does cloud-init merge the user-data from the seed directory with the cloud.cfg file?
<rharper> bit48: user-data overrides values found from /etc
<bit48> That's what I figured. Thanks.
<rharper> robjo: I've updated https://bugs.launchpad.net/cloud-init/+bug/1843634
<ubot5> Launchpad bug 1843634 in cloud-init "cloud-init misconfigure the network on SLES" [High,In progress]
<rharper> if you have more details about the wicked service names and what names cloud-init.service should use to wait for DHCP to be completed; that'd be most helpful
<robjo> This doesn't look right, cloud-init-local has "Before=network-pre.target" and the network setup should be complete after cloud-init-local, right?
<rharper> not quite;  cloud-init-local runs before networking; in particular it will bring it an interface to read the IMDS for what the config should be; then write out a config to the system, *before* the system networking service starts
<rharper> so that cloud-init is feeding network config *to* the OS networking service
<rharper> then cloud-init.service wants to run after the OS's networking services have started (and their wait for online service if they have one) so that when cloud-init.service runs, it expects networking is active.
<robjo> hmm, sorry cannot dig in right now, need to finish a presentation I am supposed to give next week and am off tomorrow then traveling. I'll find some time to look at this
<rharper> robjo: sure, thanks
#cloud-init 2019-09-15
<pingiun> I cannot seem to get cloud-init to work on ubuntu 18.04 on ec2. The script is placed in /var/lib/cloud/instances/*/user-data.txt but not executed
<pingiun> I tried with a very basic test from the ubuntu wiki
<pingiun> #!/bin/sh
<pingiun> echo "Hello World.  The time is now $(date -R)!" | tee /root/output.txt
<pingiun> also, are logs from the user data script saved somewhere?
