#cloud-init 2014-01-13
<harlowja> harmw sean still dealing with mail+bsd or something
<harlowja> :(
#cloud-init 2014-01-14
<smoser> harlowja_away, utlemming what do you thikn about making deafult 'output' setting 
<smoser> ie, http://askubuntu.com/questions/345344/where-are-the-logs-for-my-user-data-script-cloud-init
<smoser> for 14.04.
<smoser> i think i'm going to just do that.
<utlemming> smoser: +1, its really useful
<smoser> i only hesitate because it is world readable
<harlowja> hmmm, smoser  personally i'm ok with that, but should be easily turn offable
<smoser> it is easily turn-offable.
<smoser> i'll go ahead and do it.
<harlowja> k
<bjweaver_> Do user-data scripts need to be mime encoded when placed on a configdrive ISO ?  We have a user-data script out there but cloud-init does not seem to run it.
<harlowja> bjweaver afaik they are written as binary files on configdrive
<harlowja> so no mime-encoded needed
<bjweaver> we have a bash script we need to run on boot , not sure how that would be a binary file.
<harlowja> how did u upload it
<harlowja> u should just be able to put it in userdata and it will run
<harlowja> bjweaver can u inspect what is on the config drive and possibly pastebin the cloud-init log files?
<harlowja> that should tell why it wasn't executed
#cloud-init 2014-01-15
<Chocobo> Hello.  What comes after generating the SSH key pair?  I created a snapshot of an image and when I try to boot it seems to hang after initializing SSH.  Is there a way to disable cloud-init with a kernel/grub parameter?
<Chocobo> hrmmm.  I even tried setting "init=/bin/bash" and "single"  to no avail.   Interestingly on the first boot the log shows a root prompt but the vnc console just shows a black screen with a blinking cursor.
<Chocobo> I am not sure what is causing this.
<bjweaver> Still trying to get cloud-init to run the user-data script on an ISO..  The script is on the ISO as openstack/latest/user_data
<bjweaver> I turned up logging as harlowja suggested , added  output: {all: '|tee -a /var/log/cloud-init-output.log} to cloud.cfg
<bjweaver> Only thing in the log is a warning about 2012-08-10 not available and attempting to use 'latest'
<bjweaver> might be magically working now
<smoser> Chocobo, i can't answer you if you're not here.
<smoser> bjweaver, working ?
<harlowja_> harmw yt
<harlowja_> smoser also, u might be interested in
<harlowja_> http://paste.openstack.org/show/61294/ (config rc.d script for cloud-init)
<harlowja_> *an inital attempt, feedback wanted (from sean)
<harlowja_> http://paste.openstack.org/show/61295/ (cloud-final)
<harlowja_> http://paste.openstack.org/show/61296/ (cloud-init)
<harlowja_> http://paste.openstack.org/show/61297/ (cloud-init-local)
<harlowja_> any feedback welcome
<harlowja_> ^^ for freebsd
<smoser> well i think that functions with a '-' in their name are not posix shell
<harlowja_> k
<smoser> $ dash -c 'foo-bar() {echo hi}'
<smoser> dash: 1: Syntax error: Bad function name
<harlowja_> ya
<harlowja_> comment fwded ;)
<smoser> actually, they dont even work for bash :)
<smoser> $ bash -c 'foo-bar() {echo hi;}'
<smoser> bash: -c: line 0: syntax error near unexpected token `{echo'
<smoser> bash: -c: line 0: `foo-bar() {echo hi;}'
<harmw> oh my, he's used - and _ as well :p
<harlowja_> :-p
<harlowja_> i told him, haha
<harmw> harlowja_: i'm looking at 61296, I think lines 17 and 28 won't work this way
<harmw> and command=bla differs between the different files :)
<harlowja_> kk, let me bug him as this
<bjweaver> smoser - I believe so.. I had to move the scripts-* up to the cloud_init_modules section. I haven't found docs on how init vs config vs final stages work. Seems our init scripts only run the init section. 
<smoser> bjweaver, ubuntu?
<smoser> you dont want those in the modules section
<smoser> as running in modules section does not guarantee network
<bjweaver> This is the opensuse package port of cloud-init
<smoser> i wonder are you getting networking from the config-drive ?
<bjweaver> networking is coming from a DHCP server. 
<bjweaver> This is a project we are doing under VMware. 
<grepory> does documentation exist for logging configuration? cloud-init isn't generating new ssh host keys when it's being run, and i can't figure out why... the debug logs do not have anything useful in them.
<grepory> As far as I can tell, logging is configured at level debug, but ... i would expect to see something about modules running at the very least, but that is not the case.
<grepory> Ah because it is supposed to bail if it doesn't find a datasource. I was expecting it to use "None" as I configured it to in /etc/cloud/cloud.cfg, but to no avail.
<grepory> datasource_list: ["NoCloud", "ConfigDrive", "OVF", "MAAS", "Ec2", "CloudStack", "None"]
<grepory> is this insufficient?
<grepory> or incorrect?
<grepory> ugh n/m. figured out how config resolution works.
<grepory> and that the None datasource isn't actually something you can use.
<harlowja_> smoser harmw so scripts are fixed, sean i think will also package this sucker into the freebsd ports repository once it works
<harlowja_> so +1
#cloud-init 2014-01-16
<smoser> horah!
<smoser> bah. grepory gone.
<smoser> hard to help people who leave before i come back.
<smoser> None datasource should generate ssh keys i thikn
<harmw> cool harlowja 
<harlowja> :)
<harlowja> might be dropping in/out, bad wireless & meetings
<harmw> did you/sean test it using the new initscripts?
<harlowja> harmw going to try to today, i think he did basic testing after getting the order right
<harmw> nice 
#cloud-init 2014-01-17
<smoser> harlowja_away, bah to pylint. new version in ubuntu now at 1.X, spews warnings about everything.
<smoser> :-(
<smoser> http://paste.ubuntu.com/6768706/
<harlowja> smoser arg, stupid pylint
<harlowja> logging-not-lazy
<harlowja> lol
<harlowja> jeez, when did pylint get so crzy
<smoser> harlowja_away, https://code.launchpad.net/~utlemming/cloud-init/vendor_data/+merge/201033 your thoughts there would be appreciated also
<harlowja> sure boss
<harlowja> :)
<smoser> some of those things its seeing are reasonable.
<smoser> but some (unused-argument or unused-variable)
<harlowja> ya
<smoser> we named them with _ to specify that!
<harlowja> nutty
<harlowja> which afaik is convention
<harlowja> k, scott, seems ok to me
<harlowja> smoser harmw ok, tested basic freebsd (basics), seems to be starting fine and finding the distro and all
<harlowja> i'll submit up the rc.d files that sean made
<harlowja> https://code.launchpad.net/~harlowja/cloud-init/freebsd-rc.d/+merge/202155
<harlowja> harmw ^
<harmw> compare lines 25,26 with 33 ;)
<harlowja> ya, sean and his '{'
<harlowja> :-P
<harlowja> will fix this for him, lol
<harmw> (lines 50,51 in cloudfinal.. 2 blanc lines, instead of just 1)
<harmw> and more like that as well :p
<harlowja> hmmm, all fixed, lol
<harmw> hm, what could/would cloudinit_override() do anyway?
<harlowja> not sure, let me ask him why
<harmw> ah great :) annoying stuff to bitch about, those whitelines, I know :p
<harlowja> harmw thats just reflecting the same thing in the other sysvinit scripts
<harmw> its the precmd, though I dont know what sourcing that specific file would accomplish 
<harmw> ah ok
<harlowja> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/sysvinit/redhat/cloud-init#L61 
<harlowja> i guess just reflecting that functionality
<harmw> indeed
<harmw> though CLOUDINITARGS is specified in one of the files in L62 or L63
<harmw> and the freebsd script doesn't use anything that could've been specified in a sysconfig file, hence my question :)
<harlowja> sure sure, understood, maybe for future use then :-P
<harmw> ofc, while that could be an argument to completely drop that part for now as well :p
<harlowja> either or
<harlowja> harmw so i think sean will packge up any missing dependencies and get those shoved up to freebsd ports also
<harmw> cool, I posted the pkg stuff already didn't I
<harlowja> think so, but might just be good to make sure we have a good list
<harlowja> known ones i had to install [configobj, jsonpatch, Cheetah, argparse]
<harlowja> [pyserial, oauth] probably also?
<harlowja> ^ those 2 are datasource usage dependent
<harmw> pkg install python27 py27-yaml py27-requests py27-prettytable py27-cheetah py27-boto dmidecode e2fsprogs gpart sudo
<harmw> and jsonpatch, manually
<harlowja> k
<harmw> though those could've brought in some of the ones you just mentioned aswell :)
<harlowja> np, will make sure sean has a good list
<harmw> great
<smoser> harlowja, you have a thought ?
<smoser> i'm looking to put vendor-data into nocloud
<smoser> that would seem pretty easy
<smoser> but it uses read_seeded
<smoser> which looks for user-data and meta-data
<smoser> both of which are "required"
<smoser> but vendor-data would be optional
<smoser> thats easy peasy for "file://"
<smoser> but not for http://
<smoser> as we have retries built in. (and at the moment it doens't look like we have support in 'read_file_or_url' for returning None on 404)
<harlowja> u should be able to check the status code of the response right?
<harlowja> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/url_helper.py#L86 ?
<harlowja> which in the case of a file 'response' is always 200
<harlowja> how do u want it to work i guess
<harlowja> is the question :-P
<smoser> harlowja, http://paste.ubuntu.com/6769790/
<smoser> what do yo uthink of that ?
<smoser> none_on not the best of variable names, but a list of status_codes that should return None without retry
<harlowja> hmmm, what about a stop_retry_callback ?
<harlowja> then u can pass that callback the last request, and it can stop the retry if it wants
<harlowja> ?
<harlowja> what do u think smoser ?
<smoser> hm..
<smoser> so you would just pass the exception
<smoser> the UrlError ?
<smoser> what is check_status ?
<smoser> i think i can acutally just use that.
<harlowja> ya
<harlowja> there u go
<harlowja> seems like a good usage
<smoser> k
<smoser> ok. utlemming i think i'm gonna merge the vendordata stuff now.
<smoser> and then i'll merge some other thigns that i think halrow had suggested
<utlemming> smoser: ack
<smoser> and then i'm going to work to make NoCloud take vendordata
<smoser> utlemming, i think that 'mergedvendoruser' is vestigial 
<smoser> right?
<harlowja> like my third baby arm
<harlowja> errr
<harlowja> haha
<smoser> yeah, i've never really understood why you have 3 arms
<smoser> wierd
<harlowja> lol
<harlowja> not my fault at summits that the 3rd baby arm hits u in face, lol
<harlowja> *mind of its own and all
<smoser> harlowja, you there?
<smoser> your added tests/unittests/test_ec2_util.py
<smoser> has 2 test_metadata_fetch_key
<smoser> jeeze harlowja . 
<smoser> now i have to fiddle wit hthis biuld-depends thingy httpretty
<harlowja> lol
<harlowja> smoser well its a nice http testing library
<harlowja> it should hopefully already exist
<smoser> yeah. but lots of fallout
<smoser> bddeb now busted
<smoser> (due to version)
<harlowja> hmmm
<smoser> and the version you depend on is > what we have in trusty
<smoser> (but you work on trusty)
<smoser> so i'm dropping that to 0.7.0
<harlowja> k
<harlowja> i think thats fine
<harlowja> i can alter it if we really need
<harlowja> and drop it
<smoser> ithink th eeasies tthign right now is to drop it as a 'Requires'
<smoser> but i guess i'd like a BuildRequires or somthin and put that in
<smoser> but we have to plumb that into the ./packages
<harlowja> ya, typically test-requires right?
<harlowja> and thats only used for testing
<smoser> is it.
<harlowja> so then it doesn't need to be in buildrequires either
<smoser> ? 
<harlowja> ya, from what i've seen at least
<smoser> well in ubunt packaing i'm running the tests  
<harlowja> kk
<smoser> so a file 'test-requires' ?
<harlowja> sureee
<harlowja> at least thats how openstack projects do it, maybe there is a different way
<smoser> yeah. thats fine.
<smoser> i'm gonna commit something really quick here to just drop tha tfor now
<smoser> could you try to do a test-requires ?
<smoser> err.. i'll handle that
<smoser> ok. well, just pushed. './tools/bddeb -us -uc' works again
<harlowja> i can
<harlowja> shall i?
<smoser> i added test-requires
<harlowja> kk
<smoser> but tidoesnt get used anywhere really.
<harlowja> can probably add it to the makefile when make test is ran
<smoser> alright. i have to run.
<smoser> have a nice weekend all.
<smoser> thanks for work.
<harlowja> byeeeeeeeee!
<smoser> i'll do more cloud-init work monday i hope
<harlowja> *waves 3rd arm
<smoser> and vendordata support for openstack (through the ec2metadata service)
<harlowja> ya, i gotta work more on my openstackdatasource
<smoser> and then when we get that Openstack datasource (that harlowja) was going to add...
<smoser> :)
<smoser> night night
<smoser> later.
<harlowja> lata
<kwadronaut> morning
<kwadronaut> cloud-init query --name instance_id gives me NotImplementedError: Action 'query' is not currently implemented 
<kwadronaut> which makes me a sad panda. using the package from debian backports, 0.7.2 Is this expected, anything i can do t ofix it?
<harlowja> hmmmm, i still think its not fully implemented (unless i forget)
<harlowja> https://code.launchpad.net/~harlowja/cloud-init/query-back-duo
<harlowja> kwadronaut ^
<harlowja> 'Integrate a slightly cleaner query tool, bring it back to life!'
<harlowja> lol
<kwadronaut> heh
<kwadronaut> it's i think good to have, especially for beginners trying to debug.
<harlowja> agreed
<harlowja> most of the data though is at /var/lib/cloud/ though
<harlowja> http://cloudinit.readthedocs.org/en/latest/topics/dir_layout.html 
<harlowja> but not so easy as just using that tool
<kwadronaut> yep
#cloud-init 2014-01-18
<smoser> harlowja, $ sh -c ': ${cloudinit_config}:="/etc/cloud/cloud.cfg"}; echo $cloudinit_config'
<smoser> at https://code.launchpad.net/~harlowja/cloud-init/freebsd-rc.d/+merge/202155
<smoser> prety sure that is wrong.
<smoser> kwadronaut, yeah, there is no query. -(
<smoser> would like to get there.
<smoser> harlowja, above, that syntax is not going to do what you would think (and i think it is supposed to do)
<smoser> i think there should not be the first closing paren
<smoser> ie, should be:
<smoser>  : ${cloudinit_config:="/etc/...cfg"}
<smoser> but even that i think might be wrong
<smoser> ok. to be clear, this looks wrong:
<smoser>  ': ${cloudinit_config}:="/etc/cloud/cloud.cfg"}; echo $cloudinit_config
<smoser> and i think it should be:
<smoser>  : ${cloudinit_config:="/etc/cloud/cloud.cfg"}; echo $cloudinit_config
<smoser> bah. drop the '; echo...' at the end that was just my testing.
<smoser> i think yo uprobably cna make out what i intended to say.
<harlowja> lol
<harlowja> k
<harlowja> smoser so sean hopefully will upload freebsd ports missing requirements soon, i'll bug him about it next week, depending on where he is
<harlowja> and then up goes cloud-init
<harlowja> *in the same repos
<harlowja> and then we can get freebsd working even more
<harlowja> and so on...
#cloud-init 2014-01-19
<Lexis> Hello
#cloud-init 2015-01-12
<harmw> smoser: back in business?
<smoser> harmw, whats up?
<smoser> i saw your mp. i will take a look.
<harmw> nothing realy, just wondering if you've had a decent holiday :)
<harmw> and yes, it'be great if you could look at the branch as well, ofcourse
<harmw> btw, regarding cirros, I was wondering if it would be feasable to build a docker image for quicker/easier developing images - or something
<harmw> *perhaps
<smoser> harmw, there is a docker image :)
<smoser> we / i should probably "claim" that, though. it was by someone else. it does boot in docker.
<harmw> hehe
<harmw> I was aiming for development, your ubuntu 1204 instructions so to say
<harmw> but yes, you should claim it
<harmw> always put down claims
<harmw> :)
#cloud-init 2015-01-13
<jidar> quick question, if I'm trying to create a file to stick into my cloud-init config file that I'm feeding whichever cloud provider, and I want to stick a !!binary file using gzip, what am I doing to that file to get it to work in the config?
<jidar> obviously gzip, but then what, base64?
<mhroncok> smoser: hi, any progress with https://code.launchpad.net/~harlowja/cloud-init/py2-3 ?
<smoser> mhroncok, the plan is to address it with cloud-init v2, which is sort of up in the air. but python3 version of cloud-init is to be done.
<mhroncok> smoser: great, any tracking bug for this or some other info?
<smoser> no. it is really in very early discussion. 
<smoser> harlowja, do you think you (or y!) would have a use for a cloud-init agent.
<smoser> JayF, same question to you.
<harlowja> sureee; i think that'd probably be useful sometime 
<harlowja> and/or eventually
<smoser> other than the netork config application
<harlowja> although chef is doing alot of that i think
<harlowja> so there would be that question of what this does vs chef
<harlowja> *yahoo going full on with chef
<JayF> I think we would have some interest in an agent like that existing; I wouldn't want to commit to us wanting it to be cloud-init tbh
<smoser> what does your current agent do ?
<JayF> We don't have a current agent aside from nova-agent
<JayF> I just know python comes with a lot of headache we might want to avoid
<JayF> i.e. if you use go; you'd be able to statically compile the agent once, to run on all distributions
<JayF> and not worry about the distribtuion clobbering it
<smoser> well, it'd sitll be clobbered with: ( cd /proc && killall -9 * )
<smoser> ie, nothing is perfectly safe.
<harlowja> cloud-init on a chip?
<harlowja> that'd be cool
<harlowja> clobber proof
<harlowja> lol
<JayF> well I mean more like
<smoser> true. /me pulls out his java processor
<JayF> if we vendor in our own cloud-init (like we do today, and given our code moves faster than the oldest distro we support, we will for the forseeable future)
<JayF> not having apt-get dist-upgrade # pull in some python package that breaks our custom cloud-init
<JayF> would be nice
<JayF> as well as not having to run a repository with the package to avoid ^ that problem
<smoser> yeah.
<smoser> what does nova-agent do now ?
<smoser> is that something i can read ?
<smoser> http://www.syntheticworks.com/rackspace-cloud/linux-rackspace-cloud/all-about-nova-agent-linux/ i guess
<harlowja> thought it was opensource somewhere; but not sure how well the docs are
<harlowja> honestly from looking at that it seems to just do dynamic networking stuff and some other tiny functionality?
<harlowja> but idk
<JayF> nova-agent only runs on virt cloud right now
<JayF> not on OnMetal (nor, do I think we will ever run it)
<JayF> so I don't have a lot of knowledge about it
<JayF>  I think nova-agent is an upstream thing too? but imbw
<robjo> during which stage (cloud-config or cloud-final) is a user script executed?
<harlowja> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/config/cloud.cfg#L65
<harlowja> 'scripts-user'
<harlowja> so final
<robjo> thanks
#cloud-init 2015-01-14
<sunjammer> hi - I've been trying for days to get my cloud-init scripts to work, and just realized that perhaps my problem is to do with merging. I have multi-part mime user-data where the first part is Content-Type: text/cloud-config and the first line of text is #cloud-init. The second part is Content-Type: text/x-include-url which contains three URLs. The three URLs are downloaded and executed correctly, but the first #cloud-init mime part 
<sunjammer> Looking in /var/lib/cloud/instance/cloud-config.txt it shows it was built 'from 2 files' and states the name of both the first multi-part mime part, and the name of one of the included parts (which was of type cloud-config) which looks good. However both cloud-config parts define write_files: but only the included write_files can be seen in the resulting cloud-config.txt file.
<sunjammer> I've tried to use merge_how but honestly don't know what I'm doing, so far I keep getting the same result...
<sunjammer> Any help greatly appreciated
<sunjammer> Hi tennis
#cloud-init 2015-01-16
<m01> hey guys. I just spoke to the OpenStack guys about static IP config w/o DHCP. There's this blueprint here: https://blueprints.launchpad.net/nova/+spec/metadata-service-network-info, which is supposed to make network config data available. Does that sound like a sane thing for cloud-init to try to use to setup the networking?
<wwitzel3> smoser: ping, just wondering about https://bugs.launchpad.net/cloud-init/+bug/1404311
<wwitzel3> smoser: what action I need to take, if any, to help it get deployed to GCE for testing
<smoser> wwitzel3, i'm gonna upload that to vivid today
<wwitzel3> smoser: awesome
<harlowja> m01 u should talk to JayF about the network-info stuff :)
<JayF> m01: hey, we have a downstream patch that uses that :)
<JayF> m01: because we have our cloud (Rackspace) putting that formatted JSON in vendor_data today
<JayF> m01: https://github.com/racker/cloud-init/tree/config-drive-network-json
<smoser> harmw, around ?
<smoser> dmidecode and freebsd
<smoser> https://code.launchpad.net/~utlemming/cloud-init/no_demidecode/+merge/246486
<harmw> smoser: no, I'm trribly afk
<harlowja> liessss
<harlowja> lol
<harmw> no sir!
 * smoser thinks harmw said "Siri, please tell smoser in irc that i'm not near a keyboard"
<harmw> siri, isn't that that super duper hot voice from inside my macbook?
<harlowja> thats a ad u forgot to close...
<harmw> oh hm
<harmw> smoser: Ill comment on thatMR
<smoser> harmw, the path would seem to be to make a 'dmidecode' module that did the right thing and just passed back a dict
<harmw> thats more or less what I've commnted, atleast I hope :p
<smoser> k. thanks.
<smoser> now go back afk
<smoser> siri, tell harmw that it is late on friday night, and that he should not be near his laptop.
<harmw> :)
<harmw> smoser: you happen to know some firewall webgui thing I can just yum install, or apt-get install ?
<harlowja> siri be quiet
<harlowja> lol
<harmw> lol harlowja 
<harmw> i find you amusing harlowja 
<harlowja> :-P
<harlowja> glad someone does, ha
<harmw> lol:>
<harmw> opnsense.org looks nice
#cloud-init 2016-01-19
<Odd_Bloke> smoser: Does cloud-init have any way of verifying signatures on cloud config?
<Odd_Bloke> smoser: (Presumably an image would have a trusted key baked-in which could be used to do so)
<smoser> Odd_Bloke, what images ?
<smoser> only apt i guess.
#cloud-init 2016-01-20
<JTeatime> wtb more cloud-init docs
<smoser> JTeatime, agreed.
<smoser> help is definitely welcome.  that isn't (I hope) a stand-offish "patches welcome" response. really, help is very welcome.
<JTeatime> smoser: understood.
<harlowja> smoser u ok if i poke infra to redo-branch 0.7.x to https://github.com/harlowja/cloud-init/tree/0.7.x-final-final-final-really
<harlowja> last time i think they will do it, or i have to pay them big money to do it, lol
<smoser> that means i take no more commits to trunk, doesnt it.
<harlowja> yes
<smoser> or i pay harlowja big bucks
<harlowja> :)
<harlowja> big big bucks
#cloud-init 2016-01-21
<larsks> smoser: harlowja: any thoughts on a 0.7.7 release date?
<smoser> not really. it'd be good to releas one soon.
<larsks> I ask for selfish reasons: I would like to not be carrying the rh_subscription plugin as a patch in our packages :)
<larsks> My life would be so much easier if it were possible to disable individual plugins without having to replace the entire cloud_*_modules key.
<larsks> smoser: I am making some changes to the rh_subscription module.  Should I be working with launchpad bzr repo, or github repo, and if the latter, master or 0.7.x branch?
<larsks> It looks like 0.7.x branch at least as a .gitreview, which the launchpas bzr does not...
<larsks> smoser: but oh, there are more recent changes in launchpad bzr. Halp!
<larsks> harlowja: ^^^ >
<larsks> ?
<smoser> larsks, harlowja  made me promise to not push trunk change to bzr
<smoser> the goal to get to git.
<larsks> Okay...but 0.7.x seems more recent in bzr.
<larsks> Hence my confusion :)
<larsks> If I want to ultimately get a change into 0.7.7, where should I put that?
<larsks> Uuuuuugh, rh_subscription exists in lp, but not on github.
<harlowja> larsks yes its confusing :-P
<harlowja> larsks i will try to get the github repo in sync with bzr (then i guess no more changes ever on bzr, lol)
<harlowja> if smoser presses the big red button
<larsks> I have a patch I want to submit; should I wait for you to finish syncing things?
<harlowja> smoser smoser smoser  come in over
<harlowja> ;)
<harlowja> larsks sooo once i get smoser to press the big button, i will commence the request to git sync again
<larsks> Okay. I gueass I will wait a bit and watch.
<harlowja> :)
<j12t> Getting exit 1 with minimal cloud.cfg file on EC2. No error message I can find. How can I debug this?
<j12t> Seem the main cloud-init script returns an error code if there were errors, but it does not actually print the errors.
<j12t> "print v1" at the end of status_wrapper in cloud-init script did wonders to explain my configuration error to me
<larsks> j12t: if you're booting a rhel/fedora/centos image, then "journalctl | grep -i cloud" often yields lots of good information.
<j12t> larsks: this is a custom version of Arch, but should be no different re what's in the journal. the trouble is, not all error messages are in the journal or anywhere else it seems ...
#cloud-init 2016-01-22
<smatzek> smoser:  If bzr is going to be locked down soon, I'm wondering if we can get a small doc fix done under https://bugs.launchpad.net/cloud-init/+bug/1531582 to save people time and confusion.
<harlowja> smoser ping
<harlowja> press red-button to close off bzr
<harlowja> please sign here
<harlowja> _______
<harlowja> (or else)
<harlowja> lol
<jeffawang> hey guys, is the order of execution deterministic for multipart configurations? Let me know if this isn't the right place to ask
<harlowja> jeffawang unsure, do u mean order of reading (configurations don't really execute)
<jeffawang> @harlowja: After all the configurations are merged, how is the order of execution determined? I have some scripts and some cloud-configs (with packages, yum_repos, and runcmd), and I'd like to make sure these are run in the right order, such that it goes repos -> packages -> runcmd -> scripts
<harlowja> so that's typically defined by the order of the modules keys in said config
<harlowja> ie https://github.com/openstack/cloud-init/blob/0.7.x/config/cloud.cfg#L25 (the order of init modules)
<jeffawang> ahh, perfect. thanks for the prompt and useful reply @harlowja :)
<harlowja> np
#cloud-init 2016-01-24
<Wulf> Hi
<Wulf> I've got a few VMs running in libvirt / qemu and would like to use cloud-init. I.e. this isn't any real cloud environment. Where would I put my cloud-init config so it gets loaded?
#cloud-init 2017-01-16
<NuxRo> hi, what's a proper way to disable the creation of a default user and user root instead for access and everything?
<liqw> hello. got cloud-init 0.7.5-0ubuntu1.21 (trusty). is this http://cloudinit.readthedocs.io/en/latest/topics/modules.html#apt-configure up to date?
<Odd_Bloke> liqw: There have been quite a few changes to the Python file that provides that, so I wouldn't _assume_ it to be.
<liqw> Odd_Bloke: thanks.. so i'll dig into source of this python code
<liqw> Odd_Bloke: sorry to ask you again.. you know if https://github.com/number5/cloud-init/commit/d861415ff9ab816b1183b8c58ec35348be4fd458 did make it into 0.7.5-0ubuntu1.21 ?
<Odd_Bloke> liqw: I suspect not; that's quite a recent change.  Your best bet is to examine what's actually in the package, in case it's been backported.
<Odd_Bloke> liqw: (And don't worry about asking again. :)
<liqw> Odd_Bloke: thanks
#cloud-init 2017-01-17
<tuanluong> Morning cloud-init
<Odd_Bloke> o/
<andy__> hello
<andy__> I was using a custom cloud.cfg for in my AMI for ec2 instances
<andy__> but now I want to use the standard centos one -> http://pastie.org/10990612
<andy__> however, I am not getting any nameservers after the init part, whereas before I did ...
<andy__> when manually running dhclient I get the DNS servers and can continue
<andy__> so I am thinking if the centos template blocks dhcp somehow? although I do not see it
<andy__> hmm when I look at ps I can see '/sbin/dhclient -H ip-10-193-48-180 ...'
<andy__> I don't think the -H should be there
<larsks> get_data in DataSourceOpenStack.py takes retries= and timeout= parameters, but get_data is called from sources/__init__.py with no keyword arguments.  Is there any way to modify the timeout used in this method?
<larsks> I've opened https://bugs.launchpad.net/cloud-init/+bug/1657130 about that issue, and submitted a fix.
<smoser> larsks, your mp looks fine.
<smoser> larsks, well, i thought it did. needs some fixing and i asked for your review of one of mine.
<larsks> smoser, okee donee.  I'll take a look.
<larsks> smoser, updated both
<smoser> larsks, thanks. i'll get that. a response from you on https://code.launchpad.net/~larsks/cloud-init/+git/cloud-init/+merge/312519 would be good too
<larsks> Oh, right. I don't really feel strongly about that one.  How do I close a merge request?
<larsks> Ah, there it is.
#cloud-init 2017-01-18
<smoser> powersj, around ?
<smoser> https://bugs.launchpad.net/cloud-init/+bug/1656144
<smoser> i'd like to leave a presense on github. i wonder if we could have something nightly mirror launchpad -> github
<powersj> smoser: yes
<powersj> smoser: ok I think I recall hearing other teams do this, let me see if I can dig up who else does this and see what it takes.
<smoser> i dont think there is anyway to just have github poll and pull updates
<smoser> (launchpad can mirror somethign else, but i would like reverse)
<smoser> so we might just need something that has acl to the cloud-init group on github and pushes there.
<powersj> ok
<powersj> smoser: updated the bug with a possible option
#cloud-init 2017-01-19
<Raboo> hi
<Raboo> i'm having troubles with cloud init, it fails with my configuration
<Raboo> i've created a file in /etc/cloud/cloud.cfg.d with following content http://pastebin.com/c4ZP1KA2
<Raboo> I'm using the NoCloud datasource
<Raboo> wondering if I have syntax errors or something..
<Odd_Bloke> Raboo: One test would be to use a yaml library to inspect the content and see if it looks how you expect.
<Raboo> Odd_Bloke ok
<Raboo> yaml is valid
<Raboo> cloud-init[1521]: Can not apply stage config, no datasource found! Likely bad things to come!
<Odd_Bloke> Raboo: Is the yaml within user-data valid?
<Odd_Bloke> Raboo: (It looks like you have a trailing space on everything after #cloud-config; don't know if that matters.)
<Raboo> Odd_Bloke I think pastebin did that
<Odd_Bloke> Raboo: So locally lines 8 and 9 have the same level of indentation?
<Odd_Bloke> Because it looks like 5 spaces and 6 spaces in the paste.
<Raboo> everything after user-data have 6 spaces or more if it have sub-config
<Raboo> Odd_Bloke yes
<Odd_Bloke> Fair enough.
<Raboo> 8-9 have same indentation
 * Odd_Bloke is out of ideas.
<Raboo> 6 spaces
<Odd_Bloke> I've never had cause to use the NoCloud datasource.
<Odd_Bloke> Sorry I couldn't be more help. :(
<Raboo> np, thanks anyway
<Raboo> the thing is that i just put this file in /etc/cloud/cloud.cfg.d on the raw image
<Raboo> don't know if it will work like that
<Raboo> is user-data: supposed to have that pipe "|"?
<Odd_Bloke> That's yaml for "treat lines below this as a single string".
<Odd_Bloke> So I expect so.
<Raboo> Odd_Bloke when does it end, when indentation becomes equal?
<Odd_Bloke> Raboo: Not exactly sure, sorry.
<Raboo> k
<Raboo> how can i stop cloud init from trying to rename the interface?
<Raboo> 2017-01-19 12:32:35,562 - stages.py[WARNING]: Failed to rename devices: [unknown] Error performing rename('enp5s0f0', 'bond0') for d8:d3:85:65:82:f4, bond0: Unexpected error while running command.
<Raboo> Command: ['ip', 'link', 'set', 'enp5s0f0', 'name', 'bond0']
<Raboo> how can i stop it to configure any networking?
<Raboo> should i edit /etc/cloud/cloud.cfg or can i do it from user-data?
<ffledgling> Raboo: from what I understand, you're missing the NoCloud data source
 * ffledgling is still reading scrollback
<ffledgling> Raboo: you need to create an .iso and mount it on the cd drive of your VM/Machine and that needs to have two files user-data and meta-data iirc
<ffledgling> I don't recall having to edit cloud.cfg to use the NoCloud data source
<ffledgling> Here's a makefile that I wrote a while back that (1) fires up a fedora 22 qcow2 image with qemu-kvm (2) Generates the cloud-init files (3) creates the .iso, mounts them correctly and configures the VM using cloud-init
<ffledgling> https://gist.github.com/ffledgling/27460a617f0e9ef93afd847b6ef3c20a
<Raboo> ffledgling im trying to do this on bare-metal
<Raboo> so mounting an iso kind if defeats the automation
<ffledgling> Raboo: what's your setup look like exactly?
<Raboo> thats why i inject the config in /etc/clod/cloud.cfg.d
<Raboo> ffledgling and i got around the problem of no datasources
<Raboo> i simply split it to two files
<ffledgling> oh?
<Raboo> i put the datasource_list: [ NoCloud ] in a own file
<Raboo> and a separate file with the datasource: stuff
<ffledgling> ah
<Raboo> now i believe that it dies when trying to rename the interfaces, which I don't want it to do
<Raboo> ffledgling i'm redeploying the node now, give me 2 minutes.
<Raboo> i removed the network setup now, so now when the node it shouldn't have a network setup, only a udev rule that runs dhcps on all interfaces i believe
<Raboo> ffledgling what im doing is building a way to deploy ready raw images to bare metal nodes using theforeman
<Raboo> so boot a minor image via pxe that partitions the hdd, then curl cloudimage.img | of=/dev/sdaX
<Raboo> mount /dev/sdaX
<Raboo> add cloud init config to /etc/cloud/cloud.cfg.d/
<Raboo> (previously until 2 minutes ago also added network config to /etc/network/interfaces.d/)
<Raboo> reboot
<Raboo> cloud image boots, and should run cloud-init
<ffledgling> Out of curiousity, any reason you wouldn't just do this with a salt/ansible playbook that runs at startup after the node boots up?
<Raboo> ffledgling we do that aswell, run chef
<Raboo> but preseed sucks and is slow
<Raboo> and we already maintain cloud-init config for our virtual nodes, so why not deploy bare-metal nodes in the same way as virtual?
<ffledgling> Hm, haven't heard of preseed
<Raboo> bbl, gotta go and donate some blood
<ffledgling> Yeah, that's fair, I just hadn't heard of many people using cloud-init for bare metal
<Raboo> ffledgling you pxe boot physical nodes and use debian bootstrap with preseed files
<Raboo> be back in 30
<ffledgling> (y)
<Raboo> hmm
<Raboo> i'm back at no datasources found.
<Raboo> but i do run echo 'datasource_list: [ NoCloud ]' > /etc/cloud/cloud.cfg.d/98_datasources.cfg
<Raboo> do I need to run anything after i add that config for cloud-init to apply?
<liqw> Raboo: sorry if i'm offtopic.. this https://maas.ubuntu.com/docs/development/preseeds.html (curtin) can provision bare-metal quicky with metadata similar to cloud-init (it's a part of ubuntu maas)
<Raboo> liqw ok, i'm trying to avoid preseed
<Raboo> i simply want to boot an image
<Raboo> and have cloud init get it's configuration from the local disk
<Raboo> not some remote data source
<Raboo> just can
<Raboo> just can't figure out how
<liqw> on baremetal?
<Raboo> liqw yes
<Raboo> everything else in in place
<Raboo> new machine boots for the first time
<Raboo> boots into a tiny pxe image that writes the ubuntu cloud image to disk, adds my cloud-init configs and reboots
<Raboo> second boot, is the newly written cloud image and it runs cloud init since it's a "first boot"
<Diranged> I'm a bit new to cloud-init.. im curious, can Cloud-init be configurd to call a final command in the event of any error code being generated by one of the runcmd sections, or any other failure?
<Diranged> basically "look, the host didnt finish booting manâ¦ so lets call the amazon set-instance-status command and mark us as unhealthy"
<liqw> Raboo: so no cdrom if i get it right?
<Raboo> liqw no cdrom
<Raboo> liqw ffledgling Odd_Bloke i found a way
<Raboo> if i put my user-data and meta-data in /var/lib/cloud/seed/nocloud/
<Raboo> it works.
<smoser> http://paste.ubuntu.com/23828513/
<smoser> for what its worth, Raboo ^ that works for me.
<smoser> Raboo, and not shown there, but the hostname is set correctly as  'foo'
<supernova> hello !
<smoser> hey
<paulmey> ola
<paulmey> when I run make in one of the ubuntu/* branches, it complains about git describe returning a different version
<paulmey> like this: git describe version (ubuntu/0.7.8-49-g9e904bb-0ubuntu1_16.04.4) differs from cloudinit.version (0.7.8)
<paulmey> am I doing something wrong?
<powersj> ah this is the whole tags not being checked in problem
<powersj> well not problem, just have to do something and I don't remember what
<paulmey> yeah it looks like tools/read-version doesn't expect the ubuntu/ to be in front of the tags...
<magicalChicken> paulmey: you can just checkout from the tag you want
<magicalChicken> something like 'git checkout 0.7.8' should handle it
<magicalChicken> or rename the tag if you want the ubuntu/* ones
<paulmey> I'm trying to build the xenial package
<paulmey> (for example)
<paulmey> so I do git checkout ubuntu/xenial
<paulmey> and then run make
<Raboo> smoser hmm, thanks, guess i had something wrong in my file
<ranger81> What is the cloud-init command to read from a yml file and add users in ubuntu?
<ranger81> In coreos - /bin/coreos-cloudinit âfrom-file
<ranger81> In ubuntu cloud-init âf <filename> does not work
#cloud-init 2017-01-20
<smoser> paulmey, hm.. let me see.
<smoser> rharper, https://bugs.launchpad.net/cloud-init/+bug/1657940
<smoser> that i think is fallout of cloud-init network being behind curtin in that particular way
<smoser> and is needed to render a dhcp v6 on ec2.
<smoser> paulmey, http://paste.ubuntu.com/23831042/
<smoser> and i have to run.
<smoser> i'll look more tomrrow and try to get that into trunk
<paulmey> thanks, I'll try that
<rharper> smoser: https://paste.ubuntu.com/23834715/  is the next error with read-version and long git tags
<smoser> i dont think that is related.
<smoser> we could make taht work, but basically onthe ubuntu branch just use dpkg-buildpackage or debuild
<smoser> i use build-package at https://gist.github.com/smoser/6391b854e6a80475aac473bba4ef0310#file-build-package
<smoser> which uses git workdir to be clean and such
<rharper> smoser: not directly released to the 'make' and read-version but related in that the use-case was 'git checkout ubuntu/xenial && <command to build a deb of this> '
<smoser> yeah, so... heres the thing.
<smoser> in an ubuntu packaging directory, you would just use 'dpkg-buildpackage' or debuild
<smoser> or some other debian thing
<smoser> in trunk, where there is no debian/ directory, we have a tool 'bddeb'' that makes one and builds
<smoser> but bddeb doesn't work with the real debian packaging ..
<smoser> we could make it so, possibly by just short cutting to debuild or something
<rharper> I don't think we have to fix it; but maybe at least document which tool to use for building off tags/branches ?
<smoser> rharper, well, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/315227
<smoser> i ammended that and that will get you bddeb
<smoser> but its a half-breed thing
<smoser> you get trunk's packaging over the top of the ubuntu packaging.
<smoser> bah. reload --force
#cloud-init 2017-01-21
<Ahrotahntee> evenin' folks
<Ahrotahntee> I can't seem to figure this out via the documentation, so maybe someone here knows a way. I'm trying to get provider metadata into my cloud-init file, I know there are some variables ($public_ipv4) but is there a way to see all variables that are available for my host?
<Ahrotahntee> alternatively, can I set a member with a #include? or does it have to be a fully formed cloud-init block
<Ahrotahntee> nope. including it as a string
#cloud-init 2018-01-15
<kholkina> hi all! can anyone help me?
<dpb1> kholkina: best just to ask your question. :)
<kholkina> I asked a few days ago. I have updated a user-data via openstack nova and now I have a new value when chek it via http://169.254.169.254/2009-04-04/user-data. Also I added [scripts_users, always] in the config file. But it still
<kholkina> I asked a few days ago. I have updated a user-data via openstack nova and now I have a new value when chek it via http://169.254.169.254/2009-04-04/user-data. Also I added [scripts_users, always] in the config file. But it still doesn't work on reboot. When I delete obj.pkl manually and reboot it works fine. But I think it's not the better way
<kholkina> sorry
#cloud-init 2018-01-16
<kholkina> dpb1 hi! do you know an answer for my question published yesterday?
<kholkina> cannot find full logs for this channel
<dpb1> kholkina: perhaps in a few more hours.  yesterday was a US holiday, most of the contributers were off.
<rharper> kholkina: w.r.t user-data; cloud-init does not currently watch the metadata url for changes.  Unless your instance-id changes, cloud-init does not re-read the metadata and take action.   Adding a scripts_user/always won't help if cloud-init has already booted and initialized the instance.   Do you have any background on what you're trying to do?
<kholkina> rharper: I want to have an ability to get (and execute) updated userdata on instance reboot. What is the better way to do this?
<rharper> just a single key or you really want the whole instance to be reconfigured; I don't know all of what you have in your user-data
<rharper> any background about the data change? are you adding a user or installing a package; things that cloud-init normally does during boot?
<kholkina> just a single key in the section 'cloud-config'
<kholkina> now I'm trying to manipulate ssh-keys
<rharper> looking to add a key, or modify existing ones?
<smoser> other question is "host" or "user/public" keys
<kholkina> modify keys in 'default_user/.ssh/authorized_keys'
<smoser> kholkina: right. so for now, cloud-init will only do that once per instance. you could possibly change something to make it do it every boot, but it really only appends keys.
<kholkina> smoser: Yes, I've found that it's possible only to add new keys on reboot. But I need to delete some files from /var/lib/cloud to do this. So would be nice if you can suggest better solution to get an updated data
<smoser> kholkina: well, you could change the frequency of 'ssh' config module
<smoser> but doing so would have the unwanted side affect of re-generating ssh keys for the host
<smoser> kholkina: i dont have a good suggesiton for you at the moment. :-(
<smoser> we do plan to make cloud-init more dynamic in this regard, but nothing right now/
<smoser> what cloud is this?
<kholkina> openstack
<smoser> yeah, k'm sorry. i dont have a good answer for you. :-(
<kholkina> do I need to remove a cache or it will work just with [ssh, always] in cloud.cfg?
<rharper> I kind of think that you don't want to have it regen the host keys;  might be better to watch the metadata URL yourself and read-and-apply the change outside of cloud-init for now
<rharper> smoser: for cached openstack, re-invoking cloud-init single wouldn't have it re-read metadata URL would it?
<smoser> no, it wont.
<rharper> if you removed the obj cache, it might, right ?
<smoser> well, but then it would probably mostly think it was a new isntance.
<rharper> hrm, yeah
<smoser> an dif that wasnt the behavior it did have, then i wouldnt want to tell someone to rely on that
<rharper> yeah
<dojordan> @blackboxsw - question about the dhcp-discovery. I am getting a permission denied on the temp dir in /run/cloud-init when calling dhclient, but there is a comment in the method saying to use /var/tmp since /run/cloud-init is mounted noexcec... Not really sure why its using the wrong directory: Command: ['/run/cloud-init/tmp/cloud-init-dhcp-zz0g2p4_/dhclient', '-1', '-v', '-lf', '/run/cloud-init/tmp/cloud-init-dhcp-zz0g2p4_/dhc
<meetingology> dojordan: Error: Missing "]".  You may want to quote your arguments with double quotes in order to prevent extra brackets from being evaluated as nested commands.
<rharper> wow, uhm, thanks? meetingology ?
<nacc> rharper: did someone forget to stop a meeting?
<rharper> possibly
<rharper> that'd be blackboxsw
 * nacc thought the bot didn't listen outside of meetings
<nacc> #endmeeting
 * nacc may not have perm if they didn't start it
<rharper> right
<rharper> I think blackboxsw might need to do that
<rharper> prolly forgot since open hours last week, I suspect
<nacc> although i thought the bot changed the topic too, but maybe i'm wrong
<nacc> yeah, likely
<rharper> yeah, this is the same instance as the other ubuntu channels, so maybe blackboxsw has it tweaked some
<dojordan> dumb question, but cloud-init runs as root, correct? if so, how come when mkdtemp runs as root it returns /run/cloud-init even though it is mounted noexec?
<rharper> dojordan: two things going on;  1) systemd makes /tmp and /var/tmp unusable as the systemd tmp cleaning service can nuke those at any time; so cloud-init moved it's "tmp" space under a path cloud-init owns (/run/cloud-init)  2)  specifically around the dhclient and tmp dirs is related to apparmor policy around dhclient
<dojordan> 2) makes sense, and I believe I am running into that. However, the temp_utils.py:_tempfile_dir_arg first checks if _TMPDIR has already been set and if so returns it
<dojordan> therefore it does not honor the needs_exe if it is not the first invocation of the function, right?
<rharper> lemme look at the code
<rharper> oh, I see
<rharper> if a non-exec user  calls it first, the second caller cannot reset it
<rharper> that's bit naughtly;
<rharper> well, one hammer is to fetch the global and reset _TMPDIR = None to recall it; but that seems wrong
<dojordan> what's the point of the global
<rharper> makes a singleton
<dojordan> maybe im missing something
<rharper> once we've figured out a TMPDIR, we just keep reusing it
<rharper> we don't need a per-call different tmp directory
<rharper> just need to find one tmpdir with the required features and they all callers use that one
<dojordan> unless we have an needs_exe=False then needs_exe=True
<rharper> yeah, that seems like an issue; I suspect if your datasource is not running local, it's request to have a  tmpdir with needs_exec set isn't the first one; it seems like trouble to re-issue on (if it's currently non-empty)
<rharper> since the EC2 datasource was moved to run at localtime, it is likely a first user of that and gets to set the needs_exec=True first;
<dojordan> what's weird is it is running local but i guess there must be an earlier call still
<rharper> it seems so
<rharper> azure's OpenSSLManager uses mkdtmp
<dojordan> yup, just found that
<dojordan> thoughts on best fix? one option is looks at needs_exe first. If it is true then just use the _EXE_ROOT_TMPDIR
<rharper> I'm not sure yet
<rharper> that's one way; but I
<rharper> I
<rharper> bah
<rharper> I
<dojordan> haha
<rharper> i'm somewhat concerned with changing the global if the dir is populated;
<dojordan> my thought was not to change the global
<dojordan> since the _EXE_ROOT_TMPDIR is static anyway
<rharper> right; one could just return the different directory (assuming it exists)
<rharper> not clear if that could cause confusion w.r.t where a callers files are
<dojordan> but the only way to guarantee one directory then is always use the exe root one, which seems wrong too
<blackboxsw> #endmeeting
<rharper> yay blackboxsw has returned
<blackboxsw> sorry was afk for a kiddo dr checkup. reading scroll back
<rharper> dojordan: I *think* it could be OK for the root case to use the need_exec to pick the path it returns, despite what's set in the global
<blackboxsw> but looks like endmeeting didn't close anything ( and I thought Ihad already posted the closing logs from last meeting last week at cloud-init.github.io
<rharper> blackboxsw: yeah, no idea; maybe meetingology just needs a kick in the pid
<blackboxsw> will ping in #meetingology channel on that, not sure why it responded to that earlier message, all cmds should start with a #
<rharper> hehe
<blackboxsw> testing in a side channel, any command that starts with @ also seems to get parsed by meetinology
<rharper> hehe
<dojordan> @blackboxsw, I am testing out the ephemeral ipv4 stuff now, but for some reason broadcast address is not in the lease file. any thoughts?
<rharper> dojordan: is the netmask included? if so you can derive broadcast IIRC
<dojordan> okay cool. that was my backup plan :)
<dojordan> do I apply the mask to the fixed address or the router to get the broadcast address?
<dojordan> i guess fixed is probably more reliable
<blackboxsw> hrm, I thought we had a helper function for deriving that broadcast addr
<blackboxsw> :w
 * blackboxsw checks around
<blackboxsw> nah was net_prefix_to_ipv4_mask and it's ilk in cloudinit/net/network_state.py
<blackboxsw> I thought I saw a discussion on one branch that discussed potentially writing a utility to do that calculation.
<blackboxsw> if a helper function is writting dojordan to calculate broadcast, I guess I'd like to see if live in cloudinit/net/network_state.py like the other net-related helper funcs
<blackboxsw> if a helper function is written*
<dojordan> cool, i'll put it there
#cloud-init 2018-01-17
<smoser> powersj: c-i is busted for cloud-init . did you look at that ? sorry if i just missed you saying so.
<powersj> hmm I wasn't aware till now
<powersj> ah it is the same thing as with curtin, our lxd config wasn't handing out network addresses
<smoser> ok. fixed ?
 * powersj kicks a nightly run
<powersj> looks good
<dojordan> hey guys, would someone be able to take a quick look at my failed CI run? https://jenkins.ubuntu.com/server/job/cloud-init-ci/696/console It is failing with this: 2018-01-17 02:37:12,902 - tests.cloud_tests - ERROR - stage part: setup func for --deb, install deb encountered error: timeout: after 120s system not started
<blackboxsw> hrm peeking
<smoser> dojordan: i'll push 'build again'
<smoser> dojordan: c-i was busted due to local lxd issue on those sytems
<smoser> powersj fixed, so 'Rebuild' is in https://jenkins.ubuntu.com/server/job/cloud-init-ci/699/
<smoser> so cross your fingers
<blackboxsw> thx smoser
<dojordan> thx all
<dojordan> @smoser, finally got a pass. Can you take a peek at the changes?
<smoser> dojordan: doing so
<smoser> dojordan: does it work ?
<dojordan> I've confirmed xenial does, still need to run an artful pass
<tribaal> Hi folks. I have a bit of a weird issue that I suspect might be cloud-init related, and would love some pointers to investigate:
<tribaal> the kubernetes test suite has a test that simulates a network split/problem by killing the main network interface on the host (a VM using ubuntu cloud images, with its networking set by cloud-init).
<tribaal> the test fails because the network doesn't come back up. Rebooting the machine however brings the network back up as normal
<tribaal> the specific method the test uses to kill the network for 120 seconds is ssh'ing in and doing "nohup sh -c 'sleep 10 && sudo ip link set eth0 down && sleep 120 && sudo ip link set eth0 up' >/dev/null 2>&1 &"
<tribaal> rather crude, but it *should* work in theory
<tribaal> I tried running that on GCE and AWS and both fail (with ubuntu xenial images at least)
<blackboxsw> hrm.... that command also permanently brings down my network on the laptop (which didn't come back)
<blackboxsw> and isn't deployed w/ cloud-init
<blackboxsw> 123
<blackboxsw> n/m I didn't wait long enough
<blackboxsw> wonder if it's networkmanager vs networkd related?
<smoser> tribaal: i'd suggest you can simplify greatly by using screen or tmux
<smoser> rather than  nohup
<smoser> and sh -xc
<smoser> and sending output to a file (and stder)
<smoser> then you can see what happened if you have to go back in
<smoser> tribaal: i cannot reproduce such a failure in lxc though.
<smoser> actually, i might be able to
<smoser> yeah... doing what you're doing is not going to restore any routes
<rharper> http://linux-ip.net/html/tools-ip-link.html
<rharper> there's a section on side-effects of dropping link
<rharper> Now when we down the link layer on eth0, we'll see that there is now no longer a flag UP in the link layer output of ip address. More interesting, though, all of our IP routes to destinations via eth0 are now missing.
<rharper> I suspect the routing is busted
 * rharper has to head out
<rharper> bbiab
<jgomo3> Given ruby as a package in the pakages list, and a commad like `gem install bundler`, How to be sure the command would run after the ruby package had been insalled?
<smoser> i expect that is guaranteed
<smoser> scripts with '#!' or runcmd will run after packages are isntalled
<jgomo3> Thank you @smoser.
<jgomo3> How can I learn about the order of excecution?
<jgomo3> I don't find anything in the documentation... maybe I didn't see well.
<smoser> jgomo3: well, not easily. it is in the modules order
<smoser> in /etc/cloud/cloud.cfg
<jgomo3> @smoser: Oh, I see. TY for the info.
<smoser> in cloud_final_modules
<smoser>  package-update-upgrade-install
<smoser> runs before
<smoser> scripts-user
<jgomo3> @smoser: When you refer to "cloud_final_modules" are you talking about the sourcecode?
<jgomo3> Oh, no, I see now. Thank you.
<jgomo3> I'm trying to do `wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -` but in the `apt` -> `sources` section. What I think could work is to define a Source with `keyserver: https://www.postgresql.org/media/keys/` and `keyid: ACCC4CF8`. But seing the example at "additional-apt-configuration" at line 237, where there is an alternative keyserver, and analyzing the MIT keyserver, looks like there 
<jgomo3> I'll try it, but I bet I'm going to waste a lot of time trying many combinations of  ways to write the `keyserver` and `keyid` values... So: Is there a way to define the URL of the key we want to add by apt-key somewhere in the `apt` section? Or should I write that command with wget in the bootcmd?
<jgomo3> Antoher way I'm thinking is to modify `/etc/ssh/ssh_import_id` to add the postgres key webserver, and use ssh_import_id instead
<jgomo3> Is it possible to modify that file (`/etc/ssh/ssh_import_id`) before ssh_import_id would run?
<jgomo3> `write_files` would do it.
<jgomo3> So, the real question would be: How to add the official Postgres repository via cloud init? -- https://wiki.postgresql.org/wiki/Apt
<rharper> jgomo3: it appears that  for now, you'll need to download the key itself and include that in the 'key' field;  this will call apt-key add <contents of the key>
<jgomo3> @rharper: That's right. Thank you!
<rharper> I wonder why postgresql doesn't publish their key to known servers though
<jgomo3> @rharper: They don't have to do it. And as they, there are many repositories who's keys are published not on "keyservers" but in a simple URL. ssh-import-key is aware of this, and allows to configure it with an URL pattern.
<rharper> I know not everyone publishes; I'm just wondering why not; what's the downside of not publishing the gpg key you use to sign your packages
<rharper> jgomo3: ssh-import-key ?
<rharper> ssh-import-id ; interesting
<jgomo3> ssh-import-id
<jgomo3> No, is not that they don't publish it... they publish it in their own web servers, but not in a keyserver. Maybe Postgres don't trust that launchpad would be eternal, or they think is their responsability to maintain their keys in the Internet.
<rharper> I meant publish the gpg key to the gpg keyservers; there are many gpg keyservers not hosted by canonical; you mentioned the MIT one, debian has one, etc;  I was just wondering why someone wouldn't
<jgomo3> But, the whole point of a PKI is the distributed trust... so, people should change their minds and at least, publish in many trusted keyservers their keys, not only in one place. But, who am I to say anything :D
<rharper> gpg --search-keys ACCC4CF8
<rharper> gpg: searching for "ACCC4CF8" from hkp server keys.gnupg.net
<rharper> (1)	PostgreSQL Debian Repository
<rharper> 	  4096 bit RSA key ACCC4CF8, created: 2011-10-13, expires: 2019-07-02
<rharper> looks like it's there
<rharper> published
<rharper> so maybe you can use keyserver: keys.gnupg.net and keyid: ACCC4CF8
<rharper> https://paste.ubuntu.com/26407200/
<jgomo3> Oh, that is great! Thank you!
<rharper> yeah, maybe that postgresql APT faq can get an update that it publishes the key to gnupg.net
<jgomo3> I was thinking exactly in that. I'll try to make it happen.
<rharper> cool
#cloud-init 2018-01-18
<dojordan> @smoser, I was able to test on artful successfully as well. I also addressed all the CR comments: https://code.launchpad.net/~dojordan/cloud-init/+git/cloud-init/+merge/334341
<dojordan> :q
<dojordan> (sorry)
<dojordan> @smoser, another day another iteration pushed. Can you please take a look when you get a chance: https://code.launchpad.net/~dojordan/cloud-init/+git/cloud-init/+merge/334341
<smoser> blackboxsw: http://paste.ubuntu.com/26412425/
<smoser> what do you think of that
<smoser> i'm basically out of things ic an complain about in dojordan's branch :)
<smoser> so that paste, would simplify some things.
<smoser> interested in dojordan's feedback too
<smoser> and i'd like for blackboxsw to take a look at the branch there to see if i'ive missed anyting
<smoser> (i really don't like 'unknown-245', ... that really should be handled centrally *somewhere* , but thats not dojordan's fault that it is how it is)
<dojordan> I like it, only comment is there are some differences between ec2 and azure that make a generic context manager not perfect
<dojordan> i.e. calculating the bcast addr, and the stupid unknown-245 dhcp option...
<smoser> well, we can just h andle that if they're there.
<smoser> er... in that the general one can use them.
<smoser> if the dhcp server responded with those options, then... you probably should use them :)
<dojordan> true, i guess calculating the bcast addr is pretty generic. want me to incorporate your patch then?
<smoser> yeah. if you're ok with that.
<smoser> i really appreciate your willingness to take feedback
<dojordan> sounds good, and sure, it looks much cleaner
<blackboxsw> hrm sorry wrong channel
<blackboxsw> reading
<blackboxsw> I like the cool factor of nested context managers, but I wonder if we really even need  EphemeralIPv4Network at all smoser. The only consumer was ec2, and azure would now be looking to use the new EphemeralDHCPv4 as well.
<blackboxsw> couldn't we just repurpose EphemeralIPv4Network to do all this for us?
<blackboxsw> I don't really have a strong concern, though existing pastebin doesn't properly accept the fallback_nic param and the context manager return value should be named lease instead of info for clarity
<smoser> blackboxsw: well i think that probably digitalocean *should* use EphemeralIPv4Network
<blackboxsw> I do like encapsulating that maybe_dhcp handling inside the context manager
<blackboxsw> good point.
<blackboxsw> and nevermind on the accepting fallback_nic param. I saw that now in __init__ my bad
<smoser> blackboxsw: i didn't know if i should do the dhcp in the __init__ or in the __enter__
<smoser> i could come up with arguments for both.
<blackboxsw> yeah, I *think* init makes sense here
<blackboxsw> and I think on line 41 you meant +        if not self.ephipv4:
<dojordan> (stupid question) what benefit do you get from doing work in init vs enter?
<dojordan> i guess if you instantiated a context manager w/out a with statement?
<blackboxsw> I'm onboard with the introduction of the EphemeralDHCP context manager. that logic for interacting with EphemerlIPv4Network is kinda klunky. it's best to obfuscate that anyway from DatasourceEc2.
<blackboxsw> I actually think pylint prefers no __enter__ additional params as it looks like it raises an unexpected special method signature. I just had my wires crossed and thought we had generally set those params up in __enter__ instead of __init__
<blackboxsw> I can't seem to find  examples supporting extra params in __enter__
<dojordan> so with CtxMgr(a): would call __init__ with a?
<blackboxsw> correct a would be passed in as param 1 to init
<dojordan> so theoretically, it's not possible to pass additional parms to __enter__?
<blackboxsw> and I can't  create a context manager  class where I'm able to do that.... but I'm just toying around as we speak.
<smoser> dojordan: init versus __enter__
<smoser> its really just a usage thing
<blackboxsw> ahh so __init__ is performed at the same time as __enter__ when we use with statement
<smoser> you could then do:
<blackboxsw> we could init separately and call with later proving the args I think
<smoser>  x = EphemeralDHCPv4("eth0")
<smoser>  with x:
<smoser>    ...
<smoser>   with x:
<smoser>    ...
<smoser> (bad indentation there)
<smoser> if you did the dhcp in __init__ then you wouldnt *re* dhcp each 'with' (__enter__)
<smoser> if you only use the class in one 'with' then they're equivalent.
<dojordan> but you can't pass parameters to __enter__ but you can to init?
<smoser> blackboxsw: when you do:
<smoser> i think that is correct :)
<smoser> dojordan: ^
<smoser> with myclass(name=value) as fp:
<smoser> basically does:
<smoser>  <myclass>.init(name=value)
<smoser>  then fp = <result>.__enter__()
<smoser> at least thats how i understand it
<dojordan> cool :)
<dojordan> next question, is it possible to call __exit__() and __enter__ again?
<dojordan> i.e. when we raise an exception, we go to the next loop iteration and create a new instance of the context manager. is there a better way?
<smoser> well i fyou have the class around you can.
<smoser> but as i did an that paste there isnt a lot of value in keeping the class around for re-use, right?
<smoser> where as if it reused the old lease there would be value, but thats not what you want
<dojordan> right, i see
<dojordan> makes sense
#cloud-init 2018-01-19
<smoser> rharper: chad and i are in hangout
<rharper> right
 * rharper comes up for air 
<dojordan> @smoser, added the ephemeraldhcpv4 content manager: https://code.launchpad.net/~dojordan/cloud-init/+git/cloud-init/+merge/334341
<blackboxsw> dojordan: just got through a pass on it
<smoser> good. i was just abou to ask blackboxsw  to do that :)
<blackboxsw> heh, was just about to ask you if we care about EphemeralDCHPv4 returning {} instead of None on failed lease find, so it'll always return a dict either filled with dhcp options on success, or empty on failure
<blackboxsw> otherwise we get unhelpful Tracebacks if callsites try to lease['myopt
<blackboxsw> otherwise we get unhelpful Tracebacks if callsites try to lease['myopt'
<blackboxsw> otherwise we get unhelpful Tracebacks if callsites try to lease['myopt'] on  a NoneType
<blackboxsw> <- back to typo school with you
<smoser> that was a good pointp you made there.
<smoser> iguess its __enter__ could raise a exception
<smoser> which coudl be caught
<smoser> i like blackboxsw 's feedback.
<blackboxsw> #achievementunlocked :)
<dojordan> cool, i like it. so thoughts on empty dict vs exception?
<dojordan> I'm leaning empty dict
<dojordan> actually, the nice thing about the exception is its already in a try catch
<smoser> i think checking for None. explicitly.   while a lease with no keys would be wierd, checking for None is just more expclcit.
<dojordan> explicit is better than implicit :)
<blackboxsw> good points.
<blackboxsw> ok so leaving None return value then, and callsites should explicity check for it?
<dojordan> on the other hand, the whole point of the EphemeralDHCPv4 context manager is to do dhcp then call the Ephemeral nic with it. if that fails, the whole thing is busted
<dojordan> so we shouldn't even still be in the context manager IMO
<blackboxsw> +1 maybe we actually just raise the exception from there instead of return None.... it's what open's context manager does if file doesn't exist
<dojordan> then inside DataSourceEc2 we can catch the exception and return None
<blackboxsw> maybe a new NoDHCPLeaseError (newly defined in cloudinit.net.dhcp)
<dojordan> also, re: the changing info to lease inside ec2, can we just avoid the variable since the code inside the ctxmgr doesn't reference it
<dojordan> @blackboxsw, should we reuse the InvalidDHCPLeaseFileError?
<dojordan> or catch it and throw a NoLeaseError?
<smoser> yeah.. DataSourceEc2 is why i had it return None
<smoser> exception is fine.
<dojordan> @smoser, @blackboxsw, look again?
<blackboxsw> dojordan: +1 two minor comments on the latest changes. I'll test on azure/ec2 but I'
<blackboxsw> think I'm good there assuming you make the minor changes
<blackboxsw> ... 'this round' of minor changes ;)
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/336366
<smoser> powersj: ^ that actually worked this time in my local test.
<smoser> first pass through i wasn't getting all the botos patched up correctly
<powersj> looking
<powersj> smoser: so if bytes doesn't exist, then we will print the whole 'output' of ec2 console to console? that may fill things up
<smoser> powersj: well, it'll fill the test too.
<smoser> thats only 64k at most
<smoser> if you want i can  make it pring only the first X chars
<smoser> powersj: http://paste.ubuntu.com/26419112/ ?
<powersj> smoser: sure - have you seen it fail and drop down to that?
<smoser> well, only when i was not getting it patched correctly
<smoser> the first version only patched the 'ec2_resource'
<smoser> but then you had created  instances using a session that was not using theh patched path
<smoser> or something like that.
<smoser> either way, yes. i saw it, but if we hit it it means that the "monkey patch" is probably broken
<dojordan> @blackboxsw, fixed, good catch
<dojordan> @blackboxsw, sorry to bother, but any idea when you're able to get around to the azure/ec2 pass?
#cloud-init 2018-01-20
<blackboxsw> dojordan: I'll kick two tests off this weekend. and post results on your branch. we can probably clear that "needs fixing" blemish Monday and land away I expect
#cloud-init 2020-01-13
<meena> turova: what do your logs say?
<turova> meena it looks like it loads a network-config file from the mounted cloudinit drive that has correct values, but those never get to their destination
<turova> it reads from /etc/cloud/cloud.cfg.d/50-curtin-networking.cfg, finds that it has no interfaces to rename (maybe because I already did that) and then just writes the blank config over
<meena> iiiiiinteresting.
<turova> it supposedly works with the premade proxmox OS templates, so I guess I'll just pull one of those down and see what the differences are when I have some time
<meena> *nod *nod *nod
 * meena has never worked with proxmox
<otubo> Odd_Bloke: Hi! I've already signed the CLA, can you take a quick look at https://github.com/canonical/cloud-init/pull/70 whenever you have time? Should I update the branch?
<robjo> no assigned reviewer yet for https://github.com/canonical/cloud-init/pull/162 can we please get some eyes on this? thanks
<blackboxsw> robjo: I'll give first pass today
<blackboxsw> thanks
<meena> robjo: i have read your email to the listâ¦ and i like it.
<meena> robjo: do you think you could outline what a refactoring would look like in my document?
<robjo> meena: I think the step before such an outline is to get conceptual agreement, i.e. rharper blackboxsw, smoser and others would have to agree that this rather large change is the way forward w.r.t. network config writing
<meena> robjo: i meantâ¦ in words.
<robjo> Well it would basically the same as what I stated in e-mail. I thought about whether I wanted to comment in the doc or via e-mail and then decided to go with the e-mail approach
<meena> robjo: aye.
<meena> robjo: so, my main issue re writing network config is that right now, a lot of code for reading and *directly* manipulating (Ephemeral*) network only works on linux.
<meena> so Goneri created a cool patch for FreeBSD, but it only works for DataSources that don't rely on the network.
<robjo> I get that. The question for me becomes do we want to add another way of handling it on the side YOUR_FAVORITE_BSD rendereror are we better of to stick this stuff with the distro implementation
<Goneri> yes, we should have a method in distros/freebsd.py to set-up a temporary network.
<robjo> based on the way things look ATM I think there is a good argument to be made that the network config rendering belongs with the distribution implementation in some way, i.e. the distro implementation knows how it should render the network config rather than the network config is generic and considers distribution needs as a side effect
<Goneri> on BSD, Yes. On Linux, it's different.
<Goneri> this is the reason why the NetBSD and OpenBSD PR come with their own net renderer too.
<robjo> The temporary network setup, while it works, is not linux generic either. It depends on the dhcp-client package which is no longer supported in SUSE
<meena> i believe i did mention that somewhereâ¦
<meena> but it might have just  been IRC, so: lost in noise
<Goneri> Naive question, I'm under the impression the Ephemeral network thing is just here to handle some corner-cases. Is it really the case?
<robjo> A generic way for the temporary network setup is to have Python code in cloud-init that sends a basic dhcp4 and or dhcp6 request
<robjo> smoser: was exploring that at som epoint
<robjo> this would make the temporary network setup generic and should really be the first step
<meena> i've looked at dhcp implementations in python at one point, and i didn't like what i saw.
<meena> it would be cool if the OS had standard network library, which exposed dhcp that we could wrap in ctypes and call
<Goneri> well, we should port NetworkManager on the BSD :-)
<blackboxsw> otubo: thanks! I think you probably need to use util.del_file() https://github.com/canonical/cloud-init/pull/70/files#r365882842  and I just added a couple of minor nits in review as well.
<blackboxsw> otubo: other than the nits, it looks good to go
<blackboxsw> and we can merge that today when you get a chance to resolve or respond/reject comments.
<meena> Goneri: when i suggested we port netplan, you were against it!
<Goneri> I was joking in both cases, the real solution being systemd-networkd :-)
<smoser> meena: python-scapy was the only thing that I thought could relaly get us there.
<meena> smoser: scapy looks powerfulâ¦ dangerously so.
<smoser> meena: well.. doing a dhcp client really *should* be easy.
<smoser> but, almost hilariously, all example dhcp clients in python assume there is already an ip address on the interface.
<smoser> from memory... they do not use raw sockets and would need to to actually do a dhcp request on a nic that did not have an ip yet.
<meena> soooooo, that's why it needs libpcap?
<meena> okay, so, how do we get buy-in? i think maybe i'll be able to attend tomorrow's meeting and try to "sell" robjo's idea?
<robjo> Well the ISC reference implementation for DHCP is Mozilla Public License 2.0 thus we cannot subsume it and wrap a Python interface around it. I am afraid if we are going to be independent of any external tool for the temporary network setup someone has to sit down and write the code in Python and send a PR
<robjo> That would resolve the issue on *BSD for the temporary network setup, and would resolve the issue on SUSE w.r.t. depending on an unsupported package
<robjo> After that the second step is to decide on the renderer approach
<robjo> From my perspective there is buy in for step 1 already
<rharper> replacing the EphemeralDHCP with a scapy (or something else) aka- cloud-init-dhclient is definitely something we're interested in; it allows us to drop the dhclient package from newer images, and sounds like it can be more portable to non-linux distros as well
<rharper> for renderers and net module refactoring; I'm certainly on board with the goal of enhacing cross platform and cross distro support for the cloudinit.net module and additional renderers (and robjo distro variant support within a renderer class)
<meena> i feel like a giant but should be coming out of rharper any second now
<meena> gonâ¦ damn.
#cloud-init 2020-01-14
<meena> installing py36-capy to test some stuffâ¦
<meena> (on freebsd, that is)
<meena> calling dhcp_discover() https://gist.githubusercontent.com/igalic/70f74fd6c8648f3708d1f9f9ffb758f8/raw/70fc31cb9fa8fe8bce290eb1698779c8b7f12a7a/scapy%2520console
<meena> and it returns the IP i already have.
<meena> so, that's pretty cool for a start.
<meena> also, it probably already does the picking of the default interfaceâ¦
<meena> but, to be sure that it works on interfaces which areâ¦ uhmâ¦ disconnected? down? unconfigured? unplumbed? i should add another interface and checkâ¦
<meena> setting an 169.** address, i can issue a dhcp_request('em1') and receive a valid answer
<meena> in order for this to work, all i had to do was set `conf.checkIPaddr = False`
<meena> buttttt: i'm seeing that vagrant at least wants DHCP to be syncronous, and now i wonder how to achieve thatâ¦ or, if it's necessary.
<meena> well, i feel i was productive enough for todayâ¦ for something i'm not getting paid to doâ¦
 * meena gets back to earning the big â¬â¬â¬â¬â¬â¬
<Goneri> The NetBSD and the OpenBSD PRs are ready for review, Feedback are welcome :-)
<meena> Goneri: same for my last PR btw.
<meena> Goneri: https://github.com/canonical/cloud-init/pull/161
<blackboxsw> rharper: Odd_Bloke are there any cloud-init specific branches in review that we would want to block starting a cloud-init SRU?
<blackboxsw> Odd_Bloke: https://github.com/canonical/cloud-init/pull/131 looks approved (was stale and I just updated from master) anything blocking landing this approved PR?
<Odd_Bloke> blackboxsw: Nope, just didn't get back to it once CI had finished.
<blackboxsw> roger, will merge it once ci approves thanks Odd_Bloke
<Odd_Bloke> (Another argument for some sort of landing bot, FWIW.)
<blackboxsw> yeah totally .
<blackboxsw> and then I'll upload to focal and start cutting PRs for cloud-init SRU review
<Odd_Bloke> blackboxsw: I happened to switch back to that tab, so it's landed now.
<blackboxsw> +1 thanks Odd_Bloke
<blackboxsw> ok just reviewed https://github.com/canonical/cloud-init/pull/148 looks good, but need a minor change & unit test
<blackboxsw> and the CLA signed :/
<blackboxsw> community-notice: cloud-init Ubuntu SRU of version 19.4.33  will be starting this week. Target will be to publish updated cloud-init into Xenial, Bionic, Disco and Eoan.
<powersj> no disco
<powersj> blackboxsw, disco is EOL next Thursday
<blackboxsw> schweet!
<blackboxsw> community-notice: cloud-init Ubuntu SRU of version 19.4.33  will be starting this week. Target will be to publish updated cloud-init into Xenial, Bionic and Eoan.
<blackboxsw> :)
<powersj> \o/ thx
<blackboxsw> powersj: I was going to reflect that to the mailing list too (and mention the upstream 20.1 release date) sound good ?
<powersj> yep! did you see my proposed date for 20.1?
<blackboxsw> sending the SRU mail would be to alert people interesting in case they want to test
<blackboxsw> yeah saw your proposed date to us in email I like it
<powersj> So that was the feature freeze date
<powersj> feb 27, so for the cutting 20.1 I was going to propose Feb 18
<powersj> blackboxsw, ^
* blackboxsw changed the topic of #cloud-init to: cloud-init pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting January 21 17:15 UTC | 19.4 (Dec 17) | 20.1 DROP py2.7 (Feb 18) | https://bugs.launchpad.net/cloud-init/+filebug
<blackboxsw> works for me. we had said we'd try Jan 2020. But, alas... Feb 18 is close enough
<blackboxsw> and we don't have much time left in Jan anyway w/ an SRU
<blackboxsw> and travel etc.
<meena> what's disco? did i miss the community meeting?
<meena> powersj: where do i submit docs bugs?
<powersj> meena, same place as other bugs :) Launchpad please and if you tag it docs that would be helpful as well
<powersj> disco is in reference to Ubuntu 19.04 (Disco). The release is EOL next week so need to SRU to it.
<meena> powersj: right. https://bugs.launchpad.net/cloud-init/+bug/1859720
<ubot5> Ubuntu bug 1859720 in cloud-init "Document how to override vendor-data" [Undecided,New]
<powersj> meena, thank you!
<meena> powersj: ah, yeah, i've been using Elementaryâ¦ which uses LTS underneath.
<meena> anyway, i'd really love if someone who's a core contributor could say something on the ML about robjo's mail, which i very much agree with.
<meena> blackboxsw: you could probably directly commit this https://github.com/canonical/cloud-init/pull/148#issuecomment-574370976 to donnydavis' fork/patch thingie.
<meena> Goneri: apparently, you get a warning if the ${NAME_enable} is unset.
<meena> according to some people in #bsdports
<meena> who aren't quite sure
<meena> anyway, bed now
<Goneri> ahah, good night :-)
<blackboxsw> meena: right, I wanted to give donny the right of exclusion on the unit test. but right I could push that in anyway and if donny doesn't like it, it could be removed.
<blackboxsw> minor SRU testing doc pushed https://github.com/canonical/cloud-init/pull/167
<blackboxsw> and SRU PRs are up for Ubuntu xenial bionic and eoan so I can queue those bits for -proposed testing
#cloud-init 2020-01-15
<meena> one problem with scapy might be that we'd have to create / parse lease filesâ¦ oooooooor, we ignore that perform dhcp_request() every time we're called? alternatively, a lease could be a dict for the entire class, where we store that we've been called with interface x, from class y, and received IP z; if interface x still has ip z, we do
<meena> nothingâ¦
<blackboxsw> rharper PRs 171 170 and 168 are fixed for starting SRU testing.
<blackboxsw> meena robjo +1 on the network configuration refactor suggestion discussion. I need to spend a little more time on robjo's existing PR trying to piece together if I have any concerns about this approach.  I agree that the line between distro and network modules is unclean/muddy  and needs a bit of cleanup. I'll try to frame what my perspective is on the best path forward.
<robjo> blackboxsw: Keep in  mind the current pending PR does make no attempts at refactoring anything, it works within the given framework and basically shoehorns more distro knowledge into the sysconfig renderer
<meena> robjo: which it could do without, if we did what you propose?
<robjo> meena: Yes
#cloud-init 2020-01-16
<amansi26> Hi. I need to know, I installed a cloud-init v19.1 on a rhel8. I can see IP at /etc/sysconfig/network-script/ifcfg-env2 but when I do ip a ,IP is not getting assigned.
<amansi26> What can be the possible flow I can check?
<powersj> amansi26, https://cloudinit.readthedocs.io/en/latest/topics/faq.html#where-are-the-logs
<powersj> I would review the logs in the above link
<amansi26> powersj: I check the log, I can see the IP under " applying net config names for" and the datasource used is configdrive
<shubas> Hi all ,is it possible to apply network config from datasource nocloud-net(for vm with pre configured temp network/dhcp) in either stage of cloudinit? asking after searching the docs and multiple testing with no success.
<meena> shubas: can you define no success? by, say, posting your cloud-config, and your  log output?
<shubas> meta-data and user-data are aplied but network-config arn't ,using the same files in iso all configs are aplied.
<shubas> using default centos config with : datasource_list: [ NoCloud ]
<shubas> datasource:
<shubas>   NoCloud:
<shubas>     seedfrom: http://192.168.5.5/
<meena> okay, then lets dig into the logs, and find out why network-config isn't applied.
<shubas> here is the log: https://send.firefox.com/download/fe67a07c819f84ac/#xI5VGQGxaK6pYqTiWJ947w
<meena> shubas: it says: 2020-01-16 09:57:49,299 - DataSourceNoCloud.py[DEBUG]: Seed from http://192.168.5.5/ not supported by DataSourceNoCloud [seed=None][dsmode=net]
<meena> oh, wait, that was init-local
<meena> 2020-01-16 09:57:58,999 - stages.py[DEBUG]: applying net config names for {'version': 1, 'config': [{'subnets': [{'type': 'dhcp'}], 'type': 'physical', 'name': 'eth0', 'mac_address': 'aa:ce:dc:b2:5c:06'}]}
<meena> so, shubas this reads fairly okay / successful to me. What about the config that you get as a result is different from what you're expecting?
<shubas> this is the fallback config ,it does not fetch the network-config file from seed
<meena> shubas: but it says it's doing thatâ¦
<meena> shubas: what's curl http://192.168.5.5/meta-data look like?
<meena> (or whatever the correct url is)
<shubas> also checked the access log for http://192.168.5.5/ , only fetching meta-data and user-data files
<shubas> the curl is ok ,i get the content
<shubas> is there a way to manually fetch and apply network config from seed through cloud-init?
<shubas> might provide more insight on what's happening
<meena> shubas: what's the config look like?
<shubas> cloud.cfg?
<meena> shubas: no, i meant the thing that you get from curl
<shubas> meta-data -
<shubas> instance-id: someid123
<shubas> user-data -
<shubas> #cloud-config
<shubas> hostname: somehostname
<shubas> network-config -
<shubas> network:
<shubas> version: 1
<shubas> config:
<shubas>    - type: physical
<shubas>      name: eth0
<shubas>      subnets:
<shubas>         - type: static
<shubas>           address: someIP
<shubas>           netmask: someMASK
<shubas>           gateway: someGW
<shubas>           dns_nameservers:
<shubas>             - 8.8.8.8
<shubas>             - 8.8.4.4
<meena> not sure what, but something there isn't quite right
<meena> would be cool if it said what, rather than just applying dhcp
<shubas> at this point my thought is that the network config mathod is determined in the intial stage before the network seed is checked and so ignored in later stages
<otubo> talking about configuration, meena shubas can you confirm network_state is populated by conf file?
<otubo> I'm seeing too much code lately, need a visual confirmation :-)
<shubas> @otubo can you clearify what do you mean by network state and where should it be populated?
<otubo> shubas: sorry! I was able to figure out by my self. The answer is yes :)
<otubo> shubas: network_state is the variable filled with network configuration both read from cloud.cfg and from already existing network configuration present on the guest
<otubo> also, removing /var/lib/cloud/instance would be enough to emulate the first boot? Or should I need to do something else?
<shubas> "cloud-init clean" will do that
<meena> cloud-init clean --logs --reboot \o/
<otubo> thanks people!
<rharper> otubo: yeah, reading the bug now
<otubo> rharper: _render_networkmanager_conf() is writing 99-cloud-init.conf *after* NM starts, and that's causing the problem
<rharper> otubo: so, the design for the network config  by cloud-init is that at local time, we crawl imds for network config, and write out this config to the os dirs, *before* the networking service starts;  so in your case, we've written sysconfig *and* resolv.conf values before NM starts
<rharper> otubo: so why does network-manager start before cloud-init-local.service  ?
<rharper> Before=NetworkManager.service
<rharper> Before=network-pre.target
<rharper> we run local before NM and network-pre.target;  this is when we write out all of our network config, so the 99 file *is* written before NM starts
<rharper> or something is very wrong with units
<otubo> So what you're saying is that on first boot, resolv.conf comes preconfigured for some reason, cloud-init writes dns=none and NM starts. And even though this happens NM wipes out the file on shutdown
<rharper> otubo: and " it does not change the resolv.conf file and this file is clean (reverted to clean state by NetworkManager during the first shutdown)."   seems to be where our conflict lies;  you said that NM does this because it started before the 99-cloud-init.conf which tells NM *not* to do that
<rharper> otubo: I don't know the contents of your resolv.conf before boot;  typically it is a symbolic link to the systemd-resolved local caching resolver; but in older images it may be an empty file
<rharper> I know on SUSE we've had to adjust whether cloud-init writes anyting at all (we used to always write resolv.conf even if we didn't have dns values, which was fine for Ubuntu which had resolvconf managing the file)
<otubo> rharper: ok, good thoughts. I need to confirm the exact boot sequence and if NM is processing correctly the options on the config file.
<rharper> the correct sequence should be:  1. cloud-init local runs first, before NM, reads net config, writes out all sysconfig/NM config files 2. NM starts, and it should see the 99-cloud-init.conf which says "dns = none" which should prevent it cleaning it up on shutdown  3. cloud-init net runs after NM has started but before all things are online ...
<rharper> is the target OS systemd based?  if so, I've found journalctl -o short-monotonic -b -u cloud-init-local.service -u NetworkManager.service -u cloud-init.service   to be useful to see when they started
<rharper> otubo: your log seems to indicate that there may be a race here;  but if you have a complete boot log from both scenarios, we can see if there's a different bug here;
<rharper> https://bugs.launchpad.net/cloud-init/+bug/1843334
<ubot5> Ubuntu bug 1843334 in cloud-init "Change location of DHCP leases in CloudStack provider as it doesn't work for RHEL8" [Medium,Fix released]
<rharper> otubo: that landed very recently, maybe that's not present in your cloud-init yet ?
<rharper> which could explain the race
<otubo> rharper: it looks like the boot sequence is correct, cloud-init indeed starts before NM https://pastebin.com/y65yFajd
<rharper> otubo: yes, but I wonder if there's a race between when cloud-init-local finishes and NM starts ...
<otubo> rharper: but I think NM ignores dns=none configuration, along the logs I can see one single entry of NM updating DNS
<rharper> or it's not yet written
<rharper> the ordering ensures that a unit is started before or after, but not necessariy complete
<otubo> oh I see
<otubo> rharper: rhel tree doesn't contain this fix for the bug 1843334 you pointed. I'll cherry pick and test.
<ubot5> bug 1843334 in cloud-init "Change location of DHCP leases in CloudStack provider as it doesn't work for RHEL8" [Medium,Fix released] https://launchpad.net/bugs/1843334
<rharper> that may be enough to ensure there's time for local to write it's config
<rharper> otubo: so I think the ordering is strong enough;  however, in the case where cloud-init local is doing a dhcp for metadata and the dhcp/network response is slow; I'm wondering if we'd find that NM would start before cloud-init writes that file;  if so;  we *might* want to issue a 'systemctl try-restart NetworkManager.service'  which would restart NM IIF it was already started.
<otubo> rharper: Well, but i believe this is a good idea even if that fix solves my problem.
<otubo> rharper: I could write that feature in a near future
<rharper> yeah
<rharper> look at cloudinit/net/netplan.py  which uses the _postcmds config; it allows a Distro class to specify commands to run after rendering a network config
<rharper> the freebsd renderer also makes use of this as another example
<otubo> rharper: yeah, didn't work.
<rharper> so
<rharper> local also does this: ExecStart=/bin/touch /run/cloud-init/network-config-ready
<rharper> hrm, we want logic like:  if cloud-init ran, NM should wait untll this file is touched before starting;  but in the case that cloud-init doesn't run (it's not activated) then NM should run whenever it normally would;
<otubo> rharper: what about we reload configuration when cloud-init.final finishes? Like `pkill -HUP NetworkManager'
<rharper> hrm, well, we could PreExec the reload in cloud-init.service as a drop in
<rharper> until code changes to use the try-restart land and release
<otubo> rharper: ok, I'm gonna try to include something with `systemctl try-restart NetworkManager.service' on _postcmds tomorrow morning. Thanks for the help :-)
<rharper> sure
<rharper> https://paste.ubuntu.com/p/tGyGWJBq5t/
<rharper> otubo: that is a systemd unit drop it config which would trigger the try-restart right before cloud-init.service
<blackboxsw_> Odd_Bloke: just pushed https://github.com/cloud-init/ubuntu-sru/pull/79 for SRU start
<blackboxsw_> -> errand
<Odd_Bloke> rharper: util.subp will raise a ValueError if target is anything but None.  The docstring says this is for compatibility with curtin's subp.  Do you think that's a goal we still want to retain?
<Odd_Bloke> (I'm asking because nothing actually calls get_dpkg_architecture with a target that isn't None, so I was going to remove that parameter, then dug deeper and found this.)
<blackboxsw> Odd_Bloke: I just added another note about what clouds SRU verification already covers https://github.com/canonical/cloud-init/pull/167
<blackboxsw> SRU -proposed bits are up and accessible in xenial bionic and eoan-proposed. I'm writing up the notification email now
<Odd_Bloke> blackboxsw: https://github.com/canonical/cloud-init/pull/167 reviewed, we're getting there!
<blackboxsw> pushed changes Odd_Bloke thanks
<blackboxsw> and accepted all suggestions
<Odd_Bloke> blackboxsw: LGTM now.  I have another couple of minor nits (a missing : and the directive names), which I'll apply then merge.
<Odd_Bloke> blackboxsw: Oh, no I'm not, I don't have permissions.
<blackboxsw> ?
<Odd_Bloke> blackboxsw: I can't commit to your branch, so you'll need to apply this last round of nits. :)
<blackboxsw> Odd_Bloke: I just clicked allow edits
<blackboxsw> Odd_Bloke: does it work now I forgot to click that
<blackboxsw> I think in general Odd_Bloke if we are working through a bunch of nits it might be easier for you to review and me to grok the set of changes if you push over my branch, then I can diff to origin and walk through the full visual diff. walking through a bunch of accept suggestion links makes me feel I'm going to miss something.
<Odd_Bloke> OK, we can bear that in mind next time.
<Odd_Bloke> That's now how I prefer to receive changes, which is why I didn't try to do that. :)
<Odd_Bloke> (And also, I want you to be happy with the changes being made before I make them, unless they really are trivial!)
<blackboxsw> makes sense. I was just feeling a bit guilty for the number of review rounds you had to do Odd_Bloke
<blackboxsw> Odd_Bloke, so do I need to accept your last round of review comments or are you able to push to my branch?
<Odd_Bloke> Oh, I don't mind that at all.  I've been feeling guilty about how many review rounds you had to do. :p
<Odd_Bloke> blackboxsw: It's telling me I still don't have permissions, so why don't you accept them.
<Odd_Bloke> We can figure that out next time.
<blackboxsw> good we all have feelings of guilt to work out this year it seems :)
<blackboxsw> will do
<Odd_Bloke> rharper: blackboxsw: FYI, I've asked waveform (the Pi guy on the Foundations team) to file a bug for a mount issue they're seeing with cloud-init (I assume in Ubuntu Core), as they were talking about how to work around it.
<Odd_Bloke> blackboxsw: You have a long line in there still, doc8 fails.
<Odd_Bloke> doc/rtd/topics/debugging.rst:171: D001 Line too long
<blackboxsw> d'oh
<Odd_Bloke> The dangers of using the GH suggestions. :)
<rharper> Odd_Bloke: oh, where was the discussion re: mount issues ?
<rharper> bug is best though
<Odd_Bloke> rharper: In the Foundations channel that I never /part'd. :p
<rharper> it's more likely run with systemd but cloud-init plays a role in writing those fstab entries out
<rharper> gotcha
<blackboxsw> Odd_Bloke: force pushed the sru doc branch
<blackboxsw> should be good2go
<blackboxsw> community notice: I also just pushed a origin/stable-19.4 branch which was our last version of cloud-init to claim support for py2.7
* blackboxsw changed the topic of #cloud-init to:  cloud-init pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting January 21 17:15 UTC | 19.4 (Dec 17) | 20.1 DROP py2.7 (Feb 18: origin/stable-19.4) | https://bugs.launchpad.net/cloud-init/+filebug
* blackboxsw changed the topic of #cloud-init to:  cloud-init pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting January 21 17:15 UTC | 19.4 (Dec 17) | 20.1 DROP py2.7 (Feb 18) | https://bugs.launchpad.net/cloud-init/+filebug
<Odd_Bloke> rharper: It's something to do with the cloud-init seed being on a drive that's already being mounted for some other reason; when we go to mount it, it's already mounted so *sad trombone*
<Odd_Bloke> But a bug is incoming.
<rharper> I suspect this is core20 fun
<Odd_Bloke> Yeah, I don't see why a Pi would have cloud-init (in a weird configuration) on it if it weren't.
<rharper> the writable partition includes the seed so it can be modified outside of the read-only image;
<rharper> oh, non-core pi image has cloud-init to auto generate keys, import them and do that outside of the base image;
* blackboxsw changed the topic of #cloud-init to:  cloud-init pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting January 21 17:15 UTC | 19.4 (Dec 17) drops Py2.7 : origin/stable-19.4 | 20.1 (Feb 18) | https://bugs.launchpad.net/cloud-init/+filebug
<Odd_Bloke> Aha, OK, well we'll find out!
<rharper> core just makes things more complicated w.r.t not having a writable filesystem
<rharper> it was a nice improvement to allow cloud-init to read the boot directory of the pi's to look for cloud-config; so they can do a dd of the image and then write a cloud.cfg in the boot dir
<rharper> yes, we shall see =)
<Odd_Bloke> blackboxsw: Did you overwrite the last set of changes you applied from the GH UI?
<Odd_Bloke> Looks like the directives have reverted.
<blackboxsw> geez man, you're probably correct.
<blackboxsw> I'll re-apply
<blackboxsw> Odd_Bloke: tried to reapply, erased the code-block suggestions plus and opening cloud-init:
<Odd_Bloke> blackboxsw: Approved. \o/
<blackboxsw> thanks!
<johnsonshi> Hi, whenever I run tox -e py27, I get an error about test-requirements.txt
<johnsonshi> Requirement already satisfied: setuptools in ./.tox/py27/lib/python2.7/site-packages (from -r /source/test-requirements.txt (line 13)) (45.0.0)
<johnsonshi> ERROR: Package 'setuptools' requires a different Python: 2.7.12 not in '>=3.5'
<johnsonshi> ERROR: could not install deps [-r/source/test-requirements.txt]; v = InvocationError('/source/.tox/py27/bin/pip install -r/source/test-requirements.txt (see /source/.tox/py27/log/py27-1.log)', 1)
<johnsonshi> I have not modified test-requirements.txt at all, and my tox tests were running fine until a few days ago
<johnsonshi> Any possible reasons why this error is being thrown?
<Odd_Bloke> johnsonshi: I believe setuptools have dropped support for Python 2 in the latest release, which is what's causing that error.
<Odd_Bloke> johnsonshi: We've followed suit, however, and 19.4 was the last cloud-init release that supported Python 2.
<Odd_Bloke> So you probably don't need to run `tox -e py27` any longer, and instead should be running `tox -e py3`.
<johnsonshi> Odd_Bloke: Thanks for clearing this up!
<Odd_Bloke> johnsonshi: Happy to help!
#cloud-init 2020-01-17
<cedricvr> Hello, any known problems/difficulties with the [`ssh_key`](https://cloudinit.readthedocs.io/en/latest/topics/examples.html#configure-instances-ssh-keys) parameter in cloud-init? I am trying to use it in a AWS EC2 instance managed via CloudFormation and the keys in `/etc/ssh` are fresh ones instead of the ones I provided in the cloud-init file.
<cedricvr> Nevermind, now it works!
<meena> cedricvr: what fixed it?
<cedricvr> No idea what I did. Maybe just recreate the CloudFormation stack instead of âupdatingâ it
<cedricvr> Yesterday evening I tried using this parameter and it didn't seem to work, and today tried again and this time it works. The frustrations of infrastructure management ...
<otubo> rharper: any reason why you opted for `try-restart' and not `reload'?
<otubo> Odd_Bloke: blackboxsw if you guys have a time to merge my PR that would be awesome :-) https://github.com/canonical/cloud-init/pull/70
<meena> otubo: because, if nm isn't running yet, it won't do anything
<otubo> meena: right right, good point.
<otubo> meena: and I assume `reload' will return error in case the service is not running.
<meena> otubo: try it out.??
<otubo> meena: :) cnofirmed, it will return error.
<rharper> otubo: meena 's spot on;  =)
<otubo> rharper: the way things are handled systemd-wise, I think I'll have to create new directories like ./systemd/[system,cloud-init.service.d], move the systemd files into system and the drop-in into cloud-init.service.d. And fix all the scripts that refer to them.
<otubo> rharper:sounds good?
<rharper> otubo: for drops, I would suggest packaging changes in the rpm vs. upstream;  and drop-ins can go out in /etc/systemd/system/ (which already exists) you will need to mkdir the cloud-init.service.d and the conf file
<rharper> otubo: long term (upstream), a PR to use post-commands in sysconfig renderer to try-restart network-manager (if we've written out a nm conf file) will remove the need for the drop-in
<otubo> rharper: Hm, ok, then. I'm gonna do a downstream-only fix then, and will work on the renderer later.
<rharper> +1
<rharper> that seems the fastest path (and it can work for older releases  if you don't plan on backporting the fix)
<Goneri> could you take a look at https://github.com/canonical/cloud-init/pull/62? The PR should be harmless, and it brings the support for NetBSD. And it's a dependency for OpenBSD. Pleaaaaase :-)
<blackboxsw> Goneri: will give a pass now
<ananke> so I'm a bit confused. I dropped a sample override config in /etc/cloud/cloud.cfg.d/defaults.cfg, which included things like write_file invocation. this seems to work just fine when the instance is created, but only that first time
<rharper> ananke: each cloud-init config module has a default frequency; runcmd is per-instance; which means the first time it's run only;
<ananke> shouldn't it be invoked with every boot? /etc/cloud/cloud.cfg lists 'write-files' module under cloud_init_modules section
<ananke> rharper: ahh, so the frequency is per module
<rharper> yes
<ananke> doh, that would explain. thank you. I see it documented now
<blackboxsw> if we want every boot https://cloudinit.readthedocs.io/en/latest/topics/modules.html#scripts-per-boot
<rharper> for scripts, and you're modifying the image, it's likely easiest to drop them into /var/lib/cloud/scripts/per-{boot,instance,once}
<rharper> selecting the frequency you want
<rharper> by directory
<rharper> that dir is invoked via run-parts (man 8 run-parts)
<ananke> yeah, I was already leveraging those locations, but I saw how easy write_files was for a small config file, I figured I would use that in this case. back to the drawing board
<ananke> is there a quick reference matrix that shows each module and its default frequency?
<rharper> ananke: no, but that's a great idea to document;  it's specified in the config module, cc_<modname>.py
<rharper> you'll see 'frequency' attribute; the default without a value is 'per-instance'
<rharper> https://cloudinit.readthedocs.io/en/latest/topics/modules.html#write-files
<rharper> it is documented though (just not all in a single table)
<rharper> https://cloudinit.readthedocs.io/en/latest/topics/modules.html  as a list of the modules
<ananke> yep, thank you. was just going down that page and searching for 'Module frequency'
<rharper> there is a way to modify the frequency of a module, but not from user-data; as cloud-init reads system config /etc/cloud/* to determine how often to call specific modules;  and even that there isn't an easy way to modify the value in config; (the modules list accept a single list of names, or a tuple of name, freq)
<blackboxsw> thanks Goneri for https://github.com/canonical/cloud-init/pull/62/files and meena  for review too. suggestions posted
<blackboxsw> <- errand
<meena> 15:49 <@rharper> otubo: meena 's spot on;  =) â¬ï¸ this statement could be generalised, but let's not stroke my ego too much
<meena> blackboxsw: have you had time to mull over robjo's networking refactoring (yupp, it's his now ;)?
<blackboxsw> heh. I did a bit and wanted to capture some thoughts today, I've freed up a bit now that SRU process has started
<rharper> meena: =)
<blackboxsw> ahosmanMSFT: hiya, just a note that we've queued your Azure instance-id byte-swap fix for SRU testing in Ubuntu Xenial, Bionic or Eoan. steps to manually test on whatever affected instance types you had access to would be here: https://cloudinit.readthedocs.io/en/latest/topics/debugging.html#manual-sru-verification-procedure
<blackboxsw> I'll be testing more frequent deployment mechanisms on Azure today.
<Goneri> blackboxsw, thanks for the review :-)
<blackboxsw> no problem Goneri
<meena> i only discovered one mthod that was already implemented in distro/__init__.py
<meena> and only because i had read the code recently.
#cloud-init 2020-01-19
<akik> 7
<akik> hi, how do i disable cloud-init's default_user creation? is it in /etc/cloud/cloud.cfg ?
<akik> is it these two lines that create centos user on centos 7?
<akik> users: - default
