#cloud-init 2014-02-24
<harmw_> meh, still no openstack community newsletter
<smathews> hey guys, im using 0.7.4 and noticed that with the config drive source that when it checks the uuid for previous installs it fails because the length of the uuids differ (extra white space) so it runs on_first_boot everytime, is this a known bug (maybe fixed already)
<smathews> ?
<smathews_> example http://pastie.org/8769066
<smathews> apologies if I missed a responce, the web client keeps disconnecting me
<smoser> smathews, why does it find different content in the uuid ?
<smoser> smathews, ie, that doesn't look like insane behavior from cloud-init's perspective. why should it be expected to trim data ?
<smathews> my guess is that when it writes there is an extra white space at the end of the uuid in the file
<smathews> I dont think the logic here is incorrect, or that it should have to trim
<smathews> maybe the write of the uuid is incorrect
<smathews> also I think this is related, since if the uuids were matching it wouldnt run the network config https://bugs.launchpad.net/cloud-init/+bug/1275098
<smathews> on subsequent launches that is
<smathews>         util.write_file(iid_fn, "%s\n" % iid)
<smathews> looks like its from the newline
<smathews> thoughts?
<smoser> smathews, it looks like you're right.
<smoser> harlowja, ^
<harlowja> hmmm, newlnes!
<smathews> heh
<harlowja> https://code.launchpad.net/~harlowja/cloud-init/strip-iid-newline/+merge/208028 
<smathews> so, while we are on the topic of config drives, can I make a request for in on_first_boot to write the files before applying the network config?
<smathews> this would be helpful for when writing static routes
<harlowja> seems fair
<harlowja> any objections smathews ?
<harlowja> smoser ^
<harlowja> lol
<smathews> heh
<harlowja> i assume smathews  doesn't have objects :-P
<smathews> hmmm I generally dont object with myself... 
<smathews> although lol
<harlowja> :)
<smoser> is it not easy to not write the carraige return?
<smoser> clearly strip() should be safe
<smoser> but we shoud just not write baad data
<smoser> or even bad data
<smoser> harlowja, one othe rhting i meant to mention to you.
<harlowja> smoser likely it already exists out there with "\n" for quite a few versions, so better to just strip?
<smoser> the datasource for openstack right now reads from "latest"
<smoser> which is imo wrong
<harlowja> ?
<smoser> i'm pretty sure it tries "latest" in the metadata service.
<smoser> this is somewhat memeory, but there were 2 things i wanted to fix there.
<harlowja> k
<harlowja> ya, which version to use?
<smoser> a.) i thought it was reading from "latest" by default. as opposed to GRIZZLY or HAVANA (their YYYY-MM-DD respectively)
<harlowja> sure, seems so
<smoser> b.) it was warning when it tried one. warn goes to stdout, which goes to console and log as a warn
<smoser> and that wasn't really a failure case.
<harlowja> agreed
<harlowja> repairing
<smathews> smoser, thoughts on writing files before apply network in on_first_boot?
<smoser> smathews, i dont follow
<smoser> writing files as in 'write_files' ?
<smathews> yeah, in on_first_boot apply_network happens before write_files
<smathews> Id like to switch that
<smathews> it'd be handy for writing static routes and such
<smoser> smathews, i have to thikn about it. sorry . have to run right now
<smathews> no worries
<smathews> thanks for your help!
<harlowja> smoser https://code.launchpad.net/~harlowja/cloud-init/ds-os-adjust/+merge/208031
<harlowja> whenver u get free
#cloud-init 2014-02-25
<smoser> harlowja, https://code.launchpad.net/~smoser/cloud-init/run-status/+merge/208056
<smoser> i'd like your thoguhts on that.
<smoser> perhaps you have a way to make it do roughly equivalent mbut more configurable (currently just hard coded paths)
<harlowja> hmmm
<harlowja> scanning
<smoser> the goal is to allow something other than "post your logs" to debug with :)
<harlowja> :)
<smoser> i'm not un-happy with it, other than:
<smoser> a.) it kind of relies on /run being tmpfs
<smoser> b.) paths aren't configurable
<harlowja> hmmm, ya, is /run on freebsd (not really sure)
<harlowja> u could also hook in the callback into stages.Modules
<smoser> stages.modules ?
<harlowja> ya, just an idea
<harlowja> could update the json everytime that thing runs a module
<harlowja> why json, not yaml?? :-P
<harlowja> only other thing i can think of is it seems like status_wrapper could take some more options in, 
<harlowja> in a way its like status_wrapper should have 3 functors
<harlowja> 1, pre-function stuff
<harlowja> 2, the actual function called
<harlowja> 3, the post-function stuff
<harlowja> status_pre, status_go, status_post
<harlowja> or something
<smoser> harlowja, yeah, that is a simplification. 
<smoser> i think i'm gonna merge whats there though. 
<harlowja> smoser okdokie
<harlowja> smoser do u remember which branch had the jsonschema code i was working on a while ago?? think i lost it ,lol
<smoser> no. i dont remember.
<harlowja> np
<harlowja> smoser i should probably restart that work, adding jsonschema validation for the different modules, still think thats useful?
<smoser> i thikn useful, yeah, but not going to get into 14.04.
<harlowja> np
#cloud-init 2014-02-26
<smoser> harmw_, around ?
<smoser> https://bugs.launchpad.net/cloud-init/+bug/1285185
<harmw_> :)
<harmw> doesn't look good
<smoser> doesn't look good ?
<smoser> oleg diagnosed the failure well
<smoser> ubuntu shows 'inet addr:xx.yy', freebsd shows 'inet xx.yy'
<smoser> i'm thinking of doing a more specific hack
<smoser> http://paste.ubuntu.com/7001006/
<harmw> no no, the bugreport is excellent
<harmw> its just a shame I allowed it to break
<smoser> compared to oleg's solution which was to allow overriding fields
<harmw> inet 10.14.92.96  netmask 255.255.255.0  broadcast 10.14.92.255
<harmw> thats Fedora
<smoser> http://paste.ubuntu.com/7001019/
<smoser> i'll update the comment then. seems freebsd is that also, no ?
<smoser> http://lists.freebsd.org/pipermail/freebsd-hackers/2011-April/035015.html
<harmw> inet 10.9.0.1 netmask 0xffffff00 broadcast 10.9.0.255
<harmw> freebsd 10
<smoser> right.
<smoser> so we're not handling the hex on freebsd
<smoser> which is fine
<smoser> well, not fine, but another bug
<smoser> does this look reasonable: http://paste.ubuntu.com/7001024/
<smoser> (reasonable as a "just get something to fix this")
<harmw> hm yes
<harmw> btw, which net-tools version does ubuntu install?
<smoser> harmw, can you pastebin ifconfig output on a freebsd?
<smoser> $ dpkg-query --show net-tools
<smoser> net-tools	1.60-25ubuntu2
<harmw> centos 6 ships with that version aswell, showing addr:
<harmw> fedora 19 installs 2.10-alpha
<kwadronaut> harmw: http://packages.ubuntu.com/ you can find it there for the different ones
<harmw> h ok
<smoser> well, collect whatever you have easy access to
<smoser> we really should have tests for this with examples
<harmw> +1
<kwadronaut> it's going to stick to 1.60 for the next couple of years in ubuntu methinks
<harmw> http://paste.ubuntu.com/7001095/
<smoser> harmw, thanks
<harlowja> harmw just checked with sean, he'll be back tommorow, he was on vacation, still catching back up
<harlowja> told him the cloud-init channel misses him
<harlowja> lol
<harmw> ok, great
<harlowja> so i guess after he gets back into the swing of things he'll be back on here
<harmw> yea :)
 * seanwbruno returns from the madness
<harlowja> seanwbruno harmw woot
<harlowja> welcome back from the other side
<seanwbruno> dude
<seanwbruno> its dark over there.
<harlowja> come out of your cave?
<harlowja> lol
<harlowja> are they putting u in the basement again seanwbruno ?
<harlowja> *did they take your redstapler?
<seanwbruno> nah, was away from the office for a week.
<seanwbruno> also, is EVERYONE sick here in Sunnyvale?
<seanwbruno> geezus.
<harlowja> ya, i had sniffles yesetday, seems better today
<harlowja> seanwbruno u got all the legal stuff resolved, so u signed canonical stuff right?
<harlowja> think u said something about that
<seanwbruno> I got *approval* to sign it.
<seanwbruno> I haven't done the deed yet or whatever
<harlowja> ah, k, deed doing shouldn't be much then
<harlowja> i think its just http://www.canonical.com/contributor-license-agreement/submit
<harlowja> and put smoser as the project contact
 * harlowja but i might be wrong, did that a long time ago
<smoser> that sounds good enough
<harlowja> k
<harmw> ah nice, seanwbruno 
<harmw> say harlowja , you happen to known anyone that did the openstack exams from red hat?
<harlowja> is that like a doctor checkup harmw ?
<harlowja> where RH is the doctor
<harlowja> cough?
<harmw> lol
 * harlowja didn't know there was an openstack exams
<harlowja> wonders what that is
<harlowja> lol
<harmw> http://www.redhat.com/training/courses/ex210/
<harlowja> interesting
<harlowja> RH already has examns, crazy
<kwadronaut> they fist did the cups training, and since then they *love* to print paper.
<harlowja> i give u free exam
<seanwbruno> ok, buttons "pushed" on launchpad and license signing thing.
<harlowja> Price:$600 
<harlowja> i give u examn for half off
<harlowja> 300$
<harlowja> josh discount
<harmw> does that include a flight and hotel? :>
<harlowja> :-/
<harlowja> virtual flight
<harlowja> virtual hotel
<harmw> hehe
<harlowja> harmw so thats a negative, don't know anyone
<harmw> :)
<harlowja> for 300$ i give u history of openstack and for $350 i can give u a video of me that explains openstack
<harlowja> lol
<harmw> lol
<harlowja> including bonus material
<harlowja> PG-13
<harmw> 13?
<harlowja> lol
<harmw> nvm thn 
<harmw> :p
<harlowja> u got me interested, i wonder what is in this exam, lol
 * harlowja wonders if i would pass
<harmw> well, me to
<harmw> though I'm not realy into classroom training in this case, I'd rather just pick up a book on the subject
<harlowja> def, and the last book that i saw was a copy/paste of the openstack docs
<harlowja> lol
<harmw> :|
<harlowja> http://shop.oreilly.com/product/0636920021674.do
<harlowja> i never understood that book
<harlowja> came out in 2011, when openstack still figuring itself out
<harmw> jeez
<harlowja> was pretty much online docs with some comments
<harlowja> was weird, lol
<harmw> i can imagine
<harmw> http://www.redhat.com/training/courses/cl210/course-exam-outline
<harlowja> interesting
 * harlowja not sure i'd pay 600$ for that, if u know the right people u can get it for free, lol
<harlowja> i've done all the install, configure, package stuff with http://anvil.readthedocs.org/ ,lol
<harmw> hehe, does that include the piece of paper as well?
<harlowja> u got a printer harmw ?
<harlowja> lol
<harmw> :p
<harlowja> seems like an introductory course though
<harlowja> how to install, and do basic management
<harmw> looks quite like it indeed
<harlowja> u get to Y! office, give u josh discount
<harlowja> lol
<seanwbruno> "training includes free lunch in Y! cafeteria"
<harmw> yeah, but let me get some weeks vacation for that first - plus flight money :p
<harlowja> and free bathroom usage
<harlowja> no fees
<harmw> Price:2465 
<harmw> thats for the online tut
<harmw> *tutorial
<harmw> includes a hdd though :>
<harmw> 'Upon submitting your registration and payment, you will receive an e-mail notification with instructions on how to access course content. Approximately 3-4 business days after, you will receive the pre-loaded hard drive via priority mail.'
<harlowja> for real?
<harmw> http://www.redhat.com/training/courses/cl210r/
<harlowja> bahaha
<harlowja> thats funny
<harlowja> i guess they aren't using there own dogfood
<harlowja> seems like u could host the class in a openstack cloud
<harlowja> and provide people with snapshots
<harlowja> and provide people access to the snapshot with the class + software on it
<harmw> I see you can pretty easily roll a y! thingy yourself :p
<harlowja> likely
<harmw> btw, I'm still having timout between nova and neutron causing all kinds of mayhem
<harmw> moving keystone to a dedicated piece of hw didnt help afterall
<harlowja> harmw ah, neutron
<harlowja> what kind of timeouts, boot timeouts?
<harlowja> harmw https://review.openstack.org/#/c/74832/
<harlowja> that might help u ;)
<harlowja> i bet they won't tell u that in the RH training, that nova <-> neutron is racy, lol
<harmw> when I boot a new instance, everything goes just fine - up until cloud-init contacts the nova's metadata api, which in fails becaus it can't communicate with neutron. Running a curl against it later on when the instance has given up on metadata it runs ok 
<harmw> lol!
<harmw> let me dig up the exact error
<harmw> (ill read up on that review in a sec)
<harlowja> harmw its proably due to that race
<harlowja> 'Make libvirt wait for neutron to confirm plugging before boot'
<harlowja> :)
<harlowja> so nova can boot before the VIF is plugged, and if the VIF isn't plugged it would make sense that during boot it can't talk to any metadata 
<harlowja> or anywhere else actually
<harmw> indeed, but in this case the instance does manage to contact nova
<harmw> its nova that in turn can't communicate with neutron
<harlowja> hmmm
<harmw> 2014-02-16 10:58:04.683 26729 DEBUG nova.network.neutronv2 [req-55651799-a048-479a-8366-5b9ba38042b6 None None] XXX failed: Connection to neutron failed: timed out _get_auth_token /usr/lib/python2.6/site-packages/nova/network/neutronv2/__init__.py:49
<harmw> 2014-02-16 10:58:04.685 26729 ERROR nova.network.neutronv2 [req-55651799-a048-479a-8366-5b9ba38042b6 None None] Neutron client authentication failed: Connection to neutron failed: timed out
<harmw> (the XXX is a personal debug line)
<harmw> 2014-02-16 10:58:04.687 26729 ERROR nova.api.metadata.handler [req-55651799-a048-479a-8366-5b9ba38042b6 None None] Failed to get metadata for instance id: 506b3116-bae2-4729-9cdd-6147e720ab1d
<harmw> 2014-02-16 10:58:04.687 26729 TRACE nova.api.metadata.handler ConnectionFailed: Connection to neutron failed: timed out
<harlowja> hmmmm
<harlowja> def weird
<harlowja> harmw u might try the #openstack-nova channel
<harmw> k, will do
<harlowja> *i mean
<harlowja> the #openstack-neutron channel
<harlowja> or both
<harlowja> :-P
<harmw> ah yea, thats exactly why I just love openstack
<harlowja> so many channels
<harlowja> lol
<harmw> indeed, indeed
<harlowja> harmw neutron is working otherwise right?
<harmw> yea yea
<harlowja> ya, weird then
<harmw> like i said, i can curl 169.xxx just fine afterwords
<harlowja> under heavy load?
<harmw> haha, definately not :p
<harlowja> odd :(
<harlowja> bb
<harmw> nova list just took 21s... I rly need new hw
<harmw> [!!] Joins performed without indexes: 172265
<harmw> thats just awfull (from mysqltuner.pl)
<smoser`> harlowja, did you find something ?
<smoser`> ont he above?
<harlowja> smoser??
<smoser> err.. harmw i meant.
<harlowja> k
<harmw> I only just askd in #openstack-neutron
<harmw> about mysql, I configured it to log not-indexe stuff
<harlowja> harmw i think part of the issue with nova list is that it just sucks
<harlowja> lol
<harlowja> don't try to list all the things
<harlowja> lol
<harmw> i only have 4 instances...
<harlowja> oh
<harlowja> lol
<harmw> its my homelab :)
<harlowja> i know ~1000 i gets bad
<harlowja> *it gets
<harlowja> *or did once
<harlowja> might be fixed now
<harlowja> https://bugs.launchpad.net/nova/+bug/1176446 ?
<harlowja> see last comment
<harmw> shm
<harmw> so according to https://bugs.launchpad.net/nova/+bug/1228384 that should be in havana 2013.2.2
<harlowja> seems like it
<harmw> ah, and was release last week or so
<harlowja> ah
<harlowja> there u go
<harmw> let's see if those are in the RDO repo
<harlowja> ah, RDO
<harmw> since feb 14
<harmw> nice
<harlowja> if u want newer repositories, my anvil project buidls rpms 
<harlowja> and can build the newer versions ;)
<harlowja> automagically, ll
<harlowja> *lol
<harmw> nice :)
<harlowja> someday i'll get there redhat people to use it
<harlowja> https://anvil.readthedocs.org/en/latest/topics/examples.html#building
<harmw> but for now Ill stick to rdo, and patch/build whatever I need myself 
<harlowja> k
<harlowja> np
<harmw> ah, I did patch build something myself, to patch https://bugs.launchpad.net/neutron/+bug/1251459
<harmw> should've done that before running the update though :p
#cloud-init 2014-02-27
<anotheral> question if anyone has time
<anotheral> regarding awscli and cloud-init on ubuntu 12.04
<anotheral> nvm - fixed it
<smoser> harlowja, did you point me once at some tool that would convert package names ?
<smoser> from debian -> redhat or some such?
<harlowja> hmmmm, smoser not sure :(
<smoser> python library i though.
<smoser> i remember that at some point someone pointed me at a library and said "maybe cloud-init could use this to normalize package names across distros"
<harlowja> smoser hmmm
<harlowja> smoser https://github.com/jordansissel/fpm ?
<harlowja> maybe that, not sure
<harlowja> i've made a similar tool, but its anvil ;)
<smoser> nah.
<harlowja> hmmm
<smoser> this isn't for *building*
<smoser> this was for:
<smoser> apt-get install libvirt-bin -> yum install libvirt
<smoser> or
<harlowja> oh
<harlowja> ya
<smoser> apt-get install openssh-server -> yum install sshd
<harlowja> https://bugs.launchpad.net/cloud-init/+bug/785570 ?
<harlowja> packagekit
<harlowja> maybe that
<smoser> yeah, i think thats what it was :-(
<smoser> but thats not what i want
<smoser> i want a big table mapping debian names to suse names
<harlowja> lol
<harlowja> not sure then :-P
<harlowja> smoser 14.04 soon right?
<harlowja> like next month/
<harlowja> ?
<harmw> smoser: you're ok with this patch? https://bugs.launchpad.net/cirros/+bug/1184826
<harmw> just in time for 0.3.2 :p
<smoser> harmw, nice. i almost can't believe i dont hvae that.
<smoser> the format is the only thing i'd bikeshed on. i'd like for it to be the same as cloud-init.
<harmw> hm hm
<harmw> sounds reasonable
<harmw> ill work something out tonight
<smoser> thanks.
<natorious> is there a way to schedule a job w/o starting the scheduler?
<natorious> wrong chan
<harmw> smoser: you're liking this? https://bugs.launchpad.net/cirros/+bug/1184826
<harlowja> seanwbruno for freebsd stuff would the first step be to upload the missing ports to freebsd ports?
<harlowja> thats probably like 3-4 python deps
<seanwbruno> yepper, I need to grab all the things and construct something that looks like a freebsd port makefile thing.
<harlowja> k, i'd save the cloud-init package for last, harmw the current code doesn't write out the network config right?
<seanwbruno> I wonder if fbsd ports knows how to interact with lp
<seanwbruno> hrm
<seanwbruno> let me poke
<smoser> harmw, that looks great.
<smoser> harmw, can you just verify that /usr/bin is not necessary ?
<smoser> i just don't lik ehardocding paths
<dgarstang3> How can I get cloud-init to umount a directory ?
<smoser> dgarstang3 you want it to not mount /mnt ?
<smoser> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/examples/cloud-config.txt#L178
<dgarstang3> smoser: /mnt is already mounted as part of the ubuntu amazon image. s'ok, I think adding [ umount, /mnt ] to bootcmd should do it...
<smoser> dgarstang3, its not already mounted as part of the imags.
<smoser> cloud-init mounts it
<smoser> per its default configuration
<smoser> if you tell it not to, it wont
<smoser> mounts:
<dgarstang3> smoser: that's the default? How do I override that default?
<smoser>  - [ephemeral0, null ]
<smoser> see the default listed at lin 174
<dgarstang3> smoser: i see. ok I shall add that. 
<dgarstang3> smoser: but hang on... i know that typically it uses the software raid... - [ephemeral0, null ] wouldn't cover that
<smoser> ?
<smoser> what typically uses software raid ?
<smoser> doing the above will tell cloud-init to not write an /etc/fstab entry for ephemeral0 (/dev/sdb)
<dgarstang3> smoser: there's more than one ephemeral disk
<dgarstang3> smoser: typically, something, takes all empehemeral disks and puts them into a dm raid volume and mounts on /mn
<dgarstang3> sorry, /mnt
<dgarstang3> It doesn't seem like saying - [ephemeral0, null ] would stop that, since there's more than one ephemeral disk involved
<harmw> smoser: nope, full path isn't required
<dgarstang3> I'll try - [ephemeral0, null ] - [ephemeral1, null ] - [ephemeral2, null ] - [ephemeral3, null ]
<harmw> harlowja: it doesn't? I realy dont know actually, it's been a while :p
<harlowja> np
<harmw> could fairly well be the case though, it missing tons of stuff anyway 
<harmw> btw, thanks for pointing me to that nova slow list bug :) it takese 3seconds now
<harlowja> woot
<harlowja> harmw on how many instances?
<harmw> haha, just 5
<harlowja> ya, hmmm
<harlowja> still to slow, lol
<harmw> yea i know, but thats more likely due to the crappy hw :)
<harlowja> some of it, lol
#cloud-init 2014-02-28
<harmw> smoser: 0.3.2 :)
<harmw> smoser: regarding r293 and r294, I can only use file injection or rebuilding cirros from source as a means to configuring the proper resizefs mode, right?
<harmw> or I could use cmd execution (in this case: sed or just echo) in userdata, though that would probably come in to late for resizefs to occur on the current boot
<smoser> harmw, fil injection or image modification for sure.
<smoser> userdata will happen too late.
<smoser> harmw, the plan would be for in 0.4 or something to have a newer kernel
<smoser> and newer krenel gets magic fast online resize
<smoser> so i'd enable it by default.
<harmw> ah ok
<harmw> fair enough
<harmw> perhaps it'd be nice to have 2 prebuild images online, one with and one without resizefs 
<harmw> since I'm to lazy to roll my own image :p
<smoser> i really dont want to complicate things. its really easy to patch an image before upload.
<smoser> well, actually for cirros its more difficult isnt it :)
<smoser> i'm open to suggestions on it.
<smoser> the newer kernel will magically make everythign wonderful.
<smoser> (i'm actually not kidding, its up to 100x faster)
<harmw> true, but I can imagine that'll take atleast a year :p
<smoser> for online resize.
<harmw> yea you've told that bfore, sounds promissing fo
<smoser> well, yeah. i know. i suck. and the other thing i want in 0.4.0 would be some udev like thing (maybe 'mdev') which is theonly way i can reasonably support running on microsoft hypervisor (name eludes me now)
<harmw> hyper-v
<harmw> yes, I noticed a bugreport on it 
<harmw> but let's just do a 0.3.2 first, ok :)
<harmw> would cirros benefit from a tool to show more environment stats? hypervisor, ram, cpu and the likes?
<harmw> basically extending cirros-status
<harlowja> smoser anything u need from me releated to bug https://bugs.launchpad.net/cloud-init/+bug/1136343 ?
<smoser> harlowja, nah. thats fix-released in cloud-init.
<smoser> i made it public today
<smoser> as juerg opened a dupe of it.
<smoser> i have no intent of SRUing that to 12.04
<harlowja> k
<harmw> smoser: any thought about my question earlier on?
<harlowja> harmw for the network settings writing question, http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/distros/freebsd.py#L220 
<harlowja> probably something simple to write, i can even do, but not sure what the freebsd network format is :_P
<harlowja> :-P 
<harmw> it's just some stuff going into /etc/rc.conf
<harmw> shouldn't be that hard, indeed
<smoser> harlowja, sure. extending cirros-status would be fine. and useful i think.
<harlowja> sounds good to me smoser 
<harlowja> lol
<harmw> lol
<smoser> bah. harmw ^
<smoser> harmw, yeah, definitely 0.3.2 is not including hyperv support.
<smoser> or a 3.13 kernel :)
<harmw> indee
<harmw> harlowja: I guess I can look into the _write_network stuff later tonight
<harlowja> sweet, if u want, i can try to figure it out also, but freebsd experts would probably do it quicker ;)
<harmw> haha
 * harmw is by no means an expert
<harlowja> more expert than me :-P
<harmw> harlowja: openstack doesn't put any network stuff in metadata, I have to use configdrive for that right?
<harlowja> correct
<harlowja> but u can just expect that its the format at http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/distros/net_util.py#L24
<harlowja> if it isn't provided, then just don't write
<harmw> I'm including that indeed
<harmw> TypeError: _read_hostname() takes at least 2 arguments (1 given)
<harmw> did I break that (freebsd, ofcourse)
<harlowja> did u do self._head_hostname?
<harlowja> self._read_hostname
<harmw> this is a new checkout from launchpad, I didn't touch thta part of the code
<harmw> lets dive in :)
<harmw> got it
<harmw> jeez
<harmw> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/distros/freebsd.py#L85
<harmw> the filename parameter is not supposed to be there
<harlowja> hmmm, that would do it :-P
<harmw> but sucks I didn't notice the error before
<harmw> AHA
<harmw> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/927
<harmw> smoser: you broke it :p
<harmw> harlowja: https://code.launchpad.net/~harmw/cloud-init/freebsd-static-networking
<harlowja> woot
<harmw> I havent tested that just yet though
<harmw> since I'm not using configdrive atm
<harlowja> harmw should be easy to add a unittest that can verify what was written
<harlowja> that 'simulates' input and the right output
<harmw> yea 
<harmw> could you write one? or should I do that?
<harlowja> if only i know what the expected output should be ;)
<harmw> (I dont know how laborintensive such a thing is)
<harlowja> http://bazaar.launchpad.net/~harmw/cloud-init/freebsd-static-networking/view/head:/tests/unittests/test_distros/test_netconfig.py can probably be extended
<harmw> ah ok
<harlowja> def test_simple_write_ub(self): -> def test_simple_write_bsd(self):
<harmw> Ill look into that, thanks :)
<harlowja> np
<harlowja> that'd be sweet, seanwbruno if u wouldn't mind looking that over when it arrives
<harlowja> since u probably can verify all that stuffs :)
<smoser> harlowja, what'd i break?
<harmw> you introduced a parameter to a function, causing it to break once called (freebsd.py) :)
<harmw> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/distros/freebsd.py#L85
<smoser> hm..
<smoser> fix?
<harmw> just undo the change on that line
<harmw> (dropping the filename argument)
<smoser> pylint cries
<harmw> ooooooh
<smoser> for not a good reason.
<smoser> its odd. but since it is in the parent class and even declared an abstractmethod there
<smoser> we shoudl have the same
<smoser> so maybe this: http://paste.ubuntu.com/7012683/
<harmw> hm yea, I was thinking the same
<smoser> harmw, can you just verify that that fixses it really quick
<smoser> and then i'll commmit
<smoser> sorry for breaking
<harmw> sure, gimme a sec
<harmw> smoser: works :)
<smoser> pushed
<harmw> harlowja: I've extended that netconfig test a little, how should I proceed?
<harlowja> harmw commit it and there u go ;)
<harlowja> ?
<harmw> oh, hm
<harmw> I was expecting some of just running that test here, local
<harmw> harlowja: its commited
<harmw> I've pushed some other fixes in freebsd.py as wll
<harlowja> ah, u should just be able to run nosetests and have it locally run
#cloud-init 2014-03-01
<harmw> harlowja_away: tests ran fine, only complains about resolv.conf stuff
<harmw> so after thats worked out ill submit a merge request
#cloud-init 2015-02-23
<smoser> jseun, i'd like to have better stuff available there too.
<smoser> shameless:
<smoser> plug: 
<smoser>  https://www.openstack.org/vote-vancouver/presentation/going-beyond-intel-openstack-on-other-architectures
<smoser>  https://www.openstack.org/vote-vancouver/presentation/title-evil-superusers-howto-launching-instances-to-do-your-bidding
<smoser>  https://www.openstack.org/vote-vancouver/presentation/system-containers-have-your-cake-and-eat-it-too
<Odd_Bloke> harlowja_at_home: I've updated that merge proposal, by the way.
<smoser> the second has the great harlowja_at_home as a speaker, so you definitely "Would Love to SEE!" that.
<harlowja_at_home> lol
<Odd_Bloke> harlowja_at_home: It turns out the response has already been fixed upstream.
<harlowja_at_home> woot
<Odd_Bloke> harlowja_at_home: Make sure you're sitting down before opening this link; this is how it's done: https://github.com/apache/cloudstack/blob/master/systemvm/patches/debian/config/opt/cloud/bin/serve_password.sh
<harlowja_at_home> lol,
<harlowja_at_home> sad
<harlowja_at_home> bash http server scares me 
<harlowja_at_home> lol
<harmw> smoser: multiarch sounds like a nice talk :)
<smoser> yeah. i'td probably be forcused more on ppc64 as that s what we ewre recently playing with
<smoser> but would like to try with arm stuff also
<harmw> :)
<harmw> smoser: did you release a new c-i version lately btw?
<smoser> no.
<smoser> still working out python3 bugs
<smoser> https://bugs.launchpad.net/cloud-init/+bug/1423972
<smoser> a bunch of those ones.
<harmw> hmk
#cloud-init 2015-02-24
<harlowja> Odd_Bloke https://github.com/apache/cloudstack/commit/a72f14ea9cb832faaac946 still scares me, haha
<harlowja> that whole code scares me, lol
 * harlowja thought cloudstack was in java, doesn't java have a built-in http server :-P
<harlowja> aka, http://docs.oracle.com/javase/6/docs/jre/api/net/httpserver/spec/com/sun/net/httpserver/HttpServer.html or one of the other ones...
<Odd_Bloke> smoser: harlowja_away has (I believe) signed off on https://code.launchpad.net/~daniel-thewatkins/cloud-init/cloudstack-passwords/+merge/250028; could I get a merge?
<smoser> Odd_Bloke, from the bug..
<smoser> "One major issue we need to decide whether or not we want to integrate this in to the main CloudStack data source (as in my patch), or if it's divergent enough to merit its own (derivative) data source. Direction on this would be much appreciated.
<smoser> "
<smoser> why would we not want it in main datasource?
<smoser> merged
<Odd_Bloke> smoser: That was before I realised that this was deployed as part of CloudStack, I think.  Thanks!
<smoser> ok, thanks. and good work.
<smoser> httpretty doesn't seem to like cloud-init test suite much.
<smoser> harlowja, if you're bored https://bugs.launchpad.net/juju-core/+bug/1424900
<harlowja> hmmm
<harlowja> 'bytes -> text -> bytes and lose data in the process.'
<harlowja> fun fun, lol
<smoser> i hate python
<smoser> thats what i think is happening, but might be wrong. 
<smoser> we're close to having some tests, i tihkn the tests push the code through the code path properly , but just do not actually validate it properly.
<smoser> anyway, if you want to take a stab :)
<smoser> i'm out, though. have a good one.
<harlowja> kk
<tennis_> Is there any reason to run cloud-init on vagrant/virtualbox  ? 
<harlowja> tennis_ because u want to?
<harlowja> lol
<harlowja> smoser btw, i'll be out tommorow -> sun; skiinggggggg
<tennis_> well, since the concept of meta-data doesn't really exist on vagrant/virtualbox, then it doesn't seem very useful.
<harlowja> then i guess don't run it :-p
#cloud-init 2015-02-25
<mso> hi all, can someone point me into the right direction ?
<mso> i have a datasoruce "nocloud" that i can (re)run when the machine is up, however, it does not work during boot
<mso> the nocloud seed is configured in the cloud.cfg
<mso> i am actualy totally lost where to go now, the environment is running ovirt and i am trying to deploy machines through foreman
<kol> Hello , where do I make changes in cloud-init  ? I see a cloud-config.txt file , but it is always empty even when i run cloud-init , i'm not able to figure it out :/ 
<kol> or is it like it like i have to edit in etc / cloud . cloud.cfg ?
<smoser> kol, you can put config files in /etc/cloud/cloud.cfg.d or edit /etc/cloud/cloud.cfg.
<smoser> /var/lib/cloud/instance/cloud-config.txt is written by what is read in from user-data.
<smoser> if you've not passed any '#cloud-config' in user-datta, then it will be empty.
<Odd_Bloke> smoser: LC_ALL was set to C because that's what it is set to in some clouds (e.g. Azure), and it changes Python's encoding/decoding behaviour.
<smoser> hm..
<Odd_Bloke> smoser: So I think with your change, the test in http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/1059.1.2 would pass without my fix.
<Odd_Bloke> s/the test/the test I added/
<smoser> why does encode/decode change?
<smoser> is it just that encode() changes ? with no 'utf-8' arg ?
<Odd_Bloke> Because Python determines the default encoding it should use from the environment.
<Odd_Bloke> (If you don't specify one)
<smoser> right. so we should always be specifying one.
<smoser> there is lots of slop right now in that respect.
<Odd_Bloke> smoser: Agreed; setting LC_ALL=C will help catch places we're assuming that encoding will be UTF-8 when it may actually end up as something else.
<smoser> Odd_Bloke, yes, it will . and apparently will also catch places in other code (httpretty)
<smoser> which idont want to catch :)
<Odd_Bloke> smoser: :)
<Odd_Bloke> smoser: Can we pin to an older version of HTTPretty in the meantime, or do we need some of the more recent behaviour?
<smoser> 0.8.0 would be fine.
<smoser> and we could pin to that.
<smoser> (that is what is in vivid)
<smoser> at some point in the future, it might be useful to have tox environments that describe versions of dependencies on a particular release
<smoser> so i could:
<smoser> tox -e trusty-34
<smoser> er.. trusty-py34
<smoser> hey, strikov 
<smoser> so yeah, looking at bug 1424900
<strikov> hey
<smoser> the issue is that python2 made all this magic.
<smoser> and we didn't have to think about what might be in content of a read() (whether that be http or file)
<smoser> all "just worked"
<smoser> but in python3 you have to be much more careful
<strikov> correct, you need to know what you read
<strikov> because when you read ascii but think that you read utf -- bad things happen
<smoser> and the python3 port of cloud-init tried to keep us from having to know such things.
<smoser> but unfortunately that just isntt going to work
<strikov> well, it treated everything as utf
<strikov> basically we don't want t have binary -> string -> binary convertion at all for userdata
<strikov> can we read it to binary objject somehow?
<strikov> because to me it looks like python-requests converting userdata to utf-8 string
<smoser> i think we need to be able to
<strikov> when you call it from readurl
<smoser> python-requests definitely does support reading binary data
<smoser> as i use that for cloud-init
<strikov> maybe requests can handle that for us?
<smoser> but i think the way we're using it doesnt
<smoser> er... above, "as i use that for simplestreams"
<strikov> yeah, i got it
<strikov> OR we may generate correct utf-8 bitstream on the server
<strikov> maybe that's the better way
<strikov> because right now (I assume) server generates ascii bitstream
<strikov> i.e. raw bytes
<strikov> hm, i thought a bit and server-based way seems wrong
<strikov> smoser: I wonder why userdata content is marked as text/html on metadata server
<strikov> smoser: Not sure, but marking differently may solve the issue
<smoser> well, we need to treat it as bytes.
<smoser> basically, we need to treat as bytes, and try to uncompress, if that doesnt work, then try to encode as utf-8
<strikov> smoser: how about that; read_url returns bytes all the time; we have default translator (http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/sources/helpers/openstack.py#L256) which convert data to utf-8 string; userdata is the only field which doesn't use this translator
<strikov> smoser: url_helper.py has UrlResponse object; right now it has only contents() method which refers to _response.text of python-requests; we may add raw() method which would refer to _reponse.raw which is a file object
<smoser> strikov, right
<smoser> strikov, i was investigating that
<smoser> http://paste.ubuntu.com/10409455/
<smoser> that is where i was right now
<smoser> basically doing raw.read() rather than .text
<strikov> smoser: awesome!
#cloud-init 2015-02-26
<arksi> does anybody have suggestions on how to detatch and run in background a command through runcmd?
<harmw> nohup or cmd &
<smoser> arksi, is cloud-init waiting if you use & ?
<smoser> runcmd:
<smoser>  [ sh, -c, 'mycommand >/var/log/mycommand.log 2>&1 </dev/null &' ]
<smoser> something to that effect
<arksi> thanks
<arksi> I will find out if it waits
<arksi> I was trying
<arksi> [ setsid, program, & ]
<smoser> arksi, well that will probably just fail
<arksi> It does ;)
<smoser> as it will try to execute 'setsid' with the arguments: 'program' and '&'
<arksi> I will try your recommendation and see how it goes
<arksi> right
<smoser> if you just quote it, it will pass it to shell
<arksi> I didn't know we could use hardquotes like that
<arksi> right
<arksi> makes sense
<arksi> good idea
<smoser> ie, if the entry is a string it passes to shell
<arksi> I will let you know how it works
<arksi> right
<arksi> [ /bin/sh, -c , ... ]
<arksi> is clever
<arksi> thanks a lot
<smoser> just to be cleare, above:
<smoser> runcmd:
<arksi> Right
<smoser>  - "setsid program &"
<smoser>  - "another program"
<arksi> Does runcmd spawn a subshell?
<smoser> will do what you want. invoking sh -c 'setsid program'
<arksi> spawn subshell and exec?
<arksi> right
<smoser> if its a string, it goes to shell
<smoser> if its an array, no shell
<arksi> ah
<arksi> so it is equivalent to invoking system() if it is an array?
<smoser> generally speaking i hate passing things to shell
<smoser> :)
<arksi> same heh
<smoser> i would have though tsystem would go to a shell
<arksi> you are right
<arksi> system() invokes the shell and executes as a separate process
<arksi> exec() family replaces the current process with the program being exec'd
<arksi> system() = fork(), execve() and wait()
<arksi> I have to run a stupid program that checks for runlevel heh
<arksi> while cloud-init is running (on CentOS 7, at least), if you invoke runlevel command it reports 'unknown'
<arksi> (why anybody in this century is still checking runlevels is a mystery to me...)
<arksi> so I cannot invoke this program directly through cloud-init, hence I am trying to get it to detatch from init process and run in the background after waiting for runlevel 3
<arksi> horrible stuff :)
#cloud-init 2015-02-27
<faizal> Hi
<faizal> Any one online
<faizal> Need a help
<strikov> faizal: Just ask. You'll get the answer (sooner or later) if someone knows it :)
<faizal> Ok
<faizal> We are using cloud init to create users in all our servers. .I have written the cloud config script and uploaded it to s3. ..then I wrote a wrapper shell script which will download the cloud config script from the s3 and will run the command "cloudinit init "
<faizal> Our plan is to put this wrapper script in cron job and run every 30 min
<faizal> So if we want to add a user we will just edit the cloud config script and upload to s3
<faizal> The problem I am facing is
<faizal> When I run the script manually it is working fine
<faizal> But from cron job it is showing some error
<Odd_Bloke> faizal: Can you pastebin the error somewhere?
<faizal> Where can I paste
<Odd_Bloke> faizal: http://paste.ubuntu.com/
<Odd_Bloke> faizal: (It's worth noting that cloud-init is designed to run when initialising a server, it's not really intended for this use case.)
<Odd_Bloke> But I'm happy to see if I can unblock you.
<faizal> Pasted
<Odd_Bloke> faizal: You need to give us the link. :)
<faizal> 10449492
<faizal> Can you see
<Odd_Bloke> faizal: I can; in future just put the whole URL in the channel. :)
<Odd_Bloke> faizal: And could you also paste your configuration?
<faizal> Ok
<faizal> http://paste.ubuntu.com/10449576/
<Odd_Bloke> faizal: Which user was failing to create?
<faizal> When i run the command manually the user is getting created
<faizal> But from crontab it Is showing error
<faizal> It is not creating any user when running from cronjob
<Odd_Bloke> faizal: Sorry, let me rephrase: all of the users in that configuration have already been created, so I'm not sure what in the configuration is causing failure.
<faizal> I added a new user named "newuser" in the cloud config and ran the script from cronjob
<faizal> That newuser is not getting created when running from cronjob but is getting created when running manually
<Odd_Bloke> faizal: Oh, you know what, this is probably a PATH issue.
<Odd_Bloke> faizal: You'll have useradd in your PATH when you run it manually, but PATH won't be set in cron.
<faizal> ohh
<faizal> So how to fix
<faizal> Any clue
<Odd_Bloke> faizal: Set PATH in your crontab.
<faizal> Can you tell me how. ..I  am just a beginner in Linux
<faizal> Got it
<faizal> Thanks odd_bloke
<faizal> Its working fine now
<Odd_Bloke> faizal: :)
<faizal> You saved my weekend
<Odd_Bloke> faizal: You might want to look at configuration management software rather than cloud-init for this sort of ongoing usecase.
<Odd_Bloke> faizal: But I'm glad I could help. :)
<faizal> Yeah
<larsks> Odd_Bloke++ :)
<faizal> This was a task from my manger and I am a new joinee
<faizal> You saved me
<faizal> :-)
<smoser> well, as faizal has quit, it doens't help so much, but with cloud-init 2.0 we do want to have some mechanism for accomplishing what he wants.
<Odd_Bloke> smoser: Presumably we'd want to be able to do it against a cloud's metadata source.
<Odd_Bloke> (Rather than an arbitrary source, like faizal was using)
<smoser> one way or another, information in some format would get to cloud-init and it would act on it.
<smoser> i odnt think s3 is terrible, its a datastore, and can be addressed over https also.
<smoser> so : cloud-init set-config https://foo.bar/wark
<smoser> doesn't seem so bad.
<smoser> but it is treating cloud-init like puppet in a lot of ways.
<smoser> and i'm not so sure how far down this hole we'll go
<Odd_Bloke> Yeah, that would be my concern.
<harmw> my god, the Horizon spice console is looking horrible
<harmw> :p
<harmw> smoser: was cirros building on 14.04, or still at 12?
<smoser> i think 0.3 builds on 12.04
<harmw> hm hm
<harmw> ok
<smoser> but i think i mvoed it to 14.04 in a 0.4 branch
<harmw> I want to setup a new builvm, to get back to playing with it a littl
<harmw> and as much as I'd like to play in docker, I'm going the qemu-way first :p
<smoser> i think i'd go from lp:~smoser/cirros/buildroot-upgrade
<harmw> ah yeas, I was looking for that
<harmw> and that builds on 14?
<harmw> smoser: wasn't there some magic just-build-the-damn-thing script?
<harmw> (or did I create that myself, once)
<smoser> looking
<smoser> http://bazaar.launchpad.net/~smoser/cirros/buildroot-upgrade/view/head:/doc/create-release.txt
<harmw> looking that,didn't you make  build-release or something?
<harmw> oh
<harmw> stupid 
<harmw> thats actually being refrenced from inside that .txt
<harmw> VER=0.3.3 && ./bin/build-release $VER 2>&1 | tee build-$VER.log
<harmw> ok
<smoser> right.
<harmw> hm, what's this... the vm is suffering from bitrot, so it seems
<smoser> the vm ?
<harmw> instance
<smoser> you just launch new ones all the time :)
<harmw> no no, I have one ubuntu 1404 instance :)
<harmw> and it's corrupting my downloads
<harmw> hm, mtu 1450 seems to go better (instead of 1400)
<harmw> smoser: you have access to some PPC64 hardware right?
<harmw> thats IBM PowerPC, right?
<smoser> i do, yeah.
<smoser> https://code.launchpad.net/~smoser/cirros/powerpc at one point was close.
<harmw> we have Power6 and 7 at work
<harmw> would it be conventient to have access to either one of those archs to build, say, cirros?
<smoser> well, cirros is cross-building.
<smoser> so it would build on x86_64 is fine.
<smoser> i think that both of those arches should support 'ppc64 kernel'
<smoser> at somepoint i'd like to have a ppc64el one also, and that would only run on power8
<harmw> ok
<harmw> but those are all crosscompiled
<smoser> right. buildroot is always crosscompile
<smoser> so from a build perspective, powerX are not terribly useful
<smoser> but for testing they are
<harmw> yup
<smoser> i got to run. later man. have fun.
<harmw> :)
#cloud-init 2015-03-01
<Faizal> Can we run just one module for ex:- users-group module when we run the cloud init manually?
<Faizal> When we run "cloudinit init"  it will execute all the modules... Right...?
<Faizal> What if I want to run just users-group module
<Faizal> Pls help
<izalmul> Anyone here
<harmw> always nice when ppl drop questions and leave...
<kwadronaut> didn't we have irc logs somewhere? 
<kwadronaut> no need to stick around, just grab the rss of the public logs ;-)
<andreyst> Hi all, cloud-init doesn't seem to execute user-data for some reason
<andreyst> I see that it reads it and puts in userdata.txt
<andreyst> But I see no script output (it writes to /tmp/test.log), when I manually execute bash user-data.txt it works ok
<andreyst> What could be the reason?
<andreyst> Do I have to explicitly specify the need to execute user data in config somehow?
<larsks> andreyst: what does your user-data look like?
<andreyst> just a simple bash script
<andreyst> @larsks #!/bin/bash then some echos
<larsks> Hmm.  And looking in /var/log/cloud-init.log (or cloud-init-output.log) are there any errors (or any indication that it executed)?
<andreyst> larsks: I see its contents in /var/lib/cloud/instance, in user-data.txt and scripts/part-001
<andreyst> larsks: I see it is getting read, I see no errors and no indication that it was executed whatsoever
<andreyst> also I see on the internet some reference to /etc/init.d/cloud-init-user-scripts, I don't have this file for some reason
<larsks> Anyway, to answer one of your earlier questions, you shouldn't need to perform any explicit configuration to get it to execute.
<andreyst> larsks: yeah, that's how it worked for me earlier
<andreyst> thought something might have changed since then
<larsks> What distribution are you booting?
<andreyst> ubuntu@ip-10-222-222-127:/var/lib/cloud/instance$ uname -a
<andreyst> Linux ip-10-222-222-127 3.13.0-44-generic #73-Ubuntu SMP Tue Dec 16 00:22:43 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
<andreyst> it is AWS provided ubuntu server ami
<larsks> Do you have the AMI ID handy?  I can try booting it myself and see what happens.
<andreyst> yeah, sec
<andreyst> ami-234ecc54
<andreyst> it's in eu-west-1
<larsks> Okay. Let me see about getting that booted...
<andreyst> this is my userdata: http://pastie.org/9992180
<andreyst> oh man, this dumb version actually works. Guess it is something in the script after all! 
<larsks> Yay!
<andreyst> Thanks a ton for your help, man!
<larsks> Eh, I did nothin'.  Glad it's working, though.
<andreyst> you listened
<andreyst> you know that's almost always a solution :)
<larsks> Yup! :)
#cloud-init 2016-02-29
<esierrap> hi, is it possible create new partitions with cloud-init? The only possibilty is to use runcmd?
#cloud-init 2016-03-01
<smoser> harlowja_at_home, around ?
<smoser> http://paste.ubuntu.com/15248500/
<smoser> gudge
<smoser> fudge
<harlowja_at_home> extra fudge please
<harlowja_at_home> markerlib
<harlowja_at_home> wtf is that
<harlowja_at_home> ha
<smoser> harlowja_at_home, its fixed. python-virtualenv upgrade fixed it
<smoser> it came in a couple hours before i lost a couple hours
<harlowja_at_home> u lost a couple of hours in space
<harlowja_at_home> lol
<esierrap> hi
<esierrap> why my script in /var/lib/cloud/scripts/per-once doesnt run at first boot?
<esierrap> i have the module in /etc/cloud/cloud.cfg
<smatzek> esierrap.  per-once runs once ever.  It won't run on first boots if you snapshot/boot snapshot.
<smatzek> you might want per-instance instead
<smoser> thanks smatzek
<smoser> rharper, thoughts ?
<smoser> https://code.launchpad.net/~smoser/cloud-init/trunk.disable-clouddinit/+merge/287580
<smoser> the thing i'm not sure of is if i like the single 'touch /etc/cloud/cloud-init.disabled'
<smoser> rather than
<smoser> echo "disabled" > /etc/cloud/cloud-init.somethinghere
<smoser> simplicity is nice, but the 'disabled' string as configuration option means that i could also have
<smoser>  'no-modify-net'
<smoser> or something to that effect
<smoser> but then that quickly goes into the realm of cloud.cfg
<rharper> smoser: we can certainly do the latter overtime;  unless you have more than a few in mind I'd prefer the simpler one;  one simple switch
<smoser> rharper, well, the reason for the string.
<smoser> is that i can then have the contents of that file have the same as cloud-init=<value> on the command line.
<smoser> and then on that command line, i can say something liike: cloud-init=no-dynamic-net
<rharper> we can do both, right ?
<smoser> i think it just gets confusing then.
<smoser> you can do this or that or the other thing.
<smoser> pick the right one
<rharper> assuming the presence of the disabled file would negate the other options entirely; the file is like a big simple switch
<smoser> right. but it sa boolean.
<smoser> and i have  already a desire for a 3 way switch.
<smoser> on, off, static-networking
<rharper> sure; I'm saying we can do both; I don't think it's confusing;  one big on/off, and if it's on; then you can also do other stuff;
<rharper> or delay the use of the content control
<rharper> if I see disabled file, then we don't run;  that's a patch and will keep working;  we can introduce the content switch (on,off,no-net) at a later time
<rharper> and it still works with our without the .disabled switch
<rharper> smoser: I;m find with the n-way switch; if we want to introduce the big off hammer later, we can also do that as well
<rharper> or not at all
#cloud-init 2016-03-02
<rharper> smoser: would you expect trunk cloud-init to run inside trusty as-is ?
<smoser> rharper,  yeah, i'd think so
<smoser> dont remember though.
<rharper> smoser: ok, it doesn't; that does make injecting a deb of cloud-init into the various target releases somewhat problematic;
<smoser> oh. trusty needs python2
<smoser> right ?
<smoser> is that it.
<smoser> try
<smoser> ./packages/bddeb --python2 -v -S
<smoser> i'm trying that
<rharper> ah, that makes sense
<smoser> well, it does install
<rharper> ph
<rharper> it installs
<smoser> but needs python-jinja2 and python-oauthlib the second of which might not be necessary
<smoser> as i think it tries one and then the other
<smoser> and jinja2 also
<rharper> it errors out running
<smoser> we can probably make it build wihtout those deps
<rharper> /bin/sh: 1: exec: cloud-init: not found^M
<rharper> I'm injecting the deb; dpkg --install; apt-get -f install ; that picks up package deps
<smoser> seems to work for me
<smoser> (with --python2)
<rharper> right
<smoser> i have to run
<rharper> and built inside trusty sbuild
<rharper> ok
<smoser> right.
<rharper> Keeping session: trusty-amd64-0c52d381-b0b8-48af-a299-5ed976f72b21
<rharper> E: cloud-init_0.7.7~bzr1163-1.dsc: amd64 not in arch list or does not match any arch wildcards: all -- skipping
 * rharper sbuilds harder
<Odd_Bloke> smoser: So I now have a few different things that are pending merges in to trunk; I've been holding off on that because of overheard conversations about switching to git, but that hasn't happened yet (AFAIK).
<Odd_Bloke> smoser: Is trunk still frozen for the move, or can we continue using it?
<smoser> Odd_Bloke, i can't right now be delayed by moving to git.
<smoser> i'd like to be on git. and i'd welcome you or someone setting up a git branch that synced trunk
<smoser> on launchpad i think.
<Odd_Bloke> smoser: I'd like that too, but also don't have the time.
<Odd_Bloke> smoser: So can we safely merge things in to lp:cloud-init for the time being?
<smoser> yes
<Odd_Bloke> OK, cool.
<smoser> and i'm ok for you to do that.
<smoser> do the merges yourself, that is, and push.
<Odd_Bloke> smoser: Ack, thanks.
<smoser> some things:
<Odd_Bloke> smoser: Also, I'm going to have a patch for UUID endianness for trusty soonish.
<Odd_Bloke> (Luckily, it doesn't affect wily onwards. \o/)
<smoser>  a.) tox must work
<smoser>  b.) do not add dependencies
<smoser>  c.) commit message in 'git style' format
<smoser>     short description, blank line, longer description
<smoser>  d.) update ChangeLog with roughly short description, and please give people credit there
<smoser>  e.) when a bug is fixed, please --fixes lp:XXXXX on the commit
<smoser> Odd_Bloke, so its cloud-init's job to flip endieanness of a uuid ?
<Odd_Bloke> smoser: The problem is that the kernel with the bug is in the wild.
<smoser> i think basically we just have to test both possible endianness
<Odd_Bloke>     if current_instance_uuid == reported_instance_uuid:
<Odd_Bloke>         # The endianness of the instance UUID has not changed
<Odd_Bloke>         return reported_instance_uuid
<Odd_Bloke> NAILED IT
<Odd_Bloke>     if current_instance_uuid == reported_instance_uuid:
<Odd_Bloke>         # The endianness of the instance UUID has not changed
<Odd_Bloke>         return reported_instance_uuid
<Odd_Bloke> Oh FFS.
<Odd_Bloke> How do clipboards in X even work?
<Odd_Bloke> smoser: http://paste.ubuntu.com/15266719/ is what I have currently.
<Odd_Bloke> Untested as yet.
<smoser> if current_instance_id in (big_endian(value), little_endian(value)):
<smoser>  good
<smoser> can you put such a function in util ?
<smoser> i'm expecting that very soon i'll be having the same thing on openstack
<Odd_Bloke> smoser: So ATM it only affects Azure on trusty and maybe precise (I haven't checked there yet, focussed on getting trusty fixed first).
<smoser> http://paste.ubuntu.com/15266739/
<Odd_Bloke> smoser: So this was all going to be in an Ubuntu patch.
<smoser> interestingly. azure uses system-uuid
<smoser> openstack uses product-uuid
<smoser> would both be vulnerable? or is product-uuid safe due to its different location?
<Odd_Bloke> smoser: Where does OpenStack use product-uuid?
<smoser> see thatpaste.
<smoser> and then followed paste is http://paste.ubuntu.com/15259269/
<Odd_Bloke> smoser: I think the OpenStack data source uses the metadata API to get the instance ID.
<Odd_Bloke> smoser: It may be that that's also presented via DMI, but I don't think it's used.
<Odd_Bloke> (And so it should be fine)
<Odd_Bloke> Anyway, lunch is ready, I'll BBIAB.
<smoser> Odd_Bloke, read the message :)
<smoser> i'm looking to change that. or at  least to not re-hit the MD (which might be there) if the dmi data says that i am the same insatnce i was last time.
<Odd_Bloke> smoser: Oh, I didn't understand it was a proposed change. :)
<Odd_Bloke> smoser: Oh, hmph, we remove the /var/lib/cloud/instance link before we determine instance ID.
<smoser>  Odd_Bloke see /var/lib/cloud/data/previous-instance-id
<smoser> and honestly, your check here should really run before we remove that link
<smoser> because what you're doing is fairly definitive
<smoser> and we only remove that link so that we go searching again, which we wouldnt need to do.
<smoser> make sense ?
<Odd_Bloke> smoser: Not quite, no.  Looking at bin/cloud-init, the bits under '# Stage 4' don't do anything but clean things up if we aren't running with --local...
<Odd_Bloke> (Perhaps I don't understand enough about when we run cloud-init at boot, and how)
<smoser> right. you are adding somethign that is smarter than that.
<smoser> so basically first thign that happens is cloud-init --local runs.
<smoser> and it removes /var/lib/cloud/instance/
<smoser> if it had 'manual_cache_clean' then it would nto remove that link
<smoser> and it would not go looking for a datasource again
<smoser> it'd just re-use the cached data.
<smoser> which is what you want it to do.
<smoser> so
<smoser>  if manual_cache_clean:
<smoser>     do not remove
<smoser>  else:
<smoser>    if my_quick_check_for_instance_id() == the_previous_instance_id:
<smoser>    do not remove
<smoser>  else
<smoser>     remove link
<Odd_Bloke> But my quick check for instance ID is Azure-specific, other things might will need to hit metadata APIs etc.
<Odd_Bloke> And we haven't yet determined a data source at this point.
<Odd_Bloke> I'm not saying that we shouldn't do that, I just don't think I can do it in the scope of a fix for a critical bug (that only affects old versions of cloud-init). :)
<smoser> 2 othe rthings that allow me to support you making such a possibly invasive change
<smoser> a.) /var/lib/cloud/data/previous-datasource is there. meaning you could look and noly consider this if the previous datasource was azure
<smoser>   i dont think this is necessarily required, though, given
<smoser> b.) its a UUID ! this thing should never be seen *anywhere* randomly twice
<smoser> the failure path i guess is if you are a new instance on hardware, where that value is actually the node's uuid.
<smoser> and you happen to have ran on this system when someone captured a new image.
<Odd_Bloke> smoser: Or if the cloud sets the same value for all instances.
<smoser>  and it happened to hit the same instance id that your "proper" check found ?
<smoser> if a cloud has all their instance ids with the same instance-id, then yes. we are foobarred.
<smoser> i think we can strongly suggest people not do that.
<Odd_Bloke> :D
<rharper> smoser: I think the lxd check isn't quite right;  IIUC, if one does not specify a lxd: dict in cloud-config, then it shouldn't attempt to install lxd;  we do a lxd_cfg = cfg.get('lxd'); which if yaml doesn't include a lxd section returns None;  the next check if not lxd_cfg and isinstance(lxd_cfg, dict);  is True and False for lxd_cfg of NoneType;  this means we fall through and attempt to install lxd;
<smoser> you're probably right.
<rharper> adding a not isistance(, dict) I think achieves what is wanted; assuming that if someone adds just lxd section (empty dict); that will install lxd ..
<rharper> cool, with a fix in that logic;  I don't get lxd attempting to install in trusty (and thus breaking since it needs to be installed via backports);  I;ll file a bug, and send you a MR
<rharper> there's also another lurking error in one of the error paths; pyflakes cloudinit/config/cc_lxd.py will show it
<rharper> smoser: I packaged a python3 version of cloud-init built inside a xenial sbuild;  it installs to /usr/lib/python3.5/dist-packages ;  the python3 interpreter on vivid doesn't know to look there;  contrast with curtin, which uses /usr/lib/python3/dist-packages which works for any python3.x IIUC;  shoudl we do the same when packaging cloud-init ?  or do we need to build a new cloud-init deb for each release?  that does mak
<rharper> e testing changes to cloud-init across all release a longer process since it requries N rebuilds
<smoser> hm... the packaging doesn't use /usr/lib/python3.5 at all. right ?
<smoser> it dh_python that does that.
<rharper> oh
<rharper> hrm, maybe the tools/build-deb in curtin does something different
<rharper> I built the curtin package on my xenial host, so I don't think an sbuild of curtin would change the python path
<rharper> build curtin inside xenial sbuild; that is
<smoser> well i'm certain that i have never typed that path
<rharper> I agree
<rharper> now to look at the difference
<smoser> tools/build-deb does with dh_python too (curtin)
<rharper> maybe newer debhelper ?
<rharper> 7 vs 9 ?
<smoser> well thats just declaring the minimum
<smoser> it uses what isthere.
<smoser> rharper, i know what it is.
<smoser> see in curtin/trunk/debian/rules
<smoser> we build for python in $(PY3VERS)
<smoser> it has somethignto do with that.
<rharper> smoser: yeah, I was looking at the rules in curtin which specifies 2 and 3; looking to see if I can build with those changes
<suro-patz> hello folks!
<suro-patz> cloud-init 0.7.x on rhel7 is failing to execute due to dependency on urllib
<suro-patz> am I missing some requirement?
<suro-patz> https://github.com/openstack/cloud-init/blame/0.7.x/cloudinit/util.py#L49
<suro-patz> harlowja_at_home: ^^
<harlowja_at_home> suro-patz, sup
<harlowja_at_home> is six installed?
<harlowja_at_home> or a new enough six?
<harlowja_at_home> suro-patz, i'd check the version of six
<harlowja_at_home> causeit works for me
<suro-patz> harlowja_at_home: yes, that's where the issue was
<suro-patz> the rhel7 image i was using had six-1.3 :(
<harlowja_at_home> ya, sounds old
<harlowja_at_home> to old
<suro-patz> Do you think it makes sense to put a lower bound in cloud-init requirements
<suro-patz> harlowja_at_home: ^^ ?
<harlowja_at_home> nope
<harlowja_at_home> oh
<harlowja_at_home> yes i mean
<suro-patz> ok, I will go ahead and submit a request
<harlowja_at_home> suro-patz,  https://github.com/openstack/cloud-init/blob/master/requirements.txt#L8
<suro-patz> this days is it on github/gerrit or ol bazar stuff?
<harlowja_at_home> oh nm, the 0.7.x branch not have that
<suro-patz> oh cool
<harlowja_at_home> smoser, where 0.7.x stuff happening now-adays
<harlowja_at_home> :-P
<suro-patz> yes, 0.7.x does not have
<harlowja_at_home> i think smoser thought launchpad for 0.7.x and git for 2.0 stuff
<harlowja_at_home> he'll know best
<suro-patz> okie dokie
<suro-patz> harlowja_at_home: thanks !
<harlowja_at_home> np
<rharper> smoser: I was unsuccessful in teaching cloud-init to do both python2 and 3 ; without more study at least.  Currently I'm just using the python2 version of the build any place that doesn't have python3.5 (which I belive is only xenial)
#cloud-init 2016-03-03
<smoser> harlowja_at_home, still in bzr. will be for the next month or so at leas.t
<harlowja_at_home> k
<smoser> just ahve too many othe rthings at the moment to get there.
<smoser> an deven to bothe rtyping well
<harlowja_at_home> lol
<rharper> smoser: when did you want to chat about cloud-init networking bits?
<suro-patz> does tox tests for cloud-init(0.7.x) succeeds for py27?
<suro-patz> smoser: harlowja_at_home: ^^
<suro-patz> I am getting ImportError
<suro-patz> (py27) [suro@oxy-dev cloud-init (0.7.x)]$ python -c "from cloudinit import log as logging; import logging.config"
<suro-patz> Traceback (most recent call last):
<suro-patz>   File "<string>", line 1, in <module>
<suro-patz>   File "cloudinit/log.py", line 24, in <module>
<suro-patz>     import logging.config
<suro-patz> ImportError: No module named config
<suro-patz> (py27) [suro@oxy-dev cloud-init (0.7.x)]$
<harlowja_at_home> suro-patz, ok, no idea why that happens
<harlowja_at_home> probably the order u importing things
<harlowja_at_home> since u just overwrote the logging import
<harlowja_at_home> to be cloudinit.log
<suro-patz> exactly
<suro-patz> I was wondering it should be observed by others too
<suro-patz> wanted to confirm
<suro-patz> on master-branch it works fine
<suro-patz> is it possible that on branch-0.7.x it is not maintained?
<harlowja_at_home> lol
<harlowja_at_home> suro-patz, so u yourself are causing that issue
<harlowja_at_home> ask yourself what your code does there
<harlowja_at_home> it renames cloud.init log as logging
<harlowja_at_home> then tries to import logging.config from that
<harlowja_at_home> which will obviously fail
<harlowja_at_home> or afaik that's what its doing
<harlowja_at_home> so in other words, idk why u doing that
<harlowja_at_home> bbl
<suro-patz> harlowja_at_home: I just mimicked what cloud-init was failing at
<suro-patz> ERROR: Failure: ImportError (No module named config)
<suro-patz> ----------------------------------------------------------------------
<suro-patz> Traceback (most recent call last):
<suro-patz>   File "/home/suro/cloudInit/cloud-init/.tox/py27/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
<suro-patz>     addr.filename, addr.module)
<suro-patz>   File "/home/suro/cloudInit/cloud-init/.tox/py27/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
<suro-patz>     return self.importFromDir(dir_path, fqname)
<suro-patz>   File "/home/suro/cloudInit/cloud-init/.tox/py27/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
<suro-patz>     mod = load_module(part_fqname, fh, filename, desc)
<suro-patz>   File "/home/suro/cloudInit/cloud-init/tests/unittests/test_util.py", line 12, in <module>
<suro-patz>     from cloudinit import importer, util
<suro-patz>   File "/home/suro/cloudInit/cloud-init/cloudinit/util.py", line 55, in <module>
<suro-patz>     from cloudinit import log as logging
<smoser> rharper, 90 minutes ?
<rharper> smoser: sure
<suro-patz> harlowja_at_home: never mind, this error happened on repo where I had run tox for master branch once and then switched to 0.7.x. As I created a fresh repo, I do not see the issue any more - tox works for 0.7.x for py27
<smoser> rharper, i have added fix and test for lxd.
<smoser> that was busted. pushing that now
<rharper> smoser: nice!
<rharper> did you get the fix on the init_cmd dict ?
<rharper> used unknown variable
 * rharper is preparing a pep8/pylint/pyflakes MP 
<smoser> oh. i just enabled pyflakes
<smoser> trunk runs that on package build and in tox now
<rharper> smoser: cool
<smoser> pushed that and lxd to trunk.
<smoser> if you wanted to clean up ./tools/run-pep8 i'd be fine with that.
<smoser> pep8 is such a freaking moving target.t ahts the biggest pain.
<smoser> it has been clean before, and then we get a newer version in new ubuntu release and starts failing.
<rharper> well, I doubt it's been that clea
<rharper> with the I am the King, do what I say
<rharper> but yeah; I like curtin's make check
<rharper> I think it keeps the source tidy;  it does move, but I don't think that much
<rharper> the first thing that seems challenging is that cloud-init is doing module path insertion in bin/cloud-init instead of the shell wrapper like in curtin (py2or3) would be useful here
<rharper> but not sure how much of the world you want to move/touch w.r.t that sort of thing
<kwadronaut> http://netfilter.org/projects/nftables/
<kwadronaut> oops
<kwadronaut> wrong window
#cloud-init 2016-03-04
<smoser> rharper, trunk now builds with /usr/lib/python3/ rather than 3.5.
<rharper> smoser: nice!
<rharper> https://code.launchpad.net/~raharper/cloud-init/make-check-cleanup/+merge/288041
<smoser> i'm fighint stupid systemd.
<smoser> i swear it worked before
<smoser> but now my disable doesnt seem to
<rharper> heh
<jgrimm> boo
<smoser> well, at least i understand this now.
<rharper> smoser: cool;  I've a reliable mechanism to collect data from the target without using cloud-init; as well as running After=multi-user.target;  I think we can also use cloud-final.target as well;  as even if cloud-init is disabled the target is still present for ordering;  that'll have to be confirmed once you've a disabled branch to test;
<rharper> annoyingly though since I'm running this as a systemd service, one cannot ask systemd how far along in the boot process one is; systemd-analyze says; still booting; try back later; which is quite annoying; but I can take proc/uptime samples; and if there are other cloud-init eyecatchers, we can look for those as well
<Odd_Bloke> smoser: Would you be able to release trunk in to xenial, please?
<smoser> Odd_Bloke, can you sniff my ppa ?
<smoser> and then i'll do that.
<Odd_Bloke> smoser: Your PPA?
<smoser> or if yoiu'd rather: https://code.launchpad.net/~cloud-init-dev/+archive/ubuntu/daily
<smoser> set that up last night. to daily build trunk with a xenial packaging branch.
<Odd_Bloke> smoser: Oh, nice!
<Odd_Bloke> Just kicked another recipe build off to pull my merges in.
<smoser> Odd_Bloke,  you did not enable the bigstep datasource
<Odd_Bloke> smoser: "enable"?
<Odd_Bloke> smoser: Oh, you mean add it to the default config?
<smoser>  cloudinit/settings.py
<smoser> i'll tell you the same thing i told utlemming yesterday
<smoser> when you merge, please make a git style commit with
<smoser>  summary <= 74 chars
<smoser>  blank line
<smoser>  longer description
<smoser> and add entry to 'ChangeLog'
<smoser> (like the summary line)
<smoser> and please give people credit in that line when appropriate
<smoser> and if you fix a bug, in the merge commit, please --fixes lp:XXXX
<Odd_Bloke> smoser: Ack.
<Odd_Bloke> smoser: https://code.launchpad.net/~daniel-thewatkins/cloud-init/bigstep/+merge/288105
<smoser> Odd_Bloke, when you dont have that file there..
<smoser> i dontrecall. does cloud-init warn that the ds raised exception ?
<smoser> can you just try and catch the load_file and return false if not there ?
<rharper> trusty pep8 (1.4.6) is fickler than xenial pep8 (1.6.2);  make check under trusty sbuild fails; fixing now;
<smoser> rharper, see what i said
<smoser> :)
<rharper> eh, 3 lines
<smoser> thats not so bad.
<rharper> pep8 is easy;  getting newer init system levels in trusty
<rharper> another storey
<rharper> I don't want to tackle now; but I suppose cloud-init ought to run as well on trusty (and know what it can and cannot due, ala curtin)
<rharper> smoser: https://code.launchpad.net/~raharper/cloud-init/trunk.fix-trusty-pep8/+merge/288126
<smoser> Odd_Bloke, aorund ?
<smoser> Odd_Bloke, bah
<smoser> Mar  4 20:29:05 x2 [CLOUDINIT] util.py[DEBUG]: failed read of /sys/class/dmi/id/product_name#012Traceback (most recent call last):#012  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2143, in _read_dmi_syspath#012    key_data = load_file(dmi_key_path)#012  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1187, in load_file#012    return decode_binary(contents)#012  File "/usr/lib/python3/dist-packages/clou
<smoser> dinit/util.py", line 88, in decode_binary#012    return blob.decode(encoding)#012UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
#cloud-init 2017-02-27
<rharper> smoser: so, how about this one... http://paste.ubuntu.com/24080787/   ;  rendering an eni with bonded vlans; cloud-init-local renders to disk, networking is all up and fine; then in cloud-init (net mode) we somehow attempt to rename the bond0; which fails, but then somehow, at the end of the service, we've dropped our routes.
<rharper> it's possible I've broken something; this is my v2 pass-through branch; though config isn't v2/netplan/systemd at all;
<rharper> the rename may not be the issue; but we should filter out non-physical devices when we collect the current_info;  in this case, it creates a dict with the bond0 which has the same mac as the interface; but we return bond0 as the name;
<rharper> currently looking for something in sys ;  I think we can filter by the set of names in /sys/devices/virtual/net
<rharper> but that fixed it;  I'll send a PR tomorrow
#cloud-init 2017-02-28
<powersj> magicalChicken: https://jenkins.ubuntu.com/server/job/cloud-init-integration-x/103/console
<powersj> Looks like the qemu migration tests were running at the same time as the integration tests. And the integration tests attempted to delete a snapshot. Why would cii do that?
<magicalChicken> powersj: a couple things here...
<magicalChicken> the reason the snapshot is being used is as the backend of the 'launch container from existing container' lxd feature
<magicalChicken> it looks like there was already an image using one of the ones being cleaned up by the cloud tests
<magicalChicken> actually, i guess it was during the image being created originally, not during a launch
<magicalChicken> maybe there was a uuid collision between images?
<magicalChicken> the old version of the tests have bad logic for naming images, and don't check existing names for uniqueness or use any namespacing, and we had run into collisions once before I on jenkins I think
<magicalChicken> the fix is in the merge proposal for the new tests
<magicalChicken> also, this could be connected to cloud tests deleting the image it uses by default instead of preserving, maybe it tried to remove an image that was being used by the qemu tests
<magicalChicken> my guess is that its a image name collision though, somehow
<magicalChicken> in any case, I think the new version of the tests may fix it
<magicalChicken> i can try to reproduce the issue otherwise
<magicalChicken> the attempted delete was just due to lxd trying to clean things up, the original failure was that an image already existed that it tried to overwrite
<smoser> rharper, well it shoudlnt be trying to rename network devices at 'init'
<smoser> but only in init-local
<rharper> smoser: hrm
<smoser> rharper, its to late to run at init time ... networking is already up
<rharper> I completely understand
<rharper> I don't know why it triggered the rename path a second time
<rharper> I'll get a cloud-init.log for your here in a bit
<rharper> there are multiple calls to apply_network_config in main ;  I've not sorted out if we're taking a path that this then multiple times
<rharper> smoser: http://paste.ubuntu.com/24084999/;  when 'init' runs, it checks the network config, in doing that, it validates that interfaces are named properly, but this time the mac of the interface to be renamed matches a bond; then we attempt to bring it down and fail.  basically with bonding, we have a mac that points to two interfaces (one physical, one virtual (the bond))
<smoser> right. i was about to type that. it does try to rename, but should have come up with "nothing to do"
<rharper> right, it should
<rharper> but when it looks up the interface by mac
<rharper> it finds bond0
<rharper> for what it thinks is interface0
<rharper> which is true; they share a mac due to how bonding works
<rharper> so then it thinks it needs to rename bond0 -> interface0
<rharper> since we only rename 'physical' interface (we fetch only 'physical' items from the config; however, when we look up the phys interface by mac, we get two answers (bond0, interface0) ;  but it gets bond first (sorting I guess)
<rharper> however, we should reject virt interface names when we're looking for interface info for renaming
#cloud-init 2017-03-01
<jgrimm> rharper, smoser: does the following bug still need cloud-init trackers?  note resolvconf moved to its own bug.
<jgrimm> https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1636912
<smoser> reading
<jgrimm> yeah, its a mess. i'm not sure why its in verification-needed state even
<jgrimm> noticed is as pending sru was showing it yellow .. even tho resolvconf doesn't track it since it has its own bug. https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1649931
<smoser> jgrimm, i removed the tasks.
<smoser> for cloud-init
<jgrimm> cool, i figured so. thanks
<smoser> i wish tobias wolf would have written more info
<smoser> "Thanks for bricking our servers"
#cloud-init 2017-03-02
<larsks> harlowja: around?  more net/sysconfig.py questions.
<harlowja> larsks ya
<harlowja> i can try to answer :-P
<larsks> harlowja: meh, I muddled through :).
<harlowja> :)
<harlowja> kk
<larsks> More changes to net/sysconfig.py for ipv6 support.
<larsks> harlowja: if you have a chance to review the merge that would be awesome...
<harlowja> sureeee
#cloud-init 2017-03-03
<smoser> larsks, around ?
<smoser> on https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1657130
<larsks> smoser: Around!
<smoser> bah. never mind. was wondering why you didn't use 'retries' in the wait_for_metadata_service call to wait_for_url
<smoser> but retries is kind of builtin there
<smoser> so never mind.
<smoser> i was just looking at logs. you did good.
<larsks> Phew!
<rharper> smoser: bug for attempting to rename a bond:  https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1669860
#cloud-init 2018-02-26
<blackboxsw> meh 2FA on hangout
<blackboxsw> smoser: I'll try the manual ovf test from our docs and validate whether we can see regression for this SRU
<blackboxsw> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1737704 and  https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1731868 are the two cases I wanted to cover
<ubot5`> Ubuntu bug 1737704 in cloud-init (Ubuntu) "Cloud-init fails if iso9660 filesystem on non-cdrom path in 20171211 image." [High,Fix released]
<ubot5`> Ubuntu bug 1731868 in cloud-init (Ubuntu) "cloud-id: enable ESXi 6.5.0" [High,Fix released]
 * blackboxsw grabs a coffee
<smoser> blackboxsw: thank you
<blackboxsw> hrm, puppet in general doesn't seem to work on bionic
<blackboxsw> can't even use the cmdline tooling for documentation
<blackboxsw> root@b1:~# puppet --help
<blackboxsw> /usr/lib/x86_64-linux-gnu/ruby/2.3.0/openssl.so: symbol SSLv2_method, version OPENSSL_1.0.0 not defined in file libssl.so.1.0.0 with link time reference - /usr/lib/x86_64-linux-gnu/ruby/2.3.0/openssl.so
<blackboxsw> root@b1:~# echo $?
<blackboxsw> 1
<blackboxsw> I'll file a bug against puppet packaging in bionic. but I don't think it'll block the branch that's up for review
<blackboxsw> .. though maybe it should
<blackboxsw> as we can't currently install a functional ubuntu-packaged puppet for bionic. get a  traceback on Command: ['service', 'puppet', 'start'].
<blackboxsw> but I think that looks like ssl-related issues with my example config. I'll validate that before holding up the branch
<rharper> blackboxsw: you can ask in #ubuntu-devel  nacc or others may already know what's up there
<blackboxsw> rharper: confirmed PEBKAC. But puppet itself may be brittle.
<rharper> hehe
<blackboxsw> I think I had an SSL config issue w/ puppet config I specified
<rharper> yeah
<blackboxsw> but, the puppet commandline falls over and doesn't even let you read it's own man/help info if misconfigured
<blackboxsw> trying to confirm that now, and will file an upstream puppet issue if that's the case
<nacc> blackboxsw: rharper: did you figure it out?
<blackboxsw> nacc: I think I had a stale bionic container and was getting ruby ssl errors (not providing SSL_V_1_0_0 or some such dependency issue.
<blackboxsw> nacc: I can't reproduce on fresh bionic containers
<blackboxsw> if I can reproduce, I will (I had blown away my stale lxc to see if I could reproduce on something new, but no dice)
<nacc> blackboxsw: ack
<nacc> blackboxsw: to be clear, puppet has dep8 tests that do test the cli
<blackboxsw> thanks nacc. I'm not sure how I got my old container misconfigured. I shouldn't have removed it. I do think I had either a version mismatch in ruby ssl dependencies that prevented puppet's tooling from running. But, using the same puppet configs on my new instances work just fine.
<nacc> blackboxsw: it's possible you got bit by a bionic transition?
<nacc> blackboxsw: there was an ssl one recently
<blackboxsw> I'll chalk it up to that as tracebacks were complaining about missing const definition for SSL_V1
<blackboxsw> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/339373 approved
<smoser> blackboxsw: thanks.
<brunobronosky> Any idea why my attempt to init chef isn't working? http://termbin.com/3kyu
<smoser> brunobronosky: cloud-init think the world is fine it looks like
<smoser> cat /run/cloud-init/result.json
<smoser> maybe chef has some output ?
<smoser> systemctl status chef ?
<brunobronosky> It's odd to me that chef got installed, but not from the source I requested (taken straight from the cloud-init docs example). And this:
<brunobronosky> 2018-02-26 19:16:55,045 - cc_chef.py[DEBUG]: Skipping unknown chef template key 'run_list'
<brunobronosky> cat /run/cloud-init/result.json | tb > http://termbin.com/8bto
<blackboxsw> brunobronosky: those skip messages are ok..  (though noisy) it just means those key values aren't interpreted but the chef template generator I think. Trying to see aobut your sources comment
<blackboxsw> I'm re-reading tip cc_chef module code right now to refresh
<blackboxsw> brunobronosky: did you see a /etc/chef/firstboot.json on your system?
<brunobronosky> userdata file: http://termbin.com/57yw
<blackboxsw> I'd expect that firstboot.json to contain ypour run_list
<brunobronosky> Oooo. it does contain my runlist.
<blackboxsw> yeah those Skip messages, are noisy and confusing.... I think they should be dropped as the code specifically knows it
<blackboxsw> should ignore/skip them
<blackboxsw> ok so I see per your cloud-init.log that chef was installed via apt command, which is what your config specified I think.
<blackboxsw> brunobronosky: oops
<blackboxsw> http://termbin.com/57yw
<blackboxsw> ok looks like our doc may have a typo
<blackboxsw> I think the apt configuration is wrong
<smoser> yeah
<smoser> missing sources
<blackboxsw> apt:\bsource1: source1:
<blackboxsw> apt:\bsources: source1:
<blackboxsw> geez typo city
<blackboxsw> pasting
<smoser> apt/sources/source1:
<smoser> rather than
<smoser> apt/source1/
<blackboxsw> https://pastebin.ubuntu.com/p/RyKqPQQcN8/
<blackboxsw> yeah
<blackboxsw> that key is bogus.
<brunobronosky> aha!
<blackboxsw> pultting up a trivial
<blackboxsw> brunobronosky: smoser https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/339715
<blackboxsw> thanks for raising the question brunobronosky
<blackboxsw> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/339720 rharper or smoser if you get a chance. here's the set hostname approach which I've validated on vsphere. It fixes the DHCP request containing the stock cloud-image hostname 'ubuntu'. I hadn't added the init_main unit test to get coverage on the set_hostname call, but I wanted to bounce the idea off you guys before I sink  time into
<blackboxsw> the init-loca CLI unit test.
<rharper> k
#cloud-init 2018-02-27
<blackboxsw> rharper: finally got to what you were saying in standup about UserDataProcessor for read_file_or_url
<rharper> cool
<blackboxsw> yeah looks like we handle the exception without raising it
<blackboxsw> and log the warning anyway.
<blackboxsw> so it's a no fail attempt
<blackboxsw> we either have processed user-data or not and log an error.
<blackboxsw> this would be a problem if we created a semaphore for set_hostname module, but since we're calling it directly we don't have to worry â¢
<rharper> so, ideally we
<rharper> d want to know in stages if we failed to process all user-data parts
<rharper> and if so, call update a second time;
<rharper> or, if we keep that inside the update() method (even in the userdata class, not sure about where we capture that knowledge)
<rharper> but a second call would do nothing if it already processed everything, vs a second call (after networking is up) and it would "finish" processing
<rharper> blackboxsw: yeah; that does work well
<blackboxsw> right it feels like UserDataProcessor would need an instance attr that says "fail hard in error" or something.
<blackboxsw> so we could react to a failed #include
<blackboxsw> will see where it makes sense
<blackboxsw> can we have gzip'd #includes too?
<rharper> I dunno
<blackboxsw> nope doesn't look like it gzip exploding happens after #include attempts
<blackboxsw> ok I'll put something up and test it
<blackboxsw> thanks again gents
#cloud-init 2018-02-28
<smoser> rharper, blackboxsw would like some thoguhts on https://code.launchpad.net/~d-info-e/cloud-init/+git/cloud-init/+merge/340112
<brunobronosky> Anybody here have experience with cloud-init + chef? I'm trying to use http://cloudinit.readthedocs.io/en/17.1/topics/examples.html#install-and-run-chef-recipes and it is not respecting my intent to install the chef omnibus and is installing chef/xenial package instead.
<blackboxsw> brunobronosky: did you specify  install_type: "omnibus" in the chef: #cloud-config section?
<blackboxsw> I thought I had validated this in a review on chef module a few months ago
<blackboxsw> if install_type: 'packages' is specified as per that example, cloud-init will pull from distro packages
 * brunobronosky does not read comments good! https://github.com/cloud-init/cloud-init/blob/ubuntu/17.1-0ubuntu1/doc/examples/cloud-config-chef.txt#L94
<brunobronosky> Yep. That is exactly what it is doing. Sorry.
 * blackboxsw thinks that example is confusing/muddled, it tries to comment multiple behaviors at the same time.
<blackboxsw> no worries brunobronosky good news is, it's on our radar to update schema definitions and docs for modules like chef this year, so this should be better documented as we get through this module
<brunobronosky> I need to go back to http://media.tmz.com/2016/02/06/0206-derek-zoolander-vh1-films-paramount-pictures-01-1200x630.jpg
<blackboxsw> I generally also go to the specific module docs too when looking at how to configure it. But, unfortunately chef is not currently that helpful. http://cloudinit.readthedocs.io/en/latest/topics/modules.html#chef
<blackboxsw> excellent link brunobronosky . Glad you've got my same appreciation for good/bad movies.
<blackboxsw> hrm, ok so a datassource's metdata is only updated by the std datasource.get_data() which gets both user-data and meta-data, either we can decouple those  in all datasources to process metadata alone, or we add a cache_clear option to the datasource if errors are raised during userdata processing. either way looks like this hostname setting early  is going to be a bigish branch
<blackboxsw> I'm more inclined to go the UserDataProcess raises errors approach and the datasource base class could handle that exception and avoid caching metadata  userdata on the datasource in that case (because we know we'll be re-running in a later cloud-init stage
<brunobronosky> Woot! I have finally got my "cloud-init chef bootstrap" POC done. It's pretty functional, but I need to find a way to use the value of the ECS Name tag in the user-data file which is of type cloud-init.
<brunobronosky> But anyway, to do it I used and undocumented feature `exec_arguments` that I found in https://github.com/cloud-init/cloud-init/blob/ubuntu/17.1-0ubuntu1/cloudinit/config/cc_chef.py
<brunobronosky> But an error I encountered along the way raised an interesting question. http://termbin.com/k4m4
<Gaffel> What version of cloud-init is that?
<brunobronosky> I fixed that error by quoting the entries like so http://termbin.com/ymz6 But I think it's a bug that YAML is passed directly to cmd.extend because util.subp is never going to like int, or any other non-strings.
<brunobronosky> Gaffel 17.1
<blackboxsw> brunobronosky: your example could also be   exec_arguments: '-d -i 1800 -s 20'      and that'd work. I think it's worth a bug against cc_chef, it's taking the proper yaml (including int types and blindly appending it the to subprocess call on the cli, it shouldn't be calling util.subp with a list containing non-string items
<blackboxsw> brunobronosky: on the failed node you could call ubuntu-bug cloud-init and it'd prompt you for the information to file a bug
<brunobronosky> ha! didn't know that. nice.
<blackboxsw> that's:      ubuntu-bug cloud-init  on the commandline of the failed node
<blackboxsw> yeah it's a recent addition it collects all cloud-init logs and user-data if you let it so you don't have to
<blackboxsw> ok fixed up my hostname setting branch to take into account errors and retry. tested ec2 & vsphere. adding unit tests now
<blackboxsw> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/339720
#cloud-init 2018-03-01
<do3meli> anyone able to tell me how to activate debug logging for a specific module?
<smoser1> do3meli: you can't turn it on per module like that.
<smoser1> generally at this point DEBUG gets written to /var/log/cloud-init.log
<smoser1> youc an run individual modules: cloud-init single --name=apt_configure --frequency=always
<do3meli> thanks smoser1, i realized that and found the correct log file later on ;-)
<smoser1> do3meli: your mp at https://code.launchpad.net/~d-info-e/cloud-init/+git/cloud-init/+merge/340112 needs to be rebased to master
<smoser1> git fetch origin
<smoser1> (or git fetch upstream)
<smoser1> dependening on what is the name of the upstream one
<smoser1> then
<smoser1> git checkout master
<smoser1> git merge --ff-only
<smoser1> git checkout salt-freebsd-patch
<smoser1> git rebase -i master
<do3meli> hmm de rebase would just pick 2 commits:
<do3meli> pick dbc395c salt-minion module now works on FreeBSD
<do3meli> pick bee5b16 adjust formatting
<do3meli> would that be correct?
<smoser1> correct
<smoser1> those are the 2 commits that you have made
<smoser1> right ?
<smoser1> and you just want those moved on top of master
<smoser1> we *had* fixed the deb install issue you saw in trunk
<smoser1> but you are off an older version of trunk, so you dont have that fix.
<smoser> do3meli: that make sene ?
<smoser> sense
<do3meli> hmmm...really. i am confused. because created a brand new branch from upstream/master (in my case origin. but its the upstream)
<do3meli> i followed the steps in HACKING.rst as you mentioned to me yesterday.
<do3meli> well. anyway. let me try again ;-)
<smoser> powersj, rharper, blackboxsw, dpb1 . i think i had written this somewhere before. but... lost it.
<smoser> https://hackmd.io/KwUwDAnAxgJgbARgLQA4DsAzCSAsBDHFVCBDJKNPAIzmDQgGY0pgg===
<smoser> that is just a copy-and-paste if a user contributes that has not signed CLA.
<Odd_Bloke> It looks like the most recent cloud-init in artful doesn't handle GCE user data any longer.
<Odd_Bloke> I'm filing a bug now (unless someone points me at an existing one).
<blackboxsw> ok odd_bloke, ssh handling is a bit different in gce now. certain expired keys removed etc.
<blackboxsw> please file away, will check details. 'ubuntu-bug cloud-init'
<Odd_Bloke> blackboxsw: https://bugs.launchpad.net/cloud-init/+bug/1752711
<ubot5`> Ubuntu bug 1752711 in cloud-init "cloud-init no longer processes user data on GCE in artful" [Undecided,New]
<Odd_Bloke> Yeah, this seems more fundamental than that.
<blackboxsw> thanks looking now
<Odd_Bloke> Hopefully I'm just doing something wrong.
<Odd_Bloke> That wasn't filed with ubuntu-bug; is there a way I can add the ubuntu-bug information after the fact?
<blackboxsw> cloud-init collect-logs
<rharper>  #/bin/sh
<rharper> that's missing the the bang, no ?
<smoser> Odd_Bloke: apport_collect
<blackboxsw> Odd_Bloke: ^ cloud-init collect-logs:   that'll grab a tarfile of all data/logs to attach to bugs when filed after the fact
<smoser> err... apport-collect
<Odd_Bloke> rharper: Ah, yeah, so it is.
<Odd_Bloke> rharper: We are seeing this in automated testing as well, which doesn't have that typo.
<Odd_Bloke> (Also, would that cause cloud-init to not even store it in /var/lib/cloud?)
<rharper> huh, the % cat thingy made me thing it was a real cat
<rharper> nothing in /var/lib/cloud at all ?
<rharper> did cloud-init run ? cloud-init status says ?
<Odd_Bloke> rharper: That was a real cat from the actual file I manually ran this with.
<rharper> ok, then the file is broken, it's not going to execute, right ?
<Odd_Bloke> I've just reproduced with the bang.
<rharper> ok, did you use something else besides /tmp ?
<rharper> systemd likes to clean /tmp and /var/tmp for you
<rharper> touch /ubuntu-is-awesome
<Odd_Bloke> Trying now.
<Odd_Bloke> Yeah, still not working.
<rharper> ok; and any way to get the logs of those and attached ?
<Odd_Bloke> Yeah, will do so after this meeting.
<smoser> blackboxsw: http://paste.ubuntu.com/p/s7DQJrvTKn/
<blackboxsw> smoser there's aslo a 2nd part that handled encoded user-data
<smoser> its there still
<blackboxsw> ahh right you're paste manipulates that content
<rharper> smoser: blackboxsw: Odd_Bloke: It looks like the recent change to GCE source dropped setting md['user-data'] since the introduction of recursive query of instance/attributes URL;
<rharper> blackboxsw is confirming
<Odd_Bloke> Ack.
<Odd_Bloke> Thanks for the speedy response. :)
<blackboxsw> smoser your fix is good.
<smoser> the main is useful :)
<rharper> yes
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/340248
<smoser> i'll add a unit test
<blackboxsw> smoser: when's EOD for you ? I will shepherd that in once I re-test on gce with changes. Also time to  expand our manual tests to include  user-data on all clouds
<smoser> ok. i have 20 minutes.
 * blackboxsw will flag regression to stop xenial from getting an update
<blackboxsw> (given that it's blocked
<blackboxsw> good pt. will deploy and check
<smoser> blackboxsw: good. thanks.
<blackboxsw> yeah bionic is good, I see things working there in bionic.
<blackboxsw> checking the changeset
<smoser> why would it work in bionic?
<blackboxsw> strike that on bionic, I provided the same #cloud-config\n hostname: <myname>  as when I used gcloud to create the instance. so I mistakenly thought user-data was observed.   It is not: bionic fail too
<blackboxsw> Odd_Bloke: thanks for the bug, I just refiled one with the regression tags etc https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1752729 and dup'd yours
<ubot5`> Ubuntu bug 1752711 in cloud-init (Ubuntu Artful) "duplicate for #1752729 cloud-init no longer processes user data on GCE in artful" [Critical,Confirmed]
<blackboxsw> landing the fix today. bionic should be fixed by tomorrow and we'll SRU a cherry pick to artful
<blackboxsw> landing the fix today. bionic should be fixed by tomorrow and we'll SRU a cherry pick to artful/xenial
<Odd_Bloke> Ack, thanks!
<blackboxsw> Odd_Bloke: do your automated tests run against bionic?
<Odd_Bloke> blackboxsw: They do, but we've been having problems in the pipeline before we get to GCE image publication/test.
<blackboxsw> ahh gotcha.
<blackboxsw> smoser: bionic merge is up https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/340252
<blackboxsw> and your fix has landed in tip
<blackboxsw> xenial cherry pick https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/340256
<blackboxsw> artful cherry pick https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/340255
#cloud-init 2018-03-02
<smoser> Odd_Bloke: https://git.launchpad.net/cloud-init/commit/?id=e626359a6ea
<smoser> that is being questioned on bug 1747705
<ubot5`> bug 1747705 in cloud-init (Ubuntu) ""ssh_pwauth" always true on CloudStack datasource with password" [Medium,Confirmed] https://launchpad.net/bugs/1747705
<smoser> can you give some more history there?
<Odd_Bloke> smoser: Looking.
<Odd_Bloke> smoser: Commented.
<smoser> Odd_Bloke: thanks.
<smoser> rharper,  blackboxsw . this is actually surprising to me.
<smoser> http://paste.ubuntu.com/p/c4PVSgYVcq/
<smoser> x = []; x.append("foo");
<smoser> is almost twice as slow as
<smoser> x = ["foo"]
<smoser> (although using hte term "slow" for either is a liberal use of the word)
<dojordan> Hey @Smoser, when you get a chance can you take a look at this? We recently found this bug when testing cross vnet preprovisioning: https://code.launchpad.net/~dojordan/cloud-init/+git/cloud-init/+merge/340546
<blackboxsw> will get some review comments on that today. can chat about it Mondayish.
<blackboxsw> smoser/rharper/Odd_Bloke:  SRU fix for artful is published and I validated user-data working on gce
<Odd_Bloke> blackboxsw: Thanks!
<blackboxsw> thanks for catching it before xenial got updated too
<rharper> bah, already left
<rharper> I wanted to get more details on how to test that
<rharper> =/
#cloud-init 2020-02-24
<meena> shibumi: unless you only have one interface and use dhcp, then it's pretty much clear what to do ;)
<otubo> meena: rharper Odd_Bloke thanks a lot for the always good review. I'll consider all the comments and will update the PR leter on. :-)
<meena> \o/
<Odd_Bloke> otubo: :)
<meena> Good morning Odd_Bloke â have you fixed all my bugs yet?
<Odd_Bloke> meena: Yep, any that remain are contained entirely within your head.
<Odd_Bloke> (Good morning!)
<meena> cool.
<meena> i should put my head back into development / testing then.
<Odd_Bloke> shibumi: Interesting!  I'm not super-familiar with Arch's ecosystem, is that discussion about the official Arch cloud images?  And is the proposal to set net.ifnames=0 on all cloud images?
<shibumi> Odd_Bloke: yep and yep
<shibumi> Odd_Bloke: but we still have issues with cloud-init
<rharper> shibumi: generally we do no rely on the predictiable names;  even before systemd/networkd lo is a standard interface name;  w.r. t to things like eth0;  there is no guarantee that a virtual nic will get the name "eth0";   and even if there is a nic named "eth0" it may not be the nic you intend; as soon as there are more than one interface there needs to be some way to ensure that the config is applied to the correct interface.  We cover many
<rharper> scenarios such as shutdown, remove nic, boot up; many platforms may modify the nic properties (mac address) between boots;  multiple nics may be added either dynamicall, or staticly (offline)
<rharper> cloud-init does attempt to ensure the config is applied to a specific nic; as such, it will either utilize udev to emit rules akin to the 70-net-persistent.rules ; or if on systemd, emit configuration files which match the specific nic (independent of the name);  cloud-init also will rename interfaces to match a network-config provided so that when the OS networking layer comes up; the existing on-disk config will still apply;
<rharper> w.r.t renaming nics; the eth* namespace is problematic since the kernel effectively owns/manages this namespace and renaming devices within the namespace race with the kernel itself (and additional nic adds/removes);
<rharper> I would strongly advise against encoding net.ifname0 in images as this leads to other challenges down the road where images are utlized in multi-nic scenarios, ordering of devices based on the platform you run upon (different virt technologies do not always provide a persistent address (pci bus slot) between reboots or other operations)
<shibumi> rharper: do you have a different solution to the problem we encounter?
<rharper> did you find out why you don't get a 'lo' interface ?
<rharper> I saw some discussion of something symlinking the systemd defaults config
<rharper> it wasn't clear why that was done;
<shibumi> We thought it's maybe related to  ln -s /dev/null /etc/systemd/network/99-default.link
<rharper> shibumi: ok, re-reading;  w.r.t the default image; you do not need to bake in any existing network-configuration; cloud-init already provides a default of  dhcp on the best interface (where best is a heuristic that cloud-init uses , skips certain interfaces like bonds or bridges, nics that have no carrier, etc;)  it's better than just assuming that you can dhcp a kernel device named eth0
<rharper> Ubuntu images do not bake any network config in them and rely on cloud-init to auto generate this config;  on platforms which do provide network config (Like openstack, azure, ec2)  the datasource reads the platform network config and then cloud-init will render that to the OS renderer
<shibumi> rharper: ok interesting. Btw is this the right method to enable cloud-init in images? https://github.com/archlinux/arch-boxes/blob/master/provision/cloud-init.sh
<rharper> so, I believe if you can sort out getting your lo interface back, and remove the baked in netplan config;  the image will generate a dhcp on the best interface on the fly
<shibumi> I was unsure if we enable too much
<rharper> shibumi: if you have systemd, then that's too much
<shibumi> rharper: we have systemd
<rharper> cloud-init uses a systemd generatator to dynamically enable cloud-init IIF there is a datasource
<shibumi> rharper: so I can actually just install cloud-init and I don't need to enable the services manually?
<shibumi> or do I need to enable just 1-2 services from the 4?
<rharper> ideally yes, though it depends on your distro packaging
<rharper> zero if you install the cloud-init-generator
<shibumi> what do you mean with 'install'? do I need to enable that cloud-init-generator service? or is this just a binary that I need to drop in the right place?
<rharper> it's a binary (script) that's put in the place where other systemd generators are installed
<shibumi> cloud-init usr/lib/systemd/system-generators/cloud-init-generato
<shibumi> we install it there ^
<rharper>  /lib/systemd/system-generators/cloud-init-generator
<rharper> looks right to me
<shibumi> rharper: lib is a symlink to /usr/lib in arch ;)
<shibumi> ok cool
<rharper> so you could skip your service enables and boot that image with NoCloud datasource  , I think your Issue has some steps on creating a datasource iso and attaching that to the VM to test
<rharper> ## create a disk to attach with some user-data and meta-data (require cdrkit)
<rharper> $ genisoimage  -output seed.iso -volid cidata -joliet -rock user-data meta-data
<rharper> so the volid 'cidata' is one of the bits of info that cloud-init generator will detect and then enabling all of the cloud-init services
<rharper> shibumi: the goal of the generator is to not have cloud-init run if there is nothing for it to do (ie if someone just booted your cloud image not on a cloud) it shouldn't try to call out to Ec2 metadata services,
<shibumi> rharper: well, so if somebody boots up the qcow2 image without cloud image datasource the image will have actually no network configuration (if I remove that network configuration) is this correct?
<rharper> shibumi: yes; and it depends on your goals for the image if you want that behavior or not
<shibumi> rharper: good question. Would be cool if we could satisfy both (normal users who want to boot that up in qemu and cloud users)
<shibumi> rharper: are we gonna to have problems if we just leave that dhcp configuration for ethernet devices there? cloud-init should change this via netplan, is this correct?
<rharper> the question you have to answer is for the local boot, do they, or do they not want networking enabled by default;  in the case where no network is available (but they have a nic) then you sit and wait for networking service to come up
<rharper> which can be 120s
<rharper> that's not a nice experience
<shibumi> or true
<rharper> shibumi: you can use a more generate match in your baked in config, of match: {'name': 'en*}; which will match any ethernet device;  as you'll never know the mac address of the VM you boot upon
<rharper> however, if the user runs the image on a platform which has finer grain control over network-config and wants to provide static IPs in addition to dhcp on the interface, there may be network config conflict
<shibumi> mh I see
<rharper> now, the distro class controls this
<rharper> see cloudinit/distros/debian.py:_maybe_remove_legacy_eth0()
<shibumi> then the best choice is that we declare the images as 'cloud only' and refer to the vagrant images if somebody wants to use a box on their laptop
<rharper> you could add  step like this in arch distro class to detect if you should remove the baked in config or not
<rharper> or augment it
<rharper> another thought is a drop-in config in systemd which could remove this baked in config file if cloud-init is going to run
<rharper> so, you could have a cloud-init-local.service.d/10-remove-config.conf ; in which you have [Unit] ExecStartPre=/bin/rm -f /path/to/baked/in/config
<rharper> if cloud-init is enabled, that would run and remove your baked in config (and cloud-init will handle bringing up networking)
<rharper> but without a datasource, cloud-init does not run at all , and your built-in config would be used instead
<Odd_Bloke> shibumi: You shouldn't need baked-in config for Vagrant to work; "DHCP on the first interface" is appropriate behaviour there (in my experience).
<shibumi> Odd_Bloke: vagrant and qcow2 images are a different thing in our project
<shibumi> Odd_Bloke: vagrant used netctl for Arch Linux in the past (that is deprecated and therefore not supported right now)
<shibumi> the newest version will use systemd-networkd instead
#cloud-init 2020-02-25
<DanyC> hi, anyone able to help me out understand why cloud-final.service on a new started EC2 instance is in inactive (dead) state? same state is for cloud-init.target. However if i ssh and manually start cloud-final.service everything is okay. If i run "cloud-init modules --mode=final" everything gets applied, no errors
<DanyC> i came across https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1623868 which shows similar behaviour however that bug is present on my cloud-init version .. hence i'm v confused
<ubot5> Ubuntu bug 1623868 in cloud-init (Ubuntu) "cloud-final.service does not run due to dependency cycle" [High,Fix released]
<DanyC> *that bug is fixed
<meena> DanyC: which now? fixed or present?
<DanyC> meena: what i meant to say is that the above bug it should be fixed in my latest cloud-init version hence i suspect i get only the same symptom but different root cause
<meena> DanyC: aye
<meena> DanyC: which version of cloud-init do you have?
<blackboxsw> https://github.com/canonical/cloud-init/pull/225 up to add ubuntu focal cloud-integration tests to cloud-init
<blackboxsw> thx powersj on the review there. I'm plugging that into CI and saw the expected failure with ssh_import_id in our tests too
<powersj> yea tests \o/
<blackboxsw> powersj: do you know again where that re-sync cloud-init code url is on launchpad. I can't find it
<powersj> blackboxsw, I think last time we said it was a jenkins job
<powersj> which surprised me
<blackboxsw> ahh right,https://jenkins.ubuntu.com/server/job/cloud-init-github-mirror/
<blackboxsw> right I forgot about that
<blackboxsw> so that we could better control when we have the urge to "do it now"
<blackboxsw> anyhow, didn't matter because the integration tests work off of daily ppas. which has uploaded, but still waiting on package publish
<Odd_Bloke> smoser: blackboxsw: I've updated https://github.com/canonical/cloud-init/pull/211/ with an approach that gives us two xenial tox envs: one for CI (with the "right" version of pytest) and one for local dev (with the working version of pytest, still in the default list executed).  Could you re-review, please?
<smoser> Odd_Bloke: i dont mean to cause problems.
<Odd_Bloke> smoser: How would you expect that failure to manifest?
<Odd_Bloke> `./tools/tox-venv -h` does work, and lists both environments in its output.
<smoser> ./tools/tox-venv xenial-dev
<smoser> probaly wont work.
<smoser> i ugess it doesnt parse that section at al
<smoser> all
<Odd_Bloke> smoser: I've just commented (https://github.com/canonical/cloud-init/pull/211/files#r384146914) and it looks OK to me?
<Odd_Bloke> Yeah, I think it just ignores that section because it doesn't start with [testenv:
#cloud-init 2020-02-26
<ausfestivus> whats the best way to get some help with a cloud-init problem here?
<meena> ausfestivus: asking a question is a good start.
<ausfestivus> lol
<ausfestivus> sure
<meena> n.b.: most devs (the people with @ in front of their nick) are somewhere in North Americaâ¦ so if us, mere mortals cannot help, you'll have to wait for them to wake up.
<ausfestivus> im trying to debug why a script is not running when its placed in /var/lib/cloud/scripts/per-once
<ausfestivus> on ubuntu  18.04 on azure as part  of a  scale set
<ausfestivus> the script IS in the above path
<ausfestivus> root@cptadotfvm000000:/var/lib/cloud/scripts/per-once# lsinstall-ado-agent.sh
<ausfestivus> root@cptadotfvm000000:/var/lib/cloud/scripts/per-once# lsinstall-ado-agent.sh
<ausfestivus> apols for the formatting
<ausfestivus> im following the steps at https://cloudinit.readthedocs.io/en/latest/topics/faq.html#how-can-i-debug-my-user-data
<ausfestivus> to debug the user data
<meena> we've recently improved the documentation on per-X, or so i hopeâ¦ i wonder if it's released on Read The Docsâ¦
<ausfestivus> when I run the `cloud-init query userdata > user-data.yaml` command, I get a bunch of python gunk. I suspect because the VM has python2 and cloud-init needs python3
<meena> ausfestivus: that shouldn't really matter. cloud-init should be using the correct python.
<meena> ausfestivus: can you pastebin the python gunk somewhere, please?
<ausfestivus> standby
<meena> i can't do anything else
<meena> well, i could sleep, but i don't think it's appreciated at $customer's office.
<meena> unless you have 20 years seniorityâ¦
<ausfestivus> whats the best way to paste console output in here?
<ausfestivus> webclient here
<ausfestivus> ```
<meena> ausfestivus: you copy/paste the output to your favourite pastebinâ¦ or gist.github.com or whatnot, and then post the link here
<ausfestivus> ```cptuser@cptadotfvm000000:/tmp$ sudo cloud-init query userdataTraceback (most recent call last):  File "/usr/bin/cloud-init", line 11, in <module>    load_entry_point('cloud-init==19.4', 'console_scripts', 'cloud-init')()  File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 893, in main    get_uptime=True, func=functor, args=(name,
<ausfestivus> args))  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2558, in log_time    ret = func(*args, **kwargs)  File "/usr/lib/python3/dist-packages/cloudinit/cmd/query.py", line 124, in handle_args    instance_data['userdata'] = util.load_file(user_data_fn)  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1372, in load_file
<ausfestivus> return decode_binary(contents)  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 146, in decode_binary    return blob.decode(encoding)UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte```
<ausfestivus> yea okay
<meena> ausfestivus: not like thisâ¦ :P
<ausfestivus> https://gist.github.com/ausfestivus/cdd1635b9340cb2f61ee83eed071f04e
<ausfestivus> ^^ python  gunk
<meena> ausfestivus: can you run `file /var/lib/cloud/scripts/per-once/install-ado-agent.sh`
<ausfestivus> ```cptuser@cptadotfvm000000:/tmp$ file /var/lib/cloud/scripts/per-once/install-ado-agent.sh/var/lib/cloud/scripts/per-once/install-ado-agent.sh: Bourne-Again shell script, ASCII text executable```
<ausfestivus> the script is fine
<ausfestivus> if I run it from there it does the job fine
<meena> ausfestivus: it would be nice to know which script it's complaining about
<ausfestivus> hence why im  trying  to get some debug info from cloud-init
<ausfestivus> hence the python gunk after following the debug steps in the docs
<meena> oooh, waaaitâ¦ is it userdata it's complaining about?
<ausfestivus> it doesnt complain
<meena> ausfestivus: python-gunking about
<ausfestivus> oh
<ausfestivus> `sudo cloud-init query userdat`
<ausfestivus> `sudo cloud-init query userdata`
<meena> that works?
<ausfestivus> no
<ausfestivus> gives the gunk
<meena> i assumed from # that you're root.
<meena> ausfestivus: it's odd, according to docs, gzip compressed userdata should just be decompressedâ¦
<meena> and your userdata is.
<meena> here's a cut from /usr/share/mime/magic:
<meena> [50:application/gzip]
<meena> >0=^@^B^_<8b>
<meena> that means, gzip is identified by a magic byte sequence of 0x8b
<meena> ausfestivus: you can try play around with https://cloudinit.readthedocs.io/en/latest/topics/analyze.html â or read the logs yourself, to figure out if this same behaviour happens on boot, cuz if it does that's a serious bug (or misconfiguration, or misunderstanding)
<ausfestivus> are you saying  thefile in per-once should be mime-encoded?
<meena> ausfestivus: no, i'm saying that your userdata is gzipped. that's what "python gunk" is saying. and for reasons unclear to me, cloud-init is choking on that, rather than ungzipping it, before trying to decode it as text
<meena> the only thing i can imagine going wrong is double gzip encoding, or, a bug.
<ausfestivus> afk dinner
<ausfestivus> @meena how can I debug this problem further? Any tips?
<ausfestivus> the debug tools in the FAQ are not working for a reason I cant decipher
<meena> ausfestivus: pastebin your logfile, and i can help take a look. also pastebin your /var/lib/cloud-init/instance/userdata (or whatever the actual real path is)
<ausfestivus> you want both log files?
<meena> sure
<meena> if you're unsure about your logs, cuz they're a lot, you can cloud-init clean --logs --reboot
<meena> to get a new set of fresh logs
<ausfestivus> cloud-init-output.log - https://pastebin.com/0CiR0yFEcloud-init.log - https://pastebin.com/iG6Up8SP
<ausfestivus> grrr
<ausfestivus> standby
<ausfestivus> cloud-init-output.log - https://pastebin.com/0CiR0yFE
<ausfestivus> cloud-init.log - https://pastebin.com/iG6Up8SP
<ausfestivus> I have a userdata.txt and a userdata.txt.i both are binary data
<ausfestivus> err correction
<ausfestivus> .i isnt
<ausfestivus> userdata.txt.i https://pastebin.com/qacVGm4p
<meena> ausfestivus: how many boots is this? did you run clean --logs --reboot?
<ausfestivus> just one boot, didnt run clean --logs --reboot
<hkominos> Good morning guys. I wanted to ask the channel if cloud init works with only python3 installed. I am trying to build centos8 with cloud-init but I am facing issues. (even the upstream image of centos8 is showing missing deps for cloud-init)
<meena> hkominos: the latest version has dropped python 2.7 support
<meena> hkominos: but, you may wanna describe exactly what problems you're facingâ¦ or pastebin some stack-trace or similar.
<meena> how are you trying to build cloud-init, etcâ¦
<otubo> rharper: quick question (because we talked about this last time): regarding BZ#1748015, I managed to fix this by adding "ExecStartPost=/usr/bin/systemctl try-restart NetworkManager.service" to cloud-final.service.
<otubo> (I thought the bot would resolv bugzillas as well :-) - [cloud-init][RHEL7] /etc/resolv.conf lose config after reboot (initial instance is ok) - https://bugzilla.redhat.com/show_bug.cgi?id=1748015
<ubot5> bugzilla.redhat.com bug 1748015 in cloud-init "[cloud-init][RHEL7] /etc/resolv.conf lose config after reboot (initial instance is ok)" [High,Assigned]
<otubo> oh it does :-D
<otubo> rharper: do you know if this race is seen in other distros as well? Shall I send a pull-request or just keep it downstream?
<meena> otubo: i'd open a PR. better safe than sorry
<meena> i can imagine an insigificant update in systemd to create this race where it wasn't there previously
<otubo> meena: true that!
<hkominos> hi meena!. Yes python 2.7 has been dropped (correctly in my ipinion). SO the stock Centos8 image that comes from upstream here https://cloud.centos.org/centos/8/x86_64/images/ has cloud init 18,5 installed. However a dnf update shows for example that 19.2 is available BUThttp://cpaste.info/29lw/. In my case even I have a prebuild 19.4 that I use in
<hkominos> our centos7 images which works fine but i see even more dependancies missing http://cpaste.info/29lx/. So i am trying to figure out if is a packaging issue or a cloud-init issue. (or other)
<meena> seems pretty badly packaged
<hkominos> so 19.2 is hosted upstream but cannot download it. 19.4 is packages and hosted by me. Dnf shows dependancy errros BUT all the binaries are actually visible by rpm -qa . So does that leave me with a badly build centos8 upstream image ?
<hkominos> I just feel that the 18.5 cloud init might be causing problmes
<meena> hkominos: you have more than one cloud-init installed?
<hkominos> no.
<hkominos> only 18.5 is installed in the upstream image. I am simply unable to upgrade to any other binary
<meena> i wouldn't exactly call a bunch of python files strewn over your filesystem a "binary"
<hkominos> ? what do you mean ?
<meena> hkominos: if cloud-init was a self-contained binary, it might be easier to update.
<meena> welcome back hkominos. i hope all your sorrows are taken care of, since we last met.
<hkominos> hahahahaha.
<hkominos> naaaaaah still things do not work. THat is why I turn to IRC for help
<meena> hkominos: how did you build 19.4?
<meena> how was the 19.2 package built?
<hkominos> of course I did not document all this :P. It looks like I used a spec file for the older 18.5 cloud init and just changed the binary that it pointed to. to build 19.2
<hkominos> http://cpaste.info/29m7/
<hkominos> like so
<meena> hkominos: my company proxy doesn't trust  cPaste.info
<hkominos> ok. attempt2
<hkominos> https://pastebin.com/hNCwe1Xk
<hkominos> perhaps ?
<hkominos> ok. so the spec specifies python requirements
<hkominos> that I do have. But it cant find them
<meena> hkominos: have you seen https://github.com/canonical/cloud-init/blob/master/packages/redhat/cloud-init.spec.in ?
<meena> missingâ¦ thingsâ¦ are replaced with: https://github.com/canonical/cloud-init/blob/master/packages/brpm
<meena> you can call it from the Makefile
<meena> make rpm DISTRO=redhat
<hkominos> first time i see this!
<hkominos> Will get on it right away. thx
<rharper> otubo: I suspect that if Ubuntu images only had network-manager we might see the same issue
<rharper> otubo: would appreciate your thoughts on https://github.com/canonical/cloud-init/pull/226
<blackboxsw> meena: thanks for being so active
<blackboxsw> it adds a lot to the cloud-init project
<hkominos> meena I will work on this tomorrow. But even with the spec file that you provided i get tricky issues. f.e. https://pastebin.com/tBgPrXBK
<hkominos> Do i need python 2.7 to build cloud init ?
<rharper> hkominos: for centos7, at this time, 2.7 is required, for python3 in centos8 etc, there's a branch with a modified spec file; that work has not yet landed, but you can get a look at those changes here: https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/368845
<hkominos> thx!
<AnhVoMSFT> rharper: just checking if there is anything else we need for this PR: https://github.com/canonical/cloud-init/pull/54
<powersj> AnhVoMSFT, I don't think there is - we need to loop back and get that as part of our next SRU, but Chad can correct me if I am mistaken
<rharper> AnhVoMSFT:  no, powersj is right; we can land that and it would get picked up in the next SRU
<AnhVoMSFT> thanks rharper powersj
<meena> blackboxsw: ð
<meena> btw, any of you seen ausfestivus issue? i was able to pin-point the exception. but not the reason why it's happening.
<ausfestivus> Do you sleep meena? :-p
<meena> ausfestivus: between 23:00 and hopefully 7â¦
<meena> but my baby daughter is still pretty unpredictable.
#cloud-init 2020-02-27
<otubo> rharper: I was going over irc logs and found out we agreed to do the NM dropin as a downstream fix, then we would have some time to work on the _postcmd fix on the sysconfig renderer. I'm gonna close my PR, then.
<otubo> rharper: I actually already have a draft using _postcmd, I'll be pushing my branch soon.
<DanyC> hi folks, i came across https://bugs.launchpad.net/cloud-init/+bug/1020695 and while i see it has never implemented, anyone has another suggestion on how i can achieve the same but in an early cloud-init stage - ie: not final stage (runcmd/ script-user) ?
<ubot5> Ubuntu bug 1020695 in cloud-init "Add variable for local IP address to /etc/hosts manager" [Low,Triaged]
<meena> otubo: cool.
<meena> DanyC: there's some ideas here in smoser's (unmerged) branch: https://code.launchpad.net/~smoser/cloud-init/lp1020695/+merge/163216
<meena> DanyC: i think i'd revive that bug / patch
<meena> DanyC: but, can you explain why you need it super early?
<DanyC> meena: sure thing i can. First let me give you the full picture (you might have seen i've asked various q recently) ;) . All i'm trying is to snapshot an EC2 and create an AMI from it with an application configured so i can later create multiple dev env from it. Now i have an application which doesn't cope very well with changing the IP and at the same time it does also look in /etc/hosts file (in addition t
<DanyC> o its own "cache").
<meena> ooff. I've seen thoseâ¦
<DanyC> and because of all that my final cloud-init stage doesn't kick/ run due to cloud-final.service not being up which is being held by my app service
<meena> (my opinion is, like always, a bit different. because i have a config management fetish, in particular puppet)
<meena> I'd keep /etc/hosts as created by cloud-init in the AMI, and ensure that the application is *not* auto-started (in the AMI)
<DanyC> so i can't use runcmd, i can't use script-user. I tried bootcmd but it doesn't seem to work if i'm trying to "update" the IP in the /etc/hosts file with the info from ec2metadata
<meena> then have cfg-mgmt run, fix up /etc/hosts, and whatever else the app needs. and only then enable and start the app.
<DanyC> hence my assumption that maybe ec2metadata is not available and so i was looking for s'thing else.
<DanyC> meena: but with a cfn-mgmt that will need to be run in user-data no? and if the final stage doesn't kick then i fail to see how it will help me
<meena> uuuhâ¦ there's a switch somewhere to disable ec2metadata after the first boot, actuallyâ¦
<DanyC> i don't need to be switched, i need to work in the bootcmd stage
<DanyC> *to be switched off
<meena> i know, it just occurred to me, that you're trying to access it, while it's maybe switched off.
<meena> anywayâ¦ let's collect all the things we know so far.
<meena> first up: what do you think of my idea of only enabling & starting the app once everything is in place?
<DanyC> meena: i wished i could do that, sadly i have other 10 services depending on the one which is holding cloud-init
<DanyC> the only option i see is to be able to update the /etc/hosts file with the current IP of the new EC2 and bounce the service so i can then let cloud-init other stages kicking in
<DanyC> but to do that it seems is harder than i initially thought, is a catch 22. Not to mention i can't change the silly app :facepalm
<meena> DanyC: then all 10 services are disabled and stopped until everything is fixed up.
<meena> What's the point of having them running, if they don't run correctly?
<meena> heck, you could even make it a systemd service that all of them depend on!
<meena> after networking, you run, update_etc_hosts.service, it fixes up /etc/hosts, and *then* all services can be started.
<meena> think outside the box :P and inside another box!
<DanyC> interesting, and i guess you saying i should have in the cloud_init_module stage a step to run before update_eth_hosts module ? or i've misunderstood you ?
<meena> i don't really know what the best way is to bring that service onto the AMI.
<meena> how are you bringing the other services onto the AMI?
<rharper> otubo: ah, ok;  I do still wonder why the ordering is not enough;  my reading of systemd documentation suggests that NM should not be started before cloud-init-local.service has run to completion;   but I don't have the journal of when you saw the failure.
<Goneri> blackboxsw, I think I've addressed all your comment here: https://github.com/canonical/cloud-init/pull/62
<Goneri> blackboxsw, up to date prebuilt images are available here: https://bsd-cloud-image.org/
<blackboxsw> Goneri: Thanks for the ping.  Out of curiosity, how often are the prebuilt images updating cloud-init?
<blackboxsw> as in, I wonder if that's a good point of reference which you control that could be referenced if people find bugs/issues on bsd.
<blackboxsw> once bsd changes are all integrated upstream in cloud-init I guess we could discuss that further
<Goneri> blackboxsw, my goal is to autobuild them as frequently as necessary (e.g: after every merge, or even every PR), but it's still a work in progress
<Goneri> for now, I still trigger the build manually.
<blackboxsw> rharper: we can land this branch now right? https://github.com/canonical/cloud-init/pull/54
 * blackboxsw is scrubbing PRs since I'm in the mood :)
<rharper> blackboxsw: yes
<rharper> it lands to ubuntu/xenial
<blackboxsw> ok will get that landed today
<blackboxsw> sorry Goneri, one more pull request landed on master I saw your force push :/
<blackboxsw> will wrap up review on that next
<Goneri> awesome :-)
<meena> blackboxsw: you can probably close some of mine
<blackboxsw> yeah, it's about time PRs don't age as well as fine wine.
<meena> btw, can someone who's better at python than me, explain why i did this: https://github.com/canonical/cloud-init/blob/master/cloudinit/util.py#L1824 and how can i do this (in a thread-safe way? without spawning / forking?) so that i just say: find me a libc.
<sarnold> what problem are you really trying to solve?
<meena> sarnold: make this code work on NetBSD, and on the next version of FreeBSD if it changes that .7
<sarnold> meena: aha; I think I'd try a loop over several potential libc pathnames, and populate that list with the paths to current and future libc libraries
<blackboxsw> Goneri: quick volley of comments {% if variant in ["freebsd", "netbsd"] %}
<blackboxsw> oops
<blackboxsw> Goneri: on https://github.com/canonical/cloud-init/pull/62 I mean
<blackboxsw> something concerns me a bit with the shifting cloudinitlocal to after networking in startup scripts as is diverges from upstream behavior and that could impact our future work
<blackboxsw> as we'd have to take into account that netbsd is different in this regard
<Goneri> blackboxsw, this is totally a mistake. I'm actually surprised it just works
<meena> i could use https://docs.python.org/3/library/ctypes.html#ctypes.util.find_library
<Goneri> I will fix the order. thanks blackboxsw for the review!
<blackboxsw> Goneri: yeah me too a bit, I would've thought it would have introduced a startup service broken dependency loop or something
<sarnold> meena: oh that looks a lot nicer
<meena> but i think i'm gonna have to read the actual code to see what that does on the systems before using it.
<blackboxsw> meena: have a url for me of the PR that you'd like looked at first?
<blackboxsw> to refresh my memory. I'll try to get a pass on it today
<meena> blackboxsw: nope.
<blackboxsw> heh, will grab one of 'em and work through it
<meena> blackboxsw: all i want is the networking stuff on the Mailinglist to get a response :P
<blackboxsw> meena: I'll try to get an update to that then. I think the direction that robjo is going with current prs (with flavors on sysconfig renders) is probably the best approach at the moment, but I think sysconfig renderer is the most contentious/dirty of our cases because suse and rhel differ so much in network config flag support. I'll try saying something smart about that on the mailinglist, what blocked me was
<blackboxsw> examples & suggestions there.
<lachesis> hi all, i'm having a problem with ssh_pwauth setting on DigitalOcean. I am shipping an image that has its own /etc/ssh/sshd_config file with `PasswordAuthentication no` already set, but also with a `MatchUsers` section that enables PasswordAuthentication for some specific users (for somewhat silly reasons that are hard to fix). however, cloud-init is doing a broad string replacement, so it's disabling ssh password auth even for that particul
<lachesis> ar user.
<lachesis> how can i prevent this? also, where can i open an issue about this bug?
<lachesis> i'd really like to mask that setting out in the image... i suppose i could do this by editing the /usr/lib/python3/dist-packages/cloudinit/config/cc_set_passwords.py file to just disable that check
<meena> blackboxsw: the examples & suggestions provided, or the ones missing?
<meena> blackboxsw: re PRs, i think this one can probably be closed: https://github.com/canonical/cloud-init/pull/69 from what i understand from rharper . and i'm wondering if we shouldn't revert the previous work i did there.
<lachesis> i tried putting `ssh_pwauth: unchanged` in `/etc/cloud/cloud.cfg` but that didn't help
<rharper> lachesis: could you file a bug and include the tarball from 'cloud-init collect-logs'  and any provided user-data ?   we can look into the issue and see what's going on
<blackboxsw> lachesis: you can file a bug here: https://bugs.launchpad.net/cloud-init/+filebug to give a bit more context about the problem.
<blackboxsw> sorry, would help if I hit enter
<lachesis> :) yeah i'll file there... i am not providing any user data but it is possible that DO is without me... let me see if i can track down where cloud-init fetches that, probably some magic 169 address
<rharper> lachesis: right, I see you were adding the config via /etc/cloud/cloud.cfg vs. user-data; that's good enough
<blackboxsw> lachesis: if you ever need to check what userdata and vendordata cloud-init sees:   `sudo cloud-init query userdata` or `sudo cloudinit query vendordata` or `sudo cloudinit query --all`
<lachesis> ok yeah it looks like ssh_pwauth is coming in from vendor data
<lachesis> bug report: https://bugs.launchpad.net/cloud-init/+bug/1865082
<ubot5> Ubuntu bug 1865082 in cloud-init "ssh_pwauth: no disables PasswordAuthentication in MatchUsers block as well as globally" [Undecided,New]
<rharper> blackboxsw: man, we really should have collect logs pull in  /etc/cloud/cloud.cfg, /etc/cloud/cloud.cfg.d/*.cfg ...
<blackboxsw> yeah we really should
<blackboxsw> it is a pain point when we have to debug/triage
<rharper> lachesis: replied; was this run from an instance with the /etc/clouc/cloud.cfg including the added 'ssh_pwauth: unchanged' ?
<blackboxsw> though we omitted it because it could contain sensitive info
<blackboxsw> but maybe we lump it into the user-data question in apport
<rharper> I mean, any of them could;  so can user-data to some degree
<rharper> right
<blackboxsw> yeah
<lachesis> negative, there was no cloud.cfg in this case. i can rebuild the image and regenerate with that set, but it'll take a few min
<rharper> lachesis: you can add that to the ecisting instance
<rharper> and then run: sudo cloud-init clean --logs --reboot
<lachesis> and reboot it?
<lachesis> will do
<rharper> that will run like "new instance"
<rharper> the code reads that it should exit out without calling the ssh_utils path which reads in sshd ;
<lachesis> hmm ok that restarted my box, and it hasn't come back up yet :/
<lachesis> it's a cow, not a pet, so i can just destroy and rebuild it, no big worries, but it will prevent me from getting the logs this time :)
<rharper> well, that's not nice of DO ...
<rharper> lachesis: so DO does provide a *lot* of vendor-data scripts; it's possible that they are including a ssh_pwauth: no  setting by default
<lachesis> they definitely are... i included the query --all result in the tar file in that bug
<rharper> in which case, that can override system config; which leaves you with having to fight them or disabling vendor-data in your image; so you can set system-config;
<lachesis> mm i see, ideally i'd just be able to disable that particular setting and/or that setting would be smart enough to avoid messing up my MatchUsers
<lachesis> i am pretty strongly tempted to patch the python file and be done with it lol
<rharper> heh
<rharper> 2020-02-27 20:50:04,362 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/f1/sem/config_set_passwords - wb: [644] 24 bytes
<rharper> 2020-02-27 20:50:04,362 - helpers.py[DEBUG]: Running config-set-passwords using lock (<FileLock using file '/var/lib/cloud/instances/f1/sem/config_set_passwords'>)
<rharper> 2020-02-27 20:50:04,363 - cc_set_passwords.py[DEBUG]: Leaving SSH config 'PasswordAuthentication' unchanged. ssh_pwauth=unchanged
<rharper> 2020-02-27 20:50:04,363 - handlers.py[DEBUG]: finish: modules-config/config-set-passwords: SUCCESS: config-set-passwords ran successfully
<rharper> lachesis: so that
<rharper> what I expect to see if you
<rharper> your value makes it into the combined cloud config
<lachesis> sry how do i get logs from that cloud-init clean run?
<blackboxsw> lachesis: logs live in /var/log/cloud-init.log
<lachesis> obviously, thx :)
<lachesis> 2020-02-27 20:47:12,316 - util.py[DEBUG]: Read 2964 bytes from /etc/ssh/sshd_config
<lachesis> 2020-02-27 20:47:12,317 - ssh_util.py[DEBUG]: line 97: option PasswordAuthentication already set to no
<lachesis> 2020-02-27 20:47:12,317 - ssh_util.py[DEBUG]: line 103: option PasswordAuthentication updated yes -> no
<lachesis> 2020-02-27 20:47:12,317 - util.py[DEBUG]: Writing to /etc/ssh/sshd_config - wb: [644] 2963 bytes
<blackboxsw> the cloud-init clean --logs  removes your old /var/log/cloud-init.log so it'll only contain the current boot
<lachesis> $ cat /etc/cloud/cloud.cfg
<lachesis> # The top level settings are used as module
<lachesis> # and system configuration.
<lachesis> ssh_pwauth: unchanged
<lachesis> wait is that a space?
<lachesis> doh
<lachesis> it's not a space, the _ just got lost somehow?
<lachesis> `ssh_pwauth: unchanged`
<lachesis> maybe my xchat font is borked... but there is an _ showing in vim
<lachesis> but i imagine the vendor-data just overrode my config there
<rharper> that's what I'm thinking
<lachesis> ok im gonna go with the dumb patch cc_set_passwords.py option :)
<lachesis> thanks for your help folks, i love an active open-source IRC channel :)
<rharper> lachesis: yw
<blackboxsw> rharper: just repushed  https://github.com/canonical/cloud-init/pull/214 with doc updates
<rharper> blackboxsw: cool, did you see my comments in the review re: SRU blocker text / system_info() json encoding potential issues ?
<blackboxsw> rharper: I had and responded to both
<rharper> k
 * rharper reviews 
<blackboxsw> I think SRU blocker is a no because we added new fields before across SRU boundary for platform/subplatform.
<rharper> and we mark the fields experiemental correct ?
<blackboxsw> rharper: I did in the ds in the base, but we can/should add those good thought
<blackboxsw> doing that now
<rharper> so we're not baking things in; though I suspect telling folks they can now use this in their jinja template might make it less useful if it will change on them
<rharper> blackboxsw: well, I do think things like system_info/variant, etc
<blackboxsw> that's true
<rharper> won't be going away
<rharper> so those don't have to be experiement, ie, they are fixed values at this point
<blackboxsw> right it's already been a requested  feature here once or twice
<rharper> we already have runtime code that looks at os.variant etc
<blackboxsw> rharper: correct, nothing changed there, just surfaced that key in instance data
<rharper> yeah; but once added and SRU'ed, it can't change without potential regressing user scripts
<blackboxsw> that's all part of stock util.system_info
<rharper> so, we should be happy with them;
<blackboxsw> right, we would be unable to change names once SRU'd
<blackboxsw> maybe it's worth bikeshedding on key names?
 * rharper was just reviewing them 
<blackboxsw> as we certainly don't  change v1 keys
<blackboxsw> the rest of the dict doc is up for changes in general as we don't promise that part won't change.... just v1 keys
<blackboxsw> as v1 is the generalized output
<blackboxsw> s/generalized/standardized
<rharper> the 'cfg' head is going to be free-form, no?  if someone modified their /etc/cloud/cloud.cfg (or added a sub file) then there's going to be whatever in there ...
<rharper> blackboxsw: reviewed
<blackboxsw> thanks rharper yeah, that cfg probably should be root-readonly
<blackboxsw> it could have just about anything
<fredlef1> I've been looking at the life cycle of instance_link.  I see it's mostly managed in Init::_reflect_cur_instance(), called in stage6, after a valid data source has been found. But it's also preemptively deleted in Init::_get_data_source(), before initiating a search for available data sources.
<fredlef1> The preemptive deletion was added in 2016 in 0x0964b42e5 (quickly check to see if the previous instance id is still valid). Does anyone remember what made the deletion outside of reflect_cur_instance() necessary/desirable?
<rharper> fredlef1: hi, I believe the goal is to not trust the on disk object cache (which is only present on datasources which implement check_instance_id());  unless we can confirm the current instance id is the same as what's on disk (either in cache, or via the symlink);  this deletion happens during cloud-init-local time;  on subsequent stages (init, config, final) they all use the trust flag
<rharper> fredlef1: do you have a particular bug you're looking at or something else?
<fredlef1> rharper: thanks. That's useful information. Unless I'm mistaken, the disk object cache is always created but it only gets used if check_instance_id() is implemented or if manual_cache_clean is set to True.
<fredlef1> rharper: I'm looking at ways to gracefully handle a reboot where the datasource is not available anymore
<rharper> fredlef1: yes, I believe that's true, we do write it but it won't be used unless the ds implements the function ; its disabled by default in base source class (cloudinit/sources/__init__.py
<rharper> fredlef1: this is manual-cache-clean: True
<fredlef1> rharper: not quite.  I don't want to reuse the cache if the datasource is available, so as not to miss updates/changes to user-data and network interfaces.
<fredlef1> Basically, if the datasource is available, we should always crawl it but if it is not, I want a way out
<rharper> interesting
<rharper> well
<fredlef1> I'm testing a patch that modifies _get_data_source forcefully load the cached data if we failed to find a valid data source and the cached datasource still matches the config.
<fredlef1> I should probably hide that behind a configuration option to make it more palatable
<fredlef1> Does that sound senseless ?
<rharper> not senseless;  I generally like the idea of using cached datasource object under the following criteria 1) the platform is the same, 2) the datasource is the same 3) metadata service is down;  4) and I think we can compare the object's ds.instance_id value to what's written to /var/lib/cloud/data/instance-id
<rharper> currently we won't re-use the object unless the source implements check_instance_id() ;  normally this is done via some non-network verification, some platforms encode instance-id in platorm data (like dmi system uuid etc);  the ec2 instance-id is not present in system info that I'm aware of;   but you could attempt to fetch instance id via metadata service; and return true and we'd use the object;  in your "metadata service is down scenario";  then
<rharper> you could do the other fallback checks I mentioned (1, 2, 3) and return True only if those hold
<rharper> this would restore the obj from cache
<rharper> beyond that, we'd need to sort through the other hits to imds, like the .network_config property
<fredlef1> rharper: I looked at implementing check_instance_id as we do have a way to get the instance id on newer instance types but it turns outs that implementing check_instance_id for the EC2 datasource would conflict with public documentation from AWS.  In fact, I'm rather planning to implement a DataSourceEc2::check_instance_id() that returns false has
<fredlef1> a place to document why it should not be implemented.
<rharper> ok,
<fredlef1> I'll keep your criteria for reusing the cached datasource object in mind. Thanks for that.
<rharper> that's reasonable;   so, then I suspect we'd need to modify _get_data() to handle this scenario
<rharper> but possibly deal with the removal of the object cache , which is why you're asking =)
#cloud-init 2020-02-28
<blackboxsw> just uploaded latest cloud-init into Ubuntu focal 20.04 [ubuntu/focal-proposed] cloud-init 20.1-5-g67c8e53c-0ubuntu1 (Accepted)
<xiaofwan> Hello guys. I'm building RPM package in Fedora 31 with mock. Some "python36-xxx" like dependencies blocked build. Is there special reason to use python36-xxx instead of python3-xxx? Thanks!
<rharper> xiaofwan: no;  specific reason; the spec file needs some updates;
<rharper> we have someone actively working on updating our spec to build with newer centos/fedora;
<rharper> xiaofwan: the general plan would be for py2 support to remain only on stable-19.4;  for py3, I suspect that centos7 would remain py2 as well, for master, py3, we can get it working with newer python;
<xiaofwan> rharper: make sense! Appreciate!
<rharper> xiaofwan: another thing to note, historically, the unittest/make check hasn't been run in the rpmbuild due to hard-to-find depdenencies in the rpm repositories;  I do have hope that rawhide or other newer Fedora repos have new enough versions to enable that;  if you end up with something working, please consider submitting a PR  https://github.com/canonical/cloud-init/pulls
<xiaofwan> rharper: Nice! I'm working on fixing 'make rpm/srpm' on my centos and fedora mock env. Of course, I'll send PR after that. :)
<rharper> \o/
<amansi26> Hi I am seeing a scenario where when I install cloudinit 19.1, then add 9 additional volume of 2 GB each (Toal 10 volume [9 additional + 1 boot volume]), cloud init local service is failing with http://paste.openstack.org/show/790124/
<amansi26> With 8 volume everything goes well
<amansi26> Can anyone suggest something?
<rharper> amansi26: it looks unrelated to storage add;  you're datasource is no longer present;  what datasource is normally present ?
<amansi26> ConfigDrive
<rharper> is the config drive still attached after adding those volumes (is this hotplug, or offline add, and then start back up) ?
<amansi26> Didn't get you
<amansi26> Then I tried to manually make the services up. It was failing . So changed /lib/systemd/system/cloud-init-local.service with http://paste.openstack.org/show/790125/. Then rebooted . All the services are running
<rharper> how do you attach your additional volumes;  do you shut down the instance; run some commands to attach the volumes; start the instance back up; or do you run commands while instance is running, then reboot ?
<amansi26> I add the instance in running state itself
<rharper> it's failing because it could not find your config drive
<rharper> is it possible that while adding the new volumes this removed the config drive ?
<amansi26> no
<rharper> well, are you sure ?
<rharper> you only showed the log indicating that cloudinit did not find a datasource
<amansi26> If that will be the case it should fail for adding 8 volumes also
<rharper> if you use config drive, it uses either 'CONFIG-2' or 'config-2'  to find this on block devices
<amansi26> how can I check that?
<rharper> cloud-init runs blkid -c /dev/null -o export
<amansi26> rharper: aah.. you got it write. datasource is getting removed and setting it to none
<rharper> is that expected (the removal) ?
<rharper> is this Openstack or IBM CLoud ?
<rharper> I know that IBM Cloud has some first boot/second boot transitions w.r.t config drive, IIRC
<amansi26> no The removal is not expected
<amansi26> IBM Cloud
<rharper> well, it may be expected
<amansi26> So you are suggsting 8 is the limit?
<rharper> https://github.com/canonical/cloud-init/blob/master/cloudinit/sources/DataSourceIBMCloud.py
<rharper> no
<rharper> just that the config drive "goes" away some times due to how IBM CLoud is provision
<rharper> if you read the datasource docs there; it's "complicated"
<amansi26> Was IBM Cloud DataSource is recently introduced?
<rharper> no, it's been here for a few years
<rharper> initial release was  March 2018
<amansi26> Quite new then
<rharper> relatively speaking, yes
<rharper> vs. say OpenStack
<amansi26> I will try using IBM Cloud DataSource replacing configDrive
<rharper> I'm confused
<rharper> either you're on IBM cloud and we detect that we are on it;  or you're on an OpenStack cloud that's not IBM CLoud
<amansi26> As a Datasource I am using ConfigDrive
<rharper> cloud-init should detect whether you're on IBM CLoud (with a config drive) vs.  just a ConfidDrive on Openstack
<rharper> so if you're seeing ConfigDrive instead of IBMCloud; then either there's a bug in the detection, or there's been a change to IBM CLoud such that we no longer detect it properly
<Goneri> @blackboxsw, ok, I was wrong yesterday regarding the initrd. That's something I acutally did on purpose to be able to run the 4 cloud-init calls in the correct order.
<Goneri> @blackboxsw, and I can hardly do something different. I think the situation will evolve if someone starts a port of cloud-init for NetBSD.
<Goneri> @blackboxsw, but I don't think this is a serious problem.
<rharper> ahosmanMSFT: Ill take a look, thanks
<ahosmanMSFT> rharper: thanks, I updated the PR
<rharper> ahosmanMSFT: cool!
<johnsonshi> Is there a way to build a deb package without running the unit tests? I am testing Azure telemetry, so I need to have an intentionally-buggy cloud-init to test our telemetry systems. I intentionally added a raise exception statement which will obviously cause provisioning failure. But I cannot build a deb package as it runs the unit tests and obviously fails.
<johnsonshi> Whenever `make deb` is run, it runs ./packages/bddeb. Looking at the file, I have no idea how it works and been spending the past few hours trying to find where the unit tests are actually invoked during debian package build.
<johnsonshi> So in order to build a buggy cloud-init deb package, I would need to disable unit testing during deb build. Thanks! :)
<sarnold> I don't think I've seen any standardized way to pass in build flags to disable tests
<sarnold> (I can't recall seeing non-standardized ways either, but that feels likely that someone has escape hatches..)
<sarnold> johnsonshi: is the package in question using debhelper? https://manpages.debian.org/jessie/debhelper/dh_auto_test.1.en.html  "If the DEB_BUILD_OPTIONS environment variable contains nocheck, no tests will be performed. "
#cloud-init 2020-02-29
<blackboxsw> sarnold thanks. johnsonshi yes that is correct, you can run DEB_BUILD_OPTIONS=nocheck make deb  to avoid running tests on deb creation in cloud-init
<ethi1ca4l> hello
<ethi1ca4l> please, I need help with cloud-init
<ethi1ca4l> I want to create image for DigitalOcean with debian 8 but If I install cloud-init in debian 8 so I don`t see DigitalOcean in config....I work long time with this but no successful
