#cloud-init 2014-08-25
<smoser> harmw, merge proposal comments . see you tomorrow.
<harmw> smoser: thanks, I think/hope I understand the issues now :p
<vbernat> Hey!
<vbernat> I am trying to boot a cloud-init-enabled Ubuntu image
<vbernat> The Precise one.
<vbernat> I don't have a DHCP on the network, so I need to provide one in the meta-data
<vbernat> I have an ISO image with /meta-data and /user-data
<vbernat> but it doesn't seem to be mounted
<vbernat> [    1.087935] EXT4-fs (vda1): re-mounted. Opts: (null)
<vbernat> cloud-init start-local running: Mon, 25 Aug 2014 14:27:15 +0000. up 3.40 seconds
<vbernat> no instance data found in start-local
<vbernat> mountall: Event failed
<vbernat> cloud-init-nonet waiting 120 seconds for a network device.
<vbernat> cloud-init-nonet gave up waiting for a network device.
<vbernat>  
<vbernat> how could I debug that?
<vbernat> It seems that the DataSourceNoCloud can only be found once we have network...
<smoser> vbernat, by default, even if it *did* find it... it would not "claim" it.
<smoser> as the default operating mode for NoCloud is 'net'.
<vbernat> smoser: what's the easier way to change that?
<smoser> but then still the network insertion is < wonderful.
<smoser> cloud-init needs work to be able to take network information from a datasource and do that well.
<smoser> the easiest thing to do (sadly) is open the image and insert the /etc/network/interfaces file that you want.
<smoser> or run a dhcp server. which... also seems sane to me :)
<vbernat> smoser: thanks, modifying the image is not that hard (guestfish)
<smoser> vbernat, if you trust the image, its easier than guestfish
<smoser> http://ubuntu-smoser.blogspot.com/2014/08/mount-image-callback-easily-modify.html
<vbernat> smoser: looking at the source code more closely, if I put dsmode: local in meta-data, wouldn't that work?
<smoser> maybe.
<smoser> it is intended to work.
<vbernat> hummm, seems to work
<smoser> but cloud-init editing /etc/network/interfaces while the system is coming up is possibly not guaranteed.
<vbernat> but doesn't make the network up just after that
<smoser> it could be racy essentially.
<vbernat> yeah
<devicenull> hmm, 
<devicenull> I've been running into "WARNING! Your environment specifies an invalid locale."
<devicenull> and it gives a couple invalid apt commands for fixing it
<devicenull> namely 'sudo apt-get install language-pack-en'
<JayF> hey smoser :) I work with pquerna and will be working to upstream some of that work. I'm going to start off with bumping the version of openstack configdrive that cloud-init reads (so it gets the newer-enough version to see vendor_data.json)
#cloud-init 2014-08-26
<harlowja> JayF hey, thanks, that should be a simple adjustment i think
<JayF> Yeah, I have an idea of what needs to be done. Haven't made much progress today other than reading the CLA :( and going to learn how to bzr
<harlowja> ya, good ole bzr 
<harlowja> smoser when we moving to git ;)
<smoser> JayF, thats great. there are some outright bugs in it that i want to fix in short order, and get them back to 14.04 also.
<smoser> so help is definitely welcome.
<smoser> JayF, bzr is roughly equivalent really.
<smoser> bar branch lp:cloud-init trunk
<smoser> cd trunk
<smoser> bzr commit -m "foo"
<smoser> bzr push lp:~jayf/cloud-init/my-fixes
<smoser> there is some doc in the tree that shows basically that work flow.
<smoser> harlowja_away, or harmw do you have any icehouse clouds up that have latest stable release ?
<smoser> or juno 
<smoser> https://bugs.launchpad.net/nova/+bug/1361357
<harmw> Im running icehouse, yes
<harmw> don't know if I'm suffering from that though, since my cloud sucks :p
<harmw> hm, I did notice it took ages to crawl the metadata.. just didn't think much of it at the time
<harmw> Ill see if I can look/confirm this tonight when I'm done here at $work
<smoser> harmw, i'm not sure i'm right. but something recently in our upgrade from 2014.1.1 to 2014.1.2 regressed that bad.
<smoser> i really hate pypi. tried to run devstack, and transient failures related to 
<smoser> DistributionNotFound: No distributions at all found for xstatic-rickshaw>=1.4.6.2 (from horizon==2014.2.0.dev366.g987c83a)
<harlowja> smoser think there is icehouse somewhere, although we don't use the metadata service :-P
<smoser> well you smell
<harlowja> lol
<harmw> ah, I was wondering who's responsible for murdering my nose
<harmw> that explains it
<smoser> harlowja, ba.
<harlowja> ??
<harlowja> i didn't do it, lol
<smoser> pretty sure read_metadata_service is double loading everything it loads
<harmw> instant backup! +1
<harlowja> ya, instant backup
<harlowja> what if the bits get corrupted man
<harmw> the horror
<harlowja> exactly, i save u
<smoser> thank you harlowja 
<harlowja> np
<harlowja> np
<smoser> :)
<smoser> you're redundant on read *and* write
<smoser> man. thats good.
<harlowja> lol
<harmw> smoser: I wnt cirros/ci to print MTU values some day, agreed?
<smoser> ok. 
<JayF> smoser: just now getting around to seeing your messages, so there's no step to change branches before the commit with bzr? the only thing that makes a 'branch' (as I'd call it in git) is when you push your commits to another remote ref?
<JayF> smoser: sorry if I'm mixing the terms :)
<harmw> smoser: checkout my re.compile goodness :>
<harmw> I wonder who ever wrote test_generic.py
<JayF> Should I expect tests to pass against master? make test pylint pep8 as the hacking.rst file says to do shows a ton of failures
<JayF> it looks like there may be some unmocked stuff expecting certain physical attributes (like there's a test that appears to be running blkid, unmocked)
<JayF> Hmm. even just pylint/pep8 isn't passing...
<JayF> smoser: assuming you'd accept a pair of commits to fix pep8 and pylint?
<smoser> JayF, pylint just sucks.
<smoser> pep8 does too really.
<smoser> i really, really hate that they arbitrarily change what they think is good.
<smoser> at one point (precise) ./tools/run-pep8 and ./tools/run-pylint preduced no warnings
<JayF> there are ways to ignore checks you don't like, or opt into only certain ones
<smoser> yeah, but they change the defaults
<smoser> and add new crap
<smoser> and make it the default
<JayF> the issues I've seen all seem fairly reasonable so far, but that's all opinion
<smoser> but i'm open to anything that gets a static check across pretty much any distro
<smoser> i'd love that.
<JayF> So does that mean the answer is no you wouldn't or yes you would but you wouldn't reccomend trying
<JayF> b/c I'm already about a third of the way through, haha
<smoser> yes, i'd take that, and very happily if you told me that it wasn't going to start having new things complain in 6 months
<smoser> just because
<smoser> (and i'm ok to ditch pylint too)
<smoser> it seems pyflakes is much more popular
<JayF> I'll make pep8 pass as it sits on my box, remove pylint from the Makefile and the hacking.rst, and that should be a reasonable starT?
<smoser> sure. what is your pep8 version?
<smoser> i've 1.5.6
<JayF> $ pep8 --version
<JayF> 1.5.7
<JayF> freshly installed in my venv using pip
<JayF> that might actually be a way to get static results. lock the version of pep8
<JayF> that way you bump the version, you also fix pep8 issues created by bumping version (or make pep8 can ignore the new ones)
<harmw> smoser: 
<harmw> AssertionError: '/etc/rc.conf' not in {'/etc/resolv.conf': <tests.unittests.test_distros.test_netconfig.WriteBuffer object at 0x807298a10>}
<harmw> that's what I got on that netconfig test
<smoser> JayF, yeah. i agree. peip and venv is a solution.
<JayF> smoser: https://code.launchpad.net/~jason-oldos/cloud-init/pep8-fixed/+merge/232289 how does that look? Pretty sure I bzr'd correctly :)
<harlowja> smoser where did u see the double-read?
<harlowja> in loading openstack metadata?
<smoser> harlowja, i htink is it basically does a full get in a stat
<smoser> for does_exist.
<smoser> let me find
<harlowja> kk
<harlowja> hmmm
<smoser> yeah, i'm pretty sure that _path_exists
<smoser> is doing a get
<smoser> and throwing away 
<harlowja> hmmm
<harlowja> ya, i believe thats cause openstack doesn't support head requests for the metadata service, at least i think thats true from what i remeber
<smoser> right. maybe we just need to cheat and re-use or something?
<smoser> honestly, i am only aware of this because something is bad wrong
<smoser> in openstack
<harlowja> bad wrong in openstack
<harlowja> haha
<smoser> on ec2, you crawl the whole datasource in 0.07 seconds
<harlowja> smoser sure, i can add re-use in
<smoser> in openstack, each get causes you ~.5 or something
<harlowja> lol
<harlowja> :-/
<smoser> which is clearly stupid
<harlowja> def
<harlowja> clearly something
<harlowja> lol
<harlowja> brb
<harlowja> alright smoser  adding caching
<harlowja> how can openstack metadata take 0.5 sec per request :-/
<smoser> JayF, i'll take that MP for sure. but i need you to sign the contributors agreement.
<JayF> smoser: just did
<JayF> literally about 60s ago finished it
<JayF> had the tab open since yetserday
<smoser> you put my name on contact ?
<JayF> 'smoser / cloud-init' is what I put in that field
<smoser> deal. thanks.
<JayF> because I wasn't exactly sure what it wanted :)
<smoser> good enough.
<JayF> so if you're OK with that, I'm going to do my other changes based on that branch to ensure I don't break pep8 again :)
<JayF> this will be the fixes for upgrading the configdrive version + parsing vendor_data.json properly
<harlowja> smoser https://code.launchpad.net/~harlowja/cloud-init/os-caching/+merge/232303
<JayF> It worries me a little that I wrote https://code.launchpad.net/~jason-oldos/cloud-init/upgrade-configdrive and it didn't break tests
<JayF> heh
<harlowja> deep breaths 
<harlowja> thou shall not worry
<JayF> https://code.launchpad.net/~jason-oldos/cloud-init/upgrade-configdrive/+merge/232308 
<JayF> harlowja: Your name is /very/ familiar to me. Have we worked together on anything before?
<harlowja> unsure
<harlowja> lol
<harlowja> JayF where are u from :-P
<JayF> I'm thinking opscode/chef?
<harlowja> negative
<JayF> I work on rackspace.com/onmetal / Openstack Ironic
<JayF> before that did devops for a slew of different Rackspace products
<JayF> I used to be JasonF on IRC
<harlowja> kk, might have seen u before then, i was the one of the yahoo hosts of the midcycle meetup at yahoo before this
<harlowja> before the last meetup
<harlowja> if u went to that one, u had to sign in with me, haha
<harlowja> gatekeeper harlow
<harlowja> lol
<harlowja> JayF the code that mocks is @ http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/tests/unittests/test_datasource/test_openstack.py#L83 ; from what i remember we aren't simulating a full version compliant metadata service, but if u want to make that better, feel free
<JayF> Aha! Totally where I know you from then. That's the first meetup we were at, when 'teeth-agent' became 'ironic-python-agent' in Sunnyvale
<JayF> Well that's for metadata service
<JayF> I'm working on ConfigDrive
<harlowja> same thing
<JayF> we don't run a metadata service (slow or otherwise)
<harlowja> :-P
<JayF> hmm. I will look and see if I can make those tests better
<harlowja> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/tests/unittests/test_datasource/test_configdrive.py#L67 then
<harlowja> they both use the same code, just read from different places
<JayF> I wonder if there's a similar patch to mine for the metadata service
<harlowja> one http:// the other file://
<JayF> to add support for the newer version of data and the vendor_data.json
<harlowja> not afaik
<harlowja> i haven't made one, ha
<JayF> well what I'm saying is, one might be needed
<JayF> I just don't have a great way to test it
<harlowja> gotcha
<harlowja> could be
<harlowja> btw, i'm interested why doesn't the onmetal stuff use the metadata service?
<smoser> JayF, merging that soon. i did a bunch more cleanup of pylint remnants
<harlowja> isn't config-drive much harder to make work on baremetal ?
<harlowja> if u can share....
<JayF> Metadata service in the real world is not easy for $network_reasons
<JayF> I don't understand them all specifically, but smart people assure me it's difficult
<harlowja> right, sure
<JayF> for configdrive, we patched cloud-init a while back to not exclude partitions when looking for configdrive
<JayF> so we just write down an iso partition at the end of the disk that is properly labelled and contains a configdrive
<JayF> and that's been working golden
<harlowja> ah
<harlowja> is that gonna be the way that most people using ironic do it?
<harlowja> or just onmetal folks
<JayF> It's the only way IPA supports right now
<JayF> and that's upstreamed
<JayF> well, wait
<JayF> I don't know if IPA + configdrive is upstreamed
<JayF> because lots of features blocked on getting the ironic-nova driver in
<JayF> we have a stack of things running in OnMetal that will land quickly when K opens
<JayF> configdrive, long-running-agents, decom
<harlowja> cool
<JayF> what I'm working on with cloud-init is support for the new proposed nova-neutron JSON file with networking information
<harlowja> gotcha, ya, sounds faimilar
<JayF> we have a local patch (https://github.com/pquerna/cloud-init/pull/1) that we're using to build configs
<JayF> because bonding with vlans on top isn't supported in cloud-init yet...
<JayF> so my hope is that you guys will let us upstream this version, that uses network info opportunistically if it's in vendor_data.json
<JayF> and hopefully when K releases, and the 'real' JSON support lands, we'll port the stuff we're doing over to the proper upstream openstack format
<JayF> I'd strongly prefer us not have to hold patches, on cloud-init especially :)
<harlowja> cool
<harlowja> JayF https://review.openstack.org/#/c/85673 u saw that i hope?
<harlowja> ^ was a way to make the openstack network info less ubuntu-like
<harlowja> and other stuff
<harlowja> not sure if that one got superseded or died or other
<JayF> Josh sits across from me :)
<harlowja> kk
<JayF> he wrote our vendor_data.json patch that writes, basically that exact info, out to our vendor_data.json
<harlowja> cool
<JayF> we're using it right now with any onmetal images other than debian
<JayF> and works really well
<harlowja> nice so plan is to still have https://review.openstack.org/#/c/85673 someday
<JayF> in K
<harlowja> by 2050
<harlowja> ah
<harlowja> kk
<harlowja> less than 2050 then
<JayF> yeah, so I'd *really* like to get our patches integrated with you guys even before then
<JayF> just looking for that format under a key in vendor_data.json
<JayF> then when the 'real' support is finalized, we can integrate that into the configdrive bits and go use it
<JayF> I figure y'all will be quite glad to not have the /etc/network/interfaces file as a pseudo-api anymore?
<harlowja> +1 from me, ha
<smoser> harmw, around ?
<smoser> can we fix your test failure ?
<smoser> harlowja, what do you think about htat cache ?
<smoser> i can see caching being a pain at some point.
<smoser> if the files in /files/ are big
<JayF> I personally have seen people inject, using cloud-init, full rpms (of cloud-init, actually, it's pretty insane and recursive)
<JoshNang> gah i hope the network json spec gets merged before 2050
<harlowja> wtf
<harlowja> lol
<smoser> well, the goal is to be able to do such silly things.
<harlowja> ok, i can reduce it to an existence cache only
<smoser> well, then we'd pull it again.
<smoser> which would be what i was annoyed by
<smoser> i'm just asking.. do you think its worth it.
<harlowja> ah
<smoser> really we need to fix the MD
<smoser> to not suck
<harlowja> ya, i'd rather be able to like use a HEAD request :-P
<harlowja> as far as worth it, hmm
<harlowja> smoser other option is to have the cache exist say in /var/lib/cloud/data/cache 
<harlowja> and when a item is read, write it there and first check there
<JayF> harlowja: -1
<harlowja> k
<JayF> harlowja: what about diskless nodes?
<JayF> that may only have a small ramdisk for writable /
<smoser> harlowja, yeah, thats true.
<smoser> that would seem not insane. 
<harlowja> except for the i only have a floppy as a disk node
<harlowja> 1.44 mb ftw
<smoser> hm.. i dont know.
<harlowja> or fix the os metadata service to actually respond to head requests
<JayF> Why not do the optimization where it belongs? File a bug in Openstack to get HEAD added to metadata service, and a bug about how freakin' slow it is, then let them get faster instead of oyu :)
<harlowja> and then i can see if i can find the code that i had that used that before i found out it didn't exist, lol
<smoser> JayF, well, it should be reasonably quick
<JayF> should is a funny word :)
<smoser> it does cache and hold the cache
<smoser> https://bugs.launchpad.net/nova/+bug/1361357
<smoser> but something seriously regressed recently.
<harlowja> kk, u opened one, thx smoser 
<smoser> (my diagnosis in the summary is wrong, but on that same cloud before upgrade  my small amount of logs still show 3 seconds and such to crawl the MD)
<smoser> i dont know that head is all that big of a deal really.
<smoser> harlowja, what are we using 'exists' for ?
<smoser> i think the MD is fairly well designed to list what is in it from the entry point
<smoser> right?
<harlowja> so a couple things, checking version support
<harlowja> and then checking files exist before reading (Required or not)
<harlowja> i can probably cut some of these out though
<harlowja> let me give that a shot
<smoser> the version support is listed
<smoser> it takes a query
<smoser> but that list of versions should be returned
<harlowja> k, let me see if i can optimize a few of these away
<JayF> https://code.launchpad.net/~jason-oldos/cloud-init/upgrade-configdrive/+merge/232312 should be good to go, I'll look at improving tests but I'll do that in another merge req since this one still passes tests :x
<smoser> JayF, that is wrong .. 
<smoser> + # TODO(pquerna): this is so wrong.
<smoser> lets fix that a differet way.
<JayF> heh :)
<JayF> How do you suggest?
<smoser> read it in as a string and keep the string.
<JayF> I left it be for now because I can assure you that vendor_data.json parsing works with it as written (primarily because we're using it, right now)
<JayF> hm. is there an equivalent to git commit --amend in bzr? or how to edit a commit?
<smoser> its just a different mentality.
<smoser> you dont.
<smoser> you fix it
<smoser> and commit
<smoser> you could if you wanted 'bzr uncommit'
<smoser> and bzr commit 
<smoser> with the same message (and i do do that)
<JayF> you'd prefer keep the intermediate commit or uncommit/recommit?
<JayF> I don't care, you tell me :)
<harlowja> smoser alright, https://code.launchpad.net/~harlowja/cloud-init/smart-read/+merge/232316
<harlowja> check that out, bb, food
<smoser> JayF, i'm fine with either way really.
<smoser> (network back)
<smoser> bzr just more accepts that humans make mistakes and does not expect you to try to hide them.
<smoser> then, you merge in and have a merge commit and all that history stays around.
<JayF> smoser: Well, git workflows differ :) I squash now primarily because openstack/gerrit insist upon it
<JayF> for bzr and cloud-init I'll do w/e else
<smoser> yes, i know.
<JayF> smoser: trying to grok what you meant by "read it in as a string and keep the string."
<JayF> smoser: you want me to parse self.vendordata_raw somewhere else?
<smoser> let me lok
<JayF> iirc pq said when he did it before, it was assumed to be yaml and caused tracebacks
<JayF> but at this point I'm still mainly porting our patch so I know less about it than I should
<smoser> JayF, start here
<smoser>  http://paste.ubuntu.com/8153171/
<smoser> then fix up the other code to not expect a dict there.
<smoser> hm..
<smoser> it is .json. so its clearly json
<JayF> yeah, I don't get why there's any value in parsing it later when it's explicitly a json file
<smoser> JayF, can we just do
<smoser> its always a dict, right?
<smoser> so at https://code.launchpad.net/~jason-oldos/cloud-init/upgrade-configdrive/+merge/232312
<smoser> line 13 is pointless check
<smoser> so then can we just do:
<smoser> if 'cloud-init' in vd:
<JayF> I don't know if it's always a dict or not
<smoser>  self.vendordata_raw = vd['cloud-init']
<JayF> or let me correct
<JayF> I don't know the object results.get('vendor_data') is returning
<smoser> ah. ok. yeah.
<smoser> hm..
<smoser> well, its the json.load()
<smoser> right
<smoser> ?
<smoser> err.. load_json
<smoser> i'm sorry
<smoser> i have to run. i will be in tomorrow and we can sort this out more.
<JayF> I'll dig in and clean it up
<JayF> smoser: rfr -> https://code.launchpad.net/~jason-oldos/cloud-init/upgrade-configdrive/+merge/232312
#cloud-init 2014-08-27
<harmw> smoser: I'm to sucky a python programmer to fix the test on my own, so a little help or example would be helpfull yes :)
<smoser> JayF, so you're the only one using vendordata on openstack to my knowledge. at least with cloud-init.
<smoser> JayF, https://etherpad.openstack.org/p/cloud-init-vendor-data
<smoser> harmw, i'll see if i can't help ith that test today.  harlowja wrote the other ones there.  cloud-init suffers badly from having too many test case mechanisms.
<siert> smoser: I did an update on #1246485 - not sure if the suggested solution from alexius is perfect, but it works in my situation. also affects latest version...
<smoser> siert, just fyi, man page for 'hostname' recommends against using 'hostname -f'
<smoser> :)
<siert> smoser: hm, ok. but the RHEL install guide is slightly more ambiguous - "either as a fqdn or as a short host name"
<smoser> siert,  thank you for loking at this.
<smoser> while it seems straight forward, i think its probably as much as you'd hope.
<smoser> my experience is that any time i touch 'hostname' or '/etc/hosts', something breaks.
<smoser>  https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1052664
<siert> the /etc/hostname should always have the short name only. on rhel, when there is no hostname defined in /etc/sysconfig/network, then /etc/hostname is used as a fallback, to set the systems hostname.
<siert> debian/ubuntu have the short name in /etc/hostname and the rest is read from /etc/hosts.
<smoser> hm.
<smoser> having 'hostname' return a fqdn seems broken to me.
<siert> ubuntu 14.04 returns hostname version 3.15. rhel based systems: net-tools 1.60 and hostname 1.100 (2001-04-14)
<siert> maybe the difference is in the version of 'hostname', let me see... :s
<smoser> well, on ubuntu, it returns whatever you set it to.
<smoser>  hostname
<smoser> brickies
<smoser> er..
<smoser> $ hostname
<smoser> brickies
<smoser> $ sudo hostname brickies.foobar.example.com
<smoser> $ sudo hostname
<smoser> sudo: unable to resolve host brickies.foobar.example.com
<smoser> brickies.foobar.example.com
<smoser> $ sudo hostname brickies
<smoser> sudo: unable to resolve host brickies.foobar.example.com
<smoser> $ hostname
<smoser> brickies
<smoser> and that initially gets set from the content of /etc/hostname
<siert> yeah, by /etc/init/hostname.conf (hostname -b -F)
<smoser> right.
<smoser> http://manpages.ubuntu.com/manpages/trusty/en/man1/hostname.1.html
<smoser> "historically this file was supposed to only contain the hostname and not the full canonical FQDN. ..."
<smoser> siert, i know it sounds like i'm paying too much attention here, but i've just been bit many times.
<smoser> basically my experience is that anything you do breaks something.
<siert> thanks for you time, I gtg. will look into it tomorrow, maybe there's another way of getting rhel to behave the same as ubuntu when it comes to hostnames.
<JayF> smoser: Is that a bad thing? Good thing? Heh
<JayF> We've used vendor_data.json internally for stuff for a while, just now integrating it with cloud-init
<smoser> JayF, well, i would not want to break anyone's usage
<smoser> and i was proposing changes in consumpiton of it
<smoser> even though that consumption i spretty broken at the moment.
<JayF> smoser: not sure I grok what you're trying to suggest in that etherpad
<JayF> Have you had feature requests along those lines from other folks using vendor_data?
<smoser> that is how vendor data works
<smoser> in other datasources
<smoser> and actually , if there is 'cloud-init' in the network metadata service on openstack, then it probably works like that now.
<smoser> (well, for the 'dict' case)
<smoser> ie: 
<smoser>  {'cloud-init': 'this-is-processed-same-as-user-data'}
<JayF> Hmm. I honestly don't like that format, but you say that's what's already being used for other data sources (including, I presume, metadata service vendor_data?)
<smoser> what dont you like about it?
<JayF> let me make sure I fully understand
<JayF> so I get what you're saying for the dict case
<JayF> the cloud-config-archive stuff looks strange, and I'd be curious as to how that fits into a 'list' as far as the json is concerned?
<JayF> and for the string case I'm just not sure what source data would lead json.loads() to simply return a string
<JayF> you know; here's actually the primary reason I'd want vendor_data treated somewhat differently
<JayF> with regards to what we're using it for; network configs; I'd not want that to be something a user could override or influence with user data
<JayF> makes me wonder if today a user can inject a /etc/network/interfaces in order to give themselves broken networking in the existing status quo 
<smoser> as far as what would give a string:
<smoser> $ python -c 'import json, pprint; pprint.pprint(json.dumps("hi mom"))'
<smoser> '"hi mom"'
<JayF> gotcha
<JayF> in Ruby that would throw an exception :)
<JayF> ...I think?
<JayF> yeah for sure --> JSON::ParserError: 746: unexpected token at 'hello world'
<smoser> i thought it would in python earlier this morning
<JayF> so that makes a little more sense
<smoser> i dotn knwo that it does really.
<smoser> why would you not be able to represent a string as json
<JayF> I'm saying your etherpad makes more sense :)
<JayF> knowing it doesn't bail in python
<smoser> ah.
<smoser> so anyway.
<JayF> So for my use case, I think with what you've proposed
<JayF> hmm
<JayF> so vd['cloud_init'] could /not/ be a dict?
<JayF> if that's the case, it'd 100% not work for what I'd like to do
<JayF> well, I say 'like to do' but am-actually-doing in a branch
<JayF> I think there are cases for sure where I'd want to send multiple data formats into my vendor_data
<JayF> like where we want to, I guess in a sense introduce the network json format proposed for nova into vendor_data
<JayF> in the format you lay out I don't think that's doable
<smoser> hm..
<smoser> no i think it is 
<JayF> howso?
<smoser> cloud-init is very forgiving on usre-data. it pays attention to a fairly small set of explicit things, and ignores all the others.
<smoser> and this would be the same.
<JayF> but you said in the etherpad
<JayF> vd['cloud_init'] could be a list or a string
<JayF> if the goal is to have json encoded network information
<JayF> and cloud_init only reads from that key
<JayF> then the only way to get json into that key would be to serialize it into a string... which seems obtuse
<smoser> ah. i se. so you want to put that into 'network-config' and have cloud-init read it from ther.e
<smoser> i am ok with that. 
<JayF> or into vd['cloud_init'][network_config]
<smoser> wlel, i dont know.
<smoser> so 2 ways we could do that
<smoser> a.) 'network_config' is not explicitly for cloud-init. so it would be at
<smoser>    vendor_data['network-config'
<smoser>   as it will eventually (we hope) be a sibling of meta-data.json (or possibly inside meta-data.json)
<smoser> ie, its not explicitly cloud-init.
<JayF> yes exactly
<JayF> that makes sense
<JayF> so today have cloud-init look for vd['network-config'] and md['network-config'] and do things with them
<smoser> b.) cloud-init handle a part  of type 'network-config'
<smoser>  the way it handles cloud-config now.
<JayF> probably have some versioning concept so whatever differences come out between our 'prerelease' vendor-data version of it and the final openstack release will be able to be smoothly migrated
<smoser>  ie, either explicit mime type, or '#network-config' (startswith)
<smoser> yeah. i agree on vendor.
<JayF> where would the #network-config or mimetype come from if it's just a json dict?
<JayF> I'd say just look for a key, that key may change in the future, and also have it specify a version
<smoser> well, in cloud-config-archive, yo ucan supply mimetype
<JayF> hell, have the initial vendor_data key be something like rackspace_network_config to completely ensure we don't eat the namespace
<smoser> and in just a string, if it startswith('#network-config'), then it gets handled as that.
<JayF> I'd strongly prefer not having a metaformat in this case, given it's json data I don't want to have #network-config followed by a json doc
<smoser> hm..
<JayF> because all we've done is add a magic string at top that breaks all other json parsing tools
<smoser> yeah, i think that is fine.
<smoser> oh. to address one other thing above.
<smoser> yes, the usre can break things.
<smoser> rackspace seems very much to want to magically stop users from being able to break things.
<smoser> but i dont aare.
<smoser> don't care.
<smoser> from my perspective, the user is in charge.
<JayF> I'd rather software be in charge and prevent the user from doing bad, broken things they probably didn't intend
<JayF> but it's fine, I get why you think about it that way :)
<smoser> if they say "put this file in /etc/network/interfaces" i'm not going to say "i'm smarter than you".
<smoser> same as 'rm -rf /etc'
<JayF> I get it, I don't agree but that's a-okay
<smoser> i understand why you all want to protect users. but there is only so much you can do.
<smoser> fwiw, vendor data is disable-able by user entirely.
<JayF> haha
<smoser> by design. if they want to feed user-data to turn it off, that is fine.
<JayF> if someone does that on an onmetal server
<JayF> they're going to be paying a lot of money for a server that doesn't have internet
<JayF> hah
<JayF> So path forward:
<JayF> cloud-init stuff handled the way you specified
<JayF> look for a separate key in vd for network configs
<smoser> i put a bit more in that pad
<smoser> yeah
<smoser> interesting.
<JayF> right now that key is 'network_info'
<JayF> fwiw
<smoser> thats fine.
<smoser> so i think we should stuff YYYYMMDD in there just right away
<smoser> when / if it gets into openstack, it will live at metadata-url/YYYYMMDD/network_info.json
<smoser> and that YYYYMMDD would fall out there.
<JayF> not injected into meta_data.json?
<smoser> i dont have strong feelings there.
 * JayF needs to read that spec again
<smoser> it certainly could live in meta-data.json
<smoser> in which case the YYYYMMDD would certainly fall out (as it would be provided by the openstack version)
<smoser> do you think we need to build the version info into network_config
<smoser> that was the real question
<smoser> when it appears in vendordata
<JayF> I think versioning the key is 'fine'
<JayF> I dislike all options I can think of
<JayF> and that seems the easiest to implement :)
<JayF> gimme a sec and I'll give you a good example of a network json format
<JayF> fresh off a configdrive
<smoser> versioning the key.
<smoser> as in network_config_YYYYMMDD ?
<JayF> aye
<smoser> i don tknow. i tihnk you're optimizing because you dont think it iwll live there long
<smoser> :)
<JayF> I mean, the reason I'm working with you now is so we don't hold these patches for long
<smoser> yeah. 
<JayF> would be great if cloud-init support was mostly there when it goes into openstack master
<JayF> and as a bonus we don't have to hold a patch
<JayF> although the hard part is building custom cloud-inits for a ton of operating systems :)
<smoser> so i just think this is cleaner... its more explicit:
<smoser> for version in vendor_data_json.get('network_config', {}):
<smoser> versus:
<smoser> for ver in [k.split("_")[2] for k in vendor_data_json if k.startswith("network_config_"):
<JayF> so you'd do
<smoser> (like in the pad)
<JayF> network-info: { "20140101": {network info here } }
<smoser> right.
<JayF> smoser: so this is what we're putting down looks like right now --> https://gist.github.com/jayofdoom/b035067523defec7fb53
<JayF> smoser: so the patch I posted earlier that pquerna did is currently, on ubuntu, fedora, and centos, already parsing and generating configs against that
<JayF> which is pretty nice :)
<smoser> ?
<JayF> note that I redacted specifics of the server I'm using and knocked out other vendor_data stuff we use
<smoser> really? where is that work by pquerna ?
<JayF> https://github.com/pquerna/cloud-init/pull/1
<JayF> that's what we're running right now, that patch on top of latest release (.7.5 iirc?)
<smoser> i'd not seen that. 
<JayF> I thought I linked it in here earlier
<JayF> my apologies :)
<smoser> you probably did. but if i remembered or correctly processed every thing people say to me
<smoser> then i'd be pretty awesome
<smoser> :)
<JayF> heh, that's what logs and git/bzr are for, right?
<smoser> so this supports bonds
<smoser> and vlans
<smoser> and mtu
<JayF> Which is why we had to do that :)
<JayF> the interfaces file parsing didn't handle it
<JayF> and we wanted to use the network json instead
<smoser> this is awesome.
<JayF> if you spin up a rackspace onmetal server, you get network like that (10gbit bonded, vlans handed down over that bond)
<smoser> i was disapointed by onmental minimum $500/month
<JayF> I actually think there's a bug, because ubuntu trusty and debian anything can't both use the same /etc/network/interfaces file to achieve that setup
<smoser> kind of turns me off from trying it :)
<JayF> we charge by the hour
<smoser> but monthly minimum.
<pquerna> by the minute
<smoser> no?
<pquerna> there is a $50 minimum for support fee
<pquerna> (unforunately)
<smoser> dwonder why i thought it was $500.
<pquerna> well
<smoser> maybe that was something else.
<pquerna> there is a $500 option
<harlowja> JayF so we can get free onmetal
<harlowja> thx much, lol
<harlowja> then i don't need to request it here @ yahoo, lol
 * harlowja getting vms is easier than getting hardware :-P
<pquerna> managed infrastructure -> $50, managed operations -> $500
<pquerna> managed ops = rackers will configure / operate software, etc, etc.
<pquerna> managed infra is what most... of 'us' types want.
<harlowja> def
<harlowja> none of that stuff that makes people do things
<pquerna> heh
<smoser> well http://www.rackspace.com/cloud/servers/onmetal/
<smoser> still confuses me
<smoser> so i dont think i simply mis-remembered.
 * JayF brb shortly
<smoser> raw infrastructure: $500/server/mo.
<smoser> but i guess you're saying really that is :
<pquerna> so, thats billed per-minute
<pquerna> those are just monthly prices to be 'nice'
<pquerna> (le sigh)
<smoser>  $50 + minutes * (500/30/24/60)
<pquerna> right
<pquerna> $0.68493/hr or $0.0114155/min
<smoser> pquerna, ipv6 support in that patch ?
<pquerna> no :(
<smoser> ok. so see, you're not perfect
<smoser> :)
<smoser> this is really neat though.
<pquerna> we don't do ipv6 right now for.... sigh. switches.
<pquerna> one day i'll be able to run ubuntu on my switches. it'll be great :)
<pquerna> JoshNang: actually, i guess we should make the nova/neutron changes to support v6
<smoser> ok. so i'm out for thursday -> tuesday.
<pquerna> JoshNang: even if our networks don't support it right now
<smoser> yeah, there is work to do on that front too
<pquerna> JoshNang: and do the cloud-init work
<smoser> dreamhost has it 
<smoser> but your network config format *has* to support it
<JoshNang> pquerna: oh definitely.
<pquerna> +1
<smoser> and cloud-init should generically be able to support it too.
<pquerna> JoshNang: then at least we can write test cases on both sides
<JoshNang> when we originally drew up the json format, we had ipv6 in there. i just don't think it made it into the spec explicitly
<smoser> on dreamcompute you only get a public ipv6 by default.
<pquerna> nice
<pquerna> the future
<JoshNang> awesome
<smoser> yeah, but i had to find somewhere to ssh hop through to get to my instances there :)
<smoser> ok.
<harlowja> JayF u see my comment on https://code.launchpad.net/~jason-oldos/cloud-init/upgrade-configdrive/+merge/232312 ?
<harlowja> i can't quite remember if the new inline comment feature sends emails or not, lol
<smoser> i'm out the rest of the week. after US EOD. and out monday too. 
<smoser> but i really would lik to get this stuff in quick
<smoser> harlowja, you able to patch harmw's test case for him ?
<smoser> https://code.launchpad.net/~harmw/cloud-init/freebsd/+merge/231024
<harlowja> hmmm, lets see
<smoser> pquerna, or JayF can you get me as much data as possible on what you have for network config ?
<pquerna> (i'm also out starting Friday until... September 24th, wedding/honeymoon, so JayF/Josh will be on this)
<harlowja> wtf, why u guys going on vacation
<harlowja> no good, lol
<smoser> pquerna, well congratulations on that. sir.
<harlowja> u found a female, impassible
<harlowja> lol
<harmw> harlowja al rescate!
<harlowja> smoser do u have someone looking into https://bugs.launchpad.net/nova/+bug/1361357 or tbd?
<smoser> i've got someone trying to verify if it regressed between 2014.1.1 and 2014.1.2
<smoser> but thats been slow going.
<smoser> i *think* it is neutron related.
<harlowja> kk
<harlowja> hmmm
<smoser> i as happy to see the ML post
<smoser> that seemingly its not just us
<harlowja> ya, not just u 
<harlowja> thats always good
<smoser> we really need to do something there.
<smoser> even the 3 seconds that it took before it regressed is laughable
<smoser> on amazon, the ~ 20 requests it takes to crawl their MD service takes ~ 0.07 seconds 
<harlowja> :-/
<harlowja> ya, it makes no sense
<harlowja> the time it takes for such a simple service should be like < 0.01 seconds
<harlowja> no reason for it not to me
<harlowja> *not to be
<pquerna> weird. i thought the MD service put it all in memcache anyways. hrm.
<JayF> harlowja: it didn't email me
<smoser> pquerna, yeah, its definitely supposed to be.
<pquerna> we don't yet deploy MD service, but its on the short list of things i want.
<pquerna> its just silly that we don't
<pquerna> so... lets stop doing silly things.
<harlowja> less silly is good
<harlowja> but u don't want to remove all the silly
<harlowja> thats no fun
<harmw> smoser: whre should I run that script of yours, inside cirros? (regarding the nova bug)
<harmw> or would just any vm suffice
<harmw> since I'm running openstack-nova-api-2014.1.1-2.el6.noarch
<harmw> old comparison code:  0.103947877884
<harmw> new comparision code:  4.36138916016
<harlowja> :-/
<smoser> harmw, what script did you run ? 
<smoser> the wget ?
<smoser> i dont recal.
<harlowja> how is 4.2 more seconds even possible
<harlowja> thats insane
<smoser> ec2metadata is a command on ubuntu or cirros
<smoser> harmw, so what are your bencharms there ?
<harmw> hhe
<smoser> bencharms.
<harmw> this is from a centos7 instance in my cloud :P
<smoser> bench marks
<smoser> but "old" and "new"
<harmw> I took the script at https://bugs.launchpad.net/nova/+bug/1361357
<harmw> and it printed that
<smoser> oh. nah. 
<smoser> that was red-herring.
<smoser> i should just change the description of that bug.
<smoser> i'll do that now
<harmw> hhehe
<harmw> well, if you have some benchmark I could run
<smoser> the string comparison code is massively slower than it was 
<harmw> just let me know
<smoser> but that is actually by design
<smoser> harmw, 
<smoser> time ec2metadata
<smoser> just run that. 
<harmw> ah, and in which package should I find that binary?
<harlowja> smoser i don't get how even crappy string comparison can slow down by that much
<harlowja> its not like massive strings are being compared right
<harlowja> where massive is in the MB range
<harlowja> its gotta be something else than that
<smoser> right.
<smoser> it was redherring
<harlowja> ya, does the metadata server now use the conductor and object model
<harlowja> https://github.com/openstack/nova/blob/master/nova/api/metadata/base.py#L130 
<harlowja> that thing adds on alot of new RPC requests and such
<JayF> harlowja: https://code.launchpad.net/~jason-oldos/cloud-init/upgrade-configdrive/+merge/232312 I don't see any comments on this at all
<harlowja> JayF ah, it saves unsaved comment
<harlowja> thats not cool
<harlowja> odd
<harlowja> oh, u have to press the save comment button
<harlowja> weird
<harlowja> JayF ok try now
<JayF> totally got an email this time
<JayF> harlowja: I guess I don't see how that's cleaner in this case; it's more lines of code to do the same thing?
<JayF> harlowja: if there's a difference in functionality help me understand please 
<harmw> smoser: you've got a direct link on ec2metadata binary?
<harmw> or is it included with c-i?
<harlowja> JayF version can be passed in as a function argument, so the code i was doing was just ensuring that u aren't going to rescan the same version twice
<harlowja> aka someone passes in openstack.OS_GRIZZLY 
<JayF> see I almost was tempted
<JayF> to not even want to use that kwarg anymore
<JayF> because I didn't see any instances where it was being used
<harlowja> JayF kk, then lets drop it?
<harlowja> seems fair to me 
<JayF> harlowja: is there a better way to get the merge req to update other than deleting and making a new one (in lp?)
<harlowja> afaik u can just do bzr ci, and push and it the merge request will update
<JayF> bzr ci?
<harlowja> bzr commit i guess
<harlowja> guess thats my shorthand
<smoser> harmw, sorry. not in cloud-init. sorry behind on reading.
<smoser> here....
<JayF> I've been doing bzr commit -> bzr push lp:~jason-oldos/cloud-init/upgrade-configdrive
<smoser> this will work if you have sane cloud-init
<smoser> http://paste.ubuntu.com/8151758/
<JayF> and either I'm not patient enough to see the merge req update or it never does
<harlowja> JayF ya, that should be fine
<JayF> me being impatient very likely
<harlowja> JayF it should show 'commit updating' or somethign usually
<harlowja> http://paste.ubuntu.com/8161416/ is what i sent to a co-worker who i think will get involved with cloud-init
<harlowja> the basics
<smoser> harlowja, well, the string comparison is really by design. like explicitly complex to avoid magic optimizaiton that would give informatin away.
<JayF> s/ci/commit/
<JayF> and that matches what Iw as doing
<harlowja> smoser ya, i've put something similar in another part of code that uses hmac stuff
<harmw> smoser: I think my cloud sux :P
<harmw> crawl of openstack took 17.666 seconds
<harmw> crawl of ec2 took 14.452 seconds
<harlowja> harmw lol
<harlowja> super-speed
<harlowja> https://github.com/stackforge/osprofiler/commit/034987f083cfa1aff2347a4f (for the other place i've used that hmac comparison func)
<harlowja> * https://docs.python.org/2/library/hmac.html#hmac.compare_digest
<harlowja> anyways
<harmw> hehe, neutron-server taking 40% cpu :p I should realy upgrade this Atom server
<harlowja> :(
<smoser> harlowja, so something is odd in your 'smart-read' branch
<harlowja> could be
<harlowja> lol
<smoser> time nosetests tests/unittests/test_datasource/test_openstack.py
<smoser> not good. it takes like ... (still going)
<harlowja> hmmm
<harlowja> kk, let me see what i can find out
<smoser> 65 seconds
<smoser> but returns success (which is worse)
<harlowja> probably retrying i think
<smoser> harlowja, i was going to do this:
<smoser>  http://paste.ubuntu.com/8161712/
<smoser> to remove the duplication in find_working_version... put the differencein _avialable_versions
<harlowja> kk
<harlowja> that seems ok
<smoser> harlowja, wrt the openstakc metadata service and conductor...
<smoser> hopefully we cache the InstanceMetaData()
<smoser> and reuse it.
<smoser> so yeah, you'd hit that once, and that would suck, but it is supposed to be good for 15 seconds after tat.
<harlowja> sure, even without caching  there is no way it should take that long :-P
<harlowja> i mean i can render most of yahoo in +10 seconds
<smoser> https://github.com/openstack/nova/blob/master/nova/api/metadata/handler.py#L38
<harlowja> ya, i'd almost want to disable that cache for figuring out the root cause
<smoser> https://review.openstack.org/#/c/102212/
<smoser> hm.. wonder if that is in 2014.1.2
<harlowja> smoser alright, https://code.launchpad.net/~harlowja/cloud-init/smart-read/+merge/232316 updated
<smoser> harlowja, hey. ok. i just merged that.
<smoser> harlowja, if you could fix (or help harm) fix his bsd bug
<smoser> that'd be good
<smoser> one of his tests fails on linux
<smoser> and then you are welcome to push that to trunk
 * smoser feels really bad for always taking so long to do things for harm.
<harlowja> smoser checking
<harlowja> will help boss
<harlowja> lol
<harlowja> harmw alright, let me see if i can get your thingy adjusted, will get u a patch i guess
<harlowja_> harmw if u get a chance, http://paste.ubuntu.com/8163345/
<harlowja_> applying that to your freebsd makes it work for me
<harlowja_> hopefully fine with u, should just be apply to apply it and resubmit your branch
#cloud-init 2014-08-28
<harmw> harlowja_away: thanks, I see you fiddled with the distro code as well :p Ill try to merge it tonight, thanks
<harmw> my problem was just the unittest, but this is fine as well :)
<vbernat> Hey! Is there a way to disable cloud-init after first boot?
<vbernat> Once the host is provisioned, I don't expect cloud-init to modify anything
<harmw> vbernat: c-i will configure the lot at first boot, and will check at consequtive boots if it needs to do something
<harmw> so there is no real reason to disable it
<vbernat> harmw: I am using NoCloud provider and if it is not able to detect its CDROM, the machine will try to get meta-data using the EC2 way
<harmw> ah, and you umount the cd drive causing the instance to fallback to ec2
<vbernat> I don't want cloud-init to fight with puppet on each boot, I would feel more confident if it wasn't there.
<harmw> which slows it down
<vbernat> yes
<harmw> hm, I believe there is some option to have it just stop 
<vbernat> I have tried an empty /etc/cloud/cloud.cfg or an almost empty one, no luck
<harmw> hm, specifying an empty datasource perhaps
<vbernat> I have tried datasource_list: [] but no luck
<harmw> ah ok
<vbernat> in /etc/cloud/cloud.cfg
<harmw> yea
<harmw> smoser: this would be where you fly into the scene and make some clever one-liner :p
<vbernat> I could just uninstall cloud-init but I don't know how to uninstall it while ensuring it runs to completion
<vbernat> If I uninstall it while it is setting up the machine, I think this will break badly :)
<harmw> nah, there's probably an easy solution here
<vbernat> using ds= on the kernel command line doesn't make a change
<harmw> I read that was broken, not sure in which version though
<vbernat> how do I know what is run during "cloud-init final"?
<vbernat> I could disable cloud-init init scripts from here if I could run a command
<harmw> you could ofcourse just pass in some ordinary cmd in your user-data, like systemd disable cloud-init
<harmw> but I'm not confortable with that, so smoser or harlowja_away should just wake up - they'll know :)
<vbernat> yes, that would be a solution but then I may skip some tests
<vbernat> but if I understand cloud.cfg correctly, runcmd is run during "config"
<vbernat> but I could disable cloud-init in the user scripts since they are run in final module
<vbernat> I'll try that
<vbernat> harmw: well, that seems to work reliably
<harmw> :)
<harlowja_> i wake up
<harlowja_> harmw ya fiddled with distro code, it was chopping off some of the values, so fixed that :)
<harlowja_> i guess thats what tests are for (to find that, ha)
<harlowja_> harmw ie, re.compile(r'^(\w+)="?(\w+)"?') was chopping things in quotes that were like spaces and such
<harlowja_> like "a   b" 
<harlowja_> anyways
<harmw> ah lol
<harmw> hehe
<harmw> stupid
<harmw> *me
<harlowja_> all good
<harmw> ill go ahead and apply it in a whil
<harlowja_> kk
<harmw> AttributeError: 'TestNetCfgDistro' object has no attribute 'patchUtils'
<harmw> would that be my pthon env missing some packages?
<harlowja_> hmmm
<harlowja_> patchUtils is like a cloud-init thing i made
<harlowja_> so probably not env
<harmw> oh ok
<harlowja_> are u deriving from the right class
<harlowja_> patchUtils is in http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/tests/unittests/helpers.py#L161
<harlowja_> so u have to derive tests from that sucker if u want to use that function
<harmw> oopsiedoopsie
<harmw> I had some old lines of code in thre
<harmw> it works now :)
<harlowja_> kk
<harmw> ok, done
<harmw> pushed
<harlowja_> cool
<harmw> smoser: ping
<harlowja_> harmw '10 files modified (has conflicts)' :-/
<harmw> the hell
<harlowja_> ya, must be a conflict somewhere
<harmw> and I did a merge from trunk like 5 days ago
<JayF> Conflicts could easily be caused by my pep8 fixing patch
<JayF> that went in two days ago
<JayF> that touched a lot of files
<harmw> aha!
<harmw> well, since I'm way to lazy Ill just let harlowja_ sort it  :>
<harlowja_> :-/
<harlowja_> lol
<harlowja_> i'm just gonna let smoser  handle it
<harlowja_> lol
<harmw> :P
<harlowja_> responsibilty deferred :-P
<harmw> +1
<harlowja_> pass the buck ftw
<harlowja_> lol
<harlowja_> harmw btw, a guy from yahoo (not sean as u know) i think gonna get involved with the freebsd 
<harlowja_> guy pinged me yesterday
<harmw> going by the name of hiren?
<harlowja_> https://www.linkedin.com/in/hirenpanchasara 
<harlowja_> damn
<harlowja_> how u know all this stuff
<harlowja_> lol
<harmw> yea well
<harmw> :p
<harlowja_> is the freebsd people like a group of like 10 people that all know each other
<harlowja_> and live in a communue?
<harlowja_> freebsd commune
<harlowja_> lol
<harmw> lol
<harlowja_> harmw added a few more review comments to that freebsd one
<harlowja_> beasite?
<harlowja_> wtf
<harlowja_> lol
<harmw> ah lets see
<harlowja_> *beastie
<harlowja_> lol
<hiren_> :-)
<harmw> haha yea, given the linux world is using fedora/ubuntu
<harlowja_> hiren_ yo yo
<hiren_> howdy
<harmw> ah look who's there
<harlowja_> ah i get it, reference to the freebsd demon picture thingy
<harlowja_> magic!
<harmw> no shit sherlock
<harlowja_> so hiren_  i think https://code.launchpad.net/~harmw/cloud-init/freebsd/+merge/231024 is just going through final adjustments
<harlowja_> if u want to look it over, please do (i'm no freebsd expert that might miss something there)
<hiren_> k, let me
<harlowja_> hiren_ but after that goes in, we can start building freebsd images and seeing what to do next :-P
<harlowja_> harmw did u need anything else put in freebsd ports (maybe hiren_ knows people or something that can make this happen faster?)
<harmw> nope, the dependencies are taken care of already
<harmw> if this is merged and version bumped Ill submit the c-i port
<harlowja_> cool
<hiren_> yep! Super excited to see this finally happening :-)
<hiren_> harmw: yes do let me know if you need any help from ports people. 
<harmw> nah, excitement will kick in once smoser version bumps :>
<hiren_> heh
<harmw> ok, though last time that was no problem
<hiren_> cool.
<harmw> marino talked me through some bits, got everything in just a few days
<harlowja_> mario?
<harlowja_> the mario?
<harmw> read
<harmw> damn
<harlowja_> ;)
<harlowja_> lol
<harmw> :P
<hiren_> ha.
<hiren_> harmw: cool.
<harmw> harlowja_: diff comments, wtf is up with that :p
<harmw> I can see a conflict, with the <<< and such
<harlowja_> ya, thats bzr/launchpad for u
<harlowja_> i've never understood why it shows commits like that
<harlowja_> lol
<harlowja_> *commits that have conflicts
<harmw> hmk
<harlowja_> +<<<<<<< TREE
<harlowja_> 9	def _resize_ufs(mount_point, devpth):
<harlowja_> 10	return ('growfs', devpth)
<harlowja_> 11	+=======
<harlowja_> 12	+def _resize_ufs(mount_point, devpth): # pylint: disable=W0613
<harlowja_> 13	+ return ('growfs', '-y', devpth)
<harlowja_> 14	+>>>>>>> MERGE-SOURCE
<harlowja_> that one right
<harlowja_> pylint stuff ;)
<harmw> yea
<harmw> but thats all?
<harlowja_> seems like it
<harmw> since it puts the rest of the patch there as well
<harmw> which kinda draws away my attention :>
<harmw> (though I get it, since its still part of the full merger
<harmw> )
<harlowja_> i'm gonna say its all smoser  fault
<harlowja_> lol
<harmw> ofcourse
<harmw> like you always do :P
<harlowja_> :)
<kamikazemicrowav> Iâm using the âset RANDOM passwordâ feature of cloud-init and I just noticed that its being logged to the console when its performed which means its getting logged to â/var/log/boot.logâ. Is there a way to make cloud-init not do that?
#cloud-init 2014-08-29
<harmw> smoser: get to it :P
<JamesHarrison> Looking at http://cloudinit.readthedocs.org/en/latest/topics/examples.html#adjust-mount-points-mounted I can't figure out how I'd specify a NFS share, or if that's even supported - any clues on this? ie, mounting 192.168.1.1:/path on /some-mountpoint with the nfs vfstype
<ndonegan> JamesHarrison: https://github.com/number5/cloud-init/blob/26ea5244585c25a396be139336948154222675f4/doc/examples/cloud-config-mount-points.txt
<ndonegan> Basically, you put it in the same format as you'd put it in /etc/fstab, just that each field is and element in a list.
<JamesHarrison> yep, got it working in the end, had an unrelated syntax error which was confusing me, thanks!
<harmw> harlowja_: you mind waking up smoser ?
<harmw> so now they both got me on /ignore, great
#cloud-init 2015-08-24
<minfrin> Hi all. Does anyone have a workable example of the X-Merge-Type header? I have read the docs at http://cloudinit.readthedocs.org/en/latest/topics/merging.html a number of times, however I am struggling to understand what to put in the header. What I am after is two cloud-clonfigs in a mutipart, and the second multipart must be in addition to (not in replacement of) the first.
<minfrin> My understanding is the "list()+dict()+str()" gives "replace" behaviour.
<minfrin> Just tried with "X-Merge-Type: list(extend)+dict()+str()", and now the first part works (previously it didn't) and the second part is ignored.
<Odd_Bloke> smoser: I'm going to be looking at https://bugs.launchpad.net/cloud-init/+bug/1460715 this week; do you have any high-level thoughts/guidance?
<claudiupopa> Odd_Bloke: hi! did we reach an agreement regarding stackforge / openstack renaming? What was decided?
<Odd_Bloke> claudiupopa: I think we decided that we can just ignore it for now.
<Odd_Bloke> claudiupopa: We may have to rename at some point, but that point is not now.
<Odd_Bloke> (And, being unspecified, may well never come)
<j^2> hi!
#cloud-init 2015-08-25
<smoser> Odd_Bloke, i think "use sectors".
<Odd_Bloke> smoser: Cool, that was my plan. :)
<smoser> the one annoyance is that i'm not 100% certain that sectors is always 512 bytes
<Odd_Bloke> smoser: It looks like the reporting changes have broken the MAAS DataSource: https://launchpadlibrarian.net/215516689/cloud-init.log
<Odd_Bloke> smoser: That's attached to https://bugs.launchpad.net/cloud-init/+bug/1488507
<smoser> Odd_Bloke, right . looking at that now.
<smoser> fudge
<smoser> thats a packaging thing
<smoser> i think
<gamename> hi guys.  How can I tell in the main system log if cloud-init ran?
<gamename> or can I?
<kwadronaut> 'main'? usually you will want to check something like /var/log/cloud-init.log
<kwadronaut> and we're not all 'guys' ;-)
<gamename> kwadronaut: ok, but what if I can't access /var/log/cloud-* ??  Are there indicators in the main syslog to at least indicate cloud-init's ran?
<gamename> kwadronaut: The context is aws ec2 instances.  Getting the console log is easy.  Nothing else is.
<kwadronaut> boot log might have something, haven't looked at ec2 in a long time. stick around, someone else might know.
<gamename> tks
<smoser> gamename, "main system log"
<smoser> systemctl ?
<smoser> or upstart
<gamename> smoser: ok
<smoser> i'm asking.
<smoser> which were you using ?
<gamename> smoser: neither.  I have an aws ec2 instance which I think may not have run the cloud init code.  But, I can only get access to the console log of the aws instance. Nothing more.  I'm locked out.  Was looking for hints as to how I can start debugging.
<gamename> smoser: One option on aws is to get the console log.  That works.  Was hoping for some indication in it ...
<smoser> can you post the console log ?
<smoser> if its ubuntu you should see cloud-init writing things to it.
<gamename> smoser: yes.  Standby ...
<gamename> smoser: No, its CentOS 7
<smoser> i dont have a lot of experience there, testing a quick
<smoser> heres a centos 6.6 with cloud-init: http://paste.ubuntu.com/12194874/
<smoser> see cloud-init in that log
<gamename> smoser: Here's the log https://gist.github.com/gamename/0407dccb5af2d35c6061
<smoser> and heres a 7 http://paste.ubuntu.com/12194878/
<smoser> it'd seem likely that your networking didnt come pu
<smoser> up
<smoser> i guess.
<gamename> smoser: yup.  Based on your examples, I think that's the case .
<smoser> the centos 7 log i showed is from a http://cloud.centos.org/centos/7/images/ image.
<smoser> on openstack. i can't vouch for the correct (or useful) installation cloud-init on arbitrary ec2 image
<gamename> smoser: Can they be imported into AWS as an AMI?
<gamename> smoser: understood about vouching for the image.
<smoser> maybe
<smoser> i ran that one on a openstack.
<smoser> i really dont know.
<smoser> you could use ubuntu :)
<smoser> and our images "just work"
<gamename> smoser: Sorry, I'm stuck with CentOS 7.  Lotsa reasons too icky to get into here.   Well, that's a start. Thanks.
<kwadronaut> or debian, and ask in #debian-cloud ;-)
<gamename> kwadronaut: Hmmmm ... there is a #centos-cloud channel I think.  Worth a query to them too...
<kwadronaut> gamename: i'm happy in my debian-ubuntu comfort zone.
<smoser> gamename, did you modify that image at all ?
<smoser> or is it a "stock" image from some provider
<smoser> if stock, then i'd ask the provider of the image
#cloud-init 2015-08-26
<jaksi07c8> hi
<jaksi07c8> I'm using cloud-init with the cloud image of ubuntu trusty
<jaksi07c8> and sometimes it fails to install the packages (packages: section of the cloud-config file)
<jaksi07c8> it actually fails to configure them, running `dpkg --configure -a` after cloud-init has finished solves the problem
<jaksi07c8> what could be the cause of this? the environment is the same, and sometimes it works, sometimes it doesn't.
<Odd_Bloke> jaksi07c8: Does /var/log/cloud-init.log have anything related?
<Odd_Bloke> jaksi07c8: (And is this an official Ubuntu image, or a derivative?)
<jaksi07c8> Odd_Bloke: it's the official image
<jaksi07c8> no, i didn't find anything in the logs
<Odd_Bloke> jaksi07c8: Is it specific packages or can you reproduce with any packages?
<jaksi07c8> Odd_Bloke: there's a fixed set of packages to be installed
<jaksi07c8> sometimes all of them are installed
<jaksi07c8> sometimes none, sometimes just a few
<jaksi07c8> these are actually our own packages from our own apt repository, so the environment is even more fixed
<Odd_Bloke> jaksi07c8: And these packages definitely reliably install when not using cloud-init?
<jaksi07c8> is the standard error also present in the logs?
<Odd_Bloke> Perhaps.
<Odd_Bloke> jaksi07c8: Could you paste your cloud-init.log somewhere?
<Odd_Bloke> smoser: https://code.launchpad.net/~daniel-thewatkins/cloud-init/shim_fixes/+merge/269199 fixes some problems that Azure have found with the snappy Azure stuff.
<jaksi07c8> Odd_Bloke: could it be that cloud-init runs apt-get install in the background, not waiting for it to finish?
<jaksi07c8> because what I'm experiencing now is that if I shut down the VM right after cloud-init finishes (the login prompt appears), the packages are not installed
<jaksi07c8> if I wait a little, everything is fine
<Odd_Bloke> jaksi07c8: cloud-init won't block the login prompt appearing.
<jaksi07c8> mhm... I'm actually replacing an old script which did a preseed-based install of a regular ubuntu server image. The method I used to detect when the installer finishes was that I checked if port 22 was available on the host (sshd has started)
<jaksi07c8> is there a way to tell if cloud-init has finished everything, and the machine can be safely shut down?
<Odd_Bloke> jaksi07c8: Yeah, that won't work with cloud-init.  See http://cloudinit.readthedocs.org/en/latest/topics/examples.html#reboot-poweroff-when-finished
<Odd_Bloke> jaksi07c8: If you need to detect whether cloud-init has finished running while the machine is still up, then you can examine /var/lib/cloud/data/status.json.
<jaksi07c8> ah, great, thanks a lot for your help!
<Odd_Bloke> :)
<Odd_Bloke> claudiupopa: smoser: Are we going to do the meeting (in 53 minutes; don't panic) in here or in the hangout?
<claudiupopa> I would prefer it to be here, I guess there isn't much to talk for today.
<smoser> Odd_Bloke, here is fine the.
<smoser> then
 * smoser reboots
<Odd_Bloke> smoser: claudiupopa: Meeting time!
<smoser> woot
<claudiupopa> great.
<smoser> https://review.openstack.org/#/q/project:stackforge/cloud-init+status:open,n,z
<smoser> so the one merge open right now is Cladiu's
<Odd_Bloke> I think that's waiting for another +2 and a workflow +1.
<Odd_Bloke> Unless I have, once again, forgotten how this process works. :;
<smoser> ok. that doesn't have a +1 from Odd_Bloke
<Odd_Bloke> *:p
<smoser> so i was just going to ask that.
<Odd_Bloke> I do remember reviewing this most recent version, not sure why I don't have a review on it.
<Odd_Bloke> Let me log in and see if I have anything I haven't saved.
<smoser> Odd_Bloke, ok. and i'll look at it a bit after this.
<smoser> so lets say that is done, and we should have smoser's comments shortly or a +2/+1
<smoser> Odd_Bloke, did you do anything for main ?
<claudiupopa> Okay, that sounds good for me.
 * smoser has been on holiday for 12 days (very nice)
<Odd_Bloke> I was at LinuxCon/Plumbers last week, so I haven't done anything.
<smoser> oh yeah. had forgotten.
<Odd_Bloke> Yep, I'd reviewed everything but hadn't actually +1'd.
<Odd_Bloke> +1'd now.
<smoser> ok then. so if you contineu to look at that, th'ad be good.
<smoser> i have been doing some reporting work in cloud-init 0.7.x and using it in curtin also, much to Odd_Bloke's dismay
<Odd_Bloke> Yep.
<Odd_Bloke> ^_^
<Odd_Bloke> That yep was to the previous line. :p
<smoser> i will have that submitted to review in 2.0 sometime soon i hope.
<smoser> then we have https://trello.com/b/HoPNdiTI/cloud-init-development-roadmap
<smoser> Odd_Bloke, should take main executable
<smoser> and i'll add  a task for 'webhooks reporter'
<smatzek> I was on vacation the week Cladiu's review came out and it got lost in the pile of email. I'll try to get a good review of it this week but don't hold up the merge on my part.
<claudiupopa> I'll probably take the new config module architecture, in order to have soonish a work-in-progress cloudinit v2 that does something.
<smoser> claudiupopa, cool
<Odd_Bloke> cloud-init 2.0 doing something? Controversial.
<Odd_Bloke> I thought we were just going to implement all the fun bits and then do something else. ;)
<smoser> exactly
<smoser> ok.
<smoser> (thanks smatzek for comment above. any help you have is welcomed)
<smoser> thats about all we have then.
<Odd_Bloke> Cool.
<Odd_Bloke> A brief AOB point: do we have a list of everyone we should ping at the start of a meeting? :p
<Odd_Bloke> I can only type sm<TAB> once, so only one of smoser and smatzek are going to get pinged unless we make a list. :p
<smoser> yeah, i know. :) quite rude of smatzek to take collide on my first and last initial.
<smoser> smb is also guilty of that.
<smoser> (/me realizes both of them probably came into existence before him)
<Odd_Bloke> Now you're calling them old; real smooth. :p
<smatzek> claudiupopa:  A pre-emptive review on ConfigDrive.  If you're going to use 0.7.x as a starting point, can you make util.mount_cb and def find_candidate_devs(probe_optical=True): do the mount and candidate find in a way that can be overridden by the OS distro?
<smatzek> smoser: thanks :)
<smatzek> or have it call to the distro for that support?
<smoser> smatzek, well claudiupopa cares about this legacy OS that comes from some vendor in washington.
<smoser> which apparently doesn't even have a 'mount'
<smoser> so he'll probably abstract it somehow :)
<smatzek> an AIX port of cloud-init exists outside of 0.7.x trunk and this is one of the things that is different in that port.  I know if at least 1 other legacy OS, besides the washington one that a port is being created for that also has different ways it handles ISOs.
<smatzek> I hope to put patch sets out to bring the AIX items into 0.7.x and definitely 2.0 once I clear some legal hurdles.
<smoser> cool.
<claudiupopa> Sorry, I was dragged into other meeting for a while. smatzek, smoser: probably it will be abstracted, so we won't care too much about mounting from an interface perspective.
<smoser> claudiupopa, ok... very bad nitpick
<smoser> your commit message width i think is > 80
<jaksi07c8> hm.. is there a "right" way to set up static routes with cloud-init?
<jaksi07c8> I mean I can just run a command like `ip route add 1.2.3.4 via 5.6.7.8`, but that's not the prettiest
<smoser> claudiupopa, commented on your https://review.openstack.org/#/c/209520/
<smoser> jaksi07c8, no. in the future we expect much better /  more aware network knowledge, but for now... i think thats the best you can do .
<smoser> :-*(
#cloud-init 2015-08-27
<harlowja> hellooooo
<harlowja> smoser shall i put cloud-init on https://wiki.openstack.org/wiki/Stackforge_Namespace_Retirement#Active_Projects_to_Move ?
<harlowja> i added it, feel free to remove if wanted :-P
<Odd_Bloke> harlowja: I don't think we have any interest in moving.
<harlowja> i don't think there is a choice?
<harlowja> lol
<Odd_Bloke> Man, this entire thing has been so shoddily managed.
<Odd_Bloke> They said that things could just stay until some undefined point in the future.
<Odd_Bloke> Has that point been defined as October 17th now?
<Odd_Bloke> Or is this just "now you can move, we'll do them all at once"?
<Odd_Bloke> Yes, that appears to be the case.
<Odd_Bloke> If you were going to decide on a date less than 3 months from the original decision, why not just decide IN THE ORIGINAL DECISION.
<Odd_Bloke> Well, whatever.
<Odd_Bloke> harlowja: Thanks for staying on top of this. :)
<harlowja> Odd_Bloke np
<harlowja> http://lists.openstack.org/pipermail/openstack-dev/2015-August/073071.html Odd_Bloke
#cloud-init 2015-08-28
<smoser> harlowja, reading.
<smoser> so 'remain under stackforge/' is not an option ?
<smoser> Odd_Bloke, is that your reading ?
<Odd_Bloke> Yes.
<smatzek> what's up?
<smoser> the wiki is clear in stating that.
<smoser> the email is not
<smoser> smatzek, basically
<smoser>  https://github.com/stackforge/cloud-init -> https://github.com/openstack/cloud-init
<smoser> while i do not really upstream git appearing under "openstack/" namespace, i like it more than dealing with managing my own gerrit
<smoser> and other infrastructure that comes "for free".
<smoser> so for now, i think "openstack/"
<smoser> claudiupopa, did yous ee comments on https://review.openstack.org/#/c/209520/ ?
<smoser> wonder if Odd_Bloke or harlowja think i'm off my rocker there.
<harlowja> smoser Odd_Bloke openstack/ it is then, its fine with me, just a namespace, i don't think to many people will be confused by it
#cloud-init 2016-08-29
<cn28h> smoser: any thoughts why cloud-init would not create the cloud-user user? seems to be the root of the issue we're seeing.  The cloud-init.log says the user is created, but afterward running "id cloud-user" returns no such user... full log is here: http://dpaste.com/2V9JTX3
<cn28h> line 633 says it's adding cloud-user
<cn28h> unless I'm misconstruing the meaning of that log
<cn28h> well, couple lines later it outputs and actual useradd cmd
<cn28h> hm
<smoser> cn28h, it sure seems like it created it
<smoser> can you give me an example of the user-data you provided ?
<cn28h> https://paste.fedoraproject.org/417015/98181147/
<cn28h> here's the user data
<cn28h> and yeah.. the logs sounds awfully like it was created
<cn28h> other than that getpwnam() failed (and that checking afterward says it doesn't exist)
<smoser> cn28h, maybe you collected a stale log ?
<smoser> those dates are 6 days ago
<smoser> hm.. never mind. the getpwname error is same time frame.
#cloud-init 2016-08-30
<rharper> smoser: that MP you just proposed diff looks big (it includes a README.source and debian/changelog entry; expected ?
#cloud-init 2016-08-31
<blaisebool> hello, how can I run an ansible playbook bootstrapped with cloud-init ?
<smoser> rharper, hm.
<smoser> rharper, bah. i must have branched ubuntu/devel. will re-do from master.
<smoser> rharper, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/304415 looks saner now.
<blaisebool> hello, how can I run an ansible playbook bootstrapped with cloud-init ?
<smoser> blaisebool, cloud-init doesn't have any specific support for ansible.
<smoser> but a simple '#!' as user-data that does whatever you would normally do should work for you.
<smoser> cpaelzer, https://bugs.launchpad.net/cloud-init/+bug/1618572
<smoser> i can't reproduce either. i had a simplified version also.
<smoser> i suspect it might be related to overlayroot
<cpaelzer> hmm
<cpaelzer> ah you are here
<cpaelzer> only saw in #server before
<smoser> so maas ephemeral environment is running overlayroot
<cpaelzer> roaksoax not here
<cpaelzer> :-/
<smoser> /etc/apt/.... thats on overlayroot
<smoser> i'll try on a serverstack
<smoser> and let you know how i did it.
<roaksoax> cpaelzer: here
<cpaelzer> roaksoax: hi
<cpaelzer> roaksoax: just relaized discussing with smoser that you were missing here
<cpaelzer> roaksoax: smoser is assuming it might be related to overlayroot and tries to recreate on serverstack
<roaksoax> cpaelzer: ack!
<roaksoax> cpaelzer: i'm setting up a different environment too
<roaksoax> cpaelzer: so you can access there
<smoser> i suspect this is overlayroot as the cause.
<smoser> we've seen surprisinggly little issues with overlayroot, but it does have some wierdities
<roaksoax> smoser: but wouldn't that also affect the chroot ?
<roaksoax> smoser: but wouldn't that also affect the chroot for curtin ?
<smoser> roaksoax, no. because the chroot is not on an overlay
<smoser> this is overlayroot related, almost sure.
<smoser> i just did this:
<smoser> a.) launch instance in openstack (serverstack) of yakkety
<smoser> b.) echo "overlayroot=tmpfs" | sudo tee /etc/overlayroot.local.conf
<smoser> c.) sudo reboot
<smoser> console log: http://paste.ubuntu.com/23116093/
<smoser> horay!
<smoser> seems like kernel regression in overlayroot to me
<roaksoax> fun!
<smoser> ok. suck.
<cpaelzer> grmbl
<cpaelzer> why can't there ever be a day where things just work :-)
<cpaelzer> smoser: since you call it a regression - is the same working for xenial or before?
<cpaelzer> smoser: I don't know if you had the time
<smoser> so what i know so far
<smoser> trusty , i can do
<smoser>  boot image
<smoser>  sudo sh -c 'echo overlayroot=tmpfs > /etc/overlayroot.local.conf && reboot'
<smoser> everything fine
<roaksoax> smoser: btw.. on that note, any ETA on getting cloud-init to xenial (with the apt support ?)
<smoser> that seems to stop working at wily (systemd)
<smoser> roaksoax, well, why would i SRU that when it would break you
<smoser> (as you shown)
<smoser> :)
<roaksoax> smoser: but if you say that's a kernel regression :) then that at least would get into -proposed
<roaksoax> smoser: so it is ready to land once kernel fixes the kernel ?
<smoser> well, theres stills ome things i do not understand there.
<roaksoax> smoser: oh I see!
<smoser> i dont think cloud-init is at fault
<smoser> but we may end up needing to make changes
<roaksoax> smoser: yeah doesn't seem to be that case
<roaksoax> right
#cloud-init 2016-09-01
<smoser> rharper,
<rharper> smoser:  y
<smoser> your merge proposal, user has to provide the user name ?
<smoser> not just the email addr
<rharper> a name
<smoser> right ?
<rharper> it doesn't have to match, and it isn't used
<rharper> I didn't play with how cloud-init merges that data
<rharper> it expects a name field to build the name, config parameters
<rharper> it'd be a much more invasive change to allow a user config with snapuser only IIUC
<rharper> but I might have missed something
 * rharper tests quickly 
<rharper> smoser: no, it's not required, so I can update the docs/example
<smoser> rharper, but then it can't be the default user.
<rharper> it's not supposed to be the default user
<smoser> ok. i think fine.
<rharper> there are many other issues that are yet resolved in snappy land w.r.t what the expections are for devices that are "online" vs "offline" and whether "local users" (users not fetched from the store SSO) are allowed or not
<rharper> the MP provides enough to allow online devices to fetch creds from the SSO, and  if you want to make 'admin' level users, you can do that with the -users syntax in cloud-init on snappy systems
<rharper> I filed two bugs
<rharper> ssh-import-id doesn;t work for users since they stripped the tool from the image
<rharper> https://bugs.launchpad.net/snappy/+bug/1619423
<rharper> also, I filed, https://bugs.launchpad.net/snappy/+bug/1619420
<rharper> also due to removal of normal stuff from the image
<smoser> i have to run. it looks good
<smoser> an example would be nice
<smoser> but ...
<smoser> i have to run now.
<rharper> I'll fix up the example
<rharper> I think I did put one in the existing user-groups
<rharper> smoser: later
#cloud-init 2016-09-02
<rharper> smoser: http://paste.ubuntu.com/23124387/ -- looking at the event reporter output;  with the context manager, we get a start/finish for each manager;  however, for the stage runners, we don't seem to get the 'start' event;  in the paste, notice we have an extra finish, which is the other half of the 'start' of the module .. any idea why we don't get that first start event ?
<smoser> how did you give it the config you wanted ?
<smoser> rharper, ^ ?
<smoser> i think maybe it didn't know of the config until after it started/
<rharper> this is defaults
<rharper> I'm parsing cloud-init.log output to recreate the records
<rharper> if you look at cloudinit/cmds/main.py, it's the args.reporter which sets up the event stack with the name.
<rharper> it does seem like the first event isn't "published" to the logger;  so maybe it's just getting chewed  somewhere in that util.log_time
<smoser> rharper, try turning of rsyslog
<smoser> in /etc/cloud/cloud.cfg.d/05_logging.cfg
<smoser> comment out - [ *log_base, *log_syslog ]
<rharper> ok
<rharper> I'll give that a shot
<rharper> smoser: that didn't help; but I'll dig in and see what's up in main.py w.r.t the __enter__
<rharper> 2016-09-02 17:48:13,667 - util.py[DEBUG]: Cloud-init v. 0.7.7 running 'init-local' at Fri, 02 Sep 2016 17:48:13 +0000. Up 1.0 seconds.
<rharper> 2016-09-02 17:48:13,669 - util.py[DEBUG]: Writing to /var/log/cloud-init.log - ab: [420] 0 bytes
<rharper> smoser: I think that clobbers the log file right after we've published the start event
<rharper> so we miss the start event for each stage invocation, if that's true
<rharper> let me remove that and see what it looks like
<smoser> well, it does say 'ab'
<smoser> but i guess quitei possibly athe first message was not flushed, and the filehandle probably still open
<rharper> yeah;  still chasing it; it's just getting lost in logger;
<rharper> I can see it getting published via the LogHandler and it's configured
<rharper> we don't have this issue with curtin, so something is different
<rharper> smoser: I don't understand how it can be fixed, but basically, since we don't "reconfigure reporting" until after the Init stage is created (and we call apply_reporting_config() with the initial cloud config ) then we lose the first set of reporting events (logging gets them but they're not ever emitted to /var/log/cloud-init.log)
<rharper> it isn't terrible since not a log happens, but in terms of having matched pairs of events (start/finish); there's no point to starting the reporting event stack until after Init.stage2  is complete
<smoser> rharper, yeah. i gues reconfigure_reporting could re-send the current one.
<rharper> smoser: I don't know how we'd be able to replay previous events;
#cloud-init 2017-08-28
<fxpester> hi all, how to debug datasource discovery ? `cloud-init -d init` not initializing it, can`t debug configdrive descovery
<smoser> fxpester,
<fxpester> hihi
<fxpester> in 'normal' logs I can see  `Command: ['mount', '-o', 'ro,sync', '-t', 'auto', '/dev/sda1', '/tmp/tmpxamurqwn']`
<fxpester> but can`t find a way to reproduce it
<smoser> fxpester, possibly you dont have a module loaded at that point in boot that you need and gets loaded later ?
<smoser> what kernel /distro is this ?
<smoser> if you could post both /var/log/cloud-init.log and /var/log/cloud-init-output.log tha'td hhelp
<smoser> and also a dmesg maybe
<Tugz> hey all
<Tugz> question: how do i know which modules cloud-init 0.7.5 supports? This is on cnetos 7 and im trying to use disk_setup and fs_setup without success :(
<blackboxsw> hey Tugz, checking  /etc/cloud/cloud.cfg should list module names that are configured for your environment.
<blackboxsw> look under cloud_init_modules, cloud_config_modules and cloud_final_modules sections for the complete list
<blackboxsw> of enabled modules
<blackboxsw> I'd expect - disk_setup listed under cloud_init_modules section
<blackboxsw>  - growpart
<blackboxsw>  - resizefs
<blackboxsw>  - disk_setup
<blackboxsw>  are on my ubuntu instance
<rharper> Tugz: this may be related, https://bugs.launchpad.net/cloud-init/+bug/1672833 , the tl;dr was that the disk_setup module wasn't included in the /etc/cloud/cloud.cfg in some older releases;
<ubot5> Ubuntu bug 1672833 in cloud-init "AWS NVME SSD (i3 family) ephemeral storage fails to mount via fs_setup" [Medium,Invalid]
<rharper> https://bugs.launchpad.net/cloud-init/+bug/1672833/comments/4
<rharper> hackery to fixup in that comment if that's the case for your cent7 image
<Tugz> hey blackboxsw where is this section?
<Tugz> oh its on the .cfg found it
<Tugz> thanks
<blackboxsw> Tugz: older releases of cloud-init may not have the specific key name, but yeah /etc/cloud/cloud.cfg
<Tugz> how do i know if this version of cloud init supports the modules?
<blackboxsw> if the module exists on the config modules directory for your version then it is "available" on your platform.  Modules are all named cc_*py:     'rpm -ql cloud-init | grep cc_'
<blackboxsw> the module is then enabled only if it is listed in /etc/cloud/cloud.cfg
<blackboxsw> so different platforms/images can "disable" a module by not listing the named module in /etc/cloud/cloud.cfg
<blackboxsw> even if it is installed in /usr/lib/python2.7/site-packages/cloudinit/config/cc_*py
<blackboxsw> as a note, we just talked about the potential of surfacing a cmdline utility to do this introspection on your behalf to easily report availability && enabled/disabled for given features/modules
<Tugz> blackboxsw, thanks fo rthe valuable info. much appreciated
<blackboxsw> no worries Tugz, happy hacking
<blackboxsw> hrm looks like we had dupes in https://trello.com/c/CAjwe8LX/273-cloud-init-sru-zesty-xenial for https://bugs.launchpad.net/cloud-init/+bug/1701097
<ubot5> Ubuntu bug 1701097 in cloud-init (Ubuntu Zesty) "eni rendering of ipv6 gateways fails" [Medium,Fix committed]
<blackboxsw> It see it twice in our SRU list
<rharper> blackboxsw: well, we get one for free then
<blackboxsw> yeah already at 70%. man long day
<blackboxsw> verifications 'r us
 * blackboxsw needs to get a branch up to include net-convert in "cloud-init  devel" at some point
<blackboxsw> it'd assist with half of these
<blackboxsw> too many other fish to fry at the moment
<rharper> blackboxsw: heh
 * rharper has PYTHONPATH=`pwd` ./tools/net-convert.py  in history 
<rharper> but, yeah, would be nice to have it in subcommands
<rharper> also, s/network_data.json/json in the input format parameter
<blackboxsw> smoser: I'll probably need an assist validating SRU bug #1638931
<ubot5> bug 1638931 in cloud-init "enable AliYun datasource by default" [Medium,Confirmed] https://launchpad.net/bugs/1638931
<blackboxsw> as I have no aliyun acct
#cloud-init 2017-08-29
<fxpester> hi all, I have problem on xenial with configdrive, looks like  /tmp isn`t created yet but cloud-init tryes to use already
<fxpester> so, mount failed
<fxpester> cloud-init 0.7.9
<fxpester> I`m sure I`ve seen this bug on launchpad, but can`t find it now
<fxpester> hm, /tmp is present in image
<fxpester> error is 'Stderr: mount: mount point /tmp/tmpmdrxpk3u does not exist' if I run `cloud-init init --local` everything is working correct
<fxpester> finally found bug - https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1683974
<ubot5> Ubuntu bug 1683974 in cloud-init (Ubuntu) "Cloud-init fails to mount datasource from iso disk sometimes" [Undecided,New]
<robjo> smoser: Do I need to do another "merge proposal" in LP after I made the requested changes or do you get notified automatically after I pushed the changes?
<smoser> robjo, i get email when you comment.
<smoser> but i dont know if i get updates if you just git push or git push --force
<nacc> smoser: i don't think you do, but i'm not 100% either
<smoser> fwiw, i consider it always ok to say "smoser, why haven't you responded"
<smoser> most of the time the answer is "busy"
<smoser> but...
<nacc> smoser: we use work-in-progress and needs-review state changes
<smoser> it is ok to bother me.
<smoser> nacc, yeah, i often times do that.
<smoser> thanks
<robjo> OK, so the suseIntegrate branch has been updated with the requested changes and is ready for another review
<smoser> thanks. i'll take a read.
<nacc> robjo: i do think it's a bug in LP that there isn't a notification for that, fwiw :) I think github notifies watchers
* smoser changed the topic of #cloud-init to: reviews: https://code.launchpad.net/~cloud-init-dev/cloud-init/+git/cloud-init/+ref/master/+activereviews | next meeting Tue, 05 Sep 2017 16:30:00 +0000
<smoser> blackboxsw, ^
<blackboxsw> thanks smoser  +1
<smoser> blackboxsw, i just took your ec2 branch and ran it on an instance and got ipv6. horay!
<blackboxsw> superb smoser
<smoser> 2017-08-29 16:19:04,367 - stages.py[INFO]: Applying network configuration from ds bringup=False: {'version': 1, 'config': [{'type': 'physical', 'name': 'eth0', 'subnets': [{'type': 'dhcp4'}, {'type': 'dhcp6'}], 'mac_address': '06:8d:72:65:39:7c'}]}
<blackboxsw> I'm reworking the url_helper Oauthlib ImportError branch just a bit to make it testable.
<blackboxsw> already landed larsks' initial branch making the import optional
<blackboxsw> thx larsks
<blackboxsw> smoser: are you waiting on a 2nd round of reviews for https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/329397?
<smoser> blackboxsw i wasnt necessarily. just didnt see it as high priority , and then you talked about common datasources and things.
<smoser> i dont think it really hurts anything to have it
<smoser> also i didn't ever actually test it on an instance, so i wasn't going to pull it until i did so :)
<smoser> i tested that 'read_md' worked, but not that the whole boot worked
<blackboxsw> smoser: gotcha, that makes sense. I wasn't sure if that was something you were thinking of dropping ultimately because of the common datasource discussion as well. Yeah I do think that common datasource would solve some of that problem. It'll admitedly be a bit until we have common datasource methods landed (probably 2 branches to be written).
<blackboxsw> I think your adding a main is good for the moment until we get around to a cohesive/standard base DataSource api definition for all datasources. This work also ties in a bit with the JSON storage/non-pickle datasource on disk.
<smoser> blackboxsw, did you have other comments on that MP ?
<smoser> i'll give a quick test and then comment and pull if you approve too
<blackboxsw> smoser: just a nit added about allowing us to inject the testable metadata address instead of initializing to a global.
<blackboxsw> and +1 there
<smoser> blackboxsw, i'm going to add a 'log_time' around it too
<smoser> (which ... had it already had the log time, we'd have probalby not ended up doign this)
<smoser> blackboxsw, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/329397
<blackboxsw> +1 there when unit tests finish
<blackboxsw> CI pass rather
#cloud-init 2017-08-30
<basilAB> I was trying to use cloud-init (version:  0.7.5) on ubuntu-14.04. I am finding issue in appending ssh key to authorized from 'OpenStack' metadata. But it is working with 'ConfigDrive'
<basilAB> https://pastebin.com/0JPwU8fK
<robjo> smoser: Can take another look at https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/329616 and merge or let me know what else needs fixing?
<niluje> :<
<smoser> whats up niluje
<smoser> robjo, ... looking now
<smoser> basilAB, is that your holw /var/log/cloud-init.log ?
<basilAB> smoser: yes. 'grep' with 'failed'
<basilAB> my host has python2.7 and python 3 on the host and can see all the modules called under python2.7
<afranc> When does cloud-init detect payload as `text/x-not-multipart` ?
<afranc> I passing a shell script as a user data but when i start some extra commands, the ShellScriptPartHandler detects the content as `text/x-not-multipart` ending it up as an `Empty payload`, logs - https://pastebin.com/raw/sd32FqJw
<smoser> fun
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/329926
<smoser> robjo, i responded in mp
<smoser> basilAB, can you paste the hole log ?
<robjo> smoser: thanks will get back to it today
<smoser> blackboxsw, see merge above. powersj interested in your feedback there too
<smoser> afranc, what was the problem ?. can you post an entire log and tell me what you expected ot happen and what didn't?
<blackboxsw> grabbing it now
<powersj> smoser: ah yes I saw this fail recently on my artful box
<smoser> i think that the MP above is really all we can do, but that means somehow we  have to have jenkins test runners have python3.5
<smoser> as i really want c-i to report failures that would be seen in xenial
<smoser> i guess alternatively we could un-pin jsonpatch
<smoser> or pin at a newer (artful) version and just live with the risk of differences
<basilAB> smoser: Here is the full log https://paste.ubuntu.com/25432350/ (The problem which I was checking was to see why ssh key available in the metadata is not getting appended to ubuntu user)
<basilAB> I can see the user getting created
<smoser> 2017-08-30 14:45:00,079 - cc_final_message.py[WARNING]: Used fallback datasource
<smoser> that means it did not find a metadata service to use.
<afranc> smoser  i have posted both user-data and cloud-init.log which is failing - https://pastebin.com/raw/zEtB6XDf
<blackboxsw> smoser: per your branch on xenial is it worth us adding a python3.6 tox env to better exercise 3.6 issues?
<smoser> basilAB, your networking seems to not have come up.  in 14.04 you have to have an /etc/network/interfaces that defines basically "dhcp on eth0".
<smoser> cloud-init assumes the systme is set up to configure networking.
<afranc> smoser  if i have a user-data like this https://pastebin.com/raw/wiDx027d, the ShellScriptPartHandler detects the content as `text/x-shellscript` writes it to `/var/lib/cloud/instance/scripts/part-001` and executes it fine
<smoser> blackboxsw, well that becomes hard to run then from xenial
<afranc> smoser but if the user-data has additional commands like in https://pastebin.com/raw/zEtB6XDf, the `ShellScriptPartHandler` handler detects it as `text/x-not-multipart` and complains it as `Empty payload of type text/x-not-multipart` and writes it as `Writing to /var/lib/cloud/instances/af23b054-f125-4821-8639-084933ad7244/cloud-config.txt - wb: [384] 0 bytes` and do not execute the entire shell script from user-data
<afranc> smoser i wanted to know why adding extra commands, make the `ShellScriptPartHandler` detect the content as `text/x-not-multipar`
-basilAB:#cloud-init- I have'source /etc/network/interfaces.d/*.cfg' in '/etc/network/interfaces . Interface configuration: https://paste.ubuntu.com/25432392/
<smoser> afranc, i suspect you might not have an 'eth0' device?
<smoser> something made networking not come up
<smoser> er.. sorry. not afranc above, but basilAB .
<afranc> ok
<basilAB> hmm.. I do see eth0 up with an IP assigned from DHCP after cloud-init run. Are you saying it is a timing issue ?
<smoser> afranc, nothign created the user 'ubuntu' i think
<smoser> afranc, what image are you using ?
<afranc> smoser  Ubuntu 14.04.2 LTS
<smoser> from cloud-images.ubuntu.com ?
<smoser> or did you build one
<afranc> smoser, i build one .
<afranc> smoser  i am confused because at some cases shell script as user-data works
<smoser> afranc, well, cloud-init failed to find the datasource.
<smoser> so it didn't get your user-data
<afranc> smoser doesnt this confirm `2017-08-30 14:48:06,025 - util.py[DEBUG]: Crawl of openstack metadata service took 36.475 seconds`
<afranc> and this log - https://pastebin.com/raw/v4215hr8 is a working case using the same cloud.cfg
<smoser> afranc, i'im confused a bit on what happened there.
<smoser> bouncing the networking is wierd. 14.04 doesn't do networking that well. i do not think that the format of the user-data is your problem.
<afranc> smoser  this is a log from file from a working use case - https://pastebin.com/raw/wh2yyfQZ
<smoser> afranc, well, a log of *only* that boot would be good too
<smoser> the one you posted has multiple boots
<smoser> and i'd really suggest sanity checking *anything* against an official ubuntu image.
<afranc> this is another log file from non-working use case - https://pastebin.com/raw/UESQGp7S
<afranc> ok
<afranc> i was only seeing the difference in the way the `ShellScriptPartHandler` detecting the user-data
<afranc> for a working use case it detects the user-data as `text/x-shellscript` and the case where it doesn't work, it detects it as `text/x-not-multipart'`
<afranc> ok. i will do the sanity check on the image
<smoser> afranc, possibly you have a space at the beginning of it ?
<smoser> it only will read the first line. and expects it shoudl start with '#!' to be a shellscript
<afranc> i see. i will double check
<afranc> how does it detect the content as `text/x-not-multipart` ?
<smoser> it doesnt know what it is. thats just fallback.
<afranc> i see
<smoser> rharper, fwiw, i think i agree with xnox's mp for upstart in artful.
<smoser> he doesnt delete the upstart/ jobs just removes the packaging of them in artful
<smoser> and if all other ubuntu packages have done that, then cloud-init in ubuntu shoudl too.
<rharper> smoser: ok
<afranc> thanks smoser  i will dig this further
<rharper> smoser: IIUC, when there is an artful+1;  it will get branched from master, so we'll need to re-apply the same MP to that from master;  no?  So, it seemed to me that upstream should need some changes so that a downstream would opt-in to those files being packaged; I dunno
<smoser> no. master has no debian/
<smoser> it has packages/debian
<smoser> and packages/debian/rules *does* have 'upstart' in it. but we can drop that. it doesnt really hurt anything.
<smoser> basically the plan is that ubuntu/devel branch will always go on being ubuntu/devel (right now that is artful)
<rharper> also, downstream branches also have a cloud.cfg that'll have the emit upstart and other upstart related runtime files
<smoser> the first time we need to SRU to artful, we create ubuntu/artful branch. (or we can do taht whenever artful releases... doesnt really m atter).
<rharper> generally I was hoping that we'd comb through master to find out what a 'non-upstart' code base would need to look like
<blackboxsw> rharper: were you referring to this locale branch in standup today? https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/329152
<rharper> yeah
<blackboxsw> that you mentioned you had updated yesterday?
<rharper> y
<blackboxsw> last commit pushed was 08-16
<rharper> can't be
<blackboxsw> I don't see the changes that were discussed between you and smoser
<blackboxsw> hmm refreshing
<blackboxsw> maybe launchpad is falling apart. I'll try a clone and see if I see changes
<rharper> ok
<rharper> it looks fine to me in lp
<rharper> maybe because of rebase?  I dunno
<nacc> commit authorship date != push date
<nacc> and if you do amending, etc. the date won't necessarily change
<rharper> nacc: thanks
<nacc> i think LP uses author date, rather than commit date (or v.v). I can't recall
<nacc> but i've seen similar artifacts
<blackboxsw> hmm I'm only seeing cf1c4106545b6acceded30ae9f3f65910277335e
<rharper> cf1c4106545b6acceded30ae9f3f65910277335e is where it's at
<rharper> blackboxsw: see nacc's comment
<nacc> blackboxsw: you could try and look at `git log --pretty=full` to see the commit and authorship dates both
<nacc> i think that's the argument
<nacc> oh 'fuller'
<blackboxsw> nacc +1 on that.
<blackboxsw> hmm. rharper I thought you made references to authorship changes you made yesterday. I was expecting to see some changes that we determine needs_regen value based on a call to 'locale -a'
<blackboxsw> maybe I misunderstood from standup. Was your intent to avoid the extra exec in querying whether the local was already generated?
<rharper> blackboxsw: we're not querying what's present via locale -a; rather the change was to read the distro default locale config file, and if that has a set value, to assume the image is built correctly and not regenerate then
<rharper> it handles if the file is present but has no value, or if missing, and will regen as needed there
<blackboxsw> ok gotcha, thanks for the correction. ok, makes sense to rely on images to be properly built
<rharper> I think the path we took was that the image builder should be in control of what the default lang for the image is, and declare that in the correct distro location;, cloud-init will read that and respect it;  if an image is built with locales but left unset, then cloud-init will regenerate the cloud-init default locale
<rharper> blackboxsw: right
<rharper> my most recent change was the keep around the value present in the default locale config file and have that be separate from the cloud-init default/fallback value so we can reason on whether we need to update the locale config file, as well as regenerate
<blackboxsw> gotcha, and you must've squashed your commits then
<blackboxsw> which is why I didn't see updates
<blackboxsw> ok
<rharper> blackboxsw: I did
<rharper> one of the reasons smoser liked bzr (-n 0 would show you the evolution of the code)
<dpb1> cloud-init summit notes posted: https://lists.launchpad.net/cloud-init/msg00094.html
<smoser> man. look at that active mailing list
<smoser> 3 messages in 2 days!
 * smoser just sent https://lists.launchpad.net/cloud-init/msg00095.html
<dpb1> smoser: I almost couldn't find the archive link!
<dpb1> smoser: can I modify the review to be a bit.ly link?
<dpb1> (in topic)
<Odd_Bloke> unsubscribe
<dpb1> Odd_Bloke: you always have to be the center of attention
<smoser> dpb1, i will.
<Odd_Bloke> dpb1: I find it's better for everyone.
<smoser> Odd_Bloke, is that if you're the center of attention or if you leave the playground (unsubscribe)
<dpb1> smoser: thx, can we also put up a bitly for the email list? like, | mail list: http://pad.lv/~cloud-init |
* smoser changed the topic of #cloud-init to: reviews: http://bit.ly/ci-reviews | mail: http://bit.ly/ci-mail-archive | docs: http://cloudinit.readthedocs.io/en/latest/ | next meeting Tue, 05 Sep 2017 16:30:00 +0000
<smoser> how's that?
<dpb1> nice
<Odd_Bloke> smoser: Little bit of column A, little bit of column B.
<dpb1> smoser: wish there was a way to inject text into this page: https://code.launchpad.net/~cloud-init-dev/cloud-init/+git/cloud-init/+ref/master/+activereviews
<dpb1> smoser: like describing what it meant for something to be wip
<dpb1> smoser: but, oh well
<nacc> dpb1: per project? yeah, i agree
<dpb1> (going back to what we were discussing earlier about WIP branches)
<nacc> dpb1: i'd file a bug on launchpad
<nacc> tbh
<dpb1> nacc: you have the same ask for #ubuntu-server stuff too, right?
<nacc> dpb1: yeah, and i imagine any team would like to put 'policy' somewhere that's well-defined and have LP display it
<dpb1> nacc: ok, I will  no sense complaining about it in #cloud-init channel
<dpb1> lol
<nacc> if they are using git on LP, i mean
<dpb1> thx, good suggestion
<blackboxsw> rharper: approved with nits https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/329152
<robjo> smoser: https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/329616 should be all set now
<blackboxsw> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/329946 looks good. a doc nit and and a question for you smoser
<blackboxsw> nice robjo, I'll glance over that too today to catch up on opensuse context.
<smoser> blackboxsw, please do. i am basically good with it at this point. the tests bare minimum, but better than there *was* so i didnt want to tax robjo wit their creation.
<blackboxsw> +1 landing it smoser don't wait on me for that :). but yeah I had missed his branch earlier so wanted to read it a bit more
<smoser> blackboxsw, responded
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/329946
<smoser> and updated with doc removal. good catch on that . thank you.
<smoser> rharper, i had a question on your locale mp also
<rharper> smoser: here
<smoser> rharper, see mp.
<rharper> k
<smoser> you can answer here if you want more real time
<rharper> smoser: hrm, you're right;  I don't really like the idea of reading the same file twice in the default/common path; so either if we write an updated locale conf, we invalid the cached variable, or we just let it always read the file
<smoser> rharper, oh. i was just suggesting when you successfully 'apply', then you can just set the new value.
<smoser> but yeah, setting it to None would do the same thing and just force a re-read
<rharper> ok; that works
<smoser> yeah. just update it.  since you're caching it you are guaranteed to possibly be out of date
<smoser> so setting it (rather than setting it to None) doesn't make it any worse
<rharper> right
<rharper> we're not really protecting against file system writes underneath us
<rharper> possibly if we were to be daemonized and called a second time to switch locales;
<rharper> smoser: blackboxsw: thanks for review, working on an update
<smoser> blackboxsw, so your change at https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/329872
<smoser> that does change to doing a 'import oathlib...' every time oauth_headers is called
<smoser> i guess if you like the clarity it seems the performance is not terribly worse off.
<smoser> >>> timeit.timeit("bool(oauth1 is None)", setup="import oauthlib.oauth1 as oauth1")
<smoser> 0.2964163040742278
<smoser> versus
<smoser> >>> timeit.timeit("import oauthlib.oauth1 as oauth1")0.531060776906088
<smoser> (default is 1M runs)
 * rharper wants is 2  tenths of a second back
<smoser> powersj, updated https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/329926
<smoser> 2 tenths of a second per 1M runs, rharper
<rharper> the average over 1M runs
<rharper> no?
<rharper> or total time?
<rharper> yeah, you're right
<rharper> or is that per loop? I'm confused now
<smoser> that shows total time
<rharper> python3 -m timeit '"-".join(str(n) for n in range(100))'
<rharper> 10000 loops, best of 3: 30.2 usec per loop
<smoser> timeit does
<rharper> https://docs.python.org/3/library/timeit.html
<smoser> the python -m might be different.
<rharper> timeit.timeit('"-".join(str(n) for n in range(100))', number=10000)
<rharper> 0.3018611848820001
<smoser> the above definitely ran 1M imports in .3 seconds
<rharper> nope
<rharper> pretty sure that's time per loop
<smoser> nah. i'll show
<rharper> I mean, there are words on that page
<smoser> timeit.timeit("time.sleep(1)", setup="import time", number=3)
<smoser> 3.0026550928596407
<rharper> that is really  bizarre that the cli and the library differ ; =(
<smoser> blackboxsw, rharper please re-review https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/329811
<rharper> k
<smoser> and blackboxsw https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/329811
<blackboxsw> re-reading both
<blackboxsw> hmm that's the same link
<blackboxsw> and testing mkdtemp on permissions set.
<smoser> blackboxsw, hm.. second ment to be
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/329946
<smoser> blackboxsw, i will come back in later tonight and will pan on doing an artful upload.
<blackboxsw> sounds good smoser thx
<smoser> and https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/329872 is in now.
<rharper> smoser: blackboxsw: I'll try to have an update to locale branch soon so it can get reviewed prior to smoser's return tonight
<blackboxsw> smoser: thanks on the land. So, per import attempts inside oauth_headers (rharper too) this is a performance impact to every publish_event webhook in maas right?
<rharper> ack
<blackboxsw> roger. ok.
<rharper> at least I think so
<rharper> blackboxsw: well, I think we reasoned that 1) it only imports it once 2) systems without ouath pay the most (try to import again)  3) maas systems have oath
<rharper> so, it should have no per-message impact on ubuntu (which is where the maas reporting with oauth is most used)
<rharper> non-ubuntu are unlikely to use the MAAS Datasource, so won't pay for the re-import attempt
<blackboxsw> right, ok that's reasonable. (and given that systems without  oauth would break if they tried calling oauth_headers in the first place, we're safe there).
<rharper> right, pay once and then stacktrace
<powersj> blackboxsw: FYI https://bugs.launchpad.net/cloud-init/+bug/1714117
<ubot5> Ubuntu bug 1714117 in cloud-init "tox: long CI times" [Undecided,Confirmed]
#cloud-init 2017-08-31
<smoser> rharper, blackboxsw well, it will have *some* per message hit on ubuntu
<smoser> but on the order of 5/3 of what it was.  thats roughly the ratio of difference for a 'import' of an already imported library compared to bool(variable is None)
<smoser> now in the missing library case, python will probably go statting a bunch of possible locations again.
<smoser> and there could be a larger hit
<smoser> but in reality, if you hit that code it wont go doing it again (because it raises a nonimplementederror)
<smoser> rharper, or blackboxsw around ?
<smoser> i have a fix for trunk's current busted tests
<smoser> or powersj
<smoser> http://paste.ubuntu.com/25435503/
<smoser> failure is seen at https://jenkins.ubuntu.com/server/job/cloud-init-ci/236/console
<powersj> smoser: still there?
<powersj> seems you already fixed it
 * powersj disappears again
<smoser> yeah. shoudl be good now. and uploading to artful
<smoser>    e5bbf884..f713c6e9  HEAD -> ubuntu/devel
<smoser>  * [new tag]           ubuntu/0.7.9-259-g7e76c57b-0ubuntu1 -> ubuntu/0.7.9-259-g7e76c57b-0ubuntu1
<smoser> and uploaded.  cloud-init_0.7.9-259-g7e76c57b-0ubuntu1_source.changes
<Odd_Bloke> It would be nice if ds-identify/cloud-id emitted something that would end up in the nova console log; it would make working out why an instance hasn't come up a little easier.
<blackboxsw> smoser: if you run "make deb" on your drop-ubuntu-init-switch branch does it also generate a deb for you that doesn't contain etc/init files
<smoser> doesnt ?
<smoser> i'm confused
<smoser> blackboxsw, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/329946 is updated now.
<smoser> changed the default 'init-system' for bddeb.
<powersj> Long running tests on CI system: https://paste.ubuntu.com/25439503/
<blackboxsw> ahh sure enough that's the issue I was seeing, or rather, what was still producing upstart scripts in etc/init  in my debs
<blackboxsw> +1
<blackboxsw> powersj: ahh that's in progress
<blackboxsw> good find
<powersj> blackboxsw: as in there an MP to review yet?
<blackboxsw> it's the fact that httppretty in those tests isn't mocking the min datasource version
<powersj> ah ok I do recall you talking about this
<blackboxsw> so it's reaching out to the internet and timing out
<powersj> I didn't know that it resulted in long running tests as well
<powersj> yeah
<blackboxsw> ok, larsks branch is up which would fix that. but it has a needs fixing comment
<blackboxsw> smoser: approved https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/329946
<blackboxsw> thanks for the fix
<blackboxsw> powersj: https://code.launchpad.net/~larsks/cloud-init/+git/cloud-init/+merge/329660 is the 'fix' but needs a tweak before it lands.
<powersj> yeah was looking at it - his CI run resulted in much more normal times too
<blackboxsw> larsks: do you know if you'll have cycles on the branch  https://code.launchpad.net/~larsks/cloud-init/+git/cloud-init/+merge/329660 ?
<blackboxsw> if not, I think we might patch that branch and try to get that landed as the breakage (the unit tests timing out when hitting the unmocked metadata min_version) is now something we are seeing in our CI too
<blackboxsw> thanks for this powersj  https://bugs.launchpad.net/cloud-init/+bug/1714117
<ubot5> Ubuntu bug 1714117 in cloud-init "tox: long CI times" [Undecided,Confirmed]
<blackboxsw> looking over the larsks branch, I think I'd like to limit a bit of the mocking that is being done as well as the tests in his branch are also mocking out all of the min_metadata_version urls for all metadata when we only really need to mock instance-id.
<blackboxsw> so I'll probably tweak the branch a bit and try to only mock the necessary minimum content w/ httpretty
<blackboxsw> powersj: I like your nosetests params, I'd like to capture that in tox.ini too (maybe just a comment about the additional params so we can check for costly unit tests easily in the future
<powersj> blackboxsw: nosetests --with-timer --timer-ok 1 --timer-warning 5
<powersj> that uses the nose-timer plugin
<smoser> but you'd have to pip install nose-timeer
<smoser> i think
<powersj> yes
<stanguturi> I just updated my 'merge request'. CI build failed https://jenkins.ubuntu.com/server/job/cloud-init-ci/241/ The failure is something related to the infrastructure and not at all related to my change.
<stanguturi> How to rebuild the failed jenkins build?
<powersj> stanguturi: yep, looks like we had issue getting to a mirror. I can re-kick shortly
<blackboxsw> hi sankar. good to see you again
<stanguturi> Thanks Josh, Chad
<powersj> stanguturi: CI is running again, if it fails again let me know.
<blackboxsw> as long as unit tests succeed there, I'll land that branch today stanguturi
<stanguturi> Great. Thanks Chad
<blackboxsw> powersj: larsks I've got it on the CI unit test fixes/proper mocks
<powersj> sweet
<blackboxsw> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+ref/bug/ec2-tests-unmocked-metadata for the win. I'll put it up for review
<blackboxsw> it also uses httpretty.HTTPretty.allow_net_connect = False   which raises a MockError if we try to reach out to anything on the internet not mocked
<rharper> nice
<blackboxsw> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/330040 is done powersj. it should help
<powersj> blackboxsw: style will fail, but I +1 with comment
<blackboxsw> powersj: my wife tells me that a lot
<powersj> also there are still two long-ish running tests that we should fix in another bug
 * blackboxsw fails style
<powersj> https://paste.ubuntu.com/25440455/
<powersj> LOL
<blackboxsw> powersj: actually it looks like a httpretty problem too.
<rharper> powersj: is that faster?  what was the the previous time, I thought you said 20 minutes or something crazy
<blackboxsw> I'll try to grab those too in this branch
<blackboxsw> s/httpretty/unmocked httpretty urls
<powersj> these ran 20 seconds for each enviornment so ~1min and 40 seconds total versus 20 minutes ;)
<rharper> heh
<powersj> if we fix those other two tests we can speed things up enough people might run tox before pushing ;)
<rharper> wha? I always run tox; doesn't take that long
 * powersj points at blackboxsw 
<rharper> heh
<rharper> tox -e flake8
<rharper> yeah
<blackboxsw> rharper: if I  time tox -e py27 tests/unittests/test_datasource/test_ec2.py -- -s  it takes 0m6.011s now, without the changeset 0m20.199s
<rharper> blackboxsw: that's a nice win
<blackboxsw> also certain environments seem to timeout faster.. as larsks saw the tests take ~20 mins when he runs in centos env I think
<blackboxsw> I'm making a change for the two openstack tests now
<powersj> sweet
<blackboxsw> hmm so, powersj I think we need to worry about those two in a separate bug, it's spending most of it's time I believe in a unittest callback  which does pattern matching and processing on every metadata file request. instead of completely mocking each url under the metadata service, so I think it's just a costly way to setup the mock (with a callback each time)
<powersj> yeah no need to pile on
<powersj> separate is preferred anyway
<sather> is there a way to enforce exeuction of only one module from the user-data.txt file?
<sather> like, if a specific module is present, only execute that module and do not execute any others?
<blackboxsw> bah powersj https://jenkins.ubuntu.com/server/job/cloud-init-ci/242/console we hit issues with scratch on the builder I think.
<blackboxsw> E: 10mount: mount: special device /home/jenkins/ubuntu/scratch does not exist
<blackboxsw> E: xenial-amd64-c24fe5b1-313c-4f25-b2e1-58283e767c03: Chroot setup failed: stage=setup-start
<blackboxsw> Chroot setup failed
<blackboxsw> E: Error creating chroot session: skipping cloud-init
<powersj> Shoot
<powersj> I know what the issue is I can fix shortly
<rharper> sather: in general no; however if you want to customize which modules get run; you can modify /etc/cloud/cloud.cfg which has a list of which modules will run at the different stages.  It's likely if you only run one stage that the instance generally won't be configured properly but maybe in a custom image that's not an issue
<rharper> sather: also, one can run single modules via cloud-init single --file=user-data.txt --force always --name cc_modulename; in case you just need to re-run one module
<blackboxsw> thx rharper I had a doorbell, I think we need the "--frequency always" as in:    sudo cloud-init single --file=user-data.txt --force ---frequency always --name cc_modulename
<rharper> blackboxsw: yeah
<rharper> honestly, I think the frequency should be defaulted to always if we run in single mode
<rharper> that's not a typical path, why make users do that
<sather> rharper: thanks for the info. I'm just trying to settle a security concern where we have a custom module, and only want that to run via cloud-init with a specific datasource...
<rharper> sather: interesting;  modules run independent of datasources in general
<rharper> but the module is called with the cloud object which has a datasource in it; the module could check the datasource type (or attribute) and exit out if not the datasource you want
<blackboxsw> rharper: any way for us to close https://code.launchpad.net/~wesley-wiedenmeier/cloud-init/+git/cloud-init/+merge/317589. I don't seem to have perms to mark that branch as merged
<rharper> blackboxsw: is it merged? if it's merged or superceeded, we usually "reject" and add comment about what we really did
<rharper> I'll update that
<rharper> comment that we're merged (via different branch) so marking it rejected (sorry for the harsh language)
<sather> I was actually hoping for a logic like this
<blackboxsw> I think the docstr are Reject, the actual feature name has already been mergedd
<sather> if specific-datasource; then run only this custom_module
<blackboxsw> thanks rharper
<sather> the specific-datasource is something someone did in-house
<sather> to allow for usb to be read by cloud-init
<sather> but we now just opened up our box to run any code if someone plugs a usb into it
<sather> :\
<rharper> heh
<rharper> sounds like the custom module could use some tightening up
<sather> I guess I was wondering if it's possible to do
<blackboxsw> hrm, so in-house modules should be pluggable (as cloudinit/stages calls find_module on anything installed in cloudinit/config/cc_*py. Right it seems that module could be coded to reject other datasources from inspecting cloud.datasource
<rharper> in general the cloud.cfg controls the list of modules that run by default;  It sounds like you could use a custom datasource which could be derived from say config-drive, which checked various usb devices/features before running
<sather> if custom_module present in cloud-config; don't execute any other modules ;p
<rharper> and the module itself can check that your specific derivative datasource was run before attempting to handle things.
<sather> cloud-config passed from userdata
<rharper> well, generally there doesn't seem to be a way to disable the other modules dynamically that I can think of;  but maybe smoser would know if I'm forgetting a way to do that
<sather> rharper: thakns for the help
<blackboxsw> rharper:  so to be certain, even if a custom module is loaded on an instance w/ cloud-init installed, that custom module doesn't get run even though it's discovered if it isn't listed in /etc/cloud/cloud.cfg right.
<blackboxsw> so the goal is to prevent direct injection of custom modules in cloud-init without express intent as defined in /etc/cloud/cloud.cfg right
<blackboxsw> though I *think* that custom module could be run via "cloud-init single" even if it's not listed
<blackboxsw> just validated w/ cc_ntp
<rharper> sather: sure, let us know if you get stuck or something else isn't working and we'll do our best to help you along the way
<blackboxsw> rharper: ok validated that a module name that isn't present in /etc/cloud/cloud.cfg can *not* be run even on commandline w/ --force
<rharper> blackboxsw: hrm;  I know the design allowed for user-date to write out modules to be run via user-data; so I may be missing something
<blackboxsw> 2017-08-31 22:08:27,056 - cc_ntp.py[DEBUG]: Skipping module named cc_ntp, not present or disabled by cfg
<blackboxsw> rharper: yeah I thought so too. poking around
<blackboxsw> "honestly, I think the frequency should be defaulted to always if we run in single mode"  :  +1 on that
<powersj> new cloud-id bug: LP: #1714358
<ubot5> Launchpad bug 1714358 in cloud-init "ds-identify does not find CloudStack datasource" [Undecided,New] https://launchpad.net/bugs/1714358
<powersj> blackboxsw: here is a bug for the OpenStack tests that are lengthy https://bugs.launchpad.net/cloud-init/+bug/1714376
<ubot5> Ubuntu bug 1714376 in cloud-init "unittests: OpenStack DS escape or timeout" [Undecided,New]
#cloud-init 2017-09-01
<blackboxsw> thanks powersj
<smoser> blackboxsw, i followed up on ds-identify bug.
<smoser> adn the one other new one there.
<rharper> powersj: maybe we should enable a jenkins run with that nose-timer ?  that's super nice to track every so often
<smoser> rharper, you have to add nose-timer to tox environment.
<smoser> not opposed to that happenting
<rharper> smoser: yeah; just figured that we don;'t all need to do that if we had a daily/weekly checkup via a special jenkins run
<rharper> either is fine
<smoser> its not really a huge deal to add it though
<smoser> compared to a pita for some jenkins job to kind of insert it.
<blackboxsw> morning folks
<smoser> o/
<powersj> I'd like to get a job to track over all time and if it gets too high fail.
<powersj> Just for tox
<smoser> powersj, well, by design it needs to go up over time
<smoser> at least blackboxsw is bent on that happening.
<smoser> ie, total time is somewhat related to number of tests, and a goal is to increase number of tests.
<blackboxsw> yeah, I feel it's a failure if our tox runs don't take longer
<blackboxsw> :)
<rharper> I thought jenkins jobs were easy? some template and a bash script ?  but smoser I think you're right in case someone wants to run it on their own; make it easy in our tox runs
<powersj> Yes but to go from 5 or 6 seconds to 20 is a little unexpected
<blackboxsw> as in, we didn't add more unit tests
<blackboxsw> agreed
<smoser> rharper, well,. figure out how to programatically
<smoser>  a.) create the python3 tox environment
<blackboxsw> 5 to 20 seconds means a poorly written test
<smoser>  b.) insert the nose package (pip install)
 * powersj relocates 
<smoser>  c.) run tox -e py3 and and in the '--nose-timer' args without changing the default command or arguments thaat would noramlly run with
<smoser> across possible changes to the command in trunk
<smoser> it seemed easier to me to just have trunk add the --nose-timer and even run with it.
<smoser> and kind of... why *shouldnt* you be shown that the test you're adding is the slowest one
<powersj> we could also have it show only the top 10 slowest jobs
<rharper> smoser: +1 on showing you you're the slowest
<blackboxsw> bikeshed  question--  datasource method names:
<blackboxsw> DS.get_instance_data -- crawls all instance metadata, userdata vendor data but makes no changes to the cached content
<blackboxsw> DS.refresh_instance_cache -- calls get_instance_data and applies it to the cache
<blackboxsw> any other/preferred naming suggestions? I know larsks had a suggestion about that
<blackboxsw> powersj: rharper smoser ^ any paint preference you want to throw on this shed when I rename them and standardize datasources?
<powersj> nope
<rharper> blackboxsw: generally I would expect get_instance_data to return cached data (unless you pass a nocache or something like that)  we do that with the blkid bit (and blkid itself will auto cache unless you say otherwise)
<rharper> that is most callers are going to want what's in the cache; some callers are going to know they want to ignore the cache and pull fresh
<blackboxsw> good thought, I like the idea of a no_cache param too
<smoser> i think there are maybe 3 functions oon a datasource object related to metadata
<smoser> a.) crawl the metadata service, getting everything. return that data in a object that is datasource specific.
<smoser>   (ie, the one on amazon doens't look like the one on openstack)
<smoser> b.) translate that data from 'a' into the "standard" format
<smoser> c.) update the datasource's cached values from 'b'
<smoser> blackboxsw, ^
<blackboxsw> smoser: thanks for this. +1 on that thought. right so we might need an intemediary translate implemented that could handle raw -> standard/flattened from each datasource
<blackboxsw> merged powersj https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/330111
<powersj> blackboxsw: thx!!
<blackboxsw> powersj: did you link that branch to a trello card?
<blackboxsw> if not I might attach it to https://trello.com/c/RYkSTMi8/319-ci-build-times-increasing
<powersj> blackboxsw: go for it
#cloud-init 2019-08-26
<tribaal> blackboxsw: Odd_Bloke: Nice! I'll review the SRU for what I can today (enabling -proposed)
<Nick_A> what is the correct way to pass a fqdn via user data?
<Nick_A> nvm found it
<marlinc> Nick_A, what was it?
<tribaal> blackboxsw: it seems like one of my colleagues found a problem with our new datasource, but we can't really understand exactly what is happening. runcmd: in the #cloud-config doesn't seem to be picked up. Current hypothesis is that our merge of set_passwords to run always is triggering a version of https://bugs.launchpad.net/cloud-init/+bug/1532234 (list of cloud_config_modules is *overwritten*, not
<ubot5> Launchpad bug 1532234 in cloud-init "Merging with data in /etc/cloud/cloud.cfg does not work as expected" [High,Confirmed]
<tribaal> merged. Could you help confirm?
<Nick_A> marlinc just fqdn: ____
<Nick_A> It's not necessary though - default hostname sets it as a fqdn if specified that way
<chillysurfer> how are boot records "split" for cloud-init? looking through cloud-init analyze i see that a machine i just provisioned and booted up has 3 different boot records
<chillysurfer> how did cloud-init determine to have multilple boot records in this case?
<blackboxsw> chillysurfer: boot records are based on each time cloud-init sees an init-local 'start' log in /var/log/cloud-init.log.   That log line has the format:
<blackboxsw> Cloud-init v. <version> running 'init-local'
<blackboxsw> so analyze counts the number of those logs in your log file as they are emitted each boot
<chillysurfer> blackboxsw: ahh ok i see
<chillysurfer> thanks for the explanation!
<chillysurfer> blackboxsw: so if you run `cloud-init init --local` manually though it'll create another boot record even though there was no reboot?
<blackboxsw> yeah it's all handled in either cloudinit/analyze/dump.py:parse_ci_logline which creates event json objects and cloudinit/analyze/__main__.py:analyze_show which parses those parsed log events by type/name and formats the boot output messages
<blackboxsw> chillysurfer: true. that would emit that log line and analyze would then lie I believe
<blackboxsw> s/lie/be wrong
<chillysurfer> blackboxsw: got it makes total sense! thanks!
<blackboxsw> chillysurfer: just validated that behavior
<tribaal> blackboxsw: I think the use of mergemanydict is wrong :/ Instead, we should look for the particular key we are interested in ("cloud_config_module"), and merge the list by hand (if we find that key, look for a set-password, if you find it, replace it with the two elements list ["set-passwords", "always"]
<blackboxsw> tribaal: I assume you are looking at the -proposed SRU content for Exoscale :/
<tribaal> blackboxsw: correct. Well, I found the problem in Eoan...
<blackboxsw> tribaal: this is *good*, as we can try to resolve that on Eoan quickly and get that into the current SRU
<tribaal> but I'm afraid I don't know enough about the internals of cloud-init there
<blackboxsw> which just started friday
<blackboxsw> so no big 'loss' of test/devel time. I was just starting to test clouds today
<blackboxsw> tribaal: if you could post me access to an instance (ssh-import-id chad.smith) I can ssh into it and add rharper and we can poke around
<tribaal> blackboxsw: sure thing
<tribaal> blackboxsw: I created this instance with the following userdata: https://gist.github.com/chrisglass/fb0cf860be8cf01f456dfff8e162e004
<blackboxsw> tribaal: can you also file a bug against cloud-init (as we'll need one to get it fixed in the SRU/Eoan)
<tribaal> blackboxsw: ack
<tribaal> blackboxsw: actually let me file the bug first and link all the relevant stuff there
<blackboxsw> that'd be great tribaal thx
* blackboxsw changed the topic of #cloud-init to: Reviews: http://bit.ly/ci-reviews | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting Sept 02 16:15 UTC | cloud-init v 19.2 (07/17) | https://bugs.launchpad.net/cloud-init/+filebug
<tribaal> blackboxsw: https://bugs.launchpad.net/cloud-init/+bug/1841454
<ubot5> Launchpad bug 1841454 in cloud-init "Exoscale datasource overwrites *all* cloud_config_modules" [Undecided,New]
<tribaal> blackboxsw: I imported your pubkeys in "ssh ubuntu@159.100.241.237"
<tribaal> blackboxsw: it's a test machine on our preprod, so you can break anything you want
<tribaal> bonus points if you manage to break anything more than the instance itself :)
<blackboxsw> haha! thanks, tribaal ok, yeah something going on with datasource config merging order in stages.py. _read_cfg. I'll refresh on why that merge is being overridden instead of merged ther .
<blackboxsw> tribaal: as Azure does the same type of thing :/
<blackboxsw> ahh your builtin is setting all of cloud_config_modules to your list. as you were supposing, we need to only augment  the ds config on disk with your defaults. I'll work up something.
<flipsa> Hi! I ran into a weird problem / edge-case, where ds-identify does or does not correctly detect a NoCloud datasource, depending on which version of util-linux (blkid) is installed on the system. Could somebody spare a few minutes to have a look and decide if this should be fixed in cloud-init or in the third party software I am using (xen orchestra)? I described my findings in more detail here: https://
<flipsa> github.com/vatesfr/xen-orchestra/issues/4449 Thanks!
<rharper> flipsa: thanks for reporting it; can you run a cloud-init collect-logs in the failing case?  and ideally open a bug in launchpad ?  I'd like to see what udevadm info --query=all /sys/class/block/xvdXX shows so we can see what sorts of properties were on the device;
<blackboxsw> tribaal: thanks for access to the system. I have a fix for #1841454 . I  can get it to run the modules each boot now https://pastebin.ubuntu.com/p/wTtf9JYDHs/
<blackboxsw> tribaal: rharper I'll put up a branch for this fix shortly (SRU-regression/ Eoan bug)
<rharper> nice
<blackboxsw> rharper: also openstack v2 related. fixing idempotent normalize_route works wonders for fixing unit test issues
<rharper> \o/
<rharper> I thought it might; just prevents mutuation in paths which call it multiple times
<blackboxsw> https://paste.ubuntu.com/p/kgxN8qJfcM/
<rharper> I'd like an eye-catcher though so we can see where we're getting multipath passes through
<rharper> oh
<rharper> nasty
<blackboxsw> yeah I think this mutation only happened when prefix was /0  for default routes
<blackboxsw> needed a none check instead
<rharper> indeed
<blackboxsw> +1 on tracking the multiple callers for normalization as well will add some debug
<tribaal> blackboxsw: \o/
<flipsa> rharper: https://bugs.launchpad.net/cloud-init/+bug/1841466
<ubot5> Launchpad bug 1841466 in cloud-init "ds-identify fails to detect NoCloud datastore with LABEL_FATBOOT instead of LABEL (change introduced recently in util-linux-2.33-rc1)" [Undecided,New]
<flipsa> rharper: if you need anything else, ping me...
<flipsa> rharper: the logs are almost all empty / non existent, because ds-identiy bails with exit code 1 and cloud-init doesn't even run...
<flipsa> but i guess the problem / cause of the problem are clear anyway. But if not I can clarify / do more testing
<rharper> flipsa: do you have the command that creates the partitionless disk with FAT16?
<rharper> flipsa: it seems reasonable to me to support both, but I'd like to be able to create one of these type of disks so we can verify the behavior and the fix
<flipsa> @rharper: I am not familiar with the Xen Orchestra code base, but this should be it: https://github.com/vatesfr/xen-orchestra/blob/master/packages/xo-server/src/fatfs-buffer.js
<flipsa> it's all done in a web app with nodejs. The external npm library they use is commented out on line 33
<rharper> flipsa: ok, my reading of the util linux code seemed to indicate to me that the filesystem label detection in blkid used to fallback to reading the boot record label as well, but blkid stopped doing that;  this is almost certainly going to cause more wide spread failures of where FAT16 labels were provided but now they aren't, instead all of the tools which use to get a LABEL value no longer get that.
<flipsa> yeah, as soon as people upgrade some will get bitten, quite sure
<rharper> now util linux is more accurate separating the two labels out; but now everyone else gets to fix their stuff as well.  I wonder what case really broke where blkid reported the boot label as the fs label
<flipsa> no clue
<rharper> flipsa: would you be able to duplicate the original disk?  is it something really small we could attach to the bug ?
<rharper> looks like dosfstools writes both boot record and volume label with the same value
<rharper> https://github.com/dosfstools/dosfstools/blob/master/src/boot.c#L755
<rharper> flipsa: I wonder if the ftafs would do that as well
<rharper> and we can certainly check if LABEL_FATBOOT is present as well
<flipsa> rharper: 10MiB
<rharper> yeah, I bet if you xz it, that'll drop smaller, either way if you don't mind attaching that to the bug would be great
<flipsa> rharper: see the bottom of my initial bug report where i did some experiments... dosfslabel incorrectly reads from the LABEL_FATBOOT field (if LABEL is not present), but it writes to the LABEL field but does not over-write LABEL_FATBOOT. seems like a bug as well imho
<rharper> well, it's not _incorrect_ by my reading, rather it checks in _both_ locations
<flipsa> rharper: will try. what's the size limit for bugs.launchpad?
<rharper> much bigger than 10MiB
<rharper> flipsa: we'll handle this in the cloud-init side with a fix to ds-identify to support checing LABEL_FATBOOT for cidata
<rharper> I would suggest also mentioning writing both label values to orchestra
<rharper> so they can generate nocloud data for images which won't yet have cloud-init with the fix
<rharper> I see no reason not to write the same value in both places for the cloud-init use-case
<flipsa> rharper: awesome!
<flipsa> yeah, think they were hoping that upstream fixes it, but you are right, until this is pushed to distros will take enough time to create problems for people...
<flipsa> rharper: will a dd image of the virtual disk do? don't think there's a supported way to export it from the Xen Orchestra interface
<rharper> is there only a single disk or do you have a rootfs disk and a separate cloud-init data disk ?
<flipsa> only one disk, no partitions
<rharper> and the disk is FAT16 ?
<rharper> that's surely not the Operating System disk ?
<flipsa> I am sure it's not the OS disk... but you might be right, i assumed that it's FAT16 from reading the source code of Xen Orchestra. cfdisk tells me: Label: dos, identifier: 0x00000000, no partitions only "free space". blkid says type "vFAT"
<flipsa> is there a diff between FAT16 and vFAT?
<flipsa> btw, this is the whole content:
<flipsa> .
<flipsa> âââ meta-data
<flipsa> âââ network-config
<flipsa> âââ openstack
<flipsa> âÂ Â  âââ latest
<flipsa> âÂ Â      âââ meta_data.json
<flipsa> âÂ Â      âââ user_data
<flipsa> âââ user-data
<rharper> that's just the metadata disk
<rharper> so yeah, just dd that
<flipsa> rharper: https://bugs.launchpad.net/cloud-init/+bug/1841466/+attachment/5284769/+files/xvdb.img.tar.gz
<ubot5> Launchpad bug 1841466 in cloud-init "ds-identify fails to detect NoCloud datastore with LABEL_FATBOOT instead of LABEL (change introduced recently in util-linux-2.33-rc1)" [Undecided,New]
<rharper> flipsa: thanks!
<blackboxsw> tribaal: rharper  I have a branch up that fixes #1841454
<blackboxsw> rharper: tribaal https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/371823
<blackboxsw> it allows for overriding existing sys_cfg options
<rharper> ok
<rharper> blackboxsw: so, why didn't it work ...
<rharper> vs. azure's build in ds config merging ? because of the list type ?
<tribaal> I guess yes
<tribaal> blackboxsw: I'll give it a try
<tribaal> blackboxsw: yep, building a local deb and installing to a fresh ewan instance makes it work indeed
<tribaal> runcmd runs, and get-passwords runs with frequency "always" as was initially expected.
<rharper> so, azure's built-in ds doesn't update or modify the modules list, so that's why it's not an issue;
<rharper> in azure, the ds.activate() method is used to clear our disk_setup/mount semaphors in the case that they need to reinitialize the disks due to migration;  exoscale could clean up the set_passwords semaphor that way.
<tribaal> rharper: instead of forcing the frequency?
<rharper> yes ;  in your template that you loaded, did you just copy the current value of modules and replace the one entry ?
<tribaal> rharper: I'm not sure I understand the question
<rharper> before I suggested that the datasource indicate the frequency, you said there was an in-image file which set this value
<rharper> which meant that stock images wouldn't run password on each boot
<tribaal> ah yes
<tribaal> so the in-image file does it "wrong" as well it turns out
<rharper> heh
<rharper> that's not surprising; the config is awkward in that you have to somehow know or read the current list;
<tribaal> we did not copy the list and change the value, we asssumed it would be merged and therefore just added the one entry there
<tribaal> actually that's how we detected the problem - with a bionic image in our prod. But the fix there is easy enough for us - we'll just copy the full list and tweak the frequency until we can forget about it all and use the proper datasource instead that should do the right thing for us
<tribaal> in other words as a workaround we'll copy the full list in our in-image file, until we can get rid of the in-image file :)
<blackboxsw> rharper: yeah sorry was on an errand. because of the list type, full list value overrode the entire cloud_config_modules list item
<blackboxsw> rharper: per overriding sysconfig values from the file system, I didn't see any existing cases where we did that in datasources after that initial /etc/cloud/config.d merge normally our ds's pull only ds_cfg under datasource: <dsname>: key/values
#cloud-init 2019-08-27
<blackboxsw> tribaal: for tomorrow, can you add some background on the need for the set-passwords override in the merge proposal https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/371823  .   I wasn't sure if there was a limitation in the platform that disallowed changing the instance-id in your metadata after a console operation to update passwords
<tribaal> hi blackboxsw - so regardig your question (I'll discuss it here and then add a comment on the PR):
<tribaal> indeed, we currently only set an instance-id per, well, instance. That is - each time a user decides "give me a new VM", our code sets this VM's id as instance-id. While it would be possible to change the instance id between an instance creation and its end of life, it feels weird to me - the user password is not a defining characteristic of the instance, I don't think resetting the password should
<tribaal> regenerate an instance id
<tribaal> however, resetting the user password (whatever the appropriate user is for that particular os - ubuntu, root, etc...) is a feature we offer our users and so would like to keep. The current flow is something like "press this button in the UI and we regenerate a password for you, then you reboot your instance and voila, new password"
<tribaal> but this requires the cloud-init module to be run "always" unless I'm mistaken. It's the same instance - but the password might have changed.
<tribaal> resetting the instance id would trigger other modules to re-run as well when resetting the password - for instance, I guess per-instance runs of cloud-config's "runcmd" would run again if we changed the instance id?
<blackboxsw> sounds good tribaal we were talking about this in standup at the same time as your ping here
<tribaal> ah :)
<tribaal> blackboxsw: if you want to hop in a call I can do a live demo of the feature, if that would help
<blackboxsw> sure tribaal. join if you'd like https://meet.google.com/bxm-azib-mtj
<blackboxsw> your ears are probably burning  :)
<blackboxsw> as we are talking about you
<tribaal> haha
<blackboxsw> tribaal: I have a fix, but I'm still working on unit test fixe s
<blackboxsw> will unfortunately be > 1 hr as I have an errand
<blackboxsw> pushing what I have if you want to look
<blackboxsw> deb won't build because of unit test failure
<tribaal> blackboxsw: ack
<tribaal> blackboxsw: rharper: the only thing I thought about in between is that the semaphore-nuking approach will be a little be strange from a logs perspective - the logs will show something along the lines of "set-passwords ran with frequency once-per-instance" yet it will run more than once per instance.
<rharper> tribaal: yes
<rharper> I think if we also log in the datasource what we're doing it's explainable
<tribaal> yes, that is probably enough
<blackboxsw> <- back
<blackboxsw> working on the unittest fixup now
<blackboxsw> and adding a log message about set-password frequency override
<blackboxsw> tribaal: rharper just force pushed the exoscale branch
<blackboxsw> unit tests fixed
<blackboxsw> testing now on my exoscale branch
<blackboxsw> s/branch/vm
<blackboxsw> bah get_ipath not available at that spot in the datasource. fixing
<tribaal> blackboxsw: thanks - I kicked off a test, but unfortunately it hit a bump in the road (nothing tragic as far as I can see though)
<tribaal> blackboxsw: I imported your keys in "ssh ubuntu@159.100.241.241" if you want to poke around (similar setup as the prvious machine - an Eoan snapshot on our preprod infra)
<blackboxsw> thanks tribaal
<blackboxsw> tribaal: that's stale :/
<blackboxsw> ahh tribaal I forgot to push my fix will have that up in about ~20
<blackboxsw> tribaal: I fixed for your tomorrow. testing again now on a live instance
<blackboxsw> rharper: tribaal I just tested current 576531c33b7c5bf215b5bc59dbff84eb9feaf981 of https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/371823 with success
 * blackboxsw runs to an errand
<rharper> blackboxsw: thanks, I'll review
 * tribaal tries again
<blackboxsw> rharper: per using sem_path = self.paths.get_ipath_cur('sem')  that was coming up None on Exoscale datasource, in my earlier pass on this, the datasource wasn't attached to self.paths for some reason
<rharper> it should be it's in the base class
<rharper> right ?
<blackboxsw> ahh wait I was using get_ipath() not get_ipath_cur. will try it now
<rharper> yes
<rharper> I saw the same thing with get_ipath vs cur
<rharper> smells like a bug, but I've not studied the Paths() object enough to know why
<blackboxsw> thanks  rharper. I'll make a note to file a bug about that behavior. Ok I can flip back to my previous unittests then which make use of get_ipath_cur then.
<rharper> cool
<blackboxsw> and I owe you a review
<blackboxsw> rharper: I reworked https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/371823
<blackboxsw> taking your suggestions thanks
<blackboxsw> pushing it now. and testing live
 * blackboxsw is reviewing https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/371906 now
<rharper> blackboxsw: ok
<blackboxsw> tribaal: validated latest on the vm you gave me. I've shut down that vm now so I have no more access.
<blackboxsw> rharper: per secondary vnic v1 or v2, why are we emitting netmask and gateway in v1, but not providing either a /<prefix> in addresses or routes?
<blackboxsw> rharper: also a followup unrelated question, would you expect that the global DNS defined in this sample should show up as sysconfig rendered config for eth0  as DNS1=172.19.0.12 https://paste.ubuntu.com/p/ZMVk5mNGNm/
<blackboxsw> that question is part of the openstack v2 network config outstanding unit test failures I have. I'm reflecting global DNS service entries onto every device that doesn't have dhcp or it's own DNS defined
<blackboxsw> .tox/py3/bin/nosetests tests/unittests/test_net.py on https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+ref/feature/openstack-network-v2-multi-nic
<blackboxsw> ahh so rharper v2 is lossy w.r.t. 'global dns'. Since v2 defines top top-level dns service values, (and we reflect dns values to each interface) sysconfig renderer doesn't know if it needs to setup /etc/resolv.conf
<blackboxsw> *Since network v2 defines *no* top-level dns service values*
<blackboxsw> I wonder if we need sysconfig renderer to be 'smarter' and check v2 config to see if nameservers config on each interface is the same and if identical, reflect that dns information up to top-level /etc/resolv.conf
<blackboxsw> or if not sysconfig renderer, maybe network_state needs to be smarted about it's dns_nameservers property
<blackboxsw> smarter rather
<rharper> blackboxsw: re: global dns, it's only interfaces that _have_ an IP, but not dns (and we skip subnets with dhcp)
<rharper> blackboxsw: so the goal with a global dns is to just ensure that it shows up on any interface that won't already get dns from somewhere else (like dhcp)
<rharper> blackboxsw: so, yes, in your paste, I would expect a DNS entry in ifcfg-eth0
<rharper> b2 does not define top-level dns, rather it's per interface under the type:  so ethernets: eth0: {nameservers: [], search: []}, eth1: {nameservers: [], search: []}, vlans: eth0.23: {nameservers: [], search: []}}}
<rharper> s/b2/v2
<blackboxsw> rharper: only that it is a slight difference in behavior now, instead of emitting an /etc/resolv.conf, we now append DNS1 on each etc/sysconfig/network-scripts/ifcfg-eth0
<rharper> which will end up in resolv.conf
<rharper> per distro networking
<rharper> otherwise, what's the point of DNS value in sysconfig ?
<blackboxsw> ahh right: dunno if we need to shore up that difference and have network_state.dns_servers track nameservers under each v1 'ethernets' object
<blackboxsw> good pt
<rharper> hrm, so looking at sysconfig
<rharper> we pick up dns_nameservers from the network_state iface subnet config
<rharper> that really shoud end up in resolv.conf
<rharper> blackboxsw: can you paste out a dump of your network_state dict after openstack -> v2 -> network_state ?
<rharper> I think we print the object in the unittest in a few places
<rharper> and do you have a branch up that I can see your converter ?
<rharper> blackboxsw: if we wanted to stay the same, then I think in network_state parsing v2, as we read in per-interface dns data, we could also append to the network_state.dns_nameservers (if the value isn't already present);
<blackboxsw> rharper: that last statement is what I was questioning. we could do something just like that to retain top-level DNS references, though if we had distinct DNS-per interface, just leave it hanging under the interface
<rharper> blackboxsw: some quick experiements in centos7, shows that nameserver values already in /etc/resolv.conf get overriden by DNS values from ifcfg files
<rharper> so I really think that per-interface DNS really is the right way since global isn't really a thing
<blackboxsw> rharper: so we could promote common dns values up to top-level via network_state.dns_servers settings. and allow unique dns to continue to live on the interface
<blackboxsw> rharper: that works. I can leave it per interface then
<rharper> blackboxsw: so, the only case I can think of that we should verify is that if we have a global dns in the source config, ala openstack, but no interface to put it on
<blackboxsw> rharper: we'll have global dns in network_data.json :/
<rharper> no, I mean ,no where to put it
<rharper> network_data.json either gives us a static ip
<rharper> or they dhcp
<blackboxsw> hrm right.
<rharper> so we always have a place to put the dns value
<rharper> it's not really *global*
#cloud-init 2019-08-28
<blackboxsw> right I guess if all interfaces dhcp, we would drop the dns on the floor in those cases
<rharper> yes
<rharper> expecting dhcp to provide the dns
<blackboxsw> right, ok that seems reasonable
<blackboxsw> so rharper sound reasonable to emit a warning then during network_data.json -> v2 conversion if we have global dns and no applicable interfaces?
<blackboxsw> at least it points to an impossible config presented to the instance
<rharper> blackboxsw: yeah, I think if we din we have dns in services, but no interface to attach it to, then a warning makes sense
<blackboxsw> ok will do
<blackboxsw> thx
<smoser> bug 1841697 probably needs attention
<ubot5> bug 1841697 in cloud-init (Ubuntu) "Upgrade corrupts 90_dpkg.cfg" [Undecided,New] https://launchpad.net/bugs/1841697
<rharper> db_get cloud-init/datasources
<rharper>  sed -n -e /^datasource_list:/!d -e s/datasource_list:[ \[]*// -e s, \]$,, -e p /etc/cloud/cloud.cfg.d/90_dpkg.cfg
<rharper> smoser: that doesn't like no spaces between the leading [ and trailing ]
<rharper> so: [ NoCloud, None ]  is OK, but [NoCloud, None] is not
<ivve> has anyone encountered a bug where write_files creates a dir instead of a file, regardless if there is a content?
<smoser> rharper: My only reason for raising it here is that it happend as a result of an SRU.  I don't know if anything changed in that area (I thought a new datasource or two was added), so I thought it might be a regression as a result.
<smoser> if it is just a matter of a person edited that file and wrote stuff that the crappy sed parser didn't like, then its nothing new while it shoudl be fixed isn't terribly important.
<rharper> smoser: its not an sru issue
<rharper> it was local config
<rharper> ivve: no, if you have user-data that you can share to demonstrate, please file a bug and we can look at it
<rharper> smoser: thanks for the heads-up
<smoser> rharper: yeah, i saw that and your mp. repsonded to the  mp with a suggestion.
<rharper> ohm, thans
<ivve> rharper: ok thanks
<rharper> ivve: thinking about it, write_files in cloud-init does an effective mkdir -p on the dirname of the path to the file;  and if the write failed, you would see the dir but not the content;  there should be an error in /var/log/cloud-int.log   ;
<ivve> thats exactly what happens
<ivve> 4 out of 8ish files becomes directories. i think it started when i fiddled with default user, but im not sure.. this template is 2500lines with well over 100k chars
<ivve> well not entirely true
<ivve> it creates a dir in place of the file
<ivve> say i have /path/to/file it creates /path/to/file/
<ivve> so instead of a file called file, i have a dir called file
<rharper> yeah, there should be an error in the cloud-init.log  and if you have just the write_files bits, that'd be helpful to reproduce  (I don't need the actual content of the files, just a version with similar paths, etc
<ivve> aye, i think i can provide originals
<ivve> ill just make a last check
<ivve> ok think i found it
<ivve> File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1418, in chownbyname
<ivve> guess ill try with root:root
<ivve> i think i had that before and it worked
<rharper> ivve: I see, if you don't provide a user:group in write_files content, then it will use the default user and default permissions which is root:root and 0644
<ivve> rharper: i provided elasticsearch:elasticsearch which is created in users section in cloudinit along with default
<rharper> ivve: ok, please file a bug and we can sort out what's going on;
<rharper> the log should show a write failure, but it's written by root (cloud-init) and then chmod'ed to match the config specified, so its quite strange to have a missing file here
<ivve> i was checking for spelling errors in elasticsearch but couldn't find any, i will try deploy with root:root and see if thats the problem
<ivve> perhaps the group isn't created
<ivve> damn myself for doing too many changes without commits
<ivve> hmm think i found the error now
<ivve> no_user_group: true
<ivve> and write_files is using the group of the user
<ivve> will verify, recreating the stack
<ivve> but its odd, because the group exists i groups
<ivve> elasticsearch:x:1000:elasticsearch
<rharper> interesting
<ivve> so initially i think i am at fault here
<ivve> but im suprised as well :)
<ivve> trying root:root now anyways just to block out the error
<ivve> i can always chown at runcmd
<tribaal> blackboxsw: I saw your branch landed - nice! Thanks. Do you know when the code is planned to hit eoan out of curiosity?
<ivve> rharper: looks like that was the problem. however group and user does exist
<ivve> and now i know what creates the directories
<ivve> its docker-compose
<ivve> mounting stuff that doesn't exist
<blackboxsw> tribaal: I expect I'm uploading today
<blackboxsw> probably within the next 2 hours. there is another 1-2 branches of ryan's that I'd like to get in too
<blackboxsw> one of them should land in ~15 mins and I'm reviewing the second now
<blackboxsw> tribaal: once uploaded, it won't be in official cloud images until tonight as you know
<blackboxsw> ~tonight :)
<tribaal> blackboxsw: sure, I'll run an apt-upgrade in our image thingy for preprod
<tribaal> it won't be *official* but that should be good enough
<tribaal> (for preprod at least)
<blackboxsw> +1
<rharper> https://paste.ubuntu.com/p/Xx4NpDN4SB/
<rharper> blackboxsw: ^^
<rharper> so yeah, v2 to sysconfig is going to write out the DNS0 values, so we may want to confirm if we need to have a resolvconf-only flag
<blackboxsw> +1 rharper
<blackboxsw> rharper: the master ifcfg template @ https://github.com/openSUSE/sysconfig/blob/master/config/ifcfg.template doesn't mention a supported DNS* config option.
<blackboxsw> that code I referenced can't possibly be the source of truth (too old)
<blackboxsw> but it was the best reference I'd found so far
<blackboxsw> also cat /etc/sysconfig/network/ifcfg.template  on my opensuse leap 15.1 lxc is showing basically the same, no DNS* config options allowed per interface
<blackboxsw> so I'm going with your suggestion, we'll need a resolveconf-only flag for suse to handle net config v2 'global' dns
<rharper> I don't see robjo around
<blackboxsw> yeah I was trying to tab-complete his nick :/
<blackboxsw> might send a message to the mailing list once the v2 branch is up
<blackboxsw> get some input if he has time
<rharper> y
<gcstang> I'm installing cloud-init on OL 7.7 is there any reason that the user home directory wouldn't have the permissions needed to create?
<rharper> gcstang: sorry;  cloud-init runs as root, so it has permissions to create directories in /home
<rharper> what do you expect to happen and what is actually happening ?
<gcstang> I have the user setup in cloud.cfg and during boot the cloud-init.service shows an error that it couldn't run mkdir(name, perm) when creating my user's home directory in /home/
<blackboxsw> gcstang: I think if possible, we'd like to see your userdata reproducing this failure in a bug.   Run the following on your system: sudo cloud-init query userdata   (and make sure it doesn't have any secrets).   If possible a bug filed to https://bugs.launchpad.net/cloud-init/+filebug and attach the tar.gz file emitted by "sudo cloud-init collect-logs"
<gcstang> blackboxsw the query command returns empty
<blackboxsw> gcstang: ok so on your vm you didn't provide any custom user-data then?
<blackboxsw> like providing #cloud-config to the vm
<blackboxsw> at launch time
<gcstang> like the metadata?
<blackboxsw> gcstang: correct, I was wondering if you were trying change default behavior on the vm you are launching to add new (non-default) users
<blackboxsw> by providing additional configuration metadata to the instance (we call that user-data)
<gcstang> the metadata should be supplied by our URL http://169.254.169.254/opc/v1/instance/metadata/ but I don't think cloud-init is finding that
<gcstang> the only thing provided is the ssh_authorized_keys
<gcstang> blackboxsw otherwise the only configuration is in the /etc/cloud/cloud.cfg
<blackboxsw> gcstang: ok, so sudo cloud-init collect-logs will help the most I think as it'll scrape /var/log/cloud-init.log for us (which should have logged the 169.254 crawl cloud-init tried).
<blackboxsw> it
<blackboxsw> will also show the tracebacks of user dir creation. (I'm wondering if it's opc default user on ubuntu that is causing an issue here)
<gcstang> blackboxsw tried submitting a bug several times and I get
<gcstang> Timeout errorSorry, something just went wrong in Launchpad.Weâve recorded what happened, and weâll fix it as soon as possible. Apologies for the inconvenience.Trying again in a couple of minutes might work.(Error ID: OOPS-44fe7d797c5d2d2342e5ea3320e3fc05)
<blackboxsw> interesting gcstang I've reflected that to our  IS department to check it out. gcstang can you see edit this bug? https://bugs.launchpad.net/cloud-init/+bug/1841816
<ubot5> Launchpad bug 1841816 in cloud-init "OL 7.7 fails to create a user" [Undecided,New]
<gcstang> blackboxswI can
<blackboxsw> all yours gcstang edit at will :)
<gcstang> wont' let me attach, same failure
<gcstang20> blackboxsw I was able to attach the logs and my cfg
<blackboxsw> rharper: followup sed for https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/371919
<rharper> thx
<blackboxsw> see what you think. I want to avoid python dependency in that shell script, and avoid a heavy copy/paste/hoist of 37+ lines of code from ds-identify
<rharper> blackboxsw: I don't see an update on the MP, did you update bug or MP ?
<blackboxsw> haha unsaved comments can't be seen.
<blackboxsw> fixed
<rharper> cool
<rharper> I agree w.r.t python or the bigger shell fix
<blackboxsw> the resulting sed should support  datasource_list
<rharper> I can't deny I like the simplicity of the python -c  =)
<blackboxsw> the following examples :     ds_list   : [1,2,3]    ds_list: [   1,2,3 ]   \s*ds_list:    [1             ,2, 3] etc
<blackboxsw> yeah I agree rharper on the python call, just didn't want to have to sort python2 vs python3 deps in the script
<blackboxsw> too
<blackboxsw> or have to fallback to another option if the dep wasn't available at the time for some reason.
<rharper> yaml wasn't always standard lib right ?
<blackboxsw> rharper: right, it wasn't initially. ~2013 maybe?
<blackboxsw> I just tested that sed suggestion on lxc, purging cloud-init, adding datasource_list: [1,2,3] to /etc/cloud/cloud.cfg.d/90_dpkg.cfg  and it gets correctly reformatted datasource_list: [ 1, 2, 3 ]
<blackboxsw> rharper: I was hoping to hold restarting the cloud-init SRU on the inclusion of this https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/371919 .
<blackboxsw> Do you think we should wait for it, or just restart the SRU with the exoscale fixes?
<rharper> hrm
<rharper> my initial thought is to not wait, as tonight or tomorrow will bring another thing to pick up
<rharper> I also don't think the error is that common, it requires manual manipulation of the file
<blackboxsw> ok will kickoff an upload to Eoan of tip of cloud-init then
 * blackboxsw wants one quick manual ec2 test on current SRU, just to make sure the world 
<rharper> blackboxsw: yeah, looks like just two commits, the exoscale and the oracle vnics bits
<blackboxsw> didn't break
<rharper> yeah
<blackboxsw> rharper: [ubuntu/eoan-proposed] cloud-init 19.2-24-ge7881d5c-0ubuntu1 (Accepted)
<blackboxsw> tribaal: ^
<blackboxsw> that'll have exoscale fix
 * blackboxsw now tries to regen an SRU including this content too
<rharper> great
<blackboxsw> and xenial bionic disco -proposed SRU upgrade test on ec2 doesn't error out. (there are no specific SRU bugs/code related to Ec2 in this sru)
<blackboxsw> rharper: here's what I was going to do debian/changelog wise for updating the existing SRU
<blackboxsw> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/371959
<blackboxsw> so I've added a second debian/changelog stanza for the incremented new version. both reference the current SRU bug number
<blackboxsw> I'm not referencing the upstream bug id, because we
<blackboxsw> have an SRU exception to handle validating that ourselves
<blackboxsw> "the upstream bug id" == for the exoscale fix
<blackboxsw> #1841454
<blackboxsw> bug #1841454
<ubot5> bug 1841454 in cloud-init "Exoscale datasource overwrites *all* cloud_config_modules" [Undecided,Fix committed] https://launchpad.net/bugs/1841454
<rharper> blackboxsw: hrm, so should changelog have two entries each with the SRU bug # ?
<rharper> I guess we already pushed to ubuntu/disco
<blackboxsw> rharper: I think I only pushed to chad.smith/ubuntu/disco.    I thought steve suggested that on a previous sru-regression fix we performed. I can separate it out, but then we have 2 changelog entries in the most recent commit that don't have bugs
<rharper> blackboxsw: you may be right, I'm just asking
<blackboxsw> I can double-check in #ubuntu-devel
<chillysurfer> trying to build an rpm from cloud-init source, i'm getting the error KeyError: u'RPM_BUILD_ROOT'. i've tried setting the env var to a path on my machine, but still not working. any thoughts?
<rharper> chillysurfer: I suggest using our tools/run-container
<rharper> chillysurfer: if you want to do it outside ,then you want to use packages/brpm
<chillysurfer> rharper: yep that's the script i kicked off, packages/brpm
<chillysurfer> and i get that key error
<rharper> and you have rpm-build and such installed ?
<chillysurfer> nope!
<chillysurfer> i'm guessing that's the problem
<chillysurfer> is there a list of deps for brpm?
<rharper> chillysurfer: not in there but in tools/run-container
<chillysurfer> rharper: the reason i'm doing this exercise is that it looks like our rpm doesn't inject the PACKAGED_VERSION into version.py
<rharper> ah, it uses tools/read-dependencies
<rharper> chillysurfer: I would suggest you use the tools/run-container
<chillysurfer> rharper: so it falls back to just VERSION without the package information
<rharper> that's how we create our srpms for upload to copr
<chillysurfer> and trying to unwind why that's happening
<chillysurfer> ok cool i'll do tools/run-container
<rharper> so, you can look at tools/run-centos  and see how that calls tools/run-container ,
<rharper> our ci does this, ./tools/run-centos 7 --srpm --artifact
<rharper> which sets up a centos7 lxd container, puts in the cloud-init source, runs all of the steps to prep the container environment, and then calls brpm with the right flags, and will pull out the srpm to your dir when done
<chillysurfer> rharper: awesome i'll definitely do that
<chillysurfer> rharper: dumb question... what controls the rhel repos? if this is for dumping to copr for centos/fedora, what handles the pkg distribution for rhel regarding cloud-init?
<rharper> chillysurfer: we have a copr project repo for centos; we've not access to the downstream repos
<rharper> our CI publishes daily rpms for centos7;  I've a branch to fix up our py3 build so we can publush on newer fedoras and centos8; but that' needs to get reviewed/landed
<rharper> https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/368845
<rharper> we maintain two other repos on copr, el-stable and el-testing;  el-stable hasn't been updated in quite some time, mosty for centos6 support;  el-testing get's any of our new release (19.1, 19,2) and whenever we do an SRU
<rharper> chillysurfer: the downstreams decide what to pick up and when;  typically depending on the downstream, some pick up newer cloud-inits at the point releases (19.1, 19.2, etc)  others keep an older release and then backport fixes.
<chillysurfer> ah i see
<chillysurfer> that makes sense
<chillysurfer> rharper: do they pick up source packages?
<rharper> I suspect they checkout the release tag from git
<rharper> we tag and sign point releases
<chillysurfer> rharper: ahhh i see. so in that case you aren't surprised that the packages/redhat/cloud-init.spec.in isn't applied tot he pkg?
<chillysurfer> rharper: the reason i'm bring this all up is that `sed -i "s,@@PACKAGED_VERSION@@,%{version}-%{release}," $version_pys` doesn't seem to be happening, which is specified in cloud-init.spec.in for the rpm
<rharper> the spec file is a template, so it get's rendered during the install, IIRC
<rharper> chillysurfer: see packages/brpm ; it calls tools/read-version and then runs the generate_spec_contents()
<chillysurfer> rharper: right exactly. but if the downstream maintainer of cloud-init isn't calling packages/brpm that would explain why the version isn't getting injected right?
<rharper> they don't use our spec file
<chillysurfer> rharper: they use their own spec file? that would explain why PACKAGED_VERSION remains unchanged
<rharper> chillysurfer: sounds like it could be a downstream bug in their build spec
<rharper> it's not always clear they want cloud-init --version to match what the package version value is
<blackboxsw> ok rharper vorlon confirmed on reusing existing SRU bug
<blackboxsw> will push xenial and bionic for SRU re-review
<rharper> for example if they're backporting features into an older release; they may not want to bump the version value that cloud-init returns
<rharper> blackboxsw: ok, thanks
<rharper> blackboxsw: I'm going to run an errand in just a few minutes, when I get back, I'll review the bionic/xenial MPs if you've got them up
<blackboxsw> thanks.rharper  bionic: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/371962   xenial: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/371963
<blackboxsw> rharper: dugtrio running out of space? https://jenkins.ubuntu.com/server/job/cloud-init-ci/1088/console
<rharper> maybe
<blackboxsw> couldn't create chroot
 * blackboxsw logs in to see what we need to clean up
<rharper> no, maybe just not configured
<rharper> that job typically ran on tork, but the queue has been sprayed across additional nodes
<rharper> so this looks like setup fallout
<rharper> blackboxsw: send paride a note
<rharper> maybe you can reschedule that job on tork
<blackboxsw> rharper: yeah I'll try a reschedule, and will send a note
<chillysurfer> rharper: ah ok that's really good information
<tribaal> blackboxsw: ack. I'll spin an Eoan template now and will torture it tomorrow
<chillysurfer> rharper: what's the copr repo for cloud-init?
<blackboxsw> chillysurfer: daily builds are https://copr.fedorainfracloud.org/coprs/g/cloud-init/cloud-init-dev/
<blackboxsw> testing: https://copr.fedorainfracloud.org/coprs/g/cloud-init/el-testing
<blackboxsw> we'll be updating testing today/tomorrow, but generally we don't updated except when we are performing an ubuntu Stable release update into xenial, bionic, disco (>= monthly)
<chillysurfer> blackboxsw: great thanks!
#cloud-init 2019-08-29
<ivve> is users module really run _after_ write_files?
<ivve> crap, it is
<ivve> rharper: well the problem is kinda obvious now, there is no user yet when write_files tries to use "owner:" since it is created in "users-group" afterwards
<ivve> simply put: File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1418, in chownbyname uid = pwd.getpwnam(user).pw_uid KeyError: 'getpwnam(): name not found: elasticsearch'
<ivve> not a bug, simply working as intended. the user must exist before write_files is executed..
<rharper> blackboxsw: are we ready to split up the sru manual testing?
<blackboxsw> rharper: sil2100 mentioned an hour or so ago in #ubuntu-release that he'd be reviewing the new upload
<rharper> cool
<blackboxsw> we are still waiting on the queu
<rharper> so just waiting to hit -proposed ?
<blackboxsw> we are still waiting on the queue
<blackboxsw> yeah
<rharper> can you upload to the ppa
<rharper> we can test with those
<blackboxsw> ohh right, the daily ppa, will do
<rharper> the _proposed_ ppa
 * blackboxsw checks to refresh
<rharper> not daily, plese
<blackboxsw> +1
<rharper> https://launchpad.net/~cloud-init-dev/+archive/ubuntu/proposed
<rharper> looks like we forgot on the 19.1 sru
<blackboxsw> rharper: I think we may need to add that step to the SRU procedure docs https://github.com/cloud-init/ubuntu-sru/
<blackboxsw> I'll add that as I've forgotten it
<rharper> doing that now
<rharper> =)
<blackboxsw> thanks rharper
<blackboxsw> rharper: the dput or the docs?
<rharper> the docs
<rharper> you can dput
<rharper> =)
<blackboxsw> doing that now
<blackboxsw> uploads complete. waiting on builds https://launchpad.net/~cloud-init-dev/+archive/ubuntu/proposed/+packages
<blackboxsw> ok our -proposed PPA published. I'm going through manual again EC2 rharper
<rharper> blackboxsw: ok, i'll grab azure
<blackboxsw> rharper: I added https://trello.com/c/fuJm4Rrl/25-publish-to-cloud-init-dev-proposed-repo-to-begin-testing-while-awaiting-sru-queue-approval
<rharper> thx
<blackboxsw> rharper: I added you to https://trello.com/c/p2C9s3FH/10-run-manual-upgrade-cloud-tests and checked off the clouds we are trying to test so we don't collide
<blackboxsw> rharper: I guess we'll have to adapt our manual tests to source the ppa instead of official -proposed pocket
<blackboxsw> and maybe we should just extend build-and-push to also dput the changes file to our dev -proposed ppa to prevent forgetting that next round
<blackboxsw> adding that now
<rharper> blackboxsw: yeah, that's a good idea
<blackboxsw> Improving team efficiency ++ :)
<blackboxsw> cloud-init -proposed officially got approved
<blackboxsw> rharper: straw man is up for openstack v2 . I still have a v2 global dns promotion corner case to settle with unit tests. (and various v2 test_configdrive unit test fixes)  But, it 's something to glance over as I'll finish cleanup on the dns promotion behavior tomorrow https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/372009
<rharper> blackboxsw: +1
#cloud-init 2019-08-30
<blackboxsw> rharper: volley for tomorrow https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/371895
<matthiasbaur> Hey, I try to contribute to cloud-init and been told that I have to sign a CLA. As I'm not sure if the company I working for signed one, may I talk to you in private smoser?
<smoser> matthiasbaur: talk to powersj .
<matthiasbaur> powersj may I talk to you in private?
<iliv> hey guys
<powersj> o/
<iliv> I have run into what probably is a well-known issue by know but new to me, which is if I use root EBS volume that is greater than 250G (e.g. >= 300G) for a Debian Jessie AMI based EC2 instance, that root EBS volume is partitioned with what seems like a default fallback option. Namely, 2 partitions: swap and 8GB root partition. As I undersatnd cloud-init is responsible for doing the partitioning. Is that correct?
<rharper> iliv: cloud-init can do partitioning, and I believe certain platforms have some defaults; let me look at the ec2 datasource;   do you know what version of cloud-init is in your image? do you have a /var/log/cloud-init.log from the instance ?
<iliv> Cloud-init v. 0.7.7
<iliv> this is Debian Jessie AMI
<rharper> iliv: look at our code; couldn't init doesn't do any partitioning of the root device (by default) ; it may grow or resize a root partition and it can enable/create swap;
<rharper> iliv: typically images will have the root partition be the last partition on the disk, so that cloud-init can grow the last partition size to end of disk adn resize the rootfs as well
<rharper> so I wonder if the AMI partition layout is the source of the issue;
<iliv> it's kinda weird that it grows it only if ebs root volume size is <=250G
<rharper> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=784004
<ubot5> Debian bug 784004 in cloud-init "cloud-init growpart module fails to execute due to different growpart version" [Normal,Open]
<rharper> could be related
<iliv> just to be clear, when using 500G root EBS volume I get 2 partitions: 1 swap and 1 8GB root partition. the remaining disk space is simply unallocated.
<rharper> yes, this sounds like the above bug
<rharper> well, maybe something else but related, so ebs size of 200G and you get the right size ?
<rharper> in any case; the bug mentions a downlevel version of growpart in Debian Jessie which was the source of the root partition not being resized correctly
<rharper> there are a couple workaround mentions w.r.t including an updated version of growpart in the image
<iliv> > so ebs size of 200G and you get the right size ? // up to 250G
<iliv> I tested 500G, 400G, 300G, 250G and 200G
<iliv> if it's <=250G I get a 250G ext4 root partition
<iliv> if it's >= 300G, I get only 8GB ext4 root partition
<rharper> could be a new bug then; including the /var/log/cloud-init.log will be help, whether you file a downstream or an upstream bug;  if you use newer debian AMI (with newer cloud-inti) do you see the same issue ? or is it jessie only ?
<iliv> https://dpaste.de/Zvbt/raw
<iliv> I tested this with the Jessie AMI only
<rharper> https://github.com/andsens/bootstrap-vz/issues/359
<rharper> looks like that issue
<rharper> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=868875
<ubot5> Debian bug 868875 in cloud-utils "cloud-init growpart fails to grow root volumes larger than 256GiB" [Normal,Open]
<rharper> already filed in debian
<rharper> and it looks related to the util-linux version in Jessie;
<iliv> you do all the dirty work for me :D
<iliv> thanks, appreciate it!
<rharper> np
<iliv> "No longer marked as found in versions cloud-init/0.7.7~bzr1156-1~bpo8+1."
<iliv> does that mean this bug was fixed in that version of cloud-init?
<rharper> doubtful
<rharper> it was a growpart issue (which cloud-init executes) and is part of the cloud-guest-utils package
<iliv> so, nothing cloud-init can do here as long as we're stuck with this particular AMI?
<iliv> I saw a reference to cloud-initramfs-growroot in one of the bug reports
<iliv> not sure if that could help
<iliv> but the theory goes that the growing would happen very early in the process before the filesystem is mounted
<iliv> so on the surface sounds like an option to explore
<rharper> you can disable growpart via cloud-config and then perform it manually
<rharper> however, if the issue is related to the tools in the OS (util-linux package) being downlevel, then you'd somehow need to get the newer version of the tools and use them instead
<iliv> oh, I tried that but it failed
<rharper> https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/1244662
<ubot5> Launchpad bug 1244662 in util-linux (Ubuntu) "growpart reverses changes because of partx failure" [Medium,Fix released]
<rharper> this was fixed in newer util-linux
<rharper> so, likely need t pull in a newer util-linux into the image and then try the growpart again manually
<iliv> Installed: 2.25.2-6
<iliv> "This bug was fixed in the package util-linux - 2.20.1-5.1ubuntu12"
<iliv> however, that is ubuntu package
<rharper> yes, but 2.25 is > 2.20
<rharper> it was forwarded to upstream, I can see if we still have delta, but I suspect it was accepted upstream
<iliv> well, if that's the case I saw exactly the same error message as shown in that Launchpad bug report when I tried to run growpart /dev/xvda 2
<iliv> https://dpaste.de/NiFr/raw
<rharper> https://github.com/karelzak/util-linux/commit/84ac311212c06963d616b1b9a40644842f9adabd ; landed upstream and released in 2.24,  https://lwn.net/Articles/581276/
<blackboxsw> tribaal: if you have time next week, cloud-init SRU v 19.2-24-ge7881d5c-0ubuntu1 got re-queued for Xenial/Bionic and Disco with the Exoscale fixes for cloud_config_modules which we published in tip this week to Eoan.   If you get a chance to enable <series>-proposed on an Xenial or Bionic instance you have, then you can sudo cloud-init clean --logs;  apt-get remove --purge cloud-init; apt-get install cloud-init
<blackboxsw> #version 19.2-24 sudo reboot   to test that Exoscale comes up properly
<blackboxsw> I'll send an email about that too.
<blackboxsw> rharper: thanks on SRU, minor comments and please land https://github.com/cloud-init/ubuntu-sru/pull/36/files and https://github.com/cloud-init/ubuntu-sru/pull/35/files .   Or your could file a 3rd PR that just updates all sru-templates with the new "local" template variables
<blackboxsw> I'm adding it the the ec2 manual run I've kicked off now.
<rharper> blackboxsw: ill check, I'll likely put up a separate PR for the template file
<blackboxsw> +1 rharper
#cloud-init 2020-08-24
<makara1> what's a nice way to serve cloud-init configuration?
<makara1> put 'python3 -m http.server' in a unit file?
<makara1> can I get cloud-init to download a git repo and kick off a build?
<beantaxi> How can I do the following with cloud-init: 1. Have a cloud-config session that does once-per-instance setup 2. A bash script that continues that once-per-instance setup, and 3. A bash-script that executes once-every-boot. I see in the docs how I could create a multipart mime which contains a cloud-config and 2 scripts, but not how to order them or to associate them with lifecycle. Thanks!
<otubo> smoser, I'm reworking your code here: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+ref/fix/1788915-vlan-sysconfig-rendering
<otubo> smoser, should I add signed-off-by on the pull-request>
<otubo> ?
<meena> https://github.com/canonical/cloud-init/pull/521#issuecomment-678476067 â¬ï¸  this the use-case scapy? for https://scapy.readthedocs.io/en/latest/introduction.html
<otubo> smoser, I'm gonna just leave a signed-off-by with your name, if you don't want it I can remove it. :)
<rharper> morning, blackboxsw_ fyi the daily PPA builds for bionic/xenial are failing,  https://launchpadlibrarian.net/494499147/buildlog.txt.gz  https://launchpadlibrarian.net/494499148/buildlog.txt.gz
<blackboxsw_> morning folks.
<smoser> otubo: thats fine with me.
<smoser> meena: we have a use-case for cloud-init to be able to do a dhcp request and read the response.
<smoser> one path to that is through scapy (and a seemingly simple one)
<smoser> but scapy's dependency stack is not small :-(
<smoser> the difficulty we had wehn looking at a dhcp client in python is that all the example solutions do not use raw sockets.
<smoser> and thus... have an ip address already.
<smoser> s/have/assume/
<meena> @smoser scapy would also allow to do a lot of the network discovery work without heaps and heaps of guesswork
<meena> smoser: what makes scapy's dependency stack so big? i thought it only depends on libpcap?
<smoser> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=907235
<ubot5> Debian bug 907235 in python3-scapy "python3-scapy: Please provide package without large 'Recommends'" [Normal,Fixed]
<meena> da fuck
<smoser> that is fixed in debian (and i think in ubuntu as aresult), but not in all versions of ubuntu and not in many other os
<meena> who does that? why
<meena> the way new packages get into old versions of debuntu remains a mystery
<minimal> Hi folks, so what's the current ETA for 20.3?
<meena> "when it's done" ;P
<minimal> is anything ever done? ;-)
<smoser> meena: they *don't* get into old versiosn of ubuntu.
<smoser> by design. stable things are stable.
<blackboxsw_> smoser: thanks for the review last week https://github.com/canonical/cloud-init/pull/543 in response. no longer handling non-decodable or non-gzipped content for user-data
<smoser> ACK'd thanks
<beantaxi_> Hello all ... the docs give this example of combining a cloud-config and a bash script in a single mime multipart: % cloud-init devel make-mime -a config.yaml:cloud-config -a script.sh:x-shellscript > user-data
<beantaxi_> However, I'm not sure how that specifies the run order of cloud-config vs the shellscript, or how it specifies how often the shell script runs: per once, per instance, or per boot. Thanks!
<smoser> x-shellscript run per-instance.
<smoser> cloud-config is parsed before x-shellscript.
<beantaxi_> smoser: Thanks. So cloud-config runs before shell scripts no matter what. That's fine for me, and I'm guessing if that's a showstopped then run-cmd is your friend. Just checking.
<meena> smoser: what about â¦ fixing broken packages in older versions of Debuntu? like when somebody accidentally made switched Recommends and Suggests?
<smoser> runcmd runs ~shell-script time frame.
<smoser> actually... odly based on filename. those actually (iirc) get thrown in a dir, and then run with rhnparts. and the runcmd script name is 'runCmd'. so its wierd.
<beantaxi_> As for 'x-shellscript run per-instance' ... I'm afraid I'm not sure what you mean. I do see in the code, that it looks like lifecycle gets controller by the /var/lib/cloud/per-[once|instance|boot] folders, but I'm not sure how to use cloud-init to populate those folders. Thanks!
<smoser> meena: it all kind of depends.
 * meena is a big fan of "if broken it is, fix it you should"
<smoser> if you really reallly wanted to get that change in, you have to convince someone on the SRU team that it is important enough.
<smoser> i tend to be a fan of keeping behavior the same.
<smoser> as the primary value of a platform is to put things on top of. its harder to build on a platform if the platform is changing.
<smoser> thus.... the value of a "stable platform" is almost *entirely* the stability
<smoser> but... its not me that has to be convinced, and, while ubuntu has guidelines on such things, they're not always 100% consistently applied.
<meena> smoser: well, yes, but the behaviour did change, no?
<smoser> it changed from one stable platform to the next. and thats fine.  when yo u start to move your house to that next stable platform, then you deal with it.
<smoser> i dont remember the exact values here, but lets just say:
<smoser>  16.04: little recommends
<smoser>  18.04: big recommends
<smoser>  20.04: little recommends
<smoser> thats perfectly fine. it sucks that 18.04 is got this wart there, but people that are building on it expect the wart.
<meena> aye.
<smoser> but again... these are *my* feelings towards that sort of thing.  the person who decides is a Stable Release Team member, and (in my experience) their individual opinions and behaviors differ.
<meena> but  yeah, i agree that having a dedicated python3-scapy only package without the kitchensink would be preferableâ¦ which of course won't be backported, eh?
<smoser> and sometimes even differ based on the time.
<smoser> the are humans, and as a result have a very hard time remaining consistent.
<smoser> largely RHEL and SuSE have simmilar processes/values
<meena> some people make horrible decisions if they didn't have breakfast, or if their underpants are too tight.
<smoser> so to depend on scapy, i think for the near term, the thing we have to do is make it optional, with a fallback to the questionable decision that I made some years ago to use dhclient like we do
<smoser> (see.... consistency  hard for people!)
<meena> smoser: see, if you had gone with scapy years ago, you could've kept it from balooning in size in certain versions of debuntuâ¦ and we'dâ¦ have a very different networking refactor on our hands rn ;)
<rick_h> minimal:  meena sorry, didn't open this up yet today. We're waiting on one final PR and will start the release/SRU processes. falcojr has the PR up but needs review/etc so hopefully that goes through today and we'll start the release tomorrow.
<meena> rick_h: oh, so it *is* an SRU release
<meena> also, i forgot what that means again
<powersj> stable release upgrade, the exception to the don't update stable releases
<powersj> for cloud-init we take the latest upstream code, push it to the devel release and then test it against the supported releases
<powersj> in this case xenial, bionic, and focal
<beantaxi_> smoser: I am quite confused ... but as best as I can tell: I can add bash scripts to my multi-part mime, and they will be run once per instance, unless I override scripts-user in my cloud-config to [scripts-user, always], in which case my bash scripts will execute on boot.
<beantaxi_> As for my wanting to have one script execute per-instance, and one execute per-boot ... it looks like I could only achieve that by manually copying files to the /var/lib/cloud/per-boot and -instance folders. There's no way to do that otherwise. Is that all correct?
<rharper> beantaxi_: I suggest that you you use write_files and then emit them into the different scripts-per frequency desired,  /var/lib/cloud/scripts/per-{boot,instance,once} ;
<smoser> yeah.
<smoser> it does seem useful to have a way to put those in user-data. but as it is, writing files to /var/lib/cloud/per-boot is the only real way.
<beantaxi_> Ah, so in a pinch write_files can be used to put whatever files I want, wherever I want them. That's useful.
<beantaxi_> Would it be technically possible to implement something a bit more like I'm describing, by eg allowing an optional header for a dest folder (or more indirectly specifying a 'per
<beantaxi_> ')? Or is the mime-parsing code handled by the cloud providers rather than by cloud-init
<smoser> by cloud-init
<smoser> we could absolutely improve cloud-init to do what you're after.
<beantaxi_> I'm happy to take a lame swing at it ... but I'd want to make sure I'd set expectations sufficiently low
<beantaxi_> In my specific use case, I have a little family of VM initialization scripts, where each VM has an init-image and an init-vm script. Naturally I found out about cloud-init about 5 minutes after I was done.
<smoser> https://cloudinit.readthedocs.io/en/latest/topics/format.html#part-handler
<smoser> you could use a part-handler to do what youre after.
<beantaxi_> So I'd like to use the scripts as-is -- where init-image is per-isntance and init-vm is per-boot, and also start transitioning my per-instance stuff to cloud-config, which I am sure over time I will prefer.
<beantaxi_> Ok nice. I was not quite sure what those were for.
<beantaxi_> K, this is making total sense
<beantaxi_> The part-handler docs link to a blog post from 2011 from 'Foss Boss'. Is this info likely still current?
<blackboxsw_> falcojr: did you have an azure instance up that exhibits https://github.com/canonical/cloud-init/pull/539?
 * blackboxsw_ is trying to get that review complete so we can cut 20.3
<falcojr> blackboxsw_ bah, sorry, didn't get the notification
<falcojr> I have one with the fix currently
<falcojr> I could spin one up quickly that shows the problem though
<falcojr> it just needs to have accelerated networking (on the network tab if you're in the UI)
<falcojr> I used a focal DS2 v2 instance
<falcojr> s/focal/bionic
<blackboxsw_> +1 I'll launch an instance. were you using amazon linux or ubuntu?
<powersj> O.o amazon linux on azure?
<blackboxsw_> hahah
<blackboxsw_> oops brain fail
<blackboxsw_> probably not a product that azure is currently selling :)
<powersj> :D
<falcojr> haha, it should happen on focal or bionic. I used bionic
<blackboxsw_> falcojr: strange as I've added accelerated networking on a DS2v2 instance and not seeing the delay on bionic
<blackboxsw_> ubuntu@test-b1:~$ systemd-analyze blame
<blackboxsw_>           8.553s cloud-init.service
<blackboxsw_>           8.553s cloud-init.service
<blackboxsw_>           6.857s cloud-init-local.service
<falcojr> did you reboot?
<falcojr> blackboxsw_ ^
<falcojr> it doesn't happen on first boot
<blackboxsw_> falcojr: rebooting now. sure enough     1min 31.262s cloud-init-local.service
<blackboxsw_> thanks
<rharper> blackboxsw_: if you're one a box now, I just posted some questions on that PR; would be nice to see requested info attached to the bug that PR addresses;
<blackboxsw_> +1 rharper
<blackboxsw_> I'm also seeing the type of failure mode I think we were looking for on current bionic images, like a failed "rename3" interface lying around post reboot
<blackboxsw_> with matching mac addr
<rharper> yeah, I'm super interested to confirm whether there are two hv_net nics, or if there's a different sriov driver name (we already blacklist the mlx4_core)
<rharper> blackboxsw_: I wonder if we just dropped including the driver value in the emitted netplan
<blackboxsw_> rharper: checking now. we only blacklist mlx4_core on fallback network right?
<blackboxsw_> on azure
<rharper> yeah
 * blackboxsw_ double checks
<rharper> that can't be right
<blackboxsw_> also responded first here: https://github.com/canonical/cloud-init/pull/539#discussion_r475918457
 * blackboxsw_ tries to recall how to attach binary (cloud-init.logs.tar.gz) to github comments
<blackboxsw_> if that's even possible
<rharper> you can attach to the bug it references
<rharper> mlx5_core
<rharper> looks like we'll need to blacklist both 4 and 5 (and fix non-fallback path to include drivers)
<rharper> do we know if IMDS reports the sriov interface at all ?
<blackboxsw_> checking
<blackboxsw_> rharper: nope ... http://paste.ubuntu.com/p/CFjtY7JHm4/
<blackboxsw_> well "yes" in that it only reports one interface
<rharper> yeah, I don't think it did
<blackboxsw_> and both share the the same mac
<blackboxsw_> :)
<blackboxsw_> yeah
<rharper> I recall asking in the past
<rharper> blackboxsw_: I replied;  I think the fix is good;  but let's update the commit message;  the issue here is that once we started rendering from IMDS, we regressed Accelerated Networking setups  due to not including the hv_netsvc driver name in our match;    when we were v1 rendering, that emits a udev rule which matches the MAC and   DRIVER != $blacklist ;
<rharper> the fix for netplan is what's there (binding the emitted netplan to the hyperv driver nic)
<rharper> on the fallback path, we need both mlx4 and mlx5 core (xenial running on AdvNet instances with mxl5 will likely have this hang as well )
<blackboxsw_> rharper: adding https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1830740/comments/11 while I look at this
<ubot5> Ubuntu bug 1830740 in linux-azure (Ubuntu) "[linux-azure] Delay during boot of some instance sizes" [Undecided,New]
<rharper> I wonder if sysconfig rendered instances see failures, depends on what the udev rules look like
<rharper> cool
<blackboxsw_> why wouldn't we want to just blacklist on the mlx* IMDS-based network config
<blackboxsw_> and on fallback config
<blackboxsw_> ok confirmed eth0 is driver/module: module -> ../../../../module/hv_netvsc     rename3 :   module -> ../../../../module/mlx5_core
<blackboxsw_> so yeah, just wondering if maybe we should make sure we blacklist both mlx5_core and mlx4_core in both fallback config and imds-based config we write
<blackboxsw_> so we don't unintentially ignore other device types.
<blackboxsw_> or are there any other device types that would present "hv_netvsc" for one interface and bool(not any(['mlx4_core', 'mlx5_core']))
<blackboxsw_> guess that's a question for both falcojr and rharper
<rharper> blackboxsw_: so blacklist only works for eni and sysconfig which emit udev rules
<blackboxsw_> ohhh doh
<rharper> for netplan, the 50-cloud-init.yaml needs to use the match driver=hv
<rharper> with the mac
<blackboxsw_> ok ok, so drive matching is the inverse approach for netplan
<blackboxsw_> driver* matching
<rharper> we know (for now) that any rendered interface in IMDS on Azure is hv_
<rharper> blackboxsw_: well, we could emit a driver != I think ... let me confirm
<rharper> yeah, we could emit:  Driver=!mlx4_core\nDriver=!=mlx5_core
<rharper> however, I'm not sure if netplan likes that ...
<blackboxsw_> right, or it it only honors the latter of the settings etc
 * blackboxsw_ tries a generate 
<blackboxsw_> and sees what systemd thinks
<rharper> quoted it accepts the !driver
<rharper> but not sure about a list (or multiple
<rharper> I suppose we won't have both
<rharper> well, that's not easily found
<rharper> without finding the dup mac device (which may not be present)
<rharper> I think the current code is safest
<blackboxsw_> yeah I think so too at the moment, trying a clean and dirty upgrade and reboot with new code now
<blackboxsw_> as the duplicate !driver matches may collide in systemd or netplan cfg with unintentional side-effects (or matching nothing)
<blackboxsw_> like OR instead of AND
<blackboxsw_> falcojr: at the moment, I'm still seeing 90 seconds times on reboot even with driver matching hv_netvsc
<blackboxsw_> on dirty cloud-init upgrade/reboot
<blackboxsw_> http://paste.ubuntu.com/p/7NjWHX2QTY/
<blackboxsw_> interesting but 2nd reboot was fixed.
<falcojr> I didn't see it after a clean and reboot
#cloud-init 2020-08-25
<blackboxsw_> same falcojr I'll close out on that review now and put up a 20.3 branch for review
<paride> smoser, TIL https://github.com/chrlutz/apt-repos, not packaged yet
<paride> `apt-repos dsc` + dget could finally be a good way to download a src package with its .orig from a repo without setting it up in sources.list
<meena> oh, nice
<Carbrich_R2> HI all, I am having a weird issue with cloud-init, for some kind of strange reason it gets terminated. And I cannot figure out what is causing the issue. I only have this issue with CentOS8,  You can find information on
<Carbrich_R2> https://community.theforeman.org/t/cloud-init-does-not-install-puppet-but-runs-the-final-module-and-succeeds-according-cloud-init/20155/23 of my issue. The only part that is missing that if I remove /var/lib/cloud/ and reboot the client all scripts and providers work. Sometimes 1 of 100 cloud-init works without issues.
<Carbrich_R2> on should be at but ok. As long as you guys understand it :)
<meena> Carbrich_R2: what's makecache, and why is that part of the puppet installation?
<Carbrich_R2> HI, Initially I tried a cloud config with installing puppet, later on the road I also removed that part to investigate what is going wrong. At this point I am only trying to install package "SL" with the native package provider. And that also fails.
<meena> https://community.theforeman.org/t/cloud-init-does-not-install-puppet-but-runs-the-final-module-and-succeeds-according-cloud-init/20155/26 â¬ï¸ this is interesting
<meena> the easiest way to figure out if cloud-init got updated is to check the version before and after the run. but dnf's logs should have some clues, as well
<Carbrich_R2> Even more strange sometimes let`s say 1 of 100 it works. and after removing '/var/lib/cloud/ and restart the machine again wil result in OK.
<Carbrich_R2> to anwser your qeustion meena there is nothing logged about updating cloud-init
<Carbrich_R2> in the dnf lgos
<Carbrich_R2> logs. If I do a casual ping with count 20 it stops after 8-10 tries. It executes runcmd part for example and it get a sigterm. Nothing in dnf logs etc.
<Carbrich_R2>  All end up with the conclusion that something is killing cloud-init. And I cannot figure out what is happening under the hood. Any idea`s?
<meena> I'm wondering if there's some kind of time limit on execution, but that would be new to me
<Carbrich_R2> It almost looks like some race condition with system services,  that causes this issue. I have a similar setup on our CentOS7 deployment street and that works fine.
<meena> oh, hrm, i thought we fixed those
<meena> in newer versions
<meena> Carbrich_R2: can you upgrade and run your test?
<Carbrich_R2> which versions? I updated to 19.4 yesterday
<Carbrich_R2> and tried with that version also
<Carbrich_R2> :)
<Carbrich_R2>  meena I have no problems with upgrading, but which version do you advice?
<meena> https://github.com/canonical/cloud-init/pull/227 hrm, i thought we merged this "fix"
<meena> Carbrich_R2: the current release is 20.2
<meena> https://copr.fedorainfracloud.org/coprs/g/cloud-init/el-stable/ â¬ï¸ im confused about the last release being 9 months ago
<meena> maybe these are just the SRU releases?
<meena> aaaaand I'm very confused about the versions listed there
<Carbrich_R2> meena No problem, That confused me also. But in what version is that issue fixed you are referring to?
<Carbrich_R2> meena https://copr.fedorainfracloud.org/coprs/g/cloud-init/cloud-init-dev/ I could try a Daily build, but the repo file is missing..
<Carbrich_R2> They are referring to " # Save the above yum config to  /etc/yum.repos.d/CentOS-cloudinit.repo" But there is nothing above that text.. Confusing
<meena> https://copr.fedorainfracloud.org/coprs/g/cloud-init/cloud-init-dev/repo/epel-8/group_cloud-init-cloud-init-dev-epel-8.repo
<Carbrich_R2> meena THX found it already indeed :)  clicked on one of the icons down on the page.
<meena> same
<Carbrich_R2> Cool, I1`ll have to rebuild my base image with this repo and import in vcenter before i can test. I`ll come back to you if I know the outcome. It will take some time :)
<beantaxi> 8 days ago (https://github.com/canonical/cloud-init/pull/518), make-mime.py was moved from tools to devel, and going forward will be part of the CLI. Seems like a nice idea, and the docs have been able to reflect it.
<beantaxi> However, while the code is in master, it's not in a distro, and it looks like it's not tagged by 2.2-94 either. And so obviously it's not on my ubuntu EC2. Any idea when that will all happen? And, is it considered a docs bug, when the docs on the public web refer to changes that have not happened yet?
<smoser> paride: thanks for the apt-repos reference.
<smoser> beantaxi: what is 2.2-94 ?
<smoser> readthedocs has docs for older versions also. and source code for your distro should contain docs that match exactly with your packaged version.
<smoser> with regard to "when will make-mime" become part of ubuntu, probably in the next SRU (stable release update)
<AnhVoMSFT> @rharper @smoser if the customer has security requirements on /var/tmp being noexec, is there any workaround to allow cloud-init to still do DHCP to talk to metadata service?
<beantaxi> smoser: Thanks. And sorry - I meant 20.2-94, or more precisely 20.2-94-g3d06abc2-0ubuntu1, the github tag. My fault for trying to be terse instead of copy/pasting. (Really sorry -- I
<beantaxi> 'I'd be mad if someone did that and I wasted time figuring out what they were talking about)
<beantaxi> Also thanks for pointing out the docs version option, which I now found, and it does indeed refer to tools/ rather than devel. All good.
<beantaxi> As for in which release will the move to devel happen ... I suppose that question was meta, and I'm curious about if there is an easy way to tell, or a reliably way to make assumptions, about what release a PR will be a part of, other than asking here.
<smoser> beantaxi: well, looking at the debian/changelog it appears every ~ 2m
<smoser> i think thie goal was 1m.
<smoser> AnhVoMSFT: not that i'm aware of.
<beantaxi> smoser: Thanks ... I'm now looking here for Ubuntu 18.04 -> https://launchpad.net/ubuntu/bionic/+source/cloud-init/+changelog, where the release schedule has more variance, and release dates are included at the end of the changelog, when they are included. But I'm new at this; maybe this is the wrong place to look.
<beantaxi> Either way, a heursitic of 'on the order of months' is valuable.
<smoser> beantaxi: maybe Odd_Bloke or blackboxsw_ could point on what the expected cadence is.
<smoser> some other info at https://github.com/cloud-init/ubuntu-sru and https://wiki.ubuntu.com/CloudinitUpdates
<beantaxi> That looks great. I don't think I'll need anything precise but if I do I know who to ask. Thanks!
<blackboxsw_> @ahosmanMSFT  I thought there was a recent bug/branch to try to address the noexec on /tmp/noexec  for dhcp https://github.com/canonical/cloud-init/commit/db86753f81af73826158c9522f2521f210300e2b
<Carbrich_R> meena It only worked two times from the 10 and now i am only getting to see the sigterm again.
<Carbrich_R> 2020-08-25 15:03:53,142 - subp.py[DEBUG]: Running command ['/var/lib/cloud/instance/scripts/runcmd'] with allowed return codes [0] (shell=False, capture=False)2020-08-25 15:03:54,145 - util.py[DEBUG]: Cloud-init 20.2+137.gc0450c02-1.el8 received SIGTERM, exiting...  Filename: /usr/lib64/python3.6/subprocess.py  Function: _try_wait  Line number:
<Carbrich_R> 1424    Filename: /usr/lib64/python3.6/subprocess.py    Function: wait    Line number: 1477      Filename: /usr/lib64/python3.6/subprocess.py      Function: communicate      Line number: 8552020-08-25 15:03:54,145 - handlers.py[DEBUG]: finish: modules-final/config-scripts-user: FAIL: running config-scripts-user with frequency
<Carbrich_R> once-per-instance2020-08-25 15:03:54,146 - util.py[DEBUG]: Reading from /proc/uptime (quiet=False)
<blackboxsw_> so I think that pr was to use the known dhclient path if var/tmp is noexec. If that doesn't work, then cloud-init falls back to use the dhclient in /usr/sbin/dhclient
<AnhVoMSFT> Thanks @blackboxsw, I'll take a look at the PR
<blackboxsw_> @AnhVoMSFT that said, I think ubuntu will still run into an issue with AppArmor profiles on ubuntu in such scenarios
<blackboxsw_> but it'll 'work' for other distros which don't use apparmor
<blackboxsw_> beantaxi: the plan for cloud-init releases is to try to create an upstream release (20.2, 20.3, 20.4 etc)any time we have a new significant feature to make SRU generally this will likely mean at least 4 times per year. We document the estimated upstream release date (20.3) in the topic of this channel, and Canonical cloud-generally.
<blackboxsw_> specifically for your commit, we are trying to cut that 20.3 release today (we were waitong on an azure-related fix for cloud-init related boot timeouts which just landed last night)
<blackboxsw_> I expect we'll have a release of 20.3 (containing the commitish you are interested in) into Ubuntu 20.10 (groovy) and will start an SRU process to release into Ubuntu Xenial (16.04), bionic and focal
<blackboxsw_> agreed though. we can/should add some semblance of docs with that intent
<blackboxsw_> well looks like it's that time again
<blackboxsw_> #startmeeting cloud-init status meeting
<meetingology> Meeting started Tue Aug 25 16:24:27 2020 UTC.  The chair is blackboxsw_. Information about MeetBot at http://wiki.ubuntu.com/meetingology.
<meetingology> Available commands: action commands idea info link nick
<blackboxsw_> #chair smoser rharper falcojr lucasmoura
<meetingology> Current chairs: blackboxsw_ falcojr lucasmoura rharper smoser
<blackboxsw_> hey folks, welcome to another cloud-init bi-weekly (or bi-monthly) community status meeting
<blackboxsw_> or semi-monthly
<blackboxsw_> ... anyhow. We use this platform/channel to discuss latest and greatest cloud-init, as well as ensuring that there are a couple of upstream developers present to field questions or discussion as needed.
<blackboxsw_> We gather here in this IRC channel every 2 weeks to discuss current development tasks and progress on cloud-init. All questions and side-conversations welcome
<blackboxsw_> we keep our meeting minutes from previous meetings here:
<blackboxsw_> #link https://cloud-init.github.io
<blackboxsw_> The topics we'll cover today: Previous Actions, Recent Changes, In-progress Development, Community Charter, Office Hours (~30 mins).
<blackboxsw_> #topic Previous Actions
<blackboxsw_> Looks like no carryover actions from last meeting
<blackboxsw_> #topic Recent Changes
<blackboxsw_> The following changes have landed in tip of master since last meeting (08/14/20).
<blackboxsw_> found via git log --since 2020-08-014
<blackboxsw_> #link https://paste.ubuntu.com/p/h2qxwKwvFw/
<blackboxsw_> things to note. we *JUST* cut 20.3 upstream release as of 20 seconds ago. Thanks a bunch James(falcojr) for getting the Azure timeout pr up to close out this upstream release.
<blackboxsw_> and thanks smoser, rharper and meena for all the reviews and work here.
<blackboxsw_> in the last couple weeks we got fixes to reduce boot timeouts for certain azure accelerated network instances, fix oracle datasource retries, handle compressed user-data on juju deployed machines in 'cloud-init query', and early boot dhclient will not attempt to run outside of the /var/tmp sandbox directory if that directory is marked no-exec
<blackboxsw_> thanks otubo for that noexec branch
<blackboxsw_> #topic In-progress  Development
<blackboxsw_> So thanks all for the 20.3 upstream release. We will be tagging that release and pushing that tag to master just after this meeting
<blackboxsw_> falcojr: is our release lead for this upstream release and SRU so he'll be working through the release process work items. Thanks falcojr.
<blackboxsw_> -next step on the release process is to publish to Ubuntu Groovy (20.10) and then queue up a -proposed cloud-init SRU upload into xenial, bionic and focal.
<blackboxsw_> we will then begin the SRU testing (which we hope to keep at around ~7days)
<blackboxsw_> as smoser mentioned earlier. cloud-init has to follow this process to update cloud-init in stable Ubuntu releases
<blackboxsw_> #link https://wiki.ubuntu.com/CloudinitUpdates
<blackboxsw_> An email will be sent to the cloud-init mailinglist notifying the community about the SRU under test in the event that folks have spare cycles to pitch in on some of the verification effort
<blackboxsw_> also "in-progress" paride is cleaning up a bunch of automated CI-related failures and lint issues in cloud-init, so expect some branches against cloud-init on that front as well
<blackboxsw_> #topic Community Charter
<blackboxsw_> The following topics are still topics for ongoing community development anyone new to cloud-init, or with a bit of time could easily grab one of these bitesized tasks:
<blackboxsw_>  JSON schema extensions to validate user-data before instance launch: https://bugs.launchpad.net/cloud-init/?field.tag=bitesize
<blackboxsw_> - Datasource documentation and updates
<blackboxsw_> -  cloudinit.net refactor into distro-specific networking subclasses cloudinit.distros.networking: https://bugs.launchpad.net/cloud-init/+bugs?field.tag=net-refactor
<blackboxsw_> As always: thank you all for bug contributions, PR submissions, triage and discussion participation.
<blackboxsw_> If anyone would like to be involved more than they currently are, please feel free to contact us here in IRC #cloud-init on Freenode or on the mailing list cloud-init@lists.launchpad.net and we can see how best we can get you "set up"
<blackboxsw_>  #topic Office Hours (next ~30 mins)
<blackboxsw_> his time of the meeting is really just an open door for any discussions, concerns, bugs, questions or general prodding of upstream devs to make sure existing development work is unblocked where possible.
<blackboxsw_> *This time
<blackboxsw_> while we're at it with meeting time.... I'll set the next cloud-init status meeting date in the topic of this IRC channel
<AnhVoMSFT> When do you anticipate the next SRU would be?
<AnhVoMSFT> (after 20.3)
<blackboxsw_> AnhVoMSFT: thanks for the question. I hope today for 20.3, I think 20.4 will likely align with the next Ubuntu release cycle (20.10)  so, October, 2020
<blackboxsw_> as mentioned in passing earlier, we are looking at trying to create an upstream release of cloud-init when any significant feature set has landed in tip to make release verification easier (and hopefull include less sprawling sets of broad commit streams).
<blackboxsw_> this ultimately may translated to 4-5 releases per year
<blackboxsw_> I expect a frequency of < 3 months
<blackboxsw_> #topic #cloud-init pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting Sep 8 16:15 UTC | 20.2 (Apr 26) | 20.3 (estimated Aug 19th) https://bugs.launchpad.net/cloud-init/+filebug
<blackboxsw_> next status meeting sept 8th, same time ^
<AnhVoMSFT> Thanks. There's some planned work for Azure's pre-provisioning v2 and will need some change in cloud-init. We're trying to get a sense of when cloud-init SRUs land so that we can communicate the timeline
<blackboxsw_> @AnhVoMSFT generally cadence has been ~3 months between SRUs, I think we'd like to see a slightly higher frequency than that because our last SRU was so costly. but I think expectation for this pass is likely 10/15/2020
<blackboxsw_> also, if we are looking to test features in Ubuntu Groovy 20.10 images, they don't require an SRU, so upload to that development release are possible any time (we could do that as frequently as weekly if need be)
<AnhVoMSFT> thanks for the clarification @blackboxsw_
<blackboxsw_> so as of today, I expect we'll have an SRU in October and likely something beginning of Jan
<blackboxsw_> I'll take an action for us to communicate via mailinglist the next estimated SRU once this SRU for 20.3 closes out
<blackboxsw_> #action bbsw add workitem to SRU release process to announce to mailinglist estimated next SRU timeframe
<meetingology> ACTION: bbsw add workitem to SRU release process to announce to mailinglist estimated next SRU timeframe
<blackboxsw_> and again we hope to have 20.3 SRU complete by next week
<blackboxsw_> complete, as in publish the 20.3 SRU for next week into Ubuntu xenial and later
<blackboxsw_> I expect today we publish tip of master (20.3) into Ubuntu Groovy (20.10) so expect to see it in your friendly neighborhood ubuntu cloud-images in the next day or two
<blackboxsw_> falcojr: I just annotated tag 20.3 and pushed to upstream
<blackboxsw_> so we have a signed tag
<blackboxsw_> falcojr: so next task is https://trello.com/c/KxShylli/14-upload-source-tarball-to-launchpad
<falcojr> Cool, I can jump on that
<blackboxsw_> ok I think we'll wrap up the status meeting. Thanks all for tuning in!
<blackboxsw_> #endmeeting
<meetingology> Meeting ended Tue Aug 25 17:13:02 2020 UTC.
<meetingology> Minutes:        http://ubottu.com/meetingology/logs/cloud-init/2020/cloud-init.2020-08-25-16.24.moin.txt
<Carbrich_r> meena I Was logged out, so I have no idea if you wrote back.
<blackboxsw_> meeting minutes for the cloud-init status meeting just posted to https://cloud-init.github.io/status-2020-08-25.html#status-2020-08-25
<rharper> blackboxsw_: thanks!
<blackboxsw_> np; and ditto
<blackboxsw_> it'll be good to get this release out and verified
<meena> Carbrich_r: nope, sorry. i had left to pick up my daughter
<meena> who's still not asleep
<Carbrich_r> meena No problem I also went a away to pick-up my two boys :)
<Carbrich_r> I have create a bug report on: https://bugs.launchpad.net/cloud-init/+bug/1892902 With some more log details.
<ubot5> Ubuntu bug 1892902 in cloud-init "Cloud-init received SIGTERM and is thereby unable to install packages or running runcmd completely" [Undecided,New]
<Carbrich_r> meena Now seeing ubot5 I am in a doubt if that was the right place :S
<Carbrich_r> I`ll have to arrange a few things I`ll check later on.
<meena> I've been bringing my daughter to daycare by bike, and picking her up, by bike, since Friday. and today, i can't feel my body where it doesn't hurt.
<meena> Carbrich_r: now that North America is awake, and the meeting is over, it would be good to repeat your question
<Carbrich_r> I`ll try to do it later on this evening :_
<Carbrich_r> doorbell is going
<meena> all the people with an @ in front of their name know vastly more than iâ¦ aaaaand gone
<Carbrich_r> HI All I have the strange issue that cloud-init sigterms during a runcmd or package install with the native package provider. I have create a bug on https://bugs.launchpad.net/cloud-init/+bug/1892902. That explains my issue  I hope.
<ubot5> Ubuntu bug 1892902 in cloud-init "Cloud-init received SIGTERM and is thereby unable to install packages or running runcmd completely" [Undecided,New]
<Carbrich_r> meena Thx for the tip! about U.S. part ot the world :)  That is getting awake.
<Carbrich_R> .
<ananke> I'm having a hell of a time trying to figure out why cloud-init throws an error with the following apt config: https://dpaste.com/GZMDAVVEM
<ananke> the error is TypeError: 'str' object does not support item assignment
<ananke> and here's a bigger section of the log: https://dpaste.com/F526QRPAW
<ananke> I'd appreciate any hints on what I may be doing wrong
<ananke> seems the issue persists if I remove the key part, so it's just the source
<beantaxi> Until someone more knowledgable comes along ... which wouldn't be hard ...
<beantaxi> It does not appear that your config matches the structure in the example on https://cloudinit.readthedocs.io/en/20.2/topics/modules.html#apt-configure
<beantaxi> They have sources.source1.source and .key. You have sources.source1 and .key
<beantaxi> I have zero idea if that is relevant or not. Just something I noticed.
<ananke> I was following the example from https://cloudinit.readthedocs.io/en/latest/topics/examples.html#install-and-run-chef-recipes, not sure if that differs in that aspect
<beantaxi> Yep. looks like you do match that example. Sorry - that's all I got.
<ananke> though I have nothing to lose to try the other way, thank you :)
<beantaxi> Good luck!
<ananke> ha! it worked! thank you so much
<beantaxi> No way
<beantaxi> Haha -- good on you for trying it!
<powersj> ananke, if you could, can you file a doc bug @ https://bugs.launchpad.net/cloud-init/+filebug
<ananke> powersj: will do! later tonight if you don't mind, parent-teacher conference time
<powersj> no rush, but want to remember to fix this
<blackboxsw_> falcojr: I'm reviewing your upload pr https://github.com/canonical/cloud-init/pull/548   by performing the same basic new-upstream-snapshot steps now to confirm we see nearly identical diffs
<blackboxsw_> confirmed no reall diff (other than pkging author) between your branch and mine for upload
<blackboxsw_> falcojr: https://paste.ubuntu.com/p/Jhvp3YQwZF/
<blackboxsw_> I'm confirm sbuild-it works and then I'll hit build-and-push to perform the upload
<blackboxsw_> assuming that succeeds, we are good to go on groovy release of 20.3
<blackboxsw_> bah james I missed the following diff on the ubuntu/devel branch
<blackboxsw_> https://paste.ubuntu.com/p/RhN4fpbkX5/
<blackboxsw_> putting up a minor pr now
<blackboxsw_> n/m I just changed UNRELEASED to groovy in debian/changelog.  which we should have done via a sed command on your branch, but that was my fail to catch it (why I'd like to do more frequent releases, so my muscle memory can take care of things so my brain can forget them)
<blackboxsw_> falcojr: upload to groovy complete. we can send out the release email now
<blackboxsw_> thanks @!
<blackboxsw_> community notice: [ubuntu/groovy-proposed] cloud-init 20.3-0ubuntu1 (Accepted)
<blackboxsw_> ok 20.3 release is "in" for Ubuntu Groovy (20.04) should show up in cloud images in the next day or two
<blackboxsw_> falcojr: so we can now start the SRU for 20.3 into Xenial, Bionic and Focal. (once your 20.3 release email is out to the mailing list)
<blackboxsw_> thanks again
<falcojr> Sounds good. Thank you
#cloud-init 2020-08-26
<Carbrich_R> HI, I created  https://bugs.launchpad.net/cloud-init/+bug/1892902 yesterday, is that the right place for this one? And if someone has an idea or time to help, that would be appreciated.
<ubot5> Ubuntu bug 1892902 in cloud-init "Cloud-init received SIGTERM and is thereby unable to install packages or running runcmd completely" [Undecided,New]
<Carbric__> stats p
<falcojr> blackboxsw_: looks like next steps of the SRU requires a launchpad bug. Not sure how to fill that. Do we just copy the original template and fill it in later as we verify things?
<rharper> falcojr: yeah;
<rharper> falcojr: file new bug; paste in the template; and then update the description as you go;  the first thing usually is the changelog generated from new-upstream-snapshot for the source package you SRU ;
<falcojr> thanks!
<rharper> falcojr: I almost always look at the previous SRU bug for reference
<falcojr> I can't find anything with launchpad search :D
<falcojr> ooooh, it's a different project, nm
<blackboxsw_> sorry missed that initial ping falcojr . and I think you gen'd the right bug template as you showed me https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1893064
<ubot5> Ubuntu bug 1893064 in cloud-init (Ubuntu Focal) "sru cloud-init (20.2-45 to 20.3-0) Xenial, Bionic, and Focal" [Undecided,New]
<blackboxsw_> so basically now we just fill in the ubuntu-specific functional changes that affect this SRU in the section    * <TODO: Create list with LP: # included>
<blackboxsw_> so grab those bug linked items from the Changelog file that got created due to the new-upstream-snapshot run on your ubuntu/xenial branch
<blackboxsw_> and manually redact any that list down to anything that actually affects ubuntu systems (so redhat or opensuse or freebsd stuff shouldn't be listed in the top section of the ubuntu process bug.
<falcojr> sounds good, my 3 PRs are up too
<blackboxsw_> excellent reviewing now
<blackboxsw_> and falcojr here's more context for both of us on ubuntu/xenial branch creation https://github.com/canonical/cloud-init/pull/435
<beantaxi> Potentially dumb question ... on my Ubuntu/systemd VM, if the entry point to cloud-config.service and cloud-init.service is /usr/bin/cloud-init, then where is /usr/bin/cloud-init from in the distro? I can't seem to grep a matching source file.
<beantaxi> I do see in setup.py that the entry point is cloudinit/cmd:main/main.py (and there's another one for cloud-id as well). But I don't see anywhere the package itself is being made -- no target in Makefile seem to match -- and I can't find the little cloud-init script either. My Python is weak so this could all be obvious. Thanks!
<powersj> beantaxi, the packages directory is what is used to generate the deb and rpm packages
<minimal> Hi folks, minor issue I've noticed with the 20.3 release, during boot I see on the console a "hi, now its" message which is due to a printf in cloudinit/util.py. Seems this is a left over from some testing/debugging
<powersj> fix for ^ https://github.com/canonical/cloud-init/pull/556
<blackboxsw_> haha
<blackboxsw_> thanks minimal and powersj !
<beantaxi> powersj: Thanks ... not seeing anything in there ... perhaps bits of the entry point stuff do not live in the canonical/cloud-init github repo?
<powersj> beantaxi, if I recall correctly that file itself is not in the source, it is generated as part of the build
<powersj> beantaxi, is there something you are trying to change with that file?
<beantaxi> powersj: For my own edification, I'm trying to figure out what happens on startup. I'm ok with the file not being in the source; it means I can stop stomping around wondering why I'm so dumb and can't find it
<blackboxsw_> powersj: approved. waiting on ci
<minimal> powersj: Thanks, could have raised a PR myself, just hadn't taken the time to dig through git history to see how/when that line got in the code
<beantaxi> powersj: My specific underlying issue, is I'm writing a part handler and not getting what I expect (I'm getting None for filename, and can't figure out why)). So I figured I'd look into how exactly cloud-init calls part handlers. And I think ultimately I've found that in cmd/main.py; there was just a missing piece in getting there, which I think you've now provided (ie it ain't there)
<meena> anyone had any chance to help out Carbrich_r ?with https://bugs.launchpad.net/cloud-init/+bug/1892902
<ubot5> Ubuntu bug 1892902 in cloud-init "Cloud-init received SIGTERM and is thereby unable to install packages or running runcmd completely" [Undecided,Confirmed]
<meena> oh, it's confirmed already
<blackboxsw_> falcojr: per your branches I gave you the wrong guidance on 'fixing' the quilt patches for these branches. In checking whether our daily build recipe would continue to work.  we need to be able to git clone master; git merge your/ubuntu/xenial; git merge  upstream/ubuntu/daily/xenial; quilt push -a # to apply all quilt patches without errors...... but we are getting quilt patch apply issues when merging daily
<blackboxsw_> so I need to investigate a little further taking into account how our ubuntu/daily/<release> needs to merge into ubuntu/<release> to see if we might need to refresh patches in the ubuntu/daily/xenial branch too because we had to "quilt refresh" in ubuntu/xenial
<blackboxsw_> this is all related to the behavior of ubuntu_advantage config module in Xenial/Bionic is patched specifically to "work" with the older/broken client.... once we have an ubuntu-advantage-tools pkg that is officially published to X, B and F ubuntu releases, then these patches should disappear.
<blackboxsw_> falcojr: also I'm wondering if we want to generate a new-upstream-snapshot against groovy now that  https://github.com/canonical/cloud-init/pull/556 landed because it's kindof ugly to pull back that debug print statement in our SRU too
<blackboxsw_> so we cloud do a  followup ubuntu/devel release as we did yesterday (easier this time)
<blackboxsw_> and then SRU what we have in tip of master (as we don't specifically have to SRU 20.3 we can SRU 20.3-2
<blackboxsw_> then we don't have the blemish of debug statements dumped out in /var/log/cloud-init-output.log for our SRU content
<blackboxsw_> we can discuss and see if we want to do that today or not
 * blackboxsw_ jumps into a couple back-to-back mtgs.
<Carbrich_r> @meena:  thanks for memorising.
<smoser> beantaxi: fyi, i dont know if you saw. but bug 1893064 is SRU cloud-init 20.3
<ubot5> bug 1893064 in cloud-init (Ubuntu Focal) "sru cloud-init (20.2-45 to 20.3-0) Xenial, Bionic, and Focal" [Undecided,New] https://launchpad.net/bugs/1893064
<smoser> so that will get you an update you were after.
<beantaxi> smoser: Thanks! That's thoughtful of you to give me the heads up.
<johnsonshi> Hey upstream folks, is there any guidance on when LOG.warning or LOG.error should be used?
<johnsonshi> For instance if cloud-init encounters an error when applying a config. Should that be treated as a LOG.warning or a LOG.error.
<johnsonshi> blackboxsw_ falcojr: I've added necessary UT mocks in my PR (#549) in response to missing mocks in #539 (which caused the UTs to succeed and fail depending on the machine they are run on). Please see https://github.com/canonical/cloud-init/pull/549#discussion_r477546613
<blackboxsw_> hi johnsonshi sorry will get to that review. thanks.
<blackboxsw_> falcojr: reviewed your upload PR to groovy with that debug message fix
<blackboxsw_> merged
<blackboxsw_> and upload landed.
<blackboxsw_> community-notice: new upload to ubuntu groovy with tip of master [ubuntu/groovy-proposed] cloud-init 20.3-2-g371b392c-0ubuntu1 (Accepted)
<blackboxsw_> thx minimal for the notice on that again & powers for the fix
<blackboxsw_> so falcojr we need to try to regenerate ubuntu/xenial|bionic|focal branches again; probably a git checkout ubuntu/xeniall git reset --hard upstream/ubuntu/xenial; new-upstream-snapshot;
<beantaxi> smoser: I'm sure it's my limited Python skills, but I can't run make-mime.py locally. I cannot seem to get the interpreter to be happy with both `from cloudinit import log` and `from . import addLogHandlerCL`. Either it likes one or the other (or neither). If there is a better way to run on OSX, I'm happy to switch.
<blackboxsw_> one thing I think we might need to note is that (instead of the pulling in individual commits to fix the quilt patches) I *think* we might just need to run quilt push -f; quilt refresh (like the message tells us) in order to fix the quilt patches in xenial
<blackboxsw_> falcojr: ^
<blackboxsw_> johnsonshi: I think LOG.warnings tend to get ignored. cloud-init historically tried it's best to bring up a server regardless of cloud-config shortcomings, typos or misconfigurations. But I think our general opinion on that has changed.
<blackboxsw_> If the user desired a certain config and cloud-init couldn't apply it, it might be best to LOG.error rather than expect the user to grep inadvertently discover a misconfiguration later when they try to use the desired service and have to manually fix it.
<blackboxsw_> johnsonshi: I'd be more inclined to LOG.warning if there are alternative solutions that could remedy the issue or something that is really trivial. That's pretty vague I'm sorry. We've talked in the past about trying to scrub our log levels and even introduce a trace level log due to how noisy cloud-init DEBUG logs are.  We have also discussed the possibility of cloud-init status --long surfacing log warnings
<blackboxsw_> too instead of just errors.
<blackboxsw_> Neither of which have gotten enough priority in the past few years to make it a feature in upstream cloud-init :/
<blackboxsw_> to be honest thogh, the log level scrubbing would be a fairly big undertaking and touch most of cloud-init.
<blackboxsw_> so, LOG.error is good, it stops cloud-init and alerts the user to a misconfiguration that they probably should corret before they launch and use the VM for extended periods of time. And I can attribute the quote "users don't look at logs" -- smoser (every couple weeks)
<smoser> beantaxi: it should just run as './tools/make-mime.py' in a checked out cloud-init dir. (prior to it moving to cloudinit proper)
<beantaxi> smoser: Yes, that worked fine. But the source is different now, & I actually wanted to confirm that an issue I had with tools/make_mime.py -- that it adds entries for a non-existent files, eg when I fat-finger a path name -- I wanted to confirm that was still an issue in the devel version.
<beantaxi> I'm not 100% I correctly observed the issue, and there didn't seem much point in confirming with the older version since it's going away anyway
<smoser> oh. from master, runt it as
<smoser>  $ python3 -m cloudinit.cmd.main devel make-mime
<smoser> either installed for from .
<smoser> (. == top level dir of cloud-init)
<smoser> or
<smoser>  $ ./tools/tox-venv py3 python -m cloudinit.cmd.main devel make-mime
<beantaxi> Thanks! I'll give that a try
<johnsonshi> Thanks for the clarification blackboxsw_ :)
<beantaxi> smoser: Ahhhh I get it, make-mime.py is not designed to be standalone any more. Instead, make-mime is a new command under `cloud-init devel` which simply happens to be defined in make-mime.py. Thanks for bearing with me! Incidentally it looks like it fails on missing files as expected, which is good :)
<johnsonshi> blackboxsw_: Does LOG.error stop execution? Afaik LOG.critical stops cloud-init execution.
<rharper> beantaxi: yeah, I put up a PR a few weeks back to make it available since it wasn't installed with the source;   I believe you can use --force if you are really sure (it checks path and known supported mime-types, force will complain but generate things anyhow)
<blackboxsw_> johnsonshi: soo, what scenarios could lead to IMDS metadata being invalid/corrupted?     This happens when IMDS metadata is invalid/corrupted (such as when it is missing network or interface metadata). This causes the rest of provisioning to fail.
<johnsonshi> blackboxsw_: There have been cases where the Azure IMDS service fails to return network/interface metadata in the IMDS response.
<blackboxsw_> what I mean is, in what scenario would the IMDS network config by surfaced as corrupt/invalid? IS there something else we can look at as far as the platform telling us IMDS is "ready" for comsumers or "waiting"
<blackboxsw_> like a semaphore or long poll file cloud-init could "block" on until IMDS is ready ?
<johnsonshi> This happens due to intermittent platform issues :/ There is no reliable "poll" mechanism for this specific case, as the root cause of IMDS failing to return networking metadata is only due to platform component issues.
<blackboxsw_> +1 ok
<falcojr> blackboxsw_: "I *think* we might just need to run quilt push -f; quilt refresh (like the message tells us) in order to fix the quilt patches in xenial"
<falcojr> that's what I did the first time around
<blackboxsw_> falcojr: ok, I think I did something bogus having forgotten the release steps :/. I'm trying to walk through ubuntu/xenial now  given that we've uploaded 20.3-2 to groovy now to see if I can ensure I can git clone master; git merge ubuntu/xenial; git merge ubuntu/daily/xenial; quilt push -a
<blackboxsw_> which is what our daily build (xenial) recipe does @ https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily-xenial
<blackboxsw_> so if we can't merge ubuntu/daily/xenial into the new ubuntu/xenial branch, then daily builds fail
<blackboxsw_> last SRU we added  the fix-daily-branch util to uss-tableflip repo to assist in these manners. and I'm re-reading to remind myself of the context there on what the intent is for fixing daily/xenial and the scenarios in which we expected to have to run it
<blackboxsw_> I expected we wouldn't need to run fix-daily-branch unless we ran cherry-pick on a given series. This issue might have come about as well for this SRU due to our modification of postinst scripts in xenial/bionic etc.
<blackboxsw_> falcojr: to support the grub changes
<falcojr> hmmm, interesting
<blackboxsw_> so that *may* be why we are failing to merge ubuntu/daily/xenial.... but I'm just guessing at the moment
<blackboxsw_> want to confidently triage why we can't merge u/daily/x into u/x
<blackboxsw_> as it does seem to be that we have refreshed ubuntu_advantage quilt patches at the moment
 * blackboxsw_ stops the <<hand wavvy>> guessing and gets to the bottom of it 
<blackboxsw_> falcojr: so at least part of the problem with merging  ubuntu/daily/xenial into your ubuntu/xenial PR is that new grub debian/changelog entry https://paste.ubuntu.com/p/tgHs8YJZJF/
<blackboxsw_> falcojr: which is also what's been killing our daily builds for xenial anyway for the last few days https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily-xenial
<blackboxsw_> falcojr: https://launchpadlibrarian.net/495123633/buildlog.txt.gz
<blackboxsw_> shows the same traceback on the merge I'm seeing locally
<blackboxsw_> so I think we should try to fix daily builds first by fixing ubuntu/daily/(xenial|bionic|focal) branches first
<blackboxsw_> we control those branches (and there are no consumers of them beside our daily recipe tooling)
<blackboxsw_> so we can overwrite --force push those branches anytime we wish. though I'd prefer if our tooling took care of this
<falcojr> hmmm, ok. Why would an updated changelog be causing a conflict?
<falcojr> also, reaching EOD here for me, but I can dig in the morning as well
<blackboxsw_> falcojr: no worries I'll put something up, I think we might need to just refresh the ubuntu/daily/X|B|F branches due to that debian/changelog for the grub postinst changes.
<blackboxsw_> the reason the changelog conflict is because ubuntu/daily/xenial|bionic|focal branches do re-write the debian/changelog as it drops cherry-picked patches
<blackboxsw_> so we get a conflict there as your new-upstream-snapshot run also updates debian/changelog
<blackboxsw_> and that doesn't line up with the d/changelog in the ubuntu/daily/(x|b|f) branch
<blackboxsw_> have a good one either way falcojr we can sort this first thing tomorrow
<falcojr> Thanks blackboxsw_, you too!
<blackboxsw_> johnsonshi: initial review on https://github.com/canonical/cloud-init/pull/549/files# we might be able to get two fixes with one PR (as fallback config needs to blacklist mlx5_core driver too I believe
#cloud-init 2020-08-27
<AnhVoMSFT> Hi folks, under what circumstance can we run into this error "IsADirectoryError: [Errno 21] Is a directory: '/var/lib/cloud/instance'"
<AnhVoMSFT> full output: https://termbin.com/385u
<smoser> AnhVoMSFT:i thought something went in very recently that intended to fix that.
<smoser> but fwiw, you have errors before then
<smoser> AnhVoMSFT: 0755cff078d5931e1d8e151bdcb84afb92bc0f02.
<AnhVoMSFT> ah thanks - that indeed went in recently
<AnhVoMSFT> yep, looking at the log I did see an error during writing ssh key
<smoser> once htat gets created as a directory, then its done, it wont resolve itself.
<smoser> cloud-init clean will probably fix it, or rm -Rf /var/lib/cloud/instace
<AnhVoMSFT> how did it actually get created as a directory in the first place?
<smoser> see the bug for one way
<smoser> there may be other ways that it gets created, though. so it could be only same symptom
<AnhVoMSFT> I have about 10 different cloud-init log with this issue, looking through some of them, the pattern seems to be an error during ssh-authkey-fingerprints
<AnhVoMSFT> KeyError: "getpwnam(): name not found: 'ubuntu'"
<AnhVoMSFT> not related to the last issue, there's another interesting one
<AnhVoMSFT> File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 162, in maybe_perform_dhcp_discovery
<AnhVoMSFT>     dhclient_path = subp.which('dhclient')
<AnhVoMSFT> AttributeError: module 'cloudinit.subp' has no attribute 'which'
<smoser> AnhVoMSFT: it feels like maybe you have a broken install (or maybe an upgrade or something?)...
<smoser> cloudinit.subp definitely does have which in current
<smoser> but it did move recently
<AnhVoMSFT> these are just from random cloud-init.log that were collected as part of our telemetry for failed deployments so I don't have more context into that though. The error just seems odd because like you said, cloudinit.subp definitely has which
<falcojr> blackboxsw_: my brain is in packaging hell right now
<falcojr> https://github.com/canonical/cloud-init/commit/66e114a660c53400e389f119781f378311b65108#diff-4df3321932810ca4e004e72e5587ff28R95
<falcojr> how are we getting the caplog error if we're including that?
<falcojr> also, how did the upstream daily changelog get the latest SRU changelog when the non-daily branch doesn't have it yet?
<blackboxsw_> minor branch for you falcojr https://github.com/canonical/cloud-init/pull/558
<blackboxsw_> this should resolve the error because caplog is non-std on xenial's version of pytest
<falcojr> blackboxsw_: how's this different from the other line I linked?
<blackboxsw_> reading scrollback
<blackboxsw_> ahh falcor, so that package dependency path differs from our daily build recipe debuild -S + sbuild-it path which I believe uses the hard/static file from ubuntu/xenial/debian/control during the build process
<blackboxsw_> falcojr: thx for the review. https://github.com/canonical/cloud-init/pull/558 landed for ubuntu/xenial test pkg deps
<blackboxsw_> falcojr: looks like master moved again with another commit today. So, we are ready to new-upstream-snapshot 371b392ced518e45be51089b6a67b362957b1dba to each ubuntu/X|B|F branch now
<blackboxsw_> if you could re-fresh your existing ubuntu/* PRs then we should be good to go
<blackboxsw_> falcojr: lucasmoura I just kicked off a daily build recipe for xenial to see that it 'works' with current ubuntu/xenial -> ubuntu/daily/xenial branches https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily-xenial
<blackboxsw_> once we know the builder works, we can land falcor's new-upstream-snapshots
<blackboxsw_> knowing that it didn't introduce regressions (which it shouldn't)
<blackboxsw_> falcojr: case and point about revno in the daily build recipe not quite working as we hoped with occasional build collisions https://launchpadlibrarian.net/495249390/upload_2635183_log.txt
<blackboxsw_> which seem to only happen when I manually click the build now button in launchpad
<beantaxi> I notice that cloud-init 20.3 has been announced as Released, and that there is a package for Groovy, but I do not see a package for Bionic or anything else.
<beantaxi> I am a novice at all this ... but is the idea that the cloud-init team is the cloud-init team, and it releases cloud-init code, and then different teams do the packaging for different distros? Even within Canonical? (that would make perfect sense, but that does not mean it's correct)
<blackboxsw_> beantaxi: in order for cloud-init to publish into stable Ubuntu releases we have to follow this process for ubuntu https://wiki.ubuntu.com/CloudinitUpdates
<blackboxsw_> we call this a Stable Release Update (SRU)
<blackboxsw_> falcojr: and I are working today to resolve and queue a -proposed upload of 20.3 into those ubuntu releases
<blackboxsw_> beantaxi: expectation is that once we queue uploads, the cloud-init updated packages will be available in the xenial-proposed bionic-proposed apt pockets and will be testable by anyone following these steps https://cloudinit.readthedocs.io/en/latest/topics/debugging.html#stable-release-updates-sru-testing-for-cloud-init
<blackboxsw_> we'll send out an email to the list as well about this SRU.
<blackboxsw_> generally when we do an upstream release to groovy 20.3 our team tries to queue an SRU into the stable releases within the next week.
<beantaxi> Gotcha ... so it's still packaged by cloud-iniit, but there are extra steps for making a change to a stable release. That makes perfect sense.
<blackboxsw_> that SRU testing minimally takes 7 days, but practically takes average of about 10
<blackboxsw_> yep just so stable releases get more verification (and don't break existing users)
<beantaxi> Is it generally safe, to grab a source release and install from source, over top of an apt package? (for cloud-init anyway; I imagine there's not a generally correct answer)
<blackboxsw_> beantaxi: to watch progress on cloud-init SRUs you can subscribe to the SRU process bug we create
<blackboxsw_> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1893064
<ubot5> Ubuntu bug 1893064 in cloud-init (Ubuntu Focal) "sru cloud-init (20.2-45 to 20.3-0) Xenial, Bionic, and Focal" [Undecided,New]
<blackboxsw_> we will update that bug attaching verification logs for every stable release we validate
<blackboxsw_> and eventually the Xenial/Bionic/Focal items in that bug will go to Fix Released and comments added to that affect
<blackboxsw_> details on this will come out in the SRU email that will be sent to the mailinglist
<blackboxsw_> I need to update the IRC topic since we officially "released" 20.3 to groovy 2 days ago
<beantaxi> Thanks! I just subscribed. Should be interesting to get a feel for the process.
* blackboxsw_ changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting Aug 25 16:15 UTC | 20.3 (Aug 25) |  https://bugs.launchpad.net/cloud-init/+filebug
<blackboxsw_> excellent
* blackboxsw_ changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting Aug 25 16:15 UTC | 20.3 (Sep 8) |  https://bugs.launchpad.net/cloud-init/+filebug
<blackboxsw_> also updated the topic for next cloud-init community status meeting: "Next status meeting Aug 25 16:15 UTC "
* blackboxsw_ changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting Sept 816:15 UTC | 20.3 (Aug 25) |  https://bugs.launchpad.net/cloud-init/+filebug
<blackboxsw_> let's try that again: Next status meeting Sept 816:15 UTC
* blackboxsw_ changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting Sept 8 16:15 UTC | 20.3 (Aug 25) |  https://bugs.launchpad.net/cloud-init/+filebug
<blackboxsw_> ... ok I really need to lay off the coffe
<blackboxsw_> e
<falcojr> blackboxsw_: is there something I can do to help figure out why that upload was rejected?
<powersj> falcojr, where are you uploading to?
<powersj> if it is a ppa you should get an email with a reason
<falcojr> blackboxsw_ was doing it, so I'm not sure :)
<blackboxsw_> falcojr: checking upload
<blackboxsw_> falcojr: you mean groovy?
<blackboxsw_> powersj: falcojr oooh you mean the daily recipe build for xenial that I linked
<falcojr> I was asking about
<falcojr> "falcojr: case and point about revno in the daily build recipe not quite working as we hoped with occasional build collisions https://launchpadlibrarian.net/495249390/upload_2635183_log.txt"
<falcojr> yeah
<blackboxsw_> right sorry. I'm context switching a bit today and not doing that very well.
<blackboxsw_> so the daily build recipes show here https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily-xenial and https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily-bionic
<blackboxsw_> you can dive into specific upload log failures clicking int each build
<blackboxsw_> if needed.
<rharper> falcojr: blackboxsw_:  no one uploads to daily pocket, for SRU we upload to the proposed pocket of the cloud-init-dev ppa ;
<rharper> https://launchpad.net/~cloud-init-dev/+archive/ubuntu/proposed
<blackboxsw_> right, the daily recipe just has occasional upload failures into the daily PPA because the specific version of the deb file already exists in the  daily ppa today (because I clicked the upload button multiple times)
<rharper> yes
<rharper> I think we have to wait for another commit, yeah ?
<rharper> I merged a commit from paride today so tomorrow should get new hash
<blackboxsw_> rharper: yes, it should just be a timing thing for our next commit
<blackboxsw_> so I don't think we need to resolve this as falcojr is going to put up PRs for updated ubuntu/xenial|bionic|focal today for 20.3-X
<rharper> yeah
<blackboxsw_> once that lands the daily recipe will "just work" because commitish and commit count <revno> will have incremented
<blackboxsw_> so ultimately there is nothing to do (and I think we may have already introduced the needed commit) because we are just waiting on
<blackboxsw_> [  ] ï¿¼ cloud-init - 20.3-2566-g1f3a225a-0ubuntu1+1489~trunk~ubuntu16.04.1	 which is already queued in the recipe build
<blackboxsw_> seen at https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily-xenial
<blackboxsw_> so falcojr we should be able to proceed with a PR for your ubuntu/xenial with a  new-upstream-snapshot 371b392ced518e45be51089b6a67b362957b1dba
<blackboxsw_> I think
<blackboxsw_> that'd sync the latest released commit we published via ubuntu/devel into ubuntu/xenial, bionic and focal
<falcojr> cool, sounds good
<falcojr> ok, blackboxsw_ , maybe this one is right now??? https://github.com/canonical/cloud-init/pull/555
<falcojr> I tried the merge and didn't get conflicts this time
<blackboxsw_> woot falcojr xenial recipe build completed and uploaded https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily-xenial
<blackboxsw_> falcojr: awesome getting this PR review now
<blackboxsw_> falcojr: focal merged https://github.com/canonical/cloud-init/pull/553 :)
<blackboxsw_> I'll queue an upload for focal -proposed of 20.3
<blackboxsw_> falcojr: lucasmoura over in ubuntu-release channel we get a bot message confirming queued -proposed uploads Unapproved: cloud-init (focal-proposed/main) [20.2-45-g5f7825e2-0ubuntu1~20.04.1 => 20.3-2-g371b392c-0ubuntu1~20.04.1] (core, edubuntu, ubuntu-cloud)
<blackboxsw_> hhhhhhhrm
<blackboxsw_> right  20.3-2-g371b392c-0ubuntu1~20.04.1]
<falcojr> what does that mean?
<blackboxsw_> ok falcojr that means that I have uploaded (because I have "upload" rights to cloud-init)  into the focal-proposed pocket Unapproved queue... which can be found here https://launchpad.net/ubuntu/focal/+queue?queue_state=1&queue_text=cloud-init
<falcojr> ahh, cool...just didn't know if the "unapproved" was significant
<blackboxsw_> our next SRU step (once we actually have bionic and xenial uploads queued is to ping an SRU vangaurd in #ubuntu-release channel to say, please let our uploads into the official focal-proposed xenial-proposed bionic-proposed apt pockets so that public validation can take place
<blackboxsw_> once sru vanguards "let the upload into -proposed" it goes from Unapproved -> Accepted I think
<falcojr> gotcha
<blackboxsw_> at which point public vms which have focal-proposed apt config enabled would be able to see, download and verify latest cloud-inbit
 * blackboxsw_ looks over your bionic and xenial (I'm thinking we still might get into a bit of a pickle with the ubuntu_advantage quilt patches.....)
<blackboxsw_> falcojr: so, something is wrong with ubuntu/bionic and the quilt refresh route because when we  `quilt push -a` on your branch, we want it to render an cc_ubuntu_advantage.py  that knows how to talk to the old ubuntu-advantage-tools back client, as in accept commands like "ubuntu-advantage disable-livepatch" instead of the new python client which only accepts commands like "ua enable livepatch".
<falcojr> and quilt refresh doesn't do that after I check out the old version?
<blackboxsw_> so, if we ran quilt push -f; quilt refresh, that basically drops the existing debian quilt patch, removing the config parsing and documentation that tells you things like https://cloudinit.readthedocs.io/en/18.5/topics/modules.html#ubuntu-advantage
<blackboxsw_> notice we completely reworked the supported config keys for ubuntu_advantage in tip of cloud-init https://cloudinit.readthedocs.io/en/latest/topics/modules.html#ubuntu-advantage
<blackboxsw_> falcojr: right, it would only do that if we want the final product of cc_ubuntu_advantage.py to be the value at the commitish git checkout f247dd20ea73f8e153936bee50c57dae9440ecf7^ cloudinit/config/cc_ubuntu_advantage.py
<falcojr> ohhhhh, didn't realize that about the config keys
<blackboxsw_> so I *think* what we need to do in both bionic and xenial cases is to start the quilt push -f (to apply all quilt patches that can apply) fix the end result to be the file fromgit checkout f247dd20ea73f8e153936bee50c57dae9440ecf7^ cloudinit/config/cc_ubuntu_advantage.py
<blackboxsw_> we also (painfully) need to apply a minor subp patch (because we migrated subp utilities functions int tip to a new module and that needs to be applied to the result of   git checkout f247dd20ea73f8e153936bee50c57dae9440ecf7^ cloudinit/config/cc_ubuntu_advantage.py
<blackboxsw_> soooo in short (not really short) to generate your ubuntu/xenial and ubuntu/bionic; you'll need to start over with a new-upstream-snapshot 371b392ced518e45be51089b6a67b362957b1dba
<blackboxsw_> and then basically perform the following from this PR description
<blackboxsw_> https://github.com/canonical/cloud-init/pull/435
<blackboxsw_> it has an applicable ua-subp.patch that we'll need to use
<blackboxsw_> and again, once we actually get the real python ubuntu-advantage-tools deb released to xenial and bionic , this whole mess goes away
<blackboxsw_> as we can use tip of master for cc_ubuntu_advantage because the tooling supports it
<blackboxsw_> "the tooling" == ubuntu-advantage-tools version 25.0 or later
<blackboxsw_> which is our target shortly
<blackboxsw_> falcojr: I'll push a PR up for ubuntu/xenial for you to compare against as you are doing it too
<falcojr> blackboxsw_ is there an additional ubuntu advantage commit I need to cherry pick?
<falcojr> after the initial checkout?
<blackboxsw_> falcojr: the order of commands in the PR description...
<blackboxsw_> quilt push -f
<blackboxsw_> (refresh-fix) git checkout f247dd2^ cloudinit/config/cc_ubuntu_advantage.py
<blackboxsw_> (refresh-fix) git checkout f247dd2^ cloudinit/config/tests/test_ubuntu_advantage.py
<falcojr> so just the subp one?
<blackboxsw_> then patch -p1 < ua-subp.patch
<falcojr> gotcha, I'll follow the PR :)
<blackboxsw_> falcojr: only diff in that description is the following:
<blackboxsw_> you aren't working in ubuntu/daily/xenial but ubuntu/xenial
<falcojr> right
<blackboxsw_> and your starting with new-upstream-snapshot 371b392ced518e45be51089b6a67b362957b1db instead of new-upstream-snapshot --skip-release  # to update debian/changelog and bump daily version
<blackboxsw_> the rest I think applies
<blackboxsw_> or at least I'm going through that now too to confirm
<blackboxsw_> at the end we want to quilt push -a in your local ubuntu/xenial and grep my-token in cloudinit/config/cc_ubuntu_advantage.py to make sure we have to old config docs
<blackboxsw_> then quilt pop -a to make sure all patches are popped off the stack
<blackboxsw_> falcojr: I pushed me ubuntu/xenial branch (but haven't created a PR)
<blackboxsw_> so, appropriate for diffs
<blackboxsw_> against yours
 * blackboxsw_ is creating a PR documenting what I did for reference
<blackboxsw_> 'cause this is a pain
<blackboxsw_> ... not your fault, our fault for carrying around a quilt patch like this
<falcojr> blackboxsw_: just pushed mine...I compared ours and the only difference was the names
<blackboxsw_> sorry excellent falcojr lucasmoura I added a dummy WIP PR just with that context again https://github.com/canonical/cloud-init/pull/559
<blackboxsw_> which we will reject and take falcor's I believe
<blackboxsw_> cool, ok I'm trying to build yours and confirm
<blackboxsw_> falcojr: if you could do the sameish for bionic then I think we are "good"
<falcojr> yep
<blackboxsw_> It's also EOD, so no worries if you want to grab that tomorrow. we'll probably have to release 2 mondays from now as it's late ubuntu-archive-wise anyway
<falcojr> I'll finish up this branch real quick and then call it day
<blackboxsw_> so trying to squeeze the -proposed upload in by this time may be too late to make the cut for "SRU aging" of 7 days
<blackboxsw_> thanks sir
<blackboxsw_> build-package/sbuild-it worked on your xenial branch I'm running build-and-push to upload it to -proposed queue now
<falcojr> great, also I'm not able to push tags upstream, so if we want to tag these somebody else will have to do it
<blackboxsw_> build-and-push does that
<falcojr> ah, great
<blackboxsw_> so it should work... and we need to sort your ability to push tags  in the future
<blackboxsw_> ok should be a bot message coming up in #ubuntu-release channel shortly with the xenial upload
<blackboxsw_> I'm generating my own local ubuntu/bionic now
<falcojr> ahhh, was just gonna say mines different! :)
<blackboxsw_> just pushed
<blackboxsw_> oops need --force
<blackboxsw_> ebbaff6ccf57777f729c386365f6a6969d1b6982 commitish on my branch is what I have
<blackboxsw_> diffing yours now
<blackboxsw_> +1 falcojr great work
<blackboxsw_> ok I think we are good
<blackboxsw_> I'll sort details if anything else is missing
<blackboxsw_> I have a bit more before EOD
<blackboxsw_> thanks again
<falcojr> wahoo! Cool, thanks
<blackboxsw_> ok uploads all queued in Unapproved state for xenial bionic and focal. just waiting on Ubuntu SRU vanguard to approve so we can begin testing
#cloud-init 2020-08-28
<Carbrich_r> HI, Stupid question maybe, but if a bug is confirmed, what is the workflow then?
<meena> Carbrich_r: it's discussed, and prioritise or, if someone has a bunch w
<meena> *hunch what the issue is, assigned
<meena> unless, of course, someone from the community steps up and delivers a patch
<rs_goyal__> How can I pass a user-script via vendor data? I am not sure if that is possible also.
<beantaxi> rs_goyal__: I'm new here ... but you definitely want to pass it as vendor data and not user data?
<rs_goyal__> yes @beantaxi
<beantaxi> Then I'm out of ideas :) but I'm sure someone more knowledgable will be along
<beantaxi> For my own curiosity, why vendor data? In case I find myself in the same position some day
<beantaxi> If it would talk too long to explain, no worries
<rs_goyal__> In case of user data, you need to pass it everytime you create an instance. Vendordata will be common for all
<beantaxi> Thanks!
<powersj> rs_goyal__, passing vendor data is dependent on the datasource, for example on lxd you can pass it in by setting a key: https://cloudinit.readthedocs.io/en/latest/topics/faq.html#lxd
<rs_goyal__> ok
<blackboxsw> Carbrich_r: Confirmed bug statue just means someone else confirms this looks like a valid bug, it has yet to be prioritized in any work queues. Once it is in-progress you know it's imminent. But as it currently stands any bug that transistions to Triaged/Confirmed doesn't imply anyone is working that bug. Until it has an assignee it is considered an open bug that anyone in the community with interest can provide
<blackboxsw> patches for. I'm assuming this was a question related to https://bugs.launchpad.net/cloud-init/+bug/1892902
<ubot5> Ubuntu bug 1892902 in cloud-init "Cloud-init received SIGTERM and is thereby unable to install packages or running runcmd completely" [Undecided,Confirmed]
<blackboxsw> I think ultimately "we" (cloud-init upstream folks) should have set that to Triaged. which means the same thing "real bug and unprioritized" except it carries the weight that the "bug supervisor" agrees.
 * blackboxsw changes the bug state drop down
<blackboxsw> and looks like from your attached logs there that ['dnf', '-y', 'makecache'] causes a hiccup for cloud-init process for some reason.
<Carbrich_r> @meena:  thanks for the explanation.
<meena> Carbrich_r: additionally, currently folks might be slow to respond, cuz they're in the middle of an SRU release
<meena> https://wiki.ubuntu.com/StableReleaseUpdates â sru
<blackboxsw> thx again meena for fielding context for folks
<meena> blackboxsw: i try my best, but most of my best is currently directed at my daughter, so i've been neglecting my FLOSS projects that nobody pays me me for :P
<blackboxsw> hah. fam first always :)
<blackboxsw> then FLOSS fam second :)
<beantaxi> Fyi I believe the manual instructions in https://wiki.ubuntu.com/Testing/EnableProposed only work if you are sudo
<beantaxi> I mean root
<beantaxi> This works: `echo "deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe" | sudo tee >/dev/null  /etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list`
#cloud-init 2020-08-29
<beantaxi> I was able to confirm that cloud-init devel make-mime is working as expected with 20.3 via bionic-proposed. I'm happy to note that somewhere if it helps the SRU process (I followed the Wiki but I could not find a specific bug to comment on)
