#cloud-init 2014-06-09
<smoser> Wulf, it looks to me that it needs to be 'ssh_authorized_keys'
<smoser> underscore, not -
<smoser> ah. actually, i tlooks like either 'ssh-authorized-keys' or 'ssh_authorized_keys' will work.
<smoser> as _normalize_users changes '-' to '_'
#cloud-init 2014-06-10
<Wulf> smoser: is it supposed to be fixed on http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/examples/cloud-config-user-groups.txt ? Cause it's not.
<Wulf> ah, I'll just update the bug tracker
<smoser> Wulf, i'm confuse.d
<smoser> the diff you showed and my reading of the code shows that the correct alue is 'ssh-authorized-keys'
<smoser> which is what is shown in tha tlink
<Wulf> oh, you're there. Just updated the ticket
<Wulf> smoser: yes, it's correct with the trailing "s" and in that link the "s" is missing
<smoser> ah. line 72 its missing. its present on line 26
<smoser> so we're both right
<smoser> :)
<Wulf> (:
<smoser> i'll fix that and push it right now.
<smoser> thanks for patience
<Wulf> thanks
<harmw> smoser: how nice, the logo's even on the cirros page :)
<smoser> :)
<smoser> did you aever test the arm stuff ?
<smoser> i got cirros all building on trusty and with the latest buildroot
<smoser> but then my instance ot killed
<smoser> :)
<harmw> hah, no, -still- did't setup my cubietruck
<smoser> so i lost that data. 
<harmw> ohmy
<harmw> smoser: you're not giving away time for free, right?
<harmw> I badly need more then just 24 hrs in a day :p
<smoser> i understand that.
<harmw> does ubuntu have something like armel64?
<harmw> I'm seeing this in my buildscript: [ "$TARGETARCH" = "arm" ] && xarch="armel" && flav="omap"
<harmw> and I'm wondering how easy it may or may not be to have it build arm64 stuff
<harmw> hm, lets see if the scripts builds arm in the first place :)
<smoser> arm definitely builds
<smoser> arm64 migh tneed an update of buildroot
<smoser> but i'm pretty sure i had it building
<smoser> and ubuntu does have arm64 kernels
<harmw> ok, well I'm already building with an updated buildroot
<harmw> and such
<smoser> feel free to merge propose updated buildroot stuff.
<harmw> didn't I already propose such a merge a long time ago? :)
<harmw> Ill look into that tonight
<smoser> did you?
<smoser> its quite possible
<smoser> sometimes, i'm a bigger idiot than normal
<harmw> https://code.launchpad.net/~harmw/cirros/cirros-buildroot-2014.02 thats the one, though I can't remember why I actually did not request a merge jus tyet
<harmw> and I've started https://code.launchpad.net/~harmw/cirros/various-fixes, though with just one fix yet :)
<smoser> harmw, thank you
<harmw> think you can use it?
<smoser> (haven't looked yet :)
<pquerna> smoser: http://cloudinit.readthedocs.org/en/latest/topics/dir_layout.html what do you think about adding /var/lib/cloud/data/availability-zone or would you want it approached another way?
<pquerna> (rather than... import pickle;pickle.loads(open('/var/lib/cloud/instance/obj.pkl').read()).availability_zone  :D
#cloud-init 2014-06-11
<smoser> pquerna, i want to solve the problem you are wanting to solve.
<smoser> but the goal i think is to end up with 2 somewhat-commonly formed files with metadata information.
<smoser> one would contain any thing that potentially was sensitive (ie, user-data) and one non-sensitive.
<smoser> and their on-disk permissions represent that difference.
<smoser> and then 'cloud-init query' would basically load those up and spit out answers, but you could just as easily load the json directly from /var/lib/cloud/instance/data.json 
<smoser> which i guess would ideally be in /run. 
#cloud-init 2014-06-12
<jmickle> hi all
<jmickle> if anyone is around i have 2 questions probably config related that i have been racking my brain on 
<jmickle> if someone could assist it would be greatly appreciated
<smoser> jmickle, whats up ?
<smoser> in general, "don't ask to ask, just ask"
<smoser> if someone sees your question and knows how to reply they will.
<jmickle> hey smoser first thanks for responding and sure thing i didnt know how it worked in here so i didnt want to be rude 
<jmickle> smoser: I am running cloud-init on centos with openstack and when it runs it does not seem to be adding the user to sudo
<jmickle> also growpart does not seem to be running although when i run growpart manually after logging in and rebooting it works
<smoser> on an older kernel to make it all "just work", you'll need a drakut module cloud-initramfs-growroot
<smoser> on 3.8 and later, cloud-init can do it all on mounted filesystems.
<smoser> i think though that reboot would probably get you what you needed. 
<jmickle> interesting ok
<jmickle> what about sudo ? it seems to be running the add ssh key
<jmickle> but not adding the sudo line for the user
<jmickle> though i did turn off the disable root login
<smoser> can you paste cloud-init config ? /etc/cloud/cloud.cfg and anything in /etc/cloud/cloud.cfg.d/*
<jmickle> the only thing i changed was disable root: 1 to 0
<smoser> root login should be disabled by default.
<smoser> oh. i see.
<jmickle> yeah i turned it off temporarily because i had a networkign issue at first
<jmickle> and keys werent getting added
<jmickle> so i needed to login with the root
<jmickle> could that be why?
<smoser> nah. it should still do it.
<smoser> what cloud-init version ?
<jmickle> 0.7.4
<smoser> - RH: require sudo >= 1.7.2p2-3 (with sudoers.d/)
<smoser> htats just to verifiy that its not writing the file but sudo not caring
<jmickle> im sorry i lost you
<jmickle> you mean use cloud-init >= 1.7.2?
<smoser> sudo
<smoser> rpm -qi sudo
<jmickle> ah
<jmickle> 1.8.6p3 
<smoser> k. that would seem san.e
<smoser> http://paste.ubuntu.com/7634815/
<smoser> so your config has to have a 'sudo' line for the user
<smoser> (default_user)
<jmickle> got it which the default config from epel does not
<jmickle> ok
<smoser> and also see the new style
<smoser> users:
<smoser>    - default
<smoser> rather than
<smoser> user: <foo>
<jmickle> ok
<jmickle> thank you so much
<jmickle> il try the reboot with the grow part
<jmickle> maybe it is running and just needs a reboot to kick in or something
<smoser> if you run growpart on kernel < 3.8
<smoser> you have to reboot
<jmickle> ok yeah centos 6.5 is 2.6
<jmickle> ty so much
<smoser> the fix is to add that dracut module
<smoser> and then it resizes in the initramfs
<smoser> when the filesystem is not mounted
<jmickle> ok do you have an example of that or documentation for it by chance?
<smoser> (older kernels will not allow updates to partition tables of "physical disks" that have filesystems mounted on them)
<jmickle> or is it in that config?
<jmickle> ok
<smoser> http://docs.openstack.org/image-guide/content/ch_openstack_images.html
<smoser> https://apps.fedoraproject.org/packages/cloud-initramfs-tools/changelog
<jmickle> ty
<jmickle> is there a way to pass the hostname into the node-name variable in cloud init config for chef?
#cloud-init 2014-06-13
<xmltok_> hi, does anyone know why the short hostname is perferred when setting the hostname on debian boxes?
<smoser> harlowja_away, what was that library for network config and description
<harlowja> smoser https://fedorahosted.org/netcf/ 
<harlowja> netcf
<smoser> thank you
<harlowja> np
<harlowja> smoser still seems somewhat active (as a project), so thats good
<harlowja> https://git.fedorahosted.org/git/netcf.git
#cloud-init 2015-06-08
<openstackgerrit> Joshua Harlow proposed stackforge/cloud-init: Bring over the 'url_helper' module from bzr  https://review.openstack.org/170242
<harlowja> sweet, it works
<harlowja> we have bot!
<harlowja> ha
<claudiupopa> that's cool. :d
<smoser> horay!
<harlowja> now never will miss a review, ha
<harlowja> guess i just need to adjust some review comments, lol
<harlowja> will get around to that :-P
<harlowja> smoser who else should be operator in this channel?
<harlowja> Odd_Bloke and claudiupopa ?
<smoser> i'm good with those
<harlowja> k
<smoser> do we need to do anything else ?
<claudiupopa> thanks.
<harlowja> 	flags #cloud-init claudiupopa +oO
<harlowja> soooo that one is running into issues
<harlowja> -ChanServ-	claudiupopa is not registered.
<harlowja> sooo might need to register your IRC account claudiupopa
<harlowja> * http://meta.wikimedia.org/wiki/IRC/Instructions#Register_your_nickname.2C_identify.2C_and_enforce
<harlowja> let me know when thats done and i can try to run that command again
<claudiupopa> done.
<harlowja> k, cool
<harlowja> done
#cloud-init 2015-06-09
<openstackgerrit> Joshua Harlow proposed stackforge/cloud-init: Bring over the 'url_helper' module from bzr  https://review.openstack.org/170242
<Odd_Bloke> smoser: Do you have any thoughts on moving cloud-init 0.7.x development to git in Launchpad?
<openstackgerrit> Claudiu Popa proposed stackforge/cloud-init: Add the data source base classes and the HTTP OpenStack implementation  https://review.openstack.org/188327
<smoser> i'm not really opposed to that.
<openstackgerrit> Claudiu Popa proposed stackforge/cloud-init: Add the layout for a couple of Windows OS utils, especially networking  https://review.openstack.org/188361
<openstackgerrit> Claudiu Popa proposed stackforge/cloud-init: Add the data source base classes and the HTTP OpenStack implementation  https://review.openstack.org/188327
<Odd_Bloke> harlowja_: Do you know if we can get the bot to use notices rather than messages?
<claudiupopa> hey guys. Short question, what exactly is cloudinit.sources.DataSource.ds_cfg?
<Odd_Bloke> claudiupopa: It's generally used for configuring things that are specific to a datasource.
<Odd_Bloke> claudiupopa: See Azure, for example.
<Odd_Bloke> claudiupopa: Azure requires some DHCP hijinks on first boot, which are configured under the hostname_bounce key.
<Odd_Bloke> It wouldn't make sense to have that for any other data source.
<claudiupopa> it's taken from a config file or something?
<Odd_Bloke> claudiupopa: I'm less sure about that (I've only ever really played with the built-in config for the Azure source), but line 70 of cloudinit/sources/__init__.py suggests that is the case.
<Odd_Bloke> Presumably cloud.cfg.
<openstackgerrit> Claudiu Popa proposed stackforge/cloud-init: Add the layout for a couple of Windows OS utils, especially networking  https://review.openstack.org/188361
<openstackgerrit> Claudiu Popa proposed stackforge/cloud-init: Add the data source base classes and the HTTP OpenStack implementation  https://review.openstack.org/188327
<harlowja_> Odd_Bloke don't think so, but not sure (as for notices rather than messages)
<harlowja_> Odd_Bloke https://github.com/openstack-infra/gerritbot is the code for it
<harlowja_> i've modified it somewhat previously (i have a internal version running on internal y! irc that notifies whenever a yahoo employee puts a review up)
#cloud-init 2015-06-10
<Odd_Bloke> harlowja: Oh, hm, it looks non-trivial to change it to be configurable.
<Odd_Bloke> I'll just hack it in my IRC client. ^_^
<openstackgerrit> Claudiu Popa proposed stackforge/cloud-init: Add the data source base classes and the HTTP OpenStack implementation  https://review.openstack.org/188327
<openstackgerrit> Merged stackforge/cloud-init: Add the layout for a couple of Windows OS utils, especially networking  https://review.openstack.org/188361
<Odd_Bloke> Woohoo, my client hacking worked.
<smoser> claudiupopa, http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/examples/cloud-config-datasources.txt is example datasource config
<Odd_Bloke> smoser: For clarity, when you say JSONP you mean http://jsonpatch.com/ rather than https://en.wikipedia.org/wiki/JSONP, right?
<smoser> jsonpatch
<smoser> i did say jsonp
<smoser> which yeah, i say those wrong
<smoser> http://jsonpatch.com/
<Odd_Bloke> claudiupopa: ICYMI: ^
<claudiupopa> thanks
<smoser> claudiupopa, http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/util.py#L335
<smoser> i think actuall 'read_conf_with_confd' is what is used...
<ByPasS> Hi everyone, quick question here : I'm trying to get password injection working with centos image atm. I have enabled root (disable_root 0). So the passwd is injected to root but in the /etc/shadow file I still have the double bang infront of the passwd so I'm not able to login in the console
<ByPasS> am i missing something ?
<Odd_Bloke> ByPasS: How are you injecting the password?
<ByPasS> with metadata (using kvm hypervisor)
<ByPasS> eventually id rather inject it to the defined user for security purposes
<Odd_Bloke> ByPasS: Well, if you want security, use SSH keys. ;)
<Odd_Bloke> ByPasS: Could you paste the metadata you're using?
<ByPasS> agreed but my cEO wants a way for dumb ppl to login :)
<ByPasS> gimme a min need to help a fellow co worker
<Odd_Bloke> Sure.
<Odd_Bloke> (I'm in a meeting in 10 minutes so if I start ignoring you then, that's the reason :p)
<ByPasS> np we can catch up later :D
<Odd_Bloke> ByPasS: Drop that metadata in a paste, and I'll take a look at it once I'm out.
<Odd_Bloke> (No point waiting until we're both not busy ;)
<larsks> Hello all.  Curious if there are any thoughts on https://code.launchpad.net/~larsks/cloud-init/fix-systemd-detection/+merge/260885...merge-able? Needs more work?
<Odd_Bloke> larsks: I missed that you'd added tests!  I'll try to get to it later today.
<larsks> Odd_Bloke: awesome, thanks.
<Odd_Bloke> I was about to say I wouldn't be able to merge it, but I can now. \o/
<Odd_Bloke> harlowja: You're listed as a reviewer on https://code.launchpad.net/~larsks/cloud-init/fix-systemd-detection/+merge/260885; do you want to review it, or are you happy for me to merge based on my Approve review?
<smoser> Odd_Bloke, what do you think about https://code.launchpad.net/~bbaude/cloud-init/rh_subscription/+merge/259159 right now ?
<smoser> rangerpbzzzz, ^ i havne't forgotten about you
<Odd_Bloke> smoser: I'm happy with it; shall I merge?
<smoser> larsks, and you signed contributors agreement ?
<smoser> Odd_Bloke, i'll pull it. going to pull larsks too
<Odd_Bloke> smoser: Would you mind if I do one, to get the hang of it?
<gchristensen> Hi, I'm looking to use the public hostname from the ec2 metadata service in a cloudinit file. it doesn't seem to be making it available, as it is only querying a few metadata endpoints: http://169.254.169.254/2009-04-04/meta-data/public-keys, http://169.254.169.254/2009-04-04/meta-data/hostname, http://169.254.169.254/2009-04-04/meta-data/local-ipv4, and
<smoser> Odd_Bloke, ok.
<gchristensen> http://169.254.169.254/2009-04-04/meta-data/public-ipv4. is there a way to instruct cloud-init to query ://169.254.169.254/latest/meta-data/public-hostname as well? my expectation was using $public_hostname would "just do it" but that odoesn't seem to be the case.
<smoser> please update the ChangeLog when you do.
<larsks> smoser: pretty sure I did, yes.
<smoser> did we already ahve this conversation ?
<smoser> gchristensen, sadly it doesnt do that.
<Odd_Bloke> smoser: Let me know once you've done one, and I'll do the other.
<gchristensen> smoser: ok, and no way to (easily) extend it to do that?
<smoser> well, it'd take a code change.
<smoser> oh. i'm sorry.
<smoser> "in a cloudinit file"
<smoser> ?
<gchristensen> yeah, I think that is my only interface I can use here.
<smoser> i kind of assumed you meant /etc/hosts or rendered file.
<gchristensen> oh, yeah, in a cloudinit file
<smoser> larsks, i have to and see if you have :-(
<smoser> since you're not in https://launchpad.net/~contributor-agreement-canonical/+members#active it means eiterh you didn't do it, or someone hasn't put you in that group
<larsks> Well, will go do that now.  I was pretty sure I had previously signed the canonical cla, but maybe I'm mis remembering.
<larsks> smoser: You would be the "project contact"?
<smoser> larsks, sure.
<smoser> Odd_Bloke, merged larsks's change.
<Odd_Bloke> smoser: I've merged the other one; can you give it a once-over to confirm I've done it properly?
<Odd_Bloke> (lol bzr etc.)
<smoser> i like 74 character width limit
<smoser> you didn't update ChangeLog
<Odd_Bloke> Oh, FFS.
<smoser> which i wish there was a sane way to do..
<Odd_Bloke> I wish I could follow simple instructions. :p
<smoser> i guess at some point if we're requiring nice git style messages , then we can make a tool that did that.
<smoser> humans suck
<smoser> they are completely incapable of doing anything well consistently
<Odd_Bloke> Indeed.
<Odd_Bloke> smoser: Shall I push up a commit adding the ChangeLog entry?
<smoser> sure.
<smoser> and give rangerpb credit in that style there.
<Odd_Bloke> Done and pushed.
<harlowja> Odd_Bloke whats this launchpad review thing u talk about, all i know is review.openstack.org :-P
<smoser> look at harlowja being funny and stuff.
<harlowja> :-P
<harlowja> claudiupopa will tweak that review in a sec
<harlowja> my mac is acting really slow for some reason
<harlowja> damn apple, lol
<claudiupopa> no problem.
<smoser> if only there were an operating system that was pretty and easy to use that did not come from steve jobs.
<harlowja> lol
<harlowja> i use that one @ home :-P
<harlowja> xubuntu though, not the main-line one
<openstackgerrit> Joshua Harlow proposed stackforge/cloud-init: Bring over the 'url_helper' module from bzr  https://review.openstack.org/170242
<openstackgerrit> Merged stackforge/cloud-init: Bring over the 'url_helper' module from bzr  https://review.openstack.org/170242
<openstackgerrit> Merged stackforge/cloud-init: Add tox targets for coverage testing.  https://review.openstack.org/188739
<openstackgerrit> Merged stackforge/cloud-init: Add tox targets for coverage testing.  https://review.openstack.org/188739
<a-tal> hey. i'm trying to get cloud-init going with a private cloud, the meta-data service isn't serving 2009-04-04 data, but it is using latest, what's the config key for that?
<a-tal> i see it mentioned here (http://cloudinit.readthedocs.org/en/latest/topics/datasources.html#ec2) but I don't see how it's set
#cloud-init 2015-06-11
<Odd_Bloke> smoser: Do we have a good way of specifying apt mirrors for different distros via vendor-data?
<Odd_Bloke> harlowja: Tut, tut, only 25% coverage of url_helper.py.
<Odd_Bloke> Oh, no, tox wasn't recreating a virtualenv to include new test deps.
<Odd_Bloke> harlowja: Tut, tut, only 81% coverage of url_helper.py. ^_^
<Odd_Bloke> Oh, hmph, the Windows changes have completely broken the coverage reporting.
<Odd_Bloke> To ensure that we're getting accurate numbers (including modules not touched by tests), nose imports all the modules.
<Odd_Bloke> As you might imagine, this goes poorly.
<openstackgerrit> Daniel Watkins proposed stackforge/cloud-init: Don't use --cover-inclusive; Windows modules break it.  https://review.openstack.org/190557
<Odd_Bloke> harlowja: claudiupopa: ^ fixes the coverage breakage.
<Odd_Bloke> I'm looking at a fix in nose, which will allow us to reintroduce it.
<Odd_Bloke> But, for now, there's no way around it.
<Odd_Bloke> https://github.com/nose-devs/nose/pull/924 is the nose fix.
<smoser> Odd_Bloke, i'd have to look.
<smoser> not really here today, just came in for a call.
<Odd_Bloke> smoser: No worries; enjoy your absence. :)
<a-tal> hey, does anyone know how I can configure cloud-init to use a metadata version endpoint that's not 2009-04-04 ?
<Odd_Bloke> a-tal: Which data source are you using?
<a-tal> ec2
<Odd_Bloke> a-tal: I don't think you can.
<Odd_Bloke> a-tal: But even if you could, cloud-init wouldn't know what to do with anything more recent than that without code changes.
<Odd_Bloke> a-tal: Is there something specific that you need?
<a-tal> so, it's not the real ec2, but it does respond to http://169.254.169.254/latest/meta-data/instance-id and the other endpoints using latest. you can configure the datasource metadata_urls; but that's the path/version seems to be hardcoded?
<a-tal> s/that's//
<Odd_Bloke> a-tal: Yeah, we hard-code the version because we know we'll work against that version.
<Odd_Bloke> a-tal: EC2 could completely change where everything is located in 2015-06-11, but we'd still be fine because we're pinned at that version.
<Odd_Bloke> (Where 'that version' is 2009-04-04')
<a-tal> makes sense
<Odd_Bloke> So your fake EC2 will have to provide those URLs and, crucially, actually do what the API should do in that version.
<Odd_Bloke> I would imagine that's not much of a problem (though I haven't looked at it at all), I suspect the parts of the API that cloud-init exercises aren't changed a great deal.
<a-tal> yeah. i think i'll have to try to get a rewrite rule added
<a-tal> hopefully nothing explodes :)
<Odd_Bloke> :)
<ByPasS> Odd_Bloke : hi again :)
<Odd_Bloke> ByPasS: Hola!
<ByPasS> Odd_Bloke : I realized that I had libvirt injection set to true which well from what Ive read is not needed and should be set to false for cloud-init admin password injection
<ByPasS> Odd_Bloke : back to square one :)
<ByPasS> Odd_Bloke : so I disabled libvirt password and key injection (keys are injected according to the log)
<ByPasS> Odd_Bloke : which metadata url exactly you wanted me to paste ?
<Odd_Bloke> ByPasS: So how are you expecting the password to be set at the moment?
<ByPasS> Odd_Bloke : via cloud-init + nova metadata
<ByPasS> Odd_Bloke : leet me try something I found a probable cause brb
<Odd_Bloke> Sure. :)
<frickler> how are chances to get manage_resolv_conf working on Ubuntu? we have a case where we want to override nameservers for a specific subset of instances only
<frickler> see https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1394224 and https://answers.launchpad.net/ubuntu/+source/cloud-init/+question/234041
 * frickler waves to smoser a bit ashamed about not having responded yet regarding cirros :D
<harlowja> Odd_Bloke only 81% i want 181%
<harlowja> lol
<harlowja> harder faster, stronger, lol
<harlowja> or else fired
<harlowja> ha
<openstackgerrit> Merged stackforge/cloud-init: Don't use --cover-inclusive; Windows modules break it.  https://review.openstack.org/190557
#cloud-init 2015-06-12
<frickler> smoser: thx for getting me back in, my instance here got rebooted this morning and I hadn't saved my latest irssi config yet
<smoser> cloud-init would need a resolvconf handler.
<smoser> the other thing you could do ...
<smoser> is just remove the link before publishing the image
<smoser> if /etc/resolv.conf is not a link into /run
<smoser> then resolvconf will not do anything
<smoser> you might be able to apply that change in a boothook too
<smoser> but it might be racy
<Odd_Bloke> smoser: If you could have a look at https://code.launchpad.net/~daniel-thewatkins/cloud-init/lp1464253/+merge/261849, it would be much appreciated.
<smoser> Odd_Bloke, i will i cant do it now, but 70 minutes
<Odd_Bloke> Ack.
<Odd_Bloke> harlowja: https://review.openstack.org/#/c/191081/ is my project-config change.
<Odd_Bloke> claudiupopa: I'm looking at https://github.com/stackforge/cloud-init/blob/master/cloudinit/tests/osys/test_base.py#L22-L42; does platform.linux_distribution really return 'Windows' if called on Windows?
<Odd_Bloke> harlowja: We need to improve coverage (because otherwise we won't be able to land any changes once that project-config change lands).  Do you want to/are you looking at improving test coverage of url_helper, or is that something I could do?
<claudiupopa> That's a bit misleading, but in fact platform.system will be called in that condition.
<Odd_Bloke> claudiupopa: Then I think we have a bit of a faulty test. :)
<claudiupopa> Yeah, it is, thanks for noticing.
<Odd_Bloke> :)
<openstackgerrit> Claudiu Popa proposed stackforge/cloud-init: Fix a faulty test  https://review.openstack.org/191085
<claudiupopa> Odd_Bloke: ^ here's the patch.
<Odd_Bloke> +1'd.
<claudiupopa> thanks! will need another +2 of course ;-)
<Odd_Bloke> WTB +2 rights. :p
<rangerpb> smoser, Odd_Bloke heya fellas, quick question ... can cloud-init log without rsyslog? like to a local file?
<claudiupopa> Odd_Bloke: would it make sense if url_helper.wait_any_url would return a tuple of url and response?
<claudiupopa> I have an use case where I need the backoff logic, but I also need the response, which isn't provided.
<Odd_Bloke> I don't see why not.
<Odd_Bloke> harlowja: ^
<openstackgerrit> Claudiu Popa proposed stackforge/cloud-init: Change the API of wait_any_url to return a tuple of url and response  https://review.openstack.org/191100
<Odd_Bloke> I see two different types of license header on files; which should I be using?
<claudiupopa> The less verbose one. The other should be replaced.
<Odd_Bloke> Cool.
<Odd_Bloke> Thanks. :)
<smoser> rangerpb, yes.
<smoser> rangerpb look in etc/cloud/cloud.cfg.d/05_logging.conf
<smoser> its *supposed* to figure out whether or not it can use rsyslog and not use it if it cant.
<smoser> but that doesn't work so terribly well
<smoser> short answer for you though... comment out the line about syslog. and it will just do what you want
<smoser> - [ *log_base, *log_syslog ]
<rangerpb> ok roger that thanks smoser
<openstackgerrit> Merged stackforge/cloud-init: Fix a faulty test  https://review.openstack.org/191085
<openstackgerrit> Daniel Watkins proposed stackforge/cloud-init: Don't try to generate autodoc for Windows modules.  https://review.openstack.org/191137
<openstackgerrit> Daniel Watkins proposed stackforge/cloud-init: Clean up stale auto-generated autodoc files.  https://review.openstack.org/191139
<openstackgerrit> Daniel Watkins proposed stackforge/cloud-init: Basic implementation of a reporting framework.  https://review.openstack.org/191144
<Odd_Bloke> smoser: ^ is a basic outline of the reporting framework.
<openstackgerrit> Daniel Watkins proposed stackforge/cloud-init: Fail the doc build if we have any warnings.  https://review.openstack.org/191149
<smoser> Odd_Bloke, i'd realy rather not use wget.
<smoser> that just seems like failure
<smoser> but that bug just plain sucks
<Odd_Bloke> smoser: CloudStack use wget for their equivalent functionality, so they are only every going to test with wget.
<Odd_Bloke> smoser: Using anything else is just asking for regressions in the future.
<smoser> not really.
<smoser> saying "our http endpoint might arbitrarily not be an http endpoint" is asking for regressions
<smoser> i suspect they're not testing with all versions of wget from 10.04 to 16.04 and also those found on linux-distro-X version Y
<Odd_Bloke> Sure, but they are _definitely_ not testing with any version of cloud-init. :p
<smoser> i admit that i'd probably be as opposed to your implementation of a solution in any way.
<Odd_Bloke> I don't disagree that CloudStack are asking for regressions (and that the way they implemented this to begin with is embarassingly terrible).
<Odd_Bloke> But they are and they did and it's deployed in places now.
<smoser> but it was broken in one specific way
<smoser> right?
<smoser> and now its a sane response ?
<Odd_Bloke> Yeah.
<Odd_Bloke> (For now Â¬.Â¬)
<smoser> ok. so 2 paths
<smoser> a.) your wget path.b
<smoser> b.) use url_helper
<smoser> and catch exception and fallback to either old code or wget.
<Odd_Bloke> smoser: With (b), we'd have to hit the endpoint more than once.
<smoser> and in either path, with wget, use long names in command line options rather than short.
<smoser> wget --quiet --timeout=
<smoser> ...
<smoser> we only hit it more than once when its broken.
<smoser> oh yeah, and with wget alos, do not use shell=True.
<smoser> i dont know why you did.. i dont think you gain anthing from it other than the possibility of shell expansion on funny domu_requst input
<Odd_Bloke> It was breaking up DomU_Request in a way that was breaking everything.
<Odd_Bloke> And I was sick of CloudStack and their shitty implementation, so just did it the fastest way I could. Â¬.Â¬
<smoser> _do_request(domu_request="foo\"; rm -Rf /;")
<Odd_Bloke> If we merge code that does that, we almost certainly have bigger problems. :p
<Odd_Bloke> But: point taken.
<Odd_Bloke> I'll revisit this on Monday.
<smoser> if we merge code that does what ?
<smoser> if you wantto put in the wget... i'm fine with that for now. i think its not the greatest.
<smoser> but do want no shell=True and use long command line options.
<Odd_Bloke> smoser: If we merge code that calls _do_request with those parameters.
<Odd_Bloke> Yep, I'll fix that on Monday.
<Odd_Bloke> I'm EOD'ing now.
<Odd_Bloke> smoser: Have a good weekend. :)
<smoser> k
<openstackgerrit> Daniel Watkins proposed stackforge/cloud-init: Fail the doc build if we have any warnings.  https://review.openstack.org/191149
<openstackgerrit> Daniel Watkins proposed stackforge/cloud-init: Basic implementation of a reporting framework.  https://review.openstack.org/191144
<harlowja> Odd_Bloke can also include https://pypi.python.org/pypi/doc8 if u want
<harlowja> another tool i helped mak
<harlowja> can be used for more extensive testing of doc things
<Odd_Bloke> harlowja: Nice, I'll have a look at that. :)
<harlowja> ex: https://github.com/openstack/taskflow/blob/master/tox.ini#L65
<harlowja> using it to blow-up the py27 testing env, if it has some issues
<openstackgerrit> Daniel Watkins proposed stackforge/cloud-init: Add doc8 checks to docs tox target.  https://review.openstack.org/191162
<harlowja> Odd_Bloke your reporting framework i wonder if it could benefit from http://docs.openstack.org/developer/taskflow/types.html#taskflow.types.notifier.Notifier
<harlowja> ^ anyone one of my other project
<harlowja> could split that off into its own little library or something
<harlowja> your's looks slightly similarish
<harlowja> https://pypi.python.org/pypi/blinker/ is also something similarish
#cloud-init 2016-06-13
<aps> Hi guys. Is there a way to suppress output of "apt_update" and "package_upgrade"?
<smoser> aps, as in you dont want them going to /dev/console ?
<smoser>  http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/examples/cloud-config.txt#L599
<harlowja> mgagne if u get anytime reviews on  https://code.launchpad.net/~harlowja/cloud-init/cloud-init-net-sysconfig/+merge/297115  would be cool
<harlowja> that should be the thing that lets u get rid of https://github.com/mgagne/cloud-init-fedora-pkg/blob/epel7/cloud-init-0.7.5-network-info-support.patch
<harlowja> if rharper then does a networkd (?) renderer, then the world will all be merry
<mgagne> harlowja: ok, will I need to rebuild a package?
<harlowja> likely, especially since that hasn't merged yet :-P
<smemsh> hello, i'm making a cidata volume for local attach to a vm (i.e. not cloud).  i understand that network config has to be in metadata, not userdata.  however it looks like straight debian format.  does this format work for a centos image? will cloud-config "know" how to translate it?
<smoser> smemsh,yes, cloud-init will convert the debian interfaces style format into appropriate distro config.
<smoser> this is very m uch in progress thoguh, and recently improved.
<smemsh> smoser: it won't require me to be using systemd will it?
<smoser> and harlwoja is working on the renderer for centos
<drq> is cloud-init/freebsd-configdisk included in 0.7.6? I installed py26-cloud-init on FreeBSD 10.3-RELEASE. and I'm having issues with it seeing a mounted ISO.
<smemsh> smoser: i'm using the centos6-generic cloud image, which uses afaik the "old" /etc/sysconfig/network-scripts way
<smoser> harmw_, ^ ?  drq i have very little experience there. if you wanted to let me in i could maybe quick poke around..
<smoser> smemsh, that should be fine.
<smemsh> i see that google cloud is listed as the providers that use cloud-init, but it doesn't seem to be in their stock images
<drq> On our deploy scripts for linux on VMware, we mount an ISO with the config data on it. We haven't started using FreeBSD yet, but I'm trying to get it ready. I have an ISO mounted (not real data), but it appears that it doesn't look at /dev/cd0 for anything.
<drq> datasource_list is
<drq> datasource_list: ['ConfigDrive']
<smoser> drq, do you have blkid available ? it uses that to find things.
<smemsh> i found it easier to use a vfat with a label.  it's just a mkfs -t vfat -n cidata of any old thing
<drq> is that installed as a dependency? blkid isn't found on the host.
<smemsh> is blkid on non-linux?
<drq> pkg e2fsprogs-libblkid-1.42.13 might have it. Let me install that.
<smemsh> libblkid-1
<smemsh> i assumed that just parsed stuff in /sys, surprised it works on freebsd
<smoser> drq, see tools/build-on-freebsd
<smoser> in trunk
<drq> OK. let me try to find that.
<drq> there are so many cloud-init's in github. which is it.
<smoser> none :)
<smoser> https://launchpad.net/ubuntu/+source/cloud-init
<smoser> we shoudl be moving to git on launchpad soon.
<smoser> which i might well maintain a mirror in github to help clear that confusion
<smemsh> i had same question.  openstack has one i'm assuming would be reasonable fresh
<smemsh> they don't say in the repo comment that it's a mirror though, they should
<drq> got launchpad up in browser. I'll peruse that and see what I find and chew on it overnight. Thanks. Be back in the morning.
<smemsh> latest commit october 2015 hmm
<smoser> smemsh, well, the stuff under openstack was intended to be for cloud-init 2.0
<smoser> which kind of fizzled as we 've focused more on the 0.7 branch
<smemsh> smoser: yeah howcome latest is 0.77, but next is 2.0 ? what happened to 1.0?
<smoser> well, 2.0 was picked as next just because it was intende to include windows support, which woudl be heavy rewrite. but that has been put on hold at least in the near future
<smemsh> looks like there is https://git.launchpad.net/~usd-import-team/ubuntu/+source/cloud-init
<smoser> smemsh, that is actually ubuntu packaging.
<smemsh> oh the debcheckout
<smoser> right.
<smoser> the usd-import-team branches are for all packages. they're what "ubuntu distributed development" used to be for bzr
<smoser> they're just getting started really, but they will represent what was actually in the archive
<drq> smoser -> installing e2fsprogs installed blkid and it sees the ISO and does complain about the ISO I have mounted. Now we're cooking. Thanks.
<smemsh> debian's is based on bzr1215
<smemsh> well, the jessie backport is bzr1156 and jessie is 976
<smemsh> i used lp:~smoser/ubuntu/yakkety/cloud-init/pkg guess that is 0.7 tip
<smemsh> oh nice there is lxd module
<smoser> smemsh, the network configuration rendering is *heavily* updated in the last yakkety upload (ie, really recent)
<smoser> and will get more for centos soon.
<smoser> with harlowja's work.
<harlowja> eventualy
<smemsh> hrm, well i'm hoping the latest upstream centos generic cloud images have whatever they need for it to work correctly... i will find out shortly.  working out the syntax atm (docs are a little sparse for some things)
<smemsh> wow cool you can provide cc data via /proc/cmdline??? that's really neat
<smemsh> what is the instance-id used for? i can set this to gibberish right?
<smemsh> i'm just booting in local vm
<smoser> smemsh, right.
<smoser> it shoudl be unique to this "instance".
<smoser> if you change it that image will then think it needs new.
<smemsh> to support reprovision without a data wipe / reset ?
<smoser> right.
<smoser> cloud-init supports shutdown and snapshot and new instance from taht snapshot
<smoser> and i think that is a pretty common work flow on amazon..
<smoser> a.) start a ubuntu image
<smemsh> ah so like an iterative cloning
<smoser> b.) ssh in or automated do some stuff
<smoser> c.) shutdown
<smoser> d.) snapshot
<smoser> e.) new gold master .... profit!
<smemsh> yeah that makes sense.  however i would advocate for reset, the changes should be done in the automation layer (in my case ansible) and easy to reproduce the initial parts.  but it could be time consuming, i can see where it's useful to support the shortcut (and not everyone needs a layer after cloud-init)
<smemsh> i was thinking that after reading the list of all the modules, it's like, duplicating a lot of the modules in the config/orch tool.  was thinking hm if i can do all this in cloud-init, do i even need ansible? but it's like you can't go backwards or be declarative with cloud-init so i dunno
<smemsh> i can sort of see that as happening though in some future.  like cloud-init evolving into a more general-purpose orchestration tool that knows the full state of the machine
<smemsh> the truth store / metadata can be updated and cloud-init can poll it or subscribe or something and then determine what needs to change
<smemsh> with the NoConfig i.e. using a cidata-labeled filesystem, do i have to insert ssh key data into the config file or can i use #include and file:// urls just like for real metadata sources? idea being, to copy the key data from outside onto the vfat filesystem rather than inserting in /user-data
<smemsh> oh, it would still have to be the same format anyways, i couldn't just insert contents of a file.  so that won't work.  but the question is still valid, can i split up the files that way
<smemsh> i'm guessing /user-data can contain anything that it could from whatever source.  so i guess the only question is whether file:// urls will look up from other files on the config filesystem or if it's only http scheme supported
<smemsh> are 'hostname' and 'local-hostname' just aliases?
<smemsh> doc/examples/cloud-config-resolv-conf.txt has "manage-resolv-conf: true" but should be underscores
#cloud-init 2016-06-14
<aps> smoser: I just want to silent logs for apt tasks, like with the -q flag to normal apt-get commands
<aps> I do want logs for other commands
<harlowja> ok eyes on https://code.launchpad.net/~harlowja/cloud-init/cloud-init-net-sysconfig/+merge/297115 would be nice :)
<harlowja> more data samples would be cool to
<sather> I'm having trouble finding 'official' documentation on creating a module
<smoser> harlowja, more examples of ?
<smoser> Odd_Bloke, harlowja i have shame for both of you
<smoser> sather, what kind of module you want ? config ?
<smoser> harlowja, http://paste.ubuntu.com/17334762/ <-- that is you
<smoser> Odd_Bloke, http://paste.ubuntu.com/17334843/ <-- that is you
<sather> smoser: parsing user-data and writing it to a yaml file for another service to process later
<smoser> sather, check /var/lib/cloud/instance/user-data.txt
<smoser> and /var/lib/cloud/instance/user-data.txt.i
<mgagne> harlowja: what's the best way to download a diff of this merge request?
<smoser> mgagne, the 'Download diff' link at https://code.launchpad.net/~harlowja/cloud-init/cloud-init-net-sysconfig/+merge/297115 ?
<smoser> maybe im' missing something
<mgagne> smoser: yea, I'm not geared to rebuild a new package against bzr
<mgagne> smoser: I can handle diff fine
<mgagne> oh
<mgagne> found the link
<mgagne> https://i265227674.restricted.launchpadlibrarian.net/265227674/b1aed10a-3259-11e6-8a1a-002481e91f22.txt?token=KWhCrx10JmGLMNdFlQVPDw4PBP3Cfm91
<mgagne> ok looks like I don't have the appropriate version to patch against. I used https://launchpad.net/~smoser/+archive/ubuntu/cloud-init-dev and it looks to be outdated compared to trunk.
<smoser> yeah, mgagne i was just fixing that.
<smoser> harlowja, if you run 'tox' do you get a file 'c' in top dir ?
<smoser> tests/unittests/test_cli.py that seems to create it
<harlowja_> smoser hmmm
<harlowja_> let's see
<smoser> harlowja_, i fixed.
<harlowja_> k
<harlowja_> btw, data samples == openstack network_json samples
<harlowja_> i have tests in https://code.launchpad.net/~harlowja/cloud-init/cloud-init-net-sysconfig/+merge/297115 that take that file, and test to see it gets rendered as expected
<harlowja_> should also be able to easily do the same for eni rendering
<smoser> mgagne, the daily archive shoudl be up to date shortly
<smoser> ah. harlow right.
<rharper> smoser: did you share the cloudinit-data repo  ?
<rharper> that's got I think most of the example OS config data we've seen
<rharper> cloudinit-ds-test
<harlowja_> smoser lol, W503 line break before binary operator
<harlowja_> oops
<harlowja_> i have sinned
<harlowja_> forgive me father
<harlowja_> *for http://paste.ubuntu.com/17334762/
<smoser> harlowja_, its ok. you were lucky in that Odd_Bloke did worse :).
<harlowja_> lol
<smoser> harlowja_, but for your penance...
<smoser> want to see if you can get rid of that 'c' file ?
<harlowja_> i thought u said u did
<harlowja_> ?
<harlowja_> why u not fix it yet
<harlowja_> lol
<smoser> i fixed your pep error
<harlowja_> hmmm, good point
<harlowja_> ok i fix 'c'
<smoser> thefile that creates the 'c' is tests/unittests/test_cli.py
<harlowja_> which line fixes it?
<harlowja_> lol
<smoser> :)
<harlowja_> smoser what would u think about making bin/cloud-init -->> cloud-init/cmd/main.py and then using the python tooling to build this cmdline entrypoint
<smoser> i do not believe i am opposed to such a thing
<harlowja_> k
<harlowja_> the whole test_cli gets less weird if thats done imho
<smoser> yeah. i'm not sure why Odd_Bloke didn't suggest doing that.
<smoser> i think maybe jsut when he did he wanted to be less invasive
<harlowja_> ya, cause right now it does
<harlowja_> ' cli = imp.load_module(
<harlowja_>             'cli', open(BIN_CLOUDINIT), '', ('', 'r', imp.PY_SOURCE))'
<harlowja_> which is like woah
<harlowja_> lol
<harlowja_> because smoser  i did the following on that 'c' file
<harlowja_> $ file c
<harlowja_> c: python 2.7 byte-compiled
<harlowja_> so i'm pretty sure its the imp module dynamically compiling that thing
<harlowja_> and if then just bin/cloud-init is an actual importable thing, well this goes all away, ha
<harlowja_> cause i'm pretty sure that c file is getting created way down in the import machinery
<smemsh> hello,  how do i affect changes to actual root user account (password, chpasswd, ssh_authorized_keys), not the "distro default" ? there does not seem to be a way to specify the user except if creating new one (comment says "add each entry to ~/.ssh/authorized_keys for the configured user or the first user defined in the user definition directive")
<smemsh> same if i add password/chpasswd stanza it changes my "distro default" user, not root
<smemsh_> does user-data override cloud.cfg in the image filesystem or which takes precendence?
<smemsh_> mgagne: thanks for your gist, i don't think that sets the default user though, i think that would have to be done with redefined system_info or default_user
<mgagne> link I sent https://gist.github.com/mgagne/5a571499a337975b224e358b5286feb8
<smemsh_> mgagne: but, if that does change only the root user, that would solve my issue because i don't care about the "distro user"
<mgagne> smemsh_: this is the config we use in our images
<mgagne> I can't find the ubuntu user in our image
<smemsh_> mgagne: i didn't realize you could specify a user that already existed
<mgagne> but if it's built-in the base image, there is little you can do I guess
<mgagne> we build our own images from .iso so can't tell when config is applied against cloud images
<smemsh_> mgagne: it looks like i can change the distro default user with system_info or default_user, but i don't know if my user-data will override the embedded cloud.cfg in the image or not, or if i would have to make my own images
<smemsh_> mgagne: mgagne do you know if the filesystem-supplied cloud.cfg takes precedence over the same values provided from user-data? which "wins" if both places specify?
<mgagne> let me try against a cloud image, I think I have one somewhere
<mgagne> I don,t know =)
<smemsh_> mgagne: i will try, i thought might know
<mgagne> once stuff works, I often forget about the details =)
<mgagne> works for me
<harlowja_> smoser  https://code.launchpad.net/~harlowja/cloud-init/cloud-init-fix-up-cli/+merge/297409 have fun
<harlowja_> try that, no 'c' file should appear...
<smemsh_> half the stuff in my vendor's cloud.cfg are using underscores and half are dashes.  they must both work
<smemsh_> but then i see e.g. https://bugs.launchpad.net/cloud-init/+bug/1531582 which leads me to believe they have to be underscores
<harlowja_> pretty sure we have some crappy logic to handle both
<smemsh_> harlowja_: probably it would break a billion peoples' configs if you ever decided on one
<harlowja_> ya
<smemsh_> unfortunately it causes confusion and inconsistency, maybe flag day for version 3 or something ;-)
<harlowja_> :-P
<smemsh_> harlowja_: if i have same key supplied by both user-data and the cloud.cfg in the image which will have precedence?
<smemsh_> and lists will not merge right?
<harlowja_> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/helpers.py#L259 the ordering
 * harlowja_ i forget if lists merge by default, probably not
<harlowja_> so to my understanding smemsh_ user-data overrides all
<smemsh_> ahh ok good, that makes more sense then
<harlowja_> actually i take that back
<harlowja_> its input config files (from say the cli)
<harlowja_> then config files from the env (didn't know we could do that)
<harlowja_> then user data -> then vendor data then datasource specific stuff then base
<smemsh_> L260 comment
<harlowja_> ya
<smemsh_> you mean _read_cfg() ? ok
<smemsh_> so datasource configs are user-data right?
<harlowja_> nah, user data is @ http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/helpers.py#L238
<harlowja_> datasource configs i'm not sure who uses those
<harlowja_> i guess for datasources that don't have the concept of user-data?
<smemsh_> so user-data are "instance configs" in that comment?
<smemsh_> it says highest -> lowest are: input config files -> env config files -> override instance configs -> datasource configs -> base configuration
<harlowja_> ya, that looks about right
<harlowja_> and the default merge strategy doesn't overwrite keys from later entries over the newer ones
<harlowja_> so a key 'x' in instance config will stay 'x' even if base configuration has the same key
<smemsh_> so _get_instance_configs() is the one that picks up user-data ? i had thought that was a "datasource", hrm
<harlowja_> its more complicated than that
<harlowja_> the datasource does provide the user-data, but the consumption and merging and usage is a different stage of the app
<harlowja_> there's a consumption stage and then there are merging/using that config stages
<harlowja_> that ranking is post-consumption
<smemsh_> by doing ConfigMerger once consumed
<harlowja_> ya
<smemsh_> from all sources
<harlowja_> right
<smemsh_> well i'll never say it's not configurable ;-)
<smemsh_> as long as i know my user-data will override the cloud.cfg the vendor put there i'm good
<harlowja_> :-p
<smemsh_> thanks for explanation...
<harlowja_> np
<smemsh_> btw, there is a syntax for yaml to split long unbroken lines without any spaces, i noticed your docs have long lines for stuff like authorized_keys but they could split them (not sure if this is useful to change)
<harlowja_> smemsh_ ya, there is, feel free to submit some adjustments if u want :)
<smemsh_> i would if it were on github...
<smemsh_> i guess i could set up a launchpad thingy and figure out bzr...
 * harlowja_ diverts question to smoser 
<harlowja_> more people that yell at smoser about need-for-git the better :-P
<harlowja_> where yell == encourage
<harlowja_> lol
<smemsh_> it's just too easy on github and part of normal workflow.  launchpad is some silo somewhere with different tools, not that it's bad or anything of course
<smemsh_> i know canonical likes its umbrella projects on launchpad too if they fund it which is reasonable
<ajorg> smemsh_: well, it will let you override cloud.cfg, but only using the default merge strategy.
<smemsh_> great i'm all working now.  thanks for help all, have good night.
<harlowja_> smoser if u merge https://code.launchpad.net/~harlowja/cloud-init/cloud-init-fix-up-cli/+merge/297409 then merge https://code.launchpad.net/~harlowja/cloud-init/cloud-init-net-sysconfig also :-P
#cloud-init 2016-06-15
<Odd_Bloke> smoser: ;.; Forgive me.
<smoser> Odd_Bloke, thank you for fixing
<Odd_Bloke> smoser: Phil fixed it. :)
<smoser> Odd_Bloke, what do you think about https://code.launchpad.net/~harlowja/cloud-init/cloud-init-fix-up-cli/+merge/297409
<Odd_Bloke> smoser: I'm strongly +1 on the idea; I haven't reviewed the code.
<Odd_Bloke> smoser: I think the reason I didn't do anything more invasive before is that I was fixing a trusty bug, but I can't remember exactly.
<harlowja_> https://code.launchpad.net/~harlowja/cloud-init/cloud-init-fix-up-cli/+merge/297409 also fixes flake8 issues (so thats why some changes in that aren't just cli related)
<harlowja_> cause the code moved under cloudinit so now it got flake tested
<harlowja_> (if u were wondering)
<harlowja_> lol
<smoser> harlowja_, yeah, i like it
<smoser> does the packaging still work i woner.
<smoser> harlowja_, my pep8 cries now
<smoser> http://paste.ubuntu.com/17373169/
<smoser> http://paste.ubuntu.com/17374020/ seems needed for fixing that branch
<smoser> all output in output/
<harlowja_> hmmmm, will fix that, wonder why flake8 not find that
<harlowja_> smoser what pep8 is that
<harlowja_> *(what version)
<smoser> 1.7.0-2
<harlowja_> mine doesn't show anything, let's try that version, lol
<harlowja_> ok, got same issues, its because of patcher.pathc
<harlowja_> *patch
<smoser> well, i did this: http://paste.ubuntu.com/17374020/
<harlowja_> hmmm, or u can just do
<harlowja_> patcher.patch()  # noqa
<harlowja_> :-P
<harlowja_> ok, https://code.launchpad.net/~harlowja/cloud-init/cloud-init-fix-up-cli/+merge/297409 updated with those
<smoser> harlowja_, the patcher.patch....
<harlowja_> ya
<smoser> do you think its necessary that early ?
<smoser> why not just move it down
<smoser> assuming we dont stack trace in an 'import' that early, it should be ok, no ?
<harlowja_> it patches logging, which if we do it after we import cloud.log then i think it might not be working as we want
<smoser> oh. ok.
<smoser> well, short term, your solution with a comment is fine.
<smoser>  (# noqa)
<harlowja_> ya, long-term kill patching
<harlowja_> if we get git i can figure out the kill patching :-P
<smoser> harlowja_, https://code.launchpad.net/~harlowja/cloud-init/cloud-init-net-sysconfig/+merge/297115
<smoser> comented tehre.
<smoser> and yea, lets get to git
<smoser> and /me is out
<harlowja_> lol
#cloud-init 2016-06-16
<zero_shane> hi all - relatively new to cloud-init, but I haven't been able to find an answer to this yet ... does cloud-init support setting variables which can be used subsequently in templates - example FOO=bar, then within a template {{FOO}} for expansion ?   I'd want to apply this in cloud.cfg, allowing me to set more generic/templatized templates, that can be tweaked based on the base cloud.cfg information
<smoser> harlowja_, coul dyou merge with trunk?
<smoser> there is one conflict in debian.py
<smoser> harlowja_, and if you are around...
<smoser> harlowja_, i merged for yhou.
<smoser> all-in
<smoser> and uploaded to yakkety
<smoser> so you better not have messed it up!
<coolshiva123> hi team
<coolshiva123> need a help with cloud-init on centos
<coolshiva123> anyone there?
<Odd_Bloke> coolshiva123: Ask your question, and if anyone knows the answer then they might respond. :)
<coolshiva123> I have a centos7 template and i have some scripts in per-boot
<coolshiva123> the cloud-init fails to execute them
<coolshiva123> i have a clean cloud-init.log
<coolshiva123> and cloud-init-output.log
<coolshiva123> anyway to troubleshoot why cloudinit ignores script in per boot?
<smoser> coolshiva123, are they executable ?
<smoser> you should be able to run
<smoser>  sudo cloud-init single --name=scripts-per-boot --frequency=always
<smoser> and with '--debug' you'll get mroe output to screen.
<coolshiva123> hi
<coolshiva123> these scripts are definitley executable
<coolshiva123> [root@bceglc298 per-boot]# cloud-init single --name scripts-per-boot --frequency always --debug usage: cloud-init [-h] [--version] [--file FILES] [--debug] [--force]                   {init,modules,query,single} ... cloud-init: error: unrecognized arguments: --debug
<smoser> coolshiva123, --debug has to go to cloud-init not to 'single'
<harlowja_> smoser thx boss
<harlowja_> i was gonna merge today, but i guess u did
<smoser> yeah, i fixed that.
<smoser> pylint and six is obnoxious
<harlowja_> lol
<smoser> pylint --errors-only is really helpful
<smoser> but six really tricks it out
<smoser> on curtin we run --errors-only and it catches so many things
<harlowja_> cool
<smoser> but here it has a bunch of false positives
<smoser> maybe 12
<harlowja_> smoser  i was thinking, do u still want daemon support in cloud-init ? something that uses pyudev for example to get notified of hardware changes and activates modules?
<harlowja_> that was a cloud-init 2.0 thing, but doesn't seem so hard to do it ...
<smoser> so..
<smoser> plans for cloud-init in near term.
<smoser> a.) get 0.7.8 release that works for network configuration on centos on NoCloud and ConfigDrive
<smoser> b.) get to git
<smoser> c.) support getting network configuration from network'd datasources
<harlowja_> k
<smoser> ie, in he OpenstackMetadata
<smoser> so tha'td mean in the local stage, we'd bring up networking for the localnet and check 169.254.169.254
<smoser> and then say "oh look, my networking!"
<harlowja_> smoser u might also want to comment on  https://review.openstack.org/#/c/324054/
<harlowja_> i'll eventually start a openstack-dev ML about that if it doesn't move
<harlowja_> its sorta whacky to say 'the bug is there, and we aren't gonna change it, to bad...'
<smoser> its not really a bug
<smoser> just stupid
<harlowja_> :-p
<smoser> chaing the format of an opaque string is not breaking abi
<harlowja_> agreed
<smoser> it would seem like a change in EC2 from :
<smoser>  i-abcdefg
<smoser> to
<smoser>  i-abcdefgh
<harlowja_> omg
<harlowja_> how dare u
<harlowja_> lol
<smoser> (ie, they realized at some point that 6 chars was not enough)
<smoser> this is actually true
<harlowja_> nice
<harlowja_> lol
<smoser> and it actually *did* break things (stupid things)
<harlowja_> stupid things be stupid
<smoser> that assumed 8 chars (or whatever it was) was how long that thing should be
<smoser> all that MP is doing is replacing the internal (host) interface name
<smoser> with another interface name, right?
<smoser> as long as it keeps the value as consistent as the previous value (not changing on reboot, or stop/start) then it would surely seem fine to me
<harlowja_> ya, seems fair to me
<harlowja_> smoser comment on that review :-P
<smoser> on it
<smoser> harlowja_, how do i review ?
<harlowja_> press 'A'
<smoser> i'm logged in... but i dont see a text area or anhyting
<harlowja_> ya, the GUI changed
<harlowja_> lowercase 'a'
<smoser> man.
<smoser> that is *worse* than it used to be!
<smoser> which is pretty technically impressive
<harlowja_> def
<harlowja_> no disagreements from me, it got weirder
<sather> I'm trying to play around with cloud-init modules. I want to write a simple handler that takes the user-data passed in an EC2 instance and writes it to a yaml file.
<sather> what datatype is _cfg?
<sather> in the handle function `def handle(name, _cfg, _cloud, log, _args):`
<harlowja_> sather i believe its just a dict
<harlowja_> cloud is just a http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/cloud.py#L43 object
<harlowja_> and args are i forget, lol
<harlowja_> where'd the amazon guy go
<harlowja_> damn, we lost him already :(
<sather> harlowja_: thanks
<smemsh> smoser: i see you maintain cloud-init for ubuntu.  surprisingly, in ubuntu-14 cloud images, network config via cloud-init (0.7.5) does not work because "ifup: failed to open lockfile /run/network/ifstate.lo: No such file or directory" during first boot.  is this worth filing a bug or never get looked at due to age of release?
<smemsh> smoser: or maybe it's packaging error for cloud images and should not go to cloud-init package?
<sather> could someone help me understand this error:
<sather> __init__.py[WARNING]: Unhandled no
<sather> n-multipart (text/x-not-multipart)
<sather> realized I should be using write_files module
<smemsh> the snippet in cloud-init-container.conf that does mkdir -p /run/network needs to also be done for non-containers or ifup --all fails
<smemsh> it runs _bring_up_interface() before network-interface.conf / network.conf have run apparently, which fails on ubuntu
<smemsh> due to mk /run/network
<smemsh> no
<smemsh> i'll bet the tests do not try static ip setup, only dhcp and that probably works
<smemsh> yeah, i can see, init-local banner, then ifup --all fails, then cloud-init-nonet runs (which blocks waiting for static), and finally static networking comes up, which is the one that does the mkdir -p.  so it happens too late, it can never work
<smemsh> how do i inject runcmd in there to mkdir -p? network comes from metadata, it looks like it runs before anything from user-data
<sather> grr...boss says we can't use write_files directive
<sather> because we need to not touch the metadata file
<sather> :(
#cloud-init 2016-06-17
<harlowja_> smoser do u remember about any nova-specs for dynamic metadata or dynamic updates to the config-drive or similar (for example to be able to associate a new network device with a new ip, when one is attached)
#cloud-init 2017-06-12
<asdawqweq> Hello everyone, could any one give an advice how to reset cloud-init so it runs normally after next restart. I tried to delte ../instances unfortunatelly does not work, using openstack. Any tips?
<asdawqweq> a]\
<smoser> asdsad, rm -Rf /var/lib/cloud/
<dpb1> hi smoser
<smoser> hello dpb1
<blackboxsw> good morning. hopefully all made it swiftly back home
<smoser> this is wierd
<smoser> https://code.launchpad.net/~ajorgens/cloud-init/+git/cloud-init/+merge/325424
<smoser> causes
<smoser> https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/509/console
<smoser> there are issues in the MP, but the failures there show that util.append_file() is just broken
<smoser> i'm not sure how we've not seen those failures before
<blackboxsw> hmm so octal is more strict than %s  in that %s convert None to "None".  So we've been passing in an empty self.args in the past and just not validating the output in TestGenericDistro maybe
<smoser> blackboxsw, you're correct there.
<smoser> but if you ever call append_file("/tmp", "foo\n") it will just fail
<smoser> without that change.
<smoser> as that calls write_file with mode=None
<smoser> which calls os.chmod("/tmp/myfile", None)
<smoser> should have read append_file("/tmp/myfile, "foo\n")
<smoser> above
<ajorg> It's funny. I submitted https://code.launchpad.net/~ajorgens/cloud-init/+git/cloud-init/+merge/325424 as a no-brainer test of the current merge process (last time we were still using bzr).
<ajorg> I thought, "surely this can't go wrong."
<ajorg> "... it's a two character change"
<ajorg> :-)
<smoser> :)
<smoser> (and you did so 0o3 years ago... thought i'd be clever and write that as octal, but its hard to read)
<smoser> looks like some funny ascii art
<ajorg> smoser: so if I submit a 2 char fix and you counter with an 8 line fix and a test case, I still get my name on it?
<ajorg> ;-)
<ajorg> I need to setup an Ubuntu instance to run tox on. Hopefully I can keep carving off time for it.
<powersj> ajorg: out of curiosity what OS are you developing on?
<ajorg> powersj: Amazon Linux. I'm Amazon's cloud-init maintainer, when I can find time for it.
<ajorg> yes, getting unit tests running on Amazon Linux is also somewhere on my TODO list.
<powersj> ajorg: cool :) does tox bomb out on Amazon Linux today?
<powersj> we recently got them going on centos 6/7, so I was curious how we were doing on other distros
<ajorg> powersj: I last ran the unit tests before it was moved to tox (I think?) and a bunch of the tests were failing. I'll give it another try.
<powersj> ok thanks!
<ajorg> powersj: the unit tests do work on Amazon Linux. Neat-o!
<smoser> https://code.launchpad.net/%7Ecloud-init-dev/cloud-init/+git/cloud-init/+ref/master/+activereviews
<powersj> ajorg: woohoo! thanks for trying and that is good to know
<smoser> ajorg, yeah, i'd keep your name on it.
<smoser> :)
<ajorg> smoser: heh, okay, verifying fix now.
<ajorg> bleh, how did append_file ever work?
<ajorg> it passes mode=None... chmod() doesn't accept None
<smoser> ajorg, ah. i see.
<smoser> in util, 'chmod' is not os.chmod
<smoser> and it does:
<smoser> if path and real_mode:
<smoser>    ... os.chmod()
<smoser> so calling None just does nothing
<ajorg> oh, okay
<smoser> rharper, https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/325402
<smoser> that is good ?
<rharper> I think I landed that, no ?
<smoser> yes. fixed mp to mark that
<rharper> ack
<smoser> rharper, you want me to grab https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/325404  ?
<smoser> i can drop the 'pass' line in pulling it
<rharper> yes
<rharper> thx
<smoser> rharper, do you have a link to the selinux guard python bug ?
<rharper> I think that's in the MP or commit message
<rharper> https://bugzilla.redhat.com/show_bug.cgi?id=1406520
<ubot5> bugzilla.redhat.com bug 1406520 in libselinux "calling libselinux python restorecon fails on /var/lib/nfs/rpc_pipefs" [High,Verified]
<smoser> yep
<smoser> tghanks
<ajorg> smoser: not sure if I twiddled the state of https://code.launchpad.net/~ajorgens/cloud-init/+git/cloud-init/+merge/325424 correctly
<smoser> ajorg, please write a nicer commit message. i'll make the c-i bot re-run on it and then i think it looks pretty good
<ajorg> smoser: mind showing me an example of a nicer commit message to emulate?
<ajorg> (or should I pick just any from the past?)
<smoser> just 'git log'
<smoser> basically
<smoser> Summary
<smoser> <blank line>
<ajorg> and that's for the merge commit? Or you want nicer messages on all the commits (or squashed)
<smoser> More info
<smoser> put it on the 'Commit Message' in the merge proposal
<smoser> and then i will squash with that commit message
<smoser> make sense ?
<ajorg> k, I can do that
<smoser> blackboxsw, i'd appreciate a review from you on https://code.launchpad.net/~ajorgens/cloud-init/+git/cloud-init/+merge/325424
<smoser> powersj, https://jenkins.ubuntu.com/server/view/Cloud-init/job/cloud-init-centos-6/32/consoleFull
<powersj> smoser: sup
<smoser> ^ that job, can you give me an explanation of what it does ? i think that is supposed to be unit test on trunk ? right ? and eventually i think move to use run-centos ?
<powersj> smoser: yes, exactly. Unit test + build
<powersj> the run-centos is broken atm because it is missing a dependency. I was going to wait on blackboxsw merge for dependencies to fix that up, to remove the hard-coded ones
<smoser> ok.
<ajorg> smoser: okay, I think I made the commit message better. Please be critical if I failed.
<smoser> ajorg, thanks.
<smoser> i responded there. will wait for blackboxsw but other than that, great.
<ajorg> cool, I'll try to prep a couple more of our smaller fixes today
<blackboxsw> smoser: reviewing ajorgen's branch now
<blackboxsw> thx BTW ajorg
<ajorg> I'm super happy to be pushing our fixes upstream again finally.
<ajorg> We have larger ones too, but I want to learn how you like to do things on smaller ones.
<blackboxsw> the more the merrier. I'm learning too.
<dpb1> ajorg: what are some of the larger ones?
<ajorg> dpb1: lemme look so I can give you an example...
 * dpb1 nods
<ajorg> Ah, best example: I have a large patch for the migrator module that allowed us to seamlessly upgrade from 0.5 to 0.7.x. I had to restructure some things.
<ajorg> We also have some patches that add compatibility for how our fork of cloud-init worked before RPM / Yum support was added upstream
<ajorg> Some of those may be less useful generally.
<dpb1> ajorg: ah, so re-basing those and see what was unique
<ajorg> We have a new feature too.
<ajorg> Right. Upstream took a different path, but we didn't want our customers to break, so we had to add some compat.
<ajorg> We have a new feature (write-metadata) that we use to write out /etc/yum/vars/* to configure repositories.
<ajorg> And we have a feature that lets the user choose what level of security updates to install by default.
<ajorg> Other than that it's mostly small bugfixes.
<ajorg> Some more important than others.
<rharper> larsks: do you have a pointer to your downstream systemd unit files for cloud-init?  I'd like to take a look at those instead of fumbling to come up with what you already have working
<dpb1> ajorg: that level of security updates to install sounds interesting
<smoser> rharper, http://pkgs.fedoraproject.org/cgit/rpms/cloud-init.git/
<rharper> smoser: thx
<rharper> hrm, we don't have something equivalent  to 'networking-wait-online.target'  ;  the rhel network service starts bringing up interfaces, but the network-online.target only requires that network.target has been reached, so, cloud-init.net runs before networking is completely up
<dpb1> rharper: sounds bad
<rharper> well, it's a known problem; it's a matter of how to wait effectively
 * rharper looks into ifup on rhel 
<blackboxsw> smoser: ajorg just finish review on https://code.launchpad.net/~ajorgens/cloud-init/+git/cloud-init/+merge/325424   see what you think
<blackboxsw> I wanted to use with_logs more if we could instead of mocking logging
<blackboxsw> http://paste.ubuntu.com/24843185 specifically for that branch
 * ajorg looks
<smoser> blackboxsw, did you test your s?
<smoser> decode_perms has a somewhat silly signature (which we could change) where you pass in a 'log'
<blackboxsw> smoser: ajorg I ran tox on my patch ________________________________________________________________________________ summary ________________________________________________________________________________
<blackboxsw>   py27: commands succeeded
<blackboxsw>   py3: commands succeeded
<blackboxsw>   flake8: commands succeeded
<blackboxsw>   xenial: commands succeeded
<blackboxsw>   pylint: commands succeeded
<blackboxsw>   congratulations :)
<blackboxsw> ooops
<ajorg> blackboxsw: yeah, that looks good to me, pushing that soon
<blackboxsw> smoser: +1 on that thought. I felt the same
<ajorg> (to my branch)
<smoser>  oh. i see. you did the self.logger
<blackboxsw> just didn't want to hold up ajorg's branch for that sentiment (about logging)
<smoser> er... i dont see.
<smoser> oh.
<smoser> i do
<smoser> get with the program, smoser
<blackboxsw> yeah self.logger comes now out of tests.unittests.helper
<blackboxsw> heh
<smoser> i say we change the signature
<smoser> but i do think you're being overly pedantic/brittle on
<smoser> self.assertEqual("WARNING: Undecodable permissions '999', returning default None\n", ...)
<smoser> (linke 113 of your paste
<smoser> i would (and did) assertIn
<blackboxsw> smoser: works for me I was only trying to keep with the fact that len(warnings) was being used in your tests (to assert that we didn't have more than 1 warning in the log). But yes that was overly pedantic. assertIn would work instead of assertEqual when checking logs. we don't really have to count # of warnings etc in unit tests.
<ajorg> So http://paste.ubuntu.com/24843292/
<smoser> blackboxsw, true.
<smoser> i guess i was *as* pedantic
<smoser> ajorg, yeah, that looks fine to me.
<blackboxsw> +1 from me ajorg
<blackboxsw> heh, I just like passing the scapegoat smoser
 * ajorg reruns tox because Jenkins is a cruel master
<ajorg> k, pushed that latest, anything I need to do on the merge proposal to push it along?
<powersj> ajorg: as long as it is in "Needs review" and not "work in progress" you should be good to go. I usually do like to add a comment to the merge so folks see I addresses the changes.
<ajorg> powersj: thanks
<ajorg> so I'm confused by the distros / os families thing. I've got 'amazon' added to the 'redhat' os family, and there's a module (cc_resolv_conf.py) that has distros = ['fedora', 'rhel', 'sles'] ...
<ajorg> I thought that changing that to distros = ['redhat', 'sles'] would enable it for 'amazon', but it doesn't.
<rharper> ajorg: the distro tag is mostly for documentation, IIRC, if you want a module to run, then you need to update /etc/cloud/cloud.cfg ; in there the modules list for each stage (local, init, config, final) is what controls which modules run
<ajorg> rharper: I get stages.py[INFO]: Skipping modules ['resolv-conf'] because they are not verified on distro 'amazon'.  To run anyway, add them to 'unverified_modules' in config.
<rharper> interesting, if amazon is like rhel, then you shouldn't needed it per the documentation which suggests that sysconfig to manage resolv.conf
<rharper> but it sounds like you can add a 'unverified_modules: ['resolv_conf'] to the user-data to make it work
 * rharper looks up unverified-modules in the code
<ajorg> rharper: we don't use that specific module by default. we had a customer specifically ask that we enable it for them
<rharper> ack
<ajorg> There's another module (cc_spacewalk) that has 'redhat' in the distros list. My hunch is that it doesn't work on 'rhel'
<rharper> yeah, looks like the stages code will check
<ajorg> rharper: I'm less worried about how to let a customer us an unverified module than I am about understanding how that distros list in a module is used to decide if it's verified.
<ajorg> If I'm wrong that 'redhat' is supposed to enable both 'rhel' and 'fedora' then I'll be on my merry way.
<rharper> I'd say it get's set in merge of contributions
<ajorg> I'm trying to figure out if there's a bug or a misunderstanding
<rharper> ajorg: it's likely a bug here, the distro names and variants can mismatch, a distro variant of 'redhat' should match 'rhel'
<rharper> ajorg: it's likely a bug here, the distro names and variants can mismatch, a distro variant of 'redhat' should match 'rhel'
<rharper> sorry for the dupe
<ajorg> I'll try to find out if it's something that broke recently.
<rharper> ok
<ajorg> ah, will have to revisit it tomorrow though, got a meeting
<rharper> python -c 'import platform; print(platform.linux_distribution()[0].lower())'  that's what gets the distro value, and then in cloudinit/util.py there's a mapping of variants
<ajorg> be back tomorrow, thanks for help today!
#cloud-init 2017-06-13
<smoser> blackboxsw, wonder if you ahve thoughts on https://code.launchpad.net/~redriver/cloud-init/+git/cloud-init/+merge/325207
<smoser> the 'open' is the only thing i wonder about (and the annoying method of mocking open).
<smoser> (and 'open' was my suggestion)
<smoser> the one thought i have would be just to move that hunk (line 23 -> 27) to a function 'cdrom_is_readable()' or something.
<smoser> then makes easier mocking of it
<blackboxsw> checking smoser
<rharper> running tox, on a branch recent from master, saw this;   I recall we chased this one down last week but I don't remember the fix ,  blackboxsw  smoser  ?: Problem importing module imports.py: cannot import name 'input'
<rharper> during pylint run
<rharper> huh, non fatal, but noisy
<blackboxsw> hmm I recall seeing that at some point. but I don't recall the remedy (I know I had missed deps in the past and ran  tox -r to rebuild)
<smoser> rharper, tox -e pylint ? i didn't see that here.
<rharper> smoser: pylint runtests: commands[0] | /home/rharper/work/git/cloud-init/.tox/pylint/bin/python -m pylint cloudinit
<rharper> I just ran tox at the top level
<rharper> I think it's the coverage
<rharper> report
<smoser> hm.. rharper i dont see that.
<smoser> http://paste.ubuntu.com/24850671/
<smoser> am i doing something wrong ?
<rharper> I'll blame it on my local stuff
<rharper> it's non fatal, so I'm fine; I likely have cruft or old stuff
<smoser> well, just rm -Rf .tox
<smoser> give that a try.
<rharper> k
<smoser> blackboxsw, good news.
<smoser> work to do.
<blackboxsw> do tell
<smoser> (sru x, y, z all accepted)
<smoser> "good news"
<blackboxsw> good ahasenack warned that he saw lots of bug mail, figured it'd be coming across soon
<blackboxsw> ok SRU template writeups time
<blackboxsw> trying to wrap up a review and will move over to that.
<ahasenack> blackboxsw: how many bugs are you fixing in one upload?
<blackboxsw> ahasenack: this one's big 20 bugs
<blackboxsw> last two were 12 and 15 I *think8
<blackboxsw> last two were 12 and 15 I *think*
<blackboxsw> ok onto SRU testing.
<smoser> blackboxsw, i am going afk. i will help tomorrow with sru verification. just finished my
<smoser> bug https://bugs.launchpad.net/ubuntu/+source/cloud-initramfs-tools/+bug/1666573
<ubot5> Ubuntu bug 1666573 in cloud-initramfs-tools (Ubuntu) "transient systemd ordering cycle in boot with overlayroot ver read-only open-iscsi root" [Medium,In progress]
<blackboxsw> +1 smoser
<powersj> blackboxsw: do you have an open build service account?
<blackboxsw> powersj: will check. not sure if I went through that setup yet
<blackboxsw> powersj: sure enough blackboxsw
 * rharper does subnet static route math harder, and now has it passing under redhat 
<rharper> \o/
<powersj> woohoo
<rharper> let's just say I learn something today
#cloud-init 2017-06-14
<powersj> wow all the tests failed yesterday O.o
<powersj> blackboxsw smoser rharper dpb1 Our open build service project: https://build.opensuse.org/package/show/Cloud:Tools:Next/cloud-init
<smoser> powersj, i am now 'smoser' user there.
<powersj> smoser: added
<rharper> powersj: ok, will join
<dpb1> powersj: cool
<ajorg> smoser: should all things going through subp be disconnected from tty? (disconnecting from tty is easier than I thought - the main trouble was that I was scared to defy the warnings in the subprocess documentation, no need for a cat process)
<ajorg> it's incompatible with capture=True, so I thought about having it be the behavior for capture=False.
<rharper> powersj: ok, I'm raharper there
<powersj> rharper: added
<smoser> ajorg, i'd think generally that we would not want a tty.
<rharper> tx
<smoser> but that seems a scary change.
<ajorg> smoser: https://paste.ubuntu.com/24857815/
<ajorg> agreed, it's a bit scary. easy to get something wrong
<ajorg> i can drop it for now while working on other patches
<smoser> do you have a typo there ?
<smoser> p.stdout ?
<ajorg> haha
<ajorg> yes
<ajorg> dum de dum
 * ajorg runs unit tests like a good boy before re-posting
 * ajorg ... and watches tests.unittests.test_util.TestSubp hang because subprocess is hard
<ajorg> back to the drawing board
<smoser> ajorg, yeah, we really dont want to be "in the middle" if we can avoid it.
<ajorg> smoser: there's no way to disconnect from a tty without being in the middle
<ajorg> because if you're not, isatty will find your tty
<powersj> blackboxsw: looks like the failures of integration tests occurred after your commit was accepted.
<powersj> the control file that gets generated looks a little different, before: http://paste.ubuntu.com/24857896/ after: http://paste.ubuntu.com/24857898/
<powersj> Looking now for actually differences
<ajorg> oh, 'cause I'm neglecting to write the input...
<powersj> blackboxsw: to reproduce, checking out master and doing ./package/bddeb -S should be all you need
<blackboxsw> powersj: yeah I'm on a yakkety lxc right now, found 3 missing deps that need to be pulled in.
<blackboxsw> thanks
<blackboxsw> .. and sry
<powersj> no worries, I am hoping our new CI pipeline will fix all of this :)
<powersj> tox + build + short integration + centos will be nice :D
 * powersj goes and review's smoser's branch
<smoser> powersj, thanks. i just updated comment.
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/325679
<smoser> blackboxsw, ^ you too would be nice.
<smoser> that 'fastestmirror' thing is pretty useless
<smoser> useless/obnoxious/makes a caching-proxy useless
<blackboxsw> reviewing now
<ajorg> smoser: okay, this works: https://paste.ubuntu.com/24857958/
<smoser> ajorg, i guess i'm ok with it if we add an argument 'notty=False'
<smoser> and then the yum install can set that to True
<smoser> and toherwise takes old apth
<ajorg> Sure, we can make this optional. I'm okay with that.
 * powersj goes and looks at proposed tests
 * ajorg waves at ilianaw
 * ilianaw waves back
<smoser> well, here is a week attempt at disabling mirrors
<smoser> http://paste.ubuntu.com/24858093/
<blackboxsw> smoser: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/325679 minor review comment added.
<smoser> blackboxsw, thanks.
<smoser> blackboxsw, you want me to drop the 'pairs' ?
<blackboxsw> smoser: not critical to drop as it shows what you intend if things get complex in the future (cmd != package_name) so it's ok to leave as is as guidance for me if we have to go that route . It's just unneeded logic at the moment so it always gives me a moment's pause as we don't really need that little complexity currently.
<blackboxsw> if it were a lot of complexity/logic that was added for something that we don't currently need I'd say drop it as we are over-engineering the solution, but this is just a couple lines in a script so it's good
<blackboxsw> mostly that comment was just so we could have a discussion.... aaaaaaand â
<blackboxsw> :)
<smoser> k
<smoser> i updated mp with your file suggestion
<blackboxsw> thx
<smoser> but i think yours had wierd logic
<smoser> so read mine
<smoser>  http://paste.ubuntu.com/24858229/
<smoser> thats what you had
<smoser> you were looping over each file in the list and building the bad_files sllist
<smoser> so... lots more work than necessary
<smoser> i did this
<smoser>  http://paste.ubuntu.com/24858234/
<blackboxsw> umm whoa , bad diff. right
<blackboxsw> shouldn't have nested that inside the loop , just declare bad_files outside.
<blackboxsw> checking followup
<blackboxsw> yes yes, thanks
<blackboxsw> +1
<blackboxsw> that was careless
<smoser> k
<smoser> click approve and i'll pull that.
<smoser> powersj, youcan then change the jenkins jobs to do the ./tools/run-centos
<powersj> Awesome! Once we get bddeb going again I can go back to the pipeline and get it going
<smoser> powersj, do you knwo why it is failling ?
<smoser> i just started looking, i'm confused.
<powersj> I posted this a little bit ago: the control file that gets generated looks a little different, before: http://paste.ubuntu.com/24857896/ after: http://paste.ubuntu.com/24857898/
<powersj> but that doesn't make any sense
<powersj> the error was "E: Please add apropriate interpreter package to Build-Depends, see pybuild(1) for details at /usr/share/perl5/Debian/Debhelper/Buildsystem/pybuild.pm line 192."
<smoser> no it does not
<blackboxsw> hmm so diff between deps is the new one has has procps and python-contextlib2 list in Build-depends  now.
<blackboxsw> I pulled procps down from Depends  into Build-Depends. lemme see where contextlib2 comes from
<blackboxsw> from test-requirements.txt so it now gets pulled into the control.in template
<smoser> good: http://paste.ubuntu.com/24858565/
<smoser> bad: http://paste.ubuntu.com/24858566/
<smoser> it is a bug somewhere.
<smoser> but it doesnt like the python3 at the end.
<blackboxsw> so we need to make sure python dependency is ordered before test-requirements as it was in the original control file maybe?
<blackboxsw>                ${python},
<blackboxsw>                ${test_requires},
<blackboxsw>                ${requires}
<smoser> well, moving it in that list fixes it
<smoser> "fixes"
<blackboxsw> wow strange workaround
<smoser> i was experimenting with jinja
<smoser> blackboxsw, http://paste.ubuntu.com/24858833/
<smoser> that is working for me now.. hacky still, but got trhough it
<blackboxsw> smoser: looks fine, so you wanted to make it readable again (with proper indents, one line per dep)
<blackboxsw> I've just pushed ci-deps-fixes branch, wanted to add a --ci-deps optional param to read-dependencies to pull in everything ci wants (make, sudi, tar, devscripts etc) It'll also default the python version to the python binary that's calling us  via sys.version_info.
<blackboxsw> this way ci can just :   python3 tools/read-dependencies --distro ubuntu --install --ci-deps
<blackboxsw> I'm testing it now to make sure it solves the integration test issues on fresh containers
<blackboxsw> what I had also done in that branch was pull "python" package version up to the front of Build-Requires
<smoser> blackboxsw, --ci-deps ?
<rharper> well, bleh.  bonding on rhel7 is frustrating ...   the bonding module loads and creates a bond0 device but ifup on bond0 doesn't configure it;  ifup on the slaves is skipped, so I can't see how the bond is ever going to come online;  further, the docs show the use of networkmanager
<blackboxsw> maybe just ci or continous-integration? Was trying to take a step toward your "goal" suggestion a bit.
<blackboxsw> given that we want to setup all CI  dependencies on a fresh container in order to run any build/test scripts, the --ci-deps parameter would install system packages for all requirements.txt and test-requirements.txt plus the additional base needs make, tar, sudo, devscripts
<blackboxsw> so basically on a fresh container we can now perform the following: http://pastebin.ubuntu.com/24859055/
<smoser> blackboxsw, i just didn't know the 'ci'
<smoser> it conflicts . someone chose a bad name for 'continuous-integration'
<smoser> when clearly 'cloud-init' had owned the namespace of 'ci'
<smoser> :)
<nacc> ci-ci
<blackboxsw> heh. I just reread your goals suggestion.... "test" "build" as options
<smoser> https://gist.github.com/smoser/19e65095b342e98fd4d6466e4d4fa1ac
<smoser> there i had 'test-distro', 'test-tox', 'build', 'run'
<smoser> blackboxsw, so in my hack patch above... i couldnt figure out how to pretty indent. i'm lacking jinja foo
<blackboxsw> yeah /me changes it to --test-distro
<blackboxsw> those names make sense
<smoser> blackboxsw, do you have something like... really close ?
<smoser> cause i think what i have in that hack is at least enough to get us building again
<blackboxsw> smoser: the branch it pushed it's tested and runs
<blackboxsw> was writing up a review request now
<smoser> ah. i didnt see you push a branch
<smoser> and didnt see at https://code.launchpad.net/%7Ecloud-init-dev/cloud-init/+git/cloud-init/+ref/master/+activereviews
<smoser> so i assumethat is coming
<smoser> i'd like to grab all the 'Approved' at that link, but want to have c-i functional before i do
<blackboxsw> was writing test instructions. will just post as is and work out testing instructions shortly after
<smoser> sure
<blackboxsw> pushed for review.
<blackboxsw> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/325693
<blackboxsw> adding test instructions for a fresh system.
<smoser> blackboxsw, tools/run-centos can use --test-distro
<smoser> right ?
<smoser> although test-distro isn't necessarily build
<smoser> but i think that is what we have right now
<blackboxsw> smoser: yes run-centos can use it.
<blackboxsw> changing that now
<blackboxsw> in the branch
<blackboxsw> /paste.ubuntu.com/24859315/
<blackboxsw> http:///paste.ubuntu.com/24859315/
<blackboxsw> almost
<blackboxsw> http://paste.ubuntu.com/24859319/
<smoser> blackboxsw, 2 minor reponses.
<blackboxsw> looking
<smoser> i have to run.
<smoser> i'm ok if you or rharper want to squash that and push it
<smoser> to get us passing tests again
<smoser> but i have to run
<smoser> i'll check back in in a few hours and get it then if you dont.
<smoser> thanks!
<blackboxsw> smoser: pushed
<blackboxsw> thanks
<blackboxsw> testing again now
<blackboxsw> powersj: did you install sbuild on the jenkins ci environment?
<blackboxsw> I see we are running it, but it's not available in default yakkety lxcs
<powersj> Yes I installed it but we're not running yakkety
<blackboxsw> found out what I missed, the initial apt update
<blackboxsw> ok merged to master
<blackboxsw> https://jenkins.ubuntu.com/server/view/Cloud-init/job/cloud-init-integration-y/227/ is running the test
<blackboxsw> powersj: I'm also thinking of something along these lines https://jenkins.ubuntu.com/server/view/Cloud-init/job/cloud-init-integration-y/227/ for server-team-ci
<blackboxsw> to ensure all deps are installed prior to build runs
<blackboxsw> meh I only updated my local maste 227 is bound to fail
<blackboxsw> well check back in later. dinner calls
<smoser> blackboxsw, good ?
<smoser> i see https://jenkins.ubuntu.com/server/view/Cloud-init/job/cloud-init-integration-y/227/
 * smoser will check back in in hour or so
<powersj> given it built and is running tests we should be good to go
#cloud-init 2017-06-15
<blackboxsw> yeah, unfortunately, I didn't actually merge it ;)
<blackboxsw> Permission denied (publickey).
<blackboxsw> fatal: Could not read from remote repository.
<powersj> O.o you are in the group
<powersj> https://launchpad.net/~cloud-init-dev/+members#active
<blackboxsw> yeah, git push origin master was a no go. hmm.
<blackboxsw> I'll try a fresh clone merge push
<blackboxsw> hrm fresh clone shows it merged.
<blackboxsw> so when we git push origin master does launchpad automatically mark the branch merged?
<blackboxsw> or is that manual
<blackboxsw> I was used to LP marking my branches as merged automagically for bzr branches. not sure if that's the same for gitlp
<powersj> blackboxsw: I think you have to mark it
<powersj> I don't recall doing it automatically, only in bugs
<blackboxsw> ok thx powersj yeah that was my confusion (and LP git was also showing stale data @ https://code.launchpad.net/~cloud-init-dev/cloud-init/+git/cloud-init/+ref/master)
<blackboxsw> so I *think* rev b23d9d7 is in build https://jenkins.ubuntu.com/server/view/Cloud-init/job/cloud-init-integration-y/227/ too, so ok, we should be testing what I landed
<smoser> blackboxsw, powersj it will mark merge in one of 2 ways (which are really kind of the same)
<smoser> a.) you 'git merge' && 'git push'
<smoser> b.) you squash, rebase, push --force [to your repo], then merge
<smoser> but only you can do 'b', and i prefer the squash to the merge. so really only people with commit access can end up getting that to happen unless they rebase exactly on master with no in the middle commits.
<powersj> yeah squash should be done imo
<smoser> so... often times i go and mark 'merged' and then hit the edit button for 'merged at revision'
<smoser> and put in the actual commit id
<smoser> powersj, so do we think we're happy in trunk now?
<powersj> For trunk build yes
<powersj> I do see a centos build failure: https://jenkins.ubuntu.com/server/job/cloud-init-copr-build/8/console
<powersj> and integration tests failed but due to test failures, those were during the firewall reboot, so I'll let them go for now.
<smoser> powersj, you're missing python-jinja there
<smoser> is that in a fresh container ?
<smoser> that error is horrific. it should fail with a sane error rather than attempting to render a template with 'basic' when the template declares it is jinja and it knows it has no jinja
<blackboxsw> hmm build failure on 227 above http://pastebin.ubuntu.com/24861805/
<blackboxsw> have we seen this before? " Could not resolve \'us.archive.ubuntu.com\"
<powersj> what time was that failure at? It may have been during firewall maintenance
<powersj> I should just disable tests when we have those planned...
<niluje> anyone knows how to submit a merge request with launchpad?
<niluje> the "propose for merging" button doens't seem to exist on https://git.launchpad.net/~jcastets/cloud-init/
<niluje> oh ok. Only available from code.launchpad.net, not from git.
<niluje> https://code.launchpad.net/~jcastets/cloud-init/+git/cloud-init/+merge/325740
<smoser> niluje, thanks!
<smoser> niluje, fyi http://cloudinit.readthedocs.io/en/latest/topics/hacking.html kind of explains it
<blackboxsw> ok things looking a bit better on jenkins. though integration-x is failing w/ 2017-06-15 11:44:37,781 - tests.cloud_tests - ERROR - stage: collect for platform: lxd encountered error: Failed to create ZFS snapshot: cannot create snapshot 'default/images/ee1831d4efcc74310def225ec449c1b6a49cc371a744cbbb79ea5bf21b05c3b6@readonly': dataset already exists
<blackboxsw> https://jenkins.ubuntu.com/server/view/Cloud-init/job/cloud-init-integration-x/220/console
<smoser> blackboxsw, well, at least that doesn'st smell like cloud-init code regression
<blackboxsw> amen to that
<smoser> https://jenkins.ubuntu.com/server/view/Cloud-init/job/cloud-init-copr-build/8/console
<blackboxsw> jinja template changes +  brpm changes
<blackboxsw> smoser: I can take that if you haven't already
<smoser> blackboxsw, you can. i think it needs jenkins change
<smoser> right? it is just that wherever that is run it needs jinja
<smoser> ideally it runs in a container. or it could use:
<smoser>  ./tools/run-centos 6 --srpm --artifact
<smoser> http://paste.ubuntu.com/24864792/
<smoser> it could be useful to chagne run-centos to take a '--artifact-dir' to put artifacts (the srpm) in that directory rather than '.'
<blackboxsw> that's a good call. I'll go that route
<smoser> i dont think we need the '--artifact-dir' here as the build will always take place in a clean environment
<smoser> https://git.launchpad.net/server-team-ci/tree/jenkins/cloud-init/centos.yaml is what you need to change i think
<blackboxsw> yeah we can just switch CI to use run-centos I have a couple other jenkins changes from yesterday (to get deps installed for the integration-# builds too. so I'll fold them together
<smoser> k
<niluje> smoser: yep, I read the document but didn't notice it was only available on "code.laucnhpad.net" and not on "git.launchpad.net"
<smoser> fair. i reviewed your proposal.
<niluje> smoser: "we really, *REALLY* want a positive non-network identification" -> we already discussed about this on the previous PR (which was made longtimeeee ago), and unfortunately there no way to implement the on_scaleway method without hitting a network ressource
<smoser> :)
<niluje> you suggested to call dmidecode, it wouldn't work as we don't provide such info
<smoser> yeah. feel free to link to that if you'd like.
<niluje> smoser: just a thing about vendor-data. I added them in self.metadata['vendor-data']. Is it what is expected by cloud-init?
<smoser> niluje, the datasource wants a
<smoser>  self.vendordata_raw
<niluje> is the interface documented somewhere?
<smoser> well cloudinit/sources/__init__.py
<niluje> oh
<niluje> 'user-data' and 'vendor-data' are not expected to be in metadata
<niluje> I thought so :p
<smoser> well, cloud-init doesn't get to chose where you put information on the other end.
<niluje> ok
<smoser> i'm sorry if i sound like pestering...
<smoser> are there plans (at least in the vms) to put some dmi information in ?
<smoser> or some other way of identifying "you are on scaleway"
<smoser> niluje, i'm really sorry if you've told me all this before. if I had a nickle for for all the things i'd forgotten...
<blackboxsw> powersj: just put a branch for review https://code.launchpad.net/~chad.smith/server-team-ci/+git/server-team-ci/+merge/325745.
<powersj> blackboxsw: thanks
<niluje> smoser: we have two options
<niluje> smoser: 1/ we can update our initrd to create the file /var/run/scaleway and check this file exists in cloud-init
<niluje> 2/ we can add "scaleway" in the kernel cmdline and check it in cloud-init
<niluje> this way the network check can go away
<niluje> any preference?
<smoser> since you already ahve a custom initrd, that seems fine.
<smoser> in your vm images do you have a custom initrd ?
<smoser> ie, the 'vc1s' does that have a custom initrd ?
<smoser> either is fine. i think maybe kernel cmdline seems easier to maintain long term.
<smoser> i'd like it if
<smoser> a.) your vm instance types (vc1s, vc1m, vc1l) provided some dmi data . in theory that shoudl work even on arm64 vms (i dont know if you offer those), but reality differs
<niluje> (yes we do)
<niluje> for dmi data, it's really not an option as you have some baremetal servers and we don't want to maintain several solutions to identify the hardware
<smoser> b.) your bare metal instance types, it seems like kernel cmdline is acceptable. at some point in the future when you started  being able to use stock images there, with a grub that loaded a kernel from inside the image... we'd have to figure someway out. maybe you'd have to modify the image to keep the kernel param on, or just touch a file or static config.
<niluje> and we'd like to eventually get rid of the initrd (which is currently also used for vm)
<niluje> if that's ok to you let's go for the commandline
<smoser> in vm instances dmidata shoudl be possible
<smoser> because at some point you want to get off maintaining your own images for vms (i'd think)
<smoser> and having easy check of dmi platform data that says 'scaleway' is a win there.
<smoser> even if its harder to get that same win elsewhere.
<niluje> smoser: what about adding three checks ? dmidecode + cmdline + /var/run/scaleway?
<niluje> this way we can just update the cmdline for now and/or update the initrd, and we have the flexibility to add dmidecode on our vm later when we'll (hopefully) allow to run custom images/kernels
<smoser> that is fine with me.
<niluje> awesome :)
<niluje> last thing, what's the LP #id you want me to put in the commit message?
<niluje> the MR id?
<smoser> no. bug number
<smoser> i you want to file a bug, you can.
<smoser> that'd be nice.
<smoser> but if you dont', then thats fine too
<niluje> ok
<niluje> thanks
<niluje> I'll get back to you with improvements and unittests (and not in weeks, this time)
<smoser> powersj, i suspect it'd be possible to get a freebsd vm into our jenkins right ?
<smoser> in a low tech sort of way... powersj gets a vm image of freebsd or does an install, adds it to a libvirt and adds jenkins to it ?
<smoser> then we could run freebsd c-i on that system.
<powersj> smoser: yeah
<powersj> assuming the jenkins client (java) runs on freebsd
<rharper> https://docs.openstack.org/image-guide/freebsd-image.html
<smoser> "You can specify up to 1 GB additional RAM to make the installation process run faster."
<powersj> nice!
<powersj> lol
<smoser> any more than that, and its going to fail :)
<smoser> blackboxsw, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/325760
<smoser> i'd appreciate read of that if you had some minutes
<blackboxsw> grabbing right now
<blackboxsw> powersj: https://jenkins.ubuntu.com/server/job/z_integration_y/1/console looks pretty good for the 2nd part of validation on https://code.launchpad.net/~chad.smith/server-team-ci/+git/server-team-ci/+merge/325745
<blackboxsw> smoser: ci didn't like first pass. https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/325760
<blackboxsw> but reviewing code now
<smoser> its running the second now.
<powersj> blackboxsw: your merge looks good to me
<smoser> i dropped the pylint ignore on platform.dist()
<smoser> which is unfortunatlye deprecated (but so was linux_distribution())
<smoser> :-(
<smoser> https://bugs.python.org/issue1322
<rharper> aw bummer
<NostawRm> long shot, but I'm trying to find anyway that I can specify some network configurations because I'm not happy with Openstack's IP selection. I just saw that using user data is not an option, so I'm seeing what other options I have
<NostawRm> I  know I could just make a script to use but that's less than ideal :/ having it done proper by cloud-init would be better imo
<smoser> NostawRm, i'm not sure that you can do that .
<smoser> if openstack told you what your ip is, then its quite likely not going to route traffic if you take a different one
<NostawRm> I thought not. Surely if I could, I would have found it right now :/
<NostawRm> well, its a single interface/port with multiple IPs
<NostawRm> and openstack just picks the first one in the list, but if a new one is added that is then the first one in the list, then say on a rebuild, the 'main' IP would change
<NostawRm> don't know if that explains it well heh
<blackboxsw> smoser: minor comment about suse on https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/325760
<powersj> smoser: re: CI pipeline I have something here: https://jenkins.ubuntu.com/server/job/cloud-init-pipeline-ci/
<powersj> I would like to keep the run times below 10 mins, so what I am going to do is work on creating parallel stages going and write a short integration test that hits as many areas as possible rather than running all the lxd-based tests.
<powersj> Then plan would then be to replace the daily build/unit test with this and replace the per-merge CI tests with this as well, so every merge has coverage here.
<smoser> http://doodle.com/poll/tq43i9876sw2vqca
<smoser> larsks, ^ rharper powersj dpb1
<blackboxsw> https://jenkins.ubuntu.com/server/view/Cloud-init/ all the green balls
<blackboxsw> :)
<smoser> blackboxsw, thanks for the suse . i will fix.
<smoser> blackboxsw, actually..
<blackboxsw> yeah was wondering what our story is with suse and templating?
<smoser> yeah
<smoser> see comment
<powersj> smoser: FYI http://paste.ubuntu.com/24866805/
<powersj> that just happened
<powersj> centos7
<smoser> thati sodd
<smoser> powersj, i'll get it fixed.
<blackboxsw> did I mention I love the cI
<blackboxsw> did I mention I love the ci -- continuous integration for the win
<rharper> *whew* ;  all but one integration test passing,   the mixed ipv6 mtu and device mtu case (which we have to workaround in ubuntu as well);
<rharper> smoser: do you prefer a MR per bug/issue, or would you like a PR with all of the fixes and bugs?
<smoser> well, i think one per issue is better.
<rharper> ok
<smoser> powersj, blackboxsw i can recreate that in a centos 6 container (./tools/run-centos 6)
<smoser> but i dont understand it
<smoser> why would this not show failure:
<smoser> $ python -c 'from cloudinit import util; print(util.is_FreeBSD())'
<smoser> False
<smoser> or
<smoser> $ python -c 'import platform; print(platform.dist())'
<smoser> ('centos', '7.3.1611', 'Core')
<smoser> http://paste.ubuntu.com/24867035/
<smoser> oh. because he mocked platform.platform
<smoser> blackboxsw, powersj https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/325771
<powersj> smoser: there will be tox failures on that
<smoser> i think fixed
<smoser> powersj, ^
<smoser> blackboxsw, ^ a review would be good. or rharper
<blackboxsw> smoser comment added
<blackboxsw> doesn't look like _check_freebsd_cdrom returns anything
<rharper> ?
<rharper> true or false
<rharper> it's just code motion for mocking
<rharper> smoser: why don't you need the PY3 in mock anymore ?
<smoser> blackboxsw, you reviewed an older version
<smoser> and caught a valid concern
<smoser> which is now fixed
<smoser> we mock just earlier now, so dont need PY3
<smoser> look at updated diff. you see i remove usage of it.
<rharper> smoser: gotcha, we're mocking the function, so no need to mock open
<smoser> yeah.
<blackboxsw> minor assert param reorder. but +1
<smoser> blackboxsw, ?
<smoser> i'm running a ./tools/run-centos and tox on feature/freebsd-variant
<smoser>  https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/325760
<smoser> bah
<smoser> blackboxsw, you're right. it shoudl ahve been reordered
<smoser> sorry
<smoser> it is pushed now, and i just pushed https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/325760 also
<blackboxsw> +1
<blackboxsw> will watch jenkins
 * smoser goes out. i'll check back in in a few hours
<rharper> later
#cloud-init 2017-06-16
<niluje> this build fails on jenkins: https://jenkins.ubuntu.com/server/job/cloud-init-ci/4/
<niluje> but it works locally
<niluje> any idea why anyone?
<niluje> oh, found I guess.
<dpb1> what did you find
<niluje> it was working locally because I was running unittests on a scaleway instance
<niluje> ok, should be fixed
<niluje> just have to wait until the build is retriggered
<dpb1> niluje: missing a mock?  something like that?
<niluje> exactly
<dpb1> k
<niluje> annnd that's actually not good yet, there's still mocking issues :p
<dpb1> hehe
<blackboxsw> morning folks
<blackboxsw> powersj: I'm looking over the proposed changes failing https://jenkins.ubuntu.com/server/job/cloud-init-integration-proposed-x/73/console  it's related to the branch smoser landed last night
<blackboxsw> he switched bash calls in unit test setup. trying to reproduce the issue here
<powersj> blackboxsw: so that test uses what is in proposed, which shouldn't change unless he did another upload
<blackboxsw> hmm, ohh right, I just keyed in on the error traceback mentioning bash and made a quick assumption. right. Looking more closely
<niluje> https://jenkins.ubuntu.com/server/job/cloud-init-ci/5/nodes=metal-amd64/console
<niluje> any idea why the permission denied here?
<niluje> the mock seems ok to me
<blackboxsw> niluje: looking over the changes that landed there. it feels like the unit test is leaking a call .
<niluje> i'm wondering if it's not because the tests are run as non root
<niluje> when running tox locally I have a RuntimeError: Failed running ['git', 'describe', '--match=[0-9]*'] [rc=128] (, fatal: No names found, cannot describe anything.
<niluje> lol ok
<niluje> git remote add lp https://git.launchpad.net/cloud-init && git fetch lp fixes the issue :p
<niluje> my clone doesn't have all the tags
<powersj> yeah have to have the tags so it can run maketarball which uses the tags to determine the version.
<niluje> maybe it could be optional
 * niluje found the error -_-
<blackboxsw> niluje: looks like you are mocking 'http://169.254.42.42/user_data/cloud-init but get_data calls just "user_data/cloud-init" is that right?
<niluje> that's the not the issue
<niluje> I'm mocking the HTTP call correctly
<niluje> but I'm using a requests adadpter to bind on a specific source port (required by the Scaleway API)
<niluje> which requires to be root
<niluje> so I also need to fake the bind
<niluje> will be good in a second
<niluje> annnddd it should work now.
<niluje> sorry for the failing builds :(
<blackboxsw> thanks for the alert & fix
<niluje> blackboxsw: is that you who retrigger the build?
<niluje> or is it automatic?
<niluje> retriggers*
<blackboxsw> niluje: I'll check the builder to see ( I can retrigger if you've already pushed
<niluje> (I pushed)
<powersj> if you push again, we check for new reviews every 15mins.
<niluje> oh, ok
<niluje> A week or two ago I tested jenkins blue ocean
<niluje> I see your are using jenkins-job-builder
<niluje> using a Jenkinsfile + blue ocean is really a nice alternative
<niluje> https://s3.amazonaws.com/img0.recordit.co/QRyeaiscAS.mp4?AWSAccessKeyId=AKIAINSRFOQXTN4DT46A&Expires=1497631105&Signature=SipNfzd6Mk1vE5DP8KjcCE1972g%3D
<niluje> if you want to see what it looks like :)
<niluje> (it was just a job to discover the features of the blue ocean interface, it basically does nothing)
<powersj> wow that is a clean interface
<niluje> yep
<powersj> I do have a new merge review pipeline coming, I was hoping to have it up today, but ran into an issue with jenkins-launchpad-plugin. I'll try to resolve that Monday.
<powersj> Essentially for every merge we will run this: https://jenkins.ubuntu.com/server/job/cloud-init-ci-nightly/
<niluje> and with this setup, you only need to create a job from jenkins and fill a git URL
<niluje> containing the Jenkinsfile
<niluje> so since the job configuration is saved on git, you no longer have to use jenkins job builder :)
<powersj> yeah that would be nice
<niluje> can you show me the corresponding jenkinsfile powersj ?
<powersj> http://paste.ubuntu.com/24873098/
<blackboxsw> thx powers for the build kick
<niluje> oO
<niluje> is it the full jenkinsfile powersj ? or just a part of it
<niluje> I thought they had to start with "pipeline {"
<powersj> That's what I have inside the pipeline section right now. It isn't checking out from git, so I'm not sure if that affects it.
<niluje> ok
<niluje> I thought the first block inside a "stage" block needed to be a "steps" block
<niluje> http://blackhole.brmzkw.info/2017-06-16/Jenkinsfile <- one of my tests with jenkinsfile
<powersj> interesting
<powersj> it does say "required" on the doc site
<niluje> powersj: the "steps" section?
<powersj> yeah
<niluje> yeahh. build is passing. Thanks for your help
<niluje> yep, without steps I had an error
<niluje> honestly I'm totally blind when working with jenkins
<niluje> they have the worst documentation possible
<niluje> in case you are interested (because I lost some time to search how to do it): it is possible, in a pipeline, to interactively ask for administrator input (like "do you want to approve this build? yes/no" or "can you fill this text input so I can later use the content int he build? <input value>")
<niluje> I have Jenkinsfiles doing that
<niluje> it is also possible to use the jenkins API to fill these inputs
<powersj> that is cool, the thing I would like to figure out is parallel steps
<niluje> wait for it :p
<powersj> that way I can reduce our CI time
<powersj> haha
<niluje> powersj: http://blackhole.brmzkw.info/2017-06-16/JENKINS.md
<niluje> (if you see errors feel free to report :))
<niluje> i wrote this documentation for my coworkers who discover jenkin
<niluje> s
<powersj> nice thx!
 * powersj look's at proposed failures
<powersj> blackboxsw: I am 90% certain it is due to me updating the version of pylxd on the system
<powersj> blackboxsw: it is. we are currently playing games with the version of tox + pylxd + lxd because of some incompatibilities between the versions. The backported code does not work with the latest lxd + pylxd combo, even when running with a tox specified version of lxd.
<blackboxsw> bah
<powersj> because we have two good clean runs of proposed I am going to say we are good, disable the job for now so we don't keep randomly running and failing and re-enable for the next proposed test.
<powersj> bad timing in we had an upload to proposed + broken pylxd + major update to integration tests
<blackboxsw> powersj: I was wondering  if this is also impacted by  read-dependencies now attempting to upgrade system-packages each integration test run now as well
<powersj> It may, but on an LTS I would hope it would not break
<ajorg> is it okay to delete a branch in my fork once it's been merged?
<blackboxsw> ajorg: should be fine there
<blackboxsw> ok 4 SRU bug verifications needed still
 * blackboxsw is setting up an azure account
<blackboxsw> found two other bugs that were part of the SRU, but weren't listed in our trello card. adding them now
#cloud-init 2018-06-11
<cpaelzer> smoser: blackboxsw: yeah we had no extra chrony love pre Bionic
<cpaelzer> and getting them to work in container was due to that >=Bionic as well
<cpaelzer> but reading the backlog you found all that
<cpaelzer> anything left open with it?
<mgerdts> blackboxsw the notes say that the last bi-weekly meeting was 13 days ago.  The topic says the next meeting is in 7 days.  Are we rounding up due to a holiday a couple weeks back?
<rharper> mgerdts: likely, a few folks on the team are out today, so we figured we'd need a bit more time to have additional work land before the next meeting
<mgerdts> Is smoser one of those that's out?
<dpb1> mgerdts: yes
<dpb1> mgerdts: and blackboxsw :)
<dpb1> holiday season unfortunately
<mgerdts> Are they out all week?
<dpb1> scotts back on wednesday, blackboxsw later today
<mgerdts> ok, thanks
<ffledgling> Hello, running into some trouble w/ configuring dns for an image that uses cloud-init. Cloud init doesn't seem to pick up changes from either network-config nor from user-data's resolv module
<ffledgling> Is this functionality broken? (Using a nocloud provider with a fedora 28 image, cloud-init v17.1)
<rharper> ffledgling: do you have a paste of your cloud-config/user-data and the cloud-init.log available ?
<ffledgling> rharper: I can give you the former, where can I find the cloud-init.log?
<rharper>  /var/log/cloud-init.log
<ffledgling> rharper: log - https://paste.fedoraproject.org/paste/ez66J-WWe6NeZkh5mTASUg
<ffledgling> rharper: configs - https://paste.fedoraproject.org/paste/6y2H38~zsLG0PyZ4JMPkOw
<rharper> ffledgling:  it looks like the cc_resolve_conf module is not enabled by default,  in /etc/cloud/cloud.cfg , under cloud_config_modules: list, upstream doesn't have a  - resolve_conf  in the list;  I suspect that's why the user-data module isn't called
<ffledgling> rharper: I see, but then it's not picked up from network-config either
<rharper> where is your network-config stored?
<rharper> what datasource are you using, generally one cannot provide network-config via user-data ; it has to be part of the data-source (or in the cloud config on disk)
<ffledgling> it's all in an iso mounted into the VM; source is nocloud
<rharper> things like NoCloud can use it, or ConfgDrive
<rharper> ah, perfect, lemme look at the log then
<ffledgling> The other stuff in the config like ip, gateway are picked up fwiw, so it's just the DNS that's causing trouble
<ffledgling> This seemed to encapsulate the symptoms - https://bugzilla.redhat.com/show_bug.cgi?id=1473890 ; but I'm not sure if it's still a problem upstream
<ubot5> bugzilla.redhat.com bug 1473890 in cloud-init "cloud-init sysconfig renderer does not render DNS or GATEWAY" [High,Closed: eol]
<rharper>  /etc/sysconfig/network-scripts/ifcfg-eth
<rharper> that's fixed, you can try cloud-init from copr repos
<rharper> https://copr.fedorainfracloud.org/coprs/g/cloud-init/cloud-init-dev/
<rharper> what's in your ifcfg-eth0 ?
<rharper> do you see any DNS_ settings there?
<ffledgling> https://paste.fedoraproject.org/paste/rlRkQVQ0GJ8lesH4Qp3hNg
<mgerdts> The "pick up changes" part of your question makes me suspect  https://bugs.launchpad.net/cloud-init/+bug/1765801
<ubot5> Ubuntu bug 1765801 in cloud-init "network should be optionally reconfigured on every boot" [Undecided,Confirmed]
<rharper> that's not related directly to you query on dns entries being rendered in sysconfig in a NoCloud scenario
<ffledgling> nope, I don't even generate the resolv.conf myself, not touching anything network directly, everything goes through cloud-init
<ffledgling> It generates the file, but it's always empty
<ffledgling> rharper: how do I try a newer version of cloud-init with an existing image? If that's possible at all?
<rharper> yes
<rharper> add the repo to the image, if you can get into it, then upgrade cloud-init, and then you can  rm -rf /var/log/cloud-init* /var/lib/cloud/{instance*, data}  ; and reboot
<rharper> trunk renders your network-config like this in sysconfig: https://paste.ubuntu.com/p/g5k9j3XpqV/
<rharper> that should fix your dns issue
<ffledgling> let me try that
<rharper> you may find that 17.1 has this bug in it:  https://bugs.launchpad.net/cloud-init/+bug/1712680
<ubot5> Ubuntu bug 1712680 in maas-images "cloud-init re-generates network config every reboot overwriting manual admin changes on CentOS." [Undecided,New]
<rharper> in which case, comment #11 is helpful ,for workarounds, https://bugs.launchpad.net/cloud-init/+bug/1712680/comments/11
<ffledgling> Hm, don't think that's my issue - since I'm destroying and recreating the image from scratch every time, so this is after first boot
<rharper> oh sure, I mean, if you make a local change, you may still see this issue
<ffledgling> ah, good to know
<ffledgling> rharper: so... I can confirm the newer version works
<rharper> great!
<ffledgling> Now the question is, how do I get this in the default image?
<rharper> that's a fedora question
<rharper> there are some folks from the project in here
<ffledgling> would you know who?
<rharper> ffledgling: larsks is with RH, may know a good connection
<larsks> What do I know? I deny everything.
<rharper> hehe
<rharper> larsks: where would ffledgling file a request to get the fedora cloud images with an updated cloud-init
<rharper> https://fedoraproject.org/wiki/Cloud_SIG looks like a place to start
<larsks> You could file a bugzilla against Fedora...https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora
<rharper> mailing list died off in 2015 though
<ffledgling> larsks: that, I was going to do, but there was already a bug for fedora 26 that seems to have died...
<ffledgling> is that still the best way?
<larsks> There has been some talk recently about rebasing the Fedora package on the latest cloud-init release.  The primary package maintainer I think is just busy.
<larsks> Someone recently showed up volunteering to help with package maintenance, but I'm not sure where that stands right now.
<larsks> I would say go ahead and open the bug, ping me with the url, and I am happy to prod some folks.
<ffledgling> larsks: okay, thank you!
<ffledgling> If the package update is not a major process then I can probably volunteer to get this one out as well, but I suspect that might not be how it works
<ffledgling> larsks: https://bugzilla.redhat.com/show_bug.cgi?id=1589972
<ubot5> bugzilla.redhat.com bug 1589972 in cloud-init "cloud-init + nocloud ignores DNS configuration" [Unspecified,New]
<ffledgling> Let me know if there's more detail needed there
<larsks> ffledgling: thanks, looks good.
<blackboxsw> mgerdts: afternoon/evening. yeah I had a little break this morning. I tried sending out an email to cloud-init@lists.launchpad.net to notify folks about the shift. (and changed the IRC topic to reflect that as you saw).
<blackboxsw> rharper: I'm taking over https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/347727 to get some unit tests over it (and to also fix the issues with unicde in custom-scripts too)
<rharper> blackboxsw: ok
<blackboxsw> rharper: btw, https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/347559 should be all done per review comments
<rharper> k, reviewing
<blackboxsw> avoided warnings unless device-level != subnet-level mtu value
<blackboxsw> and added unit test coverage of that
<rharper> ci says no
<rharper> is that glitch or needs fix to land before it says yes ?
<blackboxsw> bah, pycodestyle. pushing fix now:wq
<blackboxsw> and apparently saving that message in irc too
<blackboxsw> rharper: pushed. I had left a debug maxDiff = None in there that isn't required.
<blackboxsw> looks like smoser's branch covers the string conversion of all mime types. ok, so we don't need separate changes for x-shellscript or cloud-boothook
<blackboxsw> ok just unit test coverage exhibiting the uncode problem
<rharper> nice
<blackboxsw> unicode rather
<blackboxsw> cpaelzer: way late on the response back to you. but nope, artful//chrony is not a supported thing. so nothing to worry about it. I just need to fix cloud-init's CI to avoid trying to test chrony deployment on artufl
<blackboxsw> cpaelzer: way late on the response back to you. but nope, artful//chrony is not a supported thing. so nothing to worry about it. I just need to fix cloud-init's CI to avoid trying to test chrony deployment on artful
<blackboxsw> well on artful inside containers only
#cloud-init 2018-06-12
<cpaelzer> blackboxsw: you were late to reply, but you beat it by replying twice :-)
<lucidguy> Ok, can you not set ssh-authorized-keys for the root account?
<lucidguy> via cloud-init
<blackboxsw> lucidguy: you can, just provide a cloud config like the following :
<blackboxsw> #cloud-config
<blackboxsw> disable_root: false
<blackboxsw> ssh_authorized_keys:
<blackboxsw>   - ssh-rsa AA.....
<blackboxsw> found per http://cloudinit.readthedocs.io/en/latest/topics/modules.html#ssh
<lucidguy> I tried that, minus the disable_root
<blackboxsw> cloud-init automatically disables the root user for securtiy
<blackboxsw> cloud-init automatically disables the root user for security. yeah have to turn that default feature off
<lucidguy> Thanks blackboxsw
<blackboxsw> no problem
<lucidguy> I hope its also not an issue with this centos6.9 cloud image
<blackboxsw> powersj: rharper https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/347824 for integration test breakage on artful containers per https://jenkins.ubuntu.com/server/view/cloud-init,%20curtin,%20streams/job/cloud-init-integration-lxd-a/377/console
<powersj> \o/
<blackboxsw> bah good review powersj fixing that conditional was bogus :/
<blackboxsw> cut-n-paste fail
<blackboxsw> pushed e5baa98996aacd7a1906fad4a77a0aaa679f5827 powersj
 * powersj looks
<blackboxsw> thx .landing once jenkins is happy again on that branch
<blackboxsw> salt_minion you are next on my radar buddy: https://jenkins.ubuntu.com/server/job/cloud-init-integration-nocloud-kvm-b/212/console
<powersj> blackboxsw: will both of those make us all green again?
<blackboxsw> powersj: I think that will do it for cloud-init. (integration-b and  integration-c are both salt_minion I think (on all platforms)
<powersj> sweet
<blackboxsw> so it feels like a bug in how cloud-init writes salt minion keys.
<blackboxsw> in bionic++
<blackboxsw> but hadn't really gotten to a proper solution yet. (that citest --preserve-instance param will help now)
<blackboxsw> another one for readability in jenkins rharper powersj https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/347833
<rharper> looking
<powersj> oh yes please
<rharper> indeed
<blackboxsw> yeah I finally hit my limit with that giant dict of hopelessness
<blackboxsw> so much time wasted searching the browser for FAILURE: or FAILED: etc
<blackboxsw> good thoght rharper will drop all those indents
<blackboxsw> rharper/powersj: updated the decription with the new output   I'm all ears for other formatting thoughts
<rharper> lemme see
<blackboxsw> powersj: rharper eureka. found the salt-minion error for jenkins on bionic ++
<rharper> \o/
<powersj> sweet
<blackboxsw> newer versions of salt-minion auto-move /etc/salt/pki/minion files over into /var/lib/salt/pki/minion/* no logs tell you about the behavior. even setting log_level:trace gives me no info about this
<blackboxsw> though I did see that trace presented an authentication request to the salt master I had to set up. it contained a new path that I hadn't written:  Initializing new AsyncAuth for ('/var/lib/salt/pki/minion', 'b3', 'tcp://10.195.58.13:4506')
<rharper> sneaky
<blackboxsw> man. that was a bit tedious. but I know know how to setup salt_master on ubuntu to auto-accept new-client certs :)
<blackboxsw> I'm going to write a tiny blog on this
<blackboxsw> s/know know/now know
<rharper> blackboxsw: yeah, good content with some cloud-init in there = )
<blackboxsw> ok salt_minion integration test passes on bionic/cosmic
<blackboxsw> putting up another small mp.
<blackboxsw> ok it's up https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/347856
<blackboxsw> rharper: powersj  if either of you have a chance on that mp above. it could get us green on jenkins
<powersj> blackboxsw: land it ;)
<blackboxsw> thanks powersj
#cloud-init 2018-06-13
<caribou> Hello,
<caribou> I'm working on adding network support to our DataSourceScaleway
<caribou> So far, I'm able to get the config written on the first boot, but it gets overwritten by subsequent reboots
<caribou> I believe that this is caused by the DEP_NETWORK dependancy
<caribou> If I understand correctly, Removing this DEP_NETWORK dependancy means that our DataSource is responsible for bringing up the network, right ?
<smoser> caribou: a local datasource is one that can be identified positively by local resources.
<smoser> it may declare supply network configuration for the system.
<mgerdts> As I was reading though https://hackmd.io/NUUO4nndS4CXTItl8Rs6Nw?both, I was a bit confused by lines 59 & 70.  Should those both start with 'nic'?
<smoser> i have to agree with you. i made the change there, but there are other occurences of 'nic', . lets ask rharper when he is in.
<mgerdts> ok, sounds good
<rharper> smoser: mgerdts:  yeah, it was nic there;   there is a general rough equivalence between nic and network;  but specifically hotplug of a NIC (which is really a udev event on kernel subsystem "net") is a rudimentary way of providing an indication to the guest that it should query metadata service for updated configuration;  you can imagine that a provide may indicate this in other ways and may want to reconfigure
<rharper> "networking" whether or not a NIC has been added or removed
<rharper> I'm happy to bikeshed on the schema we want to be more precise
<KingJ> I can see a fix for LP#1774666 has been merged in to trunk, what's the best way for me to get that on my system? Wait for the next build of the cloud-init-dev/daily PPA?
<rharper> yep
<rharper> bug #1774666
<ubot5`> bug 1774666 in netplan.io (Ubuntu Cosmic) "Bond interfaces stuck at 1500 MTU on Bionic" [Undecided,Confirmed] https://launchpad.net/bugs/1774666
<KingJ> rharper: Thanks, i'll keep an eye on the PPA
<rharper> KingJ: for Bionic, we need to do an SRU;   We'll do a new upstream snapshot during the SRU process and the daily ppa https://launchpad.net/~cloud-init-dev/+archive/ubuntu/daily  get get the bionic package before the SRU is complete; you can test that out there but once the SRU has been started, the package will also showup in bionic-proposed
<KingJ> What would the rough timelines be around the SRU / proposed / daily build? (apologies, this is the first time i've been blocked on something where the fix isn't quite out yet, but I appreciate it's not an instantaneous thing!)
<rharper> we can get a new upstream snapshot landed possibly today, maybe tomorrow, then the next days ppa will have the package, the SRU takes processing time to verify the changes once it's uploaded to -proposed, and then verification that it resolves the issues and no regressions, then a 7 day wait in -proposed for other testing and general wait before it's copied to -updates
<KingJ> Great, i'll keep an eye out - thank you for the information :)
<rharper> sure
<mgerdts> rharper: thanks for clearing that up
<blackboxsw> smoser: pushed pycodestyle fix to handle utf8 chars in cloud-config branch
<blackboxsw> jenkins is testing now https://jenkins.ubuntu.com/server/job/cloud-init-ci/79/
<blackboxsw> just kicked off https://jenkins.ubuntu.com/server/job/cloud-init-integration-ec2-b/24/ which should give us a green now that salt_minion test fixes are in tip
<powersj> smoser: how do we 1) add bionic and 2) update devel to cosmic here: https://code.launchpad.net/~cloud-init-dev/+recipes
<smoser> rharper: responed https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/347698
<smoser> powersj: getting that
<smoser> powersj: https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily-devel
<smoser> bah.
<smoser> Distribution series. had 'bionic', not 'cosmic'
<smoser> fixing
<powersj> ok so that just required selecting a different distro?
<blackboxsw> ok fixing this today https://bugs.launchpad.net/cloud-init/+bug/1776701
<ubot5`> Ubuntu bug 1776701 in cloud-init "ec2: xenial unnecessary openstack datasource probes during discovery" [High,In progress]
<runelind_q> If I understand the workflow correctly, I apply an lxc profile to an ubuntu 18.04 container which influences cloud-init, which in turn influences netplan?  I'm just trying to figure out the best way to add a static DNS (ipv6) server
<mgerdts> Does this mean that I will soon be able to not `route add -reject <link-local-address>` before running tests?
<blackboxsw> runelind_q: my most common use-case for driving changes in cloud-init though lxc would be 'lxc init ubuntu-daily:bionic myb1; lxc config set user.user-data - < mycloudconfig.yaml;   lxc myb1 start;'
<blackboxsw> cat > mycloudconfig.yaml <<EOF
<blackboxsw> #cloud-config
<runelind_q> looks like some of those values can just be set in the lxc profiles rather than passing another yaml file
<blackboxsw> runelind_q probably so. lxc provides cloud-init with the proper-full network config which ends up in /var/lib/cloud/seed/nocloud-net/network-config on the booted container
<runelind_q> cool, I'll take a look there
<runelind_q> lots of abstraction going on :)  It's like Inception networking
<blackboxsw> lxc also provide vendor_data as distinct from user-data (which both get merged to provide cloud-init customization)
<blackboxsw> yeah definitely runelind_q, so when you provide a profile checkout that seed directory to see how lxc mapped it out. cloud-init consumes those to "do the right thing" with netplan
<rharper> to do network-config in lxd, you need to specify that separately; you cannot user-data in network-config
<rharper>  https://github.com/lxc/lxd/blob/master/doc/cloud-init.md
<blackboxsw> rharper: can't you define the network-config via the same type of mechanism (right a profiles would allow you to differentate network-config for user.user-data)
<blackboxsw> just saw the link, good one
<rharper> yeah, it's just like user.user-data but it's a different file/template (network-config)
<blackboxsw> ok good deal. So, if using the CLI instead of profiles :    lxc init ubuntu-daily:bionic myb1; lxc config set myb1 user.network-config -< mynetwork-v1-or-v2.yaml; lxc start myb1;'
<blackboxsw> and that should ultimately show up in /var/lib/cloud/seed/nocloud-net/network-config I would presume
<blackboxsw> on the container
<rharper> yeah, was going to paste a quick  config to add the additional dns entry
<blackboxsw> thanks rharper
<rharper> that said, I don't know if we have a way to append additional DNS entries on an interface using DHCP
<runelind_q> I guess I could also see if containers will listen to DNS servers being advertised via SLAAC
<blackboxsw> shouldn't that come via the dhcp option when the client requests a lease?
<runelind_q> this is all ipv6
<blackboxsw> yeah /me was thinking option 23 and option 24 in dhcpv6 :(
<rharper> almost there
<runelind_q> stay on target
<rharper> =)
<rharper> I've got tone
<runelind_q> yeah, not so much for RDNSS
<runelind_q> oh, there it goes.
<rharper>  https://paste.ubuntu.com/p/9rPcBrGX3x/
<rharper> I don't have dhcp6, so I can't tell, but on my system, it puts the static v6 dns in the list; not sure about priority; this falls into dark magic of networkd/systemd-resolved
<runelind_q> yeah, it doesn't show up is resolv.conf, but v6 DNS works fine Â¯\_(ã)_/Â¯
<blackboxsw> ok jenkins integration-ec2-b is finally green again. only known cloud-init CI errors remaining are ec2-x (because of openstack warnings) and  the -cosmic tests failing because of our ppa
<blackboxsw> both in flight
<blackboxsw> ok need to sort unit tests, but will be proposing the following as a fix for OpenStack datasource to avoid unneeded metadata probes on ec2
<blackboxsw> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+ref/bug/1776701-openstack-local-no-probe-on-ec2
<blackboxsw> heading to an early lunch.
<smoser> blackboxsw: 2 fixes in review-mps pushed
<smoser> https://github.com/CanonicalLtd/uss-tableflip/commit/15b5e593c48aba5f24fc921a11385e7fa8f4eb29
<smoser> https://github.com/CanonicalLtd/uss-tableflip/commit/66fb6a5e9df8db41e96a13d6f1d7ce913bd22028
<rharper> runelind_q: sorry, resolv.conf is always going to point to 127.0.0.53 which is systemd-resolved caching nameserver (much like previous resolvconf in Ubuntu Xenial);  you can run:  networkctl status or systemd-resolved --status  and you can see the set of DNS endpoints you've got configured
<blackboxsw> ok no probe fix is up https://trello.com/c/0s2gnxyX/835-openstacklocal-should-not-probe-metadata-on-ec2
<blackboxsw> now onto joyent/azure hot/cold plug
<blackboxsw> powersj: :(  looks like integration-c w/ lxd is an issue https://jenkins.ubuntu.com/server/view/cloud-init,%20curtin,%20streams/job/cloud-init-integration-ec2-c/5/console
<blackboxsw> on lxd and ec2
<powersj> yeah noticed that now that we have a daily cosmic deb
<powersj> I can take a look tomorrow if you want
<blackboxsw> man it's like the red won't go away :/
<powersj> heck of a lot better though
<blackboxsw> powersj: yes please, I think I need to finish a strawman for a mtg tomorrow w/ joyent
<powersj> and largely due to me switching from building master to using the built daily deb
<blackboxsw> yep, I was gonna say, "all powersj fault"
<powersj> totally
<blackboxsw> PEBPAK  (problem exists between Powers and keyboard)
<blackboxsw> has nothing to do with code-quality mind you
<powersj> ;)
#cloud-init 2018-06-14
<caribou> smoser: thanks for the info. I based my changes on the DataSourceOpenstack & DataSourceEc2 so it should be good. Merge Request to follow soon
<smoser> caribou: ok. its fine to share early if you want to make sure you're going in the right direction.
<caribou> smoser: I'm about to send it; do you people want an LP bug to go along with it or simply the MR ?
<caribou> smoser: matter of fact: https://code.launchpad.net/~louis/cloud-init/+git/cloud-init/+merge/347973
<smoser> k. thanks.
<caribou> smoser: but for some reason, now that I packaged it it fails on me
<caribou> & I got jenkins failure. I'll work on that
<caribou> smoser: I fixed the issue that might have caused the failed Jenkins build. Any way to re-launch the build ?
<smoser> caribou: https://jenkins.ubuntu.com/server/job/cloud-init-ci/83/
<caribou> smoser: yeah, I saw it & found the reason. I'll fix that tomorrow, it's getting near EOD
 * caribou makes a note not to run unittests as root
<caribou> hmm, sorry I saw build-82 which also failed for the same reason. 83 will fail as well
<smoser> caribou: ok. thanks.
<blackboxsw> ok decoupling azure-hotplug  from on-boot maintenance out of this wip branch today https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/347991
<blackboxsw> smoser & rharper I've started decoupling maintenance event content into this WIP branch https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/348000
<blackboxsw> I wanted to throw that up as a very broken/basic first start, so if there are initial suggestions to where we want to move content or behavior I can tackle that as I start adding unit tests and iron out the details.
 * blackboxsw runs for an early lunch
<rharper> blackboxsw: ok, I'll look
<smoser> blackboxsw: i just tagged taht 'work in progress'
<smoser> so pushes wont have ci on them and such
<blackboxsw> +1 smoser thanks.
<blackboxsw> ok starting to work on that branch in earnest now
<blackboxsw> smoser: and rharper: if you guys get a chance to look over the fix for xenial-Ec2 that'd be great to avoid OpenStack init-local discovery
<blackboxsw>  side-effects https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/347937
<rharper> yeah
<blackboxsw> it's also blocking our CI being green (and the next SRU)
<blackboxsw> thx
<blackboxsw> smoser are you peeking at any fixes in cloud-init  the CI lxd-cosmic issues per  https://github.com/lxc/lxd/issues/4649?
<smoser> blackboxsw: almost have
<blackboxsw> ok, I won't touch it, I had created a card to look at it, but I want to get the joyent/azure stuff in shape first
<blackboxsw> card is here https://trello.com/c/bYpOVdxU/836-fix-lxc-31-301-collision-pre-existing-lxdbr0-automatic-bridge-creation
<smoser> blackboxsw: ok. running tox locally, fixing flake8 and pushing for review
<smoser> (and then i'll actually test :)
<smoser> blackboxsw: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/348005
<blackboxsw> thanks for the reviews: addressed review comments on dont-probe-openstack-ec2 branch https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/347937
<blackboxsw> tried addressing a couple of the initial comments on the cold-plug Event handling branch to maintain network on boot https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/348000
#cloud-init 2018-06-15
<caribou> Hello, could something explain the fact that all unittests pass in the jenkins context but when the pkg builds, one of them fail ?
<caribou> and moreover, it builds fine here on X/B & C
<caribou> FYI https://jenkins.ubuntu.com/server/job/cloud-init-ci/91/console
<smoser> blackboxsw or rharper would like more feedback on https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/348005
<rharper> y
<smoser> caribou: sorry give me 5 minutes i'll poke a bit.
<caribou> smoser: thanks. no real rush as I'm onto something else & won't get back at it until monday
<smoser> caribou: oh. well, the failure is simply because the build environment did not have udeveadm settle
<caribou> smoser: anything I need to do on my side ?
<smoser> caribou: sorry... rough day.
<smoser> i'll try to take a quick look
<smoser> i'll follow up on your MP
<smoser> blackboxsw: you typed: "just a nit"
<smoser>  https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/348005
<smoser> but .... what nit ?
<rharper> the whole thing ?
<rharper> =P
<blackboxsw> haha, it didn't save my comment.
<blackboxsw> smoser: comment attached
<blackboxsw> dropping unused assignemnt
<blackboxsw> smoser: rharper if we are cutting a cosmic branch today for release, I'd like to see https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/347937 landed if we could
<smoser> http://paste.ubuntu.com/p/xFjYFNggh2/
<smoser> that is me testing... the lxdbr0 stuff. looks like we need to fix.
<rharper> smoser: =(
<rharper> looks like there is existing config that auto creates we'll need to sort out
<rharper> lxc profile list default or something dumps all of the set config
<smoser> yeah, fun
<rharper> it's friday, what else were you doing =P
<smoser> blackboxsw: yeah, i'd like to get that too.
<smoser> this is a pita
<blackboxsw> referring to ssh locale.... or lxc
<smoser> lxc
<blackboxsw> ehem, sorry for the bad review without an integration test run :/
<smoser> my fault. i just assumed.
<smoser> and this is waht that leads too
<smoser> to
<blackboxsw> yeah running commands on an existing install which had lxc seemed to make sense as a validation point, but I should've actually run through the whole system during an install to see what fell out
<blackboxsw> rharper: per openstack variable names comment, what use-case are you thinking about where a user might want
<blackboxsw>  VALID_DMI_PRODUCT_NAMES or VALID_DMI_CHASSIS_ASSET_TAGS  and not just the detect_openstack() function instead?
<rharper> not the user, any other part of cloud-init that would want to know what values OpenStackDS says are valid product names to positively identify OS
<rharper> my point was mostly, if we find out that we have another PRODUCT name or asset tag, there's a single place to add it to a  list
<rharper> rather than find where in the code an indiviual value is being used
<blackboxsw> ahh you are thinking DataSource base class.detect_valid_ds or something
<rharper> yes, that'd be the next path; we've not yet migrated the logic from ds-identify into the DS classes themselves for a path which doesn't use ds-identify
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/348005
<smoser> that now passes bionic and xenial
<smoser> but doesnt have tests
<blackboxsw> return volley https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/347937
<blackboxsw> I addressed smoser's and rharper's comments, with exception of the docs reporting what DMI info we look at on openstack. I don't really know which way to go here. I like the discovery detail, but it is something that will change frequently, and it'll leave us with some significant gaps in docs if detection moves beyond the docs.
<rharper> it doesn't really change that much
<rharper> the set of valid DMI product names is static, until we know otherwise
<blackboxsw> also, ds-identify and python Datasource.get_data detection aren't aligned currently in terms of logic :/
<blackboxsw> the exception is probably Ec2 and openstack datasources which are fairly well aligned now
<rharper> they don't have to mirror each other in implementation but certainly we don't want one to say we're on OS when we're not
<smoser> blackboxsw: can you just add a comment about /proc/1/environ that that is how nova-lxd identifies itself.
<smoser> (in the doc changes)
<blackboxsw> rharper: right, but I was wondering what we end up documenting in rtd in those cases. the ds-identify detection behavior, or the python detection
<smoser> and then... i really dont think you need to cmention chassis_asset_tag of OpenTelecom...
<blackboxsw> smoser: +1
<rharper> I don't follow
<rharper> if they're not reporting the same thing, then where do they differ ?
<smoser> i'm fine with a 'Discovery' that just describes general intent, but does not contain the implementation in english text.
<rharper> the detection is the same, I think the behavior w.r.t strict mode results in different paths
<rharper> right, I just meant to document that, we currently positively recognize the following values DMI_PRODUCT = [1, 2] as users of the OpenstackDatasource
<rharper> no need for anything more than disclosing which values and from where we check
<blackboxsw> rharper: right, it's the strict mode setup that allows some datasource detection to be a little looser in get_data() when returning False
<blackboxsw> so datasource might have a subset of checks
<blackboxsw> which we ultimately should align I think w/ ds-identify behavior
<blackboxsw> but, "not there yet" â¢
<rharper> I'm still not really following, it sounds like you're suggesting that the datasource as is now would say yes when ds-identify says no
<rharper> and if that's the case, what scenario is that ?
<rharper> that smells like a bug
<rharper> effectively, non-strict mode returns maybe when we cannot positivley identify it; the DS is going to do the same (it'll run the network probe if it can't positively know via the DMI values, etc)
<blackboxsw> rharper: well that's what this branch just fixes, openstack datasource determined that it wasn't valid on ec2 platforms because OpenStack.get_data happened to probe the metadata service to find that it wasn't openstack... whereas ds-identify detects that quickly via DMI product_name/chassis_asset_tag checks
<rharper> well, no, in non-strict mode, ds-identify still has to say maybe, which means we need to probe over network
<rharper> so, AFAICT things are still aligned, we're adding support for strict mode in the Ds class itself; but not setting it to strict mode only, which is just fine
<rharper> I mean, it can be in cosmic, I think; but IIRC this is onlyi nXenial which isn't in strict mode; so we know we're probing, and this is a detect short-circuit to prevent probing when we absolutely know we're on OS
<blackboxsw> rharper: yes this issue was only in Xenial and I think you are right it's due to non-strict vs strict mode
<rharper> I don't see any divergent behavior here, so maybe it's just some phrasing; but the branch you have looks fine, I think does the right thing on xenial (non-strict mode) ;
<rharper> and on a non-ds-identify OS, where strict=true, then the get_data() would say no without positive id, right ?
<blackboxsw> yeah, I'm making hand-wavy suggestions, I'll have to find concrete examples we can talk about monday. (I thought I had come across a couple)
<rharper> so that should mirror what we get with ds-identify
<rharper> ok
<blackboxsw> rharper: end result in all cases I think the python code does mirror results from ds-identify.  I think there may be some gaps where the datasource does more work than it needs to to vet environments as DS_NOT_FOUND that ds-identify does. Some of that behavior is because the get_data() actually has does addtional probing of the metadata services etc. But some of that work could be avoided with a tighter exit NOT_FOUND
<blackboxsw> detection.
<blackboxsw> wow typos. ok. I'll dig up a couple examples
<rharper> ok, so optimizing but we don't have differing results; which I certainly fine
<blackboxsw> ci needs fixing smoser on your lxc brnach
<blackboxsw> rharper: correct, sorry for the fuss, just optimization (early exit on known invalid datasource detection)
<rharper> ok
<smoser> fudged
<smoser> d key sticks down
<smoser> rharper, blackboxsw https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/348005 i'm happy with that now, and have tested in x, b, c
<smoser> test coverage is not 100% though. it doesnt test the lxc command error path.
<smoser> but running in x, b, c does test that for exit code = 1
<rharper> we weren't testing lxc error path before either, right ?
<smoser> well before the rework i had 100% coverage on the newly added code
<rharper> oh, hehe
<smoser> but yeah, if lxc fails with other than 1, its going to bomb out
<smoser> which is kind of fine..
<smoser> i have to run. i'll check back in in a few hours
<rharper> ok
#cloud-init 2018-06-16
<smoser> blackboxsw: there?
<smoser> blackboxsw: i approved https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/347937
<smoser> feel free to land.
<smoser> and if you review mine https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/348005 feel free to approve and land.
<smoser> or rharper . or give feed back and i can fix
<smoser> i'd like to have an upload maybe tomorrow or something to get an image with this in it by monday
 * smoser goes afk for a bit
<blackboxsw> smoser: yeah kiddo bed time. I had to fix xenial against my branch
<blackboxsw> mismanaged unit test mock once I switched to get_proc_env
<blackboxsw> unit tests passing landing now after a test run
<blackboxsw> running a couple integration tests on your branch and will cut a cosmic mp
<blackboxsw> ok landing https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/348005 now
<blackboxsw> just waiting on review-mps
<blackboxsw> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/348113 posted for cosmic release
<blackboxsw> for rev 18.2-77-g4ce67201-0ubuntu1
<blackboxsw> linked to card https://trello.com/c/LvsbD0Qe/839-cosmic-upload-release
<blackboxsw> ok I'm outta here
<blackboxsw> deb built fine on my end
<smoser> hm..
<smoser> how is
<smoser>  https://jenkins.ubuntu.com/server/view/cloud-init,%20curtin,%20streams/job/cloud-init-integration-nocloud-kvm-c/31/consoleFull passing
<smoser> it ran with 18.2-1890-gfaa6f07-0ubuntu1+1614~trunk~ubuntu18.04.1
<smoser> and TestLxdBridge test passed... how did it do that ?
<smoser> blackboxsw, rharper, powersj food for thought.
<smoser> oh. thats why...
<smoser>  https://jenkins.ubuntu.com/server/view/cloud-init,%20curtin,%20streams/job/cloud-init-integration-nocloud-kvm-c/31/artifact/cloud-init/results/nocloud-kvm/bionic/
<smoser> icloud-init-integration-nocloud-kvm-c actually runs bionic
<smoser> hm...
<smoser> powersj. i tried to fix that with
<smoser>  http://paste.ubuntu.com/p/RCpQRWwWFW/
<smoser> and submitted a pull request
<smoser>  https://github.com/canonical-server/jenkins-jobs/compare/master...smoser:fix/nocloud-kvm-c-use-cosmic?expand=1
<smoser> but it says
<smoser> "This repository has been archived by the owner. It is now read-only."
<powersj> smoser: new repo is at https://github.com/CanonicalLtd/server-jenkins-jobs
<smoser> ah.ok.
<powersj> I have updated it with your patch
<smoser> k. thanks
<powersj> thank you for looking at it :)
<smoser> yeah... kicked of a ec2-c build also.
<smoser> i'll get an upload in tonight if both of those are green.
<smoser> bah. more failure. unrelated to lxd. possibly is keyserver downtime.
<smoser>  https://jenkins.ubuntu.com/server/view/cloud-init,%20curtin,%20streams/job/cloud-init-integration-lxd-c/35/console
<smoser> so i'm build-and-pushing
<Serim> TESTING TESTING
<Serim> TESTING TESTING
#cloud-init 2020-06-08
<falcojr> If I compare http://archive.ubuntu.com/ubuntu/pool/main/c/cloud-init/ to https://launchpad.net/~cloud-init-dev/+archive/ubuntu/proposed, why is the PPA behind proposed? My understanding was that it took some time for the package to make it to proposed, and we also pushed it to the PPA and could use that in the mean time for testing. Am I
<falcojr> misunderstanding something?
<lucasmoura> Hey everyone, I just realized that for xenial, we are dropping the jsonschema requirement from the package. In that case, our schema tests for https://git.launchpad.net/cloud-init/commit/?id=8bcf1c06 are not properly applied, since xenial will not perform any verification on the schema file as can be see here: https://github.com/canonical/cloud-init/blob/4261ae538563d262bc76b8c55f7cc0c8abb14b00/cloudinit/config/schema.py#L90
<lucasmoura> Is that something we are okay with for this release or should we change that behavior on xenial ?
* Odd_Bloke changed the topic of #cloud-init to: Migration to travis-ci.com in progress, expect some CI weirdness | pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting June 16 16:15 UTC | 20.1 (Feb 18) | 20.2 (Apr 28) | https://bugs.launchpad.net/cloud-init/+filebug
<Odd_Bloke> powersj: Thanks for the help on Friday!  Full disclosure, I'd forgotten I didn't have admin on the new cloud-init repo, which is why you get ping-and-run'd. :p
<Odd_Bloke> falcojr: Hmm, looks to me like we didn't upload the most recent version to the PPA. :/
<Odd_Bloke> lucasmoura: We are happy for the schema behaviour to not be present on xenial.  (We don't introduce new dependencies in stable releases without substantial justification, and xenial released before the schema code landed in cloud-init.)
<lucasmoura> Odd_Bloke, ack
<Odd_Bloke> rharper: You should probably sign the CLA too. :p
<Odd_Bloke> powersj: GH/Travis are very confused about your README PR, I think we may need to throw that one away and start again.
<Odd_Bloke> OK, I think it might have been confused because the commit you created had state attached to it in a weird way; I just `commit --amend`ed it locally, and it's now only showing a single in-progress Travis check.
<rharper> Odd_Bloke: ack
<Odd_Bloke> paride: I've just sent an email to the cloud-init ML to answer your "Mhh, how is it that Travis CI didn't run on this PR?" Q. :)
<Odd_Bloke> (And have already applied the prescribed fix in this case. :)
<paride> Odd_Bloke, great, thanks
<powersj> Odd_Bloke, based on your email I assume we are in a happy state now?
<Odd_Bloke> powersj: Yep, thanks!
<powersj> sweet
<smoser> I'd like to have someone take a look at https://github.com/canonical/cloud-init/pull/416 sooner than later.
<smoser> for obvious reasons, re-basing or merging is probably going to be a pain again.
<blackboxsw> lucasmoura: thanks for the update on the SRU
<blackboxsw> https://github.com/cloud-init/ubuntu-sru/pull/99#pullrequestreview-426391003
<blackboxsw> smoser: grabbing it now
<blackboxsw> lucasmoura: just a minor set of fixes on your PR #99, then we'll land it
<lucasmoura> blackboxsw, Thanks for the review. I have updated the code
<blackboxsw> smoser: review posted. thanks!
<smoser> blackboxsw: responded
<blackboxsw> lucasmoura: mergedhttps://github.com/cloud-init/ubuntu-sru/pull/99
<blackboxsw> checking smoser
<blackboxsw> +1 smoser
<blackboxsw> launching a container now to double check smoser and will merge
<blackboxsw> and we can follow up w/ separate pr for runparts
<smoser> is_exe doesn't seem unreasonable as a method exposed from a 'subp' module.
<smoser> but since we have no other callers, your argument seems sane.
<blackboxsw> yeah I kinda figured the same. I was checking for any  other users of X_OK... hrm looks like cloud-init/cloudinit/config/cc_chef.py
<blackboxsw> could use that too.
<blackboxsw> smoser: ok maybe we leave it public
<blackboxsw> smoser: sorry bout that. and we can fold in cc_chef  to use subp.is_exe?
<blackboxsw> or separate PR
<blackboxsw> ?
<smoser> so you're
<smoser> oops.
<smoser> um... i dont really feel strong.
<smoser> i can do it. or not.
<blackboxsw> ok let's leave it public, as I can imagine other callers using subp.is_exe in the future
<smoser> ok.
<blackboxsw> and chef could be adapted to it with a separate pr
<blackboxsw> or this one if you want
<smoser> ok. back to public.
<blackboxsw> woo hoo. and around we go
<blackboxsw> going through a deployment now
<smoser> blackboxsw: can you land? thanks for review.  and very fast review of 148 files ;)
<blackboxsw> smoser: just checking the a set of chef changes for temporary file handling that appears to be in your branch. I think it's just locally when I diff against master. but I'm double checking before merging (as I'm hovering over the merge button)_
<blackboxsw> yeah part of your PR. ok and documented. n/m
<blackboxsw> smoser: merged thx
<smoser> i moved subp_blob_in_tempfile into cc_chef
<smoser> as it was the only caller of that
<smoser> and that method used util.writefilei and subp
<smoser> so which woudl have caused recursive import
<smoser> that was just the easiest way out since only one caller
<meena> blackboxsw: https://github.com/canonical/cloud-init/pull/416#issuecomment-640032968 any comment on (my comment)?
<blackboxsw> meena: heh, I saw that happy face on your comment and interpreted as a joke given the complexity of the cloudinit.distro.net refactor :/
<blackboxsw> meena: specifically what do you see in that which would be significantly different on BSD*?
<blackboxsw> or rather, across each linux distro
<meena> well, right now, we my handle enabling and starting services on BSD, in modules, where we do it on Linux,it's all duplicated code
<meena> i actually thought, for puppet, we could use puppet to do it for us platform independently :D
<meena> (and probably the same for chef, but i don't know chef, so i can't speak for chef)
<blackboxsw> heh, right....   just get people to swallow the big puppet config mgmt solution pill.   There goes our boot time :/
<meena> all you'd do is puppet apply, not the full thing with server and agent and database
<meena> (in fact, despite being a Big Puppet Cheer Girl, i don't use the server/agent setup at all)
<meena> but, yeah, running ruby from python just to execute service { puppet: ensure => running, enabled => true } is overkill,overly specialised to that one module,and implementing very basic service handling in cloud-init would benefit more services
<blackboxsw> meena: yeah I'm a puppet fan too.  Thanks for more context. I was misreading your PR comment related to the move of subp and didn't realize you were actually talking about the init services. right, those *could/should* be distro-specialized as there is a very different approach on many distros
<blackboxsw>  sorry you were talking about a separate refactor. and I *think* that would make sense to generalize that service enable/chaining/start etc. right now the jinja templates for that and the config templates are getting a bit busy, which indicates that we may need to develop a  better way to deal with it
 * blackboxsw tries to wrap up all the discussion you good folks have been having on the cloudinit.(distro).net refactor per https://github.com/canonical/cloud-init/pull/391/files
<blackboxsw> Odd_Bloke: looks really good https://github.com/canonical/cloud-init/pull/391/files initial questions and I can followup
<blackboxsw> smoser: cloudinit/subp.py:377:17: F821 undefined name 'logexc'  on https://github.com/canonical/cloud-init/pull/420
<smoser> blackboxsw: i don tknow what to do about that
<smoser> https://paste.ubuntu.com/p/37rbHM6XQq/ ?
<blackboxsw> yeah smoser that looks good
<blackboxsw> don't want to pull in util.logexc
<blackboxsw> smoser: have you seen this before? I now see two travis CI  builds queued for your PR and I can't trash the first one https://github.com/canonical/cloud-init/pull/419
<blackboxsw> it's preventing me from merging via github UX
<smoser> blackboxsw: no
<smoser> not seen nit
<meena> y'all ever gone a day of using computers without reporting at least one bug?
<powersj> blackboxsw, see Odd_Bloke's email
<powersj> re: transition to travis-ci.com
* Odd_Bloke changed the topic of #cloud-init to: Migration to travis-ci.com in progress, see https://lists.launchpad.net/cloud-init/msg00284.html | pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting June 16 16:15 UTC | 20.1 (Feb 18) | 20.2 (Apr 28) | https://bugs.launchpad.net/cloud-init/+filebug
<blackboxsw> ahh oops right. I had triggered rebuild, but that didn't quite clear it up. though smoser just put that PR up within the last hour, so I don't that qualifies as a PR generated before the move
<Odd_Bloke> blackboxsw: Huh, yeah, that is weird.
<Odd_Bloke> Particularly as both checks are pointing at the same Travis build ID. :/
<blackboxsw> smoser: is it worth, closing and reopening you RP?
<smoser> meena: when i left Canonical, i looked at how many bugs i'd opened.  I think it was 1 every 3 days for 10  years (including weekends/holidays)
<blackboxsw> PR rather https://github.com/canonical/cloud-init/pull/419
<smoser> blackboxsw: thats fine. i'll just re-submit
<blackboxsw> yeah something is amiss there as smoser's other PRs from today were running CI fine
<blackboxsw> thx
<meena> smoser: you'd think it'd get better, eventually?? huh?
<smoser> i dont recall the number that were still open ;)
<blackboxsw> good a happy single CI run in progress on https://github.com/canonical/cloud-init/pull/421
<meena> let's not try to look too closely
<meena> I'm going to look very closely into sleep now
<blackboxsw> falcojr: lucasmoura Odd_Bloke I uploaded 20.2.45 to https://launchpad.net/~cloud-init-dev/+archive/ubuntu/proposed/+packages I think the build finally completed in that PPA as I now see the right version present in apt policy
<blackboxsw> so we should be able to use that PPA for testing
<Odd_Bloke> https://github.com/canonical/cloud-init/pull/414 <-- this has two non-committer +1s, if a committer wants to take a quick look
<blackboxsw> lucasmoura: could nits on https://github.com/cloud-init/ubuntu-sru/pull/103/files
<Odd_Bloke> blackboxsw: Thanks!  (And thanks to James and Paride for the earlier reviews!)
<lucasmoura> blackboxsw, Thanks for the review. I have updated the code
<blackboxsw> lucasmoura: a small response on renaming terminate_instance function. and +1
<lucasmoura> blackboxsw, done
<blackboxsw> lucasmoura: merged. also with trello cards if you ctrl-c on the PR url and ctrl-v while highlighting the card, it attaches the PR link to the card top-level so that trello github automation will surface whether the PR is merged
<blackboxsw> lucasmoura: per this trello card https://trello.com/c/clHH7aDU/1680-add-code-to-terminate-ec2-instance  it looks like you attach links in a comment, which doesn't surface github PR status on the card
<blackboxsw> awesome looks fixed now
<blackboxsw> then I can see a "1 merged" icon on the card without opening it (as well as
<blackboxsw> ci results too)
<lucasmoura> blackboxsw, Cool feature :) I will use on my next cards. Thanks for pointing that out
<blackboxsw> thanks
<blackboxsw> lucasmoura: for tomorrow https://github.com/canonical/cloud-init/pull/390/files review done
<blackboxsw> for ec2 mirror disabling
<lucasmoura> blackboxsw, ack
#cloud-init 2020-06-09
<aqualung> Hello - which is the best module to create an empty directory on filesystem?
<meena> aqualung: probably runcmd https://cloudinit.readthedocs.io/en/latest/topics/modules.html#runcmd - that should be very easy to express in sh.
<aqualung> meena: That's how I've been doing it, was wondering if there was a better way.
<blackboxsw> Odd_Bloke: falcojr lucasmoura ok cloud-init is officially accepted into the Ubuntu $release-proposed queues
<blackboxsw> I can now send out the SRU  call for testing email
<falcojr> sweet
<lucasmoura> great
<blackboxsw> so no more ppa:cloud-init-dev/proposed needed
<blackboxsw> and our goal is to clear all testing in 7 days if we can. I'm starting to chop away at SRU tasks now
<blackboxsw> merged https://github.com/canonical/cloud-init/pull/423 thanks falcojr and candlerb
<falcojr> thanks!
<blackboxsw> falcojr: or lucasmoura: just put up SRU tooling work item review (which is a 'one-time' trello card on our SRU board) https://github.com/canonical/uss-tableflip/pull/53
<blackboxsw> minor script cleanup. I'm now grabbing Azure cloud manual verification test
<meena> https://github.com/canonical/cloud-init/pull/422/files/0c345d445d4aef54faa18af9f97290f6f5808ab9..7027a42e661c936582808c32be97c818c7058c8b what really annoys me about this view (usually followed from a notification) on github is that it's showing a single commit, but not the commit message
<Odd_Bloke> Yeah, it so unhelpful.
<meena> in general, the disconnect between reviews and commit messages feels jarring (coming from subversion and reviews over mailing lists, of all things, where people would quite frequently remark on, or demand changes to the commit message)
<meena> it's been ten years, and i still feel most Web based VCS forges are just barely as good as commit emails to a mailing list
<meena> \/rant
<meena> aha.
<meena> 23:39 <meena> \/rant â¬ï¸ i made a mathematical symbol, but i forgot what it means. is it all rants? or some rant
<mruffell> Odd_Bloke, rharper: Just wanted to say thank you for your help with cc_grub_dpkg last week, thanks!
<mruffell> I will go and test the packages in -proposed now, to double check we are good to go
<rharper> mruffell: sure
#cloud-init 2020-06-10
<Odd_Bloke> smoser: I wonder if you have any thoughts on https://github.com/canonical/cloud-init/pull/369?  I don't love the idea of pushing more info into the log which will be unused in almost every situation.  I am considering suggesting writing out to a separate file (perhaps in /run/?) to avoid this.  What do you think?
<Odd_Bloke> (Asking because I know you have Opinions about our logging. ^_^)
<Odd_Bloke> mruffell: Of course, thank you for your work!
<Odd_Bloke> meena: Being able to run CI is a big step up from commit emails IMO, but I agree that a lot of the actual review functionality isn't much different (there's only so many different ways to comment on patches that are ultimately just text file, lol).
<lucasmoura> Hey everyone, does anyone have an idea to manually test this feature ? https://git.launchpad.net/cloud-init/commit/?id=87cd040e
<lucasmoura> My understanding on it is that we must fake the endpoint to return 403. But I could not find a way to properly test this
<Odd_Bloke> lucasmoura: I don't see why you would need 403 for that commit (it looks like it should affect all interactions with the AWS IMDS, to me), is that definitely the one you meant to link to?
<lucasmoura> Odd_Bloke, Nope, my bad here. This is the right commit https://git.launchpad.net/cloud-init/commit/?id=1f860e5a
<Odd_Bloke> lucasmoura: Right, that one makes more sense! :p  Yeah, I think all we can test is the happy path (i.e. we still come up on AWS); we do sometimes ask the clouds to test changes that require messing with their IMDS, so we might be able to do that in this case.
<Odd_Bloke> blackboxsw: Can you add anything about asking clouds to test changes that require IMDS manipulation?
<smoser> Odd_Bloke: i had a partial comment from a week ago or so saying something to that effect.
<smoser> its very easy to have a knee jerk reaction that the log should have shown you exactly the information that you found would have helped out in this very rare scenario
<smoser> i've recently come more and more to agree with https://dave.cheney.net/2015/11/05/lets-talk-about-logging
<smoser> i think the real solution is
<smoser> a.) cloud-init log a lot less by default
<smoser> b.) easily turn on debug logging, which is then a firehose
<Odd_Bloke> Yeah, agreed, I like that as a long-term goal.
<Odd_Bloke> smoser: So is your position that we shouldn't include this change at all because it's log spam, or that we should put it in a separate file, (or that we should allow it as-is, because it's a drop in the logging pond as things stand)?
<smoser> i'd say its log spam
<smoser> (which ... 90% of cloud-init log is, so its a hard position to hold)
<smoser> hindsight is 20/20
<smoser> but i like the separate-file even less than i like the log spam
<smoser> really we need to be able to enable debug logs easily
<Odd_Bloke> OK, fair enough; I was at least thinking that that would go away across boots, but it's not great either.
<Odd_Bloke> So I wonder if the thing to do is to accept this one as-is, and then come up with a Plan For Logging which we can add to HACKING.rst and then expect future submissions to follow?
<smoser> its hard to tell someone "no that can't go in the log". when there is so much crap in the log.
<Odd_Bloke> Definitely.
<smoser> so, mostly i agree with you. let this in, and then future policy.
<smoser> but people have wanted to (and have) put stuff in the log as a result of kernel bugs in one release or uninitalized memory usage in a subprogram.
<smoser> stuff that is so very unlikely to ever have another 'hit' on usefullness
<Odd_Bloke> So I guess, at a minimum, we need to reach an agreement on what log levels should be used (and for what), and determine the mechanism(s?) by which you can opt into the firehose of debug messages.
<Odd_Bloke> (I think we want users to be able to opt-in without image modification, for example, else there'll be a lot of bugs that we can't get enough info on.)
<smoser> yeah. :-(
<smoser> which leads you to "log a ton of non-important information by default in the event that sometime it might be useful"
<smoser> :-(
<smoser> i've said this before, but as i do things now, i'm much more strict than i was when cloud-init started.
<smoser> fail loudly and exit failure.
<smoser> because cloud-init's behavior of "well... stuff might have gone wrong, check the log for WARN messages"
<smoser> results in missing *more* bugs
<Odd_Bloke> Yeah, there's definitely a fine line to balance between "break very obviously" and "do enough that people can at least access the instance to debug".
<Odd_Bloke> Because if you don't do the latter, then you _need_ to log a bunch of stuff to the serial console to be sure that people will be able to debug the problem well enough.
<blackboxsw> falcojr: thanks for the note at standup. I'll scrub your sru verification results now too
<falcojr> cool, thanks
<blackboxsw> Odd_Bloke: lucasmoura agreed we have sometimes asked for assistance from the specific cloud author on certain functionality that we land. In the case of https://git.launchpad.net/cloud-init/commit/?id=1f860e5a the author was fred-lefebvre. i have in the past emailed microsoft folks notifying them that their patch landed and is in testing and that it could be validated by following
<blackboxsw> https://cloudinit.readthedocs.io/en/latest/topics/debugging.html?highlight=proposed#manual-sru-verification-procedure
<blackboxsw> I mentioned microsoft, but the same applies to aws. If we can notify them of the change in -proposed. they can get a chance to test.  We could choose to highlight @fred-lefebrvre in a comment on the merged PR https://github.com/canonical/cloud-init/pull/216 that it is available for testing (then we are pretty sure the author gets a notification of this)
<falcojr> python 3.5 is cool
<falcojr> https://imgur.com/a/DalMKsc
<Odd_Bloke> https://www.youtube.com/watch?v=3epfRPCtGJA
<falcojr> it's mad about an unprintable unicode character
<falcojr> if I try to print the variable in middle log line I get
<falcojr> UnicodeEncodeError: 'ascii' codec can't encode character '\u03b5' in position 74: ordinal not in range(128)
<falcojr> not sure why it's trying to encode ascii when system locale stuff all says utf-8, and if I open a python shell manually I can print it fine
<falcojr> only happens on xenial though
<Odd_Bloke> Is it because it's writing to a file that's opened with ascii encoding?
<punkgeek> Is there any way to detect os (linux or windows) in cloud-init?
<powersj> punkgeek, hey - cloud-init does not support Windows at this time. Inside cloud-init we do have various methods for determining which Linux or bsd distro we might be running on
<blackboxsw> punkgeek: for windows you'd probably be using cloudbase-init instead of cloud-init as they are separate projects.  But in cloud-init upstream let's you detect what linux/bsd distribution you are running on via `cloud-init query distro`   or in jinja templates in your #cloud-config userdata file
<powersj> ^ nice
<blackboxsw> punkgeek: https://cloudinit.readthedocs.io/en/latest/topics/instancedata.html#using-instance-data shows examples of providing #template: jinja\n#cloud-config
<blackboxsw> in that you can use {{ v1.distro }}      or even the short alias {{ distro }}
<meena> cool
<blackboxsw> yeah, `cloud-init query --all` will give you a list of any keys which could be provided in #template: jinja\n#cloud-config user-data
<punkgeek> Aha thank you, what differences between cloudinit and cloudbase?
<blackboxsw> punkgeek: my understanding is https://github.com/cloudbase/cloudbase-init   is windows-only and only supports a subset of the config modules that cloud-init upstream supports https://cloudinit.readthedocs.io/en/latest/topics/modules.html
<blackboxsw> this channel is related to upstream cloud-init, which as powersj mentioned. does not support windows.
<blackboxsw> from their splashscreen https://cloudbase.it/cloudbase-init/  it looks like they have plans to support bsd at some point, but generally very windows focused.
<blackboxsw> + looks like they support 4 datasources(cloud platforms) instead of upstream cloud-init's 21 https://cloudinit.readthedocs.io/en/latest/topics/datasources.html
<punkgeek> Thank you so much
<blackboxsw> no worries
<blackboxsw> falcojr: thanks for https://github.com/cloud-init/ubuntu-sru/pull/107#pullrequestreview-428284569   2 minor comments and we can land it
<punkgeek> I want to deploy a shell script into the machine that does not have internet connectio, what is the best way? Here is my shell shell script: https://github.com/autovmnet/tools/blob/master/vm_config.sh
<lucasmoura> Hi everyone, I am trying to manually test this PR https://github.com/canonical/cloud-init/pull/234, but I am still experiencing issues with it, so I think I am missing something
<lucasmoura> I am trying to replicate the exact same case that rharper used in the discussion
<lucasmoura> Although I can see the error he describe when running cloud-init, I see another error when updating for the fixed version
<lucasmoura> It states the following error: Exec format error. Missing #! in script?'
<lucasmoura> Which makes sense, sice the example in the PR is not a shell script per se. So does the example in this PR supposed to work ? The one rharper uses to reproduce the error with a lxc container
<meena> Odd_Bloke: you should rebase https://github.com/canonical/cloud-init/pull/391 andâ¦ we should merge it, or i should start sending you patches
<Odd_Bloke> meena: There are some comments on there that I need to read through and address, and unfortunately it's dropped down my list (for now). :(
<rharper> lucasmoura: here, what's not working ?
<rharper> lucasmoura: if you have a newer cloud-init with the fix, when cloud-init processes the mime-type of the payload (which is a cloud-config, not a shell-script) then it will get merged correctly into user-data, *instead* of being thought as a shell-script; which as you see, it isn't  and it cannot be executed
<rharper> lucasmoura: for verifying that;  can you reproduce the failure with the steps in https://github.com/canonical/cloud-init/pull/234#issuecomment-604033345  ?
<lucasmoura> rharper, yes, I can reproduce the error perfectly. But the problem is that an error still happens when I try to run the same user-data on the newer cloud-init version
<lucasmoura> Just give me a couple of minutes and I will add the script I am using and the error I am receiving
<lucasmoura> I think it will be easier to explain the issue
<rharper> lucasmoura: are you using the lxc-proposed-snapshot to create a new image with the updated cloud-init  ?
<lucasmoura> No, but I am manually installing the new version in the lxc container. First I reproduce the error, than I manually add the ppa where the newer cloudinit version is and try to run it again to see if no error is raised
<rharper> and you run cloud-init clean --logs --reboot ?
<lucasmoura> Oh no, I just run cloud-init init
<rharper> you need to clean
<rharper> otherwise it's not a "first boot" any more
<rharper> so the already parsed user-data is written out as a shell-script and runparts will see the old file
<rharper> you can skip the reboot
<rharper> but you do need clean
<lucasmoura> That makes total sense
<rharper> and I suggest; cloud-init clean --logs && cloud-init init --local && cloud-init init
<powersj> https://cloudinit.readthedocs.io/en/latest/topics/faq.html#how-can-i-re-run-datasource-detection-and-cloud-init
<lucasmoura> Let me update the script
<rharper> the safer way is to always use the lxc-proposed-snapshot;  and run a completely new container
<lucasmoura> thanks powersj
<lucasmoura> rharper, do you mean reproduce the error first and then launch a container with lxc-proposed-snapshot to verify the fix ?
<rharper> lucasmoura: yeah; https://github.com/cloud-init/ubuntu-sru/pull/100/files
<rharper> in there, I have a recreate()  and then a verify()
<lucasmoura> rharper, cool. Thanks for the suggestion. I will start using it :)
<haybill> I am having a cloud-init/netplan networking problem with Ubuntu 20.04.  Cloud-init runs with local cloud-init datasource and sets up the networking configuration (static IP, vlan, etc) without any errors.  Networking is configured but not active after cloud-init completes.  I did notice that cloud-init finds all the Ethernet links down when it runs (Up column in Net device info).  I have discovered that doing a 'netplan apply'
<haybill> after cloud-init has run will made the network configuration active.  The Ethernet HW in question is a Intel 10Gb ixgbe interface.  Is this expected behavior? Is my cloud-init config missing something?
<rharper> haybill: not expected; couple of things 1) if you can, share you net config, ensuring you've a match on the ixgbe nic in your config to ensure that it's brought up  2) there was a netplan bug around bringing things online without an applyl; though it was related to wifi; might be related;  are you using -daily images or the released image (which will have -updates available) ?
<haybill> rharper, #2 I am using released 20.04 with a fresh apt update && apt upgrade.  Should I be trying something newer/etc?
<rharper> haybill: no, I just new there was a release to -updates for netplan for the wifi issue;
<haybill> rharper, I am having trouble getting you a sample networking config right now, but when I deploy 18.04 with the same tools networking is fine
<rharper> haybill: you have a server install? or using a cloud-image ?
<blackboxsw> followup on https://github.com/cloud-init/ubuntu-sru/pull/105#pullrequestreview-428359896 for you falcojr
<blackboxsw> sorry lucasmoura missed the conversation
<blackboxsw> thx rharper
<rharper> blackboxsw: yw
<haybill> rharper, it is a Dockerfile generated image (like https://github.com/packethost/packet-images).  Based on this conversation I am beginning to be worried that I am missing some new 20.04 packages for networking
<rharper> haybill: ok, are you booting a container ? and passing in the nic ?  or booting a vm ?
<rharper> either case, I would suggest testing with the Ubuntu cloud images, https://cloud-images.ubuntu.com/daily/server/focal/current/    there are VM images and root.tar.gz for containers ;  testing with this can help you track down if it's related to your image build or some other issue (or cloud-init/netplan bug)
<haybill> rharper, thx I will look into the cloud images
<rharper> sure
<lucasmoura> blackboxsw, no worries
<lucasmoura> rharper, powersj thanks for the help, just confirming that the proposed fix solved the issue
<rharper> lucasmoura: nice!
<blackboxsw> lucasmoura: ec2 sru pr reviewed https://github.com/cloud-init/ubuntu-sru/pull/102/files#
<lucasmoura> blackboxsw, ack
<blackboxsw> lucasmoura: another for you https://github.com/cloud-init/ubuntu-sru/pull/106/files#
<blackboxsw> https://github.com/cloud-init/ubuntu-sru/pull/107 merged
<blackboxsw> lucasmoura: same minor change request on https://github.com/cloud-init/ubuntu-sru/pull/108/files#
<blackboxsw> and we'll merge both
<falcojr> blackboxsw: thanks for the reviews! I followed up to your followup on https://github.com/cloud-init/ubuntu-sru/pull/105
<blackboxsw> np and strange falcojr ... I thought my eyes were seeing exactly the opposite (no system_info via cloud-config).  I'll double check again. I probably botched something
<falcojr> it's possible I'm doing something crazy too! :D
 * blackboxsw doesn't mean falcojr is strange... I wouldn't say that out loud.... erm, I mean.
 * blackboxsw walks away
<falcojr> I won't deny being strange ;)
<blackboxsw> haha
<blackboxsw> wow falcojr ok, I swear I didn't see the Resolving logs before on my side,
<blackboxsw> ok. I'm rerunning. and I think I'll take the debt of documenting the system_info via cloud-config example in cloudinit.readthedocs.io . so I actually remember that
<blackboxsw> I can see the pre-upgrade 'failure' case now
<blackboxsw> and verifying the fix per your script
<blackboxsw> thx
<blackboxsw> falcojr: ok +1 on your branch if we can add some valid URL can verify that we see the Resolving http://valid.com in logs and not foo.com
<blackboxsw> https://github.com/cloud-init/ubuntu-sru/pull/110 merged
#cloud-init 2020-06-11
<punkgeek> How cloud-init iso can automatically detect the network config version to apply static ip address on the vm?
<punkgeek>  How cloud-init iso can automatically detect the network config version to apply static ip address on the vm?
<andras-kovacs> Hi! Is there any official way to use cloud-init with VMWare? I know it sounds funny.
<rick_h>  andras-kovacs http://darrylcauldwell.com/vmware-cloud-init/ seems like a starter guide with some helpful links
<andras-kovacs> thank you!
<blackboxsw> lucasmoura: and falcojr it seems we get merge conflicts on most of our ubuntu-sru Prs trying to link or update the doc links from the readme https://github.com/cloud-init/ubuntu-sru/tree/master/20200529
<blackboxsw> what do you guys think about us just dropping that linking and we can separately merge a top-level readme update later so we don't have to rebase each PR when another verification test lands
<blackboxsw> I'd even be game the commiter (who clicks the merge button) separately just pushing a commit to update the docs after it lands
<blackboxsw> as it seems painful to constantly rebase -i master and fix a trivial conflict
<blackboxsw> what do you both think?
<lucasmoura> blackboxsw, I agree. I will remove the links on my PRs now
<falcojr> blackboxsw: if you approve, I'm fine doing the fix and merge myself...pretty I have the permissions on that repo
<falcojr> also, my network took a dump for a minute or two, so sorry if I missed a message in there
<falcojr> s/pretty/pretty sure
<falcojr> a doc after the fact works too
<falcojr> whatever's easiest for everyone else I guess
<lucasmoura> blackboxsw, I have removed the links on my PRs
<falcojr> ok, I'll stop adding the links then and we can do it all at once at the end
<blackboxsw> +1 falcojr and lucasmoura I think any of us should feel free to review/merge other's PRs on this
<blackboxsw> ok sounds good. let's do it all at the end
<blackboxsw> conflict ...resolved
<meena> falcojr: try: "pretty, and sure" â that works for me.
<blackboxsw> hah
<falcojr> nice
<blackboxsw> lucasmoura: did you git add your ec2 verification results to https://www.gnu.org/software/grub/manual/legacy/grub.html#Filesystem?
<blackboxsw> lucasmoura: did you git add your ec2 verification results to https://github.com/cloud-init/ubuntu-sru/pull/102/files?
<blackboxsw> I think we want the entire console output from the test runs on the PR in ec2-sru-20.2.45.txt
<lucasmoura> blackboxsw, Not yet to both questions
<lucasmoura> Oh sorry, I will add it now
<blackboxsw> thanks
<lucasmoura> Is there a specific folder I should put that on ?
<blackboxsw> falcojr: approved however you want to resolve the merge conflicts is good w/ me https://github.com/cloud-init/ubuntu-sru/pull/105#pullrequestreview-429154846
<blackboxsw> merged lucasmoura https://github.com/cloud-init/ubuntu-sru/pull/106
<lucasmoura> blackboxsw, just realized I should put it in the ec2 file. Updating the file right now
<blackboxsw> I'll stop pinging with each merge. I'm getting a bit noisy here.
<blackboxsw> thx lucasmoura
<lucasmoura> blackboxsw, updated
<blackboxsw> lucasmoura: clear for attaching those logs to https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1881018
<ubot5> Ubuntu bug 1881018 in cloud-init (Ubuntu) "sru cloud-init (19.4.33 to 20.2-45) Xenial, Bionic, Eoan and Focal" [Undecided,In progress]
<lucasmoura> blackboxsw, ack
<blackboxsw> lucasmoura: when you do, please also update the description of the bug "* Manual Test Results" section replacing the TODO with the name of the log file you are attaching
<blackboxsw> goal is for us to replace all those TODOs in the descr with verification logs
<lucasmoura> okay
<lucasmoura> blackboxsw, launchpad bug updated
<falcojr> anybody have experience lauching a VM with cloud-init through lxd?
<Odd_Bloke> I haven't yet launch a LXD VM, I'm afraid.
<blackboxsw> and falcojr yep, let me get you the PR. :) you need a profile
<falcojr> ahhh, ok. I was able to get it started but my cloud init data wasn't passed through
<blackboxsw> falcojr: https://github.com/canonical/ubuntu-advantage-client/pull/1090/files#diff-ade24dcbd00eae69e6f292b18c4ec67aR109-R128
<falcojr> thanks!
<blackboxsw> hrm, checking the actual question you had. I'll try providing cloud-config to that to see if it works
<falcojr> well via google I found an example of somebody using cloud init data, but they created a profile too
<blackboxsw> falcojr: ok I did the following lxc launch ubuntu-daily:focal -c user.user-data="$(cat cc.yaml)" asdf-f2 --vm --profile behave-vm   and it worked.
<blackboxsw> falcojr: here's what my profile looks like https://github.com/canonical/ubuntu-advantage-client/pull/1090/files#diff-ade24dcbd00eae69e6f292b18c4ec67aR109-R128
<blackboxsw> oops https://paste.ubuntu.com/p/yq3JMVy2Dr/
<falcojr> ahh, right, that source
<falcojr> awesome, thanks!
<blackboxsw> sorry falcojr that link doesn't take you to the right place, but basically look for lxc_create_vm_profile
<blackboxsw> .... .and if you are reviewing it ;).... j/k :)
<blackboxsw> I'll put ua-client reviews on lucas :)
<falcojr> yeah, I got it, thanks again
<lucasmoura> Hey everyone, when I set a cloud-config directive through the environment variable DEBUG_PROC_CMDLINE, how can I verify that it was actually used ?
<blackboxsw> smoser: rharper do you have any objection if we hoist the remaining scripts out of https://github.com/cloud-init/qa-scripts and into uss-tableflip?
<rharper> blackboxsw: one place is better than two
<lucasmoura> Originally I thought about looking at both cloud-init.log and cloud-init-output.log, but the output is not appearing in those logs
 * blackboxsw thinks you still have access to that repo
<rharper> blackboxsw: tableflip ?
<rharper> blackboxsw: yeah, I think so
<rharper> typically I'd expect ya'll to land to tableflip though
<Sargun> Is there any way to do variable interpolation on #include directives?
<rharper> Sargun: you can use a jinja template in your #cloud-config file
<Sargun> rharper: Cool. Thanks.
<rharper> Sargun: see cloud-init query and cloud-init devel render
<rharper> Sargun: https://paste.ubuntu.com/p/GNx8VVyQVk/
<rharper> cloud-init query --list-keys will show you the supported jinja variables
<blackboxsw> yeah rharper lucas just mentioned yesterday we had a number of dup scripts which we hadn't retired from qa-scripts
<blackboxsw> powersj: same question on qa-scripts repo in cloud-init
<blackboxsw> we were thinking of migrating remaining scripts out of there into https://github.com/canonical/uss-tableflip/
<powersj> blackboxsw, do it
<Sargun> This is super neat: include http://myhost/{{ v1.distro }}-instance-{{ instance_id }}/my-user-data
<blackboxsw> yeah it's great. anything you can do in jinja templates are viable options . conditionals loops variable replacement etc.
<blackboxsw> plus the benefit of sourcing all cloud-init query instance data as variables
#cloud-init 2020-06-12
<punkgeek> How can I detect network configuration version in cloud init automatically? because I want to write a general script for Linux distros.
<Odd_Bloke> punkgeek: What do you mean by "network configuration version"?  Could you perhaps describe your problem/requirement in a little more detail?
<smoser> blackboxsw: i'm not really sure why there was a difference.  I think powersj might have some insight.  Or such knowledge may well be lost in the recesses of past brains.
<smoser> but i think that at one point a jenkins had access to one but not the other or something.
<punkgeek> Odd_Bloke: There are 2 kinds of network type which you can see in here:  https://cloudinit.readthedocs.io/en/latest/topics/network-config-format-v1.html    and  https://cloudinit.readthedocs.io/en/latest/topics/network-config-format-v2.html
<punkgeek> Some distoros such as ubuntu 16 use the first version but in ubuntu 18 use the second one (netplan)
<punkgeek> I want to make a script that can detect the OS network type and config it automatically. For example if the Distro support netplan, Use the second network type
<Odd_Bloke> punkgeek: Have you seen the `cloud-init devel net-convert` command?
<punkgeek> Odd_Bloke: This python file? https://github.com/delphix/cloud-init/blob/master/cloudinit/cmd/devel/net_convert.py
<punkgeek> I couldn't understand this python script
<Odd_Bloke> punkgeek: I guess I'm still not sure I understand where this "script" will run, so I'm guessing at what you're trying to do a little.
<smoser> punkgeek: "Some distoros such as ubuntu 16 use the first version but in ubuntu 18 use the second one (netplan)"
<smoser> the point of cloud-init is that it can render either network config version into the correct system configuration format.
<smoser> `cloud-init devel net-convert` will read either version 1 or version 2 network config
<smoser> and write sysconfig (rhel, suse), ENI (old ubuntu), netplan (new ubuntu)
<smoser> cloud-init *is* the general script
<punkgeek> Aha, So how can I use cloud-init devel net-convert in the iso file?
<punkgeek> I should use it by runcmd command?
<punkgeek> Here is a shell script that I used to config static ip in different distros: https://github.com/autovmnet/tools/blob/master/vm_config.sh   But I want to do it by cloud-init iso
<Odd_Bloke> punkgeek: When you say "cloud-init ISO", what do you mean?
<punkgeek> Sorry i'm amature in cloud-init. Here is my cloud-init script example that i've convert it to iso file by using cloud-localds and then mount it on the vm   https://github.com/autovmnet/tools/blob/master/cloud-init.cfg
<Odd_Bloke> punkgeek: OK, great; so if you run `cloud-localds`, you should see that it has a `--network-config` option.  If you use that to specify network configuration (in either v1 or v2 formats), then cloud-init will take care of rendering it appropriately for your target distro version.
<Odd_Bloke> (And no worries, there's a lot of similarly named things, so I'm just making sure we're on the same page so I'm not giving you bad advice!)
<punkgeek> Thank you, As I understand, in the cloud-localds option, I should create two config file, cloud-init.cfg and network.cfg then convert it to iso file like this:   cloud-localds --network-config=network.cfg new.iso cloud-init.cfg
<punkgeek> Am I right?
<Odd_Bloke> punkgeek: Yep, I believe that will do it.
<punkgeek> How can I do it in a single file?
<punkgeek> Odd_Bloke: And is there any way to enter both version configuration and then, system detect version automatically and use the currect one?
<smoser> punkgeek: it doesn't matter which you pick
<smoser> cloud-init will render either version to the correct result when configuring networking.
<smoser> you declare it however you want.
<punkgeek> smoser: Thank you, so is it correct to do sth like this?:  create a config.cfg which contain: https://paste.ubuntu.com/p/rXhfRXfWPz/   and then run cloud-localds --network-config=cloud.cfg new.iso
<smoser> no
<punkgeek> how should it be? Could you please give me an example to do it?
<punkgeek> Should it seprate the network part and user part? or require anything else?
<smoser> https://asciinema.org/a/145772
<punkgeek> moser: what about this one? moser>
<punkgeek> https://paste.ubuntu.com/p/NTRXsk3Rx6/
<smoser> no.
<smoser> thats not valid yaml
<smoser> well.. it miht be valid yaml, but its not sensical.
<smoser> you have 2 top level network kesy.
<smoser> only put one of the two versions in .
<smoser> i recommend following the demo and changing one thing at a time.
<smoser> all the files for the demo are at https://gist.github.com/smoser/635897f845f7cb56c0a7ac3018a4f476
<punkgeek> The point  what  I want to enter both network-config is that because my platform cannot detect the vm distors.
<Odd_Bloke> punkgeek: cloud-init can detect the VM distro (because it runs within the VM), and it will convert your input to the appropriate format for that distro.
<Odd_Bloke> So if it's running on xenial, it will convert v2->ENI, on bionic and later v2->netplan, etc.
<punkgeek> Odd_Bloke: So is it ok to enter both version like this? https://paste.ubuntu.com/p/NTRXsk3Rx6/
<Odd_Bloke> punkgeek: No.
<Odd_Bloke> You specify your configuration once, and cloud-init converts that configuration to the appropriate format.
<punkgeek> Odd_Bloke: sorry I couldn't understand, could you give me an example?
<Odd_Bloke> (In this case, the second `network` key definition overrides the first, so you are actually only defining a v2 config.)
<Odd_Bloke> punkgeek: https://paste.ubuntu.com/p/VSrFPTzFdz/ <-- that's all you need to specify, cloud-init will take care of converting it to the required format within the VM
<punkgeek> Aha so if i enter this one and the distro does not support netplan such as ubuntu16.04, It will convert to the version one?
<dking> I would like some help troubleshooting my network configuration. I see that my network-scripts were set as expected, but they don't seem to be applied. It seems like it fell back to DHCP for some reason. I'm looking through cloud-init.log now, but I'm not sure what I'm looking for yet.
<dking> I see this in the logs: "No network config applied. Neither a new instance nor datasource network update on 'System boot' event"
<Odd_Bloke> punkgeek: Correct!  (In fact it will convert it to ENI, which is the /etc/network/interfaces format used on xenial; v1 is only used internal to cloud-init.  But your broader understanding is correct.)
<dking> I got the same when I ran "cloud-init clean -r". I'm new to debugging cloud-init issues, so any help would be appreciated.
<smoser> dking: well, you can open a bug. but i'd first explain what you're tryhing to do and paste an entire cloud-init.log .
<punkgeek> Odd_Bloke: Sorry but I've got this error:   https://paste.ubuntu.com/p/B8B6MTvrKP/
<smoser> cloud-init collect-logs will collect a lot of information that people would probably ask you
<smoser> punkgeek: drop the top level 'network'
<smoser> i think newer versions of cloud-init will figure that out, but the bug you're seeing
<smoser> the bug you're seeing is that cloud-init looks for a top level 'version' key in its network config, and you have no such key. (because you have a second level network key)
<smoser> https://gist.github.com/smoser/635897f845f7cb56c0a7ac3018a4f476
<smoser> those files are valid... if you start there you might have more success.
<punkgeek> smoser: like that? https://paste.ubuntu.com/p/8VTMTqJhXR/
<punkgeek> Aha sorry
<smoser> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1798117 intended to support your input
<ubot5> Ubuntu bug 1798117 in juju "juju sends "network" top level key to user.network-config in lxd containers" [High,Fix released]
<smoser> so i'd supsect that the guest that is there does not have that fix.
<dking> smoser: I'm not convinced that it's a bug yet. I'm trying to just get an idea of why things didn't happen as I expected. Here's the cloud-init.log: http://sprunge.us/oqGOws  And here is some context: http://paste.openstack.org/show/794705/
<dking> ...and I forgot to put it in there, but in case it helps, I'm using cloud-init version 18.5
<meena> what is delphix and how do i run it myself?
<smoser> dking: good job collecting info
<smoser> you ahve 2 cloud-init runs in the log there
<smoser> maybe try a 'cloud-init --logs clean'
<smoser> just to reduce it to one boot
<smoser> but i suspect that what is wrong is that the /etc/sycconfig is 'diry" in some way
<smoser> not certain. but it surely looks good.
<dking> Okay. I'll give that a try. That's probably from where I ran "cloud-init clean -r"
<smoser> where did you get the image ?
<dking> Okay, that may be. I had an issue when it first booted caused by the BIOS.
<dking> It's generated by bifrost. This is part of a Kayobe deploy.
<smoser> if we think that cloud-init rendered the correct sysyconfig network information
<smoser> then there are a few things that might be wrong
<dking> Okay, I ran that (actually "cloud-init clean --logs" as the other way gave an error), and now the logs are empty.
<punkgeek> smoser: Sorry I have this error now; https://paste.ubuntu.com/p/KNhDJHPRQb/
<smoser> a.) cloud-init rendered this network information *after* the system brought up networking (we try to ensure that 'cloud-init init-local' runs before system network bring up . but something might be broken there.
<smoser> b.) udev rules might not be written correctly (they handle the device renaming... not exactly sure about that for sysconfig though)
<smoser> c.) other confusing sysconfig data conflicting iwth cloud-init
<smoser> punkgeek: you gave it invalid yaml
<smoser> it can't load it.
<smoser> you can test yaml loading using yaml-dump https://gist.github.com/smoser/280dbed3e9582c53c6035525f35814aa
<smoser> sorry, though. i really ahve to get to actual work now.
<dking> Yes, they appear to be correct to me, though I could have missed something. Is there any way to tell from the logs where things go wrong? The logs are cleaned now. What is the best way to cleanly run it again? Just reboot the machine, or something like "cloud-init init" ?
<powersj> dking, sudo cloud-init --local; sudo cloud-init init; or a reboot as you suggested. https://cloudinit.readthedocs.io/en/latest/topics/faq.html#how-can-i-re-run-datasource-detection-and-cloud-init
<dking> powersj: Thank you very much
<dking> smoser: Here's the new log: http://sprunge.us/q5KrHr and here is further context, including the error I got when attempting to bring up the interface: http://paste.openstack.org/show/794706/
<smoser> dking: init doesnt render networking
<smoser> init --local
<dking> OKay. I'll try that.
<dking> smoser: Here's the new cloud-init.log: http://sprunge.us/Scz96q
<smoser> do you maybe need some vlan spuport package in the image ?
<smoser> i think that maas testing actually uses vlan and centos . so i think this is a supported path
<punkgeek> I use this script but still does not works:( https://paste.ubuntu.com/p/xh25XGpYR7/
<dking> I'm not sure. I've been using the same system to deploy servers for a while, but I did make some changes to configurations before this run, and it's happened on two different machines.
<blackboxsw> falcojr: and Odd_Bloke https://github.com/canonical/cloud-init/pull/431  daily recipe fix for bionic
<dking> If the image isn't built correctly, or if there's some problem outside of cloud-init, I'll just need to know enough to locate the bug in the other software. And I have been using VLANs before. I think, though, that we had to use a certain version of cloud-init as the older ones couldn't support them. I think it was around 18.2?
<blackboxsw> falcojr: Odd_Bloke and https://github.com/canonical/cloud-init/pull/433 for xenial
<dking> Is there any way to confirm from the cloud-init.log whether the issue I'm seeing in bringing up eht0.81 manually is actually the reason why the network isn't setup by cloud-init?
<smoser> punkgeek: you'd have to post a cluod-init log at this point.
<smoser> dking: there should be some system logs for sysconfig somewhere
<smoser> but i'm not familiar enough to know
<dking> smoser: I'll continue trying to resolve that. But what I meant was, is there some place in the cloud-init logs where it would show whether it even attempted to bring up that interface, or if it received an error?
<blackboxsw> dking: I see in your log that "2020-06-12 16:44:41,585 - DataSourceConfigDrive.py[DEBUG]: network config provided via network_json"
<blackboxsw> dking: and you can see cloud-init tries to render this network config in your logs
<blackboxsw> 2020-06-12 16:44:41,598 - stages.py[INFO]: Applying network configuration from ds bringup=False: {'version': 1, 'config': [{'type': 'vlan', 'subnets': [{'dns_nameservers': ['8.8.8.8', '8.8.4.4'], 'netmask': '255.255.255.0', 'routes': [{'gateway': '10.8.1.3', 'netmask': '0.0.0.0', 'network': '0.0.0.0'}], 'type': 'static', 'address': '10.8.1.11', 'ipv4': True}], 'name': 'eth0.81', 'vlan_id': '81', 'mac_address':
<blackboxsw> 'ac:1f:6b:d0:4d:94', 'vlan_link': 'eth0'}, {'mtu': '1500', 'type': 'physical', 'subnets': [], 'mac_address': 'ac:1f:6b:d0:4d:94', 'name': 'eth0'}, {'address': '8.8.8.8', 'type': 'nameserver'}, {'address': '8.8.4.4', 'type': 'nameserver'}]}
<blackboxsw> 2020-06-12 16:44:41,598 - __init__.py[DEBUG]: Selected renderer 'sysconfig' from priority list: None
<blackboxsw> saying that it is using the proper sysconfig renderer for the network config it writes. (which aligns with you saying the valid config appears to be in /etc/sysconfig)
<blackboxsw> might want to check timestamps of 2020-06-12 16:44:41,598 versus you system logs that say when networking is brought up
<blackboxsw> to see if that happening in cloud-init 'after' your network was already up (so the updated config wasn't read by your system)
<blackboxsw> you might be able to look through output of `journalctl --utc -o short-precise -u systemd-networkd`   on your system maybe ?
 * blackboxsw was just grasping at straws though.
<punkgeek> smose: Here is the log  https://paste.ubuntu.com/p/mZM3XbVKcx/
<Odd_Bloke> blackboxsw: Shouldn't those PRs be for the daily branch, as discussed?
<blackboxsw> Odd_Bloke: ohh whoops, force of habit. I was thinking we'd do new-upstream-snapshot to refresh patches and then run fix-daily-branch script to do that. I forgot we can just fix daily only
<blackboxsw> right right will fix them
<Odd_Bloke> Nice, thanks!
<blackboxsw> hrm. thinking it through though.
<blackboxsw> so if we only fix ubuntu/daily/xenial we'll lose that next time we run cloud-init's "cherry-pick" as 'fix-daily-branch' does a git reset --hard origin/ubuntu/$release
<blackboxsw> so somehow we might need to keep a log of quilt refreshes applied in ubuntu/daily/$release to 'fix' non-cpick quilt patches.
<Odd_Bloke> I think fix-daily-branch should abort if it finds state other than what it expects.
<blackboxsw> so we can re-apply them during the next "fix-daily-branch"
<blackboxsw> yeah I think so, I can fold that into a PR I'm working on fix-daily-branch anyway as I ran into an issue with ubuntu/$releases which have no debian/patches/cpick*
<blackboxsw> so an abort and we can sort on next cherry-pick
<Odd_Bloke> I would suggest it'll be easier to review if you separate them out instead of folding it in.
<Odd_Bloke> But other than that, sounds good!
<blackboxsw> ok will do.
<Odd_Bloke> Also worth noting that the current reset --hard behaviour will just break daily builds again in a way that we already know how to solve, so it's not necessarily the worst situation.
<dking> blackboxsw: Thank you very much for the info. I am starting to wonder now if the problem might be NetworkManager. In the past, I think I've had trouble with it using VLANs.
<smoser> dking: cloud-init doesn't attempt to bring up networking information
<smoser> it just renders it and lets the OS bring it up
<smoser> the goal is that your OS just thinks that was "statically configured" networking that was already there.
<blackboxsw> Odd_Bloke: falcojr https://github.com/canonical/uss-tableflip/pull/54
<blackboxsw> handle the no debian/patches/*cpick* case on existing ubuntu/xenial or ubuntu/bionic branches (since we had just performed a new-upstream-snapshot which dropped those cpicks)
<dking> Yep. So, I suppose that if it got the scripts into the right place, then maybe the issue is with something else in the OS.
<smoser> punkgeek: you'd need to give /var/log/cloud-init.log . not ust the console log.  but i suspect that the issue is that the nic name is not what you expected. ens192
<punkgeek> smoser: can I use it like ens* ?
<smoser> punkgeek: probably not reliably. you may be able to for systems that use netplan natively.
<smoser> but really what you need to do is specify the mac addresses
<smoser> mac addresses are something you can know from "outside".  they're properties of virtual hardware.  the names that a given OS chooses to give to that virtual hardware are not reliable
<blackboxsw> Odd_Bloke: let's try that again https://github.com/canonical/cloud-init/pull/435 to fix ubuntu/daily/xenial builds
<blackboxsw> and actually looks like I'll have to rework another patch :/ n/m Odd_Bloke    moving which into the subp module also broke another quilt patch
<blackboxsw> as unit tests now fail
<blackboxsw> after patches are applied
 * blackboxsw moves it to wip
<Odd_Bloke> blackboxsw: Ack, ping when you want eyes on it again.
<dking> smoser and blackboxsw: Thanks for the help. It helped me to narrow things down. The problem does seem to be from using NetworkManager. If I switch to network-scripts, I can start the VLAN. Now, I just have to find a way to get DIB to build using network-scripts instead of NetworkManager.
<Odd_Bloke> Looks like Rick didn't get to my doc PR before the trails called to him, so if anyone wants to do a quick review: https://github.com/canonical/cloud-init/pull/430
<smoser> dking, blackboxsw it probably would be good for cloud-init to document *somewhere* that network manager is not supported at this time.
<smoser> and would be better if it noticed somehow that your system was using network manager and said unsupported
<blackboxsw> +1 Odd_Bloke
<smoser> or alternatively, actually DTRT and rendered network manager.
<blackboxsw> smoser: I guess https://cloudinit.readthedocs.io/en/latest/topics/network-config.html#network-configuration-outputs is a bit misleading
<blackboxsw> as we specifically call out NetworkManager for netplan backend
<portdirect> how can I configure cloud-init to use Netplan when ifupdown is also present on the host?
<blackboxsw> portdirect: by default in Ubuntu focal and later netplan is preferred over /etc/network/interfaces even if both ifupdown and netplan are present
<strobelight> is there a doc that lists all the recognized keys and describes them?
<blackboxsw> portdirect: on earlier releases of cloud-init you can set the renderer priority order to prefer netplan first
<portdirect> blackboxsw: im not experiencing this, its prioritizing ifupdown over Netplan for me
<portdirect> a simple apt-get purge ifupdown gets Netplan working, but this is not viable for my use case :(
<blackboxsw> portdirect: by either:  1. providing the following config in /etc/cloud-config.d/90-override-renderer.cfg in your image: https://paste.ubuntu.com/p/ky7V4RvdXT/
<portdirect> blackboxsw: trying that now
<blackboxsw> portdirect: or 2 providing cloud-config to override this on VM boot (if you aren't creating your own images)  https://paste.ubuntu.com/p/f68TbMJRdb/
<blackboxsw> strobelight: do you mean instance data keys?
<blackboxsw> or network renderer keys?
<blackboxsw> I'm missing the context of your question
<blackboxsw> falcojr: sorry about https://github.com/canonical/cloud-init/pull/431   I should have closed it when Odd_Bloke reminded me we wanted to do this against ubuntu/daily/xenial|bionic only (as that is the whole reason we have those branches now; to avoid patch commit/refresh noise in ubuntu/xenial|bionic proper)(
<blackboxsw> falcojr: https://github.com/canonical/cloud-init/pull/435 should have more context, but the short of it is that we expect to be dropped into a quilt patch apply failure cli where we need to run quilt push -f, resolve patch apply conflicts manually, quilt refresh to save those updated patches into debian/patches/*.patch
<falcojr> gotcha
* Odd_Bloke changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting June 16 16:15 UTC | 20.1 (Feb 18) | 20.2 (Apr 28) | https://bugs.launchpad.net/cloud-init/+filebug
<blackboxsw> falcojr: also added some more context for you on this ubuntu-advantage-tools pain https://github.com/canonical/cloud-init/pull/431
<blackboxsw> https://github.com/canonical/cloud-init/pull/431#issuecomment-643460234 actually
<Odd_Bloke> rharper: smoser: Your input on https://bugs.launchpad.net/cloud-init/+bug/1883122 would be really appreciated. :)
<ubot5> Ubuntu bug 1883122 in cloud-init "`cloud-init status` should distinguish between "permanently disabled" and "disabled for this boot"" [Undecided,New]
<blackboxsw> the reason we are in a bit of a pickle on ubuntu-advantage in upstream cloud-init is because the ua-tools project accepted backwards incompatibility of the ua-client CLI  and cloud-init also was allowed to design a new set of config directives without worrying about backwards compatibility of the old shell-based ua-tools CLI versus to new ua-tools python-based CLI. As a result we've carried a bit of a patch
<blackboxsw> maintenance pain in ubuntu/xenial and ubuntu/bionic branches
 * blackboxsw can't wait to release ubuntu-advantage-tools version 25.0  to xenial and bionic so we can drop the ubuntu-advantage revert stuff
<portdirect> blackboxsw: that worked perfectly, thank you - though I put the over-ride in /etc/cloud/cloud.cfg.d/90-override-renderer.cfg
<portdirect> cheers
<blackboxsw> cool portdirect good to hear
<dking> NetworkManager is probably fine for most people. I think it's the VLANs. And it may not even be just that. I'm wondering if there could be another solution, like adding a package.
<falcojr> blackboxsw thanks for the extra context
<rharper> Odd_Bloke: looking
<blackboxsw> https://github.com/canonical/cloud-init/pull/435 and https://github.com/canonical/cloud-init/pull/436 up for daily build recipe fixes
<blackboxsw> with happy instuctions :)
<strobelight> @blackboxsw user-data and instance meta data. https://cloudinit.readthedocs.io/en/latest/topics/examples.html shows examples, but is that all-inclusive? https://cloudinit.readthedocs.io/en/latest/topics/instancedata.html instance data seems to be vendor-specific, or deployment specific, for example "availability_zone" means nothing when deploying ubuntu on my virtualbox.
<blackboxsw> strobelight: yes definitely each platform supports differing values for instance data. so your mileage varies depending on where you are deploying. https://cloudinit.readthedocs.io/en/latest/topics/instancedata.html is the best bet and running `cloud-init query --all` on your platform is the real content you can rely on.
<blackboxsw> strobelight: also I put together a couple of sample runs on some platforrms a while back: https://hackmd.io/aAJFerDRRNaoL0Q_vDRYBg  it's a bit dated, but there ya go
<strobelight> @blackboxsw thanks, I'll have a look
#cloud-init 2020-06-14
<bswinnerton> Hello, first time playing around with cloud init. Things are going great in Ubuntu, but having some problems configuring networking in Debian. I'm using a nocloud database with a v2 network configuration file to set a static ip address. The user-data and meta-data is applied successfully, but after the machine comes up, it's falling back to DHCP.
<bswinnerton> Upon further inspection, this is because /etc/network/interfaces does a source-directory for both /etc/network/interfaces.d/ (which has the correctly configured file), and /run/network/interfaces.d/ which has a file that conflicts and falls back to DHCP. Is there a way to turn off the latter?
<bswinnerton> s/database/datastore
<bswinnerton> hmm, there's also something weird going on similar to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=867921
<ubot5> Debian bug 867921 in openstack-debian-images "cloud-init: Network configuration doesn't work" [Normal,Fixed]
<bswinnerton> So to recap, when using nocloud with a v2 network config that sets a static ip address, when the Debian host comes up, a few things need to happen in order for the machine to get networking: 1) `rm /run/network/interfaces.d/50-cloud-init.cfg` (which includes the DHCP setting), and 2) `mv /etc/network/interfaces.d/50-cloud-init.cfg
<bswinnerton> /etc/network/interfaces.d/50-cloud-init`, once those have been completed, a `systemctl restart networking` can be run and everything works fine.
<bswinnerton> This is cloud-init 18.3
<meena> bswinnerton: have you tried a more recent version to see if this is fixed?
<bswinnerton> I haven't, I'm using what's built into the debian cloud images
<bswinnerton> In what scenarios would cloud-init write to `/run/network/interfaces.d/`? Is that the fallback described in https://cloudinit.readthedocs.io/en/latest/topics/network-config.html#fallback-network-configuration?
<meena> I'm trying to remember if we have deb packages pre-built anyhere
<meena> my brain isn't working very well today, so you might be quicker looking yourself
<bswinnerton> https://packages.debian.org/buster/cloud-init
<bswinnerton> They're quite old, which is why the debian images include them, I assume
<meena> no backports?
<bswinnerton> ah yes: https://packages.debian.org/source/buster-backports/cloud-init
<meena> there's https://launchpad.net/ubuntu/+source/cloud-init but i don't know how well those will work on Debian
<meena> bswinnerton: so let's try at version, and if that still fails, we'll file a bug report
<meena> and by we, i mean, can you please
<meena> s/at /that /
<bswinnerton> Sounds good, installing 20.2 now
<bswinnerton> Is there an effective way to test this with cloud-init on the guest without rebuilding the image?
<bswinnerton> Or does the image need to be rebuilt?
<meena> install, run cloud-init clean --logs --reboot
<bswinnerton> Neat, wasn't aware of `clean`. Doing that now
<meena> that will regenerate everything on the server, including ssh server keys
<bswinnerton> Sigh, that fixed it
<bswinnerton> Okay, off to pester the Debian folks. Thanks meena
<meena> ð½
<meena> if i really were useful, I'd help you pinpoint where that was fixed, exactly
<meena> but screw that, let them upgrade to the latest version :P
<bswinnerton> A quick clarifying question, even after bumping to 20.2 I notice a very long delay when trying to bring up the interface, it appears that it's trying to get a DHCP lease with "Started ifup for ens3". It then eventually times out and continues along, and sets the static IP. Any idea how I could tell it not to try and DHCP? Is that the difference
<bswinnerton> between a dsstore of local vs net?
<bswinnerton> I have a feeling it was this: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=867921
<ubot5> Debian bug 867921 in openstack-debian-images "cloud-init: Network configuration doesn't work" [Normal,Fixed]
<bswinnerton> But the dates don't quite align
<meena> I'd have to see your full config to figure that out
<meena> maybe there's a v1 config sent with dhcp?
<bswinnerton> It's using v2 and no shouldn't be DHCP, but let me throw it in a gist for a second set of eyes
<meena> cloud-init query
<meena> something or the other
<bswinnerton> Here's the user-data, meta-data, network-config that's being thrown into the nocloud datastore, as well as `cloud-init query -a`: https://gist.github.com/bswinnerton/86e2e44d4dee4b5271be92580afd056c
<bswinnerton> Started a conversation with the debian cloud folks here: https://lists.debian.org/debian-cloud/2020/06/msg00073.html
<bswinnerton> Answered my own question: it's not related to the local vs net dsstore option of cloud-localds. It still tries to get a lease on boot even with the value set to local
<bswinnerton> I bet it's because of the dhcp config that lives in /run/network/interfaces.d/
<bswinnerton> meena: any idea where files in that directory come from? Is that part of cloud-init?
<bswinnerton> Oh yeah, that's definitely it. Removing the file leads to super quick boot times
<bswinnerton> God what a mess
<meena> that a patch from Debian?
<bswinnerton> https://github.com/canonical/cloud-init/commit/a6faf3acef02bd8cd4d46ac9efeebf24b3f21d81.patch
<bswinnerton> That's the first problem, the second is the fact that a file gets added to `/run/network/interfaces.d/` that defaults to DHCP
<bswinnerton> And I'm not sure where that comes from, but am working on terrible hacks with user-data to get rid of it
<meena> logs should be helpful
<meena> maybe
<bswinnerton> Not seeing any references to that directory in the log, so it must be coming from the Debian cloud image
<bswinnerton> Is runcmd the best hook for modifying network configurations early in the boot process? It only needs to run once
<meena> network is usually the first thing to be run, so there's almost no chance of doing anything before that dhcp thing
<meena> unless
<meena> https://cloudinit.readthedocs.io/en/latest/topics/modules.html#bootcmd
<meena> bswinnerton: this could be worth a shot, if you Anna rm that file, so you don't have to modify the image :D
<meena> wanna
<meena> i am now second stage tired / exhausted.
<meena> Good evening.
