#cloud-init 2014-05-12
<harlowja> SpamapS where u staying?
<SpamapS> harlowja: Westin
<mlei> i'm having some trouble getting cloud-init to find my nocloud cloud-init data on cdrom
<mlei> i've doublechecked that the volid is cidata and that it's available at /dev/sr0
<mlei> cloud-init.log shows blkid executions that seem to not be returning the correct error code
<mlei> but when i run blkid manually i don't see anything odd
<mlei> any tips on troubleshooting this?
<dgarstang> Anyone awake?
#cloud-init 2015-05-11
<mgrigaitis> my cloud-final.service timeouts on debian jessie. Any ideas how to set timeout to bigger value?
<mgrigaitis> or remove it?
<harlowja> smoser yo
#cloud-init 2015-05-12
<smoser> hey. harlowja 
<harlowja> hey yo
<harlowja> when u want to go over https://openstacksummitmay2015vancouver.sched.org/event/b52d0c99f65d7713c61524b5a380812b
<harlowja> with vice executives
<smoser> supyeah.
<smoser> i do.
<smoser> i had on my todo to bother you about tha ttoday
<smoser> thank yo ufor beating me to it.
<smoser> can we pick a time in my normal working hours tomorrow ?
<smoser> :)
<arnaud_orange> hi guys
<arnaud_orange> does anyone know which cloud-init configuration I should use to disable pollinate to seed random ?
<smoser> arnaud_orange, 
<smoser> # pollinate hangs without socket timeout if it can't reach network
<smoser> random_seed:
<smoser>    command: null
<smoser> will do it.
<smoser> you can make 'command' anything you want.  null is yaml's "None"
<arnaud_orange> oh nice
<arnaud_orange> let me try that
<smoser> but you can make it [/bin/sh, -c, 'echo hi mom']
<arnaud_orange> Is there anything else to disable when my VM does not have access to internet, because cloud-init is taking very much time to bring my VM up 
<arnaud_orange> it's far better with your random_seed configuration, thanks smoser
<arnaud_orange> but still taken 178sec to be up
<smoser> what datasourc e?
<arnaud_orange> openstack metadata
<smoser> are you able to pastebin a /var/log/cloud-init.log ?
<arnaud_orange> smoser: http://paste.ubuntu.com/11096632/
<smoser> arnaud_orange, it would appear you lose almost 2 minutes between lines 186 and 187.
<arnaud_orange> yep
<smoser> i would suggest disabling DataSourceGCE in /etc/cloud/cloud.cfg.d/90_dpkg.cfg
<arnaud_orange> I think this is because the VM is not able to access http://metadata.google.internal./computeMetadata/v1/ is not resolvable
<smoser> right.
<arnaud_orange> what is that?
<smoser> inside the image you have to change that.
<arnaud_orange> GCE=GoogleCloudengine, right?
<smoser> right.
<smoser> remove GCE there.
<smoser> please file a bug, as such a hang is not intended.
<arnaud_orange> ok, if I do that, I won't be able to rely on the classic ubuntu cloud image
<smoser> yeah, that sucks. :-(
<smoser> file a bug, we can fix it.
<arnaud_orange> yep
<smoser> and SRU and then you can be happy again.
<arnaud_orange> no way to fix it with yaml cloud-config?
<smoser> no. 
<smoser> as this is config that is read before we can get to user-data
<smoser> ie, it tells us how to find user-data. :)
<arnaud_orange> understand
<smoser> you could try to see why dns resolution would time out on that.
<smoser> that does seem broken that a dns resolve would block for 120 seconds
<smoser> the only networking provided at that point is from the dhcp server
<smoser> so your dhcp server is possibly giving a dns server that is not reachable.
<arnaud_orange> the dns server configured on the vm is 8.8.8.8 (google)
<arnaud_orange> but the VM does not have access to internet
<smoser> well, that'd seem like a misconfiguration :)
<smoser> why is it 8.8.8.8 ?
<arnaud_orange> this is configured on my tenant network
<arnaud_orange> because, when I set a floating IP to my VM, the VM can access internet
<arnaud_orange> but the floating IP is not set on startup
<arnaud_orange> it's floating between VM 
<arnaud_orange> don't know if I am clear on that
<smoser> right. but it seems like your network is busted still.
<smoser> why give someone a dns server that they cannot reach
<smoser> i guess so that later when they get a floating ip they can.
<arnaud_orange> don't know, ask my cloud provider :p
<smoser> who is cloud provider ?
<arnaud_orange> cloudwatt
<arnaud_orange> french company
<arnaud_orange> but I think it might be the same on any openstack based cloud provider
<smoser> no
<smoser> thats definitely their configuration. and they could definitely give you access to a dns server internally that would dtrt.
<smoser> i'd bug them about it. 
<arnaud_orange> the thing is that I only want my virtual machine to have access to other machines in the tenant, without access to the internet, so not having access a an external DNS is expected
<smoser> i'd file a bug with them.
<smoser> a.) telling you dns is 8.8.8.8 is hacky at best (rather than a dns server they run)
<smoser> b.) telling you a dns server is <ANYTHING> that you can't get to is kind of silly/broken
<arnaud_orange> ok, I found a way to remove the 8.8.8.8, so a) is not an issue anymore, 
<arnaud_orange> they provide their own nameserver
<arnaud_orange> but, again, my tenant router have only one interface, in the private network
<arnaud_orange> so without floating IP, no internet access
<arnaud_orange> there is no way to disable every internet request from cloud-init point of view?
<arnaud_orange> without modifying the ubuntu image,
<arnaud_orange> ?
<smoser> well, nto really.
<smoser> but if you have dns configured correctly, then the dns should fial
<smoser> and it will move on
<smoser> quickly
<arnaud_orange> you're right
<devicenull> 2015-05-12 17:06:07,091 - util.py[WARNING]: Failed to create user foobar
<devicenull> would be nice to know why :(
<devicenull> oh, I guess adduser failed for some reason
<smoser> devicenull, /var/log/cloud-init-output.log maybe
<devicenull> yea, that's where I got that error message
<smoser> not cloud-init.log
<smoser> cloud-init-output.log
<devicenull> # less /var/log/cloud-init-output.log
<devicenull> yes, that's where that message was!
<smoser> well thats not right. 
<devicenull> turns out it didn't like the 'primary-group' setting I had
<devicenull> when the primary-group didn't exist
<smoser> log should not be going to -output.log
<smoser> oh wait.
<smoser> or i guess *its* output, and on warn, it wrote that to stdout
#cloud-init 2015-05-13
<smoser> claudiupopa, https://review.openstack.org/#/c/182764/
<smoser> or harlowja 
<smoser> and finally done.
<smoser> i hope, with such license nonsense.
<harlowja> more license fun :-P
<smoser> i think we're good now.
<smoser> and i like the tiny file header too
<harlowja> :-P
<harlowja> man i worked on cloud-init in 2012
<harlowja> i'm old
<harlowja> lol
<harlowja> back when i was a weee baby
<smoser> you are old, harlow.
<harlowja> lol
<harlowja> smoser thx dawg
<smoser> is that how they spelled dog when you were in school ?
<harlowja> lol
<harlowja> yup
<harlowja> in my gangster days
<smoser> bah
<smoser> http://logs.openstack.org/64/182764/1/check/gate-cloud-init-pep8/81fefcd/console.html
<smoser> harlowja, ^
<smoser> shoot. my fault. fix coming
<harlowja> 2 blank lines
<harlowja> haha
<harlowja> we can turn off some of that OCD stuff
<harlowja> if we want
<smoser> or smoser could just not be an idiot too.
<harlowja> lol
<tmclaugh[work]> Before I go down this road, what does cloud-config do with mime encoded data that is not text/cloud-config or text/x-shellscript?
<harlowja> some of it's just OCD imho
<harlowja> tmclaugh[work] many things
<tmclaugh[work]> I;m looking to reuse the user-data endpoint for another system in our environment
<harlowja> https://github.com/stackforge/cloud-init/blob/0.7.x/cloudinit/user_data.py#L98 can be walked through
<harlowja> thats the main mime stuff
<smoser> tmclaugh[work], ignores.
<tmclaugh[work]> okay, not sure if last comment went through due to internet issues hereâ¦
<smoser> tmclaugh[work], by design, so that you can acheive exactly that.
<harlowja> ya, i think its going to log a message though
<harlowja> https://github.com/stackforge/cloud-init/blob/0.7.x/cloudinit/handlers/__init__.py#L215 
<harlowja> and then leave it be
<tmclaugh[work]> alright, not sure if previous comment went through
<smoser> "I;m looking to reuse the user-data endpoint for another system in our environment" 
<smoser> thats last i saw of you
<tmclaugh[work]> ahh, yes, that was it
<tmclaugh[work]> That;s the last I saw too. :)
<smoser> so yeah, cloud-init will happily ignore stuff. 
<tmclaugh[work]> perfect
<smoser> it does write a wanring to console now... that says "don't knwo what to do with this"
<smoser> to that affect
<smoser> which could be considered obnoixious
<tmclaugh[work]> I need new instances to have some metadata that gets passed to Foreman for Puppet.
<smoser> theres other types, as harlowja pointed out, but if you dont collide, cloud-init will just ignore.
<tmclaugh[work]> didnât see what he said.
<smoser> https://github.com/stackforge/cloud-init/blob/0.7.x/cloudinit/user_data.py#L98
<tmclaugh[work]> awesome
<harlowja> smoser aweaome, more encoding issues http://logs.openstack.org/64/182764/2/check/gate-cloud-init-python34/fc3d2da/console.html 
<harlowja> lol
<harlowja> your favorites
#cloud-init 2015-05-14
<ramblurr> is there a way to enable/disable data sources without editing the OS image (i.e., by editing /etc/cloud/cloud.cfg)?
<ramblurr> i'm on a cloudstack infrastructure and a fedora atomic image takes quite a while to boot as it timesout waiting for the EC2 metadata service 
<ramblurr> here's a screenshot of the timeout http://i.imgur.com/omyM3qz.png
<rangerpb> hey folks, with cloud-init user data input, is there anyway to handle a value that has an ! in it without quoting ?
<smoser> ramblurr, theres not really a way to do that. i'm sorry.
<smoser> rangerpb, example ?
<rangerpb> well some folks were testing my plugin code and they had a password that started with an !
<rangerpb> if they quoted it, it was fine
<rangerpb> just checking really
<smoser> they were putting that in in cloud-config ? 
<smoser> yaml ?
<smoser> i can't control the yaml syntax really.
<smoser> but if cloud-init is invoking a shell process and not quoting correctly, then we can fix that.
<smoser> ramblurr, but if you got to that one (ec2) then everything else failed.
<smoser> its last for that reason
<ramblurr> but it did work
<ramblurr> i.e., i was able to login with my key injected by the cloudstack service
<ramblurr> perhaps it continued onwards to the ec2 source after?
<smoser> it shoudl not.
<smoser> you can paste /var/log/cloud-init.log
<smoser> harlowja, ping
<smoser> hey
<smoser> https://review.openstack.org/#/c/182764/
<harlowja> i merged!
<smoser> oh. it got merged.
<harlowja> sup
<harlowja> :)
<smoser> you have to push merge?
 * smoser is so lame
<harlowja> +2 and + approve
<smoser> ok. 
<smoser> i "fixed" that httpretty thing
<smoser> what apita
<harlowja> :-P
<smoser> Odd_Blok1, put commit message in https://code.launchpad.net/~daniel-thewatkins/cloud-init/walinux-wip/+merge/258644 and ill pull
#cloud-init 2015-05-15
<dbuechler> Good morning. Has anyone build cloud-init 0.7.7 for centos7? I have a question.
<dbuechler> I had tons of trouble getting the pre-built package (0.7.5) to read my meta-data correctly, so I opted to try 0.7.7.
<dbuechler> As far as I can tell, 0.7.7 works great, except that by default, the rpm built as sysvinit. Can I just edit the init-system line in cloud-init.spec and rebuild?
<dbuechler> (CentOS 7 is system)
<dbuechler> err systemd
<dbuechler> Damned autocorrect.
<smoser> Odd_Blok1, https://code.launchpad.net/~daniel-thewatkins/cloud-init/walinux-wip/+merge/258644
<smoser> you believe that to be ready, right?
<gamename> hi guys. I'm getting this error on my aws machines: 'May 15 15:59:42 ip-172-31-62-102 journal: [CLOUDINIT] util.py[WARNING]: Unable to change the ownership of /var/log/cloud-init.log to user syslog, group adm'
<gamename> How do I fix it? Or do I need to? 
<smoser> probalby not a big deal, bu tyou can fix it with config
<gamename> smoser: not sure what you mean by 'with config'. :)  
<smoser> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/examples/cloud-config.txt#L528
<smoser> you can change that to whatever you want. the default is syslog:adm.
<smoser> in 0.7.7 you can also have it as a list of 'user:group' strings. and it will try one after the other until one works.
<gamename> smoser: Thanks! I'll do that.
<dbuechler> Question: Can CentOS 7 be included in the rhel distro as using systemd?  The text in /etc/redhat-release is "CentOS Linux release 7.1.1503 (Core)"
<dbuechler> Ah.  Oops.  I have old code.
<dbuechler> I'll pull a fresh copy and recompile
<dbuechler> FYI: The oauthlib package in EPEL is python-oauthlib.  What's listed in brpm is python-oauth.  I had to make the change in brpm to get it to build correctly.
#cloud-init 2015-05-17
<smoser> dbuechler, i suspect we need these: http://paste.ubuntu.com/11176334/
<smoser> right ?
<smoser> please just confirm and maybe pester me with an email or submit a merge proposal and well get that fixed.
#cloud-init 2016-05-16
<larsks> harlowja or smoser: can someone confirm the minimum python version required by cloud-init?  It looks like the answer is "2.6" but want to be sure.
<smoser> larsks, harlowja says that 2.6 is busted in trunk.
<smoser> and he intendes to fix that. it shouldnt be much.
<smoser> i would like to support 2.6 but less than that i have no plan.
<larsks> Awesome, thanks.
<larsks> Someone is asking about cloud-init for RHEL 5 (python 2.4), and I wanted to be able say no with authority :)
<smoser> larsks, i think realistically speaking that requires some epel stuff to get python2.6
<smoser> its just *so* old
<larsks> Yeah, absolutely.
<Odd_Bloke> Ship a cloud-init package that bricks their machine; problem solved.
<Odd_Bloke> ^_^
<gamename> Hi guys. I'm trying to something simple.  I have a script which gets copied to the instance scripts as "part-001".  However, i need to test it by re-running init and that requires me to remove the "instance" dir where that file is now living. So, how does one test?
<gamename> Come to think of it, where does a script live before it gets copied to the instance?  That is, where does the script "foo.sh" which becomes "part-001" live before being copied to the instance subdir in /var/lib?
<Odd_Bloke> gamename: What cloud/data source are you using?
<gamename> Odd_Bloke I'm using terraform to copy the user-data.
<gamename> Initially copy it, that is.
<harlowja> larsks smoser https://code.launchpad.net/~harlowja/cloud-init/cloud-init-net-refactor has the 2.6 fixes
<harlowja> someone just needs to merge that :-P
<rharper> harlowja: hi;  so network_state isn't "designed" as I'm sure you can tell.  The constraints at the time were something roughly similar to the openstack network_metadata; but distinctly *not* network_metadata from openstack for uninteresting, yet important (internally at least) reasons.  I'm more than happy to look at modifying the internal structure to be something more useful for net-config <-> network-state <-> render
<rharper> ed-format conversions.  In my initial stab at netword output, I've already needed to do some changes when parsing to map device types to the various "network", "netdev", "links" concepts that networkd has that eni doesn't
<harlowja> rharper what were the 'yet important (internally at least) reasons.' ?
<harlowja> rharper did smoser talk to u about a tiny-lib for that network conversions stuffs
<smoser> fudge
<smoser> rharper, i need to talk to you :)
<nacc> heh
<rharper> harlowja: that it not _be_ the openstack format
<rharper> *sigh*
<rharper> smoser: here now if you want to do g+
<smoser> k. give me a bit.
<rharper> sure
<harlowja> if u need me on g+ just let me know :-P
<harlowja> rharper any more details ;)
<harlowja> spilllll the beans, lol
<rharper> harlowja: that said, I want to spend some time thinking about the attributes we'd like in that internal state, with an eye toward the various output formats we need to support;  something that's currently hard to do is things like networkd's  .network with a match rule which applies to any interface that matches;  that doesn't map cleanly to an inteface-name indexed state
<harlowja> ya, thus why i sorta think a tiny-lib could focus on that
<rharper> harlowja: hehe; I'm sure you can imaging the usual sort of reasons and one or more of those applies
<harlowja> lol
<rharper> I've not looked at all of the other config formats we support or network configs we 'write out,  sysconfig seems simple enough but I've not looked at it exhaustively;  cloud-init targets many other OS types, so it would be useful to capture their sorts of requirements and syntax;
<harlowja> ya, any freebsd folks around :-P
<rharper> I don't think we'd need to implement those backends (I'd prefer an interested party to contribute those)
<rharper> heeh
<harlowja> yup
<harlowja> i've volunteered for the sysconfig one (not hard, just gotta figure out the mapping)
<rharper> but I'd want to make sure our internal structure supported things;  IMO, the key items "devices/links", "networks/subnets",  and "matching/templating" covers I think most of what we'd need
<harlowja> likely involves knowing more about the sysconfig mapping than i currently know
<rharper> usually; I've become all-to-familiar with eni and it's helpers (bridge-utils/ifenslave/vconfig)
<harlowja> :-p
<harlowja> smoser rharper so what was the decision :-P
<copumpkin> what's the role of /var/lib/cloud/seed/nocloud/{user,meta,vendor}-data ?
<copumpkin> oh, I see
<harlowja> smoser https://code.launchpad.net/~harlowja/cloud-init/cloud-init-fix-test-times/+merge/294854 should have the fixed test times
<harlowja> retries and timeouts triggering it.
#cloud-init 2016-05-17
<ccard> Are there any tools for validating cloud-init syntax?
<ccard> I have seen examples using str_replace, where the params are given values using something like "__value__: { get_param: some_parameter }", but I'd like to do something like "__value__: { str_split: ['.', { get_param: some_parameter }, 0] }".
<ccard> However, when I try this I get the error "ERROR: Property error : cloud_init: cloud_config "str_replace" params must be strings or numbers"
<larsks> That looks like Heat template syntax, actually.  Cloud-init doesn't have functions like that.
<larsks> ccard: you probably want to ask over on #heat.
<ccard> larsks: yes, I realised that this morning :) It turns out I need OpenStack kilo at least :(
<larsks> Glad you're all set.  Or at least pointed in the right direction.
<ccard> larsks: another question: how do I get cloud-init to recognise a "network-interfaces" section, to define the network devices? I think I need to get it to use a DataSourceNoCloudNet data source, but my instance is only using a DataSourceConfigDriveNet data source.
<larsks> ccard: take a look at https://bugzilla.redhat.com/show_bug.cgi?id=1235602#c3
<smoser> ccard, its buggy.
<smoser> working on fixing
<mfisch> harlowja: smoser is launchpad still the right/current place for the cloud-init source tree
<mfisch> I'd like to make a couple doc fixes
<smoser> mfisch, yes
<mfisch> thx
<mfisch> lets see if I still remember how to use bzr ;)
<harlowja> smoser will add git eventually :-P
<smoser> yes.
<harlowja> reminds me, i gotta destroy the 0.7x branch on openstack/cloudinit
<mfisch> I even got it working
<mfisch> https://code.launchpad.net/~mfisch/cloud-init/doc-fixes/+merge/294976
<harlowja> woot :-P
<mfisch> I worked at C for 2 years but apparently forgot it all, had to google
<harlowja> :)
#cloud-init 2016-05-18
<ccard> larsks: I don't have permission to see that bug report :( "You are not authorized to access bug #1235602.
<ccard> Most likely the bug has been restricted for internal development processes and we cannot grant access."
<ccard> larsks: however, I can see https://bugzilla.redhat.com/show_bug.cgi?id=1288105 and https://bugzilla.redhat.com/show_bug.cgi?id=1288819. Those appear to be related to an interaction between NetworkManager and cloud-init on rhel 7.2, but I am using CentOS 6.3 which doesn't have NetworkManager as far as I can see.
<ccard> larsks: from looking at the cloud-init logs on the instance, it appears to be ignoring the network-interfaces section altogether.
<larsks> ccard: copied from that bz: https://gist.github.com/larsks/9065baaf59ceb2b5a811a9d59cf3caea
<ccard> larsks: thanks, that's very useful information
<gfidente> harlowja, you around?
<harlowja> gfidente i am
<gfidente> harlowja, hey I was wondering if you saw the updates for the hostname/fqdn proposal?
<harlowja> hmmm, unsure
<gfidente> harlowja, I mean this https://code.launchpad.net/~gfidente/cloud-init/try_fqdn_from_hosts/+merge/294680
<gfidente> it was missing unit tests
<harlowja> oh, i'll check
<gfidente> harlowja, addCleanup done
#cloud-init 2016-05-19
<harlowja_> smoser  ok https://code.launchpad.net/~harlowja/cloud-init/cloud-init-net-refactor/+merge/293957 should be in a decent shape to merge i think
<harlowja_> it isolates the cloudinit/net stuff a little better
<harlowja_> when that goes forward then i can start to put up the sysconfig stuff i've got
<harlowja_> which i just noticed i deleted, arg, lol
<harlowja_> nm, i got it back, phew
#cloud-init 2016-05-20
<gamename> Can cloud-init use python scripts?
<Odd_Bloke> gamename: What do you mean by "use"? :)
<gamename> Odd_Bloke ah, ok.  I mean, can a python script be used in place of, say, "instance/scripts/part-001" ?
<gamename> Odd_Bloke put another way, can "part-001" be a python script instead of a bash script?
<Odd_Bloke> gamename: Have you tried it? :p
<gamename> nevermind
<smoser> Odd_Bloke, /usr/bin/cloud-init: http://paste.ubuntu.com/16524951/
<smoser>  /run/cloud-init/main.log: http://paste.ubuntu.com/16524970/
<smoser> so something in 'import' from the stdlib is taking quite a while.
<Odd_Bloke> smoser: Hmm, interesting.
<Odd_Bloke> smoser: I assume you're going to find out which one it is next. :p
<smoser> Odd_Bloke, http://paste.ubuntu.com/16525270/
<smoser> tempfile
<Odd_Bloke> smoser: tempfile has a "from random import Random as _Random", perhaps it's that?
<Odd_Bloke> smoser: And that import will create an instance of Random() as well.
<smoser> Odd_Bloke, yeah, i think thats it. added that import specifically
<smoser> and did that first
<smoser> and that took 14 seconds
<smoser> so this really sucks, and i'd appreciate you driving a solution if you wanted :) but i think i need to be done with it at this point.
<Odd_Bloke> I'll try poking doko and pitti intermittently to get it done.
<Odd_Bloke> But we also don't really have the time. ;.;
#cloud-init 2016-05-21
<harlowja_> smoser rharper if u get some time https://gist.github.com/harlowja/d63a36de0b405d83be9bd3222a5454a7
<harlowja_> https://gist.github.com/harlowja/d63a36de0b405d83be9bd3222a5454a7#file-gistfile1-txt-L271 being the main complexity
<harlowja_> trying to understand the 3 different network configs that get mutated and passed around makes this a PITA, lol
<harlowja_> openstack network data --> some intermediate format --> some other final format
<harlowja_> == pain
<rharper> harlowja_: nice;  IMO, the complexity comes from having to understand the mapping between underlying devices and the upper layers;  I've not yet looked at the code in ifupdown2; but that cloudious ifupdown2 written in python claims to have a model that uses a graph for dependencies between interfaces;  they might have a cleaner internal state representation that we could map the various inputs (network-config.yaml, netw
<rharper> ork_data.json) and then it certainly handles writing out eni since it's an ifupdown replacement;  and I don't think networkd will be that hard after handling eni and sysconfig
#cloud-init 2017-05-15
<blackboxsw> powersj: I'm not quite understanding the need for ctest -- tree_collect or tree_run versus just collect and run
<blackboxsw> if you don't pass a --deb to tox -e citest -- run doesn't it just perform the existing run against the cloud-init in the current branch?
<blackboxsw> ... which seems to be the intent behing tree_run?
<powersj> if you use collect and run you use the cloud-init that comes with the image itself. If you use tree_* you are going to run using the local tree.
<blackboxsw> behind* rather
<blackboxsw> ohhh
<powersj> if you use --deb as you said, it will pull in that deb, but why force the developer to have a deb built locally.
<powersj> tree_* will go and build the deb for you so you don't have to have all the build deps and build env setup.
<blackboxsw> feels strange to me that citest -- run, since you have source locally in order to run these tests, wouldn't use the cloud-init tree as default test target
<blackboxsw> ahhh it's to avoid debs
<blackboxsw> ahhh it's to avoid deps
<blackboxsw> ok
<blackboxsw> ok I'm good now. thx
<blackboxsw> though it kinda feels like instead of tree_collect and tree_run we could just provide a   citest -- run --use-tree option right?
<blackboxsw> or is there more to it than that? it seems tree_(run|collect) is just a mirror of run|collect right?
<powersj> Well there are lots of options to go along with the building and I think this made the argparsing much easier
<blackboxsw> thanks. I'll keep firing questions as I go through this until I get the context.
<powersj> please do :)
<blackboxsw> powersj: do you know where the template_override verbs are used "when: - create\n-copy" etc that I see in platforms.yaml?
<blackboxsw> it seems template_overrides is only used on lxd-specific cases.
<blackboxsw> but I don't really know where the consumer of that template is
<powersj> so keep in mind we only have lxd as a platform, no other platforms are enabled right now
<powersj> as far as when, I don't know off the top of my head
<powersj> I'll try to look and see
<powersj> looks like images/lxd.py:update_templates
<blackboxsw> tests/cloud_tests/images/lxd.py yeah was just typing that too
<blackboxsw> thanks
<blackboxsw> ahh so these are lxd specifc metadata config options when building and configuring  containers
<powersj> yeah
<powersj> if I recall we added this to later add various platform specific configs/options down the road
<powersj> e.g. kvm with a couple storage devices or with multiple networks
<blackboxsw> powersj: so I'm going through tests/cloud_tests/releases.yaml  and looking over feature flags in general. Do you know where ubuntu_user or lsb_release feature actually  turns into some sort of feature setup in #cloud-config during these vm tests?
<powersj> blackboxsw: if you look at the test case yaml you will see the feature flags called out. The feature flags are not a cloud-init thing, but an integration test thing.
<blackboxsw> man josh, this is a beast.
<blackboxsw> 3700 lines. sorry it's taking me a while
<blackboxsw> sorry 3200 lines
<blackboxsw> adding another set of small nits
<powersj> blackboxsw: no worries ;0
<blackboxsw> :/ that I'm only 1/3 through it
<blackboxsw> I'm gaining knowledge though on how different parts interact... so not just a code review.
<blackboxsw> so, powersj feature flags are just arbitrary strings that some unit tests claim to require and some distros and series claim to support per releases.yaml?
<powersj> bingo
<blackboxsw> it's basically a set of bools that each release claims to support. man I thought there was more to it.
<blackboxsw> ok, so is it kindof out goal to make tests flexible enough to avoid having to exlude it based on a feature?
<blackboxsw> is the the feature flag really just intended to save time via skips
<blackboxsw> that last statement should have been a question: "or is the the feature flag really just intended to save time via skips?"
<powersj> I think no to both. For example, we want to be able to have apt based tests that only run on debian, and just skip on centos/rhel/fedora
<powersj> Yes the test case could have logic that collects what release it was on and exit before hand, but that seems more complicated than just saying this test needs x feature so don't run on y.
#cloud-init 2017-05-16
<prometheanfire> gah, cloud-init is now dieing because it thinks another network device should use dhcp :
<niluje> smoser: https://github.com/brmzkw/cloud-init-scaleway/commit/881495e7930dacf5a7195e560ec66589e44b55df
<niluje> want me to make a pull request with that on launchpad?
<niluje> sorry, it took me a long time to make the patch :<
<niluje> unittests are passing
<smoser> niluje, yes, merge proposal on launchpad would be good.
<smoser> and that looks good.
<niluje> okay
<niluje> time lost because of launchpad: too much
<niluje> :p
<smoser> it shouldnt be that bad really.
<niluje> it's better since you use git :)
<smoser> niluje, did you make one ?
<smoser> what is your launchpad user name ?
<niluje> smoser: jcastets, I'm trying to figure out how to push on launchpad
<niluje> # git push lp fix-net-cfg -v
<niluje> Pushing to git+ssh://jcastets@git.launchpad.net/cloud-init
<niluje> fatal: remote error: Permission denied.
<smoser>     git remote add LP_USER ssh://LP_USER@git.launchpad.net/~LP_USER/cloud-init
<smoser>     git push LP_USER master
<smoser> (from HACKING.rst)
<smoser> the key thing being you don't have write access to /cloud-init
<smoser> you *do* have write access to ~jcastets/cloud-init
<niluje> oh
<smoser> a "fork" button would be nice
<smoser> (on launchpad).
<niluje> yeyyy
<niluje> yes
<niluje> except that lp is not github, I don't see other advantages of it
<niluje> it's a pain to use, the interface is un-understandable
<niluje> like I don't see my git repository here: https://code.launchpad.net/~jcastets
<niluje> but I see it here: https://code.launchpad.net/~jcastets/cloud-init
<nacc> niluje: https://code.launchpad.net/~jcastets/+git
<nacc> niluje: note bzr is the default still
<niluje> okay
<nacc> niluje: that will change at some point
<nacc> (sooner rather than later hopefully)
<niluje> I guess it's a mess because I had a copy of cloud-init with bzr
<niluje> anyway here's my branch https://code.launchpad.net/~jcastets/cloud-init/+git/cloud-init/+ref/fix-net-cfg
<smoser> niluje, there is also https://git.launchpad.net/~jcastets/cloud-init
<smoser> but that doesnt show review
<smoser> s
<niluje> what do I need to do to make a pull/merge request?
<nacc> niluje: from the link you had -- propose for merging?
<niluje> correct, thanks
<niluje> https://code.launchpad.net/~jcastets/cloud-init/+git/cloud-init/+merge/324114
<niluje> here it is
<smoser> niluje, http://cloudinit.readthedocs.io/en/latest/topics/hacking.html *should* cover this all :)
<smoser> if you read it and it didn't make sense, help me improve it
<niluje> ok
<niluje> You can see a web view of your repository and navigate to the branch at:
<niluje> https://code.launchpad.net/~LP_USER/cloud-init/
<niluje> +git is missing then, don't you think?
<blackboxsw> thanks smoser for the xenial SRU closeout on https://bugs.launchpad.net/cloud-init/+bug/1684869 I'm reading through your changes now
<ubot5> Ubuntu bug 1684869 in cloud-init "growing root partition does not always work with root=PARTUUID=" [Medium,Confirmed]
<smoser> https://code.launchpad.net/~jcastets/cloud-init/
<smoser> that shows me git
<smoser> niluje, it doesnt for you?
<niluje> smoser: git *and* lp
<niluje> http://blackhole.brmzkw.info/2017-05-16/launchpad.png
<smoser> hm.. i dont know why it shows the "other repositories' there.
<smoser> but you click on the branc 'fix-net-cfg' there, and then click 'Propose for merging'
<niluje> yeah anyway the "other repositories" is shown even if +git is in the url
<niluje> I just did smoser : https://code.launchpad.net/~jcastets/cloud-init/+git/cloud-init/+merge/324114
<smoser> blackboxsw, it might make sense to have get-proposed-cloudimg get the -uefi for xenial. since it actually is the same as the yakkety+ ones are for 'disk.img'
<smoser> or at least have an option '--uefi' or something.
<blackboxsw> +1 on that thought smoser maybe --uefi param
<blackboxsw> we (Canonical) are the only consumers of those tools right?
<smoser> blackboxsw, yeah. we "you and me"
<smoser> niluje, i just put a comment on your mp.
<smoser> what do you think ?
<smoser> applied, that makes your change much shorter :) http://paste.ubuntu.com/24587355/
<niluje> smoser: you're much more qualified than me to know what's the best thing to do here, but IMO we could keep my patch but maybe change the unittests
<niluje> in the unittests, the file in DHCP_CONTENT_1 contains PROTO='dhcp' ; IPV4ADDR='192.168....'
<niluje> I guess it doesn't really make sense do have both and a valid /run/net-<iface>.cfg file won't ever contain such config
<niluje> but if it does, you probably want to have the addr in the cloud-init config object
<niluje> so my suggestion is instead of your patch remove the IPV{4,6}ADDR from the example files when PROTO='dhcp' in the unittests
<smoser> niluje, well, the files are what klibc's ipconfig generates
<smoser> ie, they are real world example
<smoser> so i'd like to keep them as they are, but we just ignore the setting of 'address' on dhcp types.
<smoser> as other "normal" ways of saying "please use dhcp" do not provide you with an address, so this way other parts of the code wouldn't unintentionally expect/rely on it.
<niluje> ok
<niluje> so let me push --force with your suggestion then
<smoser> niluje, you didn't do that , id dyou ?
<smoser> please do and then i'll pull.
<smoser> and, you dont need to --force if you do not want
<smoser> as i will squash when i merge (and use the commit message from the MP)
<smoser> rharper, i'd appreciate your quick-ish review of https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/323420
<smoser> i *think* your comments are all addressed.
<niluje> smoser: I was running the unittests, just pushed --force you can pull
<smoser> niluje, htanks
<rharper> smoser: ok
<niluje> thanks for the merge and for your patience smoser
<smoser> niluje, thanksk for contributing
<niluje> I hope the scaleway datasource will be finished in the following days
<niluje> We set ip=:::::eth0: in the cmdline because otherwise our servers with two network interfaces don't boot (because the default gw needs to be eth0 and not eth1), but unlike regular initrds we don't generate /run/net-<iface>.cfg. I don't know if I should spend some time to see how to get rid of ip= from the cmdline or how to generate a valid /run/net-<iface>.cfg file from the initrd.
<smoser> niluje, yeah, i'm not sure what the best way to do this is.
<smoser> as it is right now, kernel command line overrides the datasource configuration.
<smoser> so your datasource would be "trumped" by the cmdline
<blackboxsw> powersj: what is the intent behind -n versus --build-os for run_tree?
<powersj> -n, aka --os-name, is the OS you wish to run tests on
<powersj> --build-os is what OS you want to build the deb on
<blackboxsw> It seems like the consolidated args we are providing don't quite make sense for all actions I'm not quite sure why  BDDEB arg sets are provided as subsets for tree_collect and tree_run
<blackboxsw> as defined in tests/cloud_tests/args.py
<blackboxsw> so why would you build a deb in a different os than you would run tests on?
<blackboxsw> (maybe in the case where our build environment is limited to xenial doesn't support new series X yet?)
<powersj> exactly
<powersj> We don't exactly have the logic in there either to build for multiple OSes in a single run either
<powersj> for the case where someone has multiple -n arguments for example
<blackboxsw> ahh I see now you've also commented on the mp. thanks. was just trying to understand why we might want to have --build-os artful -os-name xenial in our tree_runs.
<blackboxsw> yes though we have documented it :)     -- tree_run --verbose \ --os-name xenial --os-name stretch
<blackboxsw> maybe we just drop the xenial example from the docs until that feature is supported
<blackboxsw> ok I'm feeling better about it. had to re-read the docs. thanks.
<powersj> :)
<powersj> blackboxsw: FYI the launchpad diff is old. I keep getting "oops: cannot generate diff" so a few of your changes are not needed
<smoser> rharper, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324126
<blackboxsw> powersj: ahh right forgot about that
<powersj> blackboxsw: you mentioned "SKIP_UNCOMMITTED_CHANGES_CHECK", where is that?
<blackboxsw> most of the comments are cosmetic at the moment. trying to wrap up the logic/code review
<blackboxsw> powersj: completely undocumented :/
<blackboxsw> hmm wait a sec. I don't see it ATM. lemme dig it up
<blackboxsw> tools/make_tarball
<blackboxsw> tools/make-tarball.
<blackboxsw> which I think is what barfs when you have uncommited changes local in the tree while running make deb
<blackboxsw> or bddeb as it were
<powersj> right, so we tar up the working tree, push to builder, extract, commit everything, and run bddeb
<powersj> I don't see in that process where we would want to use that check?
<dpb1_> smoser: reading a couple things from rharper and I need some clarification -- what is "the event json object"
<powersj> blackboxsw: and it looks like I need to fix all the doc strings... :\
<blackboxsw> powersj: smoser if you guys agree w/ docstrings adhering to https://www.python.org/dev/peps/pep-0257/ I don't mind if we leave that out of powersj consolidated branch and I can followup and clean up all the integration tests by turning a crank and fixing docstrings (since I'm the one complaining)
<smoser> dpb1_, probably a json formatrted description of the event (like a hotplug attach of a disk)
<blackboxsw> it's busy work that needn't block your branch in my mind powersj. It just makes the code more legible/maintainable long term
<powersj> we should add a pep257 test then too
<blackboxsw> not a bad check for new changes
<smoser> blackboxsw, i think you're suggesting that all unit tests should have docstrings that adhear to that pep.
<smoser> i'm not opposed to that.
<powersj> blackboxsw: "Launchpad encountered an internal error during the following operation: generating the diff for a merge proposal.  It was logged with id OOPS-e452bec09711d44ff1a102538807d0e2.  Sorry for the inconvenience."
<blackboxsw> in my comments on powersj branch, it was just that all methods added ahdere to pep257 single line docstring, blank line, additional details if needed (param descriptions etc).
<dpb1_> smoser: oh, maybe the JSON that is already reported in the fire-and-forget reporting that cloud-init already does
 * dpb1_ is starting to put those pieces together
<rharper> smoser: will re-read
<rharper> dpb1_: right, so cloudinit/reporting/events.py has the Object;  there is a as_dict() method, which is then used in the WebReporter to post to a remote URL handler
<rharper> dpb1_: cloudinit/reporting/handlers:WebHookHandler 's publish_event takes the event object, and json.dumps(object.as_dict())
<dpb1_> rharper: I'm making changes to the doc, fyi
<dpb1_> in flight
<rharper> ack
<blackboxsw> powersj: ok I've finally finished reviewing your branch. had a suggestion for a couple of the tests/cloud_tests/utils that might take a little time, but we can iterate. Thanks again. I'll peek back at the branch when you say it's ready
<blackboxsw> man that hurt my brain.
<powersj> blackboxsw: haha thanks
<powersj> blackboxsw: the util change was easy. Many calls were already using cloud-init's util functions versus the ones that pass to it. So I removed the test util calls in a few remaining places.
<blackboxsw> thanks powersj
<smoser> https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/323351
<powersj> smoser: yes
<smoser> hm..
<smoser> ah.
<smoser> i see you saw it.
<smoser>  initialisation
<smoser> you had a commit that changed that to a z and then back to an s
<smoser> i didn't realize there was a revert
<powersj> yeah launchpad was being silly and not generating a diff
<powersj> so I made a quick silly change to force it and reverted and the diff got generated
<smoser> oh. funny.
<smoser> blackboxsw, you think https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/323076 is ready, r ight?
<rharper> smoser: on that net-convert; my MP was backwards;  it's the netplan renderer which flipped the args (render_network_state should always take (network_state, target=target)
<smoser> ugh.
<rharper> my fault, I didn't notice that until i was also net-converting with sysconfig as well
 * rharper puts on the cone of shame
<smoser> bah
<blackboxsw> smoser: +1 on the make deb target.  it's only step one, there will be a followup small branch to extract dependencies from the built package (rpm|deb) files
<smoser> k
<smoser> rharper, so what was the bug ?
<rharper> #1685944
<rharper> lp:1685944
<rharper> bug 1685944
<ubot5> bug 1685944 in cloud-init "tools/net-convert: fix argument order for render_network_state" [Low,Fix committed] https://launchpad.net/bugs/1685944
<rharper> there we go
<smoser> rharper, yeah, but what *was* wrong
<rharper> the whole patch
<smoser> oh. ther netplan renderer. huck
<rharper> netplan.py should call with the right order
<rharper> smoser: xnox has a branch to fix
<rharper> lemme find
<rharper> https://code.launchpad.net/~xnox/cloud-init/+git/cloud-init/+merge/324012
<rharper> I had suggested to also fix up the unittest callers to pass target=value instead of just state, dir. but that's not strictly required
<rharper> I have both of that in a branch for fixing up some sysconfig stuff, but I've not had time to do a PR yet
<smoser> so eseentially we should grab xnox
<smoser> and revert yours
<smoser> right?
<smoser> xnox's branch
<smoser> plus
<smoser> -    r.render_network_state(args.directory, ns)
<smoser> +    r.render_network_state(ns, target=args.directory)
<smoser> in tools/net-convert.py
<smoser> i' going to revert yours now. and then i have to run
<rharper> ack
 * rharper steps out for the vet
<smoser> reverting && toxing && git pushing
<smoser> then tomnorrow get the others
 * smoser runs
<powersj> blackboxsw: https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/324136
<powersj> new merge request with fresh diff and updates to doc strings
<powersj> updated util functions as well and got rid of silly functions that you called out.
<powersj> I need to run through the tests again and then I'll mark as "needs review"
<blackboxsw> 6732 lines (+2526/-1200) :)
<blackboxsw> heh
<blackboxsw> wow
<blackboxsw> I'll browse through it now powersj to see if my other comments were handled thanks !
<powersj> ok
<blackboxsw> hrm did integration tests just fail due to the snappy changes smoser landed today? https://jenkins.ubuntu.com/server/job/cloud-init-integration-proposed-z/6/console
<powersj> blackboxsw: see my email on snappy failures
<powersj> but the centos 6 tests just failed because of new tests... sigh
<blackboxsw> thx for the email ptr.
<powersj> https://paste.ubuntu.com/24589541/
<powersj> ok added e2fsprogs to deps list for setting up container that seems to have fixed it :D
<powersj> CentOS 6 is good; 7 not so much https://paste.ubuntu.com/24589593/ seems hard coded path for mkfs.ext4
<blackboxsw> wow powersj you redid all docstrings
<blackboxsw> thanks man
<blackboxsw> I love it
<powersj> blackboxsw: np figured now is the time
<powersj> I did use pydocstyle to test them too
<powersj> few are incomplete in terms of args as I only took what was there and updated it.
#cloud-init 2017-05-17
<blackboxsw> powersj: done last pass on comments completed w/ https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/324136
<blackboxsw> yeah you really did a heavy lift here thanks
<powersj> Thanks!
<powersj> https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/324201
<powersj> smoser: ^
<smoser> powersj, responded there..
<smoser> http://paste.ubuntu.com/24594336/
<powersj> smoser: Will do, thx
<smoser> larsks, is akaris avaailable on irc ?
<larsks> smoser: i can try to find out.  Give me a second...
<akaris> smoser hey
<akaris> how are you?
<rharper> powersj: had some discussion today around the daily repo for cloud-init rpms ... is there work to do there? discussions ?
<smoser> akaris, hey.
<akaris> smoser so I just pushed a verification / valueerror if the protocol is not in ipv4 or ipv6
<akaris> smoser I'll have to test centos/rhel 5/6/7 again
<smoser> akaris, awesome. thank you. that really does look good
<smoser> i was looking at the mask2cidr stuff
<smoser> that is a mess
<smoser> :)
<larsks> akaris: thanks for taking on that patch.  That whole file is a bit hairy!
<smoser> trying to make it into some sense.
<smoser> yeah. the tools/net-convert.py is quite helpful
<powersj> rharper: ah yes there is work. We need to figure out how to automate the creation daily. I still am in the investigation phase of how to make the automation happen.
<powersj> rharper: trying to see what APIs are available or how to get those builds to happen in an automated fashion using COPR
<powersj> more than willing to add you to the group on COPR and let you play with it as well
<powersj> https://copr.fedorainfracloud.org/coprs/g/cloud-init/cloud-init/
<rharper> cool
<akaris> smoser that mask2cidr stuff ... I think I triggered it because I deleted the wrong data from cloud-init's cache
<akaris> I was just figuring out via trial/error what i needed to delete to trigger a rerun
<akaris> and when I finally got there, it turns out that I somehow still must have had some data (because the mask was now converted to an integer)
<akaris> and that obviously doesn't fit into that method
<smoser> but i dont know how you got the wrong stuff.
<akaris> smoser I think I added that in the bug report, sec
<akaris> https://bugs.launchpad.net/cloud-init/+bug/1684349       rm -Rf /var/lib/cloud/data/* ; cloud-init --force init
<ubot5> Ubuntu bug 1684349 in cloud-init " mask2cidr error with integer value - argument of type 'int' is not iterable" [Undecided,New]
<akaris> this was on RHEL with cloud-init 0.7.9
<akaris> and an OSP setup with config drive
<akaris> I'm likely just hitting some corner case because I don't know how to delete that data correctly ;-)
<akaris> as a matter of fact, if I delete /var/lib/cloud/* then with the config drive setup, I cannot rerun cloud-init
<akaris> is there some other way to make it run just as if this was right after the very first boot?
<larsks> akaris: that's odd; I'm pretty sure that should work...what happens when you try?
<akaris> lasks I think nothing happened. I'll have to bring my env back up and I can reconfirm
<akaris> these instances cannot reach the metadata agent
<akaris> the customer has a scenario where there is no L2 adjacency between the instance and OSP
<larsks> ...which should be okay, if you're using config drive.
<smoser> akaris, well yeah, that should have worked.
<smoser> i dont know what state it would have found.
<akaris> may I need to delete that data and then reboot the instance ? because I did  rm -Rf /var/lib/cloud/*   ;  cloud-init --force init     and that didn't work ... but that's weeks ago btw
<larsks> akaris: for what it's worth, I rm /var/lib/cloud/* and re-run cloud-init all the time, and I haven't run into a particular problem.
<akaris> let me recreate that test environment and test again. that will be better than speculating
<smoser> yea, that is what is wierd. that really should work. if you could get the network_data.json that'd be helpful
<akaris> larsks ok cool, I'll retry that
<akaris> smoser ack
<akaris> gimme a few hours to get that back up running
#cloud-init 2017-05-18
<blackboxsw> smoser: rharper : does cloud init always go through all cc_* modules  when trying to process a config or does it attempt to filter to only those cc_* modules for which a section key is present.
<blackboxsw> err that was a ?
<rharper> blackboxsw: no
<rharper> there are several controlling factors
<rharper> I think we've documented this; lemme find it
<rharper> not that I can find easliy
<rharper> =(
<rharper> blackboxsw: so, in the system_info configuration (defined in /etc/cloud/cloud.cfg) there are lists of modules to run at various stages
<blackboxsw> cool yeah I just missed the docs/comments explaining why as i was wondering why cc/handle needs to check for key and log a "skip"
<rharper> those are considered the default list of modules that run
<blackboxsw> so, for example,  chef is defined in the final module stage, but if there is no chef config defined why would we attempt to run the module's handle()
<blackboxsw> do some modules actually perform work if no section key is present?
<blackboxsw> I know ntp performs work if both pools and server lists are empty.(creating default pools). But that at least requires an ntp  key in your cloud-config
<farcaller> smoser: hi
<rharper> blackboxsw: so, typically if you provide top-level config for a module, it will run; some modules (if in the default list) will run whether config is provided or not;
<rharper> ntp, for example, is in the default list in /etc/cloud/cloud.cfg
<rharper> chef, however, is not
<blackboxsw> I misunderstood, I thought cloud_final_modules list containing chef & puppet in /etc/cloud/cloud.cfg means chef will be run by default. Is that cloud_final_modules treated differently than cloud_config_modules in the cloud.cfg?
<rharper> sorry, I didn't realize what was in the list
<blackboxsw> oh ok. good good
<rharper> yes, it will run if in the list;  typically the modules should have a check to see if they have config or not and quickly exit
<blackboxsw> yeah I was wondering about shortcutting that last part "e if they have config or not and quickly exit" so that they don't even get called if no config
<blackboxsw> so that the handlers get a little simpler (knowing that they aren't called if their schema-defined keys are not present).
<blackboxsw> but not something for this branch I'm working currently. Just thoughts bouncing around in my head
<blackboxsw> ... as there's a lot of empty space in there (my head
<smoser> farcaller, hey.
<smoser> whats up, farcaller ?
<farcaller> smoser: sorry, got afk for a moment
<farcaller> smoser: I got you a simple implementation of NoCloud via dmi: https://bugs.launchpad.net/cloud-init/+bug/1691772
<ubot5> Ubuntu bug 1691772 in cloud-init "provide a way to seed NoCloud from network without image modification." [Medium,Confirmed]
<farcaller> I think it's safe enough in that you need to have nocloud magic string for it to work, and it's very simple to pass through via qemu / libvirt / whatever.
<smoser> farcaller, that looks nice.
<smoser> a few thoughts
<smoser> a.) can you give an eexample of qemu cmdline that you use. (i'im guessing its like the kernel command line... you're puttting ds=nocloud;seedfrom=http://)
<farcaller> yep, I use exactly that
<smoser> b.) not sure if serial number seems right or not here (admittedly we'll be abusing something)
<farcaller> and yes, it's definitely quite an abuse, I guess other fields could work better, but I'm not sure. It's up to "cloud" owners to see what they are willing to sacrifice, I guess?
<smoser> well, if you're passing 'seed' in that way, your'e not really a "cloud" owner i dont think. right?
<farcaller> yep
<smoser> ie, if you've *acutally* got a cloud, lets take a similar path to what brightbox did
<farcaller> that's what I think too
<farcaller> it's more useful for small scale / hobby stuff
<smoser> its really good idea. and somethign iv'e wanted to do before. thanks for raising it.
<smoser>  (unfortunate that smbios is only x86... arm64 has it, but qemu bugs prevent using)
<farcaller> yep
<farcaller> it's just that ec2-like md server is still the simplest way to bootstrap
<smoser> farcaller, you can also fairly easily modify your smbios to look like ec2
<farcaller> not really, I tried that :)
<farcaller> cloud-init seems to look into sysfs for ec2-like uuid
<smoser> well... thats because it was broken
<farcaller> I couldn't find a way to fake that
<smoser> its fixed now i'm pretty sure (in trunk)
<farcaller> I see. I was experimenting with 16.10 image, I believe
<smoser> you ahve to set product serial and uuid to the same value and they ahve to start with ec2 (case insensitive)
<farcaller> didn't work for me there
<farcaller> yep, I definitely tried doing that
<smoser> commit was at 370a04e8d7b530c1ef8280e15eb628ff6880c736
<smoser> it went in last week, not in an image yet.
<farcaller> ack
<smoser> farcaller, do you know how big that can be ?
<smoser> how long the fied
<smoser> field
<farcaller> nope. I didn't check tbh. I verified that it fits seedfrom with ip address, and that's it
<larsks> smoser: I'm seeing unit test failures on fedora because DataSourceCloudStack is trying to access /var/lib/NetworkManager, which requires root access.
<smoser> selinux=off
<smoser> ;)
<larsks> smoser: nope.
<larsks> drwx------. 2 root root 4096 May 17 10:53 /var/lib/NetworkManager
<larsks> Unit tests shouldn't be trying to access anything outside of the current directory in any case.
<larsks> Or current directory + any tmpdir for the test.
<smoser> agreed.
<smoser> (i was just joking about selinux)
<larsks> Is there any chance we can get automated ci running on a centos system?
<smoser> larsks, there sure is!
<larsks> Is there anything I can do to help?
<smoser> https://jenkins.ubuntu.com/server/view/Cloud-init/
<smoser> the centos-6 and centos-7 are running now (and passing)
<smoser> powersj just got that up a few days ago.
<larsks> smoser: I am confused as to why they are passing, given the errors I am seeing locally.
<smoser> they dont run on MP but on trunk nightly
<larsks> My centos7 box has the same permissions on /var/lib/NetworkManager as the  fedora system on which I'm working.
<smoser> probably becausae they run in an lxd image that doesn't have networtk manager
<powersj> yeah I'm cheating by using lxd, don't have hardware for dedicated centos system yet
<larsks> powersj: just installing the NM package into the LXD container should trigger this problem, right?
<larsks> But I guess the real solution is to fix the unit tests.
<smoser> ii suspect it should trigger it, and yeah, we can fix the unit tests. i'll give it a try here.
<smoser> i suspect i can just mkdir -p /var/lib/NetworkManager && chmod 700
<rharper> you need to touch a file in there
<larsks> smoser: awesome, thanks.
<rharper> if os.path.exists(d) and len(os.listdir(d)) > 0:
<rharper> then it'll try to parse it
<larsks> smoser: I would vote for just installing the package, rather than trying faking it.  Because who knows what other permissions we might be missing?
<smoser> hm.. well it wasnt that easy
<smoser>  sudo chmod go-rX /var/lib/NetworkManager/
<smoser> and
<smoser> tox-venv py3 python3 -m nose tests/unittests/test_datasource/test_cloudstack.py
<smoser> still works for me.
<larsks> Iiiinteresting.  I bet that https://github.com/cloud-init/cloud-init/blob/master/cloudinit/sources/DataSourceCloudStack.py#L179 is finding one of the other directories in that list.
<smoser> oh. because i have a /var/lib/dhclient probably
<larsks> Yeah.
<smoser> yeah
<larsks> :)
<larsks> What if we just mocked out get_dhclient_d?
<smoser> larsks, i think just mock get_vr_address
<raspado> hi all, I tried to disable autmount of ephemeral storage to /dev/vdb via cloud.cfg but our instance is still creating ephemeral -> /dev/vdb, here is my example https://pastebin.com/8HkNbbEd , any other method I can take to preventing epehemeral storage from auto mounting to vdb?
<larsks> smoser: or that.
<smoser> but get_client_d would work to, and then you could even populate it
<smoser> with a file that shoudl get correctly parsed.
<larsks> Right, that was my thought.
<larsks> smoser: https://code.launchpad.net/~larsks/cloud-init/+git/cloud-init/+merge/324277
<larsks> that mocks out get_latest_lease (which was simpler than mocking out get_vr_address, since we can just return None)
<smoser> that'll do.t hanks.
<raspado> hi all, anyway to disable mounting of ephemeral0 -> /dev/vdb?
<larsks> raspado: that's not a "mount"; that's just the name of the block device.  I don't think it gets mounted anywhere automatically.
<raspado> cloud.cfg doesnt manage that?
<larsks> No, the linux kernel automatically creates block devices entries for any devices it discovers.
<larsks> That's just normal kernel behavior.
<raspado> so if ephemeral storage is available, the kernel will auto discover and allocate it to a device id? hmm is that just cloud-aware os's?
<larsks> No, that's just normal Linux.  IF there is a block device attached, THEN you get an entry in /dev.
<raspado> hmm so in my case when i build an instance that has ephemeral storage, it automounts to /dev/vdb, I was hoping I can manipulate that via cloud.cfg somehow
<raspado> ah well..
<raspado> http://cloudinit.readthedocs.io/en/latest/topics/examples.html#adjust-mount-points-mounted
<raspado> fs_setup: may be able to disable this?
<blackboxsw> powersj: rharper smoser ok here's the start of the schema validation tool for ntp with sample failures (poor formatting from by for loop) http://pastebin.ubuntu.com/24601457/
<blackboxsw> from *my* for loop
 * rharper looks
<rharper> blackboxsw: super interesting ... so, looking at the valid ones; and I'm not sure we can do much about it;  the strings for hosts or ips...
<rharper> those could be valid in the schema since we're only specifying the encoded type as string or list or dict;
#cloud-init 2017-05-19
<askb> I am running into the following errors with cloud-init start up on centos7
<askb> 2017-05-18 23:00:53,931 - util.py[WARNING]: Failed forking and calling callback NoneType
<askb> 2017-05-18 23:00:54,059 - util.py[WARNING]: Failed to disable password for user centos
<askb> 2017-05-18 23:00:54,062 - util.py[WARNING]: Running users-groups (<module 'cloudinit.config.cc_users_groups' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_users_groups.pyc'>) failed
<askb> therefore not running the init script once the instance is brought up ?
<askb> Is this a bug with Cloud-init v. 0.7.5 on centos7 ?
<askb> can anyone help with this please http://stackoverflow.com/questions/44059479/cloud-init-fails-on-util-pywarning-failed-to-disable-password-for-user-cento
<csfeathers> can anyone confirm that this is a bug or that i'm being stupid
<csfeathers> i'm adding adapters to a cloud image in VirtualBox's slot 4+, they end up as enp0s10, enp0s11 etc
<csfeathers> then /etc/network/interfaces.d/50-cloud-init.cfg tries to bring up enp0s10 where w/o the additional adapter it would be enp0s3
<csfeathers> which renders my VMs unusable b/c enp0s3 is never brought up
<csfeathers> i suspect there's a sorted() somewhere which sorts enp0s10 ahead of enp0s3
<csfeathers> am i even making sense heh
<rharper> csfeathers: you're making sense, the networking documentation may be of help if you want to control which interface is configured: http://cloudinit.readthedocs.io/en/latest/topics/network-config.html
<csfeathers> i'm not doing the config myself, i use the "ubuntu/xenial64" cloud image in vagrant
<csfeathers> i'm then using vagrant to add NICs and it breaks
<csfeathers> i dunno who is doing the configuration for those images :\
<blackboxsw> rharper: do we want to allow the following ntp config:
<blackboxsw> #cloud-config
<blackboxsw> ntp:
<blackboxsw> we do allow it (and install ntp with default 4 distro pools
<rharper> just the key?  I *think* so;
<rharper> yeah; that is what I would expect with just the key;  take the defaults
<blackboxsw> ok the code allows it, I was just wondering if we wanted to make that approved valid schema
<rharper> we provide distro templates
<blackboxsw> ok good deal
<blackboxsw> just encoding it to make sure
<rharper> ack
<rharper> smoser: when do you think we'll have a 0.8.0 release?
<smoser> i think i'd call it 0.7.10 next
<smoser> but yeah, we are in need of one.
<smoser> and i have no good reason to not call one
<rharper> ok
#cloud-init 2018-05-14
<powersj> https://bugs.launchpad.net/cloud-init/+bug/1736174
<ubot5> Ubuntu bug 1736174 in cloud-init "update_hostname module fails on CentOS/RHEL with systemd" [Undecided,New]
<powersj> someone should respond to that one
<blackboxsw> alrighty, got my ducks in a row....
<blackboxsw> #startmeeting Cloud-init bi-weekly status meeting
<meetingology> Meeting started Mon May 14 16:05:28 2018 UTC.  The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology.
<meetingology> Available commands: action commands idea info link nick
<blackboxsw> welcome folks to another cloud-init status meeting. This forum is used to communicate the recent changes,  current development efforts and host open office hours to help get quick discussion on bugs,  branches or features of interest to anyone developing (or consuming) cloud-init.
<blackboxsw> We'll go through a couple of topics as usual (Recent changes, In-progress Development, Office Hours), if there are any additional topics needed just let me know.
<blackboxsw> #topic Recent Changes
<blackboxsw> We track our upstream work publicly on trello. Feel free to participate or ask questions about any feature work that is seen up there if there are concerns.
<blackboxsw> #link https://trello.com/b/hFtWKUn3/daily-cloud-init-curtin
<blackboxsw> We have landed the following work items: beware the paste :)
<blackboxsw> * Completed release of 18.2 to Ubuntu Bionic, Artful, and Xenial
<blackboxsw> * Implement bash completion script for cloud-init command line
<blackboxsw> * Improved Softlayer datasource documentation
<blackboxsw> * net: Depend on iproute2's ip instead of net-tools ifconfig or route
<blackboxsw> * Accept-ra unset by default (LP: #1732002)
<blackboxsw> * Command collect-logs will only include most recent boot's journal (LP: #1766335)
<blackboxsw> * FreeBSD: Invoke growfs on ufs filesystems such that it does not prompt. (LP: #1404745)
<ubot5> Launchpad bug 1732002 in systemd (Ubuntu) "cloud images in lxc get ipv6 address" [Undecided,New] https://launchpad.net/bugs/1732002
<blackboxsw> * Azure: poll_imds fixes to only report 'ready' state once during pre-provisioning from Joshua Chan (LP: #1765214)
<blackboxsw> * DataSourceSmartOS: fix hang when metadata service is down from Mike Gerdts (LP: #1667735)
<ubot5> Launchpad bug 1766335 in cloud-init "Running cloud-init collect-logs inside a chroot is not possible" [Medium,Fix committed] https://launchpad.net/bugs/1766335
<blackboxsw> * DataSourceSmartOS: change default fs on ephemeral disk from ext3 to ext4 from Mike Gerdts (LP: #1763511)
<blackboxsw> * pycodestyle: Fix invalid escape sequences in string literals
<ubot5> Launchpad bug 1404745 in cloud-init "cloud-init's growfs/resize fails with gpart dependency on FreeBSD" [Undecided,Fix committed] https://launchpad.net/bugs/1404745
<ubot5> Launchpad bug 1765214 in cloud-init "Multiple success messages sent to Azure Fabric if reboot occurs during pre-provisioning" [Medium,Fix committed] https://launchpad.net/bugs/1765214
<ubot5> Launchpad bug 1667735 in cloud-init (Ubuntu Trusty) "cloud-init doesn't retry metadata lookups and hangs forever if metadata is down" [Medium,Confirmed] https://launchpad.net/bugs/1667735
<ubot5> Launchpad bug 1763511 in cloud-init (Ubuntu) "DataSourceSmartOS should default to ext4" [Medium,Fix released] https://launchpad.net/bugs/1763511
<blackboxsw> A big thank you to community involvement again. Thanks jocha(Microsoft) and mgerdts(Joyent) for the recent branch work supporting Azure and SmartOS clouds respectively
<blackboxsw> We also went through another round of StableReleaseUpdates for Ubuntu on Xenial and Artful to pull in IBMCloud platform fixes. putting Xenial and artful at 18.2-4-g05926e48-0ubuntu~16.04.2 | ~17.10.2
<blackboxsw> I think that's about it for completed development...
<blackboxsw> #topic In-progress Development
<blackboxsw> There are a couple of items being worked actively at the moment:
<blackboxsw> * SRU of cloud-init tip into bionic (should land today or tomorrow) 	18.2-27-g6ef92c98-0ubuntu1~18.04.1
<blackboxsw> * SmartOs datasource detection improvements
<blackboxsw> * Moving OpenStack datasource to get detected earlier at 'local' stange instead of 'network' stage using ephemeral dhcp client
<blackboxsw> * read_file_or_url fixes returing text content in all cases
<blackboxsw> * various upstream bug fixes
<blackboxsw> * powersj: is also investigating a move to a centralized library for our cloud testing.
<robjo> blackboxsw: w.r.t. Depend on iproute2's ip instead of net-tools ifconfig or route was this a merge of iproute2tools branch? I don't recall seeing a merge notification but am way behind in e-mail
<blackboxsw> think that about captures what upstream is working on. I think we can transition to office hours for ~30 mins for anyone to bring up ideas of interest
<blackboxsw> #topic Office Hours (next ~30 mins)
<blackboxsw> hi robjo, checking status there
<blackboxsw> I know we landed one branch on that topic
<blackboxsw> robjo: so we had a couple branches to packaging dependencies in ubuntu to call out iproute2 specifically as a hard package dependency.
<blackboxsw> robjo: and the code changes (which took in some of your branch content  and review comments) landed in rev 6d48d265a0548a2dc23e587f2a335d4e38e8db90
<robjo> OK, so I can delete my branch
<blackboxsw> https://pastebin.ubuntu.com/p/266CyDt9gD/
<robjo> thanks, so we'll get that in 18.3?
<blackboxsw> robjo: yes I think i marked you co-author on that branch and pulled in all your changes to cloudinit/config/cc_disable_ec2_metadata.py
<blackboxsw> 2	
<blackboxsw> thanks again for that, sorry for the back and forth as I hadn't seen your original branch.
<blackboxsw> robjo: definitely in 18.3
<blackboxsw> it landed a week or two after the 18.2 cut.
<stanguturi> @blackboxsw, Can someone please provide inputs for the bug https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1766538 One of our team members has uploaded the necessary log files to the bug.
<ubot5> Ubuntu bug 1766538 in cloud-init (Ubuntu) "network customization with cloud-init does not work on Ubuntu18.04 Beta2 Server" [Undecided,New]
<robjo> OK, so lets also revisit some of the other stuff I have floating about as I just did the 18.2 package for openSUSE and SLES and noticed that I am once again scarring a lot of patches :(
<robjo> blackboxsw: https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+ref/emptyStageOK should be back in your court, did you get notification?
<robjo> blackboxsw: https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+ref/noLnxDistro pending since January this may have some merge conflicts now as I had to fiddle quite a bit with the patch in my package
<blackboxsw> stanguturi: looking
<robjo> I think rharper is working on a different approach to https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+ref/chrony when can we expect that?
<robjo> and does rharper account for the fact that ntp has a different service name on different distributions?
<blackboxsw> #link https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1766538
<ubot5> Ubuntu bug 1766538 in cloud-init (Ubuntu) "network customization with cloud-init does not work on Ubuntu18.04 Beta2 Server" [Undecided,New]
<blackboxsw> #link https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+ref/emptyStageOK
<blackboxsw> ahh robjo hadn't, was on vacation Friday, will grab that/close out today
<blackboxsw> #link https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+ref/noLnxDistro
<rharper> robjo: re: ntp/chrony, that's landed, including service names for different distros; I pulled unittests and scenarios from your branch; please look over master and see if we're missing anything from your branch w.r.t use-case/scenario
<robjo> rharper: OK, if it's landed I can at least throw my branch away, and yes, will take a look at master, I take it this will be another in 18.3 item?
<blackboxsw> stanguturi: ok thanks for the logs on that bug, looks like network config parsing is falling over and OVF datasource isn't being detected https://pastebin.ubuntu.com/p/qVJxDJWZRV/
<stanguturi> @blackboxsw, Oh . Thanks. Will check with him about the test setup and update the bug. Thanks.
<blackboxsw> updated the bug with a comment there
<blackboxsw> thanks stanguturi
<blackboxsw> robjo: correct as well for ntp/chrony, we held off landing it in 18.2 because of risk
<blackboxsw> it was one of the first branches landed after the cut
<robjo> ok, leaves the https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+ref/noLnxDistro implementation of distro detection as things I'd like to get resolved, others to be addressed maybe in 2 weeks ;)
<blackboxsw> ok noLnxDistro....  I'm updating the commit comment robjo to the trailing LP: #<bug_id>
<blackboxsw> claiming a review slot on that now
<blackboxsw> good unit test coverage, thanks for that
<robjo> np
<blackboxsw> ok this can be reviewed today, not sure if why we don't already have a get_linux_distro utility somewhere, but I'll poke around today for context
<blackboxsw> ahh ahh, thanks for the bug robjo ok
<robjo> the context is that the Python implementation is going away and has been deprecated
<blackboxsw> #link https://bugs.launchpad.net/cloud-init/+bug/1745235
<ubot5> Ubuntu bug 1745235 in cloud-init "distribution detection" [Undecided,New]
<blackboxsw> thanks again
<blackboxsw> changing that status and will help you get that landed
<blackboxsw> good one
<robjo> OK, blackboxsw is on the hook for two things this week ;) distro detection and the empty modules list
<blackboxsw> #action blackboxsw review distro dection and empty modules list
<meetingology> ACTION: blackboxsw review distro dection and empty modules list
 * robjo on the hook to look at chrony support in master and report back to rharper
<blackboxsw> #action robjo review existing chrony support in master per rharper's work
<meetingology> ACTION: robjo review existing chrony support in master per rharper's work
<blackboxsw> official now :)
<blackboxsw> now if I only reviewed previous meeting's action items.... checking now
<blackboxsw> 16:51 <blackboxsw> #action blackboxsw to have discussions w/ team on datasource maintaining network on each reboot per  https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/343712
<blackboxsw> ok per last meeting, we have held a couple of discussions on hotplug vs. maintaining network only on reboot. on first blush it looks like we'd need to have these mutually exclusive, but we are just started to iron our what we want to do for our initial hotplug support in cloud-init and have to have a followup discussion about how to support both approaches
<blackboxsw> #action blackboxsw carryover network hotplug vs network maintenance on reboot-only
<meetingology> ACTION: blackboxsw carryover network hotplug vs network maintenance on reboot-only
<blackboxsw> well that was the only action item from last meeting looks like
<blackboxsw> ok I think that wraps up today's meeting.
<blackboxsw> any other parting shots folks?
<blackboxsw> Thanks again for your time. It's always a pleasure.
<blackboxsw> Next meeting two weeks, same bat time...
<blackboxsw> #endmeeting
<meetingology> Meeting ended Mon May 14 17:03:58 2018 UTC.
<meetingology> Minutes:        http://ubottu.com/meetingology/logs/cloud-init/2018/cloud-init.2018-05-14-16.05.moin.txt
 * blackboxsw heads to publish those notes now to cloud-init.github.io
* blackboxsw changed the topic of #cloud-init to: Reviews: http://bit.ly/ci-reviews | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting: Monday 5/28 16:00 UTC | cloud-init 18.2 released (03/28/2018)
<larsks> smoser: ci asplode, and I'm pretty sure it's not related to my patch: https://jenkins.ubuntu.com/server/job/cloud-init-ci/1115/console
<powersj> larsks: I'm getting those failures when running your branch locally as well, but master is fine
<larsks> That's very strange.
<larsks> The only change was a move to the add_patch method that smoser suggested.
<larsks> I guess I will take a closer look this evening. Thanks for checking.
<larsks> Meh, fixed it. Made bad assumptions about add_patch.
<robjo> rharper: looks as everything for the chrony support/ ntp naming is there, the comment I have is that this implementation is inconsistent with other concepts used
<robjo> so far we had distro specific settings in the distro class implementation, now with ntp with part of the implementation in the cc_ntp module and other parts in the distro
<robjo> I think in the long run this will be confusing and I think it is worth it to have a task to eventually move the distro specific bits into the distro implementation
<smoser> powersj: did you run something here manually ?
<smoser> https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/334992
<powersj> smoser: nope it would say started by me
<smoser> hm..
<smoser> so why did that run ?
<smoser> and https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/335406 also
<powersj> those could have been someone pressing the "rerun" link, which isn't something I have a lot of experience with
<blackboxsw> powersj: even the rerun attributes to a person I though
<blackboxsw> t
<blackboxsw> smoser: can I land https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/343738
<powersj> hmm I think so since you'd have to be logged in
<smoser> blackboxsw: i think so...
<smoser> convince yourself that we're effectively doing the same check on METADATA_SOCKFILE in DataSourceSmartOS
<smoser> i am pretty sure that as it is right now for a lxd container inside a LX_BRAND we would have the 'self.md_client.exists()' call return False and take the "No metadata device '%r' ..."
<smoser> (line 230)
<smoser> right?
<blackboxsw> smoser: ok see your followup message on the branch. for some reason I'm not highlighting in IRC today. sorry reply posted to your branch. I think minimally I'd like to see the socket file check in DataSourceSmartOS if we could.
<smoser> blackboxsw: i'm pretty sure we are checking that
<rharper> robjo: smoser and I went back and forth on this many times;  it's a struggle between having all of the ntp knowledge within the module itself, asking only distro for paths/defaults vs pushing a portion, but not all of the ntp handler into the distro class;
<paulmey> @rharper /@smoser , can I get another review of https://code.launchpad.net/~paul-meyer/cloud-init/+git/cloud-init/+merge/344538 ? Thanks!
<thomasking> Is there an issue with growpart and resizefs on Ubuntu 18.04? Both show disabled even though configs for each are enabled/auto.
<thomasking> added bug, https://bugs.launchpad.net/cloud-initramfs-tools/+bug/1771221
<ubot5> Ubuntu bug 1771221 in cloud-initramfs-tools "Ubuntu 18.04, growpart and resizefs show disabled" [Undecided,New]
<blackboxsw> responded on bug, looking for a bit more data.
#cloud-init 2018-05-15
<blackboxsw> larsks: a volley of comments for you on Why bother with the split
<blackboxsw> oops on https://code.launchpad.net/~larsks/cloud-init/+git/cloud-init/+merge/345113 rather
<blackboxsw> This can land. thank you https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/343738
 * blackboxsw kicked off an ec2-integration-x test run on the commandline (not in jenkins). So please don't kick that job in the UI
<smoser> done
<robjo> blackboxsw: noLnxDistro back in your court
<blackboxsw> powersj: can you glance at my review today https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/342010
<blackboxsw> robjo: thanks
<powersj> blackboxsw: yeah
<smoser> robjo: general opensuse question
<smoser> there is a python-jsonpatch that is zypper installable . that is python2.7
<smoser> but zypper search jsonpatch does not show a python3 version
<smoser> is that expected?
<robjo> that depends on the repositories enabled and the distribution
<smoser> well, opensuse 42.3
<robjo> if the devel:languages:python repo is anebaled python3-jsonpatch would show up or if the distribution is Tumbleweed
<smoser> and i've not enabled new repositories as doing so would require me to bother you with that question
<robjo> openSUSE 42.3 is using Python 2 by default and not many python3 packages are included by default
<robjo> openSUSE Leap 15 will have mostly python3 packages as it is by default Python3 based
<smoser> ok. so.. stick with python2 for the moment on suse
<smoser> er... opensuse 42
<smoser> right?
<robjo> Leap 15 is expected in June sometime
<smoser> k
<robjo> yes python 2 on Leap42.x
<smoser> robjo: ok. in a container i did recraet a failure of nose
<smoser> but not the one you saw
<robjo> hmm inconsistencies :( , the failure I saw was produced on an openSUSE Tumbleweed box with tox -e py3
<robjo> because I have a mix of Python2 and Python 3 on my Leap 42.3 box I cannot test there
<smoser> robjo: with https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/345630 i can run
<smoser> python -m nose tests/unittests cloudinit/
<smoser> on opensuse 42.3
<smoser> err.. hmm. i thought i could.
<smoser> err maybe not. that does fix just running that test case file, but running them all i get
<smoser> http://paste.ubuntu.com/p/YGkb42wvzp/
<smoser> i'll look more tomorrow
<smoser> powersj, blackboxsw https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/345627
<smoser> that is what is allowing me to somewhat sanely get a opensuse environment
<smoser> ./tools/run-container --keep opensuse/42.3
<robjo> I wonder what would trigger the insertion of "'http://ubuntu-mirror.suse.de/ubuntu' in my case
<robjo> that looks almost like the test is using the domain name of the test machine and thus the test itself would be host dependent
<robjo> I'll try and find some time to look into this tomorrow, have to get some docs into a state where they can be reviewed first
<smoser> no. that is doc'd in cloudinit/config/cc_apt_configure.py
<smoser> it looks for <distro>-mirror
<smoser> i'll get the tests sorted in run-container for opensuse 42.3 tomorrow.
<smoser> i have to leave "right now"
<smoser> later
<robjo> but in the test it shouldn't, or the "expected" resuts cannot be matched
<ecksit> howdy! i'm doing some user and group management using the cloud-init module that dynamic writes the file. is there a way i can set the `sudo` value to something that won't trigger the sudoers file to be created? (worth noting i haven't been able to try this on an image just yet)
<ecksit> something like `sudo: false` would be perfect but i cannot locate it in the git repository
#cloud-init 2018-05-16
<rharper> ecksit: I think if you create a user entry without a sudo: value in the dictionary, you won't get an entry for that user;  the sudoers file from cloud-init will get generated if the default user for your cloud-init has the sudo's group enabled, which it is in Ubuntu
<rharper> it's also possible we've a bug, so could be worth filing an issue with an example of the user config you're using and what's happening, vs. what you expect
<ecksit> you're right about neglecting the `sudo: value` bit but i was after an explicit way of saying that the user shouldn't have sudo instead.
<ecksit> just in case `cloud-init` decides in the future to change the default
<rharper> ecksit: we'd certainly consider it a bug to change the meaning of the sudo key in the user dictionary
<rharper> but it's also reasonable to file a bug and request to allow sudo: false to be acceptable as a way to disable sudo entry for that user
<ecksit> sure, where is the best place to lodge that? on launchpad?
<rharper> yeah
<rharper> https://bugs.launchpad.net/cloud-init/+filebug
<ecksit> thanks, will take this there!
<ecksit> lodged at https://bugs.launchpad.net/cloud-init/+bug/1771468 if anyone is following along at hme
<ubot5> Ubuntu bug 1771468 in cloud-init "Allow a way to explicitly disable sudo for a user" [Undecided,New]
<rharper> ecksit: thanks!
<ecksit> my python is weak but i can attach a patch if that will get more eyes on it
<blackboxsw> patches always welcome ecksit, we can help shepherd it in if there's an issue
<blackboxsw> it'll definitely drive that feature faster if there is a patch proposed for it
<ecksit> alright - i'll take a pass at it and see what i can do
<blackboxsw> smoser: did you already push cloud-init 18.2.41 to cosmic?
<blackboxsw> I'm guessing yes. rmadison thinks so L(
<blackboxsw> I'm guessing yes. rmadison thinks so :) rather
<blackboxsw> right that was last week Friday. ok
<smoser> blackboxsw: i did do an upload to cosmic last week, yes,.
<robjo> smoser: do we have a location for the summit? I'd like to find a hotel that's not too far away
<smoser> robjo: we do. i didnt make that public, you'll get an invitaton sometime in the next couple weeks.
<dpb1> smoser, robjo: I'm not sure we have a site nailed down yet.
<dpb1> I mean, we know a company, but not a building.
<robjo> OK thanks
<smoser> robjo: i have nose running in an opensuse 42.3 container
<smoser> the chef tets were all i had to fix (http and ssl related)
<robjo> great, I will try and take a closer look at the failure I am seeing with test_apt_v3_mirror_search_dns
<smoser> robjo: https://bugs.launchpad.net/cloud-init/+bug/1771659
<ubot5> Ubuntu bug 1771659 in cloud-init "unittests fail in OpenSuSE 42.3 with httpretty issues" [Undecided,New]
<smoser> powersj: you've probalby been over this, and i missed it.
<smoser> "jenkins is going to shut down" ?
<smoser> its not runnign tests now i dont think
<powersj> yeah I put in an RT about it
<powersj> waiting for IS
<smoser> blackboxsw: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/345630 is ready for review.
<powersj> smoser: jenkins is up
<smoser> powersj: htanks. i suspect run-container should work now.
<smoser> its running https://jenkins.ubuntu.com/server/job/cloud-init-ci/1131/
#cloud-init 2018-05-17
<paulmey> smoser:  rharper : https://code.launchpad.net/~paul-meyer/cloud-init/+git/cloud-init/+merge/344538 is ready for review
<smoser> blackboxsw: around ?
<smoser> paulmey: just hit 'save comment'
<smoser> feel free to ask questions here for th enext few minutes
<paulmey> thanks smoser
<paulmey> BTW, the error code from mount is always 32 - mount failure... so string match seems like the only reasonable option
<paulmey> also BTW, IIRC, ntfs is mounted via fuse now on ubuntu, so there is no ntfs module involved...
<paulmey> as an example where an assumption and a grep of /proc/filesystems might lead to a wrong conclusion...
<smoser> fair point.
<paulmey> on the other hand, I fully support the "string match is brittle" argument
<smoser> exit(32)
 * smoser walks afk for the night
 * paulmey is afk till tomorrow
<akik> i was able to provision a vm with libcloud to azure. i'm using cloud-init in libcloud's ex_customdata (azure customdata), but i think that's only for user-data and not meta-data
<akik> can i put the cloud-init meta-data somehow through the customdata mechanism?
<caribou> rharper: Hello, remember the other day I told you about  LP: #1531880 and that I was seeing a similar situation ?
<ubot5> Launchpad bug 1531880 in cloud-init "Failed to start Initial cloud-init job (pre-networking)" [Undecided,New] https://launchpad.net/bugs/1531880
<caribou> I'm able to reproduce it on our images. Seems to be caused by a race-condition between cloud-final.service that gets to run before cloud-init.service
<caribou> it can be easily reproduced with :
<caribou> cloud-init clean && cloud-init init --local && cloud-init modules --mode=final && cloud-init init
<rharper> caribou: interesting; given the systemd unit requirements, how do you get final to run before cloud-init init ?
<smoser> caribou: running as you are in that '&&' statement is not valid.
<caribou> smoser: yeah, I know this was  just to outline the fact that if cloud-final has a chance to run before cloud-init makes the link it will fail
<smoser> and to the best of my knowledge, systemd units enforce order of cloud-init-local.service, cloud-init.service, cloud-config.service, cloud-final.service
<caribou> but looking at my systemd-analyze plot output, the ordering seems correct
<caribou> yet, all I need to do is cloud-init clean & reboot and I'll get the failure
<caribou> FYI i'm not working off an Ubuntu Cloud Image but one of our own Scaleway images
<smoser> caribou: on that system... /var/lib/cloud/instance is a link or a directoroy ?
<smoser> can you let me in to look around ?
<caribou> smoser: a directory. I straced the execution & I clearly see cloud-final doing a mkdir /var/lib/cloud/instance
<caribou> sure let me pull in your ssh key & I'll PM the IP
<mgerdts> smoser blackboxsw rharper: I'm in need of images with all of my cloud-init fixes by June 30.  Can we wrap up my two merge proposals soon?  https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/343712 https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/344168
<smoser> the networking reboot thing is harder.
<mgerdts> what is needed to make it not so?
<blackboxsw> morning folks.
<mgerdts> morning!
<smoser> mgerdts: just work and thought oon how that will i nteract with other "dynamic" networking
<smoser> on the other... i think you need updates to your sdc:routes one. the commit message i think is not correct now.
<smoser>  yeah. you removed the global routes. need to update the commit message.
<mgerdts> If someone is doing dynamic networking that could interfere, setting maintain_networks to false (or leaving it unset, the default) will make it so that traditional behavior is maintained.
<mgerdts> will update the commit message there.
<mgerdts> I'm having troubles understanding the failure at https://jenkins.ubuntu.com/server/job/cloud-init-ci/1140/console
<blackboxsw> checking now mgerdts I just queued our CI to run against the branch so we'd have a point of reference
<blackboxsw> ahh pylint, should be easy getting you the error (it's buried above)
<mgerdts> I see it nowtests/unittests/test_handler/test_handler_bootcmd.py:150: [W0612(unused-variable), TestSchema.test_duplicates_are_fine_array_array] Unused variable 'e'
<mgerdts> this is in code I've not touched.
<blackboxsw> yep https://pastebin.ubuntu.com/p/mmrk3wWK4r/ checking
<blackboxsw> mgertds, could be that pylint upstream in ci has updated, you might need to 'git fetch master'  'get rebase -i master'
<blackboxsw> since the time that you originally posted this branch we've had a couple pycodestyle/pylint fixes land in master
<mgerdts> I'll give that a whirl.  Then git push -f?
<blackboxsw> correct mgerdts yeah then git push --force
<blackboxsw> after the rebase 'works'
<blackboxsw> works without conflicts (which I imagine it should)
<blackboxsw> we have removed pylint target in favor of pycodesyle/pyflakes. so master rebase should resolve the issue
<mgerdts> done & pushed
<blackboxsw> cheers, I'll kick it again
<blackboxsw> mgerdts: I queued another build, but a banner on our jenkins says it's going down for service soon, so we may not get a response on that build until jenkins maintenance is done
<mgerdts> ok, thanks
<blackboxsw> build 1141 here https://jenkins.ubuntu.com/server/job/cloud-init-ci/
<smoser> blackboxsw: we should do somethinng on those jobs... regex or somethign to grab errors at the end
<blackboxsw> it's a good idea smoser, same with the integration test job, we do dump a dict with results that's really hard to read as a human (and I think misread by jenkins as success too)
<blackboxsw> I think I saw an example of failures that jenkins thinks passed (but didn't)
<blackboxsw> well jenkins going for reboot now
<blackboxsw> but intregration-proposed-ec2-b  I think was false success (because salt_minion int tests fail on bionic0
<smoser> stupid humans
<smoser> pathetic
<blackboxsw> I should change the topic 'stupid pathetic humans'
<mgerdts> rharper I was just looking through your latest feedback and am a little confused by why the code I inherited from a co-worker is doing things like:
<mgerdts> rcfg = dict(...)
<mgerdts> rcfg.update({...})
<mgerdts> Instead of using update when there's just one or two items to update, why not use
<mgerdts> rcfg['network'] = route['dst']
<rharper> mgerdts: heh, so the general path is to copy keys out of the sdc: values into a config dict and the map values where the keys didn't match but are equivalent
<mgerdts> same goes for subnet just below
<mgerdts> I think that both accomplish the same thing, but one uses the dict.update method.  Is there a reason that dict.update is preferred over just doing an assignment?
<rharper> not for a single value  no
<mgerdts> I guess dict.update is used quite a bit for single values other places too.  When in Rome...
<rharper> mgerdts: heh;  I'm not fussed either way; the comment was more focused on the valid_keys, include the values from sdc:routes that have keys that we want to pickup directly, and then update the dict for any key/value pairs that need a mapping instead
<mgerdts> Yeah, I understand that.  I started asking trying to head off the next review comment that says ".update() is pointless now"
<mgerdts> But I see there's a lack of consistency in extant code.
<mgerdts> https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/344168#diff-line-62
<mgerdts> pgpws[proto] and subnet each have one key modified, but differently.
<smoser> robjo: https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/335406
<smoser> is that still relevant ? have we got what we needed for iproute2 support
<robjo> smoser: was addressed by the other commit proposal deleted
<smoser> thanks.
<smoser> robjo: you have some others too... sorry to pester
<smoser>  https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/333904
<smoser>  https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/334992
<robjo> I think 333904 is still valid
<robjo> 334992 deleted
 * powersj adds cosmic integration tests
<blackboxsw> powersj: cloud-init-ci job disappeared post reboot
<blackboxsw> https://jenkins.ubuntu.com/server/job/cloud-init-ci/1047/  etc missing now
<powersj> LOL
<powersj> same with git-ubuntu
<powersj> shhhhooootttt
<blackboxsw> I see cloud-init-ci-lander and cloud-init-ci-trigger too, hrm
<powersj> pipeline based jobs
<blackboxsw> yeah weren't in the redeploy logic/source?
<powersj> jobs are deployed by us, why it got deleted is beyond me
<powersj> I am deploying with ignored cache right now
<powersj> blackboxsw: https://paste.ubuntu.com/p/spzBwbNy8w/
<powersj> barfed on pipeline jobs
<powersj> others worked fine
<powersj> ah shoot I can't even make it by han
<powersj> hand
<powersj> rt time
<blackboxsw> ahh man :/
<powersj> blackboxsw: I cc'ed you on it
<blackboxsw> powersj: yeah, I wonder if it's a missed plugin in the juju spec
<powersj> possibly, however, he noticed we were using the pipeline plugin so he made sure the master wouldn't get turned off
<blackboxsw> like after the last #is update we didn't sync the new jenkins plugin in the spec
 * blackboxsw tries to find that spec again
<smoser> robjo: can you rebase 904 on master and push --force ?
<mgerdts> smoser: I took my best guess at parsing 'just work and thought oon how that will i nteract with other "dynamic" networking' and replying.
<mgerdts> I'm failing to see why preserving legacy behavior as the default and opting in is a problem.  Then again, I'm not sure that you've acknowledged that this is the way that it works and it is a problem.
<robjo> smoser: will do after my next call which starts in 10 minutes, got to stuff my face first
<smoser> robjo: did you find out why PATH was not set in your environment ?
<smoser> or in your users's enviroment at least ?
<robjo> What I have learned is that on Debian/Ubuntu PID 1 is actually a script and and there is some re-direction happening and in that script the path gets set, we run systemd directly and thus no PATH
<smoser> PID1 is most definitely not a script in ubuntu
<robjo> Also people chimed in and stated that blkid should not run that early in the boot process
<smoser> initramfs runs without /sbin/init == systemd, but pid1 there does exec /sbin/init (which is systemd)
<robjo> not verified that's what I got as a comment in the bug report
<smoser> and initramfs does not run everywhere, so thats not what is fixing it.
<robjo> there's also supposedly etc/systemd.conf where the path maybe set
<smoser> if you had actual evidence that blkid should not be run that early, then i'm very interested in it.
<smoser> as i have been concerned that we might be missing devices that "show up" later.
<robjo> I'll send you the suggestion one of our filesystem guys made via e-mail
<smoser> but my experience in now almost 9 years of doing this is that i've never seen it.
<smoser> if you can give me a reproduce that shows running blkid once that early in teh boot, and then running it later changed ds-identify behavior (outside of you attaching a disk late by design) then i'm willing to fix.
<smoser> i'm also willing to fix PATH, even though i think that is really not the right fix, as cloud-init will not be the only thing that has to then decide on what is a sane path... better to do that once in the distro.
<robjo> you should have an e-mail
<robjo> the thing is it appears that things work in the AWS images, but I have had a customer report and then one in openSUSE that blkid is not being found
<robjo> smoser: https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/333904 rebased and pushed
<Odd_Bloke> powersj: smoser: Did we reach a conclusion about how to enable SSHing to the instance from within it?
<Odd_Bloke> I've had a hectic couple of days, so I'm not sure if I'm just forgetting what we decided on. :p
<powersj> Odd_Bloke: did smoser propose some option? Like pushing or writing the key on the instance and use it from there?
<Odd_Bloke> Yeah, that sounds familiar.
<Odd_Bloke> Shall I take a crack at that?
<smoser> yeah. thats what i had suggested.
<smoser> the only concern i have is that we would want to make sure we do not ever copy a user's private key.
<smoser> only ones that we generated.
<smoser> i guess maybe just raise an exception somewhere if the key didn't get generated with 'ci-key' or something... i'm not sure we're generating them like that now, but that way we don't end up exposing oddbloke@demigogue's private key to the world.
<rharper> blackboxsw: what's that jq one-liner for dumping user-data out of the instance json ?
<blackboxsw> ahh got it one sec
<rharper> hrm, need to somehow escape the dash ?
<smoser> robjo: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/345786
<blackboxsw> rharper: cat /run/cloud-init/instance-data.json | jq -r '.ds."user-data"'
<rharper> ah, that's it
<blackboxsw> cat /run/cloud-init/instance-data.json | jq -r '.ds."user-data"' | base64 -d
<rharper> the "user-data"
<rharper> blackboxsw: thanks!
<blackboxsw> yeah I thought I had reference in ubuntu-sru, but turns out I didn't, had to back into it
<blackboxsw> np
<rharper> hrm
<blackboxsw> rharper: or just "cloud-init query user-data"
<blackboxsw> ohh wait too soon
<rharper> hehe
<rharper> hrm, so I've got a collect logs tar
<rharper> I ran that with the decode and I get garbage ...
<rharper> what might be in the user-data raw for a maas deploy ?
<rharper> # cat instance-data.json | jq -r '.ds."user-data"' | base64 -d
<rharper> %^ï¿½(ï¿½yjï¿½Óºï¿½
<blackboxsw> looks like a qbert curse word
<blackboxsw> hmm trying to think there thought I had a pastebin of maas cloud-init user-data
<blackboxsw> as passed through curtin
<rharper> ah
<rharper> gziped
<rharper> we do't tag that do we
<rharper>  cat instance-data.json | jq -r '.ds."user-data"' | base64 --decode | zcat
<rharper> \o/
<blackboxsw> nice, wow
<blackboxsw> we should probably tag that
<rharper> #cloud-config
<rharper> apt_mirror: ""
<rharper>  
<rharper> that seems to make cloud-init apt unhappy
<blackboxsw> rharper: we only tag "base64-encoded-keys" in cloud-init instance-data.json
<rharper> we did tag it
<rharper> just not that it's gzipped
<blackboxsw> rharper: was ds/user-data  in there
<blackboxsw> ok
<rharper> https://paste.ubuntu.com/p/VP92hb9kvG/
<blackboxsw> rharper: that's my bug against maas
<rharper> oh ?
<rharper> interesting
<rharper> shouldn't we handle it more gracefully though ?
<blackboxsw> getting the #
<blackboxsw> well not my bug, but commented on a potential solution from maas https://bugs.launchpad.net/maas/+bug/1735950
<ubot5> Ubuntu bug 1735950 in MAAS 2.3 "ValueError: Old and New apt format defined with unequal values True vs False @ apt_preserve_sources_list" [Low,Triaged]
<blackboxsw> rharper: I don't know if we can behave better
<blackboxsw> maas passes conflicting settings using the old key versus new key
<blackboxsw> I guess we could avoid a traceback and say, "hey buddy, you passed me a False and True for the same arguments, I'm trusting the new key"
<rharper> hrm, in the user-data there's nothing apt_ there
<blackboxsw> instead of the deprecated key
<rharper> except the apt_mirror: "" value
<rharper> but that is the stack trace
<blackboxsw> rharper: right maas curtin->cloud-init config always passes both I think.
<rharper> I'm asying in the cloudin config passed, there is *only* the key apt_mirror: ""
<blackboxsw> rharper: line 49 and 52 in https://pastebin.ubuntu.com/26355967/
<rharper> no other apt key at all
<rharper> yes, I understand that, I'm seeing the same stacktrace but with no apt: key at all
<rharper> the user-data only contains apt_mirror: ""
<blackboxsw> hrm. what I thought was that maas sets up apt_preserve user-data config regardless of user-data provided to maas. maybe I'm misreading
<rharper> no, you're right, it does
<rharper> but for whatever reason, the user-data from this maas deployed instance, does not contain that;  now it has a ton of extra juju stuff too
<blackboxsw> hrm.
<blackboxsw> what maas version?
<rharper> blackboxsw: not sure, I don't think it's a maas change; I don't really care about maas here, it's the apperance that user-data with apt_mirror: "" throws a stack ;  maybe that's not what's triggering it and it's the bug you pointed to, but that would mean the instance json doesn't have the accurate user-data right ?
<rharper> blackboxsw: secondary question, in our collect-logs, why don't we capture /var/lib/cloud  ?
<blackboxsw> rharper: probably a good thing to catch, it just wasn't in the original suggestion/procedure for filing bugs. I thought we pulled /run/cloud-init which has a couple symlinks to result|status.json, but it'd probably be good to see what's in /var/lib/cloud/(instance|seed) too
<rharper> right, the data dir
<rharper> including the obj.pkl
<paulmey> smoser: re LANG=C when doing mount...
<paulmey> would it be acceptable to pull the subp update_env parameter up into the mount_cb def?
<paulmey> or is there a better way of doing that?
<blackboxsw> finally approved https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/343127
<blackboxsw> landing it
<paulmey> smoser not around?
<blackboxsw> paulmey: should be back in a bit. though he may be eod
<mgerdts> blackboxsw just did a resync to master and pushed it again.  You can kick the ci tests again to resolve the failure.
<mgerdts> Or maybe I can resubmit, but don't know how.
<powersj> mgerdts: what is your merge request? I can check ci for you
<mgerdts> https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/343712
<mgerdts> oh, I guess the message I received has a "click here to rebuild" link
<paulmey> blackboxsw: ok, thanks. I'll update the MP and hope it's the right thing...
<powersj> running here: https://jenkins.ubuntu.com/server/job/cloud-init-ci/5/console
<blackboxsw> oops I had kicked again too , gonna get lots of ci votes
<robjo> smoser: https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/333904  is ready for another look
<robjo> blackboxsw: is still on the hook for noLnxDistro and emptyStageOK
<robjo> and with that I have reached EOD for me, catch you tomorrow
<blackboxsw> <-- hooked. (cheers)
<throwaway5234> Hello. Stupid question.
<throwaway5234> Documentation says to use write_files like this
<throwaway5234>  -   encoding: gzip        content: !!binary |            H4sIAGUfoFQC/zMxAgCIsCQyAgAAAA==
<throwaway5234> how do I create the H4s... part
<rharper> throwaway5234: typically you'll gzip and then base64 encode:  cat /my/binary | gzip | base64
<rharper> % echo H4sIAGUfoFQC/zMxAgCIsCQyAgAAAA== | base64 --decode | zcat
<rharper> 42
<throwaway5234> I'm mainly trying to compress a config file that's too long.
<rharper> you can use a URL for cloud config
<throwaway5234> so if I did your approach would I declare it gzip or gzip+b64
<rharper> if that's what you mead
<rharper> s/mead/need
<rharper> hrm, maybe the example needs updating, the gzip+b64 seems correct;   I wonder what happens if you just say gzip though
<rharper> yaml needs to have binary data b64 encoded anyhow , but I think best to use both, write_files will b64 decode then gunzip  the payload before writing
<throwaway5234> I appreciate you help! I was looking online for a couple hours and everyone glosses over that part.
<throwaway5234> I'm still having trouble with it. Don't know what to say.
<throwaway5234> https://pastebin.com/kMXE17tk
#cloud-init 2018-05-18
<powersj> throwaway5234: your content needs to be indented
<powersj> try running your cloud-config through a yaml validator
<powersj> like http://www.yamllint.com/
<throwaway5234> Took away a needed ! from the binary line, but that did the job.
<throwaway5234> Thanks @rharper @powersj
<smoser> rharper, powersj if people are asking questions like that... honestly the easiest thing to do is to let yaml.safe_dump do it.
<smoser> http://paste.ubuntu.com/p/4Tqk9fNnt5/
<powersj> smoser: thanks we should add that to rtd for debug
<smoser> we have lots of "how do i write yaml" questions for sure.
 * blackboxsw also uses cloud-init devel schema --config-file <your-yaml-file>, which gives you that same error. 
<blackboxsw> but admitedly that yaml traceback is not awesome for someone to debug
<smoser> and me has 'yaml-dump' which verifies the yaml and writes it to more obvious json
<smoser> http://paste.ubuntu.com/p/bKPBF7PNPt/
<smoser> i gues though binary data is not part of json
<smoser> so that would fail.
<mgerdts> Is there a plan to cut a new release sometime soon?  I'm trying to work out a strategy where (non-ubuntu) images that Joyent publishes can contain a custom build of 18.2 (or 18.5 or 18.6) without fear that vendors will release a newer package that doesn't have all of the SmartOS fixes.  18.2 came just before my fixes started rolling in.
<mgerdts> In parallel, I plan to work with debian/fedora/redhat to get our patches in their official packages.
<smoser> mgerdts: https://launchpad.net/cloud-init/+milestone/18.3
<smoser> https://launchpad.net/cloud-init/+milestone/18.4
<smoser> we're on rougly 3 month release cycle
<mgerdts> excellent, thanks
<paulmey> good morning all!
<smoser> hi paul
<paulmey> smoser: I have made some changes to my MP...
<paulmey> to get the env variable inside mount_cb, I pulled the update_env from subp to the mount_cb def
<paulmey> does that sound acceptable?
<smoser> probalby... i think probalby fine to make 'mount' always use LANG=C also
<smoser> but your change seems safer from a not break things perspective
<paulmey> ok, good
<paulmey> your other remark (don't pass down the whole dsconfig, but just the preserve_ntfs flag) was very straigtforward
<smoser>  that sort of stuff makes testing  much easier.
<smoser> i often times do the same thing as you did (maybe you were following code where i'd done that :)
<paulmey> alright, I'd call it ready for review then
<daru> hey can anyone help me, I'm trying to run a script at boot, but it doesn't seem to work, I've added my script in /var/lib/cloud/scripts/per-boot/ but it doesn't seem to work, I just want to run some iptables commands, I was wondering what am I missing, thanks!
<smoser> daru: executable ?
<smoser> is the file you've added executable
<daru> yes
<smoser> well... that *does* work. i verified it yesterday.
<smoser> can you run 'cloud-init collect-logs' and put the .tar.gz it creates somehwere ?
<daru> lemme see
<daru> is that a command on cloud-init? doesn't seem to exist
<blackboxsw> that command should exist on cloud-init 17.2 or later
<daru> Oh, I'm a bit lost, I'm running cloud-init 0.7.9 on a raspbian stretch
<daru> it looks like hypriot is not in that version yet, thanks anyway!
<blackboxsw> daru, no worries, you basically need to grab the following files to ease triage http://cloudinit.readthedocs.io/en/latest/topics/capabilities.html#cloud-init-collect-logs
<daru> oh thanks, that's really helpful, already found cloud init is trying to run it on boot but fails
<daru> is this helpful? https://paste.ee/p/nFarT#1EGl2BJ5UVLsATnNVmoILgZXSNspWoPn
<blackboxsw> daru: so /var/lib/cloud/scripts/per-boot/set_iptables_for_internet_forwarding   has an "exec format error" it may have spewed more useful stderr into /var/log/cloud-init-output.log when run.... you can also try running /var/lib/cloud/scripts/per-boot/set_iptables_for_internet_forwarding   on the booted instance to see if it emits the failure output that is usefule
<daru> it runs ok if I run it as root, and the other raspberries start having internet
<blackboxsw> daru, it makes me think that the content provided in bootcmd is not valid shell. are you running it with 'sudo /var/lib/cloud/scripts/per-boot/set_iptables_for_internet_forwarding' ?
<blackboxsw> or were you running it with sudo bash /var/lib/cloud/scripts/per-boot/set_iptables_for_internet_forwarding ?
<daru> I just did `sudo /var/lib/cloud/scripts/per-boot/set_iptables_for_internet_forwarding` and it worked (on the booted instance)
<daru> and the content is just 3 very simple iptables rules
<daru> It's probably some silly mistake of mine, but i cannot see it haha
<smoser> daru: look in /var/log/cloud-init-output.log
<daru> I've seen it, but I'm not seeing anything useful (I renamed the file btw)
<daru> 2018-05-18 18:17:12,558 - util.py[WARNING]: Failed running /var/lib/cloud/scripts/per-boot/set_iptables [-] 2018-05-18 18:17:12,573 - cc_scripts_per_boot.py[WARNING]: Failed to run module scripts-per-boot (per-boot in /var/lib/cloud/scripts/per-boot) 2018-05-18 18:17:12,574 - util.py[WARNING]: Running module scripts-per-boot (<module 'cloudinit.config.cc_scripts_per_boot' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scrip
<smoser> that woudl be log
<smoser> we want cloud-init-output.log
<smoser> which contains the output of the commands it ran
<daru> I only see that kind of info in /var/log/cloud-init.log, here is https://paste.ee/p/hoLAH#08KAuIZf1l58lU8DD5UFgazCrEmTqTTv
<blackboxsw> The file /var/log/cloud-init-output.log (contains stdout/stderr from scripts to console)   versus  /var/log/cloud-init.log which contains running debug messages about cloud-init operations
<blackboxsw> note the -output
<smoser> i suspect you do not have a '#!' in taht ?
<smoser>  /var/lib/cloud/scripts/per-boot/set_iptables
<daru> I have #!/usr/bin/bash
<smoser> is /usr/bin/bash there ?
<daru> here are both just in case https://paste.ee/p/47GrL#Ak9Z4hN6hviDUuh3dOOCF1XVOAcR99T6
<daru> let me see
<daru> oh no
<daru> it's in /bin/bash
<daru> gonna change and check, ty!
<daru> yesss, you are the best ppl!!
* blackboxsw changed the topic of #cloud-init to: Reviews: http://bit.ly/ci-reviews | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting: Monday 5/28 16:00 UTC | cloud-init 18.2 released (03/28/2018) | quotes: <daru> yesss, you are the best ppl!!!
<blackboxsw> :) thx daru
<daru> thanks a lot, gonna continue my journey!
* blackboxsw changed the topic of #cloud-init to: Reviews: http://bit.ly/ci-reviews | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting: Monday 5/28 16:00 UTC | cloud-init 18.2 released (03/28/2018)
<blackboxsw> robjo: with your noLnxDistro branch:   I'm still trying to reconcile that we already have a system_cfg: distro  value which will always be exactly what is written to your new /etc/cloud/cloud-init.user.distro in the event that someone provided --distro at build time.  The only difference is that cloud-init.user.distro only currently determines what the response from get_linux_distro() returns instead of grabbing info
<blackboxsw> from /etc/os-release info, whereas system_info:distro: <variant> determines which distro class is used.
<blackboxsw> something doesn't feel quite right, why do we need runtime get_linux_distro to get overridden by the file /etc/cloud/cloud-init.user.distro ?
<robjo> blackboxsw: You make the assumption that the user does not change cloud.cfg, which I would say is often not the case
<blackboxsw> I'm not quite understanding why we need to persist this build time variant to runtime behavior. When I build something for platform/environment X, I expect that that rpm/deb package is configured to run on target platform/environment X. If someone installs it on environment Y, there may be runtime issues I'd expect.
<robjo> if the user runs setup.py with --distro ubuntu and then puts centos in cloud.cfg things are going to go wrong and there will be no evidence as to why
<blackboxsw> robjo: if the user changes cloud.cfg to say a different distro, then I'd expect cloud-init runtime to choose cloudinit.distro.<theirchoice>
<robjo> yes, but the templates that are on the system are for ubuntu in this example and that is in some cases just not going to work
<blackboxsw> as with your branch's build time decision to build for target suse|rhel|ubuntu. Our static config templates would be created with (suse|rhel|ubuntu) in mind, so if I now install my package on a ubuntu system with suse ntp templates I can expect it to break
<robjo> smoser: asked for this to be configurable at build time
<robjo> I am OK without the build time switch
<robjo> but if we have the build time switch we need evidence of what was chosen
<robjo> I really don't care either way, I believe I worked within the guidelines provided
<robjo> I personally would have probably stuck to leaving the detection fully automatic
<robjo> but again, if the switch is there we need evidence as to what the "builder/installer" selected, or for certain issues we'll be up the creek with no answers and a lot of time wasted
<blackboxsw> robjo: we have that buildtime switch that is persisted sort of, it results in the original /etc/cloud/cloud.cfg delivered (and the template selections chosen because of our templater going down if variant X paths).  Any change the user makes after cloud-init is installed I would say we don't care about. (just like with /etc/network/interfaces files). I feel like that's outside the scope of a build-time flag given.
<robjo> OK, I will remove the file writing and reserve the right to say "I told you so" at some point in the future ;)
<blackboxsw> maybe I'm wrong here. but minimally it feels like overriding runtime behavior of get_linux_distro is going to cause us problems, because we have overridden the original build config templates, and some of the runtime system_info() behavior, but the rest of cloud-init modules make assumptions based on a system being whatever distro it claims it is. I'd be worried about fallout.
<blackboxsw> hehe
<blackboxsw> robjo: yeah I'll try to get a quick talk with smoser on this to make sure I'm not on my own island of insanity here
<smoser> if it is configurable via config... and just defaults to "auto"
<smoser> then i thinkt hat'd be enough.
<smoser> i would just like for someone to be able to clearly say "this is arch, you silly ubuntu developer."
<robjo> we have the configuration option and we have the "auto" behavior by default. The question really revolves around whether or not we keep evidence that the "builder/installer" explicitly set the distro, i.e. deviated from default auto behavior
<robjo> From my perspective, when one allows the user to do something evidence of that action should be collected and stored
<robjo> which is why I decided to write it to a file
<robjo> But again, I am not attached to the file, I'll remove to get this thing finally done
<robjo> smoser: blackboxsw ^^^^
<blackboxsw> robjo: I think I'm good with the file in /etc/cloud/ just not the runtime behavior change as a result. the file itself makes it easier to triage why rendered config content was chosen the way it was. So I think we should keep that artifact around
<blackboxsw> in a meeting on this at the moment robjo to get you an answer today
#cloud-init 2020-05-11
<oznt> hi guys, just a quick question: how does cloud init translate "set-passwords" in cloud_config_module to "cc_set_passwords" ? Can't figure this one out
<oznt> Well, I found out myself : https://github.com/canonical/cloud-init/blob/7c88f96245535aacfac9b0b04d9c0d82ac2067f4/cloudinit/config/__init__.py
<lucasmoura> blackboxsw, thansk for the review :) I will start working on the proposed changes today
<Odd_Bloke> oznt: Was that your entire question, or is there anything else we can help you out with?
<oznt> Odd_Bloke, that was the question
<Odd_Bloke> OK, great. :)
<Odd_Bloke> falcojr: I've pushed the requested changes to both https://github.com/canonical/cloud-init/pull/354 and https://github.com/canonical/cloud-init/pull/349 :)
<falcojr> cool, I'll re-review now
<falcojr> I've technically already approved on the second one, you just need another approval from a committer
<usrdev> hello all - hope you are safe and well! just a quick question. when utilizing cloud-init to create templates, i have noticed that the startup script for network configuration is severely delayed. i have cloud-init to pull a DHCP lease, and this works without issue for directly standing up VM from an ISO. i am utilizing proxmox as the HV.
<usrdev> is there something i am missing to make this DHCP lease request, quicker...
<Odd_Bloke> blackboxsw: Can you add https://github.com/canonical/cloud-init/pull/349 to your list of PRs to double-check as core committer, please?
<Odd_Bloke> We also didn't assign reviews today, and I'm first so I've taken #358.
<blackboxsw> Odd_Bloke: yes I can/will
<AnhVoMSFT> @Odd_Bloke, did you have a chance to take a look into the issue with the test_conftest having pylint error https://pastebin.ubuntu.com/p/QMkhNtShjW/
<Odd_Bloke> AnhVoMSFT: I didn't, apologies!
<Odd_Bloke> Let me take a quick look now.
<Odd_Bloke> AnhVoMSFT: Hmm, could you give me a more recent log?  I'm seeing differences in package versions, and it's hard to tell if that's just because there have been releases in the past ~week.
<Odd_Bloke> AnhVoMSFT: Hmm, actually, this is stranger than that.  I can reproduce the error if I give the exact path (i.e. `.tox/pylint/bin/python -m pylint cloudinit/tests/test_conftest.py`), but not if I only specify the directory (i.e. `.tox/pylint/bin/python -m pylint cloudinit/tests/`).  I'm also running Python 3.8 (instead of, I think, 3.5 in these logs).
<Odd_Bloke> AnhVoMSFT: And I also can't reproduce on 3.5, using tox. :/
<AnhVoMSFT> @Odd_Bloke yeah, I am having a hard time reproducing on my local machine - let me try with the .tox/pylint/bin/python. That should be the same version though
<AnhVoMSFT> yes I can repro using that particular command
<AnhVoMSFT> and I am on 3.5.2 on my xenial box (python3 --version)
<AnhVoMSFT> the build server gets a new container every time it runs the test, so whatever is in .tox isn't cached between runs
<Odd_Bloke> AnhVoMSFT: OK, I think we're dealing with some sort of pylint bug here: I see different output for `.tox/pylint/bin/python -m pylint -j1 cloudinit.tests.test_conftest cloudinit.tests.test_util` and `.tox/pylint/bin/python -m pylint -j1 cloudinit.tests.test_util cloudinit.tests.test_conftest`.  Can you confirm you see different behaviour also?
<Odd_Bloke> AnhVoMSFT: I'm filing a pylint issue now, will drop the link here once I have done so.
<Odd_Bloke> AnhVoMSFT: https://github.com/PyCQA/pylint/issues/3611
<AnhVoMSFT> yes @Odd_Bloke I can confirm I am seeing different behaviors between those two commands
<AnhVoMSFT> how did the different ordering could come up in one build environment vs the others though?
<AnhVoMSFT> Oh, is this because the container we were running the test had only one core ?
<AnhVoMSFT> @Odd_Bloke: I guess the question now is: which one of them was the right one? (I know we want a consistent behavior, but which one is the "right" consistent behavior?)
<Odd_Bloke> AnhVoMSFT: So I think it's actually more likely that the container had many cores: I think the error is only reported if test_conftest is the _first_ module that a worker process tests (as I've just posted to the issue).
<Odd_Bloke> (The number of CPUs in the host is almost certainly what pylint will discover.)
<Odd_Bloke> AnhVoMSFT: So in this case, that test is specifically testing for behaviour when the wrong arguments are called (i.e. exactly what this pylint check is warning us about :p).  So it's a valid pylint error, which we should be ignoring in this specific case.
<Odd_Bloke> However, I bet we also have a whole load of other instances of this error (or others which are similarly buggy).
<blackboxsw> lucasmoura: thanks one minor nit on https://github.com/canonical/cloud-init/pull/359
<blackboxsw> then we can land that
#cloud-init 2020-05-12
<lucasmoura> blackboxsw, thanks for the review on the apt_configure schema :) I have addressed the issues you raised
<lucasmoura> Also, I think I can address the description problem, that it is not allowing more advanced formatting like lists or notes. But should we do that on the same PR ?
<rick_h> lucasmoura:  always feel free to call out follow ups as long as what's there isn't broken.
<rick_h> lucasmoura:  more smallers prs > bigger giant ones generally
<lucasmoura> rick_h, yes, that's true
<lucasmoura> Thanks rick_h :)
<dingus9> anyone know of tooling to generate cloud-init yml configs on the hypervisor side? maybe that also pack it into an iso? thanks
<Odd_Bloke> blackboxsw: Lucas has reviewed https://github.com/canonical/cloud-init/pull/347 so it's ready for your core review BTW.
<blackboxsw> lucasmoura: rick_h as mentioned in standup, unfortunately our docs on readthedocs are auto-generated from tip of cloud-init, so if we introduce a breaking doc change it'll render to the docs shortly thereafter. So, we probably should be cognizant of this fix PR in this case. lucasmoura and I can meet today to sort it.
<blackboxsw> Odd_Bloke: +1 will check that PR out today for the dot printing. I think https://github.com/canonical/cloud-init/pull/339 is ready for review per our discussion yesterday. I used https://github.com/canonical/uss-tableflip/pull/48 to generate it. I see you have doc comments on #48. were you awaiting doc closure there before we land groovy new-upstream-snapshot?
<lucasmoura> blackboxsw, I think I have arrived in a solution for description formatting issue. I can show you the meeting or commit it to the PR so you can review
<lucasmoura> I am fine with whatever suits you best :)
<robjo> rharper: still around?
<rharper> robjo: not supposed to be, but here I am =)
<robjo> yay me
<robjo> another networking problem :( of course
<robjo> when there are multiple interfaces is there a way in the config to make sure we get the "right" assignment?
<robjo> i.e. can cloud-init rename interfaces?
<rharper> yes, it does
<robjo> the kernel discovers the interfaces in "random" order, how does cloud-init decide which interface gets associated with the config passed in by the user?
<robjo> I have a case where a 10.x.x.x network sometimes shows up on eth0 and sometimes on ethX
<rharper> right, so, if the network config includes mac addresses (and there isn't a set-name, or name specified) we look up the name of the interface by the mac
<robjo> that look up then means we get the random kernel discovered names, right?
<rharper> only the first time, afterwards we emit udev rules to bind the name to the mac
<robjo> the answer then is the user needs to specify "name" and then the mac is force to be associated with that name?
<rharper> however, if you retain eth* namespaces, you sometimes lose because of hotplug racing with cloud-init (or udev) renaming
<rharper> for manual configs, yes, for automated (platform) configs, we usually have what we need via metadata service info
<robjo> right, there is always the race condition
<rharper> correct, however, cloud-init will use the written config and manuall rename interfaces on each boot
<rharper> ideally it's already fixed byt cloud-init local time due to udev being complete
<robjo> well the metadata doesn't contain the interface names
<rharper> no
<rharper> but it has macs
<robjo> it contains "id"
<rharper> and we look it up on first boot, and then it's "fixed" in udev rules and rendered network config files
<robjo> right but somehow this still has to be stiched together
<robjo> it's the first boot that's the problem
<rharper> first boot isn't a problem
<robjo> according to the customer it is
<rharper> for each interface in openstack, we extract the mac address, and before we write udev rules or network-config to disk, we "ask" the kernel, what name do you have for this macaddress
<rharper> it says eth0; so we emit udev rule that binds MAC1 with the name 'eth0';
<rharper> we write config to disk, that includes the ip/netmask/dhcp for "eth0";
<rharper> repeat for each additional interface
<rharper> then, on next boot, udev runs and *should* already enforce the correct names;  in addition to that, cloud-init reads the config on disk (which includes the original mac to name mapping) and will use ip rename to enforce MAC1 has the name eth0
<robjo> but that is the point, we ask the kernel the name for the interface, so we depend on the discovery order of the interfaces
<rharper> sure, but only once, on the first boot
<rharper> so every boot after that can randomly change, and we will *rename* eth1 back to eth0 if eth1 has MAC1 which should be on eth0
<rharper> that happens either via udev or cloud-init
<robjo> Yes, but they want the same order every time, i.e. the 10.x.x.x network always needs to be associated with eth0
<rharper> yes, and we do that
<rharper> so if you've got the log, and udev rules and system journal, we can see if there's a bug in there; but we've had this behavior for years now;
<robjo> but there is no guarantee that the mach they give us is at the time we ask associated with eth0, it could be associated with eth2
<rharper> I see;
<rharper> there's nothing we can do about that;
<rharper> the openstack metadata would need to include a specific name
<robjo> Thanks, that's what I thought
<rharper> we originally did want to use the 'id' field to set the interface name
<rharper> but that's not part of the network_json specification;
<rharper> so we had to use the lookup
<robjo> glad I have not fallen off the rocker
<rharper> hehe, it just wasn't clear to me that the first boot pairing (which we persist) wasn't the mapping they desired
<robjo> thanks for the help
<rharper> sure
#cloud-init 2020-05-13
<maccer> hello! I'm looking for some pointers/instructions to install cloud-init on a macbook, would anyone here be able to help?
<lucasmoura> blackboxsw, I have pushed my description solution on the apt schema branch. If you want to discuss it, just let me know
<Odd_Bloke> So I did a little more poking at that pytest issue, and I think the issue is that our CiTestCase monkey-patching of util.subp causes pylint to be unable to correctly infer the type of util.subp (because it sees those CiTestCase assignments and knows that it can't infer what's going on there).  It's still not clear to me why the results have been inconsistent, but my best guess is that there is some
<Odd_Bloke> difference between the processes that means some of them do find the CiTestCase assignments (and so can't infer the call signature of util.subp, and so don't emit a message about us calling it incorrectly) and some don't (and therefore correctly infer the signature of util.subp, and so do emit the message).
<Odd_Bloke> I may have inadvertently "fixed" this in master by adding the tests in https://github.com/canonical/cloud-init/pull/343; that imports CiTestCase in test_conftest, so I would now expect all pylint workers to find the CiTestCase assignments (and therefore _never_ infer util.subp's type correctly and therefore never report the error).
<Odd_Bloke> blackboxsw: +1 on the ubuntu/devel PR! \o/
<blackboxsw> wooooooot!
 * blackboxsw is digging out from reviews at the moment.
<Odd_Bloke> Dang, they left, I'm very curious to know what that person was expecting cloud-init to do on a Macbook. :p
<blackboxsw> "take over the world", just like the rest of us
<blackboxsw> Odd_Bloke: wasn't there another person trying to get configy type things going on a mac cloud a couple weeks ago?
<blackboxsw> or maybe I inferred that from their question
<meena> has anyone made an attempt at making net/ platform independent?erâ¦
<meena> i'm looking at the code again.
<Odd_Bloke> I just noticed a gap in our fail-if-subp-is-called coverage on our tests: we configure it for pytest in cloudinit/conftest.py, which means it applies to everything under cloudinit/.  However, we have also introduced pytest-based tests under tests/.  I can see two paths forward: move conftest.py up a directory (to the repo root, so it affects all tests in the repo), or relocate the existing pytest-based
<Odd_Bloke> tests (there are 4 classes, so this isn't much work) under cloudinit/ and update our unit test documentation to prohibit introducing pytest-based tests under tests/*.  What do people think?
<meena> robjo: heyo, wanna rewrite cloudinit/net?
<robjo> ENOTIME
<meena> anyone else? :P
<meena> where did we land on? do we transform cloudinit/net into a class or a function dispatcher?
<blackboxsw> lucasmoura: sorry for the delay, your changes to schema.py look really good/appropriate. Couple minor nits on https://github.com/canonical/cloud-init/pull/357#pullrequestreview-411120453 and we are good. At some point we'll have to have a discussion about how to move to draft7 of jsonschema (as xenial/bionic/other distros don't support that). But that's a conversation for another time .
<Odd_Bloke> blackboxsw: I see you've moved the ubuntu/devel card to Daily in our board, but I don't see an upload and the PR is still open.  Did you move it over prematurely, perhaps?
<Odd_Bloke> blackboxsw: And, unrelatedly, do you know of any compelling reasons for us to move to draft7 of JSON Schema?  (Not asking you to do any investigation, just curious if you happen to know of any off-hand.)
<blackboxsw> blackboxsw: yep. premature card move turns out you have to hit <enter> to  upload :)
 * blackboxsw kicks that now-ish. it'll be ready by daily tomorrow
<Odd_Bloke> ^_^
<blackboxsw> Odd_Bloke: yes patternProperties
<lucasmoura> blackboxsw, No problem and I have fixed the typos you have commented
<blackboxsw> as it stands any cloud-config module  which has opaque keys like cc_apt_configure "sources" is not represented in schema
<blackboxsw> the schema in draft4 is permissive and just says, make sure we are a dict
<blackboxsw> it can't describe the sub-objects of each sources item
<blackboxsw> draft7 lets us describe that like this:
<blackboxsw> https://paste.ubuntu.com/p/Yf3tGrtJ6R/
<blackboxsw> without draft7 we basically have to be very permissive on nested objects below opaque keys
<blackboxsw> so as it stands, our existing draft4 schema won't error on invalid apt:\n  sources:\n      'nogood'
<Odd_Bloke> Interesting, thanks!
<blackboxsw> so same kindof question applies to curtin
<blackboxsw> and storage config validation
<blackboxsw> but that's another ball of work/big feature for a cycle
<blackboxsw> hrm Odd_Bloke lucasmoura hold the phone. I may have mapped that wrong
<blackboxsw>  /usr/lib/python3/dist-packages/jsonschema/schemas/draft4.json   in your focal/groovy Ubuntu distro shows patternProperties there
<blackboxsw> in draft4. so, hrm we might be able to get that on cc_apt_configure. I'm testing now. which would mean we could use this for lucas's current branch. But, let's leave that for a followup PR
<Odd_Bloke> blackboxsw: It's not clear to me that we need patternProperties for this case, actually: aren't the patterns for the key names (which are well-defined)?
<Odd_Bloke> *well-defined in this case
<Odd_Bloke> (Either way, agreed we shouldn't block landing the in-flight PR.)
<blackboxsw> Odd_Bloke: not for sources: https://cloudinit.readthedocs.io/en/latest/topics/examples.html#additional-apt-configuration-and-repositories
<Odd_Bloke> Oh riiight, yeah.
<blackboxsw> each top-level key under apt:sources: <any-string>: {keyid: keyserver:... source}
<blackboxsw> yeah. again it looks like a was mistaken about patternProperties being in draft7. If so, I think we can clear this up for draft4.  Also I think there may be 2-3 other config modules that don't define 'strict' schema for certain config objects living below the scope of opaque strings.
<blackboxsw> I'll add a card and we can talk about this item tomorrow/later
<amansi26> rharper: A small clarification needed. When we deploy a VM. Interface is getting assigned by cloud init, but once the VM get deployed, if we manually got and assigned some other interface. Is cloud init comes in pic there also.
<amansi26> according to my understanding, it basically takes the details from config Drive(provided by Nova) and configure the interfaces. Correct me if I am wrong.
<blackboxsw> lucasmoura: what would  you prefer you squash merge commit message be for #357.  Is this ok ? https://paste.ubuntu.com/p/NVRm9bnnqX/
<blackboxsw> I tried to trim a bit for the commit msg and went with line formatting < 80 chars
 * blackboxsw should have wrapped at 72 chars
<lucasmoura> blackboxsw, I am okay with the commit message you proposed
<blackboxsw> done merged thx lucasmoura
<meena> let's test how to this fares on Hetzner: https://github.com/canonical/cloud-init/pull/363
<amansi26> blackboxsw: Can you please clearify my above doubt?
<Odd_Bloke> blackboxsw: I think we may be able to do what we want with additionalProperties and items; patternProperties would only be necessary if we had different item schemas depending on what the key is.  (I'll push up one half of the snap change for your review before I do the other half, so you can point out whatever I'm missing. ;)
<blackboxsw> amansi26: sorry, context switching. ok cloud-init comes into the picture rendering initial network config on most datasources whenever a new instance-id is detected from the datasource metadata. in DataSourceConfigDrive case. I'm trying to refresh myself about network config behavior
<blackboxsw> so amansi26 https://github.com/canonical/cloud-init/blob/master/cloudinit/sources/DataSourceConfigDrive.py#L142 is what tells us if the system is considered new, and if new, network would be re-rendered for whatever new from openstack's network_data.json
<amansi26> So in case we add a new network, it will create a new instance id?
<blackboxsw> or rather the 'networkdata' or 'network_config' on the configdrive
<amansi26> but the config drive come in picture at the time of initial deployment. After that if we add any network, this configDrive will not get updated with the new interface details
<amansi26> blackboxsw: then how come the system will get to know a new network has to be added. Is it done through cloud init
<blackboxsw> amansi26: cloud-init generally is a boot time configuration, some datasource platforms support re-rendering network config across each boot (because metadata is expected to change providing new network information). I'm trying to wrap my head around how best to do this with configdrive
<blackboxsw> I'm trying to find you a solution that doesn't involve a hammer :)
<blackboxsw> if your configdrive metadata isn't being changed to update the config drive's metadata(with instance-id set) or networkdata (with openstack's networkdata.json format)  or network_config (which is etc/network/interfaces fomat  file content). then I don't know how cloud-init would 'known' about the new interfaces
<blackboxsw> for what it's work amansi26 rharper is working toward a hotplug solution that watches udev device adds to a vm
<blackboxsw> for what it's worth*....
<blackboxsw> that would react to seeing a new device plugged into your vm after the fact, but that would only trigger a re-crawl of metadata (ConfigDrive or openstack's network_data.json) to react to the full network config intended for that device)
<blackboxsw> so configdrive front for the moment, I believe your environment would have to surface a new system-uuid value at /sys/class/dmi/system-uuid, which tells cloud-init to re-run. Your config drive content in networkdata or network_config would have to represent the newly added device
<blackboxsw> and cloud-init would pick that up
<Odd_Bloke> blackboxsw: OK, so patternProperties actually causes us problems.  Currently we treat keys as opaque, which means if they are accidentally integers (instead of strings as they would have to be in JSON), we don't error now, but patternProperties tracebacks (not emits a warning, or fails to validate; tracebacks) if keys aren't strings.
<Odd_Bloke> (Perhaps this is what you discovered before when evaluating it.)
<blackboxsw> hrm that could be Odd_Bloke sorry for the waste there if that's the case. It's been a while since I tried exercising it. At least that would confirm, can't do this on draft4. so a noop for the moment.  My only reasong to reject my previous sentiment about moving to draft7 was that I saw the patternProperties definition today in the draft4.json  schema definition and assumed functionality was there.
<Odd_Bloke> blackboxsw: The functionality is there, but relies on keys being strings (which they _have to be_ in JSON, but do not have to be in YAML or Python dicts), so we can't use it trivially.
<blackboxsw> interesting, so patternProperties requires strings, we can't provide a '\d+' reference?
<Odd_Bloke> We could, but that would match "123456" not 123456.
<blackboxsw> ahh right right.
<blackboxsw> hrm
<blackboxsw> I hadn't gotten to that as far as implementation
<Odd_Bloke> Regardless, I still don't think we actually need it for our cases.
<blackboxsw> so Odd_Bloke to clarify: 1. you think we can find a way to describe sub-objects under opaque keys, or 2. you think we can continue to ignore strict validation of sub-objects under opaque keys
<Odd_Bloke> blackboxsw: https://github.com/canonical/cloud-init/pull/364 <-- I think we can describe sub-objects under opaque keys
<Odd_Bloke> Oh wait, I'm missing a commit there, I got distracted by testing patternProperties and didn't finish writing the commit message. :p
<Odd_Bloke> blackboxsw: OK, pushed.
<blackboxsw> Odd_Bloke: yeah commands didn't help us in this case.
<blackboxsw> fetching again
<blackboxsw> Odd_Bloke: but with assertions as dict, we don't have a definition of the properties in that dict anywhere
<blackboxsw> assertions as object rather
<blackboxsw> items': {'type': 'string'} applies only to list types
<blackboxsw> it doesn't vet object types
<Odd_Bloke> blackboxsw: I only modified commands, I haven't touched assertions yet.
<blackboxsw> hrm... .ok sorry,drawing up the counter example for commands too
<blackboxsw> Odd_Bloke: https://paste.ubuntu.com/p/FkKCdRvPwN/ should fail your schema python3 -m cloudinit.cmd.main devel schema -c this.yaml
<blackboxsw> but doesn't
<Odd_Bloke> That's still using assertions. :p
<blackboxsw> hah, oops
<blackboxsw> bummer. sooo yeah. maybe I'm wrong ... for the first time in my life...... in the last hour
<Odd_Bloke> Folks, we got him.
<Odd_Bloke> blackboxsw: Yeah, I think if you're only specifying one of them, `additionalProperties: ...` is basically the same as `patternProperties: { ".*": ...}`; both match all keys that aren't matched by `properties`.
<blackboxsw> ok excellent Odd_Bloke yes I was using the silly wrong key.
<blackboxsw> awesome. TIL
<blackboxsw> ok that'll do. good deal. so that'll work for cc_apt_configure apt:sources too I think.
<Odd_Bloke> And thinking about it, I would expect needing patternProperties is an indication that our config format is wonky; we shouldn't be switching behaviour based on substrings in configuration keys, they should all be absolute.
<blackboxsw> Odd_Bloke: with your branch, I think there is also some fallout due to simplitic failure annotation handling too. `python3 -m cloudinit.cmd.main devel schema -c snap.yaml  --annotate` traces
<blackboxsw> *simplistic error annotation tracking*
<Odd_Bloke> blackboxsw: Hmm, OK, sounds like we perhaps have a testing gap because all the tests pass.  Let me fill that gap then fix the failure(s). :)
<blackboxsw> Odd_Bloke: yes, I think annotations testing is very minimal
<blackboxsw> and since we don't have a set of bad examples in different config formats we only have a couple of cases in AnnotatedCloudconfigFileTest tests/unittests/test_handler/test_schema.py
<blackboxsw> so different object type failures and not covered at all.
<meena> hrmâ¦ cloud-init isn't recognizing that i'm on Hetznerâ¦ once againâ¦
<meena> other than that, my code seems to have workedâ¦
<meena> updated with comments on what works and what doesn't
<meena> https://github.com/canonical/cloud-init/pull/363 that is.
<meena> now, bed / sleep, etcâ¦
<meena> how do i override vendor data again?
<Odd_Bloke> blackboxsw: OK, so I think the error you're seeing is because you're using integer keys.
<Odd_Bloke> (Which people will do, so we do need to handle it.)
<blackboxsw> Odd_Bloke: strange github not allowing me to update branch in the UI now https://github.com/canonical/cloud-init/pull/347
<blackboxsw> approved btw
<Odd_Bloke> blackboxsw: https://paste.ubuntu.com/p/MmXtpXjjrZ/ <-- it's fine with string keys, not with integer keys
<blackboxsw> ahh nice. ok limited impact. but still noise. annotate is a half-baked feature that will break as we extend our schema use/validation tests. :/
<blackboxsw> community notice:  Just uploaded cloud-init 20.2+ to groovy gorilla (20.10) [ubuntu/groovy-proposed] cloud-init 20.2-20-gd10ce3ec-0ubuntu1
<meena> https://travis-ci.org/github/canonical/cloud-init/jobs/686766728 â something's wrong with the fonts here, or my eyes
<meena> given the tme, i'll say eyes.
<Odd_Bloke> meena: No I think that text is meant to be red. ;)
<blackboxsw> heh.
<Odd_Bloke> blackboxsw: Actually it's even more specific than that: the problem is using integer keys which aren't string-represented by Python the same as the way they are expressed.  So 00 is a problem because `"00" != str(int("00"))`.
<Odd_Bloke> But 0 is fine, for example.
<blackboxsw> hah
<blackboxsw> right
<Odd_Bloke> And I'm not sure there's a good way to address that, unfortunately.
<Odd_Bloke> Because ints are valid keys in YAML, when we read the YAML we lose the original string representation.  So I don't know how we would avoid it.
<Odd_Bloke> Probably the best bet is to (a) make sure we aren't using keys that would fail in any examples, and (b) document that annotation will break if you use integer keys.
<Odd_Bloke> (And (c) that patches to address that are welcome. ;)
#cloud-init 2020-05-14
<meena> whoohoo! Green! https://github.com/canonical/cloud-init/pull/363
<meena> whoa, who would've thought exchanging one file, but notâ¦ everything it touches, could somehow break the system??!
<Odd_Bloke> Weird how that works. :)
<andras-kovacs> Hi! Is there a nice way to set a short name as static host name instead of the FQDN? (with cloud-init of course)
<andras-kovacs> Or should I make it with a runcmd?
<andras-kovacs> I mean ask cloud-init to set only hostname into the /etc/hostname instead of the hostname.domain.
<Odd_Bloke> andras-kovacs: What distro are you running on?
<andras-kovacs> Red Hat...
<andras-kovacs> I think a runcmd part will be fine like hostnamectl set-hostname "${HOSTNAME/.*/}"
<andras-kovacs> It's a bit messed up scenario :(
<Odd_Bloke> andras-kovacs: https://github.com/canonical/cloud-init/blob/master/cloudinit/distros/rhel.py#L80-L91 is the code that performs the hostname set on Red Hat.
<Odd_Bloke> andras-kovacs: The hostname passed there is determined by _select_hostname below, and out_fn is self.hostname_conf_fn (so "/etc/sysconfig/network" in this case, I _think_).
<Odd_Bloke> That's about as much help as I can be for RH hostname setting though, I'm afraid.
<Odd_Bloke> (Feel free to ask more Qs though!  I just don't have the background to speak with authority.)
<andras-kovacs> Very many thanks!!
<meena> Odd_Bloke: more questions, or rather, the key question.
<meena> up on GitHub.
<Odd_Bloke> falcojr: FYI, I don't know the answers to some of your Qs in https://github.com/canonical/cloud-init/pull/367 off-hand, so I'm doing some research.
<falcojr> sounds good
<Odd_Bloke> falcojr: Could you pastebin an example traceback produced by the code in that PR?  (You should have one in cloud-init.log.)
<falcojr> sure
<Odd_Bloke> Thx!
<falcojr> https://paste.ubuntu.com/p/cH8rWmwQ4z/
<falcojr> looks like I missed a substitution
<falcojr> I'll put it in a github comment too
<Odd_Bloke> falcojr: Cool, you've presumably seen my comment then. :)
<falcojr> yep, I've been looking in some of the same areas
<Odd_Bloke> Cool, will follow up on your higher-level behavioural questions now.
<Odd_Bloke> falcojr: OK, I've replied with the behavioural questions; I've skirted around talking about implementation to give you the opportunity to think about it unbiased, so let me know if you'd like to discuss that some more.
<Odd_Bloke> s/questions/answers/
#cloud-init 2020-05-15
<tomponline> rick_h: Hi, I'm part of the LXD team and I opened a PR https://github.com/canonical/cloud-init/pull/368 to update the links to the LXD doc site. However the PR checks are asking me to sign the CLA, which I've done just now at https://ubuntu.com/legal/contributors, is there anything more I need to do?
<meena> tomponline: looks good now: https://github.com/canonical/cloud-init/pull/368/checks?check_run_id=677814374
<tomponline> meena: cool thanks
<tomponline> meena: I added myself to the signers file as per https://cloudinit.readthedocs.io/en/latest/topics/hacking.html seemed to do the trick
<meena> tomponline: this, https://linuxcontainers.org/lxd/docs/ imo, should work and not throw a 403.
<tomponline> meena: Ill mention it to stgraber
<meena> like, how do you even get from here https://linuxcontainers.org/lxd/ to any of the docs you linked?
<meena> for instance, this https://linuxcontainers.org/lxd/getting-started-cli/ links to https://github.com/lxc/lxd/blob/master/doc/image-handling.md which, again, seems "wrong"â¦
<meena> anyway, gotta go take care of bebe o/~
<tomponline> meena: there's a link from https://github.com/lxc/lxd#getting-started-with-lxd and in LXD menu there's a sub item for Documentation
<AnhVoMSFT> @blackboxsw @rharper do you have the tracking link for cloud-init 20.2 release for xenial/bionic?
<Odd_Bloke> falcojr: Thank you for pushing back on the feature flag convo, I really appreciate the productive discussion we're having. :)
<falcojr> Thanks, me too. It's a little harder as the new guy as I'm not sure if I'm just not understanding all the pieces yet, but glad it's being productive
<Odd_Bloke> falcojr: Even if we were "just" getting you up-to-speed, I would consider it productive.  That we're doing more than "just" that is excellent.
<falcojr> +1
<blackboxsw> AnhVoMSFT: hi sorry, we had planned on queuing up an SRU this week for cloud-init given that we have onboarded a couple shiny new developers (falcojr and lucasmoura)  the 20.2 SRU slipped under the radar with other current work. But, I will bring it up with the team Monday, I'd like us to start an SRU process next week.
<blackboxsw> AnhVoMSFT: I hope we can start this SRU early next week, which would mean ~2 weeks validate it and see it published in cloud images
<blackboxsw> AnhVoMSFT: can update you Monday though
<AnhVoMSFT> np, I thought the SRU had been cut (I saw 20.2 tag)
<blackboxsw> AnhVoMSFT:ahh makes sense. yeah that was the upstream release (which just released to groovy a couple days ago) Generally next item on the list is to SRU 20.2 to everywhere. We'll definitely get that very soon
