#cloud-init 2014-05-20
<harmw> jeez, smoser still on vacation?
<harlowja> seems so, i saw him at the openstack summit
<harlowja> not sure where he is now :-P
<harmw> :p
#cloud-init 2014-05-22
<r-daneel> smoser, so I was wondering if I had anything more to do for patch https://bugs.launchpad.net/cloud-init/+bug/1275098
<smoser> hey, r-daneel 
<smoser> so the reason it languisued is probably
<smoser> that i just didn't have focused time to think about it.
<smoser> its very non-trivial.
<r-daneel> you mean the logic is non-trivial ?
<smoser> in that bouncing network adapters early in boot feels strange.
<smoser> and may have unintended side effects.
<smoser> i suspect the reason the interfaces came up was because they were "left over" from a previous instance ? ie, in a "capture" ?
<r-daneel> well, I experienced only 2 cases: 1. I have no proper networking, so bring_down/bring_up has no side-effect, 2. I have wrong IP info, so  I anyway will break networking by replacing the IP
<r-daneel> the case arises when we clone a volume
<r-daneel> and then try booting a new VM from the clone
<r-daneel> the OS has already an IP (the previous - wrong - one) and has it configured before cloud-init runs
<r-daneel> of course, cloud-init does all is needed in the config files
<r-daneel> but it's too late
<r-daneel> as interfaces are set up
<r-daneel> now doing an ifup, either adds the new IP or fails totally
<harlowja_> smoser u alive!
<smoser> yeah, i dont know why my bip proxy didn't join here.
<smoser> now it will.
<smoser> r-daneel, so i'm admittedly not all that knowledgable about how boot works on centos and cloud-init.
<smoser> but in ubuntu, networking comes up in parallel to the local datasource.
<r-daneel> smoser, as far as I could understand from the code, we end up calling the _bring_up_interface() method
<smoser> if thats the case on centos (or even if its not, because i want to solve this correctly),
<smoser> then the 'ifdown' could fail with "interface not up"
<smoser> or between cloud-init takign it down and then back up, the OS could bring it up.
<r-daneel> smoser, ifdown failing does not seem to be a real issue to me
<smoser> and the cloud-init's "ifup" would fail
<smoser> r-daneel, it may not seem to be an issue.
<smoser> but you can't blindly ignore it.
<r-daneel> maybe try/catch that call and pass in case of failure 6
<r-daneel> ?
<smoser> failure 6 ?
<smoser> basically you have a fairly straigh tforward failure path, with a fairly straight forward work around.
<smoser> simply remove non 'eth0' (or all) interfaces before you "capture" and snapshot.
<r-daneel> (sorry '6' is the lower case for '?' on my keyboard :p)
<smoser> but fixing it by just going willy nilly with 'ifdown && ifup' seems to be racy
<smoser> and i'd rather have a guaranteed failure with striaght forward work around
<r-daneel> smoser, this is a volume clone, the OS is not aware of being 'freezed'
<smoser> than sometimes-it-doesnt-work situation
<smoser> r-daneel, understood.
<smoser> but you could easily "prep" before "capture".
<smoser> generally, cloud-init has made you not have to "prep" (clean). and i've wanted to make that always "just work"
<smoser> but there are some wierd cases, where I don't knwo what the right behavior is.
<r-daneel> well, I understand that proper 'prepping' would be the right thing to do
<r-daneel> for instance debian/ubuntu have that udev file you'd need to cleanup 
<smoser> debian/ubuntu should not have udev files
<smoser> unless you're using an odd MAC range
<r-daneel> but obviously, when we boot on a clone, the only issue we get is with IP configuration 
<smoser> yeah. i do understand this is an issue. and i'd like to have it fixed properly.
<r-daneel> if we try to do that very cleanly, we should check if there is a pre-existing setup or at least check if the network is setup as exepcted by our freshly installed config
<smoser> its really hard in ubuntu, and i suspect it might be in centos too (if not now, then it might be later with move to systemd)
<r-daneel> we could then only ifdown if we really already have a wrong setup
<smoser> the ordering of boot is just very much not guaranteed
<r-daneel> when cloud-init does ifup it already assumes an ordering
<r-daneel> it assumes no interface is set, and that noone will fiddle with it
<smoser> well, sort of.
<r-daneel> even worse, on centos it adds the IP to any existing config
<r-daneel> not enforcing the setup
<smoser> if there was no interface configuration, for ethX, and cloud-init writes an interface configuration for ethX , at least in current ubuntu (and i think centos) nothing is magically going to bring it up
<smoser> so that case is safe to assert "it was not up"
<smoser> ubuntu/debian 'ifdown' is terribly annoying.
<r-daneel> but cloud-init happily overwrites the existing config files
<smoser> if you remove ethX from /etc/network/interfaces and then ifdown an interface that was already up, it says "not configured"
<smoser> r-daneel, yeah, agreed. its not perfect now.
<r-daneel> if interface is non-existent, not being able to ifdown it has no effect at worse
<r-daneel> and in ubuntu/debian the call is ifup ALL, not event per interface
<r-daneel> s/event/even/
<smoser> the sanest thing to me seems to be to block any network events from occurring while you're going looking for "local" data sources
<r-daneel> again, ifdown on a non-setup interface or one that has wrong info wouldn't bother me, ... if I do changes by hand then reboot, ifdown will fail on shutdown and OS doesn't care
<smoser> then, correctly reading "existing config" and merging it with config from config drive.
<r-daneel> merging is ok, but you're likely to conflict with what is set
<smoser> but ignoring things that might fail sometimes is not helpful to anyone.
<smoser> well, if you conflict and you're blocking all networking coming up, thne you set it right.
<smoser> ie, config-drive would "win".
<smoser> (possibly allowing for cloud.cfg configuration that modifies that behavior)
<smoser> do you know if you could block all networking from coming up on centos ?
<r-daneel> but the cloud-init service in the OS comes after the OS own's script for network setup, as it seems
<smoser> i suspect i can do it on ubuntu, pretty sure you can do it fairly easily on sysvinit.
<r-daneel> so how will you prevent the OS from setting the network info ?
<smoser> there are 2 cloud-init services
<smoser> local and 'init'.
<smoser> the local can read only local datasources
<r-daneel> ok, let me check ordering ...
<smoser> and cannot presume netowrking, but can set up networking.
<smoser> then... there is the other thing that adds a twist here.
<smoser> in reality, i suspect that ocnfig drive is not long for this world.
<r-daneel> hmm ? configdrive will be removed 6
<r-daneel> ?
<smoser> its static nature is just too limiting. i suspect in time to come, config-drive will disappear as the metadata service is able to provide dynamic data.
<smoser> are you familiar with how hot-plug of network interfaces works on amazon ?
<smoser> its really nice.
<smoser> interface added, causes udev rules to fire that create and name the interface.
<smoser> those same rules say "oh, look, i'm on EC2, and that means the metadata service might tell me what IP address I should get".
<smoser> (this doesn't work on ubuntu but on Amazon Linux AMI it does)
<r-daneel> ok, I see
<smoser> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1153626
<smoser> that is a much more sane way of doing things
<smoser> and going that rougte would still mean we potentially have the issue that you were seeing (already existing config)
<smoser> r-daneel, i'm not opposed to getting this working better.
<smoser> and don't mean to sound stand-offish
<smoser> its just not as easy as it might first appear.
<r-daneel> I see your point, we use configdrive because of inherent reduction of complexity and security
<smoser> i don't really know that anything is more secure.
<smoser> if your host network is compromised, i think you have significant issues.
<smoser> and as for complexity, i think that openstack networking generally being difficult to get right is what lead to pepole wanting config drive and its initial popularity), but at least in my recent experience, that is much better sorted now.
<smoser> ie, you dont have "no route to 169.254.169.254" issues so much.
<r-daneel> exposing a common service to all tenants seems much more risky than giving access to a file on the host. Someone escaping his VM is a much bigger problem but less likely to happen
<alexpilotti> smoser: hi there
<smoser> r-daneel, but you've already exposed dozens of common services to your guests ;)
<r-daneel> smoser, such as ?
<smoser> nuetron-api, nova-api, dhcp, dns
<r-daneel> smoser, these are control planes not the VM infrastrucutre
<smoser> dhcp and dns are on vm infrastructure
<r-daneel> we have no dhcp and dns is external
<smoser> i dont know. if you can't securely route traffic from one vm to an endpoint specific to that vm, then you cant actually do networking securely between 2 vms of a single tenant.
<smoser> so yes, i agree, it is mroe complex, but its not complexity that you can actualy do without i think in the end.
<smoser> alexpilotti, whats up?
<alexpilotti> smoser: waiting for an answer on which is the minimum / recommended cloud-init version with MaaS metadata support :-)
<smoser> oh. sorry.
<smoser> our 12.04 images use 0.7.0-0ubuntu2
<smoser> so thats surely known-working
<alexpilotti> smoser: great, so with 0.7.4 we are good 
<r-daneel> smoser, when trying to keep things as secure and non-complex as possible, we found it useful to use configdrive. I agree that at some point in time we may need more flexibility and my walk the metadata-server way. For now I find it useful to have the choice between statically set information in configdrive and dynamic setu through metadata-server. 
<smoser> r-daneel, and you're certainly not alone in that decision :)
<alexpilotti> smoser: tx, doing some tests
<r-daneel> smoser, so back to our initial topic ;) as far as I remember, the 'local' and 'init' phases both ran after the OS had finished setting the IPs. Am I wrong ? 
<smoser> r-daneel, on ubuntu, local happens [possibly] in parallel with any 'auto' interfaces.
<smoser> '[possibly]' is very complex. 
<smoser> but for almost all cases i can think of they're in parallel, and nothing forces them to be serialized.
<smoser> i do not know about centos.
<smoser> sysvinit iseasy to do these sorts of things :)
<r-daneel> centos service ordering is S10network, S50cloud-init-local, S51cloud-init, S52cloud-config, S53cloud-final
<r-daneel> for ubuntu, I did not fully check. experience showed (in the logs) that we were always doing ifup on an already  up interface and ifup refused to override
<smoser> r-daneel, yeah, so on centos, it should be possible to just put cloud-init-local before S10
<smoser> the one thing that that breaks, which i dont think is a real issue is network mounted filesystems (ie, /usr/ on nfs)
<r-daneel> if you have the wrong IP, our setup prevent you from communicating (anti spoofing) if it could, we'd mess up things because of IP collision
<r-daneel> smoser, then we still have no better solution for ubuntu
<r-daneel> smoser, would it be more acceptable to do the ifdown/ifup cycle only on failure of the initial ifup ?
<smoser> maybe, yeah.
<smoser> i'm guessing one way or another we can force cloud-init local to run before and block networking coming up
<smoser> but it will probably be tricky
<smoser> (since as it is, netowrking comes up on udev events)
<smoser> (network-intreace-added)
<smoser> sorry.. net-device-added
<r-daneel> so for those relying on udev, cloud-init should already be hooked-in to 'know' what to do
<smoser> yeah. is complex though... elcoud-init explicitly actually emits the net-device-added when its inside a container
<smoser> (as lxc instances don't get those events)
<r-daneel> smoser, would it help to implement that ifdown/ifup in the platform specific files ? maybe only on ifup failure ? I understand that we're trying to march toward a future-proof solution but this will require a lot more code-diving on my part :)
<smoser> i dont mind if ifup/down is in 'distro'.
<smoser> err.. in per-distro code.
<r-daneel> ok, I'll try to come up with distro-specific code. Will get back to you for a review once done :) 
<r-daneel> smoser, thank you for your help
#cloud-init 2014-05-23
<harmw> ah, the smoser still lives :p you had a nice holiday or something?
<smoser> harmw, well, was at openstack summit, so just not around a lot from that, and then system got rebooted and bip apparently wasn't configured to auto-log in here.
<harmw> :)
<harlowja_> smoser i'm still waiting for my blinky hat
<harlowja_> lol
<harlowja_> ;0
<harlowja_> :) i mean
<smoser> you'll have to take it away from my 6 year old daughter
<harlowja_> lol
<harmw> smoser: how 'bout that cirros t-shirt
<harmw> didn't your wife mke something?
<smoser> she made me a logo. and a coffee mug with it on it.
<harmw> you've got a picture?
<harlowja_> a cirros shirt, i didn't get that, lol
<smoser> https://launchpadlibrarian.net/176196772/cirros-192.png
<harmw> you want constructive criticism or just a good 'oll 'wow nice!' :)
<harlowja_> hmmm, smoser  u need to put https://launchpadlibrarian.net/73510418/logo.png somewhere in there
<harlowja_> :)
<harmw> lol
<harlowja_> then u can wear yourself on your shirt
<smoser> i just pushed the svg to lp:~smoser/cirros/cirros-graphics
<smoser> i am most certainly *not* looking for constructive criticism on https://launchpadlibrarian.net/73510418/logo.png
<harlowja_> smoser u are supposed to add https://launchpadlibrarian.net/73510418/logo.png to https://launchpadlibrarian.net/176196772/cirros-192.png (like as an overlay)
<harlowja_> and hide 'cubswin' in there to
<smoser> on the cubswin front, their ace pitcher Jeff Samardzija has the lowest era in the major leagues, 10 starts and zero wins.
<smoser> http://ftw.usatoday.com/2014/05/jeff-samardzija-chicago-cubs-existential-nightmare-mlb
#cloud-init 2015-05-19
<dbuechler> Hi folks. Built rpm of cloud-init 0.7.7 because stock 0.7.5 doesn't read my meta-data. Don't ask me why. I don't have any idea. 0.7.7 seems to work if I execute it manually, but on boot, I get:
<dbuechler> cloud-init[1260]: File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 49, in <module>
<dbuechler> cloud-init[1260]: from six.moves.urllib import parse as urlparse
<dbuechler> cloud-init[1260]: ImportError: No module named urllib
<dbuechler> What am I missing?
<dbuechler> I just noticed smoser's comments to me.  If anybody sees him, could you please let him know that his pastebin: http://paste.ubuntu.com/11176334/  was correct.  Those were the changes that I made to brpm to get it to build correctly.
<smoser> dbuechler_, thanks.
<smoser> dbuechler_, you get that from trunk ? 
#cloud-init 2015-05-20
<mrkz> hello, Is there a [full] documentation of  all the cloud-init configuration options (/etc/cloud/cloud.cfg)?
<mrkz> looking atm http://cloudinit.readthedocs.org/en/latest/index.html but looks incomplete to me
<harmw> just out of python curiosity, how current is Flask for writing a simple REST service?
<robjo> Trying to mimic a config drive outside of OpenStack using qemu but am not having much luck, could use some help
<robjo> I have the config in ISO format and am attaching the iso file to the VM with the -cdrom option for qemu
<robjo> The device shows up as /dev/sr0 and thus I figured it should just work, but it does not appear to be picked up, any ideas?
<mrkz> robjo: maybe try #qemu ?
<robjo> mrkz: I am trying qemu
<robjo> qemu-kvm -netdev tap,helper=/usr/lib/qemu-bridge-helper,id=hn0 -device rtl8139,netdev=hn0,mac=00:16:3e:7e:18:36 -m 2048  -cdrom /work/tmp/configDrive/configdrive.iso openSUSE-13.1-OS-guest.x86_64-0.0.6.raw
<robjo> Inside the vm the ISO shows up as /dev/sr0, that based on my understanding of the code should be all there is to it
<mrkz> I mean #qemu channel, this one looks a bit inactive :/
<robjo> oh, sorry
<mrkz> robjo: I just joined and ask, but no answer so far
<robjo> it's more of a cloud-init than a qemu thing
<mrkz> robjo: so, if you're trying to mimic an openstack instance boot, shouldn't you look into nova instead of cloud-init?
<robjo> cloud-init initializes the VM and I want to test some changes to cloud init without having to set up OpenSTack
<mrkz> I'm kinda new to this, but afaik nova boot up the VM and then cloud-init runs @ boot time inside the VM and then performs all you setup in config file
<mrkz> dunno if I'm right tho
<robjo> mrkz: Yes that's how it works
<mrkz> robjo: so I quite don't understand your issue
<mrkz> also, what I do to test is rm /var/lib/cloud and then re-run cloud-init init inside VM
<robjo> When cloud-init runs it gets data from a so called data source, this can be a meta data server or can also be a so called ConfigDrive
<robjo> ConfigDrive is one of the options offered with OpenSTack and is used for example when the environment cannot have a DHCP server
<robjo> when using ConfigDrive cloud-init can write the network configuartion file and then bring upi the network
<robjo> But this implies that the config drive is properly recognized by cloud-init
<robjo> Theoretically having the config drive info in an ISO file that is attached to the VM and shows up a /dev/sr0 should be sufficient, however in my case cloud-init does not appear to find the config drive
<robjo> thus I cannot test the writing of the network configuration file
<dbuechler> Robjo: Unfortunately, I don't know anything about ConfigDrive, other than cloud-init seems to have a datasource for it.  What version of cloud-init are you running?
<robjo> 0.7.6
<dbuechler> What OS is the VM?
<robjo> dbuechler. it really doesn't matter, but it is openSUSE 13.1
<robjo> anyway I had another ISO and that one works as config drive :)
<robjo> Looks like the second ISO I generated is not quite what cloud-init expected
<robjo> I am all set
<mrkz> robjo: have you looked @ /var/log/cloud-init.log in both VM instances?
<dbuechler> Cool.
<dbuechler> robjo: Glad to hear that you got it figured out.
<dbuechler> I've got a problem of my own... cloud-init 0.7.7, built as an rpm for CentOS 7 using 'make rpm' runs on boot but bombs with:
<dbuechler> cloud-init[1260]: import cloudinit.util as util
<dbuechler> cloud-init[1260]: File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 49, in <module>
<dbuechler> cloud-init[1260]: from six.moves.urllib import parse as urlparse
<dbuechler> cloud-init[1260]: ImportError: No module named urllib
<dbuechler> I have python-urllib3 1.5 installed, so I'm at a loss to explain it.
<robjo> Couple of sources for the trouble would be that the dependencies in the rpm are not correct and when installing cloud init python-six is not pulled in
<robjo> but six.moves.urllib is imported, not urllib
<mrkz> dbuechler: but does $ python -c "import urllib; print urllib" work?
<dbuechler> Good question!  Wait 1.
<mrkz> no, as robjo said before, try this instead
<mrkz> $ python -c "from six.moves.urllib import parse as urlparse; print urlparse"
<dbuechler> Ok.  One sec and I'll find out.
<dbuechler> Returns No module named urllib
<dbuechler> I have python-six 1.3.0 installed.
<dbuechler> ...from rpm - probably EPEL.
<mrkz> so, I might be wrong, but looks like your six package lacks that
<dbuechler> Errr, looks like a stock CentOS package.
<dbuechler> Hrm.  Ok.  I'll check and see if EPEL has a six package.
<robjo> I would concur, python-six-1.3.0 is most likely too old
<robjo> I have 1.8.0 on my system and things work
<dbuechler> Ok.  I'll take a look around and report back.  I appreciate the input.
<dbuechler> Reporting back on my urllib issue - updating six in my CentOS 7 image with "pip install --upgrade six" DID solve the problem.  python-six that ships with CentOS 7 is too old.
#cloud-init 2015-05-21
<dbuechler> Hi All!  I've got a stupid question: How does resizing work in cloud-init? I built a custom image for CentOS 7 because the stock images weren't reading metadata correctly. However, it's not resizing correctly and I'm wondering if I built it wrong.  Anything I should avoid? LVM, xfs, etc?
<smoser> dbuechler, kernel version?
<dbuechler> Errrr 3.10?
<smoser> in a kernel > 3.8, cloud-init's "growpart" module calls growpart <root-device> <root-partition>
<smoser> and then resize_root module calls resizefs /
<smoser> or the equivalent.
<smoser> you need growpart to do it.
<dbuechler> So, I need to have growpart installed... in CentOS, growpart is in cloud-utils-growpart (
<dbuechler> I think)
<smoser> right.
<smoser> so that has to be there.
<smoser> and growpart only knows ow to resize mbr or gpt
<smoser> it can't lvm grow 
<dbuechler> AH!
<smoser> patches will be welcome for that.
<dbuechler> Ok.  No LVM.  Good to know.
<dbuechler> I appreciate the insight.
<smoser> i dont knwo if xfs will work or not.
<smoser> i'd have to look.
<smoser> i think we would try
<smoser> but you'd then need xfs-utils
<dbuechler> I'm pretty sure xfs-utils is installed by default in "Minimal System" configuration, but I'll double-check.
<dbuechler> I'll let you know after I rebuild the VM this afternoon.
<dbuechler> There is an issue with the stock version of python-six in CentOS 7.  I filed a bug report on it.
<smoser> you filed bug in cloud-init ?
<dbuechler> Yes. I'm not sure there's anything you can really do about it except update documentation.
<dbuechler> It was mostly an FYI.
<smoser> dbuechler, well, i'm guessing we could do something.
<dbuechler> smoser: I also filed another one regarding 'make rpm' on a tarball of 0.7.6 downloaded from cloud-init.  make-tarball bombs complaining that the directory is not a branch.
<smoser> hm.. just use the bzr branch i guess.
<dbuechler> smoser: Re six - I included my workaround in the bug report... I used pip to upgrade six (it went to 1.9.0) and that solved my problem.
<dbuechler> Can you use bzr to get 0.7.6?  I'm new to bzr, so I hadn't figured that part out yet.
<smoser> yeah.
<smoser> bzr branch lp:cloud-init
<smoser> cd cloud-init
<smoser> cd ..
<smoser> bzr branch -r 0.7.6 cloud-init 0.7.6 
<dbuechler> Ah!  Good to know.  Thank you.
<dbuechler> I have to run. I'll let you know how that new image works out. Thank you for your time.
#cloud-init 2015-05-22
<harlowja> smoser u around today in vancouver?
<harlowja> or u head back?
<smoser> here now.
<smoser> i'm sitting and working in my window office.
<smoser> pretty nice
<smoser> 3rd floor. looking at planes
<harlowja> cool
<harlowja> :-P
<harlowja> i'm in cinder cave
<harlowja> lol
<harlowja> maybe should go up there
<harlowja> probably more interesting, lol
<smoser> what are you doing in cinder ?
<harlowja> smoser ha, just listenining in
<harlowja> i'm roaming around :-P
#cloud-init 2016-05-23
<CabbageMan69> Hello all. I'm having some problems turning on debugging info for cloud-init on an instance in EC2. Is it as simple as having "verbose:\n  debug: true" in the cloud-init config?
<CabbageMan69> I'm getting the following two lines of output in my cloud-init-output.log file with that config present, and I see the following two lines ... not sure what is happening between those two steps that causes it to take 2 minutes
<CabbageMan69> Cloud-init v. 0.7.5 running 'modules:config' at Mon, 23 May 2016 05:02:58 +0000. Up 35.99 seconds. Cloud-init v. 0.7.5 running 'modules:final' at Mon, 23 May 2016 05:05:08 +0000. Up 166.10 seconds.
<CabbageMan69> For context, I'm using Ubuntu 14.04 and I'm running v0.7.5 of cloud-init (is this ancient?)
<cpaelzer> smoser: Hi, I extended the MP at https://code.launchpad.net/~paelzer/cloud-init/test-apt-source/+merge/294521
<cpaelzer> it has now the dictionary based handling and all kind of tests around it
<cpaelzer> smoser: I'll check tomorrow if I need to revise it a lot / a bit  / not - and will start to poke at the curtin portion of it then
<cpaelzer> smoser: I'll expect to have some questions tomorrow when you come online, but will let you know then
<harlowja> rharper ya that's def where the complexity comes from ;)
<harlowja> and/or figuring out sysconfig for myself, and its weirdness :-P
<rharper> harlowja: hehe
<harlowja> rharper although i think https://gist.github.com/harlowja/d63a36de0b405d83be9bd3222a5454a7/ is mostly there, just eyes on that would be cool
<harlowja> let me know if it looks dumb :-P
<harlowja> https://gist.github.com/harlowja/d63a36de0b405d83be9bd3222a5454a7/#file-sysconfig-py-L320 is the main stuffs
<harlowja> oddly i don't think redhat has a way to configure routes without an interface
<harlowja> or at least i'm not sure if there is one
<rharper> harlowja: I gave it a once over the other day; it looks quite reasonable (cleaner than my initial cut); looking to steal a lot of that;  we may find with the two implementations (and a networkd coming) that we have a base set of hooks that new config backends would implement
<harlowja> cool
<rharper> the networkd does change things up quite a bit as there is no single file to write, rather multiple files (.link for name mapping) (.network for networks which may or maynot be device specific) and (.netdev for virtual devices like bridges and bonds);  so the in-order processing that happens for eni and sysconfig don't have to apply but certainly can.
<harlowja> ya, sysconfig also doesn't have single files
<harlowja> only eni i think has a single file
<rharper> ok
<rharper> bbiab
<harlowja> smoser rharper can we (or shall i?) merge in https://code.launchpad.net/~harlowja/cloud-init/cloud-init-net-refactor
<harlowja> that way this sysconfig stuff (which that uses as a base) ; can start to get reviewed more offically :-P
 * harlowja doesn't know to well how to stack things in bzr (against a branch that hasn't merged yet)
<smoser> harlowja, can you write at very least a combined commit message and description there ?
<smoser> is it all just "refactor" did you clean up the N different internal states ?
<harlowja> sure boss
<harlowja> ummm, let me check
<harlowja> mostly refactor :-P
<harlowja> ok, updated https://gist.github.com/harlowja/d63a36de0b405d83be9bd3222a5454a7 rharper if u want to see
<rharper> looking
<rharper> harlowja: nice, lines 34 -> 70ish look like cloudinit.net common stuff that all backends could use
<harlowja> def
<rharper> harlowja: nit: L82 in the header you say on %(now)s ; would  'at' or '@' be more natural ?  L324 render_dns;  you have it as a pass, but I'd think the signature would need to take a list object to append something if/when it ever did support updating dns;
<harlowja> sure
<harlowja> rharper  yup yup, dns not done yet ;)
<harlowja> let me get that in a more offical place, ha
<harlowja> ok smoser added nice handy commit message in https://code.launchpad.net/~harlowja/cloud-init/cloud-init-net-refactor/+merge/293957 :-P
<rharper> and at the risk of exposing my ignorance;  a few of the render methods are @classmethod ;  but it doesn't appear that they're used outside of other methods within the class; what's that doing ?
<harlowja> ya, more or less just a thing i've gotten used to, no need to pass self around if not using it for those
<harlowja> and if don't need any class attributes, then just doing staticmethod
<harlowja> can make them all instance methods, but got in the habit of not doing that and using the right decorator
<rharper> I see, you didn't explicitly need to reference anything in the instance s?
<harlowja> nah
<rharper> gotcha
<rharper> w.r.t quoting, my reading on ConfigParser and INI is that no quoting should be needed; anything after the Variable= is included in the value ...
<harlowja> hmmmm, i thought most of these sysconfig files were just 'sourced' into scripts
<rharper> oh
<rharper> hehe
<rharper> I forgot
<rharper> the bash scripts
<rharper> you're right for sysconfig
<harlowja> ya, i didn't think sysconfig stuff was very smart, lol
<harlowja> and it all eventually gets parsed by some shell scripts, lol
<harlowja> so without doing quoting things go bad (especially with whitespace)
<harlowja> ie like https://gist.github.com/harlowja/5f9b3008d20d2668e8bcfc28eb491955#file-gistfile1-txt-L102
<harlowja> why it was done like this, who knows
<rharper> yeah
<rharper> I was just confused
<rharper> sysconfig is huge legacy
<harlowja> def
<rharper> at the same time, the guts of ifupdown are ... harder to read than the in-line awk in sysconfig so no leg to stand on there
<harlowja> lol
<harlowja> ya, it's all one big mess imho
<harlowja> guess thats why systemd :-P
<rharper> one mess + second mess => systemd ?  not sure what we expect then
<rharper> magic ?
<harlowja> :)
 * rharper sprinkles some systemd magic around 
<rharper> I do know that we'll now be told there _is_ one TRUE way
<rharper> whether that's right or not; at least you have an answer
<harlowja> ha
<harlowja> ya
<harlowja> mgagne if u get a chance can u look over https://gist.github.com/harlowja/d63a36de0b405d83be9bd3222a5454a7
<harlowja> its my creation, based off in part https://raw.githubusercontent.com/mgagne/cloud-init-fedora-pkg/epel7/cloud-init-0.7.5-network-info-support.patch + the refactored stuffs
<harlowja> pretty sure its right, and u can just feed it a openstack json file and see what is output
<harlowja> $ python sysconfig.py test.json
<harlowja> for example
<dmsimard> harlowja: neat, I'll look too.
<harlowja> thx
<harlowja> https://gist.github.com/harlowja/d63a36de0b405d83be9bd3222a5454a7#file-sysconfig-py-L406 == the main function that thinks get started from
<dmsimard> harlowja: just to be sure, what version of cloud init am I supposed to be running ?
<harlowja> a unmerged branch ;)
<dmsimard> (for the imports)
<dmsimard> ehhh, how do I set that up ?
<harlowja> https://code.launchpad.net/~harlowja/cloud-init/cloud-init-net-refactor/+merge/293957
<harlowja> that one
<dmsimard> can you zip it somewhere or something ? Afraid my bzr skills are pretty awesome (not)
<harlowja> sure
<harlowja> dmsimard actually u should just be able to run
<harlowja> bzr branch lp:~harlowja/cloud-init/cloud-init-net-refactor cloud-init
<harlowja> that will get u the branch
<harlowja> (into a folder called 'cloud-init')
<dmsimard> ok
<dmsimard> let me see..
 * dmsimard installs bzr ... and shivers
<harlowja> lol
<harlowja> try a few json files (from openstack); let me know if anything looks really crazy (it might be crazy)
<harlowja> since things like https://git.fedorahosted.org/cgit/initscripts.git/plain/sysconfig/network-scripts/ifup-eth are crazy
<dmsimard> harlowja: it's not crazy, it's empty :)
<dmsimard> sec
<harlowja> lol
<harlowja> k
<harlowja> running stuff like $ python sysconfig.py test.json  will just output to stdout btw
<harlowja> it doesn't write anywhere yet
<dmsimard> harlowja: https://gist.github.com/dmsimard/d6eaa8b61f3c601d833b29ef430637da this is what I'm testing against
<harlowja> k let me try
<dmsimard> and this is my output: http://paste.openstack.org/show/498378/
<harlowja> yup, looks like that to me also, ha
<harlowja> ok, let me see where things went here
<dmsimard> I have the full config drive if you want, it may or may not be the one I think I may have provided you last time
<harlowja> oh i know
<harlowja> let me see, def seems like it should have more :-P
<harlowja> one of these other format layers killing the interfaces there
<harlowja> will figure that out
<dmsimard> fwiw that's a mitaka config drive, if it's different or anything, no clue
<harlowja> k
<harlowja> u can try the example in https://bugs.launchpad.net/nova/+bug/1514082
<harlowja> i was trying that one also
<harlowja> as i figure out which layer dropped the info in your file
<dmsimard> harlowja: ah, yup, that one works fine
<dmsimard> harlowja: the whole config drive is @ https://dmsimard.com/disk.config if you want to peek at it.
<harlowja> np, not needed i think, just that one file is what matters here
<dmsimard> you should add my network_data.json to your unit tests in test_network_config_conversions after you figure it out :)
<harlowja> ya
<harlowja> ya, ok, sooo might have to poke rharper on this one
<harlowja> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/net/network_state.py#L239 is puking on your date
<harlowja> primarily because http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/net/network_state.py#L266 keys don't exist
<harlowja> and in the openstack version, they aren't getting translated ...
<dmsimard> I updated my gist to be pretty printed :p https://gist.github.com/dmsimard/d6eaa8b61f3c601d833b29ef430637da
<harlowja> and i'm not quite sure what they should translate to/from
<harlowja> it seems like that code is looking for bridge_interfaces and then interfaces (two sets of interfaces)
<harlowja> where your example only has one :-P
<harlowja> so what does it bridge to :-P
<dmsimard> it's a vm on a linuxbridge environment
<harlowja> right
<dmsimard> the vm doesn't have to configure any bridge
<dmsimard> it's done on the compute node
<harlowja> sure
<harlowja> but from my understanding of that code, its looking for a interface definitions of what to bridge to what else
<harlowja> so i'm not sure the mapping of http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/net/network_state.py#L239 and what openstack provides (is data missing?)
<harlowja> what happens is that your json file first goes into a intermediarty format that is '[{'subnets': [{u'routes': [{u'netmask': u'0.0.0.0', u'network': u'0.0.0.0', u'gateway': u'172.19.3.254'}], u'netmask': u'255.255.252.0', u'type': 'static', 'ipv4': True, 'address': u'172.19.1.34'}], 'name': u'tap1a81968a-79', 'mac_address': u'fa:16:3e:ed:9a:59', u'type': 'bridge', 'mtu': None}, {u'type': 'nameserver', u'address': u'172.19.0.12'}]'
<dmsimard> So you have a network_data.json that's linuxbridge (mine) and the one you provided from the bug is from an ovs setup. It's a pretty good test to ensure both work well.
<dmsimard> The OVS one you linked also has the notion of links
<harlowja> which then gets processed by that code into another intermediary format and then that format is what the sysconfig.py stuff works on
<harlowja> right, something is off/missing in the handle_bridge stuff here
<dmsimard> The links are probably just interfaces on the compute node (bridge tap) and the networks are bridged to these links on the compute node
<harlowja> ya, i'm gonna delegate to rharper on this one, unsure what the expected openstack translation is for that to correctly work
<dmsimard> I'm off for the day, let me know if you find out and I'll poke at it some more .. I could even pitch a few VMs at it with the real thing :)
<harlowja> ya, eventually we'll get there ;)
#cloud-init 2016-05-24
<harlowja> i almost feel like the above bridge stuff would be fixed by just assuming things bridge to eth0 if not provided a port iface to bridge to
<harlowja> or maybe bridges in openstack are something different
<rharper> dmsimard: what's the setup?  the network_data.json you posted is an empty bridge (AFAICT);   is that a running instance?  if so, can you paste ifconfig  ?  I'd suspect that the openstack providing network_data.json is missing something
<cpaelzer> I must admit I don't 100% understand what apt_old_mirrors is for - doc isn't telling a lot and the code neither
<cpaelzer> I'll dig through the commit log for it, but if one knows that part as his best buddy please feel free to share :-)
<rharper> dmsimard:  I see what's going on; thanks for the test case;
<smoser> cpaelzer, when the image is built, you can/probably-should leave apt's '/var/lib/apt/lists/' around
<smoser> that way, the next time someone 'apt-get update' they dont have to pull 20M or whatever it is of data that hasnt changed (the release pocket at least).
<cpaelzer> smoser: thanks, but you are jumping to the decision - I wanted to understand what it is actually for
<cpaelzer> other than "renaming" I don't see what it really does (by my lack of apt knowlegde about that subdir)
<smoser> but since cloud-init picks / writes a different mirror, then your *new* apt-get update is going to miss all that cache
<cpaelzer> ah
<smoser> so it renames files from the old mirror name to the new mirror name
<smoser> but you have to know the old mirror (or at least i allowed you to configure that)
<cpaelzer> that makes sense, so if old/new have lot of same content it will not pull all of it then
<smoser> probably you could figure it all out and dtrt, or possibly it was more difficult to do that.
<cpaelzer> I'd drop this feature (apt_old_mirror) then
<smoser> ?
<smoser> oh. yeah, out of curtin.
<smoser> drop
<cpaelzer> exactly - thanks for the explanation
<smoser> is that waht you meant ?
<smoser> it is useful for cloud-init still
<cpaelzer> yep
<smoser> although i admittedly don't know if it ends up being used
<cpaelzer> oh I understand the cloud-init use case now that I understand what it really does :-)
<smoser> the cloud-images may actually have apt-get clean run on them
<cpaelzer> I can tell you that the doc is rather ... nonexisting
<smoser> that is pointless
<cpaelzer> it isn't even listed in examples or such
<smoser> as anything in the release pocket is static so you should really never throw that away
<smoser> :)
<smoser> well, some things aren't really made to be configured.
<smoser> clearly documentation is good. and severlely lacking.
<smoser> but that is'nt really something that a end user would be expected to configure. so its less important.
<smoser> its just configurable rather than having a hard coded value
<smoser> cpaelzer, https://code.launchpad.net/~paelzer/cloud-init/test-apt-source/+merge/294521 has merge conflicts
<smoser> and did you think at all about the sucky fact that curtin and cloud-init's merging algorithms differ by default ?
<cpaelzer> smoser: I did think about the sucky fact (that sounds wrong) and decided to not overhaul any majore piece especially since it would change behaviour
<cpaelzer> smoser: but - I added it right away in the example
<cpaelzer> smoser: so that anybody that starts at doc/examples would be made aware that to get an equivalent mergeing he would need to set it
<cpaelzer> smoser: merge conflicts in bzr is the same as in git - i.e. something else got merged that now conflicts - right ?
<smoser> yeah.
<smoser> bzr merge ../trunk
<smoser> bzr conflicts
<smoser> bzr resolve foo
<cpaelzer> smoser: I'll look into it and (try to avoid) hate bzr :-) after I finished my config example for curtin
<cpaelzer> smoser: will upload updates later on then (hoepfulyl no too complex conflicts)
<smoser> cpaelzer, fwiw, i just checked in a xenial container and the image *does* have /var/lib/apt/ populated, so that cloud-init's renaming those should result in not having apt-get update re-download static data.
<cpaelzer> smoser: cool will look into that, but I'd expect for curtin to only hardcode the paths then (no config option)
<smoser> cpaelzer, you can drop that from curtin if you'd like
<smoser> it is probalby useful
<smoser> but a small-ish optimization compared to installing an operating system
<rharper> dmsimard: https://code.launchpad.net/~raharper/cloud-init/trunk.network-data-bridge/+merge/295591
<dmsimard> rharper: what do you need to know for linuxbridge ? What a network_data.json looks like with multiple networks and ports?
<rharper> dmsimard: yeah
<dmsimard> I can get you that.
<rharper> and if it sets any bridge properties
<rharper> under the link section
<dmsimard> Well, the json I provided is a linuxbridge environment so I'm not sure what else could be there
<rharper> currently in that patch I'll copy in any keys that start with bridge;  the rackspace config I've seen had bond related stuff that I've not seen elsewhere and vlan'; but I lack a deep list of network_data.json examples from real clouds
<rharper> dmsimard: you can set things like bridge_stp, bridge_fd, bridge_hello
<rharper> the most interesting would be a set of interfaces to add to the bridge
<dmsimard> Hmm, perhaps mgagne would have something for bonded networks
<rharper> I've got a good bond example from rackspace a while back; but others are welcome
<dmsimard> Yeah Rackspace tends to do things slightly differently, it's best not to rely just on them
<rharper> sure
<dmsimard> I'll get you a config drive with multiple networks and ports later
<rharper> thanks!
<cpaelzer> smoser: once the conflicts are understood and fixed in the file - do I do bzr commit followed by bzr resolve - or will bzr resolve alone be what I want?
<cpaelzer> docu reads like the second
<smoser> bzr resolve
<smoser> and then bzr commit
<cpaelzer> smoser: thanks
<cpaelzer> ah in the log I see now how this works (hopefully)
<cpaelzer> it is not rebasing mine onto current upstream (like git) but adding a merge commit on top of mine so that they match again in that regard
<cpaelzer> at least that is how it looks like and that also explains why the diff of that last commit was so big it was the delta of the merge essentially
<mgagne> smoser: is there anything I can do to help with bug #1577982 ?
<smoser> mgagne, i have told some people that i'll have something in yakkety by eod my friday.
<mgagne> cool, thanks for the follow up
<smoser> mgagne, would you be able to consume a branch for me ?
<smoser> if i send you a link ?
<smoser> (i dont have it now, but if i point you at a bzr branch woudl you be able to test that or do you need a ppa ? or .... how can i get you to help me on this)
<mgagne> smoser: I don't know how to build from bzr
<smoser> would a ppa build be sufficent ?
<mgagne> yes
<smoser> ok. thanks. i'llj ust ping you here when i have somethign
<mgagne> sure, thanks!
<harlowja> rharper yt
<rharper> harlowja: here
<harlowja> hey, so i was running dmsimard example openstack network json file through https://gist.github.com/harlowja/d63a36de0b405d83be9bd3222a5454a7 (updated)
<harlowja> but am wondering what your expected parts for bridges are
<rharper> dmsimard was going to see about getting another example with multiple networks and such;
<harlowja> k
<dmsimard> yeah I haven't got around to it yet
<dmsimard> extinguishing some fires first
<harlowja> i can make the bond stuff work, just when i run that
<rharper> harlowja: but what I was looking for was whether there would be additional bridge_  options set in the link
<harlowja> https://gist.github.com/harlowja/358358fc41685874a12f4db809504c19
<rharper> harlowja: I have a bond example from rackspace if you're interested
<harlowja> hmmm, bridge doesn't have the params the network_state.py stuff seems to require
<harlowja> maybe shouldn't be required?
<rharper> harlowja: I posted a MR with the fix
<harlowja> MR?
<harlowja> mapreduce
<harlowja> lol
<rharper> bah
<rharper> MP
<rharper> <rharper> dmsimard: https://code.launchpad.net/~raharper/cloud-init/trunk.network-data-bridge/+merge/295591
<harlowja> k
<rharper> I hadn't see a network_data.json with type: bridge before
<dmsimard> rharper: ok, getting you that network_data.json now.
<dmsimard> rharper: eta ~40 mins or so while the environment is spawned
<rharper> dmsimard: cool
<harlowja> smoser good to go with https://code.launchpad.net/~harlowja/cloud-init/cloud-init-net-refactor/+merge/293957 ?
<smoser> what is usedevelop ?
<smoser> and i can't use py26 by default.
<smoser> as there simply isn't a py26 anywhere on ubuntu :)
<harlowja> that's ok :-P
<smoser> other than 'deadsnakes'
<harlowja> usedevelop just is a time saving thing, in that it doesn't do a full pip install of cloud-init
<harlowja> but uses symlinks
<harlowja> i can remove that if needed
<smoser> and you're using some magic for tox that is i think not available in 14.04
<harlowja> shouldn't matter
<smoser> the 'py26' magic
<harlowja> ??
<smoser> hm.. maybe not
<harlowja> http://tox.readthedocs.io/en/latest/config.html#confval-usedevelop=BOOL
<smoser> odd
<harlowja> i can turn that off though, not that big of a deal
<smoser> yeah, that doesnt' seem too bad.
<smoser> but why chage 'python -m nose' to nosetests ?
<smoser> i think we adtually want {basepython} -m nose
<harlowja> think that didn't work on 26
<harlowja> let me double check
<harlowja> $ python -m nose
<harlowja>   /home/jxharlow/.venv/bin/python: nose is a package and cannot be directly executed
<harlowja> maybe this was something special on 2.7 + ?
<smoser> probably. but how did you drop the tesenv:py26
<harlowja> kk, i can just add the special case for 26
<harlowja> reposting in a sec
<harlowja> ok, reposted, and py26 still work, so goodie
<harlowja> and 27 it seems to, so even better
<harlowja> :-P
<harlowja> https://github.com/harlowja/remote_tox (my tool for testing these on different envs)
<smoser> harlowja, so far in curtin we've even staye daway from xi
<smoser> six
<smoser> xi = 11. :)
<smoser> harlowja, you dropped if PY26 ?
<smoser> i'm not opposed to being *able* to run tox -e py26
<smoser> i just cant have it in the default 'tox'
<smoser> as then it errors because ENOINTERPRETER and you can't distinguish that as easily from ETESTFAIL
<harlowja> smoser  unittest2 seems to handle that all just fine for tests
<smoser> oh. i see.
<harlowja> so no more need for crazy if PY26: <testcrap>
<harlowja> will take 26 out of default tox list
<smoser> harlowja, some 'from cloudinit import in cmdline.py'
<smoser> with better quoting that might make sense.
<smoser> harlowja, some 'from cloudinit import' in cmdline.py
<smoser> and would we not want 'from . import' ?
<smoser> or is that a py27 thing too
<harlowja> just a thing that i guess is something i've gotten used to from openstack land
<smoser> what do you mean ?
<harlowja> http://docs.openstack.org/developer/hacking/#imports
<harlowja> i've always thought releative imports a little pita, i guess openstack folks do also, so ya, idk
<smoser> why?
<harlowja> less clear i think (in my view)
<harlowja> but maybe its from my java days, idk
<harlowja> lol
<harlowja> but if we want to just say meh, that's fine also
<smoser> well, if we're expecting to be able to 'export' that module so that it is free of cloud-init, then 'import cloudinit' is kind of a fail path
<harlowja> fixed via tiny little script :-P
<harlowja> replace 'from cloudinit.net' --> 'from <thing>.net'
<harlowja> could just make that thing a lib in the first place :-P
<smoser> :-P~
 * harlowja is trying to shutdown a openstack project that did this kind of copy/paste for years, lol
<harlowja> finally getting rid of it, lol
<dmsimard> rharper: hai
<dmsimard> https://gist.github.com/dmsimard/d5b14f5cba05307a3008315d11520e26
<smoser> into its own proper library ?
<harlowja> ya
<dmsimard> full disk.config = https://dmsimard.com/disk.config
<harlowja> many proper libs
<dmsimard> harlowja: ^ multi-network/multi-port vm
<harlowja> the project i'm a PTL of started off with this copy/paste junk
<harlowja> dmsimard nice
<harlowja> complex examples ftw
<harlowja> with bridges :-P
<dmsimard> that was surprisingly harder to set up than I thought, the deployment we're testing only has one network, not two
<dmsimard> so I had to do some fiddling by hand
<harlowja> ya, i'm hoping the sysconfig stuff i have here handles most of those
<harlowja> cause it gets whacky real quick, lol
<harlowja> smoser  https://wiki.openstack.org/wiki/Oslo#Incubation
<harlowja> lol
<dmsimard> I'll have that dev environment for 24 hours until it self destructs (reaped by auto reprovision) so let me know if you need anything else from it
<harlowja> kk
<dmsimard> I think that gist and disk.config should have what you need
<harlowja> as many examples u can make would be sweet :-P
<harlowja> once liberty is deployed where i'm at that should help also
<harlowja> but the more the merrirer
<dmsimard> that example is mitaka
<harlowja> k
<rharper> dmsimard: thanks!
<dmsimard> rharper: np, like I said you have 24h if you need anything else :P
<rharper> dmsimard: that looks fine; the MP I posted earlier parses that just fine;  I was mostly interested to see if any other bridge related parameters might get set in the "links" entries of type: bridge; they look the same here so unless you know of an Openstack that might twiddle with bridge settings like forward delay, stp or other things like that (and have a network_data.json with those values) I think we're good with th
<rharper> e current MP to support things.
<rharper> dmsimard: hrm, actuall, your network dump from in the guest is interesting;  why doesn't the network_data.json have a the ethernet devices listed ?
<harlowja> ya, it almost seems like there is something missing :-P
<dmsimard> ethernet devices, like eth0 and eth1 ?
<harlowja> are there ethernet devices just assumed to exist
<harlowja> ya
<rharper> right, so I'd think we'd see bridge_interfaces = list_of_link_id's
<rharper> dmsimard: right
<dmsimard> I guess it's made agnostic through network0 and network1
<harlowja> eth<magic>
<dmsimard> but I don't know
<harlowja> ya, the dump of /etc/sysconfig stuff would be useful, or whatever else u got 'ifconfig'
<rharper> the network describes the ip network config of the link, the link is one of 'ethernet', 'bond', 'vlan', and now 'bridge'; in the vlan and bond case, the "link" indicates if it consumes other links
<dmsimard> harlowja: that particular VM was setup through dhcp, not with your branch of cloud init or anything
<dmsimard> but the network_data.json and the rest of the info are relevant
<rharper> http://paste.ubuntu.com/16664758/  for example this one
<rharper> dmsimard: something did the bridge config there ; I think that's what's the interesting part
<dmsimard> rharper: from what, the brctl output ? That's from the compute node, not the guest
<dmsimard> that's handled by the neutron linuxbridge agent
<rharper> ok
<rharper> well, I'm somewhat confused then, what does networking look like wihtin the guest that got the network_data.json  ?
<dmsimard> let me log in to it, sec
<rharper> cool
<dmsimard> rharper: what do you want to know exactly ? which files, commands, etc
<smoser> harlowja, is it appropriate to call a static method with self.methodname ?
<harlowja> ya
<harlowja> it is
<smoser> ok. so you dont have to do something like self.__class__.methodname
<harlowja> nopers
<harlowja> same for class methods
<nacc> smoser: in python? it's allowed in publishe documentation, i just looked this up for the importer :)
<harlowja> man, gonna make me cut out six :-P
<rharper> dmsimard: ls -al /sys/class/net and ifconfig -a would be useful ; brctl show would be useful from within in the guest
<rharper> harlowja: hehe
<smoser> i knew it would work, i just didn't knwo if it was bad form
<dmsimard> rharper: there are no bridges in the guest
<harlowja> :-/
<harlowja> openstack is weird
<harlowja> lol
<rharper> right, I didn't think so; but the network_data.json is supposed to be exported into the guest;  it appears then like the ovs type; type bridge should be interpreted as a ethernet ... I wonder why it's emitting type bridge with it's not creating one in the gues ?
<rharper> harlowja: I think the network_data/metadata is partially implemented
<harlowja> possible, https://github.com/openstack/nova/blob/master/nova/virt/netutils.py#L172 is the code for this
<harlowja> but ya, i'm unsure if its busted or missing stuff (especially for bridges?)
<dmsimard> rharper: why are you expecting bridges inside the guest ?
<rharper> the previous example, and the rackspace one which does bonds of ethernet devices and adds vlans is functional
<dmsimard> the guest has eth0, that's it, there's no bridges to be had
<rharper> dmsimard: because the linkls type is bridge, this is what the guest would get in the network_data.json
<harlowja> dmsimard what OS ?
<dmsimard> the vtap is bridged on the compute node
<rharper> ie, "make me a bridge"
<harlowja> if its rhel, i can tell u why its not implemented :-P
<harlowja> (cause the gist i have is that impl, lol)
<dmsimard> the networking on that VM wasn't setup by cloud init, it was setup by dhcp
<dmsimard> I know, I'm the one poking you about rhel/centos support ;p
<rharper> dmsimard: I'm aware of how the host networking and VM networks are configured;  I'm trying to understand why the network_data.json would say type: bridge when it's really just type: ethernet (aka, do dhcp on the guest ethernet device)
<dmsimard> rharper: some guest info https://gist.github.com/dmsimard/d5b14f5cba05307a3008315d11520e26#file-guest-info-txt
<rharper> dmsimard: right, network_data.json can be used to confer a network configuration into the guest
<dmsimard> rharper: the network_data.json wasn't used by cloud-init to configure networking
<rharper> ie, emit a network config dynamically in the guest, it could be , run dhcp on eth0, or configure eth0 and eth1 as a bridge, etc.
<dmsimard> so I don't know if cloud-init would have done anything
<rharper> it does now
<dmsimard> not for centos :p
<rharper> right, which we're trying to fix
<rharper> harlowja 's branch and all
<dmsimard> though, I *can* spin a ubuntu vm
<dmsimard> with the same network config
<dmsimard> if you'd like
<rharper> it won't matter
<harlowja> ya, don't try centos, ha ;)
<rharper> the concern is that the openstack metadata is saying the links for the guest are type bridge, which the back end of the device (bridge on the host, or tap pair) doesn't matter; the guest has a virtual nic, and it should run dhcp;  so we can either assume that type: bridge means an  ethernet device in the guest (likew do for type: ovs)
<rharper> bah, I've got relocate bbiab
<dmsimard> type bridge is probably where you would see type ovs ?
<dmsimard> so it probably just stands for "this is a linuxbridge type thing, not an ovs type thing"
 * dmsimard shrugs
<harlowja> ya, so maybe the bridge isn't what we think a bridge is, lol
<dmsimard> pretty sure it can be safely ignored
<larsks> Let me chime in: I can tell you definitely it can be ignored.  The guest doesn't care if the underlying l2 infrastructure is using ovs, linuxbridge, or something else.  From the perspective of the guest, it's just an interface.
<dmsimard> more than that, the whole links part is probably not relevant
<dmsimard> unless, I don't know, it's a baremetal use case or something like that
<dmsimard> I don't know much of cloud-init usage beyond openstack VMs
<dmsimard> mgagne: are ironic machines using cloud-init for the network config ? (bonding, vlan, etc)
<harlowja> ya, there's already code that discards the various types, but weird that it exists in the first place :-P
<smoser> dmsimard, there are
<smoser> not trunk cloud-init at the moment, but we do hope to have that
<dmsimard> smoser: have what ?
<smoser> rackspace baremetal use cloud-init and config-drive in bare metal systems for network
<larsks> dmsimard: I think network config is via puppet.  I guess mgagne can confirm...
<mgagne> dmsimard: yes, there is no other way around for centos7, can't read complex debian bonding config
<smoser> but they use out of tree.
<larsks> Oh, well, there you go.
<smoser> we would like to have that functional all in cloud-init proper
<dmsimard> ok, I wasn't sure if it was done by IPA or cloud-init, now we know :)
<mgagne> we don't use puppet in customer's instances
<mgagne> hold on
<mgagne> reading backlog
<mgagne> just to make sure I'm answering the right question
<smoser> is mgagne rackspace ?
<mgagne> I'm Internap
<mgagne> or iWeb
<smoser> JayF, woudl knwo more about the rackspace for sure.
<JayF> You should be careful telling people I know things
<mgagne> we talking about baremetal delivered to customers?
<JayF> You'll just lead them to disappointment :)
<JayF> me and mgagne know each other already
<JayF> natorious: do you have the link to our cloud-init fork for mgagne to check out?
<dmsimard> mgagne: I guess the question is more around.. is the "links" part of the network_data.json relevant at all -- It doesn't seem to be for guest virtual machines (since it's l2 knowledge from the compute node) but we were wondering if it was relevant for bare metal.
<mgagne> https://github.com/jayofdoom/cloud-init-fedora-pkg ?
<dmsimard> mgagne: i.e, referring a linuxbridge network_data.json from mitaka: https://gist.github.com/dmsimard/d5b14f5cba05307a3008315d11520e26#file-network_data-json
<JayF> mgagne: no, the one we're using now is in bazaar, natorious will know where it is
<smoser> :)
<smoser> some day. we will be done of czr
<smoser> bzr
<smoser> i promise
<mgagne> dmsimard: what specifics of links are you referring to?
<dmsimard> mgagne: would something like bonding be represented there, for example ?
<natorious> its https://code.launchpad.net/~nathan-house-0/cloud-init/cloud-init
<natorious> the pkg build scripts are all linked to the working bzr branch so you would want to fork it if your planning on doing distro builds
<mgagne> https://bugs.launchpad.net/cloud-init/+bug/1577982
<smoser> harlowja, https://code.launchpad.net/~harlowja/cloud-init/cloud-init-fix-test-times/+merge/294854
<mgagne> http://paste.openstack.org/show/498744/
<mgagne> content is here
<smoser> you did not remove nose-timer
<mgagne> let me check
<mgagne> hold on
<mgagne> Im not sure it's the right ocntent
<harlowja> smoser  oh, working on it :-P
<mgagne> nope, not the right one it seems
<mgagne> that's strange, I thought it was bonding
<mgagne> ok, let me get one
<copumpkin> is it safe to move keys back and forth between userdata and cloud.cfg?
<copumpkin> or if not, how can I make things that I would normally write in userdata happen regardless of userdata?
<mgagne> ok, issue wasn't with bonding, just ubuntu16.04, wrong bug
<smoser> mgagne, if you want see what i'm thinking: http://paste.ubuntu.com/16665427/
<rharper> dmsimard: ok; I think we have two cases for when network_data.json is being used;  in your case; (as with the ovs case I've seen) the network metadata refers to how the computenode has configured networking;  bridge, or ovs; however, it's not currently exporting any guest configuration (though it could, for example instead of ipv4_dhcp; it could be ipv4 and emit the static ip config it wanted to assign).  I'll rework th
<rharper> e MP to treat the type: bridge like type:ovs, which is to say that we will not attempt to assemble a bridge in the guest, rather each link of type: bridge means we have an ethernet device
<dmsimard> rharper: mgagne gave me http://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/metadata-service-network-info.html#rest-api-impact
<dmsimard> It looks like a good sample
<rharper> dmsimard: yeah, and notice it doesn't have the linuxbridge example you have
<mgagne> and https://github.com/openstack-infra/glean/blob/master/glean/tests/fixtures/liberty/mnt/config/openstack/latest/network_info.json
<rharper> but I think the right of it is as I stated above;  for type: ovs and type: bridge, we merely need to translate that to ethernet interfaces in the guest (rather than attempting to assemble a device of type: X in the guest)
<rharper> now that's contrasted with the other types which is somewhat in conflict if network_data.json is meant to describe the guest network
<rharper> so, I'm not 100% clear on what the right thing is if in the future we wanted to describe how to configure a bridge in the guest with the ethernet devices on the VM
<mgagne> there is no bridge on the guest
<rharper> right
<rharper> I understand that;
<mgagne> anything related to bridge is leaked details from internal implementation on the hypervisor, not the guest
<rharper> right; so I think the network_data.json you have should have type: phys instead of type: bridge in it
<rharper> if we agree that network_data.json is describing guest networking
<smoser> so that is actually leaking ?
<smoser> copumpkin, for the fast majority of things, yes.
<smoser> some things can't really take affect in the user-data but have to be in cloud.cfg.d
<dmsimard> rharper: perhaps the nova developers are a better resource to better understand what goes where and why
<copumpkin> smoser: like the list of modules for init/config/final? :)
<dmsimard> I don't think the network_data.json I gave you is inherently wrong
<smoser> the list of modules for config and final can, right? or are you reminding me of a bug, copumpkin
<smoser> they should be able to :-(
<dmsimard> It's just that the links portion is more than likely just not relevant for guest networking, although it could be for bare metal network (vlan, bonding, etc.)
<rharper> dmsimard: well, it's ambiguous; the bridges are clearly on the host; and not part of the guest
<smoser> so lets fix that.
<smoser> in theory they can. in practice maybe not.
<harlowja> smoser ok, whenever u ready https://code.launchpad.net/~harlowja/cloud-init/cloud-init-net-refactor/+merge/293957 has mini six like module now, and using the imports that should be easier to 'extract that net folder out'
<dmsimard> rharper: maybe it's a bug and shouldn't be exposed to guests, I don't know
<harlowja> rharper dmsimard ya, weird stuff, lol
<copumpkin> smoser: oh, interesting, I didn't realize that. Is there some way to disable that?
<copumpkin> not reminding you of a bug, nope!
<rharper> dmsimard: yeah from my reading of the spec; it's meant to describe networking on the "machine" executing cloud-init and fed the network_data.json
<harlowja> if it gets host info, that's odd lol
<rharper> so the generator of the json shouldn't include things that aren't meant to be configured/rendered where the json is consumed;
<smoser> mgagne, http://paste.ubuntu.com/16665693/ is where i am on making those changes described in the other paste. nothing tested yet, just looking at making the changes.
<dmsimard> take it up to the nova guys, I have no clue :(
<rharper> dmsimard: sure;thanks for the help
<smoser> copumpkin, disable what ?
<smoser> it sure would seem both broken and wrong to tell the guest what the host's networking configuration looks like
<smoser> even if only partially
<rharper> harlowja: I'm leaning on bug here; mostly because it's an under consumed/tested spec.
<copumpkin> smoser: the ability to change loaded modules on the fly. I'd like to force userdata to only use a handful of modules
<harlowja> https://gist.github.com/harlowja/d63a36de0b405d83be9bd3222a5454a7 (the updated next thing after that refactor branch gets merged)
<harlowja> rharper ya
<mgagne> I'm not sure I'm qualified to review those changes =)
<smoser> copumpkin, i dont think you can disable it. generally the goal is that the user-data wins.
<smoser> they're the ones who say waht goes.
<copumpkin> hmm
<copumpkin> that's unfortunate
<smoser> they're the "owner"
<rharper> smoser: did we get our network_data.json turned on in serverstack ?
<mgagne> dmsimard: there was a bug in earlier version where network_data.json wasn't available in the versioned path (only latest/)
<harlowja> rharper i can send out a mail to the openstack dev list, if we can't figure out wtf is going on here :P
<smoser> rharper, checking
<rharper> harlowja: ok; I suppose I should join that ml
<harlowja> (a non ambiguous network_json)
<copumpkin> smoser: I guess I could delete the cc_* modules I'm not using? :P
<harlowja> rharper  if u want, ha
<rharper> harlowja: I think the spec is pretty clear
<rharper> harlowja: well, not really but I don't want you to have to be a proxy unless you want
<smoser> yeah. you could. but they could put them back in :).
<copumpkin> what if I took out write_file?
<harlowja> rharper :)
<smoser> copumpkin, its definitely never been a intended goal
<smoser> i woudlnt be entirely opposed to having the system be able to limit things.
<smoser> but i've always been really focused on the other side... amking the user-data able to make me do anything i want with a system.
<copumpkin> the main idea would be to provide a well defined interface with a few knobs that the image maker wants tunable
<copumpkin> and otherwise making it fairly hard to do arbitrary work on the machine
<copumpkin> so the image maker might provide a custom cloud-init module that's enabled
<copumpkin> and possibly a couple of the standard ones, but write_file, runcmd, and various other "arbitrary power" ones would be disabled
<smoser> rharper, no i dont think so. you should bother openstack team
<smoser> http://paste.ubuntu.com/16665816/
<rharper> smoser: cool!
<smoser> copumpkin, yeah, i can see the point. but that has just never been a focus.
<copumpkin> fair enough
<smoser> i ahve to run.
<copumpkin> thanks for the help!
<rharper> harlowja: ok, I've joined; feel free to lob something out there and CC me
<harlowja> k
 * rharper will attempt to not look like an fool when responding 
<harlowja> meh, its ok if u do
<harlowja> lol
<rharper> haha
<harlowja> smoser https://code.launchpad.net/~harlowja/cloud-init/cloud-init-fix-test-times/+merge/294854 fixed up
<harlowja> enjoy faster testing lol
<harlowja> damn retries :-P
<rharper> harlowja: what's the net speed up there?  could you include that in the commit message
 * rharper likes nose timer
<harlowja> like u really want to know my netspeed?
<harlowja> fast i guess?
<harlowja> lol
<rharper> and setupClass
<rharper> harlowja: haha, the net speedup with the timeout=5
<rharper> versus trunk
<harlowja> oh
<rharper> heheh
<harlowja> very good vs shitty
<harlowja> but sure i can get the number
<harlowja> lol
<rharper> cool
<harlowja> rharper https://gist.githubusercontent.com/harlowja/b015a0751a954b043edb4206768085ec/raw/2ff148bd3c965bcbed5c1f43a532dc998d9219a7/gistfile1.txt let me know if makes sense (or i should add anything else); u can respond with additional questions i guess to (just i don't want to sound stupid in not understanding the issue, ha)
<harlowja> weird bridges :-P
<rharper> harlowja: I think core  questions 1)  confirm that given link to the spec (http://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/metadata-service-network-info.html#rest-api-impact)  that the info in the network_data.json represents *guest* network config (either baremetal for ironic, or a VM)
<harlowja> k
<rharper> if (1) is true; then the example you present is leaking *host* network config
<harlowja> ok, u want to followup with those questions ;)
<rharper> sure
<harlowja> kk
<harlowja> rharper  http://lists.openstack.org/pipermail/openstack-dev/2016-May/095835.html sent, will look for followup, tag team emails
<harlowja> lol
<rharper> k
<harlowja> maybe br == phy and they just want the bridge flag set (in sysconfig terms)
<harlowja> not quite sure :-P
<harlowja> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/s2-networkscripts-interfaces_network-bridge.html
<harlowja> not sure what the equivalent is in ubuntu
<rharper> harlowja: I think it's just under implemented; it clearly could have used the type: phys
<rharper> and the ironic use-case would have used type: bridge to mean, make a bridge
<harlowja> ya, perhaps
<harlowja> also put times on https://code.launchpad.net/~harlowja/cloud-init/cloud-init-fix-test-times/+merge/294854
<harlowja> 12.376s vs 1m12s
<harlowja> ;-P
<harlowja> :-P
<rharper> \o/
<harlowja> ha
<rharper> data-driven commits!
<rharper> how can smoser say no now!
<harlowja> i'll find smoser if he says no
<harlowja> in his basement
<harlowja> lol
<rharper> hehe
<rharper> relocating, bbiab
<harlowja> k
<harlowja> rharper have u seen https://github.com/openstack-infra/glean/blob/master/glean/cmd.py#L344
<harlowja> another renderer of this stuffs
<rharper> harlowja: looking
<rharper> harlowja: also; we may want to employ a version'ed converter if this type: bridge and type: ovs are bugs;  in current version or older, we can translate those thoes to type: physical for the network-config yaml
<harlowja> ya
<rharper> harlowja: that looks like a subset of our eni writer
<harlowja> ya, cough cough library
<harlowja> that all can share
<harlowja> cough cough
<harlowja> lol
<harlowja> rharper if that refactor goes in, shouldn't be to hard to make the openstack converter versioned to handle that kind of crap
<smoser> harlowja, the other slow test is ec2
<smoser> and i think its just wrong logic
<harlowja> hmmm, could be
<smoser> test_userdata_fetch_fail_server_not_found is the test
<harlowja> might be another one that is retrying
<smoser> i think the return of _skip_retry_on_codes is just wrong
<harlowja> ah, could be that to
<smoser> its supposed to say that 404 on user-data is ok.
<smoser> (and should not be retried)
<harlowja> who made that crap, lol
<smoser> http://paste.ubuntu.com/16668369/
<smoser> that makes it faster
<smoser> and faster always means better
<smoser> right?
<harlowja> yup
<harlowja> lol
<harlowja> delete all the code
<harlowja> lol
<smoser> then the other only slow one is test_seed_runs
<smoser> which does for i in range(1, 50)
<smoser>    for j in range (1, 50)
<smoser> makes a bunch of dictionaries (2500)
<harlowja> u trying to remove the speedup loop
<harlowja> lol
<smoser> so now http://paste.ubuntu.com/16668426/ and we're down to 2.6 seconds here locally. with the longgest one still being test_seed_runs
<harlowja> cool
<harlowja> btw smoser i think the shade guys would be up for considering using a tiny-lib of this net stuff ;)
<smoser> shade guys ?
<harlowja> sorry i mean glean
<harlowja> that other cloud-init thingy
<harlowja> the majority of https://github.com/openstack-infra/glean/blob/master/glean/cmd.py  is networking_json stuff
<harlowja> https://github.com/openstack-infra/glean/blob/master/glean/cmd.py#L701 ...
<harlowja> and looks like that handles (to some degree) sysconfig, eni, gentoo ?
<smoser> ./tools/export-net --toplevel=curtin | tar -C ~/your-project -xf -
<harlowja> ya ya, they don't want to vendor though i think
<harlowja> which is what that would be
<smoser> yeah
<harlowja> but they might be up for using a tiny-lib
<harlowja> or that's what i hear :-P
<harlowja> soooo how bout that
<harlowja> lol
#cloud-init 2016-05-25
<copumpkin> what does the ~ mean in cloud.cfg values?
<copumpkin> my machine comes with a cloud.cfg with "ssh_genkeytypes: ~"
<copumpkin> as well as "syslog_fix_perms: ~" and a couple of other instances
<copumpkin> oh, it's a YAML construct
<smoser> copumpkin, yeah. yaml has some nice things
<copumpkin> yeah, somehow I've been writing yaml for ages and never came across the tilde :)
<smoser> i didn't know that '~' is same as null
<copumpkin> yeah, not sure who produced the cloud.cfg with it in there
<copumpkin> it's the default cloud.cfg for centos AMIs on ec2
<smoser> [somewhat related] anchors are very useful in yaml
<copumpkin> yeah, I've dealt with those
<copumpkin> although I start preferring jsonnet or nix when it comes to that
<smoser> http://paste.ubuntu.com/16686200/
<smoser> i dont know jsonnet  or nix
<copumpkin> think json/YAML withs slightly different syntax, explicit variable binding, and functions
<copumpkin> makes it very pleasant to do repetitive things
<copumpkin> http://jsonnet.org/ and http://nixos.org/nix
<smoser> yeah, jsonnet looks nice.
<smoser> nix is a package manager ?
<smoser> how does that have to do with json or yaml ?
<copumpkin> yeah, but it's also a language with very similar behavior to jsonnet (basically the same thing with a different syntax), which they use to specify all their packages in
<copumpkin> basically a purely functional lazy language for defining config (no side effects, file writing, or any of that junk)
<copumpkin> I actually prefer it to jsonnet, but it's a less obvious match than jsonnet because nobody really thinks of it as a standalone language
<harlowja> rharper not much response on that thread, thinking i'll just go ahead and make a versioned openstack 'converter'
<rharper> harlowja: I think so; someone we have 3 reasonable ppl saying bug
<rharper> so versioned converter makes sense
<harlowja> k
<rharper> in your branch I think
<harlowja> sure
<copumpkin> if I'm writing my own cloud-init module, is there a simple way to query the instance ID?
<copumpkin> I could hit 169.254.169.254 directly, but that doesn't account for the various other ways cloud-init could get an instance ID
<copumpkin> aha, cloud.get_instance_id() looks promising
<harlowja> or 'cat /var/lib/cloud/data/instance-id'
<copumpkin> is there a reason to do that over get_instance_id()?
<harlowja> nah, if u are writing python code modules then probably prefer get_instance_id
<copumpkin> cool, thanks
<rharper> harlowja: on the network_data.json, should we propose a fix to those drivers (ovs and linuxbridge) and have them emit type: vif in their metadata hook ?
<harlowja> in nova u mean rharper ?
<harlowja> would seem to make sense to me
<rharper> yeah or neutron; where ever those backends live
<harlowja> https://github.com/openstack/nova/blob/master/nova/virt/netutils.py#L172 is where the json gets made
<rharper> yeah
<harlowja> seems like https://github.com/openstack/nova/blob/master/nova/virt/netutils.py#L250 just needs to be not just ethernet :-P
<harlowja> but vif.get('type') in ['ethernet', 'vlan', 'vos']
<harlowja> or something like that?
<rharper> the vif .type attribute is the deal
<rharper> yeah, hard to say if the metadata should fix it up or if there's some other attribute that should be set (like guest visible vif_type) ?
<harlowja> agreed
<rharper> the mapping in netutil is easy; not sure if we should push further up
<rharper> in the model
<harlowja> agreed
<rharper> k
<harlowja> can just propse a patch, see if it gets shot down :-P
 * rharper adds something else to poke at 
<harlowja> u or i can, ha
<rharper> haha
<harlowja> rharper  https://gist.github.com/harlowja/12e0a56ff0ee0182cc943f966bef8ec0
<harlowja> should fix that issue
#cloud-init 2016-05-26
<rharper> harlowja: nice
<harlowja> rharper ya, nothing special
#cloud-init 2016-05-27
<smoser> mgagne, i just pushed https://launchpad.net/~smoser/+archive/ubuntu/cloud-init-dev
<smoser> err.. pushed to that ppa
<smoser> its from this branch
<smoser> bzr+ssh://bazaar.launchpad.net/~smoser/cloud-init/trunk.fix-networking/
<smoser> it should get you closer
<smoser> the thing left is getting the device renames done. per the network config that was proviced
<smoser> provided
<smoser> so your example network_json gave 'eth0' as the names
<smoser> but i dont have code to envofrce that on first boot
<smoser> it should work after you did 'update-initramfs -u -k all' and reboot
<smoser> but clearly thats not ok.
<kodokuu> Hi, is it possible to add ssh key without start ssh service with cloud init ?
<kodokuu> in cloud env like openstack
<smoser> kodokuu, whats the target os ? tehres not a way to do it explicitly, but with boothook or user-data easily enough to stop it and have it not run subsequently.
<mgagne> smoser: link id is an arbitrary identifier: https://github.com/openstack/nova/blob/master/nova/virt/netutils.py#L273-L274
<mgagne> smoser: it's not THE interface name
<mgagne> http://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/metadata-service-network-info.html#rest-api-impact "Generic, generated ID"
<mgagne> it happens to be eth0 on our setup, we have a custom implementation of the spec which wasn't ready at the time we needed it
<harlowja> so smoser merge https://code.launchpad.net/~harlowja/cloud-init/cloud-init-net-refactor/+merge/293957 next week right??? ;)
<smoser> mgagne, the intent is that it is declaring the name.
<smoser> harlowja, yes. sure. always next week :)
<harlowja> lol
<smoser> but yes.
<harlowja> :(
<harlowja> pinky swear?
<harlowja> http://www.urbandictionary.com/define.php?term=pinky%20swear lol
<harlowja> #2 defintion there is the best
<harlowja> "an eternally binding promise (made by two people hooking their pinky thumbs together), which, if broken, will result in the culprit losing his or her pinky"
<mgagne> smoser: we should get a configdrive generated by latest upstream. it won't be eth0
<mgagne> will test against ppa right now
<smoser> mgagne, thts fine. whatever name it says, is the name that cloud-init will name it to be
<smoser> that  make sense ?
<mgagne> no
<smoser> you're declaring the nic with this mac is to be named this
<mgagne> nowhere it's written that the OS should rename the nic for that provided name
<mgagne> it's a ID, not a name
<harlowja> pinky swearrrr?
<harlowja> lol
<harlowja> don't make me get your pinky
<harlowja> lol
<smoser> hmm
<smoser> http://paste.ubuntu.com/16736838/
<smoser> mgagne, err..
<smoser> http://paste.ubuntu.com/16736864/
<smoser> rharper, ^
<smoser> so we were looking at mgagne's data http://paste.ubuntu.com/16735620/
<smoser> where the name of 'link' was something that could be used as a nicname
<rharper> right, read backscroll
<smoser> but in serverstack we see http://paste.ubuntu.com/16736864/
<smoser> where link tapf6f6405e-15
<smoser> not sure what to do here...
<smoser> "if the name looks like it was intended to be a name, then use it otherwise make up a reasonable name" ?
<rharper> smoser: right, either introduce a specific name field, which is designed to set the name, or someone makes something up; it could be a subset of the id (with limitations due to kernel naming limits);  it could be cloud-init making one up; it could be nothing and let systemd set the name;
<smoser> ok.
<smoser> for each entry in the 'links' array
<smoser>  if there is a 'name' entry, use it as the name
<smoser>  if id matches [a-z][a-z]*[0-9] and is less than length 6 then use it
<smoser>  else, use nicN, where N is the index in the 'links' array
<smoser> ?
<smoser> rharper, ^ mgagne ^
<smoser> i dont love the "if matches" stuff... so we could just ditch that and its both more sensible and predictable.
<rharper> https://blueprints.launchpad.net/nova/+spec/metadata-service-network-info  says 'name' would be part of the response... initially, the spec then replaced name and network_id with just id ... so I dunno;  I don't see why we can't use id as a identifier of nics
<mgagne> why should cloud-init be in the business of renaming interfaces?
<mgagne> blueprint is not the authoritative source of information. spec is what people review, approve and implement.
<smoser> mgagne, so that the user can declare what the name is.
<mgagne> there is no way to provide such information in OpenStack
<mgagne> and OpenStack doesn't care about that detail
<mgagne> it uses DHCP by default without configdrive. there is no way to tell an OS to rename it's interface name with DHCP
<smoser> well, yes. when there is no source of networking truth, the fallback is used. that is sane.
<smoser> but when there is someone that declares things, we want to follow that.
<smoser> you're arguing that there is nothing in openstack that is declaring that, and i think i'm ok with that.
<mgagne> "when there is someone that declares things, we want to follow that." it was never the intent of the link id field.
<mgagne> afaik
<mgagne> but I see rackspace uses it as a fallback which I disagree with.
<mgagne> there used a private implementation which doesn't behave the same as upstream. upstream will never send "eth0" in that field.
<mgagne> JayF: ^
#cloud-init 2017-05-22
<smoser> powersj, I think you opened a bug for https://jenkins.ubuntu.com/server/view/Cloud-init/job/cloud-init-style/38/console right ?
<powersj> smoser: yes
<smoser> link ?
<powersj> https://github.com/PyCQA/pylint/issues/1444
<smoser> powersj, thats a good quality bug report.
<smoser> thank you
<smoser> powersj, or blackboxsw https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324401
<smoser> fiddle.
<smoser> got an extra commit. will fix
<smoser> fixed
<smoser> blackboxsw, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324401
<smoser> the reason i use os.path.sep.join is that os.path.join("/etc", "/passwd") ends up giving you "/passwd"
<smoser> which i think is just horrible
<smoser> $ python -c 'import os, sys; print("sep.join: %s" % os.path.sep.join(sys.argv[1:])); print("path.join: %s" % os.path.join(*sys.argv[1:]))' /etc /passwd
<smoser> sep.join: /etc//passwd
<smoser> path.join: /passwd
<smoser> powersj, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324401
<smoser> do you know... do we re-use .tox in c-i ? will we have to kill something to make it refresh tox ?
<powersj> smoser: not sure I follow. What do you mean refresh tox?
<smoser> tox -e pylint
<smoser> that installs stuff into .tox/pylint
<powersj> ah ok
<smoser> and (i think) subsequent runs of 'tox -e pylint' will *not* freshen .tox/
<powersj> In CI we use a new directory for every run
<smoser> i m ight be wrong.
<smoser> ok, then yeah, that should be fine.
<powersj> yeah
<powersj> fresh check out too
<smoser> ok. so can you ack that MP ?
<powersj> oh yeah - sorry about not catching the diff. Launchpad has been doing that to me as well
<powersj> done
<smoser> yeah, annoyinng
<smoser> we shoudl probably open a bug
 * smoser does that
<seanbright> are user questions on topic here?
<smoser> seanbright, sure
<smoser> powersj, filed at https://bugs.launchpad.net/launchpad/+bug/1692571
<ubot5> Ubuntu bug 1692571 in Launchpad itself "sometimes diff shown in git merge proposal is not updated" [Undecided,New]
<powersj> smoser: thx
<seanbright> smoser: i'm actually reading a bug report now that may answer my question. back in a tick.
<smoser> k
<blackboxsw> smoser: +1 I thought that the values coming out of pkg-config  would be the root path /..././.   and since we are adding a string which setup controls, ''systemd-fsck@.service.d' we shouldn't hit that issue. But defensive programming makes sense.
<seanbright> smoser: was looking for an elegant solution to this -> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1153626
<ubot5> Ubuntu bug 1153626 in cloud-init (Ubuntu) "Multiple Interfaces and IPs not detected in AWS VPC" [Medium,Triaged]
<seanbright> oh. fancy bot.
<seanbright> but i'll just try to work with some of the suggested work arounds mentioned there
<smoser> seanbright, well, there is a path to that now..
<smoser> but it still takes some dev work
<smoser> we are hoping to address it
<smoser> seanbright, its been quite some time since we'd have loved to have a solution for that. :-(
<seanbright> i might take a stab at it
<seanbright> but no promises
<smoser> rharper, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324338 i'd appreciate a 'please fix these things' or 'approve' i've resaponded to your comemnts
<smoser> and also blackboxsw or rharper on https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324274
<rharper> smoser: ok
<smoser> and i'll trade for a bcache review now
 * smoser finds
<powersj> rharper: seems your ntp changes broke the integration tests over the weekend from https://bugs.launchpad.net/cloud-init/+bug/1645644
<ubot5> Ubuntu bug 1645644 in cloud-init (Ubuntu) "ntp not using expected servers" [Medium,In progress]
<rharper> smoser: when I examine the journalctl of cloud-init units (in xenial up through artful) I don't see any DEBUG level items, just stdout/stderr
<powersj> I noticed you made updates to the tests too, but the user-data pools are not found in the output
<rharper> powersj: likely fubar'ed the merge
<rharper> basically we run ntpd -q to dump what servers it found; and compare that with the user-data
<rharper> do we have the artifacts ?
<rharper> smoser: comparing /var/log/cloud-init.log vs. journalctl output  ; that's how I know there are messages being emitted but not showing up in the unit log ... thoughts on what changed ?
 * powersj adds a card to keep artifacts from integration tests O.o
<powersj> let me get you a copy of them
<rharper> k
<powersj> here is what ntpq_servers produces https://paste.ubuntu.com/24625221/
<rharper> that looks like the defaults
<rharper> vs. what we may have put in there
<rharper> in the user-data configuration
<powersj> ntp_conf_pools https://paste.ubuntu.com/24625227/
<rharper> yeah, and we'll need the written ntp.conf file
<rharper> and the cloud-init.log (which should log what it wrote)
<powersj> http://paste.ubuntu.com/24625234/
<powersj> My run command: tox -e citest -- run -n xenial -t modules/ntp_pools
<powersj> ah actually my local run won't have your change in cloud-init
<powersj> so I need to run with the local tree real quick and get the failure the jenkins job saw, since it runs with a locally build "daily" cloud-init
<rharper> yes
<rharper> it looks like it rendered template, but didn't restart ntpd (which is what the fix does)
<rharper> we should see a run_cmd doing systemctl reload-or-restart or a 'service ntp restart'
<rharper> in the log
<smoser> rharper, ok. back. finished your bcache review request
<rharper> thanks, I'll check
<smoser> rharper, right. you dont see those in stderr and stdout any more . you actually never were supposed to. i forget the change that fixed that.
<rharper> =(
<rharper> well, it breaks cloudinit-analyze working with journalctl output
<smoser> but by default running 'cloud-init' shouldnt just  print debug level sutff to stdout. intstead it logs to console.
<smoser> er... to log file.
<rharper> which has high res timers
<smoser> log file shoudl now have high res timers
<smoser> right?
<rharper> lemme see
<smoser> as we are not going through syslog
<rharper> micro second
<rharper> better than second level
<smoser> well, millisecond
<rharper> ah, yes
<smoser> 1000th
<rharper> ack
<smoser> milli? micro
<smoser> yeah, your'e right
<rharper> milli IIRC
<rharper> at least subsecond =)
<smoser> i suspect that we weren't really getting prcision better than that (or not *much*) better than that
<rharper> I did enjoy getting cloud-init verbose output from journal, but understand w.r.t debug
<powersj> smoser: https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/324429 fix for ntp failures
<powersj> smoser: https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/324430 fix for snappy failure on proposed/artful tests
<powersj> rharper: thx for the reviews, just updated the snappy one with my comments. 100% open to ideas on how to work around this one.
<rharper> ack
 * rharper goes to pickup kiddo
<smoser> powersj, thank you for that.
<smoser> rharper, since you had done merge back and forth, a rebase was non-trivial, and that function actually was a collision and i had to resolv by hand.
<smoser> its much easier to handle a rebase since ultimately thats what i'm after.
<smoser> it was kind of disappointing that i couldnt just 'rebase -i' and squash stuff on your branch.
<smoser> powersj, https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/324429
<smoser> clean that up and i'll grab it quick
<powersj> smoser: heh on it
<powersj> smoser: done
<rharper> smoser: I'm happy to know how to handle those in the future ... I end up doing a rebase -i twice, once when I merge in master and then again after I push to my -u branch
<rharper> I know some folks --force push but I'd prefer to not do that;   it's strange to me (and I'm sure I don't understand what git's doing) that I can't have my remote branch pull in the rebase from master and have it fast-forward to apply the actual changes
<smoser> rharper, twice ?
<rharper> in my local branch, I git rebase -i master
<rharper> replay and fix it up against master;, then I git push -u raharper ... and it fails, saying I need to merge changes
<rharper> so then I git pull from my remote, and it has the  same conflicts that I had to resolve when I did my rebase -i on master
<smoser> ah.
<smoser> so yeah, rebase and having to 'push --force' does stink
<rharper> the only thing that's worked is if I rework the patch before rebasing it
<smoser> i always do:
<smoser>  git rebase -i master
<smoser>  git push smoser HEAD
<rharper> ah
<smoser> and with --force
<rharper> oh
<smoser> but then the single branch gets pushed
<smoser> not all my branches, as yeah, that'd be scary.
<rharper> it sorta feels like we're missing something
<smoser> well, rebase pretty much is always going to require a push --force i think
<smoser> right ?
<rharper> like it seems of little use to git push -u if I'm going to have to do that
<smoser> it *is* not a strict fast forward.
<rharper> maybe; I don't think I know enough git to answer clearly
<rharper> it seems like I want to push HEAD into the remote
<rharper> and then "remotely rebase"
<rharper> ie, we should only need to do the rebase once
<rharper> the changes are *identical*
<smoser> well, push is just dumb. it pushes what you have here over there.
<rharper> since it's a mirror of operations
<rharper> I mean, I do as you say, git fetch origin, git rebase -i master (fix up and complete rebase);
<rharper> I should be able to update origin at my remote; and then replay the same rebase operations
<rharper> says the non-git-expert
<smoser> well, push is just dumb. in that all it does is set refs. it doesn't do stuff on the server othe rthan that (and check that it is "fast-forward" or not)
<rharper> yeah
<rharper> but force just overwrites things
<rharper> seems like it could figure it out but I don't really know enough
<paulmey> Hi smoser, I just saw your comment re https://bugs.launchpad.net/cloud-init/+bug/1692093 ...
<ubot5> Ubuntu bug 1692093 in cloud-init "Cloud init is re-executing fs and disk setup during reboot" [Undecided,Confirmed]
<paulmey> we actually also have the same exact issue with new instances that attach an already prepared data disk
<paulmey> it gets wiped, even when the fs_setup info matches...
<paulmey> I agree that we shouldm'
<paulmey> shouldn't run cc_disk_setup on every boot, just to catch the ephemeral disk...
<paulmey> but somehow, we also have to figure out a way to get good answers from lsblk that aren't dependent on timing...
<paulmey> left a comment on the review...
#cloud-init 2017-05-23
<askb> where can I get the latest srpm for 0.7.9 ?
<powersj> https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/324490
<powersj> smoser: ^
<smoser> powersj, i know i ran tox before pusing yesterday
<powersj> default tox run doesn't include pycodestyle
<powersj> (I can't remember why we did it that way)
<powersj> I believe because we run the tip version and not tied to a specific release
<smoser> yeah. ok. right. it doesnt run tip things for style
<smoser> powersj, the change isnt new or is iot?
<smoser> #startmeeting ubuntu-server-team
<smoser> bah
<powersj> that area of the code wasn't touched since Jan
<blackboxsw> smoser: rharper who or what delivers /etc/cloud/templates/ on a system
<blackboxsw> part of the cloud-init deb right?
<blackboxsw> do any vendors overwrite this content?
<blackboxsw> just wanted to confirm they are fully under out control (in the ./templates dir of cloudinit)
<smoser> blackboxsw, yes. under cloud-init control outside of someone having a patched cloud-init
<smoser> powersj, https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/324490
<smoser> wait.
<smoser> thinking
<powersj> Wanna update tox too?
<powersj> Or at least run pycodestyle of a specific version now that we have cleaned things up
<smoser> powersj, yeah, that was the thought. i'mlooking at doing that. its really flake8 that we'd move.
<smoser> powersj, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324497
<powersj> smoser: ok let me play with that and I'll ack in a bit
<rharper> blackboxsw: there is a built in template;  there is precedence for overriding a template (aka the apt-configure module where user-data can define the template, but I don't think we added support for that in the ntp module at this time)
<larsks> smoser: I just added a note to https://code.launchpad.net/~akaris/cloud-init/+git/cloud-init/+merge/324196 with result of testing on rhel6.  Let me know if that is sufficient.
<smoser>  larsks thank you!
<powersj> smoser: acked the flake8 upgrade, I'll pull my merge
<blackboxsw> smoser: thanks for the review. I've responded to comments and I'm adding unit test for the real template render file. will ping when done
<smoser> blackboxsw, ...
<smoser> is_ec2_explicit_datasource
<smoser> makes me have a 80 char line
<blackboxsw> hahaha
<blackboxsw> rejected :)
<blackboxsw> +1
<smoser> good enough.
<larsks> smoser: do you think that +merge/324196 is ready at this point?
<smoser> larsks, yes. i want to pull that.
<larsks> smoser: thanks, just wanted to confirm.  I have some internal pressure to get a build out but want to make sure the patch was in good shape.
<powersj> smoser: when you are bored ;) thoughts on https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/324430
<smoser> powersj, i guess thats fine .
<powersj> smoser: thx
<powersj> that pulls 2 more out of the review lane \o/
<smoser> harlowja, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324497
<smoser> help ?
<smoser> powersj, see that. interesting.
<powersj> smoser: you ran pip check on your system though right? not the tox venv?
<smoser> well, no. i ran it in 'tox-venv flake8 pip check'
<smoser> but yes, that wasnt clear
<powersj> so hacking pulls in pep8 it looks like; are you concerned about the requirement version mismatches?
<harlowja> hey hey
<smoser> harlowja, is that safe ?
<smoser> why is hacking needing those old versions. it seems to work with newer.
<harlowja> the asssertEqual -> assertIsNone should be fine
<harlowja> unsure about why hacking needing old version
<smoser> yeah, thats the question i have.
<smoser> my tox works, but i didn't think i'd have pep8
<smoser> and then went looking as to why (as it should be replaced by pycodestyle)
<smoser> and hacking is what pulled it in
<smoser> as hacking has specific versions
<harlowja> got me :-/
<harlowja> throw email to openstack-dev ?
<smoser>  https://bugs.launchpad.net/hacking/+bug/1607942
<ubot5> Ubuntu bug 1607942 in hacking "hacking not compatible with flake8 3.0.x" [Undecided,New]
<blackboxsw> smoser: have you noticed that we have different distro naming conventions for hosts tmpl vs ntp nmpl in templates?
<blackboxsw> hosts.suse.tmpl  vs ntp.conf.sles.tmpl
<blackboxsw> shall we align those
<blackboxsw> it'd be nice if we had a std convention for attempting <thing>.<distro-specific>.tmpl if exists and  fallback to <thing>.tmpl when no distro scoped  tmpl is avail
<blackboxsw> but, not sure if there are other users besides hosts, ntp and sources
<blackboxsw> s/users/usecases/
<smoser> :-(
<smoser> blackboxsw, yes please.
<blackboxsw> ok will file a bite-size bug
<blackboxsw> what's your preference "sles" or "suse"?
<blackboxsw> looks like we have a lookup OSFAMILIES in cloudinit/distros/__init__.py
<smoser> yeah.
<smoser> cloudinit/distros/sles.py:        self.osfamily = 'suse'
<blackboxsw> suse it is thx
<blackboxsw> ok https://bugs.launchpad.net/cloud-init/+bug/1693020
<ubot5> Ubuntu bug 1693020 in cloud-init "Cloud.get_template_filename should start with distro-specific template and fallback to generic template files before warning !found" [Low,Triaged]
<blackboxsw> smoser: addressed your review comments and pushed https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/324450
<blackboxsw> do we want to make all unit tests logged ? 2min 20 second runtime vs 1min 30?
<smoser> blackboxsw, good judgement
<smoser> no need to log all
<blackboxsw> ok will use the with_logs = True where needed then
<blackboxsw> thx smoser
<blackboxsw> all review comments addressed then
<blackboxsw> will have the real branch up in a few. adding a couple extra unit tests for test_schema
<blackboxsw> so it'll sit for tomorrow. and I am in review debt
<powersj> blackboxsw: smoser: https://jenkins.ubuntu.com/server/job/cloud-init-build/183/console
<powersj> Did python3-pycodestyle get added some how as a depends?
<powersj> That test attempts to build cloud-init's master in a xenial sbuild env.
<blackboxsw> checking
<blackboxsw> powersj: those deb package deps are generated automatically based on what's in requirements.txt and test-requirements.txt. hmm I'm not finding what pulls that in ATM.
<blackboxsw> maybe my branch needs updating.
<blackboxsw> it should be pulled in via ./tools/read-dependencies and python(3)?- prefix attached automagically
<powersj> smoser: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324497 that updated test-requirements.txt
<powersj> which updated build deps then
<powersj> so you can't build on xenial atm
<powersj> tbh I didn't look hard enough I thought you updated the versions in the tox file
<powersj> Looks like python3-pycodestyle is only in yakkety and newer too
<blackboxsw> yeah I haven't found it in xenial either, we'd have to backport and host the package maybe in a cloud-init-deps ppa.
<blackboxsw> powersj: do we have a cloud-init-deps ppa I don't see any ppas listed at https://launchpad.net/~cloud-init associated w/ the cloud-init team ?
<powersj> blackboxsw: I don't believe so
<blackboxsw> or do we just rely on <series>/updates etc for newer bits?
<powersj> blackboxsw: that
<blackboxsw> +1
#cloud-init 2017-05-24
<smoser> powersj, blackboxsw yeah, that kind of stinks. basically i think we *have* to detach package build dependencies form tox requirements.txt
<smoser> which i think might in-advertantly have been why it was the way it was when blackboxsw found it and thought WTH
<smoser> the package build on xenial didn't magically start depending on python3-pycodestyle just because trunk moved some versions in its tox.
<smoser> as an example, see that https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily-xenial did not break because of this change.
<askb> can anyone assist with this https://bugs.launchpad.net/cloud-init/+bug/1692424
<ubot5> Ubuntu bug 1692424 in cloud-init "util.py[WARNING]: Failed to disable password for user centos" [Undecided,New]
<askb> Is there are doc for building the latest verion on centos7 ?
<smoser> askb, can you get a cloud-init.log from the system ?
<smoser> /var/log/cloud-init.log
<askb> smoser, yup
<askb> do you want me to upload the logs ?
<askb> smoser, http://paste.openstack.org/show/610552/
<askb> smoser, forgive my ignorance, but are there any docs for building the rpm/srpm for centos7
<askb> I noticed that there is a spec.in
<askb> "make rpm" does not work
<smoser> powersj, blackboxsw https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324541
<smoser> askb, hm...
<smoser> let me find
<smoser> I had https://gist.github.com/smoser/19e65095b342e98fd4d6466e4d4fa1ac
<smoser> we're workign on getting rpm build into daily c-i, and i'm not sure what is being used for that.
<smoser> hmm.. i see ./packages/brpm is busted. odd
<smoser> askb, i thik that gist will work. brpm is failing for me here, but because i dont have cheetah installed and the spec file needs it.
<powersj> smoser: yeah the build test that I pointed out is what uses packages/brpm
<powersj> I expect that to work :)
<smoser> powersj, we need to move the spec.in to be jinja2 rather than cheetah
<powersj> and we had a centos 6 test case fail last night
<smoser> :(
<powersj> https://jenkins.ubuntu.com/server/job/cloud-init-centos-6/10/consoleText
<powersj> I'm going to go eat/wake-up and then I'll file bugs and we can chat about it after standup as far as what you want to see for fixes
<askb> are these the same errors you are getting http://paste.openstack.org/show/610557
<askb> smoser, ^^
<smoser> askb, i suspect that whatever you're seeing is fixed in a newer release. 0.7.5 is quite old.
<smoser> askb, yes.
<smoser> you'll need python-cheetah installed
<smoser> which that gist will install for you.
<smoser> ie, if you grab that 'centos-setup' and run ./centos-setup build
<askb> I am not building it with lxc ... instead straight on centos7 virt env ?
<askb> should I be using the lxc instead
<smoser> well, the script will install the deps for you
<smoser> i jsut showed how to do it all with lxc
<smoser> but all it amounts to is runing that script inside the container
<smoser> which should be not really any different than inside a centos7
<askb> you mean from the gist above correct?
<smoser> "virt env" == "kvm guest" or python virtual env
<smoser> https://gist.github.com/smoser/19e65095b342e98fd4d6466e4d4fa1ac
<askb> python virtual environment
<smoser> well, package build is going to use system ddependencies...
<smoser> powersj, i'im not sure what is changed there.
<smoser> but tests/cloud_tests is python3 only. and that syntax is not python2.6 compatible
<askb> smoser, Is the script to be run the - centos-setup - on debian/ubuntu ?
<smoser> to build an rpm you really need to be in a centos (or rhel) environment
<smoser> ./centos-setup build
<smoser> will install the necessary packages to build
<smoser> then you can run
<smoser> ./packages/brpm
<askb> oh this says "lxc: containers not found" on centos7
<smoser> and build a package
<smoser> sure. skip the lxc portion, just use centos-setup
<smoser> wget that and chmod and execute it.
<askb> smoser, I was able to build an rpm - thanks!
<askb> smoser, will update a bit later how it goes with the testing since its getting 1am here
<smoser> askb, good luck
<smoser> and sleep tight
<askb> smoser, quick question: how do I build the latest stable version/release ?
<smoser> you can git checkout 0.7.8
<smoser> and do the same ./packages/bdrpm
<askb> 0.7.9 is the head ... of the branch now ?
<smoser> sorry
<smoser> 0.7.9 is newest release
<smoser> 'master' branch is trunk
<askb> awesome! :)
<smoser> askb, i updated that gist README hoepfully to be more clear
<smoser> and to down-play the lxd thing so as to not confuse people that.
<smoser> blackboxsw, updated https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/324450
<blackboxsw> thanks+1 let's land I've added https://trello.com/c/iJdLSnh4/91-add-root-dir-lookup-for-unittests
<smoser> k.
<blackboxsw> heh, I'll steer you later or tomorrow to the real meat: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/324499 :)
<blackboxsw> it's up for review. I'm working on the old git review queue for cloud-init at the moment
<blackboxsw> minor comment and approve of https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324274
<smoser> looking
<smoser> yeah, got to add that.
<smoser> powersj, are you looking for fix centos6 ?
<powersj> smoser: I wasn't planning on looking at it today, but if you have something let me know and I can ack or test or merge or whatever :P
<smoser> ok
<smoser> blackboxsw, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324557
<smoser> can you ack that quick ? its just part of
<smoser>  https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324274
<smoser> but the part that *has* unit test
<harlowja> smoser and others http://lists.openstack.org/pipermail/openstack-dev/2017-May/117401.html
<harlowja> i responded a little bit, but feel free to :-P
<smoser> harlowja, thanks. i dont have mch to add there.
<harlowja> np
<blackboxsw> approved the branch smoser
<smoser> thanks.
<blackboxsw> smoser: I'm reviewing https://code.launchpad.net/~akaris/cloud-init/+git/cloud-init/+merge/322992 and forgot what your thoughts were on us using python's netaddr instead of trying to carry our own burden of mask2cidr cidr2mask calculations
<blackboxsw> I was debating about cobbling up a quick suggestion branch w/ unit tests for the review if you netaddr is a  likely candidate for replacing this somewhat fragile code
<blackboxsw> if you *think* netaddr is a likely candidate....
<blackboxsw> but again, maybe cloud init wants to reduce requirements on external packages for the sake of speed, dunno.
<blackboxsw> it's nice to have simple validation functions like valid_ipv4, valid_ipv6 IPNetwork.netmask builtins
<smoser> blackboxsw, python2.6 ENO_NET_ADDR
<smoser> and even the rhel 6 version of python2.7
<smoser> blackboxsw, i was looking at that too and trying to clean it up abit
<blackboxsw> won't touch it since you are playing in that space, but ping me for a review on it when you are done. I don't want to miss what approach you are looking for.
<blackboxsw> s/review/review please/ :)
<smoser> blackboxsw, http://paste.ubuntu.com/24646853/
<smoser> that is what is in my buffer right now
<smoser> i was looking at xnox's cidr branch
<smoser> (which had the same change as akaris)
<blackboxsw> smoser: why not mask = str(mask) in line 162 of your paste, that was you are properly converting to the expected type for both ipv4 & ipv6 handling.
<smoser> you just ean
<smoser>  mask = str(mask)
<smoser> if : in mask:
<smoser> well, with proper quotes
<smoser> right ?
<blackboxsw> yeah.
<blackboxsw> then you don't have to perform str(mask) twice
<blackboxsw> also that function doesn't have a return or error if a string w/out . or : is presented
<blackboxsw> does   subnet['prefix'] = mask_to_net_prefix(subnet.get('netmask'))   # Where None is returned  cause any problems? (line 52 of your paste)
<blackboxsw> heh and I have no idea what's happening on line 85
<blackboxsw> 87 rather
<blackboxsw> WIP paste I bet
<blackboxsw> +1 I like the general approach in either case. given netaddr DNE on some of our support matrix
<jbrowne> Hello, I'm running into a race condition between cloud-init and apt-daily.service on Xenial.  This was mentioned in passing in https://bugs.launchpad.net/cloud-init/+bug/1594576, but I don't see a cloud-init issue open specifically about it.  Warning: small wall of text to follow.
<ubot5> Ubuntu bug 1594576 in cloud-init "cc_salt_minion behaves badly if apt-get update fails" [Medium,Triaged]
<jbrowne> cloud-init when installing packages is encountering a lock on APT due to the apt-daily service running simultaneously.
<jbrowne> Running a dead simple user data, I see that the stock AWS Ubuntu Xenial AMI does not encounter the race condition as the timer state is:
<jbrowne> Thu 2017-05-25 15:34:01 UTC  20h left      n/a  n/a    apt-daily.timer              apt-daily.service
<jbrowne> While the timer state of my packer-built AMI (based on the stock AMI) is:
<jbrowne> Fri 2017-05-19 08:40:01 UTC  5 days ago    Wed 2017-05-24 19:19:21 UTC  4s ago apt-daily.timer              apt-daily.service
<jbrowne> Searching extensively (including IRC logs) I haven't found any discussion in the cloud-init project about this issue and am happy to open an LP bug.
<jbrowne> There's a lot of discussion about when unattended upgrades should run, etc. and if it is a systemd issue, but regardless it seems like cloud-init should be defensive about this.
<jbrowne> I would naively argue that cloud-init should retry on apt issues, but I see that was deemed unneded in another case (https://bugs.launchpad.net/cloud-init/+bug/1594967)
<ubot5> Ubuntu bug 1594967 in cloud-init "cloud-init lack of retry causes apt-get failures to be fatal" [High,Fix released]
<jbrowne> One are in which I am ignorant is how the actual AMIs are built; I'm trying to determine if something specifically clears systemd's timer states when making an image, etc.
<jbrowne> Was digging into cloud-image-utils and then decided to just post here since that area also appears to be s-mosers.
<blackboxsw> jbrowne: peeking for context.  Without context of cloud-init's intended behavior, I would prefer apt to retry upon failure, or checksum mismatch which has hit me multiple times in other projects. Let me see if I can get more context on this specific issue. Thanks for "the deets".
<jbrowne> Basically if one uses cloud-init for:
<jbrowne> packages:
<jbrowne>   - awscli
<jbrowne> and systemd has launched some other apt related process concurrently there can be a lock contention
<jbrowne> I have a slew of kinda related bugs I've found in other projects that build their own AMIs (chef, vagrant, etc.) and have hit this lock contention with cloud-init, but I didn't want to dump an entire thesis into the channel. :)
<blackboxsw> I'd naively like to see at least a simple retry fallback mechanism (maybe 3 retries on a backoff algo) before failing. Yes dpkg lock contention has hit a myriad of other projects I've been on. I'd think we could build in a simple retry/log effort before failing.
<blackboxsw> but, I'm just reading through the cloud-init code now (and your references) to see what the existing state of affairs is.
<blackboxsw> we've recently talked side-channel about the potential of adding a decorator to retry certain functions that are known to get a benefit from retries on expected failure. This might be a case where something like that could come in handy.
<blackboxsw> it *looks* like we could potentially build in a retry mechanism just in Runners.run of cloudinit/helpers.py.  I'll peek at whether that's viable.
<smoser> blackboxsw, http://paste.ubuntu.com/24647424/
<smoser> thats where i am now.
<smoser> jbrowne, why would systemd launch some other apt related process ?
<blackboxsw> yowsa smoser
<smoser> fudge.
<jbrowne> apt-daily is no longer in /etc/cron.daily.  Now on a timer
<jbrowne> If your machine is down during the trigger time (or a custom built AMI) systemd will trigger that timer during boot -> apt-get update in another process
<smoser>  http://paste.ubuntu.com/24647437/
<smoser> jbrowne, ack on that. that is a mess. cloud-init should probably disable that thing and then re-enable it.
<jbrowne> There's a lot of chatter about how this has changed when unattended updates run and conflict between not running during boot and machines that are up for short periods of time (laptops), etc.
<smoser> that is generally garbage really.
<smoser> no user can sanely run 'apt-get update' or 'apt-get install' because some system thing might be doing something.
<smoser> shoudl really be fixed at the apt level
<jbrowne> I can see the argument for using systemd timer instead of cron.daily due to it blocking other cron jobs, but changing fundamental behavior like this bugs me.
<smoser> blackboxsw, second paste. i pasted the file not diff.
<jbrowne> Also unattended upgrades is on by default now in Xenial, but that's a whole 'nother story. :)
<blackboxsw> yeah I'm on 2nd paste now. it looks a lot simpler
<blackboxsw> smoser: not sure, might have to default to string for the get on line 25 +                if (subnet.get('type', '').endswith('6') or
<jbrowne> The reason I brought this up in channel rather than just opening an LP bug was to get a sense of how the cloud-init team feels about it.  My current work around is to have packer disable the apt-daily.service in systemd and I re-enable it on launch.
<blackboxsw> if type isn't present, subnet.get would return None and you can't call endswith on that
<smoser> jbrowne, yeah. you want to file a bug ? you can file against cloud-init.
<smoser> but it seems honestly silly / broken that 'sudo apt-get update' fails 30 seconds of every day by default.
<smoser> or whatever that ends up being.
<smoser> blackboxsw, well, i think its busted if the subnet doesnt have a type
<smoser> blackboxsw, so thats my attempt at noramlizing the fields in subnets. then have to adjust callers to expect more sanity.
<blackboxsw> smoser: "no user can sanely run 'apt-get update' or 'apt-get install' because some system thing might be doing something."  yep. it is insane, but it's reality at the moment, the other issue is checksum mismatches in apt repositories to all apt users who happen to call apt update while the distro is being updated. Something about the packages.list files not
<blackboxsw> being fully written during that update causes spurious failures on any remote caller to apt update.
<smoser> well hash sum mismatches are fixed now.
<smoser> in xenial+
<smoser> everything is by hash
<blackboxsw> ahh good. I hadn't followed that fix. (because I had a workaround for it ;) )
<blackboxsw> but I agree that generally cloud-init shouldn't put workarounds in for other things that really should be fixed.
<blackboxsw> s/other things/external dependencies/
<smoser> http://www.chiark.greenend.org.uk/~cjwatson/blog/no-more-hash-sum-mismatch-errors.html
<smoser> he doesn't point to the launchpad bugs
<smoser>  https://bugs.launchpad.net/ubuntu/+source/apt/+bug/972077
<ubot5> Ubuntu bug 972077 in apt (Ubuntu) "apt repository disk format has race conditions" [Medium,Fix released]
<smoser> but then andreas hit that today on xenial... as he was using a br.archive.ubuntu.com mirror
<smoser> that i guess doesn't have the server side fixes needed
<jbrowne> I'll file an LP bug, thanks.
<smoser> please paste bug url here jbrowne
<jbrowne> will do
<jbrowne> https://bugs.launchpad.net/cloud-init/+bug/1693361
<ubot5> Ubuntu bug 1693361 in cloud-init "cloud-init sometimes fails on dpkg lock due to concurrent apt-daily.service execution" [Undecided,New]
<smoser> gracias
<smoser> blackboxsw, i'm interested in your thoguths on that pastebin .. even though its not in any real nice stae. tox fails at the moment.
<blackboxsw> smoser: yes definitely. will test it out now. could add a couple of unit tests and post you a diff if you want
<blackboxsw> smoser did you want to delete all empty/None keys from the dict?
<blackboxsw> +    for k in ('address', 'gateway', 'netmask'):
<blackboxsw> +        if k in subnet and not subnet[k]:
<blackboxsw> +            del subnet[k]
<blackboxsw> could be done w/  subnet = dict((k, v) for k, v in subnet.iteritems() if v)
<blackboxsw> also s/if 'netmask' in subnet: del subnet['netmask'] can be consolidated to    subnet.pop('netmask', None)
<blackboxsw> smoser: some minor changes to normalize_subnet http://paste.ubuntu.com/24648411/
<blackboxsw> smoser: also it seems like logic is duplicated a lot between _normalize_subnet and normalize_route with the exception that the key we are normalizing in routes is 'network|destination'  and in subnet is 'address' it could be small private helper function that takes the address_keys  and could be called inside normalize_subnet():
<blackboxsw> _normalize_net(addr_keys=['address'])     and in normalize_route as _normalize_net(addr_keys=['network', 'destination'])
<blackboxsw> that wasn't proper english at all, but hopefully you got the gist
<blackboxsw> ok last pass on this for today http://paste.ubuntu.com/24648847/
<blackboxsw> common _normalize_net_keys() called from both normalize_subnet and normalize_route
#cloud-init 2017-05-25
<smoser> blackboxsw, yeah, i got the gist.
<smoser> blackboxsw, your python2 shines through
<smoser> (iteritems() is not python3 :)
<smoser> pushed to https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+ref/cleanup/mask2cidr
<blackboxsw> whoopsie. right grr, need to be more careful
<smoser> <smoser> powersj, if your'e still arond
<smoser> <smoser> i think that centos6 migith be as easy as
<smoser> <smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324585
<smoser> blackboxsw, then... some python2.6 breakage from ntp
<smoser> :-(
<smoser> http://paste.ubuntu.com/24650034/
<smoser> there... https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324585
<smoser> that fix there too now
<smoser> and /me goes away 25 minuts late
<askb> smoser, seems like running into other ssh issues with 0.7.9
<blackboxsw> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324585 approved
<askb> smoser, here are the logs: http://paste.openstack.org/show/610611/
<powersj> I propose we use 0.8 to drop python 2.6 support ;) centos 6 dropped full update last month and is only in maitence mode if I am reading dates right.
<askb> powersj, is there a python version dependency for cloud-init to work with centos7 ?
<powersj> askb: I think since centos 7 has 2.7 if I recall it isn't an issue
<askb> powersj, is 0.7.9 tested to work with centos7, running into issues: http://paste.openstack.org/show/610611/
<askb> powersj, mainly with ssh restart failure and failed to disable password for ubuntu user
<askb> Is this a known issue ?
<powersj> I'm not familiar with any issues, if you run the command manually what happens?
<powersj> As far as testing l, all I have in place right now is unit testing and package build. Working on getting some initial integration tests going next month.
<powersj> All on centos 6/7
<powersj> I'm not too sure how the version of cloud init gets into the images or that process but also need to research that
<askb> how do you run the cmd manually ?
<powersj> Askb did you file a bug?
<askb> cloud-init single -n cc_scripts_user
<askb> yes - the original issue was with version 0.7.5 which is pretty old from centos7 repo
<askb> bug: https://bugs.launchpad.net/cloud-init/+bug/1692424
<ubot5> Ubuntu bug 1692424 in cloud-init "util.py[WARNING]: Failed to disable password for user centos" [Undecided,New]
<powersj> Sorry to be more clear, try running the failing commans "passwd -l ubuntu"
<askb> so I built an rpm with 0.7.9, now testing it again with the latest centos7 image has brought me to  http://paste.openstack.org/show/610611/
<askb> for running the cmd manually, now my user data script is not even available under /var/lib/cloud/scripts/*
<powersj> askb, Hmm I can try to look in the morning, easier when I'm not on my phone. Thanks for filing a big. Tbh I'm not familiar with the heat templates, but will look
<askb> powersj, thanks would appreciate that! ... btw ... there are several post on the net with users running into similar issues with 0.7.5 with no solution/work around unfortunately
<powersj> askb: if you have time can you add an example of those posts to the bug as well as what image you are using in the first place
<askb> sure thing!
<askb> 2017-05-25 03:48:22,464 - util.py[WARNING]: Failed to disable password for user ubuntu
<askb> 2017-05-25 03:48:22,465 - util.py[WARNING]: Running module users-groups (<module 'cloudinit.config.cc_users_groups' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_users_groups.pyc'>) failed
<smoser> blackboxsw, you have merge conflicts https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/324499
<blackboxsw> smoser: oops botched the conflict resolution in lp, clean proposal is at https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/324640
<blackboxsw> trashed cc-ntp-schema branch
<blackboxsw> s/trashed/deleted
<smoser> blackboxsw, ok. i will review locally
<blackboxsw> thanks
<smoser> blackboxsw, fyi, that has c-i Needs fixing
<blackboxsw> looking it over
<smoser> blackboxsw,
<smoser> $ ./tools/cloudconfig-schema --doc # to dump rst from the schema definition
<smoser> bash: ./tools/cloudconfig-schema: No such file or directory
<blackboxsw> smoser: are you refering to "RuntimeError: Device /dev/xdb1 did not exist and was not created with a udevamd settle."/
<smoser> i was talking about https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/324640
<smoser> blackboxsw, and also ^ . wasa the description still supposed to be valid ?
<smoser> you didnt add tools/cloudconfig-scheme
<smoser> you didnt add tools/cloudconfig-schema
<blackboxsw> smoser: just pushed thx
<blackboxsw> 4858a34
<smoser> blackboxsw, git push --force ?
<smoser> i dont see it here.
<blackboxsw> smoser: now?
<smoser> blackboxsw, https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+ref/cc-ntp-schema-validation
<smoser> "28 minutes ago"
<blackboxsw> smoser: dumb. fixed
<blackboxsw> PEBKAC
<sauloaislan> Hi!!
<sauloaislan> I'm having a problem, my cloud init does not work on openstack with network interface neutron. I'm doing a deploy with network interface neutron using a simple cloud init, but I can not log in to the VM.
<smoser> sauloaislan, can you see a console log ?
<smoser> or ideally you can get the cloud-init.log file off the system as that will likely have some info.
<sauloaislan> If I do a deploy with the network flat interface and the same cloud init I can login
<rharper> smoser: hrm, so, the .link files for rename are somewhat racy with udevd and the udev hotplug rules 80-ifupdown.rules I'm seeing the ipv6 vlan network configuration ...
<smoser> rharper, i dont think they should be racy
<rharper> well, I wish it weren't either
<smoser> it should be very deterministic. it may not do what you want. but i dont think it should be racy.
<smoser> the .link files definitely run last
<smoser> and only apply if the device hasnt already been renamed
<smoser> (that is what i recall at least)
<rharper> I need to keep digging, but if I move the .link files out of the way, it all works fine;
<rharper> my concern is that I don't at which point the .link files apply vs. when the 80-ifupdown.rules run ; this is all prior to cloud-init local;
<smoser> https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1579130
<ubot5> Ubuntu bug 1579130 in systemd (Ubuntu) "need to support renaming of devices in container and on first boot" [Wishlist,Triaged]
<smoser> theres a lot of info there.
<rharper> it appears like the interfaces are already named correctly but there's something going on with the vlan interface names
<smoser> remember the .link files get into the initramfs
<smoser> (although the udev rules would to)
<rharper> hrm
<rharper> on the second reboot ?
<rharper> or after reboot,?
<smoser> well, cloud-init doesnt actully force re-generation of initramfs
<smoser> as that'd be really heavy
<smoser> and as we found in that bug above, wouldnt necessarily fix anything
<smoser> so cloud-init makes sure it renames everthing it can by itself
<rharper> smoser: https://code.launchpad.net/~raharper/curtin/trunk.passthrough-netconfig , and XenialTestNetworkIPV6Vlan (or any of the IPV6Vlan) break
<rharper> if we write the .link files (which we do in cloud-init, but not in curtin)
<smoser> rharper, https://bugs.launchpad.net/cloud-init/+bug/1594546
<ubot5> Ubuntu bug 1594546 in cloud-init "no need to write systemd.link files" [Medium,Fix released]
<smoser> we *had* stopped writing them at one point.
<smoser> whyd'd we start again
<rharper> ISTR we thought it was harmless
<smoser> ef18b8ac4 put them back
<smoser> :-(
<smoser> rharper, i think what you're saying is that when cloud-init does netplan config it is still writing those .link files
<smoser> and netplan is (i assume) also writing .link files
<smoser> so that' dget harry
<rharper> netplan will write .link files;
 * rharper looks at ef18b8ac4
<rharper> hrm, I'm not seeing it, the links_path shouldn't be set when we instantiate an eni renderer
<smoser> oh. maybe i read wrong
<smoser> but as you see, we are creating them, right?
<smoser> rharper, thoughts on  https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324639
<smoser> it doesnt adda  unit test, but unit tests aren't easily useful in this scenario
<Odd_Bloke> Hmph, just discovered I've been sitting in #cloud-init on Canonical IRC since I last restarted weechat. >.<
<smoser> Odd_Bloke, do you know.. was https://code.launchpad.net/~daniel-thewatkins/cloud-init/+git/cloud-init/+merge/324642
<smoser> always just *wrong* or did it get changed?
<smoser> it seems odd that they would change instance/attributes/sshKeys to instance/attributes/ssh-keys while leaving the camelcase project/attributes/sshKeys
<Odd_Bloke> smoser: It did get changed.
<Odd_Bloke> smoser: Changing instance metadata keys is easier for them than project metadata keys.
<rharper> smoser: yes, they are getting created, I need to see why
<smoser> Odd_Bloke, i asked in the mp, and asked for an update.
<smoser> it seems very odd.
<smoser> pylint tells me now:
<smoser>  Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)
<smoser> which is odd.
<smoser> Odd_Bloke, if you fix that quick, i'll grab it and get it into artful end of day
<smoser> and intend to prepare sru of trunk soon
<Odd_Bloke> smoser: Updated.
<rharper> smoser: turns out that with no config passed to the renderer class, you get the defaults, which currently defaults to rendering the .link files for eni
<rharper> I think it may make sense to set in the ubuntu distro class the links path to None, which would stop rendering them
<smoser> rharper, that is what it previously had, right?
<smoser> efef1c263ab1c473fd90f3782a785edab7d02430
<rharper> hrm, we also have the 70-persistent rules
<rharper> smoser: I think so
<smoser> we do need the 70-persistent rules, i think.
<smoser> maybe not.
<rharper> well, cloud-init does the rename
<smoser> cloud-init will take care of renaming any non-virtual devices that are present when init runs
<rharper> right
<smoser> but if they are not present, then it wont do it
<rharper> and udev rules and .link files are for physical
<smoser> so i think we were relying on udev to do anything cloud-init couldnt do because it wasnt there yet.
<Odd_Bloke> smoser: Thanks for the review+merge. :)
<rharper> smoser: I think you're right; after lxd, we had to do rename in cloud-init;
<rharper> which removed the need to rely on udev for that
<smoser> except for devices that are not present when cloud-init runs
<smoser> a device could have a name and networking infromation but not a address
<smoser> and cloud-init should then arrange for it to get the right name
<smoser> even though it wont be up (and may not even be around yet) when cloud-init init runs.
<smoser> ii think
<smoser> rharper, or blackboxsw i'd appreciate https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324639
<rharper> smoser: you're thinking a manual interface?  cloud-init will still attempt to provide a name for physical interfaces in network-config no matter if it has a subnet or not
<rharper> on the control-hotplug; that may be right
<rharper> we could modify the rules and links to be rendered for missing interfaces or ones marked control: hotplug
<rharper> but it does seem strange to have some in the rules/links file but not others;  however; I don't see how we can disentangle the different rules which are attempting to do renames
<smoser>  http://paste.ubuntu.com/24658740/
<smoser> that would seem valid config. and if 'eth1' just wasnt there when cloud-init ran, you'd hope that it wouuld have still arranged for it to get the right name
<smoser> rharper, i think we just render the udev rules for everything
<smoser> we can't rely on .link files... they suck
<smoser> so we render udev rules for everything
<rharper> for all physical devices, yes
<smoser> cloud-init does the right thing and only attempts rename if it needs to
<rharper> yes
<smoser> "physical" including veth
<smoser> :)
<rharper> that bears out too, if I remove the .link files and leave the .rules one, we;re fine
<rharper> sure
<smoser> i dont understand though why the .rules would break anything
<rharper> I was referring to the possible interfaces from the network config
<smoser> i really thought they are not supposed to do anything if a udev rule named them
<rharper> I;m not sure that they do interfere at this time
<rharper> was just wondering why we did both .link and .rules
<smoser> by accident
<smoser> :)
<smoser> doh!
<smoser> so when you fix that, please add a test case
<rharper> yes
<smoser> it always feels wrong to remove code that you think you might need in the future
<rharper> heh
<smoser> but it might be the right thignto do to remove the support for rendering of the .link files
<rharper> I think so
<rharper> smoser: so I'll do a patch that just pulls .link rendering completely
<rharper> that should keep this pretty clean, and then unittest updates should all be that's needed
<askb> powersj ping
<powersj> askb: howdy
<askb> powersj, hey ...
<askb> re the auth_url message, wondering how I could help with that
<askb> you are most likely seeing the error "Missing value auth-url required for auth plugin password" because that has to be source through ~/.config/openstack/clouds.yaml or RC file or as an env variable
<powersj> ah ok
<powersj> can you try running the validation script and get the results?
<askb> basically, the template itself works pretty well, since about 200 jobs run using those template
<askb> until recently cloud-init stopped working
<askb> sure, let me check
<powersj> askb: interesting. Do you have copies of the images you used? Is it possible to see what changed to the image?
<powersj> originally I believe you said you were running a fairly old version of cloud-init, so I find it unlikely that behavior in cloud-init has changed causing the issue
<askb> when I run the validate-script on the template, it returns nothing, no errors
<askb> so that should be good
<askb> centos7 repo distributes 0.7.5 version of cloud-init
<askb> the newer images have package updates of possible dependencies
<askb> I need to dig into the images to get a proper delta of the deps between new and old image
<askb> will do it a bit later today
<askb> in the meantime, are there any basic tests for cloud-init which could be run on the instance to make sure its working fine or narrow down the issue
<askb> yesterday, while running 0.7.9 this gives a different error:
<askb> 2017-05-25 02:07:00,591 - util.py[WARNING]: Failed to disable password for user ubuntu
<askb> 2017-05-25 02:07:00,593 - util.py[WARNING]: Running module users-groups (<module 'cloudinit.config.cc_users_groups' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_users_groups.pyc'>) failed
<askb> I am unable to understand why its looking for ubuntu user on centos box ?
<powersj> As the pastebin's showed cloud-init was trying to run the passwd command when it failed, yeah the ubuntu user was also odd.
<askb> powersj, is the issue with 0.7.9 and 0.7.5 possibly the same ?
<askb> from the logs
<askb> other than the dependencies is there anything else I could provide or run some other tests
<powersj> askb: well you got the same error messages if I recall. However, I still wonder what changed on the system to cause the error in the first place. If you aren't expecting an ubuntu user, what change caused that? trace back to that first and that might determine why cloud-init is failing.
<powersj> if you can get us the images themselves that might help.
<askb> the ubuntu user error was seen with 0.7.9 and centos user error was seen on 0.7.5
<askb> ok will see if I can get the images
#cloud-init 2017-05-26
<smoser> rharper, i'd appreciate your thoughts.
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324639
<smoser> powersj, askb the ubuntu user will be used probalby if he installs from a trunk created rpm
<smoser> https://code.launchpad.net/~harlowja/cloud-init/+git/cloud-init/+merge/307485
<smoser> would fix that
<askb> smoser, would this fix the original issue with bug filed ?
 * askb 1821: Operation in progress 
<smoser> askb, i'm sorry, did you ever get a /var/log/cloud-init.log ?
<smoser> from the trunk built one would be helpful.
<askb> smoser, updating the logs now
<askb> smoser, wondering if the bug you mentioned above is expected to fix the original issue ?
<smoser> askb, logs where ?
<smoser> sorry for not following above.
<askb> smoser, oh sorry! my bad ... have updated the logs on the bug filed here: https://bugs.launchpad.net/cloud-init/+bug/1692424
<ubot5> Ubuntu bug 1692424 in cloud-init "util.py[WARNING]: Failed to disable password for user centos" [Undecided,Incomplete]
<askb> I noticed earlier today, you mentioned that 307485/324639 may fix ubuntu user issue caused while running the passwd command
<askb> Was curious if that fixes the issue in question and if I could help with testing the fix on our env
<rharper> smoser:  on the udev-settle MP,  I'm still confused about the previous comment on removing blkdev re-readpt, and when I look at the diff on launchpad, line 13 still does util.subp(blkdev_cd) ?  what am I missing?
<rharper> als, this is the link removal;  with a build of that cloud-init, I'm now passing the vlan tests that failed before
<rharper> https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/324675
<smoser> rharper, see lsat comment right now
<smoser> i think you'll understand :)
<smoser> askb, i think selinux must be stopping cloud-init from invoking 'passwd -l'
<smoser> see
<smoser>  Stderr: passwd: Libuser error at line: 124 - error creating `/etc/shadow+': Permission denied.
<rharper> smoser: ack
<smoser> rharper, do you recal something about selinux and systemd services....
<rharper> yes
<smoser> there was something. maybe larsks ?
<rharper> when I was debugging some centos7 issues
<larsks> smoser: what where when?
<smoser> thanks.
<rharper> restorecon
<rharper> failures
<rharper> IIURC
<smoser> larsks, https://bugs.launchpad.net/cloud-init/+bug/1692424
<ubot5> Ubuntu bug 1692424 in cloud-init "util.py[WARNING]: Failed to disable password for user centos" [Undecided,Incomplete]
 * larsks looks
<smoser> cloud-init is running passwd -l
<rharper> if it failed to restore a file in a path, it quit recursing
<smoser> and failing with permission denied.
<smoser> i think i joked that the solution was to turn of enforcing :)
<smoser> and i think larsks took me seriously
<larsks> smoser: no, that was a "what is the topic of this discussion" as opposed to a "what are you thinking?"
<sauloaislan> smoser i find this erro in my cloud-init log route info failed
<larsks> Why do we think this is a restorecon failure? I don't see that in the bug anywhere.
<rharper> there was an update to the library needed but wasn't released into 7.3 at the time
<sauloaislan> this test with the ironic.conf  network interface = neutron
<larsks> There was a bug in python-selinux, yeah.
<smoser> rharper, or larsks could you fill some info in on that bug if that is what you think it is ? i think you're saying there might be a bug in python-selinux ? (but cloud-init doesn't use that for invoking passwd -l to my knowledge)
<smoser> so i'm kind of confused.
<rharper> I don't know that your passwd -l is related
<smoser> sauloaislan, more info needed. paste a cloud-init.log somewhere ? and if its openstack an network_data.json
<larsks> smoser: *I'm* confused.  Looking briefly at the bug I'm not sure where selinux comes up.
<rharper> I saw log messages on a centos7 boot related to the restorecon
<smoser> larsks, well, 'selinux' is what i thought could make a process runnign as UID=0 not able to run 'passwd -l'
<smoser> the other thought was root was mounted ro, but that seemed less likely.
<larsks> smoser: but I don't see 'passwd' in that bug either.  Am I looking in the right one?  #1692424
<smoser> well. the last cloud-init.log he's posted there.
<rharper> reading it now
<smoser> the title of the bug is "Failed to disable password"
<smoser> which is done with 'passwd -l'
<smoser> log: https://launchpadlibrarian.net/321286768/cloud-init.log
<larsks> This is setting information for user "ubuntu"?  Is this actually an ubuntu system?
<smoser> the other errors in that log are a result of askb running with a cloud.cfg from trunk
<smoser> (as the confusing user)
<larsks> Anyway, if it is an selinux problem, then there would be corresponding denials logged in /var/log/audit/audit.log.
 * smoser is surprised that larsks's redhat system doesnt have a ubuntu user :)
<rharper> heh, I think this rpm wasn't built right w.r.t centos
<smoser> it was built from trunk
<larsks> It would be interested to see those to confirm the problem.
<rharper> but the centos config in the rpm (/etc/cloud/cloud.cfg)
<smoser> https://code.launchpad.net/~harlowja/cloud-init/+git/cloud-init/+merge/307485
<rharper> need to check that; when I was hand building rpms for centos I got that error (ubuntu user in centos)
<smoser> so... some history...
<larsks> It is also possible that if someone mucked about with the filesystem (e.g., file injection via virt-customize or something) that they simply forgot to relabel the filesystem.
<smoser> askb ran into a bug with 0.7.5
<smoser> i said "that is really old...."
<larsks> So: audit.log or it's something else
<rharper> larsks: ack;
<smoser> he went to try something newer by building trunk
<smoser> and got cloud.cfg from trunk (which isn't really reasonable for redhat)
<rharper> right
<rharper> harlowja: had the split config patch
<rharper> I used that in my branches
<rharper> we need to pull that back end config/cloud.cfg.<distro>
<rharper> s/end/in
<smoser> we should get the merge above so that trunk-built cloud.cfg are good.
<rharper> yes
<smoser> see mp above ^
<rharper> ack
<rharper> I used it about a month ago when I was doing initial passthrough testing
<rharper> I still have some cloud-init rpms i hand built
<smoser> askb, so, the net of all the above.... can you grab a 'dmesg' and a /var/log/audit/audit.log
<rharper> let me see if I still have my rebase dir on that
<sauloaislan> I dont have the cloud-init log but I have neutron-metadata-agent.log http://paste.ubuntu.com/24667212/
<smoser> sauloaislan, the log is inside the guest
<smoser>  /var/log/cloud-init.log
<larsks> smoser: fyi, this is the selinux-python issue: https://bugzilla.redhat.com/show_bug.cgi?id=1406520
<ubot5> bugzilla.redhat.com bug 1406520 in libselinux "calling libselinux python restorecon fails on /var/lib/nfs/rpc_pipefs" [High,Verified]
<rharper> yes
<smoser> larsks, thank youi
<smoser> rharper, https://code.launchpad.net/~xnox/cloud-init/+git/cloud-init/+merge/324010
<smoser> can you change your review there
<smoser> as he updated test cases.
<smoser> oh. wait. that wasnt your complaint.
<smoser> or may be it was.
<smoser> i have the larger set of improvement in my branch
<smoser> but his does fix an issue for him, at low regression risk and want to get that bug fix in
<smoser> mine is more invasive
<smoser> rharper, https://code.launchpad.net/~farcaller/cloud-init/+git/cloud-init/+merge/324273 you could change your NEEDS fixing too
<paulmey> Hi all, some team here on Azure is testing some changes and has an issue with Ubuntu core not starting cloud-init
<paulmey> they have modified the way they are generating the provisioning iso, but I don't exactly know how snappy/systemd pick that up in ubuntu core
<paulmey> Is this a good place for that question?
<smoser> hey
<smoser> sorry.
<smoser> paulmey, yeah.
<smoser> so... there are some  issues still outstanding with getting ubutnu core to r un cloud-init.
<smoser> rharper, has been fighting that fight and can fill you in.
<paulmey> :-)
<smoser> all the changes we think are necessary are in cloud-init currently and i intend to sru that to 16.04
<smoser> (starting today)
<paulmey> yeah I saw a couple of commits in cloud-init in that direction
<smoser> but there are some changes in the image build process that have to be made (in addition to getting this newr version of cloud-init)
<paulmey> we have core 16 running on Azure...
<paulmey> but I'm trying to figure out what change causes it to break now...
<paulmey> I don't know enough about how snappy and systemd work together to get the system up...
<paulmey> for some reason, the whole cloud-init.target doesn't get run...
<paulmey> not sure what that would be conditional on...
<smoser> its explicitly disabled in the image.
<smoser> i thikn they touch /etc/cloud/cloud-init.disabled
<paulmey> oh ok
<paulmey> nah I checked that
<rharper> oh *sigh*
<paulmey> lol
<smoser> funny, eh.
<rharper> well, the second is the bug w.r.t detection of snappy
<rharper> and then we may have other items that come up w.r.t. the network configuration that's baked in that we need to remove that was having some conflicts...
<rharper> paulmey: depending on which versions of the core.snap are included, we have to bake in a few cloud-init configurations that ensure that the cloud identification is found which will enable cloud-init units to run;
<paulmey> Odd_Bloke: you bake this image, right? :-)
<paulmey> https://www.irccloud.com/pastebin/RrttwRMj/
<rharper> hrm, well, Odd_Bloke and I worked together getting something working;
<paulmey> I see a whole bunch of these "/usr/bin/snap[825]: cmd_auto_import.go:198: error: cannot mount /dev/..." errors in the journal...
<rharper> those should have patched parts which do the right thing;   well, then next I would suspect the cloud id bits; typically in that image if cloud-init doesn't run, you can still login to it via serial console interaction; not sure if that's possible in your case though.
<rharper> noisy warnings
<rharper> if you can get into the image then getting to see /run/cloud-init/* would be useful to see if we didn't identify the platform
<paulmey> yeah, I've hacked my way in through the console
<rharper> the UC16 images run in strict mode
<rharper> is it possible that your platform doesn't identify itself in the known way ?
<paulmey> https://www.irccloud.com/pastebin/TiZP9HLY/
<paulmey> yep, that;s the tip I needed, I think
<rharper> paulmey: ok
<rharper> please ping me if you run into any more of those
<rharper> I've spent *way* more time that anyone should have
<rharper> so I hopefully can prevent you from wasting any time
<paulmey> I'll see if I can find out why ds-identify has rc=1
<rharper> it's likely the mounts
<paulmey> it's the cdrom label...
<paulmey> in the error case it is labeled DVD_ROM
<paulmey> has_fs_with_label "rd_rdfe_*" && _RET="$name" && return ${DS_FOUND}
<paulmey> ok, thanks, I can tell them how to fix it now...
<rharper> great
<smoser> wait..
<smoser> was there a bug in ds-identify paulmey ?
<paulmey> not necessarily, that is how you look at it...
<paulmey> this team of ours is changing the way they generate the iso
<paulmey> and apparently they were not giving it the same label
<paulmey> if they can't maintain the same label, I'm sure we'll propose a change to ds-identify
<paulmey> but it will take time for that to percolate into the image in the gallery as well...
<smoser> paulmey, cool.
<smoser> i'd definitely be open to more specific label... or any other way of identifying "You are running on azure"
<smoser> blackboxsw, i'm running a integratino test on trunk, and then will prepare an upload to artful and x, y, z. so witth that i'll ahve a large list of sru bugs
<smoser> (horay)
<smoser> other requests, if you're bored
<smoser> - need unit test for
<smoser>   https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324274
<smoser> - need further review on smoser mask2cidr
<smoser>   https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324677
<smoser> - need further review on smoser aliyun
<smoser>   https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324625
<powersj> :)
<smoser> powersj, its ok if i can just push 'go' ?
<smoser> https://jenkins.ubuntu.com/server/view/Cloud-init/job/cloud-init-integration-a/
<smoser> and
<smoser> https://jenkins.ubuntu.com/server/view/Cloud-init/job/cloud-init-integration-x/
<smoser> can i do them at the same time ?
<powersj> if you run -x it will autokick off y then z then a
<powersj> if you want results faster than sure just kick off a and x
<powersj> and a will get run twice, but that's ok
<powersj> and yes running at the same time is fine
<smoser> o. cool
<smoser> i'll just push 'x' then
<smoser> larsks, just an fyi, after blackboxsw and I both individually broke unit tests on centos6, i asked powersj to get c-i reporting on merge propsals with a run in a centos6 container. so that will hlpefully stop us from breaking you so  often.
<larsks> smoser: also centos7?
<powersj> I will do both
<smoser> powersj, ? i think we can do that. centos6 was the big one though as it is different in 2 ways
<smoser> a.) python2.6
<smoser> b.) not ubuntu
<powersj> We do have daily centos6/7 build and unit test runs
<powersj> Need to add that to our merge request reviews
<smoser> right.
<larsks> powersj: that would be awesome.
<powersj> and not run as root :)
<smoser> we should point out.... its not "real" root.
<smoser> runs in a unpriviledged container
<smoser> but the container thinks its root
<larsks> better than nothing! should at least help highlight issues generating config files or path assumptions, etc.
<blackboxsw> +1 smoser on sru work, I'm adding one more unit tests for ImportError of jsonschema and then my branch is out of the way. Moving on to your review requests right after
<blackboxsw> I restructured the auto-gen'd docs to include Examples for the schema
<smoser> \o/
<rharper> https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/324702
<rharper> that's needed for pass-through equivalency with curtin.net
<smoser> http://paste.ubuntu.com/24670061/
<smoser> blackboxsw, ^ there is our bug list
<paulmey> smoser: there are a couple of ideas floating around in Azure to change the source of the provisioning info and/or the recognizability (if that's a word) of the Azure environment...
<paulmey> none of these have solidified enough to take action, but I'll keep my eye on them...
<smoser> paulmey, yeah. thats good. it really does make sense from a cloud platform perspective to make yourself easily recognizable.
<blackboxsw> excellent smoser want me to start on that now or reviews. I've just pushed final changes to cc-ntp-schema-validation
<smoser> blackboxsw, well, i guess the bug list would be good to start on
<blackboxsw> starting the bug list
<blackboxsw> https://trello.com/c/VxjOm2E5/109-upload-to-artful-and-srus or something else?
<smoser> blackboxsw, twiddled all the ticky marks
<smoser> https://bugs.launchpad.net/cloud-init/+bug/1685935
<ubot5> Ubuntu bug 1685935 in cloud-init "make deb breaks due to missing debuild script" [Medium,Fix committed]
<smoser> that one i referenced in the changelog but shouldnt have.
<blackboxsw> roger smoser
<smoser> i'm heading out.
<smoser> have a nice weekend all.
<blackboxsw> have a good one, thanks
<blackboxsw> smoser: see you tuesday. forgot monday was a holiday
<paulmey> rharper, smoser : I found a better way to identify Azure and created a tracking bug here: https://bugs.launchpad.net/cloud-init/+bug/1693939
<ubot5> Ubuntu bug 1693939 in cloud-init "Switch Azure detection to use chassis_asset_tag" [Undecided,New]
<paulmey> have a great weekend all
#cloud-init 2017-05-27
<askb> smoser: As requested, I have uploaded the logs, dmesg and audit.log for the run with 0.7.5-10 on the bug!
#cloud-init 2018-05-21
<smoser> blackboxsw: did you have changes to https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/345806 that i've not seen / are not pushed there?
<blackboxsw> smoser: nope, I'm about to push the annotate schema branch now, just finished unit tests there, then I need to address your comments on os-locall
<blackboxsw> should be on that branch in about 10 mins
<smoser> ok,. i thought you had said you'd worked on that some. sorry
<blackboxsw> smoser: pushed https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/346415
<smoser> blackboxsw:
<smoser> $ python3 -m cloudinit.cmd.main devel schema --annotate --config  THIS_THING_IS_NOT_A_FILE && echo "that looks good" || echo "fail"
<smoser> that looks good
<smoser> i just found that when incorrectly using your nicely formated and useful 'to test' bit in Description
<smoser> i suspect that part isnt nwe
<blackboxsw> +1 not new, but I should fix it in this brnach
<blackboxsw> yeah the other parts (nonyaml files for scan or parse errors) used to annotate no error prior to this branch
<blackboxsw> I'll add that catch too
<blackboxsw> pushed a fix for that annotation. checking review comments
<smoser> powersj: fixed your run/container issue
<powersj> smoser for shortopts what is r?
<smoser> nothing ?
<smoser> where did you see that ?
<powersj> local short_opts="ahkrsuv"
<smoser> powersj: fixed. and fixed the other missing ones there (-n and -p werre missing)
#cloud-init 2018-05-22
<impermanence> any cloud-init maintainers in here?  I just noticed that the CentOS and RHEL package links for https://cloud-init.io/ are broken, fyi.
<smoser> impermanence: how so ?
<smoser> hhm.
<smoser> indeed they are
<impermanence> yep
<smoser> https://copr.fedorainfracloud.org/groups/g/cloud-init/coprs/
<impermanence> :)
<smoser> dpb1: ^ cloud-init.io links for centos are broken. tomorrow.
<impermanence> thanks for maintaining the project smoser.
<smoser> our pleasure.
<otubo> larsks: I'm trying to get the fedora source tree for cloud-init (not the rpm source, the real git tree) but this link https://apps.fedoraproject.org/packages/cloud-init/sources is broken (for any pakcage actually), do you somewhere else I could get a pointer to it?
<otubo> I just want to confirm latest fedora build includes this commit https://git.launchpad.net/cloud-init/commit/?id=21632972df034c200578e1fbc121a07f20bb8774
<otubo> shardy: ^
<shardy> otubo: you could download the RPM and unpack it with rpm2cpio (or install it) and verify if the commit is applied by inspection?
<shardy> otubo: I've not been involved with cloud-init for a while tbh but IIRC we package the released tarballs vs having any fedora specific git tree
<otubo> shardy: I can get the RPM and check the sources but I just want to find out the fedora git tree for it. Can't be that dificult, right? :)
<shardy> otubo: the packaging git tree is at https://src.fedoraproject.org/cgit/rpms/cloud-init.git/ if that helps
<smoser> rharper or blackboxsw i'd like a review of https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/345786 if you get a chance.
<powersj> smoser: can I get an ack on https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/346404
<smoser> landing
<powersj> thanks!
<smoser> larsks: can you quick update https://code.launchpad.net/~larsks/cloud-init/+git/cloud-init/+merge/345113 ?
<smoser> sorry for churn
<smoser> 0
<larsks> smoser: ack, looking.
<larsks> smoser: rebased. The conflict was because someone went and made spelling corrections :)
<blackboxsw> yeah larsks there was a driveby contributor last week sometime
<blackboxsw> minor pycodestyle on https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/346429 then approved
<smoser> blackboxsw: done.
<blackboxsw> ok just pushed my latest os-local changes to https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/345806  separated ec2 upgrade logic for fallback_interface from common base class
<blackboxsw> should be done pending unit test issues
<smoser> blackboxsw: i'd asked about 'use_network_json' and asked you to drop the docs ... ?
<blackboxsw> smoser: +1 on those suggestions, as per doc separation I'm interested in that as I overlooked othe doc fixes that I wanted because I didn't want to bloat too much, so it's not busy work. I'd have a separate branch anyway for other stuff
<blackboxsw> ok https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/346657 is up for doc updates
<blackboxsw> I'm outta here
<blackboxsw> see y'all tomorrow
<dpb1> o/ blackboxsw
#cloud-init 2018-05-23
<smoser> blackboxsw: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/345630
<smoser> i think thiat is ready
<smoser> and https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/346747 is simple enough.
<blackboxsw> smoser: will grab both.
<smoser> robjo: https://bugs.launchpad.net/cloud-init/+bug/1772961
<ubot5> Ubuntu bug 1772961 in cloud-init "suse (packages/suse/cloud-init.spec.in) spec does not build" [Medium,Confirmed]
<smoser> is that somethign you could look at? if necessary i could/can provide you with a system that you can './run-container' on.
<robjo> smoser: I'll take a look, probably Friday, the next couple of days will be dedicated to dealing with Specter v4 fall out
<smoser> blackboxsw: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/345630 that could use an eyeball. seems mostly non-contentious
<blackboxsw> ... will wrap it up and land it.
<smoser> do we have a bug on https://jenkins.ubuntu.com/server/view/cloud-init,%20curtin,%20streams/job/cloud-init-integration-lxd-a/357/artifact/cloud-init/results/
<smoser> or https://jenkins.ubuntu.com/server/view/cloud-init,%20curtin,%20streams/job/cloud-init-integration-lxd-b/
<smoser> hm.. why did that just now go red
<powersj> smoser: I changed the integration tests yesterday
<powersj> they now use the ppa...
<powersj> but, that means they use upstream tests
<smoser> hm.. but ppa is up to date.
<powersj> yeah so I didn't think we would have a gap
<powersj> but that is what changed
<blackboxsw> lxd-b salt_minion error we have a bug
<blackboxsw> lemme see here
<smoser> well, in -a it was chrony
<smoser> install of chrony failed
<smoser> how do you know what failed ?
<smoser> we really need a nic elist of which tests failed.
<blackboxsw> weird... /me thinks we have a salt_minion bug...
<powersj> salt_minion did fail
<blackboxsw> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1769754
<ubot5> Ubuntu bug 1769754 in cloud-init (Ubuntu) "salt-minion: public/private keys not preserved in /etc/salt/pki" [Undecided,New]
<blackboxsw> I filed against the package, should have been against cloud-init project
<blackboxsw> I can grab that so we see pretty green if folks want
<blackboxsw> I can also try to consolidate the integration test failures to a human readable output at the end
<smoser> hm...
<smoser> blackboxsw: first make c-i like os-local
<smoser> and we can land that
<smoser> rharper: if you could look at https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/344168
<smoser> i think that is close
<rharper> sure
<smoser> blackboxsw: ... i think maybe something push overwrote ?
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/345630 says merged.
<blackboxsw> smoser: I think you and I collided twice
<smoser> i had just pushed https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/346415
<smoser> yeah...
<smoser> but the tools hould fail
<blackboxsw> I tryied running review-mps on your branch
 * blackboxsw is hands off.
<blackboxsw> I'll let you land them
<smoser> hm..
<smoser> i'd have thought it would fail though.  the push should fail if its not ff-only
<blackboxsw> on my side this is what I saw
<blackboxsw> https://pastebin.ubuntu.com/p/GHWBX4bw4J/
<blackboxsw> so the merge failed, but before the push, the script made changes to the branch in Launchpad
<smoser> right.
<smoser> ok. that makes seense.
<blackboxsw> so we should do those changes only after a successful push
<smoser> so in that case it should not change the bugs.
<smoser> yeah.
<blackboxsw> +1  I'll change that now
<blackboxsw> and I'll stop running review-mps
<smoser> also i was just thinking ... since i am re-using a '--local-repo-dir'
<smoser> we should rm -Rf .tox
<blackboxsw> +1 will add it
<smoser> so that it gets re-built with any un-intentionally not pinned versions
<smoser> but first make c-i like your os-local branch
<blackboxsw> ok pushed fix for os-local. awaiting https://jenkins.ubuntu.com/server/job/cloud-init-ci/35/
<blackboxsw> moving around review-mps operations to avoid touching bugs/MPs until publish has occurred
<blackboxsw> s/publish/git push/
<blackboxsw> oops had to rebase for the httpretty changes. running again
<blackboxsw> I'm going to put up an upload for C as os-local is landing
<blackboxsw> ok cosmic upload proposed
<blackboxsw> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/346780
#cloud-init 2018-05-24
<smoser> blackboxsw: nice. i was just about to do that... came in to just go ahead and do an upload
<smoser> blackboxsw: http://paste.ubuntu.com/p/RfPQ3MDVH7/ ? for review-mps ?
<smoser> i could not figure out how to '--skip' more stuff...
<smoser> and --dry-run doesn't work so well because it dry-runs everything (even necessary read-only operations)
<blackboxsw> smoser, I'm adding an alias, dryrun == skip(bugs,publish,test)... only validate/lint the branch and print errors locally (no remote changes)
<smoser> sounds good.
<blackboxsw> I should've changed dryrun when introducing skip bugs, publish, test ... just didn't get around to it
<robjo> smoser: https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/346844  should address parts of lp#1092637
<smoser> i think you typod that number
<smoser> bug 1772961
<ubot5> bug 1772961 in cloud-init "suse (packages/suse/cloud-init.spec.in) spec does not build" [Medium,Confirmed] https://launchpad.net/bugs/1772961
<smoser> ?
<robjo> too many bugs in my Inbox :( , yes cut and paste error
<smoser> robjo: it looks like you still need the 4 changes in https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+ref/fix/1772961-suse-spec-fixes
<smoser> and then we get to
<smoser>  http://paste.ubuntu.com/p/gZpTmTW3jN/
<smoser> robjo: thanks for looking at that.
<robjo> smoser: my changes were only to address the "missing files" issue, i.e. addition to what you had already done
<smoser> ah. ok. well, just grab my commits too and take over that branch.
<smoser> if you'd like i can quickly give you access to an instance that you can use 'run-container' from.
<robjo> sorry still poking around while keeping an eye on the last of the Specter v4 image publishing
<blackboxsw> ok so changes that we landed in cloud-init for salt support freebsd changed the way /etc/salt/pki minion certs are created. Docs (https://docs.saltstack.com/en/2017.7/ref/configuration/minion.html#pki-dir)  still claim that the minion certs should live under /etc/salt/pki/minion  path, but cloud-init now checks isdir(/etc/salt/pki/minion )  and opts to use just /etc/salt/pki if minion subdir isn't present. This breaks
<blackboxsw> on ubuntu as those invalid certs in /etc/salt/pki/* are deleted. But, apparently that is intended behavior on FreeBSD. I need to find out why these files are being removed by a 'system salt-minion restart' on bionic++
 * blackboxsw is trying to resolve https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1769754
<ubot5> Ubuntu bug 1769754 in cloud-init (Ubuntu) "salt-minion: public/private keys not preserved in /etc/salt/pki" [Undecided,New]
#cloud-init 2018-05-25
<craftsmanship> What are the criteria to have a project listed on cloud-init.io? it seems like bsd-cloud-init isn't getting much love these days (or it's moved from github and the link is old).
<smoser> craftsmanship: cloud-init.io is really just cloud-init.
<smoser> we do have some bsd support (freebsd) in cloud-init, and are very open to help making that better.
<craftsmanship> smoser: is there some kind of list of the kind of help needed?
<robjo> smoser: anything else you need from me on lp#1772961
<smoser> craftsmanship: well, we dont have any automated tests for it, and I'm not even sure of how to go about installing it. so that would be two things..
<smoser> unfortunately neither of which are "easy entry"
<smoser> robjo: just go ahead and grab the patches I have
<robjo> OK
<smoser> and put them in your granch... no reason to deal with others. i just fixed things until i hit the end of my knowledge
<smoser> which was pretty quick :)
<smoser> then i called in the experts
<robjo> smoser: all set ~rjschwei/suseSpec branch should be good to go, thanks for working on this and getting tests and builds working on openSUSE :)
<smoser> robjo: http://paste.ubuntu.com/p/G8sHX95JGW/
<smoser> trying again with the trivial leading / added.
<smoser> fudge.. well.. paste fail
<smoser> http://paste.ubuntu.com/p/b9wzqvCVy5/ is how it ended
<robjo> looks to be a macro issue, could be that this macro is only defined in the build service. Since we are only dealing with newer distributions I can expand it
<smoser> robjo: oh. ok.
<smoser> yeah, just adding a / didnt help
<smoser> so your explaination makes more sense
<robjo> smoser: pushed, but we might still be in trouble because in my package I carry this patch: https://paste.ubuntu.com/p/v2D6sG484t/
<smoser> robjo: ok.. i'm fine to take fixes to upstream to figure those both out the right way.
<robjo> what's the "right" way, do we want to import util and use get_linux_distro?
<smoser> robjo: http://paste.ubuntu.com/p/5PM6GkFKNH/
<smoser> \o/
<robjo> yay :D
<smoser> is get_linux_distro the right way to figure that out ? is it a linux distro thing ?
<smoser> that means it builds and tests pass... that doenst mean the rpm is useful in any way :)
<smoser> but... steps at a time.
<robjo> well upstream systemd moved to /usr/lib, but in Ubuntu I know things are in /lib
<robjo> in openSUSE we had one version where things were in /lib then things moved to /usr/lib
<robjo> thus from that I'd say yes it is distro dependent
<hatifnatt> Hello! Is it possible to run set_hostname and / or update_hostname at cloud-init local stage? I'm running it on Debian but I have exactly same issue as described here https://serverfault.com/questions/867473/cloud-init-local-stage-on-freebsd
<rharper> hatifnatt: what cloud-init version are you running ?  18.2 release has this include  https://git.launchpad.net/cloud-init/commit/?id=133ad2cb327ad17b7b81319fac8f9f14577c04df which covers setting hostname before inital dhcp;
<hatifnatt> rharper: I suppose I'm running something old cloud-init  0.7.9-2
<rharper> hatifnatt:  yes; not sure who handles pulling upstream cloud-init into freebsd
<hatifnatt> rharper: No I'm not running FreeBSD, I'm running Debian Stretch. This link with FreeBSD is only example with issue description.
<rharper> ah, right, sorry
<rharper> but my response is the same, not sure when debian will pick up the new releases
<rharper> but the upstream deb should work
<hatifnatt> Yes possibly I can install deb from bionic. Looks like even sid is too old, it's still at 0.7.9-5
<jocha> Hi all I have a question regarding ubuntu 16.04 deployment on Azure, https://pastebin.ubuntu.com/p/fCKNpWPhc4/ :)
<rharper> jocha: the change upstream to the AzureDS would break existing behavior in ubuntu xenial, so in the ubuntu/xenial branch you found, there is a debian/patches/azure-use-walinux-agent.patch which adjust this value back to what xenial cloud-init has always had
<rharper> Description: Use walinux-agent rather than builtin fabric support
<rharper>  Upstream now uses the built-in support for instance initialization on Azure.
<rharper>  On a stable release, we want to continue to use the walinux-agent integration.
<rharper>  Upstream made this change under bug 1538522.
<ubot5> bug 1538522 in cloud-init (Ubuntu) "Calls "service walinuxagent start" in Azure data source" [Medium,Fix released] https://launchpad.net/bugs/1538522
<jocha> thanks! let me take a look and read up about this, will come back if I have any further questions!
#cloud-init 2018-05-27
<hatifnatt> rharper: Hello again, I build cloud-init package v18.2 for Debian but it's still looks like hostname updated too late. According to the logs there is no hostname modification done at 'local' stage. Here is my config and log for local part https://paste.ubuntu.com/p/P9qKRcjWmv/ please take a look if you can. Tnaks in advance.
<hatifnatt> Also I made same test with Ubuntu Bionic cloud image (https://cloud-images.ubuntu.com/bionic/20180518/) and get exactly same result - stub hostname leak to DDNS via DHCP. This image contain cloud-init version 18.2-27-g6ef92c98-0ubuntu1~18.04.1
#cloud-init 2020-05-18
<Odd_Bloke> blackboxsw: https://github.com/canonical/cloud-init/pull/364 <-- I think this is ready for review, while I file the annotation bug and prepare the corresponding change for snap.assertions.
<Odd_Bloke> blackboxsw: FYI, https://github.com/canonical/cloud-init/pull/358 is potentially a candidate for an SRU blocker; we have an SLA for it that falls before we would expect to do another SRU.
<blackboxsw> Odd_Bloke: sorry, are you suggesting we wait trying to start an SRU until this branch lands (and try to work out the patches to Xenial/bionic/focal to enable/disable changes in behavior)
<mock> Hello
<mock> I am drafting a cloud-config file and looking for functionality to load a rpm package downloaded locally to the server at provision time. Can that be done with packages module...or do I need to use runcmd?
<mock> Or is there another channel where this question should be asked?
<Odd_Bloke> blackboxsw: I'm saying that, AIUI, per SLAs we need to SRU this change by June 14th; we can either proceed with the SRU today as planned (and so do two SRUs between now and then), or we can hold that SRU until this change lands.
<Odd_Bloke> blackboxsw: There may be flexibility on that SLA, we should follow-up on that.
<blackboxsw> Odd_Bloke: thanks for clarity. let's SRU today and next SRU will be short
<blackboxsw> we've held off on the SRU process for a while anyway and so it's getting to be a larger bit of verification work
<blackboxsw> the more times we do it the faster it gets
<blackboxsw>  https://github.com/canonical/cloud-init/pull/364 < merged
<Odd_Bloke> I think there's a lot of constant cost to the process (we still manually verify on every cloud, for example), so I'm not sure I entirely agree with that.
<Odd_Bloke> Specifically with "the more times we do it the faster it gets", that is.
<Odd_Bloke> But I do agree that it's been a while, so we probably shouldn't hold off.
<Odd_Bloke> Particularly because I am going to have some review comments here, so we don't have a definite timeline on landing the PR.
<Odd_Bloke> mock: This is the right channel.  Let me make sure I understand: you want to download an RPM file at boot time and install it in the system, is that correct?
<mock> Odd_Bloke: Yes. I'm pulling an rpm from Artifactory to be installed locally on the server where the cloud-config is running.
<mock> I can pull the file--and kinda need to--using runcmd + wget to fetch it. I was looking to see if there was a way to install it once it is downloaded.
<Odd_Bloke> mock: OK, cool.  I think you're probably best off writing a runcmd entry that does this for you.  (You could potentially perform the download in runcmd and then specify a path in packages, but (a) that might not work at all, and (b) if it does, it's not a common or documented configuration so you'd open yourself up to regressions in future.)
<mock> Gotcha. Ok, I'll plan the runcmd route for download and install. I was just looking for functionality if it existed.
<mock> Thanks!
<Odd_Bloke> No problem!
<meena> rharper: thank you for your comment, i left one myself: https://github.com/canonical/cloud-init/pull/363#issuecomment-630351265
<rharper> meena: cool!  it'll be a group effort =)
<meena> rharper: thank you ð
<dgerich> good afternoon, i am trying to inject a cloud-init (cloud-config) into Ubuntu 18 provisioning via Cobbler, and can't seem to get even a simple example to work, where do i need to save the #cloud-config yaml file for it to be executed by cloud-init on boot?
<Odd_Bloke> dgerich: o/ I'm not familiar with Cobbler, unfortunately.  Could you point me at the example you're using?
<dgerich> using examples from: https://cloudinit.readthedocs.io/en/latest/topics/examples.html#
<dgerich> using cobbler to provision Ubuntu 18 via seed (Automatic Installation Template)
<dgerich> i can use wget to download a file with #cloud-config but what confuses me is where that file should be placed in order for cloud-init to pick it up and execute that config
<dgerich> i tried placing it in /etc/cloud/cloud.cfg.d/99_test.cfg but that does not seem to work
<Odd_Bloke> dgerich: cloud-init expects to fetch its user-data (i.e. cloud-config) from the cloud on which its running, not from somewhere on the instance (though there are ways of doing that).  Does Cobbler master the image for you, or is it running at instance boot time?
<dgerich> I am using it to provision VMs in a Datacenter (not on the cloud) I've read that I can use Datasource NoCloud, sorry i am still a bit confused on how its all setup
<Odd_Bloke> dgerich: Not a problem!  https://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html#datasource-nocloud is the documentation for the NoCloud datasource.
<dgerich> also did try to place it here for no effect: /var/lib/cloud/seed/nocloud/user-data
<rharper> dgerich: I think you likely want to use NoCloud-net datasource, and you can pass this via kernel parameters, including a url to where the config can be fetched;     ds=nocloud-net;seedfrom=http://10.245.168.20:34347/     ;  you can replace the http= in the seedfrom to point to a web server you can control; from there, you'd need to provide a directory with the required NoCloud files 'user-data' 'meta-data',
<rharper> dgerich: if you can use the filesystem itself, you put the user-data in the correct location, but you also need a meta-data file, which contains at least, instance-id: <some string>  and local-hostname: <some value here>   ; in addition to your user-data
<dgerich> ah I was only placing the user-data file, so I need to add a meta-data as well to same place?
<dgerich> was /var/lib/cloud/seed/nocloud/ correct location to place both files?
<rharper> yes
<dgerich> ok ill give it a try, thank you
<Odd_Bloke> blackboxsw: I see that you moved us away from flake8 in https://github.com/canonical/cloud-init/commit/80dfb3b02.  I have a spurious pyflakes warning in a branch I'm working on that I'd like to be able to ignore (rather than writing extra code to effectively do the same).  Do you have any objections to me moving us back to flake8?
#cloud-init 2020-05-19
<kqkq95> hi all !
<kqkq95> I have a conflict between cloud-init and customization of the VMware guest operating system, at the first start, the network configuration + name of the VM is configured, cloud-init starts to configure while my system reboots because of VMware who has finished his configuration, someone has already encountered this problem?
<kqkq95> at the next start cloud-init repeats the actions ..
<Odd_Bloke> blackboxsw: Still waiting on an answer to my flake8 Q from yesterday.  (This is a semi-blocker for my work to add a conftest to tests/, which is the only reason I'm nagging you. :)
<blackboxsw> The following changes have landed in master; found via git log --since 05-05-2020 https://paste.ubuntu.com/p/d2qR8pTZNY/
<AnhVoMSFT> @blackboxsw i think cloud-init doesn't really depend on the order of attached device since it's been using the device path (/dev/disk/...). I don't know if this is a new issue or if it has always been there. I can go back to older cloud-init version and check. Will file a bug
<blackboxsw> thanks AnhVoMSFT
<rharper> AnhVoMSFT: blackboxsw:  if you configure more than one additional disk, and provide a disk_setup, fs_setup and mounts;  the default merge behavior combines dictionaries, but lists are replaced, so  both disks go through disk_setup; but only one gets config for fs_setup and mounts
<rharper> https://paste.ubuntu.com/p/KbWYh2jkPw/
<rharper> AnhVoMSFT:  unfortunately, they'll need to replicate the built-in config with the additional disks;   https://paste.ubuntu.com/p/2wkyvD2zhj/
<rharper> if we changed mounts and fs_setup to also accept a dictionary; with the device-name or something to be an unique key; then, that would merge more cleanly by default  https://paste.ubuntu.com/p/4fWT9tGD7T/
<rharper> blackboxsw: ^
<blackboxsw> reading back
<AnhVoMSFT> oh i see - so disk_setup currently accepts dictionary but fs_setup and mounts accept lists, that's the issue
<AnhVoMSFT> thanks for the clarification @rharper
<blackboxsw> Odd_Bloke: sorry to miss that question about flake8. I have no objections to moving to flake8 from pyflakes. there was a flake8 dependency issue that a barely recall and I didn't have strong opinions about moving away from flake8 other than there were py3 dependency issues at the time for flake8 I believe.
<blackboxsw> and rharper thanks, I have limited use of the cc_disk_setup module, and forgot how merging would affect additional disks.
<blackboxsw> Odd_Bloke: falcojr finally responded on https://github.com/canonical/cloud-init/pull/367 I think your suggestions are solid on the approach
<blackboxsw> just some questions on whether we can include feature_overrides.py module in tip to better document our expected behavior on downstream ubuntu/* releases.
<rharper> AnhVoMSFT: if you want to file a bug with that issue, we can discuss and see how much effort to support fs_setup/mounts as dictionaries would be
<rharper> bug 1879552
<ubot5> bug 1879552 in cloud-init "Azure: cloud-init skips formatting the resource disk (ephemeral0) when there is additional data disks' configuration in user-data" [Undecided,New] https://launchpad.net/bugs/1879552
<rharper> good, you already did!
<AnhVoMSFT> yep :)
#cloud-init 2020-05-20
<meena> gonna move some code aroundâ¦!
<meena> anybody wanna help?
<meena> Odd_Bloke: dunno why i thought starting on this https://github.com/canonical/cloud-init/pull/363#issuecomment-628829489 would be the best ideaâ¦
<meena> perhaps, i shouldn't.
<meena> but, as i have noted previously, the other cloudinit/net functions are very rarely usedâ¦ perhaps thats the why
#cloud-init 2020-05-21
<Odd_Bloke> meena: OK, nice!  So I think what you've produced there is closer to what we would have part way (or perhaps most of the way, I haven't looked at the exact set of things you've "migrated") through this migration process.  I haven't fully read through Ryan's comment yet (I was off yesterday), so I'll do that today and perhaps see if I can propose an initial structure for discussion.
<blackboxsw> rharper: Odd_Bloke our debian/control depend on pyflakes (python2-version) anymore?
<blackboxsw> it seems like we should be dropping that. pkg dep right? https://github.com/canonical/cloud-init/blob/ubuntu/devel/debian/control#L11
<lucasmoura> Hi everyone, just a quick question about https://bugs.launchpad.net/cloud-init/+bug/1456277 . The ec2 mirrors should only be used by ec2 instances, right ? Or is there any other scenario that I should still enable them ?
<ubot5> Ubuntu bug 1456277 in cloud-init (Ubuntu) "cloud-init searches for ec2 mirrors regardless of what cloud its on" [High,Confirmed]
<lucasmoura> My idea here will be to enable those mirror only if the datasource platform_type is ec2, but I don't know if that is too restrictive
<powersj> lucasmoura, I think in general AWS would be happy if only instances in their cloud use their mirrors
<rharper> powersj: not sure but I don't think they are accessible from outside aws, no ?
<rharper> it's internal ips that are mapped
<powersj> http://us-west-2.ec2.archive.ubuntu.com/ubuntu/
<Odd_Bloke> blackboxsw: Huh, I'm very surprised that hasn't come up in in Py2 removal, but perhaps Build-Depends aren't checked in the same way.  Yeah, looks like that should be pyflakes3 now.
<rharper> powersj: heh
<Odd_Bloke> I _believe_ that the Azure mirrors use in-cloud-only DNS servers so that only in-Azure instances will resolve the Azure mirrors to the Azure instances, so perhaps you're thinking of that, Ryan?
<Odd_Bloke> lucasmoura: I'm not sure that bug is applicable any longer: `nslookup foobar.ec2.archive.ubuntu.com` gives me NXDOMAIN (whereas `nslookup us-east-1.ec2.archive.ubuntu.com` does return IP addresses).
<Odd_Bloke> The "any subdomain will resolve" logic has moved to *.clouds.archive.ubuntu.com, which resolves to our mirrors by default instead of EC2's.
<lucasmoura> Odd_Bloke, I think I am missing something, wouldn't it be possible for another cloud provider to set up their availability zone to a similar pattern than ec2 and enable the ec2.archive.ubuntu.com/ubuntu/ mirror ?
<Odd_Bloke> lucasmoura: You're absolutely right, I misread the report initially, my mistake.
<lucasmoura> Odd_Bloke, No worries. I will enable that mirror only for ec2 instances
<lucasmoura> Thanks everyone for the help :)
<Odd_Bloke> lucasmoura: I do worry that some cloud deployments may have configured their mirrors on the assumption that the EC2 pattern will always be searched.  (i.e. They have a `us-east-1` region, and they have their internal DNS return their internal mirrors when `us-east-1.ec2.archive.ubuntu.com` is resolved.)
<lucasmoura> Odd_Bloke, makes sense. But is there a way to know that this kind of configuration is expected ? I mean is there any info on the datasource that might suggest that ?
<Odd_Bloke> Nope, unfortunately.  So we need to think a little bit about how we want to release this change.  My initial thought is that this is _fairly_ easy to workaround: if you're hitting a problem as a result of this then you must already have some internal DNS configuration, so adding us-east-1.clouds.archive.ubuntu.com to resolve to your mirrors should be possible.
<Odd_Bloke> (Actually, you could also hit a problem if you rely on the use of the EC2 mirrors but have firewalled off the Ubuntu mirrors.  Again, the workaround is to change your existing configuration, so shouldn't be too hard to deploy a workaround.)
<Odd_Bloke> So I wonder if this is another case where we shouldn't be backporting the behaviour to current Ubuntu releases, but should change it going forward?
<lucasmoura> Yes, I agree that it makes sense to change it only going forward. Should we support this change using the feature flags falcojr is implementing ?
<Odd_Bloke> That's a good question: should this be configurable or not?
<lucasmoura> I guess it shouldn't be, but I don't know if by doing non-configurable I would generate a headache on downstream package maintainers with they want to enable the ec2 mirror lookup. I know it would be still possible to edit this using patches, but it would be harder to find it in the code
<Odd_Bloke> Sorry, let me rephrase: I think we _at least_ want to use a feature flag to capture the two different behaviours.  The question I'm asking is: should we allow this behaviour to be configurable at runtime by users (in which case it would be better as a configuration option than a feature flag)?
<blackboxsw> Odd_Bloke: right, I think https://github.com/canonical/cloud-init/pull/373 is the first upload we've tried after the official drop of pyflakes
<blackboxsw> *from the archive
<blackboxsw> which looks like it happened about a month ago
<blackboxsw> in groovy
<Odd_Bloke> Oh, OK, so I guess this did get caught by Py2 removal, just later removal than I assumed. :p
<blackboxsw> so lucasmoura and falcojr I added and edited the a comment response about what testing we look for  to validate a release. What I caught was that sbuild barked with a missing dependency error. when I tried to run `sbuild-it ../out/cloud-init_20.2-30-g8bcf1c06*dsc`
<lucasmoura> Odd_Bloke, Now I see it. I think the best approach is allowing this to be configurable at runtime by users. But how should we proceed in making this configurable ? I initially thought about adding it to cloud.cfg, but the user can interact with it somehow ?
<blackboxsw> I think this is the last step to validate that we can in fact upload a new-upstream-snapshot to the devel release. I pushed the changeset for a follow up review https://github.com/canonical/cloud-init/pull/373/files#diff-d837a1c17de0268bcea321239ddbed47
<blackboxsw> lucasmoura: mentioned that he had a different debian/changelog in his review of 373. I think tomorrow I'd like to walk through that new-upstream-snapshot creation if we could to see what diffs you were seeing lucasmoura.
<Odd_Bloke> lucasmoura: So presumably we will have this new behaviour enabled by default in groovy.  Under what circumstances would we want users to be able to switch back to the old behaviour?
<Odd_Bloke> (Apologies for the Socractic method I'm employing here. ;)
<lucasmoura> blackboxsw, I made a mistake when I reviewed that. Originally I thought that changelog was auto-generated, but after commenting that I spotted the differance, I realized that there is a commit that updates it
<lucasmoura> So I don't think I have spotted anything unusual, I was just forgot that the changelog is not produce automatically as shown in the PR
<blackboxsw> ahh; ok. that works. ok then I think we are fine here. I'll have a couple of other prs for folks to review in a moment dropping pyflakees from debian/control on all the other ubuntu/* branches we continue support for downstream releases
<lucasmoura> Odd_Bloke, hahah no problem. But what about if we create a new config on apt configure module that enables and disables ec2 mirrors ?
<lucasmoura> In this cc_configure_module function https://github.com/canonical/cloud-init/blob/master/cloudinit/config/cc_apt_configure.py#L946 we can see if the ec2 mirror is disabled and filter out the mirrors
<lucasmoura> We just need to guarantee that this config will always be true for ec2 instances, but I don't know if I config modules should perform actions based on the type of instance we are running
<Odd_Bloke> lucasmoura: You may also want to look at https://github.com/canonical/cloud-init/blob/master/cloudinit/distros/__init__.py#L845-L848
<lucasmoura> Yes, I have seen it when I was trying to understand how the mirrors were parsed from the cloud.cfg file. Do I still don't get your suggestion, do you think we alter the regex to not set the ec2_region ? Do we mean that we should allow this regex to be configurable ?
<Odd_Bloke> lucasmoura: Sorry, give me a few minutes to finish up my response to Ryan's networking refactor comment, then you can have my full attention.
<lucasmoura> Odd_Bloke, no worries
<ACHLO> https://youtu.be/QokZbqwcHwc
<Odd_Bloke> lucasmoura: OK, so "a few minutes" was evidently optimistic.  I don't really have this code in my brain at the moment (though I have touched it recently), shall we hop on a video call and pair for a bit tomorrow so we can investigate it together?
<lucasmoura> Odd_Bloke, Okay, no problem
<Odd_Bloke> rharper: meena: Just dropped a long response to Ryan's comment into https://github.com/canonical/cloud-init/pull/363 (enjoy the read ;)
<rharper> Odd_Bloke: thanks!
<sodapop> I'm trying to append to profile.d this line-> declare -x PATH=$PATH:$JAVA_HOME/bin
<sodapop> can I somehow write it as is and not the expanded
#cloud-init 2020-05-22
<rharper> sodapop:  write_files has an append mode
<blackboxsw> paride: a much belated fix for ubuntu/bionic debian/control https://github.com/canonical/cloud-init/pull/376 thanks for the reviews and ubuntu/focal PR up too https://github.com/canonical/cloud-init/pull/382
<Odd_Bloke> OK, I spent the afternoon collating the discussion from PR 363 into a single place (in-tree): https://github.com/canonical/cloud-init/pull/384
 * blackboxsw get my reading glasses.
<Odd_Bloke> rharper: meena: As you've been most involved, would love if you could look over ^ and ensure that I've accurately represented what we discussed.
<Odd_Bloke> And, yep, blackboxsw lucasmoura falcojr: please take a look too!
<Odd_Bloke> I'll proceed with the implementation of the proposed framework on Monday.
<rharper> Odd_Bloke: yes
<blackboxsw> community notice: [ubuntu/groovy-proposed] cloud-init 20.2-30-g8bcf1c06-0ubuntu1 (Accepted)  upload of near latest cloud-init is published to Ubuntu Groovy (20.10).
<blackboxsw> should be in cloud images in a day or two
<blackboxsw> I
<blackboxsw> we'll be starting the StableReleaseUpdate to Xenial, Bionic, Eoan and Focal with this changeset next week.
