#cloud-init 2014-07-14
<smoser> harlowja_away, ijw i'll read that today. thanks for the link.
<harlowja> smoser np boss
<JoshNang> harlowja: yup! same josh
<harlowja> not the same josh as me, lol
<harlowja> why u take my name, lol
<JoshNang> it's a good name :D
<harlowja> :)
<smoser> harlowja, where was your replace-template thing
<smoser> neve rmind
<harlowja> https://code.launchpad.net/~harlowja/cloud-init/changeable-templates/+merge/208994
<harlowja> smoser how was vacation
<harlowja> u get out of basement?
<smoser> i did. spent some time in indianapolis and baton rouge. said hello to mike the tiger. :)
<harlowja> i only know tony the tiger
<harlowja> he makes cereal
<smoser> i dont know if they're related.
<harlowja> lol
<harlowja> could be, cousins maybe
#cloud-init 2014-07-15
<harlowja> smoser when u gonna do some taskflow reviews?? ;)
<smoser> harlowja, https://twitter.com/jtopper/status/489102959582920704
<smoser> fun
<harlowja> lol
<harlowja> haters gonna hate
<harlowja> i'd be nice to know exactly what he wants in doc
<harlowja> instead of just hating 
<harlowja> but haters gonna hate no matter, lol
<harlowja> smoser if i go open up a twitter account and say cloud-init is awesome, will that help :-P
<harlowja> cloud-init wishlist: its the best
<harmw> lol
<harmw> he could've just come here
<harmw> tons of nice loving people here
<harmw> :p
<harlowja> probably stuck in his hating cave
<harlowja> can't get out
<harlowja> *underground bunker
<smoser> debug log is pretty sucky
<smoser> we can have our own "read mean tweets" session!
<smoser> http://youtu.be/4fB94_DndL8
<harlowja> smoser so the one thing that i can thing that make debug log better is to remove the merging stuff
<harlowja> the merging logs
<smoser> yeah. ther ei s alot of sutff that could be dropped. or dropped to a SUPER_DEBUG
<harlowja> ya, likely just dropped, ha
<harlowja> smoser harmw https://code.launchpad.net/~harlowja/cloud-init/less-noise/+merge/226940
<harlowja> thereee
<harlowja> removed the top 2 noisy ones
<harlowja> mr.twitter guy now can shutup
<harlowja> lol
#cloud-init 2014-07-16
<harmw> smoser: where is the arm disk image for cirros?
<harmw> http://download.cirros-cloud.net/0.3.2/
<harmw> I want to try one before I give my own ARM spins a try
<smoser> harmw,  there is no disk image
<smoser> because hte disk images have grub
<smoser> as the only way to boot
<smoser> i diont know how to make a image that would boot via a system loader on arm
<smoser> ie, we use grub 0.97 for disk images on intel, and the bios will load that. but arm "bios" is either non-existant or widely varied.
<smoser> at some point we'll porbably need grub2 in there, to support powerpc
<harmw> ah yes, I thought it would go down that road
<harmw> perhaps we can use u-boot
<harmw> all embedded stuff I know uses uboot
<smoser> harlowja_away, https://code.launchpad.net/~harlowja/cloud-init/changeable-templates/+merge/208994
<smoser> harlowja_away, whoohoo. my 1 commit per cycle.
<smoser> https://review.openstack.org/#/c/103193/
<smoser> :)
<harlowja> smoser not good, u supposed to fix my bug, lol
<harlowja> smoser on changeable-templates/+merge/208994, adjusting it
<smoser> does it make sense do you think ?
<harlowja> i try not to think
<harlowja> lol
<harlowja> seems to make sense though 
<harlowja> smoser https://code.launchpad.net/~harlowja/cloud-init/changeable-templates/+merge/208994
<harlowja> all updated, bonus points and all
<harlowja> crappy renderer included, lol
<harlowja> smoser what do i get for the bonus points?
<harlowja> a sticker?
<smoser> harlowja, you get a free lifetime cirros 0.3 license.
<harlowja> :-/
<smoser> unlimited cores
<harlowja> hmmm
<harlowja> questionable
<harlowja> lol
<harlowja> i mean, thanks so much
<harlowja> jee golly, i'm special
<harmw> woooaaa
<harmw> smoser: whre do I get a cirros license?
<harmw> mine's about to expire...
<smoser> fax your request to +1555-cirros0
<harmw> darn
<harmw> me don't got no faxin' machine
<smoser> hm.. bummer. 
<harlowja> smoser i tried the number, didn't work
<smoser> hm.. 
<harlowja> smoser another one, just been annoying for my poor 2.6 machine, lol,
<harlowja> https://code.launchpad.net/~harlowja/cloud-init/py26-test-fixage/+merge/227098
<smoser> looks good enough. 
<harlowja> after that i usually only see 1 failure, but its probably cause my base machine is old
<harlowja> http://paste.ubuntu.com/7805294/
<harmw> harlowja: sbruno talked to you already?
<harlowja> nope
<harmw> he apparently did some work on initscripts 
<harmw> atleast thats what he told me like 2 weeks ago :p
<harlowja> cool
<harlowja> tell him to get in here then, lol
<harmw> he said he'd bug you about it
<harlowja> kk
<harlowja> smoser harmw when u guys gonna do some taskflow stuff?
<harmw> taskflow?
<harlowja> no good
<harlowja> lol
<harlowja> http://docs.openstack.org/developer/taskflow/ 
<harlowja> my other project
<harlowja> https://wiki.openstack.org/wiki/File:Core.png (oh ya)
<harmw> +1 for stickman
<harlowja> ;)
<harlowja> find the happy man
<harlowja> extra points if u find him
<harlowja> happy stick man
<harmw> lol
<harmw> specialized worker
<harlowja> ha
<harlowja> u win cirros free edition
<harlowja> for life
<harmw> o-m-g
<harlowja> ya, i know
<harlowja> and u can use taskflow, for life
<harmw> the honor
<harmw> Ill read this tomorrow
<harlowja> cool
<harmw> almost midnight here now
<harlowja> kk
<harmw> bye!
<harlowja> lataa
#cloud-init 2014-07-17
<mgagne> anyone knows where I could find a backport of cloud-init 0.7.x for Precise?
<smoser> mgagne, i've not done one. 
<smoser> i dont think there are too many issues if you just build trunk
<smoser> ./tools/builddeb -us -uc
<mgagne> smoser: I'm looking at PPA and looking for a tutorial on how to backport stuff from trusty to precise so I can do it myself
<smoser> mgagne, yeah. thats the right path. i'  not sure off the top of my head what would be problems.
<smoser> i've done that as recently as 0.7.0
<smoser> https://launchpad.net/~maas-maintainers/+archive/ubuntu/maas-ephemeral-images
<mgagne> smoser: well, my main challenge is: how do I do this? apt-get source cloud-init on trusty, update changelog to target precise, dput to ppa and hope for the best?
<smoser> mgagne, yeah. mostly. you can build it in a sbuild chroot to test locally
<smoser> before going to ppa
<smoser> (i'd recommend that for quicker turn around)
<smoser> but one way or anotehr test it buildling locally before pushing ot a ppa
<mgagne> smoser: lets say I have a bunch of dependencies missing that would need to be backported too. How can I programmatically get the list of missing dependencies so I can automate the process?
<smoser> you shoudln't have too many dependencies
<smoser> for cloud-init ?
<mgagne> smoser: yep, it now requires python-json-patch, python-json-pointer and openstack-pkg-tools
<smoser> hm..
<smoser> let me see.
<mgagne> smoser: it's the first time I create a ppa and backport a package so I'm not familiar with backporting challenges. sorry if I ask a lot of questions =)
<smoser> thats ok. 
<mgagne> smoser: I see backportpackage command exists. Will it take care of dependencies?
<smoser> i dont know. i doubt.
<smoser> where did youthink you needed openstack-pkg-tools ?
<smoser> and python-json-pointer?
<mgagne> smoser: this guy thinks he needs it: http://packages.ubuntu.com/source/trusty/python-json-patch
<smoser> yeah, you do need that.
<mgagne> smoser: python-json-patch needs python-json-pointer
<smoser> ah. 
<smoser> hm..
<mgagne> smoser: so you see the rabbit hole I'm going into
<smoser> well, you can probaly fairly easily drop the jsonpatch 
<smoser> and its not a big deal.
<smoser> something i had hoped to develop more
<mgagne> smoser: ok, I'll try to drop it. I was hoping to not have to modify debian/control and have to version it somewhere =)
<smoser> hm..
<smoser> you'll also have to modify code to do that. i think. not much, it should be contained in 
<smoser> cloudinit/handlers/cloud_config.py
<mgagne> smoser: I just don't understand why python-json-patch needs a run-time dependency on openstack-pkg-tools though
<smoser> not runtime
<smoser> build time.
<smoser> http://paste.ubuntu.com/7810965/
<mgagne> smoser: true, my mistake
<mgagne> smoser: will try your patch, thanks
<smoser> make test will probably fail too
<mgagne> smoser: I guess I can take care of it too
<smoser> yeah, or just disable that if it even runs it in the build
<smoser> i dont remember 
<mgagne> smoser: should be trivial to add your patch and disable associated tests, will start with that for now
#cloud-init 2014-07-18
<smoser> harlowja_away, when you wake up
<smoser>  https://code.launchpad.net/~smoser/cloud-init/changeable-templates/+merge/227323
<smoser> you fix that rendering and we're good
<smoser> :)
<smoser> i'm even fine if you break your more exotic:
<smoser>  ${hostname} 
<smoser> style if you support
<smoser>  $hostname
<harlowja> smoser k, so u want $hostname support, i see :-P
<harlowja> u no like ${hostname}
<harlowja> lol
<harlowja> smoser https://code.launchpad.net/~harlowja/cloud-init/changeable-templates/+merge/208994 updated with non-brace matching as well
<harlowja> sucked over your tests as well
<harlowja> smoser an intersting question is that we could change all the existing templates to just use the basic renderer
<harlowja> instead of the jinja2 based ones
<harlowja> *for the simple ones with 'for' loop like stuff
#cloud-init 2014-07-20
<harmw> smoser: regarding https://bugs.launchpad.net/cirros/+bug/1301958
<harmw> I've got a patch ready, just not the infra to test it on :p
<harmw> (probably later this week)
#cloud-init 2015-07-13
<xenol> hello, I'd like to know if cloud-init supports configuring networking on CentOS 7 as it does on Debian-based systems
<Odd_Bloke> xenol: It should do; if not, please report bugs. :)
<xenol> Odd_Bloke: network-interfaces should also work on centos based machines?
<Odd_Bloke> xenol: Are you seeing behaviour that suggests otherwise?
<xenol> Odd_Bloke: nope, I haven't tried it out as all the examples are for Debian.
<Odd_Bloke> xenol: Which examples are you looking at?
<xenol> Odd_Bloke: well, the CI nic configuration resembles debian way too much and I am not sure if the same mechanism works for CentOS
<xenol> Maybe I was just mislead by network-interfaces syntax
<Odd_Bloke> xenol: Yeah, I think you should be fine using that on CentOS.
<eckes> Hello, I have a cloud-init 0.7.4 on OEL6.6 which is started via openstack kilo (kvm) and it has the problem that on first boot it sets the hostname to "imagename" but writes "imagename.novalocal" into /etc/sysconfig/network (and resolv.conf). I actually want the domain be part of hostname, but it should set that on first init boot as well. Any ideas?
<smoser> eckes, what do you mean domain ?
<eckes> domain name, de ec2datasource reports "novalocal" as domain. It should be part of the FQDN (uts_name, not the resolved one).
<eckes> (or leave the domain off the hostname, its fine too. I just want to avoid the machine has two different host names, depending if it is the first or second boot)
<smoser> oh. i think maybe this bug is fixed in trunk
<ByPasS> if the domain is in the metadata you can always force it within the runcmd: section aswell
<smoser> https://bugs.launchpad.net/cloud-init/+bug/1246485
<eckes> thanks for the reference
<eckes> I guess I need to produce my own image :)
<ByPasS> u can mount the image and edit it aswell
<smoser> fix was : http://paste.ubuntu.com/11873037/
<eckes> yes the bug pretty much looks like my problem, thanks
<eckes> ok works :)
<SpamapS> o/
<SpamapS> Question regarding cc_resizefs.py ..
<SpamapS> How would people feel about a governer put on it that prevents it from resizing a root partition to the point where grub cannot read it, if no /boot is detected?
<SpamapS> smoser: ^
<SpamapS> harlowja_: ^
<harlowja_> uh oh
<SpamapS> governor? anyway... :-P
 * harlowja_ goes back into the box
<harlowja_> *still in the box*
<harlowja_> lol
<SpamapS> harlowja_: who let you out?!?
 * harlowja_ it was the weekend, i got bored :(
<harlowja_> i escaped
<harlowja_> lol
<harlowja_> SpamapS a governer seems ok to me, seems to make sense to not kill the partition so that grub dies
<harlowja_> as long as it doesn't go past 65MPH
<SpamapS> harlowja_: we'd have to leave the current behavior alone.. because PXE booted things and externally booted kernel things like xen might be annoyed.
<harlowja_> k, so a configurable governer that ensures it doesn't go past 65MPH
<SpamapS> but I'm more inclined to just resize it to 1TiB and basically say "resize it again if you really need that"
<SpamapS> it seems like a unique situation that I have.. baremetal servers with one gigantic root disk exposed to the OS
<harlowja_> ya, at y! we've been trying to get out of resizing anything for anyone automatically, they can do it themselves
<harlowja_> it just takes to long
<SpamapS> harlowja_: thats my current workaround.. just disabling resie
<SpamapS> resize
<harlowja_> ya
<SpamapS> though it has also made me think what we really need is LVM because zomg thats a big RAID
<harlowja_> we've been moving to a static thing provided by openstack/ironic/vms and if users need more, they can ask cloud-init to do it, or not, but its not something they can then say 'openstack took to long'
<harlowja_> cause people complain it takes to long, and we remind them its not provisiioning time thats doing this crap
<harlowja_> its there desire to have super-big disk (that they probably don't need, lol)
#cloud-init 2015-07-14
<Odd_Bloke> smoser: I'm introducing an Azure change which starts using devices defined using udev rules for Azure; where in lp:cloud-init should these live?
<smoser> Odd_Bloke, hey.
<smoser> you're saying you'll have clooud-init install udev rules, right ?
<Odd_Bloke> smoser: Yeah.
<Odd_Bloke> (Packagers will have to take care of the actual installation, but we should provide them the rules to use)
<smoser> i guess i'm not opposed to top level 'udev'
#cloud-init 2015-07-15
<Odd_Bloke> smoser: Review of https://code.launchpad.net/~daniel-thewatkins/cloud-init/lp1411582/+merge/264831 would be much appreciated.
<smoser> Odd_Bloke, why are symlinks bad ?
<Odd_Bloke> smoser: Some of the FS tools fall over on symlinks.
<Odd_Bloke> I think mkfs.ext4, for example.
<Odd_Bloke> And to check that it's a valid block device in cc_mounts, we have to dereference it to find the actual device to check.
<smoser> why?
<smoser> not being a jerk, dont rmember. and checking a symlink to a device should be ok
<Odd_Bloke> smoser: We check in /sys/block; that uses the sda1-esque name.
<smoser> Odd_Bloke, you can also check /sys/block/ to see if it is a partition
<smoser> rather than just checking existance.
<smoser>  ie, /sys/block/sda/sda1/partition
<smoser> that file exists iff sda1 is a partition on sda
<Odd_Bloke> smoser: Oh, cool.
<Odd_Bloke> smoser: I've just taken the logic that was already there; I'll make that change if you think it's a good idea.
<Odd_Bloke> I don't really have a sense of what sort of weird block devices we might end up seeing. :p
<smoser> its incredibly inconsistent
<smoser>  https://bugs.launchpad.net/curtin/+bug/1263181
<smoser>  and
<smoser>  https://bugs.launchpad.net/curtin/+bug/1471928
<smoser> your suggestion handles the second naming convention ('p')
<smoser> the first is obscene
<Odd_Bloke> That 'p' logic was already there as well.
<smoser> right
<smoser> Odd_Bloke, claudiupopa to today's call i'd like to talk about blueprints / specs / tracking
<smoser> and what to do.
<Odd_Bloke> smoser: Yeah, that would be good.
<smoser> wish i had easier access to things to test.
<smoser> for p in glob.glob("/sys/class/block/%s/*/partition" % dev):
<smoser>     with open(p, "r") as fp:
<smoser>         if fp.read().rstrip() == part:
<smoser>             found = p
<smoser>             break
<pankaj2934> can we make cloud-init run a local script in image on first boot : default for all builds : even if user does not provide any user-data
<Odd_Bloke> smoser: claudiupopa: harlowja_: You should all have invites to a Trello org in your inbox(es).
<claudiupopa> Yeah, saw it, thanks!
<smoser> pankaj2934, put executable files into /var/lib/cloud/data/scripts/per-instance
<pankaj2934> it should be on controller or into the image
<smoser> in image.
<pankaj2934> doesnt cloud-init wipe out /var/lib/cloud whenit runs first
<smoser> alternatively, vendor-data is provided by the "vendor" , cloud-inti finds it from the datasource, and it can provide can provide boothooks or user-scripts
<smoser> pankaj2934, no. it wipes a link (/var/lib/cloud/instance , which points to the current /var/lib/cloud/instance)
<pankaj2934> does it have to be #cloud-config file or i can just put a shell script in that directory
<smoser> er... instance/<instance-id-here>
<smoser> executable file in that dir is run in run-parts style
<smoser> ie, C locale sorted order
<pankaj2934>  thanks  i will try that .
<claudiupopa> Odd_Bloke, smoser: we have a new public board for the things we need to track: https://trello.com/b/HoPNdiTI/cloud-init-development-roadmap.
<Odd_Bloke> harlowja_: ^
<Odd_Bloke> claudiupopa: Thanks!
<claudiupopa> I started moving a couple of things from the etherpad there.
<claudiupopa> No problem. ;-)
<claudiupopa> basically there are 3 columns, to do, doing and done but I'm open to other suggestions.
* smoser changed the topic of #cloud-init to: cloud-init || cloud-init 2.0 http://bit.ly/cloudinit-reviews-public http://bit.ly/cloudinit-reviews cloud-init || cloud-init 2.0 http://bit.ly/cloudinit-reviews-public http://bit.ly/cloudinit-reviews
* smoser changed the topic of #cloud-init to: cloud-init || cloud-init 2.0 http://bit.ly/cloudinit-reviews-public http://bit.ly/cloudinit-reviews cloud-init || cloud-init 2.0 http://bit.ly/cloudinit-reviews-public http://bit.ly/cloudinit-roadmap
<Odd_Bloke> claudiupopa: smoser: Maybe a column to do with specifications.
<claudiupopa> What kind of specifications?
<Odd_Bloke> The "we're working on a spec but nobody is implementing it yet" type.
<pankaj2934> i have a in /etc/cloud/cloud.cfg.d/09_pk.cfg  , it contains #cloud-config , runcmd: - touch /tmp/pk.txt  .. when the instance boots cloud-init.log has read 66 bytes from /etc/cloud/cloud.cfg.d/09_pk.cfg  but no file in tmp .
<pankaj2934> i have a file *
<smoser> pankaj2934, it should work. unless user-data provides runcmd also.
<smoser> lists dont merge well.
<pankaj2934> not providing anything in user data , target is to run a script like firstboot for every instance : even if user does not provide any user data
<smoser> why did you not like /var/lib/cloud/data/scripts/per-instance ?
<pankaj2934> put the same file there  it did not work .
<pankaj2934> should i be putting .sh file in per-instance
<smatzek> yes
<smoser> i suspect its the same issue. cloud-init final is not running. i think.
<smoser> or for /var/lib/cloud/data/scripts/per-instance if you did that after the instance booted, it wont recognize it on reboot as the marker is just on the instance-id.
<pankaj2934> i deleted the instances folder and retried
<pankaj2934> rebooted
<smoser> can you paste /var/log/cloud-init.log ?
<smatzek> you probably need to delete  /var/lib/cloud/data/instance-id along with the instances directory before reboot.
<harlowja_> Odd_Bloke claudiupopa cool
<smoser> Odd_Bloke, where di you send me an invite ?
<smoser> never mind. i see.
#cloud-init 2015-07-16
<Odd_Bloke> smoser: I'm looking at https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1381776; how do you feel about adding python-serial to cloud-init's dependencies?
<Odd_Bloke> smoser: (On precise)
<Odd_Bloke> And probably trusty, as it's missing there as well.
<Odd_Bloke> smoser: It's already installed on our normal Ubuntu images because of Landscape (via twisted), which is why we haven't seen this bug more.
<Odd_Bloke> smoser: https://code.launchpad.net/~daniel-thewatkins/cloud-init/lp1475215/+merge/264967 is a nice, easy review if you have a minute.
<Odd_Bloke> smoser: (I'm currently assuming that we're code reviewing everything that goes in to 0.7.x)
<smoser> Odd_Bloke, merged thanks.
<smoser> i'm surprised. about 1381775.
<smoser> i'm surprised. about bug 1381776
<Odd_Bloke> smoser: Thanks. :)
<Odd_Bloke> smoser: So you only need it installed if you include a DS that requires it in your list of DSes.
<smoser> i tihnk its robably just an oversite
<smoser> at very least it should be Recommends
<Odd_Bloke> But by default, you will have such a DS.
<Odd_Bloke> It's in Depends in vivid.
<smoser> ah. ok.
<Odd_Bloke> So, I agree, I think it's just an oversight.
<smoser> well then.
<smoser> fix if you want.
<smoser> i'd like to not hvae python-serial, as i think it is not pure python.
<smoser> and not really sure why you can't just open up /dev/ttyS1 and write to it.
<smoser> Odd_Bloke, https://code.launchpad.net/~daniel-thewatkins/cloud-init/lp1411582/+merge/264831
<smoser> why azure_resource-part1 and azure/resouce-part1 ?
<Odd_Bloke> smoser: The first is what we put in place with the cloud-init udev rules.
<Odd_Bloke> smoser: The second is what walinuxagent puts in place with its udev rules.
<smoser> do i care about the second ?
<smoser> i'm not really opposed to this, but just wondering.
<smoser> and maybe lets at least explain/comment to that effect
<Odd_Bloke> smoser: If the packaging for cloud-init is done badly, but the packaging for walinuxagent is alright, this will still work.
<Odd_Bloke> smoser: (So really more of a consideration for non-Ubuntu systems)
<smoser> if the packaging for cloud-init is done badly, then we should fix that :)
<smoser> i'm ok with it. but can you add a comment to why we're searching some seemingly non-existent path ?
<Odd_Bloke> smoser: Yep, good shout.
<smatzek> on the topic of adding dependencies to the cloud-init package for individual datasources, I'd rather not have them as hard dependencies.  Some of those dependencies are sometimes not available for some Linux distros on non-x86 architectures.
<smatzek> that being said I see the point of view of including them as depndencies and distros should adapt the rpm requirements list appropriately when they build the RPMs.  I just know I had trouble with this with dmidecode which was not available for ppc64 in either RHEL or SLES or both, I can't remember which.
<smoser> yeah, its hard though.
<smoser> because many people will not have need of many of these things.
<smoser> if they're building an image, they dont necessarily need all datasources functional.
<Odd_Bloke> Yeah.
<Odd_Bloke> Maybe it should be a Recommends.
<smatzek> in v2 this could be solved by packaging each datasource or groups of related datasources together in their own package which updates cloud.cfg to add themselves in.  Then image builders could would install datasource packages they want available.  This seems like a lot of extra overhead to solve this issue though.  It could be solved by Recommends and then documentation on which datasources require which extra packages above and beyond the baseline
<smatzek> requirements.
<Odd_Bloke> Yeah.
<smoser> we do have a goal of datasources being more stand alone
<Odd_Bloke> There are only a few use cases where you would actually care about having the extra handful of Python libraries.
<Odd_Bloke> So Recommends is probably the way to go, then people who care can uninstall/not install requires.
<Odd_Bloke> s/requires/recommends/
<smoser> for ubuntu i think recommends makes sense for stuff that could be annoying. and i'm not opposed to packaging datasources seperately uon ubuntu either.
<smatzek> if we go the route of recommends, the datasources that have requirements above the baseline like dmidecode which has native code (AltCloud, SmartOS), Sigma, etc should try-except around their imports to handle it gracefully and put a nice message out that says "we require package xyz for datasource abc"
<smoser> yeah
<Odd_Bloke> True.
<Odd_Bloke> So for python-serial in precise and trusty should I: (a) Depends, (b) Recommends, or (c) Recommends and code changes as per smatzek's comment?
<Odd_Bloke> (c) is a bit of PITA to do in maintenance.
<smoser> make same change as in vivid
<Odd_Bloke> Cool.
<smoser> Odd_Bloke, for 2.0, how can i render README.rst easily ? to test see if my change is sane?
<smoser> this seems to work:
<smoser>  tox-venv docs rst2html README.rst  > /tmp/out.html
<Odd_Bloke> That seems reasonable.
<openstackgerrit> Scott Moser proposed stackforge/cloud-init: README.rst: mention bugs are tracked in launchpad  https://review.openstack.org/202576
<Odd_Bloke> smoser: Do we want to track bugs in the same place as 0.7.x?
<Odd_Bloke> I worry it might become confusing.
<smoser> Odd_Bloke, i dont know. i dont want two separate projects in launchpad.
<smoser> i did make a 2.0 series, but i've not played much
<Odd_Bloke> Why not separate projects?
<smoser> Odd_Bloke, i think separate projects is more confusing than single project.
<smoser> and in 3 years, you'll say "why is this one named cloud-init 2.0"
<Odd_Bloke> Provided we can clearly delineate between 2.0 bugs and 0.x bugs, that's not a problem.
<Odd_Bloke> But if we have the same bug in each cloud-init, a single commit won't fix it.
<Odd_Bloke> And so it might be a bit harder for people looking at bugs to understand whether they should be fixed or not etc.
<smoser> https://launchpad.net/cloud-init/2.0
<smoser> i'm not really sure what all you can do with 'series'
<smoser> but i think essentially 'wily' is a series
<smoser> i thikn launchpad intends to model what we need.
<Odd_Bloke> Cool. :)
<smoser> yeah, https://launchpad.net/ubuntu/+series
<smoser> thats heavier use then i surely would hope to have.
<pankaj2934> what is the diffrence user for  /var/lib/cloud/scripts/* folder and /etc/cloud/cloud.cfg.d/ folders
<pankaj2934> use for *
<pankaj2934> once the instance is booted up i want to have cloud-init make a REST call to a service . what would be best place to place the code .
<pankaj2934> once the instance is booted up i want to have cloud-init make a REST call to a service . what would be best place to place the code .
<smoser> pankaj2934, /var/lib/cloud/scripts/* are executables
<smoser> /etc/cloud/cloud.cfg.d/ is configuration
<smoser> you might be able to just use 'phone_home'
<smoser>  http://cloudinit.readthedocs.org/en/latest/topics/examples.html
<pankaj2934> @Smoser so if i write a abc.cfg file with phone_home code in it (as from the example ) and put the file in /etc/cloud/cloud.cfg.d/ of the image anytime a new instance is created phone_home call will be made. ?  can we haev multiple phone home in multiple cfg file ?
<smoser> all config files get merged into a single config
<smoser> so "last one wins" that defines the phone_home url
<Odd_Bloke> smoser: It looks like you didn't push to Launchpad when you did the last Vivid SRU; would you be able to push that up?
<smoser> probably, let me check
<smoser> done
<Odd_Bloke> Danke.
<openstackgerrit> Scott Moser proposed stackforge/cloud-init: add cloud-init main  https://review.openstack.org/202743
<smoser> Odd_Bloke, ^
<smoser> harlowja_, we need some test coverage on logging and url_helper bad.
<harlowja_> more test coverage?
<harlowja_> on logging :-/
<smoser> http://paste.ubuntu.com/11889248/
<harlowja_> i thought it was 200%
<harlowja_> 200% covered all the things
<harlowja_> ah, the mysticism of coverage numbers :-P
<smoser> harlowja_, yu're not looking at writing test 'right nwo' are you? cause if not, i'm gonna spend next 30 minutes or so.
<harlowja_> i'm not
<harlowja_> u can :-P
<smoser> k. thank you kind sir.
<harlowja_> np
<harlowja_> lol
<harlowja_> np sir
<pankaj2934> scenario: user puta hostname:mywebserver in openstack horizon and server gets named mywebserver. but our admin team has a naming standard. is it possible to restrict usrs to pass only certain user-data and not all
<smoser> you' dhave to enforce that on openstack side.
<pankaj2934> how can that be done. i could not find any option in nova so i thought there might be somethign in cloud-init
<smoser> no. cloud-init doesnt have anythign like it.
<smoser> youd have to change openstack to just load the user-data on the way in.
<smoser> inspect it.
<smoser> you probably dont want to do that.
<pankaj2934> yes : biggest problem we have developers have learnt cloud-init and they haev started putting user-data whcih runs as root and deviates the build from standard
<pankaj2934> can we have cloud-init not process user-data but still process everything in cloud.cfg.d and scripts folder ?
#cloud-init 2015-07-17
<openstackgerrit> Daniel Watkins proposed stackforge/cloud-init: Convert reporting handlers to be instantiated.  https://review.openstack.org/202960
<openstackgerrit> Daniel Watkins proposed stackforge/cloud-init: Implementation of a simple registry.  https://review.openstack.org/202990
<openstackgerrit> Daniel Watkins proposed stackforge/cloud-init: Convert reporting handlers to be instantiated.  https://review.openstack.org/202960
<smoser> Odd_Bloke, what is ropeproject ?
<smoser> https://review.openstack.org/#/c/202960/2/tox.ini
<Odd_Bloke> smoser: A Python refactoring library which vim can use.
<smoser> you can teach me how to use that sometime.
<smoser> please
<smoser> do you have any idea why py26 was broken and is now fixe d?
<smoser> ie, initially jenkins https://review.openstack.org/#/c/202576/ failed. the 'review' comment made it happy
<smoser> er.. recheck
<Odd_Bloke> smoser: It looks like a new version of mock was released between the first and the second.
<Odd_Bloke> 1.1.3 is installed in the failing one, and 1.1.4 in the succeeding one.
<Odd_Bloke> Which would make sense; I'd heard from someone else that they'd broken 2.6 support in mock.
<Odd_Bloke> Presumably they have now fixed it.
<Odd_Bloke> smoser: The registry is just a library thing that we can use where we need it.
<Odd_Bloke> smoser: I'm going to use it in two ways in the reporting stuff: firstly, a registry of available reporters (i.e. mapping config names to actual classes), and secondly, a registry of the instantiated reporters (which the actual event reporting code will use).
<Odd_Bloke> smoser: There will probably be further changes to it, but I figured getting the basic stuff out of the way would make other changes easier to review.
<smoser> i really hate the internet
<Odd_Bloke> ?
<smoser> Odd_Bloke, i hate that random dude on the interenet uploads mock 1.1.3 and breaks my tests
<smoser> causing me to have to spend time figuring out why they're broken.
<smoser> i find that anti-productive.
<Odd_Bloke> smoser: Random dude on the Internet also wrote mock, meaning that writing our tests is much easier.
<Odd_Bloke> So I wouldn't be too harsh on him. ;)
<smoser> clearly. and he probably makes less stupid mistakes than i do.
<smoser> but being human the number of stupid mistakes per unit of time > 0.
<smoser> and the count of random dudes is > 0
<smoser> so the time lost on such things is definitely non-zero
<Odd_Bloke> I'd still argue less than the cost of reimplementing everything yourself.
<Odd_Bloke> So net gain is positive.
<smoser> i'm not complaining about using libraries
<Odd_Bloke> We should consider pinning our dependencies though.
<smoser> i'm complaining about them moving
<smoser> i complain when you break something in an ubuntu sru
<smoser> i like stable
<smoser> Odd_Bloke, i'll pll your reporting handlers instantiated
<smoser> i know its obnoxious
<smoser> i know its painful.. but please do not just throw additional changes in
<smoser> you're more than welcome to keep me honest on that also
<Odd_Bloke> smoser: 'do not just throw additional changes in'?
<smoser> ropeproject
<Odd_Bloke> Ah, yeah.
<Odd_Bloke> Yep, will avoid doing it in future.
<smoser> Odd_Bloke, you have a thought on how to fix http://logs.openstack.org/43/202743/1/check/gate-cloud-init-python27/1cf650f/console.html
<smoser> http://paste.ubuntu.com/11893139/ is what ihave, an dit does work, but what is in that review now seems prettier
<Odd_Bloke> smoser: Having a look now.
<smoser> if i dont catch stderr, it goes to console
<Odd_Bloke> smoser: Patch 'cloudinit.shell.sys.stderr' and then look at .write on the object passed to the test method.
<Odd_Bloke> The problem is trying to swap out just the 'write' method.
<Odd_Bloke> smoser: Also, you have shell.py and test_main.py; would be good to make those the same name (as it took me a while to find one from the other). :)
<smoser> k
<smoser> so you can do 'cloudinit.shell.sys.stderr' ?
<smoser> and that patches cloudinit.shell's usage of sys.stderr only ?
<Odd_Bloke> Yep.
<Odd_Bloke> (Unless anywhere else is doing something stupid like 'from cloudinit.shell import sys', but that shouldn't every make it past code review :p)
<harlowja_at_home> smoser, https://github.com/openstack/oslotest/blob/master/oslotest/output.py is what most of openstack does for this
<harlowja_at_home> perhaps u can do something similar?
<harlowja_at_home> yes yes, i know, more fixtures :-P
<harlowja_at_home> but something like https://github.com/openstack/oslotest/blob/master/oslotest/output.py#L46 might work for u
<openstackgerrit> Merged stackforge/cloud-init: Convert reporting handlers to be instantiated.  https://review.openstack.org/202960
<openstackgerrit> Merged stackforge/cloud-init: README.rst: mention bugs are tracked in launchpad  https://review.openstack.org/202576
<openstackgerrit> Daniel Watkins proposed stackforge/cloud-init: Implementation of a simple registry.  https://review.openstack.org/202990
<openstackgerrit> Daniel Watkins proposed stackforge/cloud-init: Use a registry to configure reporting handlers.  https://review.openstack.org/203093
<openstackgerrit> Daniel Watkins proposed stackforge/cloud-init: Implement a DictRegistry.  https://review.openstack.org/203094
<smoser> harlowja_at_home, Odd_Bloke http://paste.ubuntu.com/11893290/
<smoser> that looks pretty nice, and works in py27 and py34, but raises
<smoser>  http://paste.ubuntu.com/11893291/
<smoser> on py26.
<smoser> i believe that we discussed something about this , but i am not recalling details.
<Odd_Bloke> smoser: This comes back to my not wanting to use testtools' TestCase, because you can't use that assertRaises syntax on 2.6.
<Odd_Bloke> (Whereas you can if you use unittest2.TestCase)
<harlowja_at_home> trade-offs either way
<smoser> ok. i'll avoid it for now. it does look pretty though :)
<Odd_Bloke> testtools should be using unittest2 by default (at some point).
<Odd_Bloke> Oh, it does.
<Odd_Bloke> But overrides the context-manager assertRaises. /o\
<smoser> Odd_Bloke, http://paste.ubuntu.com/11893330/
<smoser> that works, does it seem sane to you ?
<Odd_Bloke> smoser: You should be able to do something like 'sys_exit_exc = self.assertRaises(SystemExit, ...)', but I'm happy with that.
<Odd_Bloke> smoser: Though maybe go for 'assertIsNotNone' instead of 'assertTrue', to make it clearer that you're trying to avoid the value you set initially.
<smoser> i tried that.
<smoser> AttributeError: 'TestMain' object has no attribute 'assertIsNotNone'
<Odd_Bloke> Everywhere or on 2.6?
<smoser> 2.6
<Odd_Bloke> *flips table*
<smoser> Odd_Bloke, with self.assertRaises ...
<smoser> can i get the exception that was raised ?
<smoser> cause otherwise i can't check the .code
<Odd_Bloke> smoser: Yeah, it should return it.
<smoser> oh. i didn't realize that. nice.
<Odd_Bloke> smoser: But maybe not on Python 2.6, because apparently everything I know about unittest isn't true there. ;)
<smoser> i dont think it does.
<smoser> http://paste.ubuntu.com/11893374/
<smoser> produces http://paste.ubuntu.com/11893375/
<smatzek_> unittest lacks various assert methods in 2.6 and it's a pain.  I ended up having to write my own that mimiced the 2.7 methods in a parent test class to UT on 2.6 when my project used to support that level.
<Odd_Bloke> smoser: What about with testtools.TestCase?
<smoser> yeah, there its fine.
<Odd_Bloke> Right, that's the superclass of cloud-init's TestCase.
<Odd_Bloke> So you _should_ be fine.
<Odd_Bloke> But, again, YMMV. :p
<Odd_Bloke> smatzek_: unittest2 FTW. :)
<Odd_Bloke> Unless you're working on a project with harlowja_at_home, apparently. ;)
<harlowja_at_home> and/or all of openstack, lol
<Odd_Bloke> harlowja_at_home: Aren't those the same thing? :p
<harlowja_at_home> :)
<harlowja_at_home> perhaps, hahaha
 * harlowja_at_home pumps out various thousands of lines of code a day (under different names)
<harlowja_at_home> don't tell anyone....
<Odd_Bloke> I've never actually seen harlowja_at_home in the same room as anyone else.
<harlowja_at_home> why testtools overrode assertRaises u got me
<Odd_Bloke> So you could theoretically be everyone.
<Odd_Bloke> That's hard science.
<harlowja_at_home> quantum space-time
<Odd_Bloke> harlowja_at_home: I'm guessing it was for 2.6 backwards compatibility before they used unittest2.
<harlowja_at_home> prob
<harlowja_at_home> and the api stuck
<harlowja_at_home> and thats how the story was written
<Odd_Bloke> Well, it's their own implementation of assertRaises.
<Odd_Bloke> So it would probably take work to make it a context manager.
<Odd_Bloke> So I suspect it's just that no-one has cared. :p
<Odd_Bloke> Perhaps I'll care at some point.
<harlowja_at_home> ya, or the api they created for assertRaises in testtools can't now be changed (since people depend on its behavior) and its just unforutante that python/unittest2 used the same name
<harlowja_at_home> but probably really easy to add a proxy method to the unittest2 one under a different name...
<Odd_Bloke> You could make it a context manager without changing the API for non-CM uses.
<harlowja_at_home> ok, or that
<harlowja_at_home> get er done
<Odd_Bloke> :p
<openstackgerrit> Scott Moser proposed stackforge/cloud-init: add cloud-init main  https://review.openstack.org/202743
<openstackgerrit> Scott Moser proposed stackforge/cloud-init: add cloud-init main  https://review.openstack.org/202743
<openstackgerrit> Scott Moser proposed stackforge/cloud-init: improve test coverage  https://review.openstack.org/203249
#cloud-init 2016-07-18
<smoser> harlowja, yeah, i agree, it is odd.
<harlowja> smoser does the canonical cloud-init syslog stuff work?
<harlowja> (in the ubuntu images)
 * harlowja hasn't checked what the cloud.cfg for logging is in one of those
<smoser> harlowja, yes.
<smoser> it does change
<smoser> which is kind of obnoxious
<smoser> ie, cloud-init local comes up and logs to a file in one format
<smoser> and then cloud-init comes up (after syslog is up) and logs through syslog to the same file (/var/log/cloud-init.log) but the log file format changes.
<smoser> i do want to support logging through syslog though, as it is just very useful
<harlowja> ya, seems the redhat version though is sorta messed up though, ha, and file is possibly a better 'actually works' default
<harlowja> vs the reverse, where syslog is the default and people do various config (that dont seem to work) to get a /var/log/cloud-init.log
<harlowja> vs just having a file log to /var/log/cloud-init.log  as the default
<smoser> holser_, around ?
<smoser> holser_, https://code.launchpad.net/~smoser/cloud-init/trunk.mcollective-cleanup
<smoser> and https://code.launchpad.net/~smoser/cloud-init/trunk.mcollective-cleanup/+merge/300394
<holser_> Ð Ñ
<holser_> Hi
<holser_> I am here
<holser_> Iâll test it tomorrow morning
<holser_> and let you know
#cloud-init 2016-07-19
<harlowja> smoser https://code.launchpad.net/~harlowja/cloud-init/cloud-init-fix-dnf/+merge/298851 ok with u?
<harlowja> one less redhat patch...
<holser_> smoser: I have reviewed your recent refactoring of cc_mcollectived. I approved it.
<smoser> holser_, great. i have some changes here to avoid the obnoxious testing of handle
<smoser> holser_, did the changes make sense to you?
<holser_> yes
<simar> hey folks, I have question regarding hostname assignment using cloud-init.
<simar> I tried creating a cloud-config within /etc/cloud/cloud.cfg.d in order to set the hostname and the fqdn for my host.
<simar> But for some reason it didn't take effect after a reboot.
<simar> Would I need the update_hostname module set in the cloud_init_modules as well for this?
<simar> Open to other ways of doing this  if there are any.
#cloud-init 2016-07-20
<Odd_Bloke> simar: By default, modules will only run once per instance (i.e. on first boot), so it may be that whatever updates the hostname is configured to only run once.
<simar> Odd_Bloke: I see, so defining a cloud-init module within cloud.cfg.d wouldn't have any effect if the hostname was previously configured by something else?
<simar> The reason I asked is because me doing so didn't result in any changes. I'm not sure if the naming of the config matters, I made sure it came in order after all other configs before that.
<harlowja> @smoser did we start getting CI???
<harlowja> automatic CI
 * harlowja just got message from diogo.matsubara+server-team-bot@canonical.com :-P
<nacc> heh, that should get renamed :)
<nacc> powersj: --^
<harlowja> :-P
<nacc> harlowja: that was the server team's old QA person
<harlowja> ah
<powersj> haha
<harlowja> nice, ha
<nacc> harlowja: powersj is the new one
<powersj> and yes.. CI is going
<harlowja> woah
<nacc> harlowja: nad i think powersj resurrected CI
<nacc> *and
<harlowja> is there any way to get the URLs accessible outside of canonical :)
<nacc> heh, probably a good thing to do :)
<harlowja> don't think i can view them (firewall...?)
<powersj> It just got through all the existing merge requests and is now on 5min cron to review
<nacc> and/or don't send internal URLs to an external list
<powersj> Yeah the CI jobs are internally ran
<harlowja> ya, internally ran is fine, having the output vieweable would be nice
<nacc> right
<harlowja> (otherwise hard to know what broke)
<harlowja> that'd be sweet
<harlowja> cool they are running though :)
<powersj> ok let me talk to smoser and rharper about it
<harlowja> cool, thx powersj and nacc :)
<harlowja> cool to see progress here
<harlowja> smoser where is git now ;)
<smoser> powersj, yeah, we want urls that are available...
<smoser> curtin too
<powersj> figured :) this would mean a new jenkins instance
<Odd_Bloke> simar: cloud-init will already have run when you launched the instance; it will have done hostname stuff and marked that as "done".  When you reboot, it will consider whether or not it needs to do it, see that it has already been done and skip doing it again.
<Odd_Bloke> simar: (This is so that, for example, if you change your hostname away from what your cloud provider sets it to by default, it won't get reset every boot)
<powersj> CI email address should be updated (at least my inbox says it is... holy cow)
#cloud-init 2016-07-21
<boolman> hey guys, How do I create an account with the password in plain text? I tried this http://pastebin.com/HkVAZT24 but it doesnt work
<boolman> i also tried in quotes, eg 'installer'
<boolman> ohh I needed to set lock_passwd: False
<smoser> boolman, yeah.
<larsks> smoser: any thoughts on when we might see 0.7.7?
<larsks> Oh, huh, ubuntu ships a package based on a pre-release checkout it looks like.
<smoser> larsks, i acknowledge we need one.
<smoser> i'd 'like to get a list of bugs that we should target, and just start killing them.
<smoser> so long...
<smoser> since last release.
<larsks> Yeah.  I'm carrying a bunch of backports in the RHEL package and would love to stop doing that :)
<larsks> I am happy to help squash bugs if you have a list of release blockers..
<smoser> larsks, are you subscribed to https://launchpad.net/~cloud-init
<larsks> I am now :)
<smoser> ok.
<smoser> bah. harlowja not here.
<smoser> :-(
<smoser> holey moley. speak of the devil
<smoser> harlowja, i said your name literally not 45 seconds before you joined
<harlowja> lol
<harlowja> i have risen
<harlowja> from a conference in arizon
<harlowja> *arizona
<harlowja> lol
<larsks> harlowja is in your irc servers, watching your messages...
<smoser> larsks, is asking fora release.
<harlowja> ah
<harlowja> has it reached 2 years yet?
<smoser> i think harlowja is outside my window!
<harlowja> we can only release every 2 years
<harlowja> lol
<harlowja> do we want to have that CI working before  a release
<harlowja> make sure things are tidy before that?
<harlowja> (might be nice to have that same CI activate the building-packages code, ensuring it works)
<harlowja> then sure?
<smoser> this is true...
<smoser> well i'd like to get a release to have something.
<smoser> and then we can add more releases.
<harlowja> sure
<smoser> hopefully less than 2 years from now.
<harlowja> lol
<harlowja> sooo the other stuff i was liking to do was to have a cloud.cfg that was for fedora
<harlowja> fedora/redhat
<smoser> the rsyslog thing is obnoxious for sure.
<harlowja> ya, that to
<larsks> harlowja: you mean, a fedora compatible cloud.cfg in the distribution?
<smoser> so i'd like to do a round of bug triage, and we can target some to a 0.7.8 release.
<harlowja> in the tree larsks
<smoser> and then we can go through and move them out if we want
<harlowja> at least for people to look at, vs having to find the source rpm, know how to extract it ...
<smoser> er... i guess 0.7.6
<smoser> er... i guess 0.7.7
<harlowja> i mean we have a brpm tool in tree, but if it uses the default cloud.cfg, that probably not gonna work out the best
<smoser> i'm so used to seeing 0.7.7~ that i have started to assume that that was the releas.e
<larsks> Okay. Any objection to just grabbing what's currentinly in the fedora packages and committing it?
<harlowja> larsks not from me
<harlowja> there is a syslog patch in the fedora package as well
<harlowja> which goes back to how its weird and i'd almost like file logging to be the default in cloud-init
<harlowja> (providing a logging config that is basic by default, and known to work)
<larsks> The only fedora rsyslog related patch appears to be a one-liner changing the match critera in the rsyslog config?
<harlowja> ya, so i've tried that one, it doesn't seem to work, lol
<harlowja> at least when i was using it
<larsks> Having said that, I think the logging is a little mucked up right now anyway.  I would rather we were just spitting everything to stdout/stderr and let the system take care of logging it.
<harlowja> larsks ya, or just dump it to a file (by default)
<harlowja> which we can do with a 1 line change
<harlowja> i'd rather not have mucked up logging, but simple stuff that should just work, lol
<smoser> alright. so very informal. lets say we do a release of 0.7.7 2 weeks from today.
<harlowja> 2 years from today
<harlowja> lol
<harlowja> wfm
<smoser> :-)~
<harlowja> can we arrange to do it on smoser bday
<harlowja> when is your bday
<harlowja> lol
<larsks> Looking at cloud.cfg...it would be nice if the distribution-specific parts were in something like cloud.cfg.d/distro.cfg or something, so that we could have a common set of modules enabled by default across distribution.  Does that make sense at all?  Right now the cloud_*_modules sections differ substantially between stock/freebsd/fedora...
<larsks> Maybe that gets too complicated.  I dunno.
<smoser> well, for more complexity there, i'd rather hav the modules declare supported distros and improve how the list is done and such.
<harlowja> smoser right, we have basic support for that already
<harlowja> it more though just results in a 'warning' if the module supports an operating system and its being ran on a different one
<harlowja> so perhaps just further stuff there
<harlowja> then we can just 'list all modules'
<harlowja> and let the runtime figure out what should/shouldn't work
<harlowja> 3https://code.launchpad.net/~harlowja/cloud-init/cloud-init-file-logging/+merge/300798 is the logging change
<harlowja> if we desire it
<harlowja> * https://code.launchpad.net/~harlowja/cloud-init/cloud-init-file-logging/+merge/300798
<harlowja> i don't think most people know that http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/stages.py#L808 exists :-P
<smoser> ok. harlowja larsks i just added a team name cloud-init-bugcontrol
<larsks> I am fine with the idea of just logging to a file by default.  Can we get file logging directly without spawning a new process like that?
<smoser> and makde it the "bug supervisor" for http://launchpad.net/cloud-init bugs
<harlowja> larsks spawn new process?
<larsks> harlowja: output: {all: '| tee -a /var/log/cloud-init-output.log'}
<smoser> you should be able to move state of bugs now i think
<harlowja> that's all in smoser realm for that one :-P
<smoser> larsks, log to a file without a process ?
<smoser> you just want to avoid the tee?
<larsks> smoser: log to a file just use the python logging api, rather than having to pipe output to something.
<harlowja> that one i think is for 'print'
<larsks> That just strikes me as weird.  But whatever works...
<smoser> right.
<smoser> output: handles any subprocesses
<larsks> Oh right, makes sense.
<harlowja> right, also subprocesses
<larsks> Neeeeeever mind.
<smoser> cloud-init makes its standard out (and inherited) go to places described in output:
<harlowja> we can alter that to adjust stdout and stderr
<harlowja> that could avoid the tee
<smoser> the python logging is configured in 05_logging.cfg
<harlowja> larsks  i mean if u really want, we could likely drop the tee
<larsks> harlowja: Eh, never mind.
<harlowja> :-P
<larsks> I'd rather get a release :)
<harlowja> :-P
<harlowja> 200 years
<harlowja> lol
<harlowja> smoser does https://gist.github.com/harlowja/e2333dd3ed219b53ef44671185177c3e actually skip anything?
<harlowja> even though things get accumulated in forced and skipped there, those lists only seem to get used for logging?
<harlowja> (or maybe i'm confused, ha)
<smoser> harlowja, you would appear to be correct :)
<harlowja> :-P
<smoser> larsks, can you try something for me ?
<smoser> so to get a realease... i added you and harllwo to bug control team
<larsks> Okay.
<smoser> and you should be able to now target things you think valid to 0.7.8 milestone
<smoser> for example... https://bugs.launchpad.net/cloud-init/+bug/1603830 seems reasonable
<smoser> can you just try to confirm that (mark it confirmed, it sure seems legit) and then target it to milestone ?
<larsks> Sure.  Let me try...
<smoser> i can spedn some time going through bugs and targetting them to 0.7.8
<smoser> and woudl appreciate it if you and harlowja did the same.
<harlowja> yes sir boss
<smoser> rharper, is also in that group
<smoser> and powersj
<larsks> That seemed to work.
<rharper> smoser: ok
<smoser> alright then.. we have the start of a bug list
<smoser>  https://launchpad.net/cloud-init/+milestone/0.7.7
<rharper> 0.7.7 or 0.7.8 ?
<harlowja> cloud-init 0.7.7 "been-to-long"
<harlowja> lol
<smoser> rharper is in the same camp as me and has just seen 0.7.7 for so long that he thought that was released :)
<harlowja> lol
<rharper> heh
<smoser> the ubuntu versions are 0.7.7~bzrREVNO
<smoser> where '~' in apt speak means less than
<smoser> dpkg speak i guess
<harlowja> whats ubuntu
<smoser> its one of those free linux things
<smoser> kind of like centos
<harlowja> ah
<harlowja> except newer?
<harlowja> lol
<harlowja> https://code.launchpad.net/~harlowja/cloud-init/cloud-init-really-skip-baddies if u want it :-P
<harlowja> that will actually skip modules, lol
<smoser> is worked_distros close to sane ?
<harlowja> in the various modules?
<harlowja> i think the redhat/yum ones probably need updating
<smoser> does larsks have a list that he runs ?
<harlowja> so no, imho, not all the modules very well use that
<harlowja> some do, some don't, ha
<harlowja> ones like cc_rhn_subscription should
<smoser> if its in their default list, i'd mark it as working
<harlowja> or cc_yum
<smoser> yeah.
<harlowja> cc_lxd also should probably have debian...
<harlowja> so there probably should be an audit...
<smoser> agreed.
<larsks> smoser: A list of modules?  Honestly, I'm not sure that anyone has paid much attention to the cloud_*_modules sections of the config.  Hence my earlier comments about moving to some sort of single list or something...
<larsks> That is, there is a config we are using, and it includes some things intentionally, like cc_rhn_subscription, but other stuff just because it was already there.
<harlowja> larsks so just some background
<harlowja> each module can optionally have a attribute 'distros' or 'osfamily'
<harlowja> the cloudinit runtime (the thing that executes those modules) will look for these attributes and potentially skip if not on a supporting distro
<harlowja> by default if a module doesn't list that attribute, its assumed to 'just work'
<harlowja> some modules do list it (but without https://code.launchpad.net/~harlowja/cloud-init/cloud-init-really-skip-baddies things aren't actually skipped anyway)
<harlowja> so we can have a single list, but we have to list those attributes correctly
<harlowja> and then the runtime itself will handle skipping on unsupported distros
<harlowja> just we have to consistently mark 'distros = ['ubuntu', 'debian']' in modules
<harlowja> so that could == single list if we use it correctly
<harlowja> bbiab
<larsks> Fair enough.  Fwiw, I would love to see an is_available() method or something that could perform arbitrary runtime checks (e.g., cc_puppet could check if puppet were available, etc), rather than checking against distro names (like, what if someone were running an rpm based distribution with apt?)
<larsks> missed him by this much...
<larsks> harlowja: Repeating myself re:the distros var, because you quit at just the wrong moment: Fwiw, I would love to see an is_available() method or something that could perform arbitrary runtime checks (e.g., cc_puppet could check if puppet were available, etc), rather than checking against distro names (like, what if someone were running an rpm based distribution with apt?)
<harlowja> lol
<larsks> But obviously not right now :)
<harlowja> larsks why not right now?
<harlowja> its not that hard
<mgagne> looks like 'name' is optional in links http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/sources/helpers/openstack.py#L547 but later is mandatory: http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/net/network_state.py#L284
<larsks> Well, if we're trying for a 0.7.7 release, something that introduces an entirely new mechanism for like that seems like a bad choice in the short term.
<harlowja> larsks well we can at least build in the mechanism right now
<larsks> Any chance we can work on github, or do I need to learn bzr + lp to submit changes for review?
<harlowja> i'll make something quick
<harlowja> smoser ^^^^^
<harlowja> smoser is mr.moveittogit
<smoser> harlowja, yeah, lets do it.
<harlowja> smoser like by EOD
<harlowja> ?
<harlowja> lol
 * larsks yays
<harlowja> to git
<harlowja> ?
<mgagne> harlowja: tested bonding + vlans with latest cloud-init and it fails.
<smoser> mgagne, :-(
<harlowja> mgagne any log u got, any output of input and what it generated?
<mgagne> for the reasons I mentioned above
<harlowja> hehe, let me find the logger for this
<mgagne> let me setup network, currently on IPMI
<harlowja> mgagne can u try https://code.launchpad.net/~harlowja/cloud-init/cloud-init-render-tool
<harlowja> it should write out files given a config in
<mgagne> explain like I'm 5 ?
<harlowja> given network_config.json ---> poof >>> turd
<smoser> :)
<harlowja> pooppy
<harlowja> lol
<harlowja> or is that el3?
<harlowja> lol
<smoser> mgagne, did that regress ?
<smoser> harlowja, for git... we have to just rid 'bzr' from everywehre
<smoser> that is cloud-init's http://paste.ubuntu.com/20342955/
<smoser> and then my debian packaging in ubuntu makes use of bzr export and bzr merge upstream
<smoser> but that is not a blocker for cloud-inti
<smoser> for version sutff. i think we were saying pbr. have to look. i think you had something on that.
<harlowja> prob
<mgagne> smoser: never managed to test bond + vlan, only tested "flat" networks (without bond)
<mgagne> smoser: testing cloud-init is really a part time job here =)
<smoser> mgagne, right. thats fine. i was just wanting to make sure it wasant regression
<mgagne> I don't think so
<harlowja> larsks https://code.launchpad.net/~harlowja/cloud-init/cloud-init-dynamic-distro-check/+merge/300805 (the basics of that dynamic distro check)
<smoser> as i was pretty sure you'd verified it working for you.
<smoser> yeah.
<mgagne> there is nothing to regress at that point
<mgagne> ok, I'm having issue with networking, can't easily fetch logs without SSH. will take more time to debug as it could also be related to our auto networking config.
<powersj> smoser, I have an initial job running the python tests now every 12 hours. I'll start looking into the vmtests and how they are run for curtin to see what we can do for cloud-init.
<smoser> powersj, yeah, that will be a fair amount of work.
<smoser> we shoudl talk some / brainstorm on how to get something reasonable into place.
<powersj> ok
<smoser> we can do a whole lot of test (including non-ubuntu distros) on lxd
<mgagne> ok, that's the best I can do in the short term: http://imgur.com/a/wVuTr
<powersj> that's how I've been experimenting is setting up lxd env.
<powersj> not sure how to do different archs other than i386 yet though
<smoser> well, we have resources for power at least
<smoser> harlow dead
<smoser> harlowja
<smoser> (summoning worked before)
<smoser> this is new to me:
<smoser>  http://paste.ubuntu.com/20345473/
<smoser> guess which of those lines pep8 doesn't like
<smoser> complaining
<smoser>  E225 missing whitespace around operator
<smoser> must of come as fallout of https://github.com/PyCQA/pycodestyle/issues/166
<larsks> smoser: I am looking at the bzr->git stuff in the scripts right now.
<smoser> larsks, nice.
<smoser> larsks, i think the plan was to use pbr sensible_version and the like
<larsks> smoser:  does anyone actually use brpm?
<smoser> harlow does
<smoser> at least used to frequently.
<larsks> Arg, he's dropped off again.
 * larsks is going to buy harlowja an irc proxy for his birthday.
<smoser> and i'd like to have something like that.. thats what i'd use for some c-i
<smoser> i use bddeb in development for testing a package.
<larsks> Okee dokee.
<smoser> mgagne, if you can manage to get the openstack network_json of the failed system that'd be helpful
<mgagne> smoser: will try =)
<mgagne> smoser: finally made it: http://paste.openstack.org/show/539268/
<simar> is this channel logged? any publicly available logs?
<simar> also how does cloud init keep a track of which modules to run on boot? let's say I write a module that changes the hostname of the machine, would it actively check /etc/hosts on boot each time to know if it's required to run that module or not?
<mgagne> IIRC there are flags created in /var/lib/cloud/instances/<uuid>/sem for once_per_instance configuration
<mgagne> otherwise some always run at boot
<simar> gotcha. thanks mgagne that seems to be right
<simar> Also if I write a custom module that lives in /etc/cloud/cloud.cfg.d/ how would I call that up from cloud.cfg?
<simar> so if its xx_modulename.cfg, i'd have to call it as modulename from within cloud.cfg under cloud_final_modules?
#cloud-init 2016-07-22
<smoser> mgagne, thank you.
<smoser> prettier at http://paste.ubuntu.com/20390677/
<smoser> mgagne, it'd be nice if you could open a bug.
<smoser> have a test case showing failure based on your json.
<smoser> thank you.
<smoser> here is test case http://paste.ubuntu.com/20392738/
<smoser> rharper, ^ . if you're sitting there doing nothing.
<smoser> i suspect this is related to not using 'id' as name
<smoser> i go now.
<smoser> good night.
<boolman> smoser: thx for confirming
<boolman> (lock_passwd: False)
<powersj> smoser, is there a good time to do a hangout today? or would Monday work better?
<smoser> powersj, today is fine with me. what works for you?
<powersj> anytime in the next hour, or after 1pm mountain
<smoser> powersj, 5m ok ?
<powersj> sure
<smoser> powersj, https://hangouts.google.com/hangouts/_/canonical.com/hangout-smoser
<smoser> powersj, so
<smoser>  https://gist.github.com/smoser/f112518132f2849443f8edfcf292c8ec
<smoser> is some stuff i just had when i was doing some testing
<smoser> it builds and hacks in the cloud-init into the container
<smoser> powersj, http://download.cirros-cloud.net/daily/20150923/README-lxd.txt shows how youc anlaunch lxc with user-data
<smoser> lxc launch xenial x2 --config=user.user-data=$(cat user-data)"
<smoser> you can also use --config=user.data=- < foo
<mgagne> smoser: so I fixed the name attribute issue. Now it's getting a bit worst.
<mgagne> smoser: eth0 and eth1 are configured and enslaved to bond0 while eno1 and eno2 exist and should be used.
<smoser> hm... what did you change ?
<mgagne> http://paste.ubuntu.com/
<mgagne> damn
<mgagne> should post first
<mgagne> http://paste.ubuntu.com/20475020/
<mgagne> now I'm trying to up the bond and slaves aren't configured even if bond-master is configured properly. I see that slave interfaces have bond configuration too (bond-mode, bond-miimoo, etc.)
<mgagne> last time, I had to enslave manually even if I followed https://help.ubuntu.com/community/UbuntuBonding
<mgagne> will continue to debug and make sure it's not something on my side
<mgagne> also, auto bond0 is missing from generated config
<mgagne> that's why network failed to start
<mgagne> same for slaves interfaces
<mgagne> to summarize: cloud-init creates and tries to enslave eth0/eth1 while eno1 and eno2 exist (and are stil configured by cloud-init?), auto stanza is missing from bond0 and slave interfaces
<larsks> Arg, brpm does a recursive rm of ~/rpmbuild.  Bad build script! Bad!
<smoser> larsks, wow. that is bad.
<smoser> shame on harlowja_at_home
<smoser> mgagne, i suspect that you didnt update the json completely.
<mgagne> update the json?
<mgagne> I'm not sure what you are referring to
<smoser> so http://paste.ubuntu.com/20475020/
<smoser> i think you  meant you updated the openstack to build and include 'name'
<mgagne> no
<smoser> but could you post the complete json there ?
<mgagne> it is not part of the spec, not part of my implementation nor the upstream one
<mgagne> I posted it already yesterday
<mgagne> let me check
<mgagne> http://paste.openstack.org/show/539268/
<smoser> well what did you mean by http://paste.ubuntu.com/20475020/
<mgagne> looks like 'name' is optional in links http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/sources/helpers/openstack.py#L547 but later is mandatory: http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/net/network_state.py#L284
<mgagne> so I forced name to exist by using the id instead so I don't have to refactor network_state.py
<smoser> right. but i suspect your force didnt' do it right.
<smoser> so i wanted to see that 'forced' json with 'name'
<mgagne> don't change a thing, I will put the id anyway
<mgagne> and name isn't part of the spec so I won't add something that shouldn't exist in the first place
<mgagne> I think there is a wrong assumption here: http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/net/network_state.py#L319
<mgagne> or something needs to be translate before it comes here
<mgagne> bond_interfaces contains the link_id
<mgagne> and I think later it can't find the actual interface in the parsed interfaces
<mgagne> maybe because physical aren't renamed to eth0/eth1 properly earlier or something like that.
<mgagne> will try that render tool and see that happens
<mgagne> what*
<smoser> mgagne, is 'id' going to be sane ?
<smoser> should i i use it?
<mgagne> well, the question is: why bother with a name?
<smoser> versus veth234234 that id gest for link veth
<mgagne> can't it be bond + auto_increment?
<smoser> well, eys of course.
<mgagne> then default to that if name isn't provided.
<mgagne> but as stated, name will never be provided unless someone provide a change in the spec
<smoser> but if the user provides networking information and says "this is what i want the bond link to be called". then its better to have it called that then creating one myself.
<mgagne> so we are adding dead code
<mgagne> I think we already talked about that
<mgagne> ok so I managed to make that render tool work
<mgagne> same behavior
<smoser>  http://paste.ubuntu.com/20492837/
<mgagne> smoser: ok, I updated the JSON to the actual physical interface names (not the link ids) are in bond_links. it now works
<mgagne> so I think cloud-init doesn't translate the bond_links (link id) to physical name
<mgagne> smoser: with oeth1, you should see that 4 physical interfaces are configured.
<smoser> http://paste.ubuntu.com/20493003/
<mgagne> yea, that's exactly the problem
<mgagne> and auto stanza is missing
<mgagne> so network is never started properly
<smoser> well, i'm not sure that is it
<smoser> di dyou open a bug ?
<mgagne> not yet
<mgagne> will opennow
<mgagne> how can I link to a specific line of a specific review in LP/bazaar?
<mgagne> revision*
<smoser> mgagne, i dont know.
<mgagne> ok, will link to latest for now
<larsks> smoser: these are the changes I'm working on for moving to github: https://github.com/larsks/cloud-init/compare/feature/move-to-git
<larsks> I  haven't tackled the build scripts yet.
<smoser> movign to git
<smoser> not git hub
<smoser> right?
<larsks> Oh yeah?  See, I though we were going to use the existing cloud-init/cloud-init repository on github.
<larsks> No?
<larsks> Just git on launchpad?  That is also fine.  I will just need to fix hacking.rst really; nothing else cares about where things are hosted.
<mgagne> here we go: https://bugs.launchpad.net/cloud-init/+bug/1605749
<larsks> smoser: well, I've updated the docs and put my changes on launchpad @ https://git.launchpad.net/~larsks/cloud-init/?h=feature/move-to-git...
<smoser> larsks, ok. thank you.
<larsks> ...but of course I can't propose them for merging against a bzr repository, and it doesn't look like LP allows diffing between arbitrary revisions.
<smoser> larsks, thanks.
<smoser> i can figure out how to diff two trees
<larsks> I figured :)
 * smoser googles :)
<larsks> If you move the existing cloud-init repository into git, we could make this the first merge request...
<smoser> yeah
<smoser> larsks, still there ?
<smoser> so your master is straight import ?
<larsks> I think so.  Does it look like not?
 * larsks checks.  Seems to be.
<larsks> The last change in my master is "mcollective: add tests, cleanups and bug fix when no config in /etc."
<smoser> just was checking.
<smoser> how did you do export / import ?
<larsks> Using git-bzr-remote...
<larsks> Well, that's mostly just import.  bzr->git.  If I had to go the other way, I would probably just using 'bzr branch', and then generate and apply a patch or something.
<smoser> no you're good.
<smoser> harlowja_at_home, had done somethign and had to mess aroudn with some tags...
<smoser> trying to find that
<larsks> I've just updated the git.launchpad.net/~larsks/cloud-init with all the tags, in case those are useful.
<smoser> he had done this
<smoser> https://gist.github.com/harlowja/8bfe7e9a19214379684f
<smoser> ok. let me see.
<larsks> I'm not sure what's going on after line 30 with with the rebasing or with any of the tag deleting, but the basic bzr fast-export --plain | git fast-import looks fine.
<larsks> Personally, I just ran 'git clone bzr::lp:cloud-init', and then pushed it wherever...
<smoser> oh.
<smoser> wow. git just has that. i didn't know that.
<smoser> so if i do that, mine will differ on shas right ?
<larsks> That should be substantially the same as the fast-export | import.
<larsks> Actually, I think you should end up with the same shas.
<smoser> yeah, i do
<smoser> ok. i'm going to push this imported branch
<smoser> larsks, thank you for your patience
<larsks> fyi, Also 'git push --tags'
<smoser> yeah. and i will first delete the ubuntu- tags
<smoser> https://git.launchpad.net/~cloud-init-dev/cloud-init/
<jgrimm> \o/
<smoser> and updated https://code.launchpad.net/cloud-init
<smoser> larsks, so you can submit yours as an MP now
<larsks> Proposed.
<smoser> larsks, link ?
<larsks> Oh, sorry, I thought you'd get a notification of some sort.  https://code.launchpad.net/~larsks/cloud-init/+git/cloud-init/+merge/300953
<smoser> well, i might ave.
<powersj> now we see if CI continues to work...
<smoser> powersj, whoops :)
<powersj> I'm glad you did this to cloud-init first
<smoser> larsks, thanks. i'm just about EOD also... so
<larsks> Me too.
<larsks> Gotta go perform child transportation, in fact.
<smoser> harlowja_at_home, ^ larsks got done in like 3 days what you've been trying to do for like 2 years!
 * smoser notes that what was actually accomplished was getting smoser to stop being lazy
<smoser> thank you both.
<larsks> Well, keeping in mind also that the build scripts are currently broken :)
<larsks> Have to take off.  See y'all on Monday....
<powersj> A bunch of CI jobs may launch in 3-4 mins. harlowja_at_home apparently wasn't allowed to launch ci jobs
<powersj> I've added the cloud-init-bugcontrol and cloud-init-dev groups to the allowed users
<powersj> smoser, So the branch name changed from lp:cloud-init to cloud-init:master (see last entry) if you look here: https://code.launchpad.net/cloud-init/+activereviews
<powersj> The trigger job ain't happy with me just switching to the new name, so still investigating.
#cloud-init 2016-07-23
<harlowja_at_home> ha smoser
<harlowja_at_home> i got stuck in southwest mess last night
<harlowja_at_home> was stuck in phoneix arizona :(
<harlowja_at_home> southwest all messed up last few days
<harlowja_at_home> smoser, very nice, git finally
<harlowja_at_home> woot
#cloud-init 2017-07-17
<niluje> blackboxsw: smoser: I just pushed the split of unittests
<niluje> I hope this is okay
<smoser> niluje, yeah. great.
<smoser> in a MP its ok to add commits or to push --overwrite
<smoser> adding commits is actually easier for the reviewer, and you can comment 'addressing review feedback' or something.
<niluje> smoser: MP?
<niluje> merge proposal?
<smoser> yeah.
<niluje> k
<smoser> mp, pr, mr, pm
<niluje> do you want me to merge the commits into one then?
<smoser> all but the last oen make some sense :)
<smoser> i'll squash it when i pull
<niluje> okay
<niluje> if you have anything you want me to change, feel free to ask :)
<smoser> powersj, blackboxsw rharper comments on
<smoser>  https://public.etherpad-mozilla.org/p/qxREmZaciZ
<smoser> are welcomed.
<powersj> smoser: made one word change, otherwise looks good
<smoser> hm..
<smoser> what did you change? i only see my color
<smoser> blackboxsw, ^ i think you were working on a way to check that group... that we could use in c-i to inform people.
<smoser> powersj, ^
<powersj> smoser: "If you have already signed the contributor agreement, please
<powersj> simply say so here." removed the s on contributor
<smoser> cool
<smoser> thanks
<smoser> blackboxsw, or rharper https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327525 . review appreciated.
<blackboxsw> morning.
<blackboxsw> will check it out
<powersj> smoser: ho
<rharper> smoser: stand up!
<smoser> gah
<smoser> rharper, blackboxsw https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327532
<smoser> blackboxsw, ....
<smoser> wonder if it'd be helpful for your ipv6 stuff
<smoser> if you grabbed the unit tests from https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324274
<smoser> i was going to grab those into a separate merge propsoal, but realized it might make your life more difficult as you might make those tests fail
<smoser> blackboxsw, well... i went ahead and grabbed.
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327534
<blackboxsw> smoser: I think this ipv6 branch is going to be rather large as is. If you wanted to land that as a separate merge proposal I can just rebase against  master to get the updates and fix the unit tests. I agree, this change will break those tests.
<blackboxsw> yeah I'll review that. it's big enough as is.
<smoser> blackboxsw, since i've got you..
<smoser> https://code.launchpad.net/~jcastets/cloud-init/+git/cloud-init/+merge/325740
<smoser> you asked for split up tests (or i asked you to ask for that) :)
<smoser> i'm good with it as it is, but i'll add some comments on the privilged source port stuff.
<smoser> i'm still kind of weary of
<smoser>  +from requests.packages.urllib3.connection import HTTPConnection
<smoser> +from requests.packages.urllib3.poolmanager import PoolManager
<blackboxsw> you think that needs to be from urllib3.poolmanager import PoolManager instead?
<blackboxsw> or just don't like using PoolManager/HTTPConnection in general
<smoser> blackboxsw, it seems that we're using an internal bit of requests
<smoser> no ?
<blackboxsw> smoser: I hadn't used requests yet in any projects I've touched. Yes, it feels a bit like that use here is diving into the repackaging of urllib3 within requests, which makes we wonder why not just use urllib3 directly in this case. The intent of the requests package seems was to make simple gets and posts or URIs a bit easier, but this invocation is diving into a more complex example which might be better
<blackboxsw> pulled using urllib directly.
<blackboxsw> or some other package
<blackboxsw> or a higher level api within requests
 * blackboxsw is reading up on the requests package
<smoser> blackboxsw, i dont have a good other solutionm, and it appears that this does work.
<smoser> even on ubuntu, which does not have the 'vendorized' urllib3
<smoser> https://github.com/requests/requests/issues/4104
<smoser> and blackboxsw look at /usr/lib/python3/dist-packages/requests/packages/__init__.py on your system.
<smoser> pints at https://github.com/kennethreitz/requests/pull/2375
<blackboxsw> now that is interesting issue.  with the imports. Ok, so this looks like a known problem and different vendors handle things differently. So this might hit us with other non-ubuntu distros.
<smoser> blackboxsw, tests/unittests/test_runs/test_simple_run.py
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327344
<blackboxsw> tox -e
<blackboxsw> oops
<smoser> bah. no harlowja
<smoser> he'd have an idea
<smoser> blackboxsw, https://code.launchpad.net/~jcastets/cloud-init/+git/cloud-init/+merge/325740
<smoser> blackboxsw, so i figured it out... the source fo the 'lost mock'
<smoser> get_cmdline()
<smoser> sets a global PROC_CMDLINE based on the result, and once its called, it is then set forever.
<smoser> not sure what the right thing to do here is
<smoser> i guess really we should just need to mock that methods use
<smoser> but *always* mock it
<rharper> smoser: nice find
<blackboxsw> thx smoser done testing https://code.launchpad.net/~jcastets/cloud-init/+git/cloud-init/+merge/325740. minor typo tweak on your comments
<smoser> blackboxsw, thanks.
<smoser> i'll pull
<smoser> please think about that PROC_CMDLINE
<blackboxsw> hrm, within unit tests couldn't we set DEBUG_PROC_CMDLINE then we'd never have the side-effect
<blackboxsw> right smoser ?
<blackboxsw> def get_cmdline():
<blackboxsw>     if 'DEBUG_PROC_CMDLINE' in os.environ:
<blackboxsw>         return os.environ["DEBUG_PROC_CMDLINE"]
<smoser> yeah... but how even to do that always ?
 * blackboxsw was thinking about python-fixtures and the EnvironmentVariableFixture specifically in our unit test __init__
<smoser> blackboxsw, i'm gonna kill you8r sscaleway
<smoser> instnace
<blackboxsw> smoser: thank you
<smoser> blackboxsw, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327344 is ready i think now. i'd appreciate a review
<smoser> and /me goes away
<blackboxsw> I need to head to the DMV to renew registration. relocating to wait in a line
#cloud-init 2017-07-18
<niluje> smoser: blackboxsw: thanks a lot for your help for the MR
<niluje> and sorry it took so long to make it good - is it that long for every datasource?
<niluje> do you have a release schedule?
<blackboxsw> niluje I think that time was due in part to multiple vacations on our end impacting the branch review, discussion and landing.  (Yes, we were thinking we held you up a bit too long on the feedback loop). As per releases, we get something into ubuntu artful fairly frequently. And then monthly-ish try to get backported critical bug fixes through an SRU process into xenial, yakkety and zesty
<niluje> blackboxsw: honestly, I think it's 100% my fault if it has been so long :)
<niluje> one of my coworker asked me to review his PR in 2015, I had to rewrite it, some unittests were failing, I had something else to do and looked back to fix the issue less than 2 months ago :p
<niluje> the first commit of the scaleway datasource was nearly 2 years ago
<blackboxsw> niluje: well we â¤ the contributions, doesn't matter how long they take, and Scaleway datasource is a bit more complex than most given the secure port requirement.
<niluje> yeah
<blackboxsw> :)
<niluje> oh and anyway
<niluje> do you still want servers for cloud-init?
<blackboxsw> niluje: it would be nice if there were a comp'd account with limited resources, like an account that would allow us to spin up one instance for integration and QA/automated testing.  We would make sure we only spin up the instance in limited capacity, such as when we are preparing to release SRU updates to make sure it doesn't break Scaleway.
<blackboxsw> but I'm not sure if that's hard to do.
<blackboxsw> or if we ping you once we are ready for an integration test
<niluje> blackboxsw: sooo I asked how our sponsorship works (I don't deal with them here)
<niluje> what we ca do is give a discount of 10e/m (which is enough to have 3 small VPN) to your organization
<niluje> but it requires to have an account with a valid credit card, which won't be charged unless you reach more than 10e
<niluje> s/VPN/VPS
<smoser> niluje, we can probably work with that. just to clarify, we're not in search of "generic compute".
<niluje> yeah I got it :)
<smoser> we were asking for resources for integration test explicitly to ensure that scaleway would work and put that into some (future) continuous integration.
<niluje> yep
<smoser> if you dont mind, i will feel free to bother you later.
<niluje> no problem, I'll be here and the offer will still stand
<niluje> thanks a lot for everything :)
<smoser> niluje, wrt releases... we are well over due
<smoser> blackboxsw, if you are around, a quick read of
<smoser>  https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327534 would be nice.
<smoser>  https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327344 too. and then i'd like to upload that to artful to un-block python3.6 in ubuntu
<blackboxsw> smoser: I'm going through that review now but it will probaby be in ~25  mins. Tuesday mornings I need to drop a kiddo at school at the same time as standup
<smoser> blackboxsw, it appears apparmour will allow dhclient to run
<smoser>  /etc/dhcp/dhclient-script
<smoser> and that does not exist
<blackboxsw> yeah, I was concerned about whether it did exist on some clouds
<smoser> so, while it has the same issues as /sbin/dhclient-script ...
<smoser> well, in ubuntu we can search if anything in the archive has that file
<blackboxsw> but maybe we could take that approach if not present.
<smoser> and i doubt it does
<smoser> $ apt-file search dhclient-script 2>&1 | grep -v '^W:'
<smoser> apparmor-profiles: /usr/share/doc/apparmor-profiles/extras/sbin.dhclient-script
<smoser> dracut-network: /usr/lib/dracut/modules.d/40network/dhclient-script.sh
<smoser> isc-dhcp-client: /sbin/dhclient-script
<smoser> isc-dhcp-client: /usr/share/man/man8/dhclient-script.8.gz
<smoser> rear: /usr/share/rear/skel/default/bin/dhclient-script
<smoser> we should pose question to security team though
<smoser> ask what they'd recommend.
<blackboxsw> smoser: I'll hit up the security channel.
<blackboxsw> oops you already did
<smoser> blackboxsw, updated https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327344 with comments . more palatable now i think
<smoser> and i actually verified that all tests/unittests/ use TestCase from helpers
<smoser> err... *now* i updated
 * blackboxsw is trying to rework register_helper a little bit on your to make it a bit more DRY 
<smoser> ):
<smoser> :)
<blackboxsw> on your other branch.
<smoser> i started that too
<smoser> oh.
<smoser> i thoguht you were going to say you were reworking reset_global-state()
<blackboxsw> heh
<smoser> to use setattr
<blackboxsw> nah
<blackboxsw> sorry for the slow goings on this. I just want to make sure I understand and tested what I've reviewed.
<blackboxsw> speed will come...
<smoser> blackboxsw, well, ifyou're looking for generalization..
<smoser>  tests/unittests/test_datasource/test_aliyun.py
<smoser> uses something similar
<smoser> thats where i started with register_helper
<smoser> but the py3.6 one
<smoser> if you can quick read that one.
<smoser> that is more important
<smoser> as it is blocking things in ubuntu
<blackboxsw> yeah definitely. committed WIP comments on your ec2, grabbing py3.6 and playing on my aartful box
<smoser> there is actually nothing python3.6 specific in those changes
<smoser> they're all just bad expectations in tests, and one actual fix
<blackboxsw> smoser: when I tweak tox.ini environment to basepython = python3.6.  I still hit an error: http://pastebin.ubuntu.com/25119991/ digging in now
<smoser> blackboxsw, i think you have a bad .pyc or something
<smoser> something is wrong
<blackboxsw> yeah I'm running tox -r to rebuild now
<smoser> as line 23 in your paste is not line 212 in my merge proposal
<smoser> ie, open cloudinit/net/netplan.py and go to line 212
<smoser>         fpnplan = os.path.join(util.target_path(target), self.netplan_path)
<smoser> is what you shoudl see
<blackboxsw> ok that plus a clean_pyc worked for me
<powersj> smoser: submitted merge for kvm backend. I have it working locally and on torkoal. Would like to see what you think. https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/327646
<powersj> smoser: I also have the workflow documented and can walk you through it if you want.
<blackboxsw> ok I'm out of your way on https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327344 . approved
<blackboxsw> back to ec2 tests branch
<smoser> thanks
<rharper> smoser: https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/327648  (sysconfig update MP)
<smoser> powersj, you want to chat in 20 minutes ?
<powersj> smoser: sure
<smoser> rharper, from
<smoser> 8da074f831ce8e4ad781163b0301d6e0b07e0d12
<smoser> er... from your link above, that first one looks fine. i'll just grab that.
<smoser> bah. i'll look more tomrorow.
<smoser> i'm reallyconfused.
<smoser> i was going to cherry pick the above, but it leaves tox broken
<smoser> well, fixed that
<smoser> 1 commit down, 11 to go
<smoser> commented in your mp
<rharper> smoser: thanks, I'll run tox on my mp (can't believe I didn't already)
<smoser> well just when i cherry picked that one
<smoser> it probably means you now have merge conflicts in another commit
<smoser> ie, i bet you didn't fix that one and fixed it later.
<smoser> i have to run.. i think the way i'll do thi sis just to work one at a time
<smoser> sound reasonable ?
<blackboxsw> smoser, approved https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327534   take what you will as far as review comments.
<smoser> rharper, http://paste.ubuntu.com/25121184/
<smoser> thats my next commit review
<smoser> and i need to run now, really this time
<blackboxsw> it feels like those httppretty tests are expensive. ~4 seconds per test, though I haven't profiled where we are spending our time in those tests yet.
<blackboxsw> see you smoser have a good one
<smoser> hm..
<smoser> :-(
<blackboxsw> smoser: nevermind, they were expensive because I had manipulated the mocked metadata urls locally and leaked calls were timing out.
<blackboxsw> real	0m0.480s ok better
<rharper> smoser: ok, looks like the IPV6_ gateway prefix gets fixed in the next commit (whic his fine)
<powersj> rharper: thx for the review
<powersj> rharper: smoser: what is the proper way to format this command via mount-image-callback: http://paste.ubuntu.com/25121627/
#cloud-init 2017-07-19
<tmartins> Hey guys, listen, when I enable "virtio-scsi, in my OpenStack environment, it speeds up a lot my storage! However, it breaks cloud-init! Both SWAP and Ephemeral are failing. Maybe that's because the devices changed from /dev/vdX to /dev/sdX ?
<tmartins> How to make cloud-init work with virtio-scsi?
<tmartins> I mean, by default, without tricks via "user_data"...
<smoser> tmartins, need more information on what is failing
<smoser> file a bug? collect the info requested there.
<tmartins> Right, I'll fill a bug report with the logs!   =)
<smoser> tmartins, please just give me apointer here.
<smoser> (also)
<tmartins> Sure!
<powersj> rharper: \o/ xkvm is awesome
<rharper> powersj: thank smoser  too
<powersj> smoser: thanks for suggesting this as well
<powersj> rharper: thoughts on how to include it with cloud-init?
<smoser> i think copy to tools/
<rharper> put it in tools
<powersj> k
<smoser> powersj, http://paste.ubuntu.com/25126584/
<smoser> that is what i recall having done for mount image callback.
<smoser> that partuclarly lets me run mount-image-callback lxd: with sudo without a password
<powersj> smoser: what is the "lxd:" part do? My ignorance of mount-image-callback is showing...
<smoser> lxd: means lxd do the same sorts of things to a container that you'd do to an image
<smoser> so i can
<smoser> $ lxc init ubuntu-daily:xenial xsmtest
<smoser> $ sudo mount-image-callback xsmtest mchroot
<smoser> er...
<smoser> $ sudo mount-image-callback lxd:xsmtest mchroot
<powersj> ok so for my use-case I need to specify a similar command string for the Cmnd_Alias in sudoers
<smoser> maybe something like
<smoser>  http://paste.ubuntu.com/25126653/
<smoser> and remember that files in /etc/sudoers.d have to be 440 (or more restricive) or they're ignored.
<powersj> ok
<powersj> thanks
<smoser> basically that is going to protect you from accidental mis-use
<powersj> yeah
<smoser> as if you wanted to exploit it, you could just put any file in /citest-images/
<smoser> and 'mchroot' is just the equivalent (available in newer 'mount-image-callback') of
<smoser>  chroot _MOUNTPOINT_
<rharper> smoser: I just push a rebase of curtin-centos on master
<smoser> ok
<smoser> packages are pita
<smoser> rharper, just noticed... i tried fedora25 lxd instance.
<smoser> our copr repo builds python2 there
<smoser> and there is no python2 in their install.. ie, we should be python3 there.
<rharper> yeah, we've not been testing the fedora path =(
<rharper> I suspect we'll have more "fun" with systemd-units under fedora as well
<powersj> smoser: thanks for the sudoers file. no more typing password all day long
<powersj> smoser: did you have any suggestions on http://paste.ubuntu.com/25126921/
<smoser> wow
<smoser> so .. the rpm version available in fedora25 for cloud-init from the daily is
<smoser>  0.7.9+211.gd1e8eb7-1.fc25
<smoser> that .fc25 i think is what handles something i'd wonderd about.
<smoser> if you have build-time logic then you'd get 2 different binary rpms for centos6 and centos7
<smoser> (and i think we do)
<smoser> whats interesting there is that means (i *think*) that a upgrade from centos6 to centos7 will get 100% new packages.
<smoser> oh.. or only stuff that got re-built. maybe they can binary copy also
<smoser> i geuss that ends up being the same as ubuntu
<smoser> yeah
<smoser> rharper, around ?
<blackboxsw> rharper: smoser powersj: so createing a tmpdir and running a copy of dhclient from there on both ubuntu and centos results in success. No apparmor/SELinux conifinement constraints hit us on either ubuntu or centos 6. Checking cent7 on aws now
<blackboxsw> cent6 images in aws have selinux enforcing (as opposed to our lxc environments when don't)
<blackboxsw> *which don't
<smoser> you'd hope they do :)
<smoser> lxc is not enforcing just because it doesn't play well with the container
<blackboxsw> ok rebasing this branch against master and fixing up unit tests.
<blackboxsw> smoser: actually I'll wait to rebase until you land this one https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327534
<blackboxsw> since those unit test changes will have merge conflicts w/ my branch
<blackboxsw> this month will be interesting to see what my aws bill is
<smoser> blackboxsw, you're welcome to grab that to trunk yourself
<smoser> if it makes it easier for you
<smoser> assertFalse might be valid in python2.6.
<blackboxsw> it's easy enough as you are adding unit tests. yeah just wanted to closeout on my minor review comments there. please do land it with whatever review comments you accept. it'll make my branch easier(smaller) when I put mine up.
<blackboxsw> smoser ^
<rharper> smoser: back
<smoser> rharper, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327739
<smoser> rharper, ^ you can see your commit (that i cherry-picked) and then some cleanups of mine.
<rharper> yes
<rharper> one question, we can already rely on 'prefix' even if we get the odd 'netmask: ffffffffffffffffffff' ?
<rharper> smoser: ^
<rharper> that's my one question in dropping the 'netmask' in subnet part and only using prefix
<smoser> have to test. where would we get that ?
<smoser> commit d00da2d5b0d45
<smoser> says
<rharper> I think I saw that in the json network from rackspace
<smoser>       * prefix will always include an integer value that is the
<smoser>         network_prefix for the address.
<smoser>       * netmask will be present only if the address is ipv4, and its
<smoser>         value will always correlate to the 'prefix'.
<rharper> in curtin, network_alias, network_mtu, and vlan_network_ipv6 all use v6 with netmask set to ff*
<rharper> so you can net-convert those and make sure it doesn't blow up
 * rharper tries out the change locally
<smoser> i think it should
<smoser> because we do
<smoser> in _normalize_net_keys
<smoser> we use  mask_to_net_prefix
<smoser> although 'ffffffffffffff' that is just wrong
<smoser> no : ?
<rharper> it's /64
<rharper> and it's a thing
<smoser> without a : ?
<rharper> oh, no
<rharper> sorry
<rharper> i was just being lazy
<smoser> then it should be fine.
<smoser> rharper,
<smoser> $ python3 -c 'import sys; from cloudinit.net.network_state import mask_to_net_prefix; print(mask_to_net_prefix(sys.argv[1]))' "ffff:ffff:ffff:ffff::"
<smoser> 64
<rharper> smoser: looks good
<smoser> rharper, IPV6ADDR=2001:DB8::10/64
<smoser> that is ok ?
<smoser> it seems that it must work for SECONDARIES
<rharper> AFAIK, yes
<rharper> it's passed to iproute2 (/sbin/ip)
<rharper> which takes prefix in v6 format
<smoser> k
<smoser> rharper, so. .. on your templatifying of the systmed unit files...
<smoser> none of those *should* be necessary
<rharper> go on
<rharper> those changes were taken from the upstream cloud-init in el today
<smoser> After and Before don't do anything
<smoser> oh.
<smoser> right
<smoser> larsks, just wanted to take them out
<rharper> there is something else besides ordering
<rharper> I think 'network' versus 'networking'
<rharper> is at lease one change
<rharper> there is no 'networking' service in el
<rharper> only 'network'
<smoser> but I can put 'After the-third-sunday-of-the-new-millenium.service'
<smoser> and it doesnt do anything unless that service will be run
<smoser> same is true of networking.service and network.service
<smoser> the one that is present will get run Before and others ignored
<rharper> you're suggestion thens is to include all of the names it might be
<smoser> thats what we have right now
<rharper> well, no
<rharper> because trunk doesn't work under el7
<rharper> it might be the default deps then
<rharper> but I know for sure, the unit file in trunk doesn't run networking early enough for vmtest to pass
<rharper> so I pulled in the changes from downstream el cloud-init
<rharper> I'm fine with tweaking it
<rharper> but I do know for sure it won't work as it is
<smoser> and then you are
<smoser> dropping
<smoser>  DefaultDependencies=no
<rharper> yes
<rharper> for el
<larsks> I am too distracted to follow right now, but note that the defaultdeps stuff was causing us all sorts of issues. Yeah.
<smoser> on centos
<smoser> did you ahve a reason for that ?
<smoser> we almost certainly need that in order to accomplish some of the things that -local does
<larsks> smoser: we kept hitting dependency loops without dropping that.
<rharper> I dropped it due to el having the change, they're network service and other things like dbus  are ordered differently
<rharper> larsks: ack, I saw that as well
<rharper> we'd have to drive in dbus early changes which aren't present
<larsks> and not just on core stuff: cloud-init got into a fight with os-collect-config, also, which only crops up on openstack deployments.
<rharper> that said, I;ve not tested el7 on GCE which does DNS resolution early
<rharper> but they likely don't have both a libnss and systemd-resolved combo like we did on Xenial
<smoser> we're ultimately going to need it.
<smoser> other wise you run really late
<rharper> well, we're not running late now
<smoser> you're running after all local mounts are done
<smoser> which is too late to accomplish some things like re-formatting an ephemeral disk
<rharper> we know that DefaultDeps in el are after local mounts?
<rharper> or is that ubuntu systemd
<smoser> yes
<smoser>  Service units, unless they set DefaultDependencies=no, automatically get a dependency on sysinit.target
<smoser> i'm fine to deal with it later.
<rharper> bleh
 * rharper feels unit rage 
<smoser> rharper, in that commit you also mixed in redhat.spec file changes
<smoser> which i'm kind of confused on
<rharper> it was needed due to how unit files were copied
<smoser> a.) you have a comment that indicates .service files are installed by setup but .target are not
<smoser> but i dont see how that is possible
<smoser> b.) i dont understand how it could have worked before.
<rharper> it copied systemd/*
<rharper> we have template files in there now
<rharper> so only copy in the *rendered* unit files
<smoser> setup.py installs the .target and the rendered .service files
<rharper> we have .target and .service files;  the spec file copied *everything*, and setup.py also copied in some
<smoser> for install
<rharper> it does now
<smoser> "now" ?
<rharper> I added render_systemd_unit
<rharper> to setup.py
<smoser> right.
<smoser> but it does that for both .service and .target
<smoser> but your comment in the spec file acts as if it does that only for .service
<rharper> we only pass in the files that are templated
<rharper> all .service files are templated
<rharper> but the .target files still need to be installed
<rharper> so we manually copy this in
<rharper> those in
<smoser> they get installed the same way that .service files do
<smoser> in your branch
<smoser> the setup.py install
<smoser> will install them
<rharper> how?
<rharper> where in the setup.py does it specify .target files ?
<rharper> oh, maybe  it does work
<rharper> we pass back non template files as-is;
<smoser> right
<rharper> I suspect one changes was made after the spec change
<smoser> why '$' ?
<smoser> err.. \$
<rharper> pre jinja
<rharper> there's a commit later that should ahve been squashed
<rharper> which drops the \$
<rharper> I wrote these before we switched to jinja in the specfile
<smoser> rharper, so i think the only change we need to spec file then
<smoser> is to drop the 'cp -p systemd/*' line
<rharper> yes
<rharper> that looks right
<rharper> and I'll drop the \$ commit in the series since you will skip that here
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327739 is merged
<smoser> i have to go afk.
<smoser> rharper, so on top of yours
<smoser> i had http://paste.ubuntu.com/25127837/
<smoser> can you see if that works ? that'd be my next MP.  your 'Templatize systemd service files'
<smoser> plus http://paste.ubuntu.com/25127837/
<smoser> checking that now with centos7. centos6 didn't have any of the systemd files and i worried
<rharper> k
<smoser> but i guess that is by design
<rharper> right, no systemd in el6
<rharper> upstart files only
<smoser> rharper, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327743
<rharper> is that supposed to be both the sysconfig prefix change *and* the template change ?
<rharper> the diff on the MP looks like it has two commits
<smoser> it does
<smoser> yours, then my cleanup
<smoser> my cleanup dropped all the $ changes from the .spec.in
<rharper> mine shows 3 commits
<smoser> and removed the mkdir and copying of .target
<smoser> reload
<rharper> k
<rharper> huh
<smoser> i'd not pushed trunk
<rharper> what's up with the delay ?
<rharper> ah
<rharper> o
<rharper> hrm , two commits, diff still wrong
<smoser> launchpad doesnt render the moves well
<smoser> renames
<smoser> :-(
<rharper> bah
<rharper> I'll look at the commit
<smoser> yeah, and please write a commit message too
<smoser> if you ACK that then i'll pull when i get back
<rharper> ok
<smoser> and we'll be in like 3/12 :)
<rharper> 3/11 (we get to drop one)
<tmartins> smoser, here is the bug report! https://bugs.launchpad.net/cloud-init/+bug/1705346
<ubot5> Ubuntu bug 1705346 in cloud-init "Cloud-Init fails to deal with SWAP and Ephemeral if virtio-scsi is enabled" [Undecided,New]
<smoser> rharper, next one ..
<smoser>  Fix sysconfig rendering of virtual interfaces with network configurations
<smoser> needs unit tests
 * rharper accepts this 
<rharper> I believe they're added near the end
<rharper> 095f7585b57d35e9de68ee99c77a500ad644c293
<rharper> smoser: what would you like?  I can see about adding a separate singular test ; try to avoid conflicting with the one coming at the end
<smoser> hm..
<smoser> well, as a standaone commit we shoudl be able to get some unit test that shows the changes.
<rharper> k
<rharper> I'll do that
<smoser> rharper, in 97a25a75143a
<smoser> use is is_ipv6_addr
<smoser> and a test would be good.
<smoser> i can work on getting some of those together too
<rharper> ok, I'm doing the virtual interface one now
<rharper> if you want to tackle 97a
<rharper> so far, no regression, I've rebuilt the curtin-centos cloud-init package and re-ran the curtin branch (with your passthrough reviews applied) and all tests still passing
<rharper> will repeat tonight with a rebase from trunk
<rharper> smoser: do you want a paste with the unittest for 095 ?
<rharper> http://paste.ubuntu.com/25128675/
<rharper> that should do; and won't collide with the test in the last commit
<rharper> I'm going to step out for a bit, will check back later
<tmartins> smoser, I've updated the bug report with the config drive!   ;-)
#cloud-init 2017-07-20
<smoser> hm... https://jenkins.ubuntu.com/server/job/cloud-init-integration-a/91/
<smoser> rharper,
<smoser> http://paste.ubuntu.com/25133128/
<rharper> yes
<smoser> i'm going to put together a "simple" bond, vlan, and bridge config
<smoser> and add tests for sysconfig rendering of them.
<smoser> that bond shows stacktrace in current trunk and seems to render sanely with your
<smoser>  "Fix sysconfig rendering of virtual interfaces"...
<rharper> yes
<rharper> the last commit in my branch fixes an issue when you have a mac_address property on the bond, vlan or bridge
<rharper> but that's easy enough to fix up when we merge that commit
<rharper> I sent you a unittest
<rharper> that covered bond and vlan
<rharper> but not exhaustively
<smoser> ah. i see  yeah, you render the 'all'
<rharper> we could add a bridge device into that config for the oen I posted, and it would cover all 3 virtual types
<rharper> that's in the final commit
<rharper> but something simpler for the current patch, <rharper> http://paste.ubuntu.com/25128675/
<rharper> is there
<smoser> which is just a combination.. i think having individual ones would be good
<rharper> sure
<smoser> oh.
<smoser> ididt see .. you did that pate yesterday?
<smoser> thats great.
<smoser> your paste there doesnt have a 'route' on the subnets... so it doesnt excercise that code that you fixed
<smoser> i'm sorry i missed your patch
<rharper> there are two paths
<rharper> we can add one with subnets (single subnet) and one with subnets with routes
<rharper> cls._render_subnets(iface_cfg, iface_subnets) is for the subnets, and cls._render_subnet_routes(iface_cfg, route_cfg, iface_subnets) for any 'routes' keys under a subnet
<smoser> hm..
<rharper> we can just add a 'routes' list if you want to hit that path as well
<smoser> so 'routes' can be a sibling of subnets ?
<rharper> so, we have static routes (top level type)
<rharper> and then we can associate a list of routes for each subnet
<rharper> which is what 'routes' is, which models network_config.json  from openstack
<smoser> http://paste.ubuntu.com/25133177/ ?
<smoser> (that doesnt seem to work)
<rharper> sorry, type: routes top level
<rharper> or key under a subnet
<rharper> subnets: [{type: static, routes: [list of routes]}]
<rharper> config: [{type: route, dest: X, gateway: y}]
<rharper> you want the former
<rharper> drop lines 29 to end
<smoser> larsks, i have a question you probalby know the answer too
<smoser> our rpms build and get a .fc25 or .el6
<smoser> on fedora do they re-build every time ?
<smoser> every release... ie, if i upgrade do i get 100% new packages ?
<larsks> smoser: packages are rebuilt for every release, yes.
<larsks> But if there's no change in version/release, then for an *upgrade*  I guess you might not get a new package.
<larsks> Just a sec, let me check.
<larsks> #fedora confirms that during an *upgrade* packages won't be replaced if there isn't an actual change in the package version.
<larsks> But honestly, #fedora seems uncertain.  Shouldn't matter in either case, though: if the package version and release are the same there should not be a difference between the two, regardless of the .dist tag.
<smoser> larsks, hm.
<smoser> but how does it know that they would differ or not ?
<larsks> I mean, that's what the package version and release are for.
<smoser> as they could differ in build-time logic
<smoser> ie, our cloud-init truunk will build an rpm on copr for fc25 and el6
<smoser> and they'll differ
<larsks> If you think your build environment is generating different code, you increase the package release.
<smoser> but you do that by design... you look at rpm build macros and such
<larsks> You'll often see exactly this in the changelog for packages: "Rebuilt for https://fedoraproject.org/wiki/Fedora_24_Mass_Rebuild"
<larsks> For example.
<larsks> In that case, you increment the release on the rpm even though you're not making any changes to the rpm content itself.
<smoser> ok.. so heres an explicit example.
<smoser> we build the trunk rpm in copr.
<smoser> it spits out a .el6 rpm and a .el7 rpm
<smoser> the .el6 one has upstart files and adds upstart jobs on installation
<smoser> the .el7 one has systemd and sets up systemd on installation
<smoser> but they have the same version
<smoser> i'm wondering how we would handle a user upgrading from 6 to 7
<smoser> it seems they'd *have* to get the .el7 version
<smoser> but if they did that'd mean that an upgrade of 6 to 7 is 100% of packages even if the package *didn't* change.
<larsks> I will say that for EL, major version upgrades aren't something in which I have a lot of confidence.
 * smoser tweets ;)
<larsks> Well, for example, there is no "live" upgrade from 6->7.
<smoser> larsks, thats fine. its quite likely i'm just thinking too much.
<larsks> Whereas fedora has e.g. the "fedup" tool for that.
<larsks> For the most part, we treat el6 and el7 as completely separate entities.
<smoser> i just assumed one could upgrade 6 to 7 or 5 to 6 to 7
<smoser> that is helpful, thank you.
<larsks> I'll see if I can set up a test just for kicks sometime this week.  Not next week, though: I'll be on vacation :)
<larsks> smoser: hey, take a look at https://wiki.centos.org/TipsAndTricks/CentOSUpgradeTool
<larsks> "Warning: use of this tool is currently BROKEN as several system-critical packages are of a higher version number in CentOS 6.7 than they are in CentOS 7 so those do not get upgraded correctly."
<larsks> :)
<smoser> rharper, so on your branch
<smoser>  TestSysConfigRendering
<rharper> y
<smoser> er... how is TestSysconfigRoundTrip different from TestSysConfigRendering
<rharper> it differs in input and how it's called; I think the latter used an eni file as input to convert and then render a sysconfig
<smoser> ok. we lost the meaning of RoundTrip along the way
<rharper> you're right
<smoser> the 'round trip' of eni's test was that it started with an ENI, read that, generated a network config, and then rendered that
<rharper> it's just doing it one=way
<smoser> yeah
<rharper> I suspect we can just remove the RoundTrip part for now
<smoser> yeah. and just merge it with the previous class.
<rharper> I don't think we necessarity need to attempt to parse sysconfig dir back into network config
<smoser> i'll adjust stuff.
<smoser> right
<rharper> ok
<smoser> we dont have a sysconfig parser at the moment
<smoser> we *did* have a eni parser
<smoser> and that was why round trip made sense
<rharper> y
<smoser> rharper, take alook at https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327821
<rharper> smoser: approved
<smoser> rharper, i'm going to let https://jenkins.ubuntu.com/server/job/cloud-init-ci/62/console finish
<smoser> before i push
<rharper> ok
<smoser> but then i have it ready to go
<rharper> ok
<blackboxsw> ok unit testing branch is up https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/327827  (part 1 of aws dhclient support in init-local stage)
<blackboxsw> now rebasing the 2nd branch on this one
<smoser> rharper, awesome. one of the tests i added exposed the bug you fixed in the next commit
<rharper> \o/
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327828
<powersj> smoser: here is the latest version of kvm backend: https://git.launchpad.net/~powersj/cloud-init/commit/?id=28e84d93c193d7db41f1cba3bf5ff7a0ea364fc7
<powersj> still stuck on mount-image-callback trying to run commands via '/bin/sh -c'
<powersj> http://paste.ubuntu.com/25134551/
<smoser> rharper, onboot seems "global" to a device
<smoser> right ?
<rharper> per interface file
<rharper> it's equivalent to auto $iface
<smoser> right.
<smoser> i think there is a bug there.
<smoser> rharper, http://paste.ubuntu.com/25134578/
<rharper> I think it's a deficiency
<rharper> eni has control per stanza
<rharper> sysconfig does not
<rharper> hrm, but the bug that is present is that you should see IPADDR0, 1, 2 3
<rharper> for each subnet
<rharper> lemme see if later commits fix that
<smoser> right. thats the bug i'm pointing at
<smoser> we only have one addr
<smoser> and rendering eni for that breaks
<smoser> is 'type': 'manual' right ?
<rharper> somethings not right; I had that fixed
<rharper> I think it's type manual
<rharper> it should be type: static, control: manual
<rharper> there we go
<rharper> that works
<smoser> rharper, http://paste.ubuntu.com/25134607/ ?
<rharper> yes
<smoser> that renders
<smoser>  ONBOOT=yes
<rharper> right, you have one interface that's control: auto
<rharper> this is just a deficiency
<rharper> there's no way to support mixed control
<smoser> nah.
<smoser> drop the second address
<smoser> same thing
<rharper> no
<rharper> you get IPADDR1
<rharper> and NETMASK1
<smoser> http://paste.ubuntu.com/25134616/
<smoser> renders
<smoser>  http://paste.ubuntu.com/25134617/
<rharper> that's a bug
<rharper> but different than the first
<smoser> thats what i was saying :)
<smoser> that is what i was saying all along
<rharper> well, you changed
<smoser> well, if you put 'type: manual'
<smoser> then it paid attention
<smoser> so thats why i tried that
<rharper> not type: manual, control: manual
<rharper> you have a subnet which is type: static, but you want control: manual (define but dont' up it)
<rharper> that works fine;
<rharper> for single subnet, with type: static, control:manual, ONBOOT should be set to what subnet.get('control') == 'manual' is
<smoser> you can try to come up with something that actually does what you want
<rharper> which is't not, which is the bug you've found
<smoser> but i dont thikn you'll find it.
<rharper> I'm not following
<smoser> take one of my yaml, and make it into somethign that you think should render ONBOOT=no
<smoser> and show me.
<rharper> http://paste.ubuntu.com/25134632/
<rharper> that works (no bugs)
<rharper> but the one you posted (single subnet control: manual: should set ONBOOT=no) and it doesn't (which is new bug)
<smoser> you get ONBOOT=yes in your example
<rharper> well, you could argue the first one I posted is bug too; I say it's a known definciency
<rharper> because one of the two subnets is not control: manual
<rharper> so youre saying please UP this subnet
<rharper> but then we say, don't up the next one
<rharper> and ONBOOT is a toggle
<smoser> and if you drop the address, so you only have 1 thing... then you *still* get ONBOOT=yes
<rharper> ie, sysconfig does not support ONBOOT per subnet (like eni does via iface
<rharper> right
<rharper> if you specify control: manual, it's missing the check on what control value is
<rharper> for single subnet, we can get that right
<rharper> which is the bug I think you originally found
<rharper> well, we could revert to aliased interface files if we really wanted to support that
<rharper> eth0 (ONBOOT=yes) eth0:1 (ONBOOT=no)
<rharper> I suspect, they could extend ONBOOT to ONBOOT{$index}
<rharper> like they've done else where
 * rharper looks a scripts to see if that's so 
<rharper> doesn't look like it on el7
<smoser> rharper, you needed this
<smoser>   http://paste.ubuntu.com/25134666/
<smoser> your fix didn't change anything. because it was in fact checking subnet['type']
<rharper> well, it did something; so I need to find if we've a broken network config
<rharper> huh, we have type: manual and control: manual
<rharper> now I need to look at docs
<smoser> it shoudl be control: manual
<smoser> as it can be type: dhcp
<smoser> control: manual
<rharper> yes, it worked in eni because the it had both type: manual and control: manual
<rharper> the subnet code ignored the type check, but had the control check like you have
<rharper> I'll fix up the test-case in curtin to use type: static
<rharper> but you're fix is still the right one for sysconfig
<smoser> rharper, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327828
<smoser> it doesnt seem to want to update, but
<smoser>  http://paste.ubuntu.com/25134735/
<smoser> is the diff against my just-pushed upstream/master
<rharper> ok, looking
<rharper> then I'll rebase my branch again
<rharper> urg
<rharper> the last commit collides with your extra unittests in test_net
<rharper> I was hoping to avoid that =(
<rharper> no, something else; me fixes rebase
<rharper> looks like indentation sytle changes
<smoser> sysconfig: enable mtu set per subnet right ?
<rharper> no, the last MAC vs HW_ADDR
<rharper> almost done; fixed up the last commit's unittests (the ipv6 /64 change, and the new unittest you had for bonds needs MACADDR instead of HWADDR (which is what the last patch fixes)
<rharper> ok, force pushed mine after a rebase on master
<smoser> rharper, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327832
<smoser> that one is OK'd by c-i. you ack, i pull
<rharper> ok, when will launchpad update with the correct diff
<smoser> i dont know that it wil
<smoser> just fetch and diff yourself :)
<smoser> or click on the individual commits
<smoser> i have the next one ready too
<smoser>  sysconfig: enable mtu set per subnet
<smoser> with tests added
<rharper> ok
<rharper> we've a new bug in the network state:  the mtu test in curtin registered a manual subnet with no address (AFAICT that's legit);  we have network_state code that checks for 'address' in a subnet and blows up if 'address' key isn't present
<rharper> the type: manual avoided that path since it was never looked at
<rharper> http://paste.ubuntu.com/25134871/
<rharper> that's a one-liner if 'address' in subnet and ':' in subnet['address']
<smoser> currently sysconfig raises a valueerror
<smoser> ValueError: No config network address keys [address] found in {'control': 'manual', 'type': 'static'}
<smoser> actually, yeah. bot in my test raise a valueerror
<smoser> saying its bogus
<smoser> eni and sysconfig
<rharper> hrm
<rharper> let me try curtin.net
<rharper> it's possible that I can just remove it altogether as I can't see what value it would add to the test-case at this time
<smoser> http://paste.ubuntu.com/25134888/
<rharper> nope, fail in curtin.net as well;  well, let's remove it and see what it looks like
<rharper> yeah
<rharper> oh, it was ordering iface stanzas
<rharper> hrm, so how do we get an iface $name manual ?
<rharper> and no ip configuration ?
<smoser> rharper, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327832
<smoser> is now pushed with my fixes on yours squashed and laucnhpad re-rendered
<smoser> no diff other than commit message versus what c-i ran at
<smoser>  https://jenkins.ubuntu.com/server/view/Cloud-init/job/cloud-init-ci/67/console
<smoser> so i will pull if you say good.
<rharper> looks good
<smoser> and then i have 7 ready and i might just include 8 in with it i just straight cherry-picked it
<rharper> ok
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327836
<smoser> rharper, so. ^ has 2 more of them.
<smoser> which leaves us just aaa7101790d5
<smoser>  Fix style issue with unused exeception variable in DataSourceAzure
<smoser> which is not related
<smoser> and i'm not sure what caught it. cause that isnt new
<smoser> and then
<rharper> right, it's not, but i wanted make style-check to pass
<smoser>   sysconfig: use MACADDR on bonds/bridges to configure mac_address
<rharper> it fails here  (xenial laptop)
<smoser> ah. well, the answer is that we dont care.
<smoser> run tox -e pyflakes
<rharper> tox passes
<rharper> am I missing an update?
<rharper> surely we run a tox flakes under xenial ?
<rharper> or is it style from $dev_release  only ?
<smoser> coding style is checked via running
<smoser>  tox -e flake8
<rharper> what does 'make style-check' run ?
<smoser> that pins versions of of things to what we accept as current targets
<smoser> all 'make' targets use whatever things are installed on the system.
<rharper> is there a reason why it doesn't use tox ?
<rharper> if that's the one-true-way  ?
<smoser> make is bascially "use what is on the system"
<smoser> we run 'make test' during the ubuntu builds
<rharper> at best it passes but most likely wrong; I'll submit a patch later;
<smoser> so that we verify the test runs against system versions
<rharper> right, but style-check could certainly do the tox -e flake8 instead
<smoser> but we do not run style checks
<rharper> right, which I'm perfectly fine with
<rharper> I'm just out-dated on how to check style
<rharper> if we have a partually useless make target, I'd like to fix it up
<rharper> that's all
<smoser> because other wise we are held to the least common denominator of styles
<smoser> its not completely useless.
<smoser> it passes for me
<rharper> but it doesn't for me
<rharper> you're on artful
<smoser> (as artful has the same (or similar) versions that we *do* care about)
<smoser> i'm not opposed to removing it... but it doesnt run when you just type 'make'
<rharper> but I've got something else that does't like it ; anyhow, we can bikeshed in the MP (if I submit it)
<smoser> and that is by design
<rharper> that's fine
<rharper> I don't want to change style-check to run anywher else
<rharper> jsut make style-check should match what we say is the right style, which is the frozen set in tox -e flake8
<smoser> nothing in 'make' depends on the internet
<smoser> thats why make doesn't run tox
<rharper> so, sed 's,pep8 $(pyflakes),tox -e flake8'
<rharper> nothing with plain 'make' depends on the style-check by design
<rharper> so why not have the style-check actually check the style with the right versions ?
 * rharper tables it for later
<smoser> once https://jenkins.ubuntu.com/server/view/Cloud-init/job/cloud-init-ci/69/ finishes i'll pull
<rharper> k
<rharper> just about done with rebase of the last few
<rharper> ok, pushed
<smoser> ok... https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327836 will pass c-i. i will pull it . that means we just have the newest commit on your branch to grab.
<smoser> sysconfig: use MACADDR on bonds/bridges to configure mac_address
<smoser> i'll take your tests that you added there and put them into the ones i added rather than adding the mis-named RoundTrip
<rharper> you merged both mtu-per-subnet and ipv6 def-route ?
<smoser> those are both in 7
<smoser> which i'll just pull now (bringing in the 2 individual commits that are there)
<rharper> k
<smoser> those are now there.
<smoser> i'm woring on the last commit.
<smoser> hora!
<rharper> ack
<rharper> no more rebase needed then
<smoser> rharper, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327838
<smoser> i just cherry picked yours and then
<smoser>  https://git.launchpad.net/~smoser/cloud-init/commit/?id=64bd6b52a57cc0a7cc3a4c689958d0c7b2f4add6
<rharper> ack
<rharper> approved
<rharper> thanks!
 * rharper ends smoser some beer
<smoser> yeah, ...
<smoser> so you dont need this in ubuntu
<smoser> you just needed it in trunk for the moment
<smoser> but i think i might just upload to ubuntu too
<blackboxsw> minor comment on https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327838
<smoser> blackboxsw, ack. i'll grab that change.
<rharper> there was at least one commit needed for eni rendering
<blackboxsw> +1 thx
 * rharper looks for the list
<smoser> i dont like using the .get() twice
<smoser> (but we were doing it anwyay)
<smoser> ie, i dont like:
<smoser> if mydict.get('key'):
<smoser>    x = mydict.get('key')
<rharper> smoser: https://bugs.launchpad.net/cloud-init/+bug/1701097  ,is needed back to Xenial
<ubot5> Ubuntu bug 1701097 in cloud-init (Ubuntu Artful) "eni rendering of ipv6 gateways fails" [Medium,Confirmed]
<smoser> but oh well.
<rharper> but only once we release (and sru) an curtin with passthrough enabled
<rharper> we should see about getting those scheduled/started soon given our timeline and desire for MAAS testing
<smoser> blackboxsw, https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/327827 looks very clean. thank you
<smoser> nice commit messages.
<blackboxsw> good deal smoser, thanks
<blackboxsw> I'll get out from under this aws ipv6 in short order I hope
<blackboxsw> smoser: to confirm, how a datasource is determined to run in local mode is by the absense of  <DatasourceModule>.datasources sources.DEP_NETWORK in the deps listing?
<smoser> right
<smoser> ok. so https://jenkins.ubuntu.com/server/view/Cloud-init/job/cloud-init-ci/72/ should finish in 12 minute and then i can pull
<smoser> i might get to that later tonight.
<blackboxsw> have a good one smoser
<rharper> later
<blackboxsw> smoser: /me is wondering why we'd want to try searching/get_data for DataSourceEc2 in the network stage if we can run DataSourceEc2 in local
<blackboxsw> smoser: /me is wondering why we'd want to try searching/get_data for DataSourceEc2 in the network stage if we can run DataSourceEc2 in init-local
<blackboxsw> n/m I think I recall now. The reason attempting the datasource in both init-local and init-network  stages would be to keep non-aws Ec2 clones running in the init-network stage so we don't impact their initialization w/ this changeset.
#cloud-init 2017-07-21
<smoser> blackboxsw, right.
<smoser> it *should* be ok to do what we're doing.
<smoser> and this path (dhcp and use that address) is actually less likely to cause any issues
<smoser> than using the ipv4 link local path.
<smoser> we can talk about that... maybe we don't need a fallback path.
<smoser> have to think more
<powersj> smoser: rharper if I want to run a single cloud-init module and pass in user data how do I do that with: cloud-init single -n hostname
<smoser> cloud-init single --name=hostname --frequency=always
<smoser> and for changing the config that that thing sees....
<smoser> i always just end up writing different content into /etc/cloud/cloud.cfg.d/99-smoser.cfg
<smoser> and re-running boave
<rharper> if you to pass in user-data, you need --file /path/to/cfg
<smoser> rharper, blackboxsw https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327532
<smoser> that is updated with commits for your feeback (and flake fixes too)
<rharper> k
<smoser> quick view of https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327896
<smoser> would be helpful too
<blackboxsw> hrm trying to test freebsd instances with the dhclient changes I can't seem to ssh to the instance w/ root@ec2-..... and the pem file. It's prompting me for a password
<blackboxsw> will check those branches now smoser
<smoser> hrm indeed. :)
<blackboxsw> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327896 approved
<blackboxsw> your extensions to mock in https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327532 remind me about adding linters which should balk if we are importing mock directly
<smoser> blackboxsw, ?
<smoser> i think it doesnt matter though
<smoser> unless you know otherwise
<smoser> i wondered about this and tested quickly
<smoser> if a test does import the 'helpers'
<smoser> and then (before or after) import mock via 'import mock'
<smoser> then the mock is still patched everywhere
<smoser> is that what you were meaning?
<blackboxsw> hrm, I lemme check on that. I probably misinterpreted what mock monkey patching we were doing there in helpers.
<smoser> blackboxsw, theres only one "copy" of an import
<smoser> just differntly namespaced based on how its imported
<smoser> right?
<smoser> that fact is part of the reason on why you can fail to un-mock something in a unit test and have another unit test affected.
<blackboxsw> smoser, ok confirmed the proper behavior as you suggested. Anything that imports helpers will have the properly patch mock.Mock which has assert_not_called since the import already "happened" in original helpers import
<blackboxsw> yeah correct
<smoser> either before or after
<smoser> import mock
<smoser> import helpers
<smoser> or the other way
<smoser> doesn't matter
<blackboxsw> so we *don't* have to worry about where we import mock as we have  enxtended mock.Mock at helper module import time
<blackboxsw> right
<smoser> as long as helper gets loaded
<smoser> ugh
<smoser> cloud-init in archive now depends on nplan | ifupdown
<smoser> updated, then purged ifupdown
<smoser> in lxc container
<smoser> cleaned out /var/lib/cloud
<smoser> reboot
<smoser> does not have an ipv4  dhcp address
<smoser> :-(
<smoser> hm.. maybe it did
<larsks> smoser: around?
<smoser> larsks, here.
<smoser> sorry... slow reply
<larsks> No worries.  So, (a) approved for cloud-init summit.
<larsks> (b) does this seem like a reasonable explanation of the ovirt issue?
<larsks> https://gist.github.com/larsks/62841738dbfad27155628a2560cb818c
<smoser> a. \0/
<smoser> larsks, that sounds pretty reasonable yes.
<larsks> Excellent. Thanks.
<smoser> larsks, i think we *could* keep both the instance id and the previous dmi id around and compare
<smoser> i think that woudl work.
<larsks> Just an extra symlink in /var/lib/cloud/instances, maybe?
<smoser> well, cloud-init loads /var/lib/cloud/instance/obj.pkl
<smoser> and that, which is the ConfigDrive class
<smoser> has the ability to look around and decide if it is new or not
<smoser> i think ... (memory here)
<smoser> we could keep the uuid from dmi and the instance id.
<smoser> and if uuid did not change, we could assume we were same.
<larsks> I guess. It seems better if ovirt were to fix things, since they're pretending to be an openstack data source.
<smoser> yes.
<blackboxsw> hrm, I'm not sure our is_connected function in cloudinit/net/__init__ works on aws cloud instances
<blackboxsw> cat /sys/class/net/eth0/carrier
<blackboxsw> 1
<blackboxsw> I'm seeing a 1 returned from my active eth0 interface (so is_connected returns False)
<blackboxsw> ahh wait a sec, sorry n/m. we check iflink first
<blackboxsw> ok ignore the peanut gallery: python3 -c 'from cloudinit.net import is_connected; print(is_connected("eth0"))'
<blackboxsw> True
<smoser> rharper, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327921 should be good now.
<rharper> k
#cloud-init 2018-07-16
<Guest78> Cheers to the maintainers for making such a great tool.  I'm trying to find documentation on starting from the cloud-init package in the case there is no image available.  In my case, I'm working on a non-major cloud that has the concept of user-data, which is just a string not backed by cloud-init.  Installing the package seems easy enough, but how can I tell cloud-init to look at my custom metadata / user data field?
<Guest78> apologies if I am totally missing the complexity of such a request
<rharper> Guest78: If your cloud does is one of the cloud-init datasources it knows how to read then you may want to look at using the NoCloud datasource which can read information from an ISO, floppy, or a directory within the image
<rharper> http://cloudinit.readthedocs.io/en/latest/topics/datasources.html
<rharper> http://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html
<Guest78> thank you @rharper it is not, but there is a metadata URL so hopefully that can be a source without much hassle
<rharper> Guest78: maybe;   do you know if it 's similar to any of the existing datasources?  like an OpenStack variant ?
<rharper> the quickest way is likely to write a shell program which uses curl/wget to pull the metadata and then you can parse out what you need from it.   Is it only meta-data or does it include user-data as well ?
<Guest78> @rharper the metadata endpoint is different and I'm honestly not sure what is similar.  it seems like I may be able to use nocloud-net in the docs you sent me "With ds=nocloud-net, the seedfrom value must start with http://, https:// or ftp://"
<Guest78> there is user_data in the meta-data, yes, it's a 16K max string
<rharper> can you dump the output from the URL  somewhere ?
<rharper> typically a cloud needs to provide a unique id (instance-id) ; and then user-data would include yaml in cloud-config format which can be processed;   if it's in NoCloud's format (one file for meta-data and one file for user-data)
<rharper> it's possible that the seedfrom=url could work
<Guest78> very cool, you mean the output from the metadata service?
<rharper> yeah
<Guest78> Ok
<Guest78> @rharper excuse the delay that info is a little too revealing, but it's a large json response where user_data is: curl http://<gateway_ip>/skytap/ | jq .user_data
<rharper> np
<rharper> the NoCloud datasource expects cloud-config, which is textoutput starting with #cloud-config\n<yaml>
<rharper> so, the NoCloud datasource isn't going to work with that.
<Guest78> ok, thanks for clearing that up
<rharper> there are more complicated ways to use NoCloud to run a program to extract the user-data/cloud-config and run that manually, but that's not going work well.  Likely the best bet would be to teach cloud-init how to consume the metadata URL directly
<Guest78> ok, great, I'll start looking into that.  For now we'll probably go with your idea of using the shell script to parse user data until we can figure that out.  cheers!
<rharper> sure
<smoser> Guest78: how did/do you sign up for skytap?
<Guest78> @smoser it sorta fell into my lap professionally.  Did some digging and if you want to check it out, one of the professional services folks would like to get you an account.  I'll DM his contact
#cloud-init 2018-07-17
<rharper> smoser: on the net-convert, the autolander failed, there was a pycodestyle failure
<rharper> tools/net-convert.py:95:13: E131 continuation line unaligned for hanging indent
<rharper> ERROR: InvocationError: '/var/lib/jenkins/slaves/torkoal/workspace/cloud-init-autoland-test@2/.tox/pycodestyle/bin/python -m pycodestyle cloudinit/ tests/ tools/'
<smoser> rharper: h.. that is odd
<rharper> yes
<rharper> https://jenkins.ubuntu.com/server/job/cloud-init-autoland-test/7/console
<smoser> oh. well not so odd
<smoser> ther ewas an error :)
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/349484
<smoser> i'll fix
<rharper> k
<rharper> you can re-mark approve and land as well
<smoser> yeah
<smoser> i'll wait on c-i though
<smoser> thanks
#cloud-init 2018-07-19
<rebrec> hi
<rebrec> i would like to use cloud-init on my vcenter infrastructure. Is there a way to do this without using the iso provisionning thing ? i expect to disable cloud-init auto start and run it from the cli with a config file passed as parameter
#cloud-init 2018-07-20
<smoser> rharper: i think i probably paste-binned you a git-propose-merge once ?
<caribou> smoser: good morning; I'm about to push the new version of my MR in our infrastructure as it fixes an IPv6 regression
<caribou> hopefully it'll be adequate for merging, otherwise I'll push a new version
<caribou> btw, I took ownership of LP: 1662345 as it is a showstopper for us to deploy cloud-init on arm64
<ubot5> Launchpad bug 1662345 in qemu (Ubuntu Xenial) "smbios parameter settings not visible in guest" [Medium,Confirmed] https://launchpad.net/bugs/1662345
<smoser> caribou: ok. i will look.
<caribou> smoser: fun thing is that your suggestion fixed an issue we have with IPv6 following our recent deployment
<rharper> smoser: hrm
<smoser> rharper: i found it.
<smoser> thanks to irc logs.
<rharper> heh
<smoser> git-propose-merge is http://paste.ubuntu.com/p/gMBWKvD42W/
<rharper> what was it again?  oh, like sparkie's git to MP
<rharper> right?
<rharper> for launchpad ?
<smoser> yeah, but sparkiegeeks' requires launchpadlib and auth.
<rharper> which is sort of the right way to do it, but  sure , we're already  logged in anyhow
<smoser> this is much faster, and works in a small subset of places . but places that i use.
<rharper> yep
<smoser> well, this just opens up the browser for you
<smoser> and you hit 'merge'
<smoser> his actually creates it.
<smoser> there are other issues i have with his. nothign that couldnt be fixed. that is definitly the right place (or more right than hacky smoser scripts)
<danMS_> smoser: i am working on https://bugs.launchpad.net/cloud-init/+bug/1779207
<ubot5> Ubuntu bug 1779207 in cloud-init "Failed mount of '/dev/sdb1' with Swap File cloud-init config" [Medium,Triaged]
<danMS_> Do you know if other cloud platforms have hit this issue?
<smoser> danMS_: by "platform" you mean cloud platform (not linux distro)
<smoser> right?
<danMS_> yes
<smoser> the issue is probably present everywhere.
<smoser> however it is dramatically worse on azure
<smoser> because of "redeploy" with same instance id.
<smoser> er... maybe not relavant to instance id.
<smoser> let me try again
<smoser> the problem exists any time there are "stale" entries in /etc/fstab
<danMS_> ok, on Azure they will get a new ntfs formatted ephemeral on redeploy or deallocate
<smoser> on Azure, stale entries occur in a regular lifespan of an instance (with 'redeploy'... that might not be the right word)
<smoser> on other platforms, the issue can occur on a snapshot -> new-instance
<danMS_> so it sounds like, when they deallocate their VM, they present the previously attached ephemeral drive
<danMS_> whereas we are not
<mgerdts> blackboxsw: Trying to update DataSourceSmartOS to use EventType.BOOT to get network reconfiguration on boot and hitting a snag.  The change is simple enough, I think: https://hastebin.com/vahizocidi.diff
<smoser> danMS_: well no one else "deallocates"to my knowledge.
<smoser> i dont know acutally.
<danMS_> i have not done a deep dive, but i did not see this issue on Ubuntu vms
<mgerdts> But when I install the new deb networking is not configured properly.  The ENI file has dhcp, and it looks like the saved configuration is for the wrong datasource.
<mgerdts> >>> pickle.load(open('obj.pkl', 'rb'))
<mgerdts> <cloudinit.sources.DataSourceNone.DataSourceNone object at 0x7fc92c313c50>
<mgerdts> That was in /var/lib/cloud/intance.
<mgerdts> *instance
<smoser> mgerdts: blackboxsw is out for a hwhile
<mgerdts> oh, ok.  Any idea what could be causing the wrong datasource data to be cached?  That is, does this sound familiar?
<rharper> mgerdts: hrm;  that's curious;   what's the recreate scenario ? new instance boot, upgrade deb , reboot ?
<mgerdts> install xenial, upgrade to bionic, reboot, install new cloud-init, reboot (ssh host key changed - bad) (did not look closely at networking config), poweroff vm, modify network config in host, restart host metadata service to be sure it picked it up, booted VM.
<rharper> mgerdts: ok;  I suppose it's best to pkl.load and print after each state to see where it changes
<mgerdts> Since then, I did cloud-init clean -l, reboot.  It put DataSourceNone on obj.pkl
<rharper> ah
<rharper> that sounds like unidentified change
<rharper> clean wipes the previous instance info
<rharper> so next boot ds-identify needs to run and pick; what does the log look like after that reboot ?
<mgerdts> https://hastebin.com/axaguxusit.txt
<rharper> no local data found from DataSourceSmartOS
<rharper> so the SmartOS DS didn;t say "yes" I have metadata/user-data
<rharper> when cloud-init called .get_data() on it
<rharper> which means that the fallback is DatasourceNone
<rharper> so the question is , why did the SmartOS DS say it had no local metadata  ?
<rharper> isn't that over the serial interface ?
<mgerdts> yeah,
<mgerdts> I'll try some debug statements in _get_data
<rharper> yeah, I don't see a return without a boolean, and the False path has logging =(
<mgerdts> that's what I was thinking
<rharper> and some debugging in sources/__init__.py
<rharper> we now do this metadata caching;
<smoser> mgerdts: you should be able to use the main too
<smoser> python -m cloudinit.sources.DataSourceSmartOS
<smoser> migth be easier to debug that way
<mgerdts> that dumps a bunch of metadata
<smoser> so i think the pickled object must have a method that is getting in the way. method or attribute i guess.
<rharper> IIUC, cloud-init clean was run
<rharper> which wipes the object
<smoser> oh. hm..
<mgerdts> yes, and verified that clean worked
<smoser> so then this is essentially fresh boot ?
<smoser> or as llose as clean can get us ?
<mgerdts> commenting the change to DataSourceSmartOS caused ENI/50-cloud-init.cfg to get static network config.  Oddly, not the right network config.
<mgerdts> https://hastebin.com/aviqacabew.txt
<mgerdts> ip should be .223 per sdc:nics, but is .222
<rharper> "ip":"10.88.88.223","ips":["10.88.88.222/24"]
<rharper> your data disagrees
<mgerdts> huh.  I guess I missed one of the ip's int he zonecfg.
<rharper> I think we only look at ips since it was a superset ?
<mgerdts> notice 222 and 223 in there
<rharper> yeah
<mgerdts> ok, so something in the get_data() path is unhappy with update_events = {'network': [EventType.BOOT]} in DataSourceSmartOS.
<mgerdts> I'll go hunting
<mgerdts> Looks like the comment in class DataSource is wrong.  This seems to work:
<mgerdts> update_events = {'network': [EventType.BOOT_NEW_INSTANCE, EventType.BOOT]}
<mgerdts> Apparently BOOT_NEW_INSTANCE is not a subset of BOOT.
<rharper> oh, yeah
<rharper> that must have been asperational
<mgerdts> :)
<rharper> Don't we log if we skip reading it ?
<mgerdts> I think DataSourceSmartOS should do: update_events['network'].append(EventType.BOOT)
<mgerdts> Doesn't look like it.
<rharper> update_metadata could use some logging in the negative path I think
<rharper> otherwise you get return False and nothing
<mgerdts> yeah, I'll add something there along with updating the aspirational comment.
<rharper> thanks for debugging that
<mgerdts> Yeah, no problem.  Thanks for the nudges in the right direction.
<mgerdts> hopefully this addresses the changed ssh host key after upgrade + reboot.
<mgerdts> would you like the fix for blackboxsw's change in a separate changeset from the smartos changes?
<mgerdts> likely: https://hastebin.com/evokubiluj.diff
<rharper> mgerdts: the ssh won't regen if the instance-id hasn't changed;
<rharper> separate is best if that's not too much trouble
<mgerdts> so maybe dpkg -i cloud-init_all.deb cuased it to get clobbered.
<mgerdts> sure, easy enough.
<rharper> I suspect we should also add a unittest on the derived Datasource
<rharper> I wonder if the actual unittests do the .append() like you did
<mgerdts> sadly unittests fail now.  So probably need some fixes there too.
<mgerdts> I think I managed to sort this out.  https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/350374 then https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/350375
<rharper> mgerdts: thatnks, reviewing
<smoser> mgerdts: you just want the first to land first ?
<smoser> the second is a superset right ? but not separate commits.
<smoser> i think.. ?
<mgerdts> yeah, the second will break without the first.
<mgerdts> due to list vs. set
<mgerdts> I tried to set dependencies in the merge proposal, but not sure if that is actually hooked into anything.
<mgerdts> awesome.  python 2.6 strikes again.
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/350381
<smoser> mgerdts: ./tools/run-container is fairly easily usable from ubuntu if you alve lxc to get you a centos/6
<smoser> set notation?
<mgerdts> yeah
<mgerdts> should be fixed now.  will CI bot automatically re-run or does it need to be nudged?
<smoser> rharper: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/350382 will fix tip-pylint
<rharper> auto reruns
<rharper> looking
#cloud-init 2019-07-15
<tombee> Hey, does anyone have any examples on how to set up LVM for mounted disks (non-root partitions) with cloud-init?
<rharper> https://canonical.my.salesforce.com/5003z00001yXYRIAA4
<rharper> Odd_Bloke: so for the Oracle Datasource issue; I believe this is what they see _on_ OracleLinux using updated cloud-init datasource
<rharper> hence the "dracut" initramfs fix at the bottom
<Odd_Bloke> Right, I think I understand the issue there, based on a quick scan through.
<AnhVoMSFT> rharper any concern in landing this https://code.launchpad.net/~vtqanh/cloud-init/+git/cloud-init/+merge/369785 ? I looked at current analyze code and it looks like it's only finding events that are of type start: or finish:, so these events won't be even touched by analyze (which is what we want)
<AnhVoMSFT> for stacking azure-ds events nicely within analyze (which will be another PR), I think one possible way to handle it would be to make an educated guess of which cloud-init phases we're in when we see those events (during analyze) based on the timestamps. Otherwise we would have to refactor the code to make args.reporter global so that other reporte
<AnhVoMSFT> rs can find it as the parent
<rharper> AnhVoMSFT: do you have a cloud-init analyze show/blame output with those new events in place ?
<AnhVoMSFT> rharper https://paste.ubuntu.com/p/mfc4YSbFQQ/
<rharper> AnhVoMSFT: right; we still have to do something for  https://bugs.launchpad.net/cloud-init/+bug/1833731
<ubot5> Ubuntu bug 1833731 in cloud-init "cloud-init analyze output not formatted cleanly on Azure" [Undecided,New]
<rharper> AnhVoMSFT: but the diagnostic event doesn't contribute to the formatting issue;  we should land something for the current formatting and your additional events as well
<AnhVoMSFT> yep, we do need to fix that, but like you said it shouldn't be the blocker for those additional events
<AnhVoMSFT> I'm trying to explore the options for fixing it
<rharper> ok
<samgilson> hey Odd_Bloke, on my MP you just approved the CI bot flagged my commit message for not meeting the style standards. I just fixed it, but is there any way to re-run it from my side to make sure it is correct?
<rharper> samgilson: no, we can poke it again
<rharper> samgilson: ok, I've remarked approved
<AnhVoMSFT> rharper What do you think about this fix for analyze? https://paste.ubuntu.com/p/zHyDKCbWVr/ - it's relatively simple. WIP branch: https://code.launchpad.net/~vtqanh/cloud-init/+git/cloud-init/+ref/UpdateReporterForAnalyze
<samgilson> rharper thanks so much!
<AnhVoMSFT> the names are now showing up correctly, but I think we need to land your fix that would let analyze process more than 2 level deep to be able to render it correctly
<AnhVoMSFT> Also I updated the merge proposal for the additional telemetry addressing your comments
<rharper> thanks;
<AnhVoMSFT> rharper if I apply your commit here https://paste.ubuntu.com/p/y3JskPgtg8/ on top of that UpdateReporterForAnalyze WIP branch and run cloud-init analyze show I get this nicely formatted output: https://paste.ubuntu.com/p/y3JskPgtg8/
<rharper> AnhVoMSFT: \o/ ;  though I think we shoudl somehow get a DataSourceAzure prefix in those somehow
<AnhVoMSFT> wrong link to the commit : https://git.launchpad.net/~raharper/cloud-init/commit/?id=fe1d854dbb63fa6c8219547473cdc02a7fb02709
<rharper> that does look much better
<rharper> would like to get a init-local/datasource/_get_data  which would give us one more indent level I think;
<rharper> well, s/datasource/$name_of_the_Datasource;  but I think it may be easy enough to set that in your event wrappers rather than some how read it from the current ds object
<AnhVoMSFT> yeah, the event wrapper can probably inject azure_ds: in there
<Odd_Bloke> rharper: https://code.launchpad.net/~xiaofengw/cloud-init/+git/cloud-init/+merge/368902 <-- did you accidentally kick off the autoland test job manually for this MP?
<Odd_Bloke> (It's not a problem if you did, just trying to understand the error message!)
<rharper> possibily
<rharper> hrm;  I know I added a ci run; but maybe I also submitted an autoland
<rharper> the comments are off
<rharper> net: skip bond interfaces in get_interfaces (detail)
<rharper> I did marked that branch approved
<rharper> but I don't see how it updated the vmware MP
<Odd_Bloke> :/
<rharper> indeed
<samgilson> hey Odd_Bloke rharper the CI bot came back around and set my MP back to needs review, the commit message should be correct now.
<rharper> samgilson: thanks
<Odd_Bloke> samgilson: Thanks!  Convincing the bot to accept your commit message is a rite of passage for cloud-init development, so welcome to the club. ;)
<samgilson> Odd_Bloke Glad to see its not just me who messes this up '=D
<Odd_Bloke> samgilson: For reference, I normally do `git commit --allow-empty`, compose in $EDITOR (vim, in my case), and then `:w /tmp/commit-msg` and use that.
<Odd_Bloke> (My vim is configured to wrap git commits correctly, so it solves the problem.)
<samgilson> Odd_Bloke thanks! I will try it for the next one and hopefully it works =D
<Odd_Bloke> Fingers crossed!
<AnhVoMSFT> rharper https://code.launchpad.net/~vtqanh/cloud-init/+git/cloud-init/+merge/370156 - I "stole" your commit (cherry-picking it) into my branch on top of my changes for the MP
<rharper> AnhVoMSFT: nice
#cloud-init 2019-07-16
<tribaal> rharper: addressed doc comments. Thanks for your review. Not sure if I should switch the branch back to "needs review" however? I'll let you do that if necessary (I have lower connectivity this week - I'm abroad).
<rharper> tribaal: replied;  we mark branches WIP to avoid re-reviewing something that's pending;  so if you've pushed changes up, you can always switch state to Needs Review
<AnhVoMSFT> rharper what is the best way to update commit's author name without code change? If I do a git rebase -i will launchpad pick it up?
<AnhVoMSFT> thanks - just updated the author for the MP https://code.launchpad.net/~vtqanh/cloud-init/+git/cloud-init/+merge/369785
<rharper> AnhVoMSFT: thanks
<rharper> Odd_Bloke: https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/370213
<Odd_Bloke> rharper: +1 on that MP; I couldn't look at test results because I don't have VPN creds on this laptop.
<rharper> Odd_Bloke: cool, I'm fixing pycodestyle
<rharper> but otherwise I think it's good to go
<rharper> unittest verifies the fix
<Odd_Bloke> Yep, +1 on whatever change you have to make.
<rharper> cool
<rharper> thanks
<Odd_Bloke> rharper: +1 on ephemeral DHCP, want me to Approved it?
<Odd_Bloke> rharper: I realised you're AFK ATM, so I Approved it.
#cloud-init 2019-07-17
<rharper> Odd_Bloke: thanks!
<rharper> AnhVoMSFT: Can you rebase with author update here too: https://code.launchpad.net/~vtqanh/cloud-init/+git/cloud-init/+merge/370156
<vmwlab> Hi looking for resources to learn cloud-init other than read the docs as they are more documentation than a learning materials . thank you
<vmwlab> are there such resources? as I  cant find any online classes or even books, only old videos or blogs from 5 years ago.. Anyone ?
<powersj> vmwlab, we recently published a whitepaper on cloud-init here: https://ubuntu.com/engage/cloud-init-whitepaper
<powersj> vmwlab, I did a talk at debconf17 about it: https://debconf17.debconf.org/talks/164/
<powersj> vmwlab, and here are the docs https://cloudinit.readthedocs.io/en/latest/
<powersj> vmwlab, and another talk last year https://events.linuxfoundation.org/wp-content/uploads/2017/12/Cloud-init-The-cross-cloud-magic-sauce_Smith_moser.pdf
<powersj> I can't find the video replay for that one
<powersj> and one more video https://www.youtube.com/watch?v=RHVhIWifVqU
<powersj> with that I am going to bed, but feel free to ask some other questions if you have any here!
<vmwlab> perfect thank you powersj
<rharper> Odd_Bloke: so if we take any branches today; then I'll need to wait till tomorrow to wait for CI to complete;  so I'm thinking to wait on https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/366667 till after 19.2 ;
<rharper> we can also land them after 19.2 and pick them up in the 19.2 SRU
<Odd_Bloke> rharper: Understood, happy for you to cut 19.2 as-is.
<rharper> ok; that's what I was thinking as well;
<Odd_Bloke> rharper: I believe https://code.launchpad.net/~daniel-thewatkins/cloud-init/+git/cloud-init/+merge/369783 is ready for re-review now
<rharper> Odd_Bloke: great!
* rharper changed the topic of #cloud-init to: Reviews: http://bit.ly/ci-reviews | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting July 22 16:15 UTC | cloud-init v 19.2 (07/17)
<Odd_Bloke> rharper: I'm struggling to put my response to your review comments in to meaningful words, so I'll probably revisit that when I'm back on Monday.
<rharper> Odd_Bloke: ok
<Odd_Bloke> And if I'm still struggling to use my words, we can hangout to nail things down?
<rharper> definitely
<Odd_Bloke> Thanks!
<smoser>  https://launchpad.net/~cloud-init-dev/+archive/ubuntu/daily/+build/17282846
<rharper> smoser: I've seen that recently; that sort of feels like the release branch was missing a patch to remove the function;  but I'm not sure why we'd see that now;
<rharper> I've a build failure from before 19.2 anyhow
<rharper> maybe a partial revert
<rharper> smoser: alright, IIUC, the revert patch  debian/patches didn't completely revert all of the changes to the ubuntu-advantage module; but it updated the unittests, so unittest expected the older code; but the cc_ubuntu_advantage.py still had newer bits.
<rharper> I've refreshed the revert quilt patch to restore the file prior to the updated ua commit;  I can pass tox before and after apply all patches; and IIUC, the daily build checksout master, but the debian dir from ubuntu/bionic;  so we should then quilt push patches from ubuntu/bionic onto master cleanly; and tox should pass.
<rharper> verifying that works, and I'll push a fix up commit to ubuntu/bionic;  and I think I need that on xenial and cosmic as well
#cloud-init 2019-07-18
<smoser> rharper: yeah, so essentially ubuntu/xenial branch needs to build with current trunk.
<rharper> smoser: I believe I've got it fix for ubuntu/bionic; will replay the same refresh on xenial and cosmic next
<rharper> powersj: https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/370329
#cloud-init 2019-07-19
<smoser> hm.
<meena> good morning happy people. how do i close a bug i opened? https://bugs.launchpad.net/cloud-init/+bug/1799674
<ubot5> Ubuntu bug 1799674 in cloud-init "tools/build-on-freebsd is outdated and can be dropped" [Undecided,Incomplete]
<meena> oh, i can just change the status to Fix committed!
<muhaha> its there any lockfile or status file which I can watch if cloud-init is successful done?
<rharper> muhaha: you can run 'cloud-init status --wait'
<Nick_A> got a strange issue
<Nick_A> we have a debian 9 image with cloud-init. you launch the image using a form that takes a root password and applies it like http://paste.openstack.org/show/7bqsoXTLs8Af9UkO4Ffu/
<Nick_A> it salts the password entered through the form
<Nick_A> however the password doesn't work
<Nick_A> but the same image works if launched with userdata: http://paste.openstack.org/show/PR9AU8ElrZ8NRNIWzRPT/
<Nick_A> other images work fine, it's just debian 9 for some reason...
<rharper> do you know what version of cloud-init ?
<rharper> maybe this bug? https://bugs.launchpad.net/cloud-init/+bug/1811446
<ubot5> Ubuntu bug 1811446 in cloud-init (Ubuntu) "chpasswd: is mangling certain password hashes" [Undecided,Fix released]
<rharper> Nick_A: ^^
<Nick_A> rharper cloud-init 0.7.9
<rharper> Nick_A: does it reproduce in newer cloud-init versions ?
<Nick_A> I haven't forced debian to use a newer version. was trying to stick with defaults
<Nick_A> trying 18.3
<Nick_A> @rharper 18.3 works
<Nick_A> sorry slack habit with the @
<rharper> no worries
#cloud-init 2020-07-13
<PandemiK> HEllo
<PandemiK> Somone there to help me understand how datasource priority is working on cloud-init ?
<Odd_Bloke> PandemiK: There are a few people who may be able to help; go ahead and ask your question and you'll find out. :)
<PandemiK> Ok, I just don't want to bother you with some noob question
<PandemiK> Thanks ;-)
<PandemiK> I'm using CI with Proxmox VE. In PVE, we cannot alter meta-data and user-data generated by PVE has iso9660 nocloud source
<PandemiK> I'm trying to add another user-data source and merge it with PVE's one
<PandemiK> So I thought I could go ahead with smabios settings
<PandemiK> So I thought I could go ahead with smbios settings
<PandemiK> But where settings something like "ds=nocloud-net;s=https://dev.altinea.fr/"
<PandemiK> The files on the iso9660 are not used anymore. Any idea howto to 'merge' them without touching the PVE user-data file ?
<PandemiK> Access to the user data file is public : https://dev.altinea.fr/user-data
<PandemiK> I tried to setup merge_how on top a the file
<PandemiK> perhaps could I add some #include ? Is that possible with a iso9660 file (cidata) ?
<MICROburst> How to set an interface to promisc mode?
<Odd_Bloke> PandemiK: So the best way to do this would be to use multi-part user-data with an include, but that would require the user-data provided by Proxmox VE to be modified.
<PandemiK> that's my fear
<PandemiK> I also wrote a little patch for PVE that add support for vendor-data
<PandemiK> That does the job (installing ssh keys and a few packages) but need to integrate a patch to PVE
<PandemiK> And having a patch into mainstream PVE would be ... long
<Odd_Bloke> PandemiK: If that is immutable, then it becomes substantially more complicated; cloud-init expects to fetch its NoCloud configuration from a single source (e.g. an ISO, or a remote URL), so changing the configuration of the instance away from PVE's ISO will mean that you no longer get the metadata which PVE provides, which is likely to cause problems.
<PandemiK> without PVE's iso9660 source, we don't have fqdn for example
<PandemiK> which means creating a file for each VM, not wanted
<Odd_Bloke> I'm not sure I follow; why would that imply "creating a file for each VM"?
<PandemiK> Is there a way to provide only vendor-data from a URL with smbios version setting ?
<PandemiK> PVE generate some interesting config inside it's 'own' user-data : fqdn and hostname parameter, derived directly from the VM's name
<PandemiK> everything else seems to be vm-agnostic so I could set it in another file (from a url for example)
<Odd_Bloke> Ah right, so you're saying you'd need to have a separate file on dev.altinea.fr for each server (and specify which one of those files to use in SMBIOS config at launch time)?
<PandemiK> exactly !
<Odd_Bloke> Yeah, I can see why you want to avoid that. :)
<Odd_Bloke> Do you create your own images, or do you using stock images provided by distros/OSes?
<PandemiK> distro's one (Ubuntu in my test case)
<PandemiK> All datasource are available, perhaps could I mix two ?
<Odd_Bloke> Nope, each boot uses a single DS, I'm afraid.
<PandemiK> I only need to have some kind of vendor-data merged with user-data to add node to puppet and install a few packages
<PandemiK> sad :-( So It would necessary go for a code change in Proxmox
<Odd_Bloke> Yeah, PVE not allowing you as a user to provide user data is breaking cloud-init's assumptions, unfortunately; maybe I'm not thinking of another way around it, but it does sound like a platform issue that's difficult for us to find a way around.
<PandemiK> ok, many thanks for your help and confirmation
<PandemiK> Time to annoy PVE devs so ;-)
<Odd_Bloke> ^_^
<Odd_Bloke> Sorry I couldn't help more!
<PandemiK> no problem, at least, I'll stop wasting time with something not possible
<PandemiK> And a little remark if there's cloud-init dev here : CI seems to NOT support ssh certificates
<PandemiK> I've been able to hack around this with 'disable_root_opts' of the ssh plugin but supporting it officially would be great
<PandemiK> despite this, the possibilities with cloud-init are almost endless, it's really exciting !
<PandemiK> (if I could provide my own datas :-)
<Odd_Bloke> PandemiK: It's funny you should mention SSH certificates, someone opened this PR related to them yesterday: https://github.com/canonical/cloud-init/pull/487 :)
<smoser> PandemiK: "having a patch into mainstream PVE would be ... long".  I'd recommend starting that process.  As the old saying goes, "the best time to plant a tree is 30 years ago.  The second best time is now".
<PandemiK> nice quote ;-)
<smoser> PandemiK: one thing that maybe could help you... i dont know if we ever did this or not. i dont think so.
<smoser> but we could make cloud-init able to put the instance id into the header of a seed request
<PandemiK> @smoser : as said, this would need to generate a user-data 'on the fly' per VM, right ?
<smoser> yeah... you're going to have to do that. i think. as yeah..
<smoser> as you don't have a way to put any differenciationg information into the url. (if i understand correctly)
<PandemiK> My goal is to let PVE generated all vm-specific datas and add only vm-agnostics datas
<smoser> some other cloud-init-like things put mac addresses of the NICs available in the headers
<PandemiK> I won't try to create a script that get the instance-id from proxmox, correlate with VM fqdn and return a specific user-data file
<smoser> so pve just does not provide any user-data ?
<PandemiK> did I mention I'm not a developer ;-)
<smoser> ah. i see. i think.
<smoser> you dont' want to provide launch-time input ?
<smoser> i guess https://pve.proxmox.com/wiki/Cloud-Init_Support is what you're using ?
<PandemiK> yup
<PandemiK> It seems there's an effort in what I'm looking for :
<PandemiK> https://lists.proxmox.com/pipermail/pve-devel/2020-July/044241.html
<PandemiK> really new but promising
<smoser> yeah... it seems that that is what you're after.
<smoser> and you'd just put in as ciuserdata base64("#include your.url.here\n")
<smoser> it is pretty crazy... it looks from their patch that proxmox is writing yaml with strings and indents.
<smoser> ie 'content .= "... mtu "
<PandemiK> I'm trying to understand their idea but yeah ... strange
<smoser> rather than creating some dict and yaml.dump()
<PandemiK> remember it's Perl ;-)
<smoser> yaml is a strict super set of json
<smoser> cloud-init reads yaml, but they could just write json
<smoser> and i'm sure there is a json for perl
<smoser> (there probably yaml too)
<MICROburst> How to set an interface to promiscuous mode?
<Odd_Bloke> MICROburst: I don't believe that we have platform-independent configuration to express promiscuous mode, so I think you'll have to configure the system to apply it yourself.  You may want to look at networkd-dispatcher which is recommended by the netplan docs (https://netplan.io/faq#use-pre-up-post-up-etc-hook-scripts) but this is not my area of expertise so I can't speak categorically.
<MICROburst> <Odd_Bloke>: So you suggest to put the stuff in write_files: ?
<Odd_Bloke> MICROburst: That's what I'd try first, yeah.
#cloud-init 2020-07-14
<blackboxsw> Odd_Bloke: https://github.com/canonical/cloud-init/pull/466/files reviewed. minor docstring/typing nit leaving as approved so you can make a call on whether it's needed or not .
<Odd_Bloke> blackboxsw: Thanks; addressed and replied, though I didn't follow exactly your request so please take a look and LMK if you're +1. :)
<blackboxsw> d'oh forgot about typing dependency
<blackboxsw> approved
<AnhVoMSFT> was there a status meeting today?
<blackboxsw> AnhVoMSFT: ahh it is today. thank you. forgot it was Tuesday
<AnhVoMSFT> no worries - maybe we skip this week and have a mega one the next time :)
<blackboxsw> #startmeeting cloud-init status meeting
<meetingology> Meeting started Tue Jul 14 16:52:35 2020 UTC.  The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology.
<meetingology> Available commands: action commands idea info link nick
<AnhVoMSFT> nvm
<blackboxsw> heh.
<blackboxsw> let's do it since we have an active attendee.
<blackboxsw> i'll make it snappy today
<AnhVoMSFT> sounds good
<blackboxsw> community notice: time for another bi-weekly (or semi-monthly if you prefer) cloud-init community status meeting
<blackboxsw> #chair rharper Odd_Bloke smoser
<meetingology> Current chairs: Odd_Bloke blackboxsw rharper smoser
<blackboxsw> welcome to another round of cloud-init upstream updates and discussion. We use this meeting as a time to gather to discuss current development of cloud-init, ask and answer questions, and generally expedite development be unblocking devs. All questions. side-conversations and interruptions are welcome
<blackboxsw> first order of bizzzznesss. setting the meeting for next time
<blackboxsw> +2 weeks from today
* blackboxsw changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting July 28 16:15 UTC | 20.1 (Feb 18) | 20.2 (Apr 28) | https://bugs.launchpad.net/cloud-init/+filebug
<blackboxsw> July 28th same time (minus 30 mins)
<blackboxsw> previous meeting minutes are here:
<blackboxsw> #link https://cloud-init.github.io/status-2020-06-30.html#status-2020-06-30
<blackboxsw> The topics we'll cover today: Previous Actions, Recent Changes, In-progress Development, Community Charter, Office Hours (~30 mins).
<blackboxsw> #topic Previous Actions
<blackboxsw> none seen last session. so we can jump to the next topic
<blackboxsw> #topic Recent Changes
<blackboxsw> The following commits have been landed on master of upstream branch since last meeting: found via git log --since 2020-06-30
<blackboxsw> #link https://paste.ubuntu.com/p/6Fn5jy8t46/
<blackboxsw> a bit of cleanup and test coverage work and CI fixups for lxd integratin testing in those commits.
<blackboxsw> Of note:
<blackboxsw> a fix for (LP: #1456277) thx lucas
<ubot5> Launchpad bug 1456277 in cloud-init "cloud-init searches for ec2 mirrors regardless of what cloud its on" [High,Fix committed] https://launchpad.net/bugs/1456277
<blackboxsw> a fix for (LP: #1884619) part of the cloudinit.net refactor for thx Odd_Bloke
<ubot5> Launchpad bug 1884619 in cloud-init "cloudinit.net refactor: is_physical" [Low,Fix committed] https://launchpad.net/bugs/1884619
<blackboxsw> and (LP: #1886531)  fix for missing /etc/fstab file path thx rharper
<ubot5> Launchpad bug 1886531 in cloud-init "cloud-init status broken in groovy lxd containers" [Medium,Fix committed] https://launchpad.net/bugs/1886531
<blackboxsw> and thanks paride for fixing our CI tests for lxd-based targets
<blackboxsw> I think that about wraps recent-changes.
<blackboxsw> #topic  In-progress Development
<blackboxsw> I was hoping today we'd be able to finally say cloud-init 20.2 has published and released to Ubuntu Xenial. Bionic, Eoan and Focal. All testing is complete, we have unblocked any of this process on our side and we are awaiting an SRU team representative to review and release the bits into Ubuntu proper.
<blackboxsw> We pinged yesterday and a few hour ago again to get this SRU reviewed and released. Expectation is that it will be released to all Ubuntu series today/tonight, so I'd expect that cloud images see that update in the next day or two.
<blackboxsw> for those watching at home, the folowing bug will be closed as fix-released once cloud-init SRU is published.
<blackboxsw> #link https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1881018
<ubot5> Ubuntu bug 1881018 in cloud-init (Ubuntu) "sru cloud-init (19.4.33 to 20.2-45) Xenial, Bionic, Eoan and Focal" [Undecided,In progress]
<blackboxsw> for context, this was a beast of an SRU (as upstream waited too long (~6 months) between to get the SRUs, which involved more verification and complexity. We will make sure to avoid some of this complexity in the future by more frequent SRUs and more requests for community validation I expect.
<blackboxsw> Also, expect that when this SRU is published, and email will be sent to cloud-init@lists.launchpad.net and a discourse post as well as a "community-notice:"  banned comment
<blackboxsw> #topic Community Charter
<blackboxsw> Community driven development  is what helps keep cloud-init active, so that you all for your contributions ( PR reviews, bugs, PR development, discussion etc).
<blackboxsw> we have a number of general goals we continue to work toward:
<blackboxsw>  - json schema coverage of cloudinit.config.cc_* modules for better error reporting on malformed user-data
<blackboxsw> - datasource documentation updates and content creation needs
<blackboxsw> - cloudinit.net refactor into distro-specific networking subclasses cloudinit.distros.networking
<blackboxsw> Bugs associated with that work are available for anyone to own
<blackboxsw> #link k https://bugs.launchpad.net/cloud-init/?field.tag=bitesize
<blackboxsw> #link https://bugs.launchpad.net/cloud-init/+bugs?field.tag=net-refactor
<blackboxsw> And details docs on existing refactor are available here
<blackboxsw> #link https://cloudinit.readthedocs.io/en/latest/topics/hacking.html#ongoing-refactors
<blackboxsw> If anyone would like to be involved more than they currently are, please feel free to contact us here in IRC #cloud-init on Freenode or on the mailing list cloud-init@lists.launchpad.net and we can see how best we can get you "set up"
<blackboxsw> #topic Office Hours (next ~30 mins)
<blackboxsw> This 'section' of the meeting is a time where a couple of upstream devs will be available in channel for any discussions, questions, bug work or PR reviews.
<blackboxsw> any topics, bugs, PRs or concerns or rotten fruit to throw are all welcome :). In the absence of dicussion, cloud-init PR reviews are prioritized
<blackboxsw> Odd_Bloke: can I merge the following as your squashmerge commit message for PR https://github.com/canonical/cloud-init/pull/466      https://paste.ubuntu.com/p/ZRdq4bYWG7/    I think it fixed LP: #1884626
<ubot5> Launchpad bug 1884626 in cloud-init "cloudinit.net refactor: wait_for_physdevs" [Low,In progress] https://launchpad.net/bugs/1884626
<Odd_Bloke> blackboxsw: You cannot (because I just did ;).
<blackboxsw> :sad trombone:
<Odd_Bloke> I just reran the RTD doc build and it passed this time (for anyone else who just got that failure email).
<blackboxsw> Ok I think that about wraps the status meeting
<blackboxsw> thanks for tuning in folks
<blackboxsw> #endmeeting
<meetingology> Meeting ended Tue Jul 14 17:50:44 2020 UTC.
<meetingology> Minutes:        http://ubottu.com/meetingology/logs/cloud-init/2020/cloud-init.2020-07-14-16.52.moin.txt
<blackboxsw> minute reposted https://cloud-init.github.io/status-2020-07-14.html#status-2020-07-14
<Odd_Bloke> otubo: FYI, I'm planning on getting back to your PRs over the next couple of days; I'm still working through a backlog from some days off last week.
#cloud-init 2020-07-15
<sdhd-sascha> Did anybody know an easy tutorial for creating an bond, with pxe and maybe install "kubeadm", "rke" or some other way to create an Kubernetes cluster? (on bare-metal....)
<Holiday> I know this isn't directly cloud-init related but I wasn't able to find an answer in #Ubuntu. Any reason why after an apt-get install cloud-init there would be 3 systemctl units on ubuntu 18.04 and cloud-init runs on boot as expected but in 20.04 after the apt install there's no units listed in systemctl and nothing runs on boot even after a systemctl enable cloud-init?
<Holiday> It looks like everything is there and good but nothing kicks off. a 'systemctl start cloud-init' fires it off and displays what I'd expect to the terminal, just wasn't sure what by apparent default would keep it from doing so on boot in 20.04
<sdhd-sascha> @Holiday I don't know. I'm only guest here... But i figured out, since years. That LTS ubuntu is only LTS after some years... It maybe better to use some older distro... (Except you are a expert, and can handle this kind ;-) )
<sdhd-sascha> Maybe i try landscape today. Some weeks ago it didn't support 20.04
<Holiday> @sdhd-sascha ah sadly hands are tied as this is for a provision portal for internal devs/users/etc to spin things up. Our home grown solution used vmware-tools and ansible so it was just "does ssh work", where as the new solution I'm assisting in rolling out we went cloud-init a couple years ago for the linux side which worked well.. until 20.04 made me a liar haah
<Odd_Bloke> Holiday: Hmm, I'm not sure off-hand.  What type of Ubuntu system are you installing cloud-init on that doesn't already have cloud-init?
<Holiday> it's a 20.04 LTS, built from kickstart to be mostly bare bones, so I did an apt install cloud-init like I did with our 18.04 build (built the same way with kickstart and the same overall packages)
<sdhd-sascha> @Holiday @Odd_Bloke : As i know MAAS use cloud-init ...
<Holiday> just confused on why the systemd files all seem there, systemctl enable doesn't yell at me about it not existing, but then the darn thing doesn't run on boot but will by hand haha. I know, ubuntu prob
<Holiday> or I just have some fub in my script that's preventing run on boot but everything is the same as the 18.04.. hence my 'what. the. heck.'
<Odd_Bloke> Holiday: Can you run `cloud-init collect-logs` on an affected 20.04 machine and upload the tarball it produces somewhere?  (A Launchpad bug filed using the link in /topic would work.)
<Holiday> rgr I'll work on that one
<Odd_Bloke> Thanks!
<Odd_Bloke> (My guess is that the cloud-init generator is tripping you up, but I'm not entirely sure; cloud-init is generally pre-installed in Ubuntu images where it would be used, so post-first-boot installation isn't a path that is regularly walked.)
<Holiday> @odd_bloke https://bugs.launchpad.net/cloud-init/+bug/1887668  I hope I put enough detail without being too like.. who knows. Did this while on a zoom haah
<ubot5> Ubuntu bug 1887668 in cloud-init "cloud-init doesn't appear to be running on boot" [Undecided,New]
<Odd_Bloke> Holiday: OK, so it looks like my guess was correct: if you look at /run/cloud-init/ds-identify.log, then you can see "No ds found ... Disabled cloud-init", which means that cloud-init is entirely disabled.
<Odd_Bloke> Looking further up in the log, you can see that "rpctool query returned 1".
<Odd_Bloke> Holiday: Which is emitted by https://github.com/canonical/cloud-init/blob/master/tools/ds-identify#L791-L808
<Odd_Bloke> So we're expecting `vmware-rpctool "info-get guestinfo.ovfEnv"` to return 0 and it instead fails.
<Holiday> so that is related to the vmware tools from my quick google.. which open-vm-tools is in place
<Holiday> okay I see running the vmware-rpctool "info-get guestinfo.ovfEnv" returns "No value found". I also ran on my 18.04 but it returns the same value
<Odd_Bloke> Holiday: Has the exit code changed between the two Ubuntu releases?
<Odd_Bloke> (`echo $?` after running it.)
<Holiday> they're the same. I'm not 100% sure that's the main cause as the software as far as I remember is forming an iso with the user rundata to inject (mounts to the VM pre-boot), and is using the root account via vmware tools rpc for the other bits (morpheus is the portal software).
<Holiday> but again I'm not super well versed in this area. The exit code for both is 1
<Odd_Bloke> Hmm, OK, could you pastebin ds-identify.log from the 18.04 system?  (https://paste.ubuntu.com/)
<Holiday> https://paste.ubuntu.com/p/hd95RyBbZY/
<Holiday> quick skim between the two systems seems everything there matches up
<Odd_Bloke> *polishes monacle and peers through it* "Disabled cloud-init", you say?
<Odd_Bloke> Oh, actually, a major difference there is uptime.
<Odd_Bloke> Holiday: If you reboot the bionic instance, does cloud-init run?
<Holiday> hrm well it did, something I changed in this clone apparently made it stop haha
<Holiday> basically it was running every time it booted/rebooted. now for some reason it stopped, I did just run an 'apt update; apt upgrade'
<Holiday> going to clone from my base again and check that one pre-updates
<Holiday> okay yeah so I cloned my base bionic to a new VM as well and on power up cloud-init runs and I see all that "calling http://169.254.169.254/<etc>' failed stuff as expected for the 120 seconds
<Holiday> interesting.. on the one I ran 'apt upgrade' on, a systemctl list-units | grep cloud once again returns no results like the 20.04 system that's not running on boot either
<Odd_Bloke> Oh, that is interesting; what version of cloud-init is that upgrading from?
<Holiday> so pre-upgrade, an apt list --installed | grep cloud returns cloud-guest-utils @ 0.30-0ubuntu5 and cloud-init @ 19.4.33-gbb4131a2-0ubuntu1
<Holiday> hrm same version after the apt upgrade
<Holiday> I did see systemd get and what not get updated (although since the base auto-updates on provision via cloud-init there were a ton of packages that got updated), so apparently not cloud-init related at all since the versions are the same but another package that's being updated
<Holiday> just needed that second set of ideas and extra thoughts/pokes to realize that. Still doesn't make sense why on 20.04 where I installed cloud-init and haven't updated after that doesn't work on boot like expected
<Holiday> funny thing is I remember getting hit by the "updating systemd killed my network" bug. I'm going to slide the cause over into that area
<Odd_Bloke> Holiday: So one thought I have: is the OVF ISO available to the instances for the entire time they're up?
<Odd_Bloke> Because if it is removed, then cloud-init will potentially no longer identify that there is anything for it to do, and disable itself.
<Odd_Bloke> (Which could be a difference between the two instances which isn't tied to Ubuntu version.)
<Odd_Bloke> And my other thought: a systemd upgrade will reload the daemon, so it will pick up new services on-disk at that point.
<Odd_Bloke> So if you install cloud-init and later (or in the same transaction, probably) upgrade systemd on one system, but don't on the other; then that could explain the difference in reports.
<Odd_Bloke> (A `systemctl daemon-reload` after you install cloud-init should have them appear available.)
<Holiday> I'll check that daemon-reload
<Holiday> the only ISO in use is with the rundata that morpheus passes over. Other than that there's no OVF or ISO as these are existing VMs within vCenter. Historically every reboot cloud-init would run (which was a bit of an 'ugh' because I'd have to wait for the 2 minute timeout before I could do something in the base VM we clone from but boy do I miss that delay now haha)
<Holiday> trying to purge cloud-init now via apt and re-installing too to see if there's any change that way
<Odd_Bloke> Yeah, so I think the problem you're having is that your instance doesn't appear to be running on a cloud, so cloud-init disables itself.
<Odd_Bloke> From what I can tell, it sounds like focal is behaving as we would expect; you've been getting away with it on bionic for some reason. :p
<Holiday> haha could be
<Holiday> guess worst case I'll make an rc.d entry to kick it off and then use the rundata to wipe that entry or something.
<Odd_Bloke> Holiday: As you build these images, you could configure cloud-init to look only for the datasource you're expecting to use.
<Odd_Bloke> That would have two effects: the 120s timeout would go away, and ds-identify won't disable cloud-init if only a single datasource is configured.
<Odd_Bloke> (Because it assumes that whoever wrote the configuration is smarter than it is, precisely for this sort of case.)
<Holiday> haha obviously I proved that wrong
<Holiday> alright I'll look into that area and see if I can figure something out there. This is the only area so far where we've used cloud-init and it initially was setup in short time as we were on an initial time crunch. I'll do some reading up in that area and see if that helps at all
<Holiday> I still just find it weird it worked before the update but not after, yet the cloud-init package version didn't change
<Holiday> arlight, time to google and read. Thank you for all the assistance, though, input and direction pointing!~
<Odd_Bloke> Holiday: If there's anything more we can do, let us know!
<smoser> https://github.com/canonical/cloud-init/pull/492 makes me think of https://youtu.be/3m6Blqs0IgY
<Odd_Bloke> falcojr: https://github.com/canonical/cloud-init/pull/493 <-- the Oracle PR
<falcojr> will take a look
<Odd_Bloke> smoser: You may also be interested in ^ as the original author of DataSourceOracle; it's refactoring DataSourceOracle to use Oracle's metadata endpoint.
<smoser> Odd_Bloke: are instance ids the same between opc/v1 and openstack ?
<smoser> +1 to oracle for not having putting HTML in their metadata responses.
<blackboxsw> community-notice: cloud-init 20.2 finally released! to Ubuntu Xenial, Bionic, Eoan and Focal. Thanks all for the patience and verification efforts! email/blog post forthcoming
* blackboxsw changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting July 28 16:15 UTC | 20.2 (Jul 15) | https://bugs.launchpad.net/cloud-init/+filebug
<blackboxsw> expect to see 20.2 in daily published cloud images within the next day or two
<Odd_Bloke> smoser: Good question!  I'll check.
<Odd_Bloke> smoser: N=1, yes.  I'll ask Oracle for confirmation.
<powersj> blackboxsw, I think the date for 20.2 should stay at Apr 28. That is when upstream cut the release.
<blackboxsw> ahh right +1
<powersj> 20.3 should then be planned for sometimes from now till sept. so we stay on track for 4 releases/year
* blackboxsw changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting July 28 16:15 UTC | 20.2 (Apr 26) |  https://bugs.launchpad.net/cloud-init/+filebug
<blackboxsw> powersj: yep, plan for 20.3 is in short order when some Oracle work is complete. (and to avoid this costly an SRU)
<powersj> sweet
<blackboxsw> will send an email to the list early next week
<blackboxsw> -> camping for a real woodsy outdoor wedding Th/Fri
<blackboxsw> :w
<blackboxsw> falcojr: followup on your https://github.com/canonical/pycloudlib/pull/3/files
<blackboxsw> or first review round rather
#cloud-init 2020-07-16
<mruffell> congratulations for the cloud-init 20.2 SRU release!
<smoser> otubo: https://code.launchpad.net/~smoser/cloud-utils/+git/cloud-utils/+merge/387530
<smoser> your test/feedback on that would be nice.
<Odd_Bloke> falcojr: Thanks for the review!  Is the "another channel" you mentioned the gist that you're putting together?
<falcojr> Yep! Let me link it here
<falcojr> https://gist.github.com/TheRealFalcon/2665d0c12ae8944c288c342507691178
<Odd_Bloke> OK, nice, I'll have a read through that this afternoon.
<Odd_Bloke> falcojr: This isn't really prompted by your document, but I was thinking more last night about how to run multiple test methods against the same instance: I wonder if an autouse class-scoped fixture would be the way to go: any tests that are in the same class will run against the same SUT.
<Odd_Bloke> (Don't know if it should be autouse, but that would avoid needing to request the fixture in every test.)
<Odd_Bloke> (I do wonder how that would interact with parameterization: if we parameterized a class to run the same tests with two different configurations, would they share the SUT?)
<Odd_Bloke> falcojr: Another stray observation for your "launch args" mark: marks can take kwargs which can be accessed the same way as args are.  I've just confirmed this is supported in xenial.
<falcojr> yeah, I wasn't sure about kwargs. Thanks for checking on that. I think a fixture might be the way to go then. I think class level fixtures like you mention make sense too, but at last as we have enough flexibility at the top-level, those sort of fixtures can come as we need them
<Odd_Bloke> falcojr: I've responded to ~the first half of your gist in a comment. :)
<falcojr> Cool, I'll take a look
<catphish> morning
<catphish> i'm looking at how cloud providers use cloud-init, most seem to provide config from a known url, usually on 169.254.169.254, but i'm wondering, how do these VMs generally establish network connectivity prior to fetching this config, are they using DHCP?
<catphish> actually, i think i have a better solution, but i'm sure i'll be back with more questions :)
#cloud-init 2020-07-17
<smoser> catphish: most of the time "dhcp on eth0"
<smoser> at least for the initial connection to the metadata service.
<catphish> smoser: thanks, i thought that might be the case, unfortuinately in my environment i don't want to rely on being able to run a dhcp and web server on all networks a vm might be connected so, so i'm looking at using a config disk instead
<smoser> catphish: hardware ?
<catphish> i don't understand
<catphish> x86-64 qemu/kvm VMs
<catphish> in a "public cloud" environment, but where some VMs might be on a private network
<smoser> catphish: so no... not hardware ;)
<smoser> "physical" is what i meant.
<catphish> oh right nope :)
<smoser> you're looking to create a public cloud ? or your wanting to use/ammend an existing one?
<catphish> i'm making one
<catphish> or rather, just adding cloud-init to one with the hope of replacing a proprietary postinstall mechanism
<smoser> if you just wanted to "ammend" an existing one.
<smoser> i think config disk (config drive) is a reasonable solution. You can put it on any partition (it doesn't have to be a separate disk... with gpt, you've got 128 partitions and spending 10MiB on a config location sin't really costing you anything)
<smoser> but if you wanted to ammend a network solution, then one way you could do it would be to provide the MAC of the nic where the metadata servcie lived in dmi info
<smoser> somehow just tell cloud-init "this is the mac to dhcp on" and then it can go on as it does
<catphish> what options can i provide in smbios?
<catphish> because that would be idea if i can force it to use a link-local IP
<catphish> i'm puzzled because the default address seems to be APIPA, but it doesn't seem to assign an APIPA IP
<catphish> ie by default it appears to try to download a config from 169.254.169.254 but never assigns an IP
<catphish> i can't see any way to use network config without ipv4 dhcp
<smoser> catphish: yeah.. i think you're prbably looking at amazon
<smoser> or at least, yes. that is how ec2 works.
<smoser> but one thing crazy about ec2... they run their server on a ipv4 link local address, as you've found
<smoser> but if you configure ipv4 link local, you wont get a response from them.
<catphish> smoser: same with digitalocean
<smoser> digital ocean i think *does* do ipv4ll
<catphish> yeah, it does, it assigns a random address
<smoser> but on ec2, if you don't request from the ipv4 address that the dhcp request would give you, then you can't talk to the metadata service
<smoser> :)
<smoser> digital ocean's is better.
<catphish> yeah, that was my interpretation too
<smoser> as it is right now, what you're wanting to do is not implemented fully.
<catphish> i'm tempted to write my own to do one of 2 things: 1) fetch the data from a virtual serial port or block device 2) use ipv6 link local properly
<smoser> config-disk (I'd suggest NoCloud) is the only real option.
<smoser> oh...
<catphish> yeah nocloud (config-disk) basically works
<smoser> serial... one of the datasources does that
<smoser> joyent's
<catphish> yeah i was just looking that that joyent thing
<smoser> it does a metadata service over serial
<smoser> yeah
<catphish> annoyingly they use a proprietary format
<catphish> but it could work
<smoser> and then you avhe to serialize requests over serial
<catphish> yeah, that's not *too* painful
<smoser> if i were going to design a new datasource now, i htink i would
<smoser> a.) provide instance-id and mac address of "the right" nic in dmi data (or possibly some other "local" manner)
<smoser> b.) use ipv6 link local to hit a well known address (or maybe provide the ipv6 address in 'a' also)
<smoser> c.) get the rest of the network configuration from the metadata service there.
<smoser> i think joyent actually did a reasonable job.
<smoser> oh... other "local" transports are available.  hypervisor <-> guest sockets .
<smoser> vsock i think is the thing
<catphish> a local (unixy) socket would be absolutely amazing
<catphish> basically i'm down to 2 options: 1) write a datasource that pulls data from the hypervisor by some kind of virtual socket 2) use an ISO datasource (basic nocloud)
<catphish> i'm currently going with option (2) because it doesn't require somehow getting my data source into every OS image i want to use, i'm running into trouble with caching though, qemu doesn't seem to refetch the iso for subsequent requests, and i'd like to to able to change the contents
<catphish> since cloud-init unmounts the iso, it should be safe to change it, then reprovision later, but something is caching it, hopefully i can address this
<catphish> one specific question - can i use an ISO to configure the network, but ALSO configure a network data source to run at the next stage?
<catphish> smoser: after much internal discussion, we're going to write a datasource :) it's going to 1) get an instance-id and security key from DMI 2) get an interface MAC from DMI 3) bring up the appropriate interface and fetch the remaining config from a well-known ipv6 link-local IP
<catphish> and of cource we're going to be dickheads and make sure our config is slightly nonstandard, and name the module after our platform, so that nobody else can use it ;)
<catphish> thank you for your assistance
<smoser> catphish: unattaching the iso will be a pain.
<smoser> cloud-init will look for the instance-id on each boot
<smoser> so you have to disable cloud-init on subsequent boots or set 'manual_cache_clean'
<smoser> and, fwiw, it doesn't have to be an ISO
<smoser> as in iso9660. it has to be a filesystem with a label.
<smoser> catphish: what would be neat... and faster development cycle
<smoser> is if  you also made it able to take instance-id, security-key and interface from environment .
<smoser> then you can iterate on lxd containers quite quickly
<smoser> actually, now remembering ... there was a pan to do a more advanced datasource for lxd
<smoser> using /dev/lxd which is a socket
<catphish> smoser: yeah, i'm not a fan of the iso (or ext2 or whatever) filesytstem idea, seems kinda messy having an extra filesystem dangling around, so yeah, going to have a proper stab at the IPv6 LL approach, the only downside is that i'll have to modify every OS image to include it
#cloud-init 2020-07-18
<uebera||> Are there known problems regarding a specified uid of a new account? "users:    - name: sys-maint    uid: 500    no_user_group: true    primary_group: adm" results in the creation of uid/gid 1000 (using cloud-init 20.1-10-g71af48df-0ubuntu5). What am I missing?
<meena> uebera||: maybe the system doesn't allow to create a user with a uid < 1000, unless you say it's a system user.
<uebera||> meena: Good point, will try to set system to true.
<uebera||> I wonder, though, why I would not see any error/warning message in that case.
<uebera||> Also, according to the documentation, there would not be any home directory in that case (which would make it difficult to populate /home/sys-maint/.ssh/authorized_keys, I guess).
<meena> uebera||: for the first question: try to doing it locally on the command line. for the second:
<meena> why does it have to be uid 500?
<uebera||> meena: Because that's the uid for said account on xx machines.
<uebera||> (Might have been better to choose 400 in the past, but oh well.)
<uebera||> meena: For the record, using "system: true" resulted in a home directory owned by root:root, the uid in use is now 998, but .../.ssh/authorized_keys have been populated. Somehow leaves me with more questions regarding uid handling.
<meena> uebera||:  ðï¸ðï¸  i have to sleep now.
<uebera||> meena: Me too. ;)
<uebera||> My bouncer doesn't need sleep, though, so if anyone can provide me with some pointers, I'm happy to go through them tomorrowâ¦ :D
<uebera||> Ok, I identified the underlying issue myself: https://bugs.launchpad.net/cloud-init/+bug/1396362/comments/4 tells you that the value for uid has to be provided as a string rather than an integer.
<ubot5> Ubuntu bug 1396362 in cloud-init "Support UID and GID specification in user and group definitions" [Low,Triaged]
<uebera||> % id -a
<uebera||> uid=500(sys-maint) gid=4(adm) groups=4(adm),27(sudo)
<uebera||> \o/ ... gn8
