#cloud-init 2014-07-07
<smoser> mnaser, mgagne a couple things:
<smoser> a.) kernel resize of online disk is orders of magnitude faster with newer kernels . that got magically fast ~ 3.5 i think.
<smoser> b.) backgrounding the resize ends up not being all that useful on the older kernels.  It should background, but it is extremely IO intensive and halts most other things.   it should work correclty, but everything will be slow.
<mnaser> smoser: i notied that, also it doesnt seem like it background cloud-init .. if nothing else in cloud-init is setup to run, it won't really be backgrounded
<smoser> it shoudl bacground
<smoser> mnaser, is the call to util.fork_cb just broken ?
<smoser> it looks right
<mnaser> smoser: so what was happening is that it was saying "Backgrounding resize" but it was still blocking
<smoser> hm.. i really doubt its broken.
<mnaser> and the proof to that even more is that the "part" that does resizing would say "took 180 seonds" let's say and it was blocked there
<smoser> hm..
<mnaser> but this is cloud-init of CentOS 6
<mnaser> so maybe that's not been tested
<mnaser> also i recall
<smoser> i dont know what is in centos 6. i'm just looking at trunk
<smoser> but it *is* crazy io intensive
<mnaser> there was an error
<smoser> and so is boot
<smoser> so originally when i did this, i found it to be almost useless.
<mnaser> with the callback being a NoneType or something
<smoser> hm..
<smoser> let me look at that again
<mnaser> also this is 10x ssd drives in raid-10 so it's pretty fast
<smoser> you're right
<smoser> its busted
<mnaser> smoser: Jul  7 12:34:20 localhost [CLOUDINIT] util.py[DEBUG]: backgrounded Resizing took 41.487 seconds
<mnaser> Jul  7 12:34:20 localhost [CLOUDINIT] util.py[DEBUG]: cloud-init mode 'init' took 41.799 seconds (41.80)
<mnaser> and then it continues afterwards
<smoser> :-(
<smoser> and i broke it when i added the util.log_
<smoser> open a bug ?
<mnaser> perhaps
<mnaser> sure
<mnaser> smoser: https://bugs.launchpad.net/cloud-init/+bug/1338614
<smoser> mnaser, try: http://paste.ubuntu.com/7760542/ ?
<smoser> are you able to easily try that ?
<mnaser> hm
<mnaser> i think i can spin up a new vm with cloud config disabling resize
<mnaser> then patch it and run the module manually
<mnaser> i assume if i run it manually it should background as well
<mnaser> heck, let me try to patch one that is already running because it's failing to fork resizefs anyways
<smoser> it should work, yea.
<smoser> mnaser, thank you.
<mnaser> smoser: fyi in the cc_resizefs.py file
<mnaser> 'args': (resize_cmd, log)) should be 'args': (resize_cmd, log)}
<smoser> yeah
<mnaser> doesnt give that warning anymore... now to test if it's actually properly backgrounding will take a bit longer
<mnaser> but let me do it, solving this will be very usefu
<smoser> mnaser, it was calling fork_cb completely wrong.
<smoser> so i'm trying to just fix how it calls it, by adding a fork_cb_kwargs and calling that one, and making fork_cb backwards compat and call the fork_cb_kwargs.
<smoser> my concern is that fork_cb might not be completley backwards compat
<mnaser> and maybe other parts thats rely on it might be broken now
<mnaser> right so just made a new vm that was not resized automatically
<mnaser> now i'll do the patch for it, delete all the cloud init stuff, reboot and *hopefully* it'll background
<mnaser> or should i just run it manually in the shell after patching and it'll be okay, smoser ?
<smoser> rm -Rf /var/lib/cloud
<smoser> and reboot
<smoser> and see
<smoser> that'll be more complete at least
<smoser> there is only one other caller though
<smoser> so i could just patch it too
<mnaser> alright so its patched up now
<mnaser> rebooting
<mnaser> its only going to do a resize from 20gb to 40gb but i think it'll still show
<mnaser> oh, it didnt do the resize again beacuse it read the user data (configdrive) and that says not too
<mnaser> hm
<smoser> ah. rihgt.
<smoser> and you can't easily override that. 
<mnaser> Jul  7 15:22:03 resize-test [CLOUDINIT] cc_resizefs.py[DEBUG]: Skipping module named resizefs, resizing disabled
<mnaser> even if i force it to run
<mnaser> from the shell
<smoser> well if you tell it to run it still reads config
<mnaser> i guess what ill do is
<mnaser> take a snapshot of this server
<mnaser> and boot another one
<smoser> mnaser, just put 'resize_root = NOBLOCK'
<smoser> right before that
<smoser> ie, force it on
<smoser> actually..
<mnaser> before where exactly?
<smoser> if you run it manually you *should* be able to pass it in
<mnaser> cloud-init --debug single -n resizefs
<mnaser> this didnt run it manually
<smoser> cloud-init --debug single -n resizefs noblock
<smoser> try that
<mnaser> i'll take a snapshot and boot from it a much larger machien so we have a feel of what it's like
<mnaser> so if anything else is disturbing, we'll see it
<smoser> mnaser, it is crazy slow.
<smoser> i have done resize on spinning disk from 2G to 2TB
<mnaser> oh ill make 20G to 80G resize
<smoser> and it took > 1 our i think
<mnaser> yeah i know that pain :P
<smoser> 1 hour
<mnaser> ssd, 20G to 1.2TB was 10 minutes
<mnaser> still al ot
<smoser> yeah.
<smoser> and it is crazy magic fast in 3.5
<smoser> its 1000x faster
<mnaser> oh well, centos :-(
<smoser> yeah
<mnaser> alright, got snapshot, booting it now
<mnaser> smoser
<mnaser> Jul  7 15:31:57 resize-test [CLOUDINIT] util.py[WARNING]: Failed forking and calling callback log_time
<mnaser> Jul  7 15:31:57 resize-test [CLOUDINIT] util.py[DEBUG]: Failed forking and calling callback log_time#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/cloudinit/util.py", line 224, in fork_cb_kwargs#012    child_cb(*args, **kwargs)#012TypeError: log_time() argument after * must be a sequence, not NoneType
<smoser> boo.
<smoser> mnaser, http://paste.ubuntu.com/7760686/
<smoser> try that
<smoser> and i apologize
<smoser> i wondered about that
<mnaser> no worries
<mnaser> now i can easily rename and reboot :p
<mnaser> btw smoser 
<mnaser> backgrounded Resizing = Backgrounded resizing
<mnaser> cannot imagine how much that bothered me in logs :-P
<smoser> ?
<smoser> oh. i see. capitalize :)
<mnaser> yes :p
<smoser> well, the 'R' makes sense.
<smoser> sort of 
<smoser> in my world
<smoser> as the 'Resizing' in the non-backgrounded path.
<mnaser> smoser: really silly
<mnaser> but
<smoser> (so grepping for 'Resizing' would get either)
<mnaser> why not just do this
<mnaser> def fork_cb_kwargs(child_cb, args=[], kwargs={}):
<smoser> mnaser, generally, http://satyajit.ranjeev.in/2012/01/12/python--dangerous-default-value-as-argument.html
<mnaser> aah
<mnaser> that's good to know
<smoser> https://docs.python.org/2/tutorial/controlflow.html#default-argument-values
<mnaser> rebooting
<smoser> see the 'important warning' there
<mnaser> always learning something new
<mnaser> Jul  7 15:48:46 resize-test2 [CLOUDINIT] util.py[WARNING]: Failed forking and calling callback log_time
<mnaser> Jul  7 15:48:46 resize-test2 [CLOUDINIT] util.py[DEBUG]: Failed forking and calling callback log_time#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/cloudinit/util.py", line 229, in fork_cb_kwargs#012    child_cb(*args, **kwargs)#012  File "/usr/lib/python2.6/site-packages/cloudinit/util.py", line 1830, in log_time#012    ret = func(*args, **kwargs)#012TypeError: 'str' object is not callable
<mnaser> smoser ^
<mnaser> tho
<mnaser> i think i should ahv esaid this earlier
<mnaser> running cloud-init 0.7.4 not trunk
<mnaser> so maybe this has to do with it
<smoser> mnaser, i'll stop bothering you and i'll just make sure it works on trunk before asking again.
<smoser> k?
<mnaser> smoser: no problem, thank you so much :)
<harlowja> smoser u might want to respond to http://lists.openstack.org/pipermail/openstack-dev/2014-July/039514.html :)
<harlowja> see last line ;)
<cloudnoob> harlowja: I emailed you about cloud-init-dev over the weekend?
<cloudnoob> I will paste my email into irc :)
<harlowja> cloudnoob howdy
<cloudnoob> It looks like the default configuration of cloud-init on Ubuntu 12.04 LTS causes a new RSA key & fingerprint to be generated when an existing (root) EBS volume is attached to a new instance[1]. I'd like to avoid this behavior on some of my instances.  After some poking around, I think removing 'ssh' from cloud_init_modules list in /etc/cloud/cloud.cfg will prevent this from happening[2]. Can a cloud-init-dev@'er confirm this is t
<cloudnoob> oh hmmm
<cloudnoob> looks like I hit a limit
<cloudnoob> 1/ It looks like the default configuration of cloud-init on Ubuntu 12.04 LTS causes a new RSA key & fingerprint to be generated when an existing (root) EBS volume is attached to a new instance
<harlowja> that would seem fine to do
<cloudnoob> 2/  I'd like to avoid this behavior on some of my instances.
<harlowja> take ssh module out
<cloudnoob> 3/ After some poking around, I think removing 'ssh' from cloud_init_modules list in /etc/cloud/cloud.cfg will prevent this from happening
<cloudnoob> 4/ Can a cloud-init-dev@'er confirm this is the case, and that this is a safe/secure thing to do for existing systems which have already been through once-per-instance setup? I'm also open to advice or suggestions on better approaches.
<cloudnoob> see http://serverfault.com/questions/294039/rsa-fingerprint-changed-after-moving-ebs-and-eip-to-new-instance for a similar writeup
<cloudnoob> also my understanding is based on the assumption that 'ssh' in this list corresponds to executing the handler defined in 'cc_ssh.py'.
<harlowja> cloudnoob i think u can also place ssh_genkeytypes: [] in userdata
<harlowja> to avoid it doing any extra key creation
<cloudnoob> ok
<harlowja> or adjust it in /etc/cloud/cloud.cfg
<cloudnoob> is that better than removing from cloud_init_modules?
<harlowja> disabling ssh makes it impossible for any user keys to come in, if they ever want
<harlowja> i think u are hitting http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/config/cc_ssh.py#L91 which the ssh_genkeytypes: [] should just stop
<cloudnoob> ah
<harlowja> aka, no more generation, but if people provide them that code path will work
<cloudnoob> okay cool
<cloudnoob> I'll try it
<cloudnoob> there are existing keys so yeah, just don't want to overwrite them
<cloudnoob> thanks for the suggestions
<harlowja> np
<cloudnoob> and thanks for the software of course
<cloudnoob> teh useful
<harlowja> np
<harlowja> thx for using it :-P
<smoser> harlowja, thanks for the heads up. responded
<harlowja> smoser cool
<harlowja> i'm not touching that question ;)
<harlowja> u can, haha
<harlowja> *not the ipv6 question, the lets get rid of metadata service
<harmw> can't we just ditch ip6?
<harmw> far easier
<harmw> everything just works with ip4
<harmw> lets keep it that way
<ijw> yo
<ijw> Since I'm here, I wanted to ask: has anyone thought about a generic format for network information (rather than the /etc/network/interfaces file that Openstack likes to template, badly, and then inject)?
<harlowja> ijw yes
<ijw> How much thinking so far?
<harlowja> ijw http://bazaar.launchpad.net/~harlowja/cloud-init/better-todo/view/head:/TODO.rst (this is updated with this, although its been thought of for a long time, ha)
<harlowja> smoser u should merge that branch :)
<ijw>   I have an MTU problem that is really not going to fit in with the current mechanisms
<harlowja> ijw i don't think its cloud-init that would have the biggest problem with changing, its the other things that write the format :)
<ijw> Win7 doesn't advertise for MTU on DHCP, and Linux has Opinions on how big the MTU can be
<harlowja> ijw https://fedorahosted.org/netcf/ has been discussed (although its xml, eck, haha)
<ijw> Given that the in-page example is a bridge it's more complex than we strictly need anyway
<harlowja> sure
<ijw> (also, you can't specify data by eth0/1/2 since they're assigned by the OS)
<harlowja> ijw the issue is that other surronding projects need to get away from writing this format also, this imho is the harder part :)
<harlowja> *neutron/nova in openstack
<harlowja> ijw ideally someone really needs to own that format change and push it through the various openstack projects
<harlowja> the cloud-init work i don't think would be that much
<harlowja> *at least to do the basics
<smoser> harlowja, we have a fail safe datasource.
<smoser> ijw, yeah, mtu sucks.
<harlowja> smoser damn gotta change that in the todo, no longer a todo then, ha
<smoser> ijw, from my perspective, it needs to be specified by the "provider" in one way or another.
<ijw> It's not even a format change - it's an additional file on the web server or config drive that could happily be ignored by anything that didn't know about it
<smoser> dhcp is reasonable for dhcp . but ideally it'd be available if not dhcp.
<harlowja> ijw sure, just write "network.xml" (or something)
<ijw> And I think if it had interface, optionally with VLAN/v4(s)/v6(es)/mtu that would be sufficient
<smoser> this advertising needs to be more part of the network imo
<harlowja> and cloud-init will use it if it exists (or if not look for the old way)
<smoser> but maybe not.
<smoser> it is very problematic
<smoser> and we're hitting it here too.
<ijw> smoser: DHCP in either the v4 or v6 forms has various issues advertising MTU.  The client needs to know, and v6's spec is written with a pedantic respect to the fact that ethernet doesn't standardise packet sizes >1500 bytes (so making bigger MTUs is hard)
<ijw> Assuming YAML wouldn't make people cry, the spec could be very straightforward, in fact
 * ijw finds an etherpad
<harlowja> :), bb, meeting
<smoser> it just seems really odd that this isn't sovled
<smoser> harlowja, merged. dont say i never did anything for you :)
<harlowja> smoser woot :-P
<ijw> By sheer coincidence someone's just talked to be about bonding, too...
#cloud-init 2014-07-08
<harlowja> smoser what are your thoughts on https://code.launchpad.net/~harlowja/cloud-init/changeable-templates/+merge/208994 (this will help in the py3 update, to have this capability)
<harlowja> that combined with https://code.launchpad.net/~harlowja/cloud-init/py2-3 should get most of the issues out of the way
<smoser> harlowja_away, oh yeah. forgot about cheetha.
<smoser> the stuff there is backward compat ? in taht it will use the old cheetah format if the template doesnt have 'template: foo' ?
<mnaser> i'm seeing issues with cloud-initramfs-tools .. more specifically with the dracut module on el7
<mnaser> growpart doesn't seem to run properly.. getting this http://pastebin.com/AH2ftLzX
<smoser> mnaser, i suspect you need https://code.launchpad.net/~juergh/cloud-initramfs-tools/dracut-growroot-pre-mount
<smoser> personally i thikn the dracut code is racey and broken
<mnaser> it looks like it smoser 
<mnaser> but now im discovering something else
<mnaser> you posted this https://lists.fedoraproject.org/pipermail/cloud/2013-May/002405.html
<smoser> juerg's solution was to make it happen before the disk is mounted
<mnaser> centos 7 comes shipped with 3.10
<mnaser> does that mean i can just use that and forget the dracut stuff?
<smoser> mnaser, i think it shoudl work
<smoser> yes.
<mnaser> will it run a reboot on the machine?
<smoser> no.
<mnaser> :>
<smoser> it uses 'update' 
<mnaser> the perks of new linux kernels!
<mnaser> heh
<smoser> kernel in 3.8 now lets you do that.
<smoser> yeah. it is really nice.
<mnaser> alright so i'm going to remove it
<mnaser> and see what happens
<smoser> yeah. 
<smoser> see growpart --help output
<smoser>  --update
<smoser> it shoudl dtrt, but requires partx and 3.8+ kernel.
<smoser> i think you can just add '--update=on' and it will fail if it doesn't think it is supported.
<mnaser> yep looks like it
<mnaser> runs the verify_ stuff
<mnaser> alright, this is good stuff, let me clear everything and try this out
<mnaser> smoser: woohoo, worked! :-)
<mnaser> thank you so much, this is a bit of a relief, making images is annoying!
<smoser> i agree making images is annoying
<smoser> i rant whenever i can that people shoudln't do it.
<mnaser> but there is no centos 7 images
<smoser> a sane distribution provider would provide them for you
<mnaser> at least not *yet*
<smoser> maybe you should chekc the sanity of your distro
<smoser> :)
<mnaser> haha :p
<mnaser> the good thing is they're moving towards making that happen
<smoser> cool. i'm honestly ok with other distros.  and i do understand that there are lots of reasons why "just use ubuntu" isn't really an option.
<smoser> but we really try to make "just use ubuntu" the obvious solution when you say "boy, this could work better".
<harlowja> smoser right the idea for https://code.launchpad.net/~harlowja/cloud-init/changeable-templates is that it will work with cheetah or jinja
<harlowja> and peopel that will want python3 will really be required to use jinja (since cheetah sorta dead)
<smoser> right
<smoser> harlowja, good
<harlowja> smoser https://code.launchpad.net/~harlowja/cloud-init/py2-3 keeps on getting bigger though, lol
<harlowja> stupid stuff changes :-P
<harlowja> but the only library that doesn't appear on python3 is i believe the oauth one and cheetah
<smoser> and oauth is supposed to be dropped anyway
<harlowja> kk
<harlowja> i don't understand why they moved so many modules around in py3
<harlowja> and then six had to come in and fix that, lol
#cloud-init 2014-07-09
<smoser> harlowja, i will take a further look at that stuff on monday
<smoser> i'm out the rest of the week.
<harlowja> np
<harlowja> smoser all good
<harlowja> have fun boss sir 
<harlowja> make sure to leave your basement during vacation
#cloud-init 2014-07-10
<linuxgeek_> hey cloud-init
<linuxgeek_> i want to use the cloudinit feature, so as a first step i started reading cloudinit on the net
<linuxgeek_> got useful info
<linuxgeek_> i did a nova boot --flavor x --image xx --userdata /root/test.yaml testvm
<linuxgeek_> the content of the test.yaml is copied from here http://cloudinit.readthedocs.org/en/latest/topics/examples.html#run-commands-on-first-boot
<linuxgeek_> now i logged into the instance
<linuxgeek_> and checked the /etc/hosts file
<linuxgeek_> it has not echoed the line
<linuxgeek_> any thoughts what i'm missing?
<JoshNang> hey, does anyone have time to review a nova spec? we'd like to add a json description of the network interfaces to configdrive/metadata service to make it more flexible. https://review.openstack.org/#/c/85673/
<JoshNang> our use case is baremetal servers (openstack ironic) with bonded interfaces and vlans over those bonds, and be able to support debian/centos/windows more easily. we're more than happy to write the patches to make cloud-init use the new json :)
<harlowja> JoshNang i'll look it over
<harlowja> linuxgeek_ can u get the cloudinit log file, might have a reason why that didn't happen, /var/log/cloud-init.log (i think)
<harlowja> JoshNang on the cloudinit side, what did u think about using to write it in, https://fedorahosted.org/netcf/ ?
<harlowja> or other?
<harlowja> JoshNang any reason why not just to use that format that netcf takes as input?
<JoshNang> harlowja: i figured we'd have to hand code a lot of it. netcf looks like it'd make life a lot easier, at least for fedora/centos/etc, and maybe we can pitch in to help with adding .deb backends 
<harlowja> damn irc client just crashed, JoshNang  what did i miss, think i missed a line 
<JoshNang> harlowja: openstack in general seems to be trying to moving away from xml. i don't have a strong opinion either way
<JoshNang> what you maybe missed: i figured we'd have to hand code a lot of it. netcf looks like it'd make life a lot easier, at least for fedora/centos/etc, and maybe we can pitch in to help with adding .deb backends
<harlowja> https://git.fedorahosted.org/cgit/netcf.git/tree/src/drv_debian.c :)
<harlowja> might just already work
<harlowja> so convert the xml format to json and be done, lol
<harlowja> or yaml, or plaintext
<harlowja> ha
<JoshNang> heh if i have to write a json to netcf gen, i'm sure i can come up with something
<JoshNang> plaintext could be fun :)
<linuxgeek_> harlowja, i do not find cloudinit directory or clound-init.log. will this be present when the image has cloudinit installed?
<harlowja> it should be, u should find a /etc/cloud/cloud.cfg that should have the config, if thats missing then that makes me wonder if cloudinit is installed at all
<harlowja> JoshNang anyway, thats just a nitpick, the format as long as its compatible would be great, then can just use netcf (which already seems to do most of what we want)
<harlowja> JoshNang when smoser  gets back i'm sure he'll have other inputs, he wants to fix some of the current brokenness that is this stuff
<JoshNang> harlowja: sweet. i'll review their format and adjust as needed.
<JoshNang> harlowja: sounds good! more input is always helpful
<harlowja> JoshNang u might be interested in https://bugs.launchpad.net/cloud-init/+bug/1153626 
<harlowja> which is where i think smoser  wants to go (much more dynamic network config, apparently already done in ec2?)
<harlowja> maybe not for your spec, but something to think about
<linuxgeek_> i'll upload a ubuntu cloud image since they have cloudinit installed. so the cloud-init.log should have info if its a success or a failure?
<harlowja> linuxgeek_ typically yes, if not u can adjust cloud.cfg log levels
<JoshNang> harlowja: ooo interesting. i definitely want to see more dynamic network config in openstack (somewhat less useful in baremetal), so if this can help, ++
<harlowja> ya, smoser  (i believe on vacation until nexst week?) will have lots to say about that
<harlowja> ^ the BDFL of cloudinit, ha
<harlowja> i'm just BDFLjr :-P
<JoshNang> ahh :)
#cloud-init 2014-07-11
<kylestev> why isnât `blkid` mocked? basically, why do I have to run tests as root on debian? ;_;
<linuxgeek_> ping
<linuxgeek_> !ping
<linuxgeek_> hey harlowja_away 
<linuxgeek_> hi harlowja 
<harlowja> linuxgeek_ howdy
<linuxgeek_> harlowja, fairly well, thanks.
<linuxgeek_> re yesterday's thingy on the cloudinit. with a trusty ubuntu cloud image
<linuxgeek_> i was able to test the http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/examples/cloud-config-boot-cmds.txt
<linuxgeek_> and it works
<harlowja> woot
<linuxgeek_> there is another cloud image which is vmdk fedora 19
<harlowja> k
<harlowja> that one might not have cloud-init built in, or does it?
<linuxgeek_> and that is not able to reach the metadata service url http://169.254.169.254
<linuxgeek_> hence the userdata also does not work
<linuxgeek_> how is it that one image works while the other does not
<linuxgeek_> is this specific to how the image is built?
<linuxgeek_> yes it does have cloud-init in it
<linuxgeek_> i can see the cloud directory in /var/lib and cloud-init.log in /var/log
<linuxgeek_> so it is a cloud-init enabled image
<harlowja> it is likely more how the image was built, what firewall settings (?) could be by default on
<harlowja> that should be the only difference really
<linuxgeek_> ok
<harlowja> and those differences could stop it from contacting 169
<linuxgeek_> is this a internal url?
<harlowja> define internal ;)
<linuxgeek_> lol :-)
<harlowja> its a special IP
<linuxgeek_> internal = within the instance and not going outside
<harlowja> not usually
<harlowja> http://stackoverflow.com/a/7499376  is an idea of what it does
<harlowja> but it varies depending on the cloud u are on (the exact details vary)
<harlowja> openstack clouds do it differently than amazon and so-on...
<harlowja> so its 'special'
<linuxgeek_> yes o~s and ec2 is so different
<linuxgeek_> and i have o~s based cloud env
<linuxgeek_> i read http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AESDG-chapter-instancedata.html
<linuxgeek_> and the stackoverflow post.
<linuxgeek_> so it says ec2 knows what instance is making the request and it knows the info abt the instance 
<harlowja> right, openstack is the same
<linuxgeek_> so ec2 can return the metadata that is requested
<harlowja> right
<linuxgeek_> so this happens even before the request reaches the special ip?
<harlowja> right, it happens during vm building time
<harlowja> so my guess is if it works for one image and not another, then its an image thign
<linuxgeek_> ok
<harlowja> thats my gut feeling
<harlowja> i'm not sure what the solution is though ;)
<linuxgeek_> the flow is, the instance makes a request to the special ip based on the userdata injected to it
<harlowja> and before that happens, neutron, nova, make it possible for that special IP to be directed somewhere
<linuxgeek_> the openstack sees this request and knows which instance is sending this request
<harlowja> ya
<harlowja> and sends back some data
<harlowja> and that goes through the pipes and pops back up in the instance
<harlowja> the instance thinks it talked to a magic 169 address and was none-the-wiser
<harlowja> ^ pipe magic
<linuxgeek_> is it nova neutron which plays a role in routing the request to the special ip via the nova/neutron security groups and rules?
<harlowja> both afaik
<linuxgeek_> i'm planning to convert the trusty.img file [which works] to a trusty.vmdk and test
<harlowja> i believe nova is the endpoint and has the instance data, neutron sets up the plumbing
<harlowja> so both are involved
<linuxgeek_> i browsed through the o~s docs and the net too, i did not find details about how nova and neutron play a role in the metadata part. do you happen to know of a doc or a guide which talks about this?
<harlowja> i do not, but if u jump in #openstack-neutron  channel u might find someone who does
<harlowja> there should be something somewhere i would think
<linuxgeek_> have u used qemu-img convert?
<harlowja> i have
<linuxgeek_> sure, i'm on the neutron channel. i'll ask arnd for the docs
<linuxgeek_> cool, have u converted a .img file to .vmdk?
<harlowja> negative, i haven't touched vmdk stuff :)
<harlowja> i have done raw -> qcow2 and qcow2 -> raw
<harlowja> but not vmdk stuff
<harlowja> linuxgeek_ mestery on the neutron channel should be able to find something
<harlowja> i thought arnd was more in glance (maybe different arnaud)
<harlowja> or maybe someone else, haha
<linuxgeek_> haha, yep sure.
#cloud-init 2014-07-12
<harlowja> linuxgeek_ hmmm, no bites on the question in #openstack-neutron, hmmm, maybe next week ?
<harlowja> never can tell if people are gonna respond
<linuxgeek_> harlowja, yep never can say if ppl will respond
<linuxgeek_> hey harlowja_at_home, in continuation of testing different os for cloudinit windows is next in line.
<linuxgeek_> i asked this on openstack asking you in case you would know
<linuxgeek_> is windows a supported deployment in openstack with esxi hypervisor?
<harlowja_at_home> unsure :-/
<harlowja_at_home> i haven't done much with windows, i know some person at yahoo got it running under kvm
<harlowja_at_home> openstack + kvm
<harlowja_at_home> but likely u want to use https://github.com/cloudbase/cloudbase-init for that
<linuxgeek_> yep, Hyper-V, KVM, and XenServer/XCP is in the supported list per the openstack guide
<linuxgeek_> there is no reference of esx
<harlowja_at_home> ya, i'm not so sure, never done much with esx
<linuxgeek_> ok sure, thanks for replying though :-). enjoy the rest of the weekend
<harlowja_at_home> np
<harlowja_at_home> best of luck :)
#cloud-init 2014-07-13
<harlowja_at_home> ijw_,  guess u saw https://review.openstack.org/#/c/85673 right?
<ijw_> Yup, which I'm pleased to see and commented on
<harlowja_at_home> cool
<ijw_> He was taking this from the 'fix it in Openstack and they will come' approach but I think mainly cos he didn't know about the cloud-init group
<ijw_> Either way, looked pretty okay; could do with a bit of refinement but I think he'll get there
<harlowja_at_home> ya JoshNang found this channel (afaik thats the same josh, haha)
<ijw_> It's at rev 13 and apparently I'm the only person with a -1 on it, which makes me feel bad, but I still don't think it'll fly in its current form
<harlowja_at_home> smoser, when he gets back hopefully has some inputs on it
<ijw_> Er, and I note that I was nagging you on there, so not only did I see it, I've been talking to you in the comments ;)
<ijw_> (tl;dr: /etc/network/interfaces sucks and we shouldn't use it as a reference form)
<harlowja_at_home> :)
<harlowja_at_home> i had to write the /etc/network/interfaces translater in cloud-init so i know your pain ;)
<harlowja_at_home> ijw_,  there i added my comment, better format == good ;)
<ijw_> OK, and I've made the comments on routing (which is also throwing /etc/network/interfaces under the bus, but hey)
<ijw_> I'm not sure I have a full mental picture of what's needed either, to be honest, I've just noted many weirdnesses of the interfaces format over time, and I'm way out of practice with the redhat way of doing things, which may be precisely the same or completely different (ditto Windows)
<harlowja_at_home> eck multicast
<harlowja_at_home> multicast bad
<harlowja_at_home> * at least yahoo folks that i know say that
<harlowja_at_home> forget why they say that though, haha
#cloud-init 2015-07-06
<openstackgerrit> Merged stackforge/cloud-init: Speed up a slow test.  https://review.openstack.org/198348
<openstackgerrit> Daniel Watkins proposed stackforge/cloud-init: Move our TestCase in to the cloudinit.tests package.  https://review.openstack.org/198308
<smoser> Odd_Bloke, how did you know about / find the mock timejump ?
<Odd_Bloke> smoser: It's just a trick I've picked up before; all it's doing is using mock's side_effect (which lets you define fake return data).
<smoser> Odd_Bloke, duh. i didnt' see the function you added.
<smoser> and googled for that method name with no results.
<smoser> now it makes more sense :)
* smoser changed the topic of #cloud-init to: cloud-init || cloud-init 2.0 https://review.openstack.org/#/q/project:stackforge/cloud-init+status:open,n,z http://bit.ly/cloudinit-reviews
<Odd_Bloke> smoser: ^_^
<smoser> Odd_Bloke, just pinged in openstack-infra fro help on https://review.openstack.org/#/c/191081/
* smoser changed the topic of #cloud-init to: cloud-init || cloud-init 2.0 http://bit.ly/cloudinit-reviews-public http://bit.ly/cloudinit-reviews
<Odd_Bloke> smoser: Thanks. :)
<Odd_Bloke> (I've done that several times, so we'll see if you get more attention than me :p)
<Odd_Bloke> smoser: You're magic!
<Odd_Bloke> smoser: Could I pick your brains about https://bugs.launchpad.net/cloud-init/+bug/1470890?
<Odd_Bloke> smoser: I have a couple of ideas of how we could fix it: (a) implement a GCE special-case like the EC2 special case, or (b) add a DataSource.region attribute which can be used in the same way that DataSource.availability_zone is used now.
<Odd_Bloke> smoser: (b) is preferable, because it should be more generally useful, but at the moment we only pass AZ through to the relevant parts of the code, so it will involve a bit more rejigging.
<Odd_Bloke> smoser: Thoughts on those ideas (or additional suggestions) most appreciated.
<Odd_Bloke> smoser: Hmm, actually threading it through wasn't too painful.
<Odd_Bloke> Because it's pretty much only this code path that uses it.
<Odd_Bloke> Though it may make the backport more painful. :/
<smoser> yeah, its a convoluted path that gets there.
<smoser> i think someone else one of hte other cpc customers wanted something similar to find their mirrors.
<smoser> i'm fine if you clean it up. old behavior needs to be preserved is all/
<smoser> Odd_Bloke, for stuff like https://review.openstack.org/#/c/198263/ i'd be nice if you referenced the other change-id i think. or at least some place said, we'll be getting code coverage gating via http://....
<Odd_Bloke> smoser: Sure. :)
<Odd_Bloke> harlowja, oh guru of all that is OpenStack CI, why hasn't Jenkins unverified https://review.openstack.org/#/c/198308/ and queued it for gate?
<Odd_Bloke> (Is it because we already have a job waiting in the gate queue?)
<harlowja_at_home> yo yo
<smoser> hey. you still holiday ?
<harlowja_at_home> nah, have oslo irc meeting on monday mornings
<harlowja_at_home> so just finished that
<harlowja_at_home> i'm always on holidy man
<harlowja_at_home> lol
<harlowja_at_home> hmmm, why isn't that review on http://status.openstack.org/zuul/
<harlowja_at_home> thats weird
<harlowja_at_home> openstack CI sometimes is weird
<harlowja_at_home> bb though, heading into work lan
<harlowja_at_home> *work land
<harlowja_at_home> Odd_Bloke, maybe jump on #openstack-infra and ask around; seems weird
<Odd_Bloke> Oh, hmph, coverage fail.
<openstackgerrit> Daniel Watkins proposed stackforge/cloud-init: Don't consider version-specific code for coverage.  https://review.openstack.org/198263
<Odd_Bloke> smoser: You actually workflow-toggled the wrong change, but it doesn't matter because it would fail anyway because of coverage.
<smoser> hah. funny.
<Odd_Bloke> smoser: harlowja: Review of the above appreciated; it should fix the coverage failure (though, of course, Jenkins will let us know if that is the case).
<openstackgerrit> Daniel Watkins proposed stackforge/cloud-init: Move our TestCase in to the cloudinit.tests package.  https://review.openstack.org/198308
<harlowja> ok bacccck
<openstackgerrit> Merged stackforge/cloud-init: Don't consider version-specific code for coverage.  https://review.openstack.org/198263
<openstackgerrit> Merged stackforge/cloud-init: Move our TestCase in to the cloudinit.tests package.  https://review.openstack.org/198308
#cloud-init 2015-07-07
<openstackgerrit> Daniel Watkins proposed stackforge/cloud-init: Basic implementation of a reporting framework.  https://review.openstack.org/191144
<smoser> does this look right:
<smoser>  http://paste.ubuntu.com/11836765/
<smoser> what is line 2 there?
<harlowja> smoser ah
<harlowja> u have discovered googles notrious anti-parsing mechanisms
<harlowja> lol
<harlowja> ^ not kidding actually
<smoser> hm..
<smoser> odd. its actuallydocumented
<harlowja> let me see if i can find why they do that
<smoser>  http://review.cyanogenmod.org/Documentation/rest-api-changes.html
<harlowja> ya
<harlowja> let me see if i can find the reason, forgot it
<harlowja> afaik they do that in other json apis google makes to
<harlowja> * http://gerrit-documentation.googlecode.com/svn/Documentation/2.5/rest-api.html
<harlowja> 'The JSON response body starts with a magic prefix line that must be stripped before feeding the rest of the response body to a JSON parser:'
<harlowja> now let me see if i can find why
<smoser> magic
<harlowja> :-P
<smoser> harlowja, before i write something , do you know of an existing tool that allows me to :
<smoser>  gerrit-grab --branch=mybranch https://review.openstack.org/#/c/191144/
<harlowja> hmmm, i thought there was one that did that
<harlowja> maybe ask on #openstack-infra they might rememb ber
<harlowja> i'm pretty sure there is something :-P
<harlowja> sooo smoser  https://code.google.com/p/gerrit/source/browse/gerrit-httpd/src/main/java/com/google/gerrit/httpd/RestApiServlet.java?r=92edc8e4812acd9972bd213c60ea6eb25a52f545#50
<harlowja> there is your reason :-P
<harlowja> 'Garbage prefix inserted before JSON output to prevent XSSI.'
<harlowja> i remember that when i made https://github.com/harlowja/gerrit_view/ (i was like wtf is that line) so ya, thats how i know, lol
<harlowja> https://github.com/harlowja/gerrit_view/#czuul and https://github.com/harlowja/gerrit_view/#cgerrit is still cool
<harlowja> and u should use them
<harlowja> lol
<harlowja> * https://pypi.python.org/pypi/gerrit-view
<harlowja> and http://paste.openstack.org/show/352612/ u can share my config, lol
<harlowja> * for czull
<smoser> harlowja_, jblaire pointed out git review -b. i requesetd merge https://review.openstack.org/199225
<harlowja_> cool
<harlowja_> nice nice, i thought git review did it somehow
<harlowja_> or something like it, ha
<smoser> that is pretty smooth, harlowja_
<smoser> ijust used gertty to do a review and that is pretty smooth too.
<harlowja_> smoser cool
<harlowja_> i haven't tried gerrtty recently
<harlowja_> i'm ok with the webui :-P
<harlowja_> but zuul i got bored oneday and made a console-ui for it
<harlowja_> lol
<borei> hi all
<borei> im trying to use cloud-init with NoCloud datasource, but how can i point that user-data and meta-data are on the cdrom attached to VM ?
#cloud-init 2015-07-08
<smoser> Odd_Bloke, i think i said vivid to you
<smoser> but i meant wily. wrt webhooks reporter
<Odd_Bloke> Oh, OK.
<Odd_Bloke> Cool. :)
<Odd_Bloke> smoser: You're going to confuse zuul again by doing workflow +1 without two CR +2s. :p
<smoser> ?
<smoser> i did +2
<smoser> i guess i didnt understand what i did wrong.
<smoser> clearly.
<smoser> explain ?
<Odd_Bloke> smoser: Will do in a mo; in a meeting ATM.
<Odd_Bloke> Oh, confusing, I would have expected it to do nothing because there was only one +2.
<Odd_Bloke> smoser: My understanding is that zuul only examines a change for gate+merge when a workflow +1 happens.
<Odd_Bloke> When that happens, it checks if there are two code review +2s.
<Odd_Bloke> If there are, it queues for gate.
<Odd_Bloke> If not, it does nothing.
<smoser> why would it want *2* code review +2 ?
<Odd_Bloke> I thought that was what was required.
<smoser> hm..
<smoser> harlowja_, why does cgerrit just show "Connecting" for me ?
<Odd_Bloke> smoser: Looking at project-config, I see no requirements for code review at all.
<Odd_Bloke> So I'm confused in one of two ways. :p
<Odd_Bloke> harlowja_: Can you explain how stuff gets in to gate?
<openstackgerrit> Merged stackforge/cloud-init: Basic implementation of a reporting framework.  https://review.openstack.org/191144
<Odd_Bloke> \o/
<smoser> harlowja_, why does cgerrit just show "Connecting" for me ?
<Odd_Bloke> harlowja_: WAKE UP
<harlowja_> whatttttttttt
<harlowja_> smoser hmmm, cgerrit showing connecting, hmmm, might not work anymore, lol, or a bug or something :-P
<harlowja_> smoser ssh keys setup and stuff?
<harlowja_> cgerrit connects into the gerrit ssh event stream
<harlowja_> smoser run -v (which should enable more logging that may show something)
<harlowja_> https://github.com/harlowja/gerrit_view/blob/master/scripts/cgerrit#L800 should get configured then
<harlowja_> -v output.log (since the screen isn't useful)
<smoser> harlowja_, can you give me an example that works for you?
<harlowja_> let me get it running again, sure
<harlowja_> see if it works :-P
 * harlowja_ hasn't ran cgerrit in a while
<smoser> its petting cpu right now.
<smoser> ah. it came back. now waiting for events.
<harlowja_> ya, so that means whenever a review or review comment comes in via the ssh event stream
<smoser> so it just sits there and polls ?
<harlowja_> negative
<harlowja_> gerrit pushes
<harlowja_> *pushes events
<harlowja_> so its push based
<harlowja_> *vs the web ui, which is poll based
<harlowja_> hmmm, i got 'socket.error: [Errno 113] No route to host' let me see, ha
<harlowja_> ok finally got it to work, ha
<harlowja_> smoser "cgerrit -v output.log -i 500 -p 100 -k "/home/harlowja/.ssh/id_rsa" -u harlowja -s "review.openstack.org" -p "29418" "
<harlowja_> that worked for me
<harlowja_> so guess it still works lol
<smoser> it pegs cpu ?
<smoser> its eating 100% of one of my cpu
<harlowja_> not for me
<harlowja_> cpu 1.0%
<harlowja_> make sure the key works
<harlowja_> by trying 'ssh review.openstack.org -p 29418'
<harlowja_> that should emit something like http://paste.openstack.org/show/355953/
<harlowja_> i've got reviews up to make https://github.com/openstack-infra/gerritlib/ better, but they just sit there for months
<harlowja_> ^ which is the lib thats used to talk to gerrit
<harlowja_> cgerrit imho not super-useful, just neat to 'see' (c) stuff, but u can't review stuff on it (gertty was made for this, after cgerrit was made)
<harlowja_> it was a fun mini-project no-matter (and inspired the guy who made gertty to make that program, lol)
#cloud-init 2015-07-10
<harlowja> http://lists.openstack.org/pipermail/openstack-dev/2015-July/069240.html if u guys intersted
<harlowja> check it out :-P smoser ^ lol
<harlowja> or anyone else interested :-P
<harlowja> (my other work)
<smoser> i refuse to acknoledge you have been given permission to work on other things
<harlowja> :-P
<harlowja> lol
<harlowja> i mean, thats a slide-set about cloud-init
<harlowja> obviousslyyyyy
<smoser> oh. let me look then.
<harlowja> lol
#cloud-init 2016-07-11
<Odd_Bloke> linuxgeek_: Your full /var/log/cloud-init.log would be helpful. :)
<holser_> harlowja: could you have a look at https://code.launchpad.net/~sgolovatiuk/cloud-init/fix_mcollective ?
<holser_> I added tested per smoser requests
#cloud-init 2016-07-13
<harlowja> sure, looking over
#cloud-init 2016-07-14
<smoser> harlowja, i'd appreciate a look at https://code.launchpad.net/~smoser/cloud-init/trunk.lp1602373/+merge/300021
<smoser> and rharper ^ you too
<smoser> and then, since you're clearly looking for something to do,
<smoser>  https://code.launchpad.net/~smoser/cloud-init/trunk.net-improve-lo-dns/+merge/298035
<smoser> rharper, fyi, this is what our openstack (serverstack) ends up readnering in /etc/network//interfaces.d/50-cloud-init.cfg
<smoser> http://paste.ubuntu.com/19331646/
<smoser> it has the 'dhcp' and dns-addresses
<smoser> i think that is in the realm of funky behavior
<smoser> resolvconf seems to be ok i geuss. so maybe nevermind.
<rharper> smoser: re: server stack;  it's setting a global dns service entry in network_data.json as well as marking the interface as configured via DHCP;  Those aren't necessarily at odds; the DHCP response might not include a nameserver; or even the same ones.  I don't think we know enough to not emit a global dns entry if one or more of the interfaces is also dhcp;
<smoser> right.
<smoser> its not really at odds, but it is pointless. i thought we (and resolvconf) might end up adding 2 dns servers of the same, or otherwise getting confused.
<smoser> i was thinking it might be like our race
<smoser> with the routes
<rharper> not sure that it's pointless;  resolveconf should collect as many values and collect them
<rharper> no different than hooks in resolveconf.d
<rharper> which can add additional values
<rharper> in serverstack case, its setting a global value, and the DHCP response may include more
<rharper> I'd think it'd rather be on the openstack side to decide to emit dns_nameserver under the 'services' section or not depending on the type of network being deployed
<harlowja> smoser will check that out
<harlowja> i've got a big godaddy module for cloudinit i'm working on
<harlowja> i'll share it with u guys, it probably has some things that can be made generic for all folks
<harlowja> but its a refactoring of things they were doing on firstboot via some non-cloud-init local triggering of puppet from a git repo and ...
<harlowja> which is duping some of cloud-init and ...
<harlowja> so i'm just fixing that, lol
<rangerpb> hey harlowja smoser suggested i follow up with you on a python related question/problem if you have a sec
<rangerpb> pertains to -> http://bazaar.launchpad.net/~bbaude/cloud-init/azure_dhcp/view/head:/setup.py#L214
<rangerpb> harlowja, smoser and i are curious if there is a way to change the install dir where line 217 ends up
<harlowja> smoser suggested me :-P
<rangerpb> yeah, guilty as charged
<harlowja> ah, hmmm
<harlowja> good question
<rangerpb> basically we'd like that line 217 to not end up /usr/bin
<harlowja> ya, i wonder if there is a way
<harlowja> i don't know one off the top of my head
<harlowja> but there probably is
<rangerpb> i havent been able to find anything either
<harlowja> if u want, can u jump on #openstack-oslo
<harlowja> lifeless (who is in NZ) has a bunch of knowledge about this stuff
<harlowja> he might know a way
<rangerpb> smoser, ill pursue that ... what do you want to do in the interim ?
<smoser> harlowja, would there be some way to just move files after setup.py ran ?
<harlowja> mv?
<smoser> ie, just hackily mv ./usr/bin/that-file ./usr/lib/cloud-init/other-file
<smoser> well, withink the context of setup.py thugh, but outside the context of entry points
<harlowja> gotta be some way here
<harlowja> did u guys try   --install-scripts
<harlowja> python setup install --help
<harlowja> shows that might work?
<rangerpb> i think the rub is he wants cloud-init into /usr/bin but not others
<harlowja> hmmm
<harlowja> well thats weird
<harlowja> lol
<harlowja> smoser might be crzy
<harlowja> lol
<rangerpb> we are trying to avoid flooding /usr/bin but like the benefit the entry_point provides as it injects the right python version, etc
<smoser> rangerpb's other idea was to have cloud-init dhclient-hook
<smoser> which i'm not really opposed to.
<smoser> and then its '--help' can even say "you probably dont want to call this"
<rangerpb> harlowja, is there a way to replicate the entry point upside but not call it out as an entry_point in setup.py ?
<harlowja> not sure
<harlowja> this is in setuptools/easy_install and all that which i don't know to much about how it works :-P
<harlowja> u may find some folks in #pypa (the pip people who also do setuptools and stuff afaik are in here)
<smoser> harlowja, thats why we have you here.
<harlowja> this gets into there territory, lol
<harlowja> ha
<smoser> if people ask me a question, they expect me to say 'go ask someone more knowledgeable'
<smoser> but when they then ask you, you're supposed to know.
<harlowja> delegation is key, ha
<harlowja> i delegate to :-P
<harlowja> pypa folks though should really know
<harlowja> i'm just a country-bumpkin who doesn't know anything
<harlowja> lol
<harlowja> smoser https://gist.github.com/harlowja/781a010bbe2634982dbb0878982852bc if u get bored
<harlowja> https://gist.github.com/harlowja/781a010bbe2634982dbb0878982852bc#file-cc_godaddy-py-L733
<harlowja> the submodule idea there would be nice to offically support
<harlowja> i'm also seeing https://gist.github.com/harlowja/41d557bb2c9dbd38a1c3c75bd834a0b6 :-/
<harlowja> which seems odd
<harlowja> `chunk_size=1024` is the default
<harlowja> unsure why that would be invalid, lol
<harlowja> thats during reading '/sys/class/net/eth0/carrier'
#cloud-init 2016-07-15
<smoser> harlowja, i'm confused
<smoser>  python -c 'import argparse; print(argparse)'
<smoser> <module 'argparse' from '/usr/lib/python2.7/argparse.pyc'>
<harlowja> ya, but pkg_resources is not finding it :-/
<harlowja> and/or redhat didn't register it right or something
<smoser> ah.
<smoser> now i see.
<smoser> python2.6
<smoser> its new in 2.7
<harlowja> ya
<harlowja> i think redhat probably messed up there argparse package
<harlowja> or forgot to register it with pkg_resources or something
<smoser> ?
<harlowja> https://bugs.launchpad.net/cloud-init/+bug/1603533
<harlowja> seems like 'needed = self.resolve(parse_requirements(requirements)) ' wouldn't fail if argparse was registered
<harlowja> but meh, also fixes it by just not listing it as a dep in 2.7+
<harlowja> smoser other question
<smoser> right.
<harlowja> any opposition to https://gist.github.com/harlowja/50a66fb9bed6e0897f1691eae3107f01
<harlowja> on cent7 there is like no content (at least in my cloudinit) in /var/log/cloud-init.log
<harlowja> and journalctl doesn't have anything
<smoser> so it actually fails to go to syslog ?
<harlowja> think so
<harlowja> now i did see the following in a redhat patch
<harlowja> https://gist.github.com/harlowja/275e35b41cd268d869b7b503d7f654d7
<harlowja> so that might fixs it
<harlowja> don't think that one was upstreamed :(
<harlowja> let me just try that fix; see what happens
<smoser> hm.
<harlowja> (gotta get these damn redhat folks to stop having secret patches, lol)
<harlowja> bbiab though, lunch
<harlowja> smoser whats weird though is that even if i use that patch (which does help) it still doesn't write debug stuff to that log file
<harlowja> so then it makes me question the value of that syslog stuff in the first place
<harlowja> or maybe the sysconfig stuff is just busted, lol
<harlowja> *syslog i mean
<harlowja> for example when working on my godaddy module, this is all the debug in the logs, lol
<harlowja> https://gist.github.com/harlowja/56b2b5109bbbd4b3441cda43a5617107
<harlowja> not so useful :-P
<harlowja> i'd rather have old-janky file that actually works :-P
<harlowja> soooo, if we defaulted to just the file stuff smoser would that be ok
<harlowja> at least its dependable :-P
<harlowja> i'll probably do that locally anyway, pita having this thing not show anything
#cloud-init 2017-07-10
<niluje> blackboxsw: hi, any idea when smoser comes back?
<Wulf> 19:28:14 <@smoser> blackboxsw, i'm out the rest of the week thoguh
<Wulf> That was quite some time ago
<Wulf> Jul 03
<Wulf> niluje: My best guess is 5 hours from now.
<niluje> Wulf: yeah he told me he was leaving for a week :p
<niluje> I wanted to know if he could give a look to the PR introducing the scaleway datasource
<blackboxsw> niluje: yeah should be in this week.
<smoser> niluje, was that a ping for me ? /me is here now
<smoser> (we have a 'cloud-init status meeting' in 67 minutes.. 4:00 GMT)
<niluje> :D
<niluje> hi smoser :) yeah, I was just wondering if you had the time to get a look to my mr
<blackboxsw> smoser
<blackboxsw> welcome back.
<blackboxsw> ..."I wanted to know if he could give a look to the PR introducing the scaleway datasource"
<smoser> niluje, i will today.
<niluje> smoser: oh, thanks a lot :)
<smoser> hi... i'm going to wait about 2 more minutes before starting a meeting here just to not have it finished before a late comer arrived.
<smoser> i've written down a agenda like thing at https://public.etherpad-mozilla.org/p/cloud-init-meeting
<dpb1> hi smoser, here
<smoser> ok... 4 minutes late.
<smoser> Hello everyone.
<smoser> This is the first 'cloud-init status meeting' (for lack of a better name)
<blackboxsw> hello hello
<smoser> this was announced / asked for feedback on time at https://lists.launchpad.net/cloud-init/msg00091.html
<powersj> o/
<smoser> and this is what we came up with.
<smoser> I'm working off a very informal agenda at https://public.etherpad-mozilla.org/p/cloud-init-meeting
<smoser> # Introduction
<smoser> ^ see above... very informal, let people raise issues and discuss things
<smoser> Goals for the meeting
<blackboxsw> sounds good. we can see what themes come up and build from there
<smoser>  * Not waste (much of) people's time
<smoser>  * Function as 'office hours' for 30 minutes after... smoser and others will be present to talk on irc.
<smoser> blackboxsw, yeah.. i am hoping that the 'office hours' portion will be used by people to help communication.
<blackboxsw> +1
<smoser> # Recent Changes / Highlights
<smoser> here... i should have a list of "highlights". but I do not have htat at this point.
 * smoser looks for a link
<smoser>  1. https://git.launchpad.net/cloud-init/commit/
<smoser>    You can always look there and see recent commits to trunk.  We always try to write readable git-style commit messages with links to bugs for more information.
<smoser>  2. https://lists.ubuntu.com/archives/ubuntu-server/2017-July/007559.html
<blackboxsw> Any recent SRU lands that we want to highlight (per 1 week ago)
<smoser>   Josh (powersj) has been writing some of these things and sending to the ubuntu server mailing list as above.
<smoser>   I guess it would be good to take the cloud-init portion and send to the cloud-init list also.
<dpb1> smoser: +1
<dpb1> smoser: I'll do that with the next one
<smoser> a highlight there... powersj will be presenting at Debconf https://debconf17.debconf.org/talks/164/
<blackboxsw> [ACTION  -smoser] Send cloud-init portion of the ubuntu-server updates to the cloud-init list.
<smoser> next topic
<smoser> # In Progress Development / Highlights
<smoser>  * Merge Proposals: https://code.launchpad.net/%7Ecloud-init-dev/cloud-init/+git/cloud-init/+ref/master/+activereviews
<smoser>  * Trello Board: [https://trello.com/b/hFtWKUn3/daily-cloud-init-curtin]
<smoser>  * Bugs: https://bugs.launchpad.net/cloud-init
<smoser>  * AWS ipv6 support: http://pad.lv/1639030
<ubot5> Launchpad bug 1639030 in cloud-init (Ubuntu) "Configure networking based on EC2 metadata source" [Medium,Confirmed]
<smoser> The last there is what blackboxsw is looking at right now..
<smoser> we're wanting to make cloud-init able to configure ipv6 networking on AWS.
<smoser> (aws has ipv6 support for a while now, but cloud-init hasn't taken advantage of it)
<smoser> Also, it looks like powersj added
<smoser>      Opensuse builds: https://progress.opensuse.org/issues/19712
<dpb1> that's me
<smoser> ah.
<blackboxsw> was going to start adding a doc here https://docs.google.com/document/d/1VWnp-29_UM_LGr1h_CTu14UmLJFu_Vu6yzZTne5x-Qw/edit?usp=sharing
<robjo> Bew aware the AWS DHCPv6 server implementation is "broken"
<robjo> s/Bew/Be/
<blackboxsw> I'll make sure the doc is publicly readable. as I think it might not be currently
<smoser> can you elaborate?
<robjo> when the server receives options it does not understand in the original request for a lease it will drop the request on the floor and you'll never get an answer
<smoser> robjo, ? in my limited knowledge it seemed to work at least enough for 'dhclient -6' to work
<smoser> robjo, oh. thats good to be aware of.
<blackboxsw> ahh interesting.
<blackboxsw> thanks! will test out what options we intend on sending to ensure we aren't falling on our face
<smoser> so, yeah, we're working on getting cloud-init repos up for open suse (as dtb pointed out above) and have also recently got builds in fedora's copr
<robjo> yes, dhclient -6 works, dhclient only send the minimal set of info to get the lease. It works because that's the client that's in amazon-linux ;)
<dpb1> robjo: is there a know "good set" of options to send on the request?
<smoser> the goal for both will be to have "trunk" builds to more easily try for users.  https://copr.fedorainfracloud.org/coprs/g/cloud-init/cloud-init-dev/
<robjo> we only figured out what we cannot sent ;)
<smoser> robjo, thats really helpful. thank you.
<dpb1> robjo: thx
<smoser> Anyone have anything else here ?
<smoser> if not we'll move onto 'open discussion', and then 'office hours'
<robjo> Well, thanks for setting this up and getting started with OBS, I no longer feel so alone :D
<smoser> # open discussion
<blackboxsw> us too :)
<dpb1> is open discussion and office hours the same thing?
<dpb1> :)
<smoser> yeah, i was goin gto comment to that affec.t
<smoser> so at this point lets consider this meeting done.
<smoser> if you have any questions, please feel free to ask.
<smoser> and more generically, always feel free to ask.
<blackboxsw> do you think we should invite meetingbot or some other bot to this so we record previous cloud-init meetings in the future?
<smoser> another link for people's usage
<smoser>  this channel is logged vi irclogs.ubuntu.com
<smoser>  https://irclogs.ubuntu.com/2017/07/10/%23cloud-init.html
<ajorg> discussion on the finalpoints here? https://code.launchpad.net/~ajorgens/cloud-init/+git/cloud-init/+merge/325857
<smoser> so you can look back if need be.  I often use that to post links in bugs.
<smoser> blackboxsw, yeah, we could/shoudl have a meetingbot for this.
<blackboxsw> there we go thx smoser
<dpb1> blackboxsw: +1, want to figure out how to do that?
<blackboxsw> dpb1: I don't mind, I wanted to get another bot in here for cloud-init branch lands, new merge proposals
<blackboxsw> etc.
<blackboxsw> just to raise the communication about what's happening in cloud-init
<dpb1> blackboxsw: ok, meetingbot should be easy.  probably a file with channels it joins somewhere
<blackboxsw> [ACTION - bbsw] get a bot running in channel
<smoser> ajorg, on that .. i can comment there. i suspect it did "just work" in python 2. but busted in python3.
<smoser> i *think* writing in binary mode should be ok to change to.
<ajorg> because of the utf8 stuff?
<smoser> as the input i *think* will be crlf adjusted.....
<ajorg> should be, by the mime decoder, right?
<smoser> i dont want to break something where a user fed 'crlf' end lines and that worked before due to writing in text mode
<smoser> ie, windows user put something in and python 'did the right thing' before but would break if we had written in binary mode.
<smoser> i'm not sure.
<smoser> but yeah, as you said above. yaml would handle it to i think.
<smoser> ajorg, ... in the aws ipv6 path, blackboxsw had some questions maybe you could help us out with
<smoser>  https://docs.google.com/document/d/1VWnp-29_UM_LGr1h_CTu14UmLJFu_Vu6yzZTne5x-Qw/edit?usp=sharing
<ajorg> hmm. for it to be executed it has to have a text/ mime type, so maybe allowing binaries there is a bad idea anyway (can't actually be used correctly)
<smoser> i had always assumed that the metdata service was put on a link local address (169.254.169.254) to in fact be "link local"
 * ajorg reads
<blackboxsw> I see folks can read this. i'm mid typing it up but you have all the context below in the doc
<ajorg> interesting
<ajorg> I can go back to the VPC team and see if we can get good answers for that. better would be to cut as a ticket via the canonical internal channels so David can track it.
<blackboxsw> basically it seems AWS network timesout when trying to reference the link local 169.254.169.254 metadata service. We're thinking it's because the either that the metadata service rejects requests from an IP not allocated to the specific instance (like a static 169.254.169.200 address being dropped).  Or potentially because the metadata service response might be coming from the router/gateway address.
<ajorg> In Amazon Linux I think we create a static route on the first network device, so that whatever else happens the 169.254 address should work
<blackboxsw> but that's just a hypothesis from our side. :)
<blackboxsw> thanks ajorg for the help
<ajorg> sounds like a good hypothesis to me.
<ajorg> I'll reach out to someone here. Please let me know if you cut a ticket about it so I can subscribe to it.
<blackboxsw> my next pass just after this meeting will be allocating the link local 169.254.169.10 to eth0 and setting up a route that'll point at the default known gateway which was given via dhcp  and we'll see if we get a response.
<blackboxsw> +1 ajorg will do
<smoser> ajorg, a static route to where ?
 * ajorg checks code to be sure
<ajorg> correction...
 * ajorg finds the hardware token that will let him check the code...
 * smoser launches a amazon linux ami
<robjo> Amazon-linux has a bunch of scripts that handle all the route setup etc. Unfortunately the scripts are not consumable by other distros as not published via GitHub or other repo. In an effort to get multiple interface support working properly across clouds we are working on networking scripts that will eventually show up on GitHub, hoping by the end of the week
<robjo> hopefully that will be useful to others
<ajorg> 169.254.169.254 0.0.0.0         255.255.255.255 UH    0      0        0 eth0
<blackboxsw> ooh good info robjo. Is there an existing amazon github repo somewhere that we should subscribe to?
<smoser> hm...
<smoser> $ ip route
<smoser> default via 172.31.0.1 dev eth0
<smoser> 169.254.169.254 dev eth0
<smoser> 172.31.0.0/20 dev eth0  proto kernel  scope link  src 172.31.2.240
<ajorg> Just a static route straight to that one host.
<ajorg> (from the IP given by DHCP)
<ajorg> yeah, like that
<smoser> i dont *thinK* htat should fix it for us.
<ajorg> blackboxsw: not an official one, yet, and unfortunately it's specific to Red Hat / Fedora style network scripts.
<robjo> no, I have asked repeatedly and Dave has been trying to move the needle at AWS but there appears to be no interest for AWS to maintain their scripts in GitHub thus we are doing our own thing now, I ran out of patience
<smoser> currently we do basically:
<smoser>  ip addr add 169.254.<rand>.<rand>/16 dev eth0
<smoser>  ip link set dev eth0 up
<smoser> (where 'rand' is random.randint(1, 168) and random.randint(0, 255))
<blackboxsw> understood ajorg  & robjo
<ajorg> config for that route isn't in those scripts, it's in a flat file:
<ajorg> http://paste.ubuntu.com/25062274/
<ajorg> smoser: that's probably not going to work, i suspect you're right that it's checking the source IP and dropping anything not expected
<smoser> ajorg, so your route above... that is *really* wierd
<ajorg> IIRC this is actually why we use a static route, because if it comes from eth1 instead of eth0 it will also be an unexpected address.
<smoser> in that it says "don't send this to the gateway"
<smoser> right?
<smoser> which i'd have thought would break stuff.
<ajorg> correct that it says that, less correct that it's weird
<smoser> well, its wierd because we have no such route
<ajorg> it says that packets to that host should just go straight there
<ajorg> right, that is a bit weird
<ajorg> fair enough
<smoser> it seems wierd to me though. because i think i have the following scenarios
<ajorg> packets headed to the gateway probably get captured, but if i knew for sure i'd still be bound by the "we don't talk about internal implementation details" thing
<smoser> a.) ubuntu current default behavior.... requests to MD address get sent to the default gateway
<ajorg> by captured i mean they never actually reach the gateway
<smoser> b.) amazon linux default behavior... requests to MD explicitly do not go to gateway
<smoser>   but have a 172.X.X.X (or 10.0.X.X) source address, as provided by dhcp
<ajorg> IIUC all that's needed is for the source address to be correct and possibly for the packet to be on the correct interface
<smoser> c.) our attempt at ipv4 link local... requests to MD not through the gateway, but with a 168.254.X.X source address.
<ajorg> so if the source address is not what's expected, it doesn't get caught by whatever is routing it to the instance metadata service
<ajorg> if it is what's expected it does get routed
<ajorg> we added the static host route so that the source address will always be correct
<ajorg> do you know if you run into problems if you have multiple interfaces? (ipv4)
<smoser> (in 'b', it seems somewhat wrong.. as those are "martian" packets i think... per linux rp_filter)
<ajorg> (static route forces it onto eth0, which has the correct address)
<smoser> ajorg, yeah, multiple nics arent really addressed yet. and it wouldnt surprise me if we hit things like that.
<ajorg> someone ported our ec2-net-utils scripts to ubuntu and posted them to github... lemme see if i can find that
<smoser> ajorg, well...
<ajorg> https://github.com/ademaria/ubuntu-ec2net
<smoser> i think what we're agreeing on is that the source address has to be the ipv4 that is handed out by the dhcp server on "eth0".
<ajorg> correct
<smoser> and (to my knowledge) the only way to know that is in fact dhcp
<smoser> which we were hoping to avoid.
<ajorg> ah
<ajorg> reason to avoid?
<smoser> (as with dhclient... it will have side affects)
<ajorg> I can imagine reasons, but I might be wrong
<smoser> the goal is to bring up the networking required to see the metadata service, read the configuration there, render /etc/network/interfaces (or appropriate configuraiton per distro) and then let the distro bring up the interfaces.
<ajorg> ahhh... that makes sense.
<smoser> dhclient runs scripts and such on 'up', and leaves state in /var/lib/ and other things...
<ajorg> I totally get how that's desirable.
<ajorg> And that works on other clouds?
<smoser> we may end up having to resort to that, but, its less than ideal.
<smoser> well on digital ocean, they have ipv4 link local working correctly :)
<smoser> and it *feels* to me like a platform change coudl occur on amazon to support doing the right thing if the source address is a 169.254.X.X address
<blackboxsw> as in, no default gw needed, just the static link loval IPv4 addr and metadata responds.
<ajorg> it's worth pushing for, i think.
<smoser> blackboxsw, yeah... so ajorg says we dont need the gateway
<smoser> we need the source address.
<smoser> which i said was "wierd" :)
<blackboxsw> ahh right sorry crossed my wires with his above
<smoser> i'm pretty sure that is a "martian" packet
<ajorg> bbiab, trying to find the right team to ask about this internally
<smoser> ajorg, thank you for the discussion.
<blackboxsw> thx ajorg . so time check smoser?
<smoser> robjo, thanks for being here.
<blackboxsw> shall this wrap up the meeting?
<smoser> lets say meeting ended.
<blackboxsw> great discussion folks.
<robjo> smoser: Thanks again for getting this started
<smoser> i think the difference i had between "open discussion" and "office hours" was just to have one be part of "meeting" and one just "people are around"
<smoser> (in my head at least)
<smoser> ie... we wrap up meeting in ~ 20 minutes or something, but office hours remain... someone can know that they can come and just ask questions and have some expectation  that someone will respond.
<smoser> anyway.
<smoser> #end meeting
<smoser> thanks everyone
<smoser> the next meeting will be on the 24th.
<blackboxsw> yes that makes sense smoser
<blackboxsw> thanks again
<powersj> smoser: I just submitted pylxd update merge and I did submit a fix for artful tests last week
 * ajorg returns
<ajorg> so new theory on the link-local thing (without having found the right sources internally yet)...
<dpb1> ajorg: (btw) will you be there on wednesday?
<ajorg> yup
<dpb1> great
<ajorg> it may be that the problem is simply that it's too soon. it takes a little while for the network stack (on the cloud side) to be fully ready, and your packets might be dropped in the mean time
<ajorg> so if you want to try again w/ link-local but add a big sleep before it, you might find that that works
<ajorg> obviously that wouldn't be desirable, but it would help explain things
<blackboxsw> hmm ajorg how big a sleep, my wget was retrying for 10 mins
<ajorg> gah, well that's plenty big
<ajorg> was this with other networking unconfigured?
<blackboxsw> heh
<blackboxsw> this was a single eth0 static config @ 169.254.71.10 or something close to that IP.
<ajorg> k
<blackboxsw> around line 150 http://paste.ubuntu.com/25036317/ any lines prefaced with Chad were my additional network setup ( I tuned the retries down to 2 on the wget in this run though.
<blackboxsw> as the 20 retries and 15 min default timeout were too larege
<blackboxsw> *too large
<ajorg> blackboxsw: I've added more details on the ticket here and CC'd David Duncan on it. We'll try to get you some answers. Meanwhile you might be stuck with having to rely on DHCP.
<smoser> blackboxsw, http://paste.ubuntu.com/25062706/
<smoser> results of:
<smoser>  addr:
<smoser>  addr: http://paste.ubuntu.com/25062713/
<smoser>  link-local: http://paste.ubuntu.com/25062728/
<smoser> in my mind that basically proves ajorg right. you have to have the "correct" source address.
<ajorg> I like being right, except when it makes someone sad.
<smoser> well, in blackboxsw and my assesment before talking to you, we thought we'd need to send through the gateway, not have the right source address.
<smoser> so you helped diagnose quicker.
<ajorg> glad to be of service then
<smoser> ajorg, so. https://code.launchpad.net/~ajorgens/cloud-init/+git/cloud-init/+merge/325857
<smoser> current behavior in cloud-init is t fail if no '#!' at the front.
<smoser> changing to your suggestion "defaults" to using /bin/sh if there is no shebang (per the shells' implementation of "read the first 2 bytes")
<ajorg> right
<smoser> i really think it better to make convince / teach people to use '#!'
<ajorg> okay
<ajorg> i'm actually totally okay with dropping this one
<smoser> (and i'd like to support binaries... write_files definitely should support binaries, and almost certainly did in python2 ... thats a bug we need to fix)
<ajorg> (or carrying a local patch if our customers don't want to learn
<smoser> just failing with no idea why isn't great...
<smoser> so we could try without shell=True
<smoser> and on exception look at the file, and if it did not start with '#!' warn
<smoser> or rather just adjust the warning message to say 'did you mean to start your data with "#!/bin/sh"?'
<smoser> i realize that cloud-init warnings are probably not read by most people, but at least that gives a hint
<ajorg> I'm more inclined to giving the warning. It requires peeking, but that's not exactly terrible.
<smoser> and only shows a small bit of noise when in the case that the user gave a binary that just exited non-zero
<ajorg> right
<smoser> the additional noise just in the message above, as if you sent '/bin/false' it would still WARN that it exited non-zero
<ajorg> or isn't there a specific exit code for "I couldn't even try to execute that"?
<ajorg> 126?
<smoser> http://paste.ubuntu.com/25062793/
<smoser> you were right.. it should raise a OSError errno 8
<ajorg> cool
<smoser> you want to re-work this to handle that ?
<ajorg> I'll have to write a note on the PR for now, have to do my "real" job for a while today.
<ajorg> but yes
<smoser> yeah. thanks. i'll comment in the pr
<ajorg> ah, great. thank you!
<blackboxsw> smoser: nice work on the validation there w/ the route added
<blackboxsw> ok. so you validated w/ the small script that link-local fails. ok
<blackboxsw> I had looked at the wrong paste for a second and thought you had it succeeding
<smoser> right. fails with ipv4ll address
<smoser> passes with the "right" source address.
<blackboxsw> yeah so smoser, where shall we go from here for today?  You going to continue working on DataSourceAWS ipv6 support using dhclient while we determine if aws can suppport link-local source addresses?
<blackboxsw> *while we wait to see if AWS can add support for link-local source addresses
<smoser> blackboxsw, you can take a look at the dhclient path.
<smoser> blackboxsw, but if the thing actually works..
<smoser>  http://code.activestate.com/recipes/577649-dhcp-query/
<smoser> does seem consumable
<blackboxsw> right smoser and the primary reason we'd use the python client instead of dhclient in ephemeral is to avoid the side-effects right
<smoser> correct
<smoser> and to get cross-distro
<smoser> basically it would be (it seems) more distro agnostic
<blackboxsw> makes sense to me
<blackboxsw> ok figured out how to do a simple dhcp discover using scapy. not sure if that dependency is too large for cloud-init though
<dpb1> smoser ^
<dpb1> oh
<dpb1> he gone
<blackboxsw> I'll see what it pulls in. but it's a fairly simple python security library that only strictly depends on python.  lots of suggests for the package. but anyway. I'll have about a 20 liner that should give is a tiny python dhclient which can perform our dhcp discovery and without any side-effects
<blackboxsw> just tested locally and had no problem identifying my dhcp server as well as using a obtaining a new dhcp ip. problem I
<dpb1> blackboxsw: sounds nice
<blackboxsw> see with this and with the link smoser sent was that it requires an IP configured on the interface to start with in order to use the socket to broadcast the dhcp discover packaets. feels like we are putting the cart before the horse as we don't truly know what I'
<blackboxsw> see with this and with the link smoser sent was that it requires an IP configured on the interface to start with in order to use the socket to broadcast the dhcp discover packaets. feels like we are putting the cart before the horse as we don't truly know what IP to give the instance on a given aws private net
<dpb1> better to not maintain our own code too
<blackboxsw> right
<blackboxsw> make it pretty much a 5-liner for just a simple dhcp client request http://paste.ubuntu.com/25064349/
<blackboxsw> but I'm still missing how we bring up the initial  interface so scapy can talk on the socket as the udp broadcast message isn't permitted w/out some viable(up) interface. will have to think on this over dinner
<Putti> Is anyone here from canonical that could check where my contributor agreement has lost? Launchpad link to my account: https://launchpad.net/~j.kylmala
<larsks> Putti: I suspect most of the canonical folks are going to be here us business hours (central/mountain/pacific time, I think).
#cloud-init 2017-07-11
<Putti> ok, I have time :)
<smoser> powersj, https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/327130
<powersj> smoser: yep
<smoser> "with the upgrade to 2.15"
<smoser> what upgrade to 2.15 ?
<powersj> fixes our pylxd issue with lxd >= 2.15
<smoser> so i can run this on artful again ?
<powersj> yes
<smoser> see updated commit message
<smoser> what do you think?
<smoser> the versions are too close and get confusing (i was confused 2.15 -> 2.2.3 -> 2.2.4)
<smoser> ie, pylxd versions are close to lxd versions and thus confused smoser's poor little brain
<smoser> also a question on https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/326951
<Putti> smoser, are you able to see what is the status of my canonical contributor agreement? Launchpad: https://launchpad.net/~j.kylmala
<smoser> Putti,
<smoser> if you've signed, then eventually you'll be put into the group
<smoser>  https://launchpad.net/%7Econtributor-agreement-canonical/+members
<smoser> and you will show up there
<smoser> however, the process is manual.  you sign, it goes to someone, they read, add you to the group
<smoser> so there is a lag of a couple days i think normally.
<smoser> what i do is
<Putti> I did it like two weeks ago, and have seen new members being added in that time, so I wonder if it has lost somewhere?
<smoser> a.) look at that group
<smoser> b.) ask the person
<smoser> if you *say* you've signed it, then i
<smoser> i'm willing to accept it and assume that you'll appear in the group eventually
<Putti> I send the group admins also message via the launchpad but no answer
<smoser> because if you're willing to lie to me, then you would just as well have filled the thing out and lied there.
<smoser> i'll ping though and see if i can't find a response for you.
<smoser> was there a merge proposal you wanted me to look at ?
<Putti> smoser, thanks
<Putti> I send some patches, but I guess I should create a branch? :D
<Putti> I have to look into how that is done.
<smoser> Putti, yes.
<smoser> the HACKING.rst should explain. if you have questions, please feel free to ask.
<smoser> http://cloudinit.readthedocs.io/en/latest/topics/hacking.html
 * niluje pokes smoser
<smoser> niluje, believe it or not i do have a tab open
<niluje> ahah
<niluje> smoser: if you're willing to lie to me, then you would just as well....
<niluje> tell me you have the tab open
<niluje> :D
<niluje> ok joke, no rush :)
<smoser> :)
<smoser> thats right.
<smoser> niluje, so questions here ?
<smoser> err..i 'll review in linel there.
<niluje> smoser: I actually have to go in a few minutes
<niluje> but if you have some time here I'm free tomorrow to answer your questions here if you prefer
<niluje> -here
<smoser> niluje, i'll reviw
<smoser> in the merge propsoal
<smoser> thanks for patience
<niluje> ok
<niluje> I hope it'll be ok this time :-)
<niluje> see you tomorrow, thanks again
<smoser> hm..
<smoser> you went through requests.packagse.urllib3.connection
<smoser> to get your HTTPConnection
<smoser> i wonder if that is intentionally hidden.
<blackboxsw> :q
<blackboxsw> oops
#cloud-init 2017-07-12
<niluje> thanks smoser :) let's review your review
 * Putti got added to the contributor agreement group! :)
<niluje> smoser: I still don't understand anything to the launchpad interface, but all your comments have been addressed :)
<niluje> I cant rebase my commits into one if you want me to
<Putti> oh, the tox tests take like 5-10 min to run :/
<Putti> they are contacting the outside world?
<Putti> whee, first merge request done. Was kinda tedious â I have to look up if I can integrate it somehow with my command-line flow.
<niluje> Putti: if you are doing some hacking/debugging
<niluje> run tox -e <env>
<niluje> like tox -e py27
<niluje> or to run a single test tox -e py27 -- cloudinit/unittest/xxx.py
<niluje> or tox -e py27 -- cloudinit/unittest/xxx.py:Classname or Classname:test_name
<Putti> ok
<niluje> or call nose directly: .tox/py27/bin/python -m nose --tests cloudinit/unittest/xxx/.py -x -s
<niluje> tox has always been super slow for me for everything
<Putti> niluje, but maybe huge part of the slowness is just that it contacts all the web servers?
<Putti> for me the test tries to contact google but fails
<Putti> maybe the slowness is because of this: "FAIL: test_get_data_returns_false_if_not_on_gce (tests.unittests.test_datasource.test_gce.TestDataSourceGCE)"
<Putti> I will link the full log, a minute...
<Putti> http://paste.debian.net/plain/976118
<Putti> The ISP I am currently connected redirects to their own web page if the resource is not found, maybe the unit test gets confused by this?
<niluje> lol
<niluje> anyway IMO a unittest should never hit a network resource
 * Putti agrees with that
<niluje> but I'm pretty sure they don't
<Putti> I guess a temporary fix is for me to use different dns.
<niluje> which env is it? py27?
<Putti> correct
<Putti> py27
<niluje> sec
<niluje> yeah nevermind
<niluje> but I guess if you find the place where a network resource is hit, cloud init maintainers will be interested in a fix
<Putti> not too hard to find: cloud-init/tests/unittests/test_datasource/test_gce.py :)
<Putti> I might look into how to fix it
<powersj> Putti: if you find a test is escaping out to the network please file a bug. However, I am fairly certain nothing attempts to get out.
<powersj> Those calls should be mocked, as those metadata services should not available outside the cloud providers network
<Putti> it seems like the GCE test mocks the request but somehow it lets the dns request pass through.
<blackboxsw> Putti: I can peek at that today too if you don't end up tackling it
<Putti> blackboxsw, thanks, I'm pretty new to cloud-init and also I have very minimal experience with unit tests in python so if you just have time it would be nice. Though, I'm so curious about this problem, so we will see if I figure out the fix (in case one is needed) for this before you.
<blackboxsw> Putti: +1 and welcome to the club, I'm relatively new to cloud-init too.  Putti appreciate your efforts fire off any questions you might have and we can field them. to speed unit test debug cycles you can put a pdb.set_strace() over in the unit test you care about and tox -e py3 -- --tests tests.unittests.test_datasource.test_gce.TestOpenStackDataSource -s
<blackboxsw> Putti: +1 and welcome to the club, I'm relatively new to cloud-init too.  Putti appreciate your efforts fire off any questions you might have and we can field them. to speed unit test debug cycles you can put a pdb.set_strace() over in the unit test you care about and tox -e py3 -- --tests tests.unittests.test_datasource.test_gce.TestDataSourceGCE -s
<blackboxsw> put you probably already know that
<Putti> I actually didn't know that, thanks :)
<blackboxsw> *but...   I need to go find my typo-free keyboad
<blackboxsw> :)
<Putti> blackboxsw, I have some ideas for a possible fix. 1. We could mock cloudinit/url_helper.py's readurl() in the test_gce.py so that "allow_redirects=True" will be changed to False. Or the metadata.google.internal domain could be changed to 169.254.169.254 (this seems to be some sort of special ip) like some other datasource tests already do.
<blackboxsw> Putti: checking that unit test now to see exactly what you mean.
<Putti> Here is some information about the magic ip address: https://askleo.com/why_cant_i_connect_with_a_169254xx_ip_address/
<blackboxsw> right that address, 169.254.169.254, is link local, which means those packets (if they hit your network) shouldn't be forwarded beyond your router. so expectation is that link-local ips will timeout on your local  network. But in the interest of expedience in unit tests, instead of long timeouts we should cut those tests off to avoid lengthy timeout waits. (or we should ensure that our timeout length is as
<blackboxsw> small as possible.
<blackboxsw> s/cut those tests off/mock those external queries to 169.254.*.* so that the expected response is immediate/
<Putti> the question is: how do we do that?
<Putti> ooh, I see, mocking them
<Putti> mhm, but aren't they already mocked? It is just dns that is slowing down, and now there is no dns
<Putti> I mean if we use ip address
<Putti> Hmm, I wonder if this is something that should be fixed in one of those http request mocking python modules we use. But as a temporary fix for us would be removing host names and use ip addresses instead.
<blackboxsw> Putti yeah strange about your pastebin. Since that unit test is mocking platform_reports_gce() to return False, I'm not certain why your env is attempting to access metadata.google.internal hostname
<blackboxsw> hmm Putti that mock doesn't really work it seems. If I put a pdb.set_trace() in  cloudinit/sources/DataSourceGCE.py get_data():
<Putti> yes it doesn't ... I got a new idea why, a minute!
<blackboxsw> https://www.irccloud.com/pastebin/1xAbn5DV/
<blackboxsw> we shouldn't get there if we .tox/py3/bin/python3 -m nose --tests tests/unittests/test_datasource/test_gce.py:TestDataSourceGCE.test_get_data_returns_false_if_not_on_gce -x -s
<blackboxsw> but we do
<blackboxsw> yeah that mock setup in that test is broken
<Putti> I'm trying now adding "_set_mock_metadata" to the test
<Putti> erm "_set_mock_metadata()"
<Putti> mhm, I guess there is no point in adding that... I guess I need to focus on reading the code for a minute :P
<blackboxsw> Putti: can you test this patch? http://paste.ubuntu.com/25076047/
<Putti> yes, a moment
<blackboxsw> yeah I don't like mocks, they don't always behave as you'd expect them. In the case of this unit test the setUp() declared the mocked return_value for cloudinit.sources.DataSourceGCE.platform_reports_gce as True. Then the subsequent unit test tried to override that return value incorrectly to False.
<blackboxsw> using the patch decorator on the specific unit test we can ensure that the unit test has properly overridden that return_value to False instead of setUp's "True".
<blackboxsw> the problem with our tests was that our broken mock as actually still returning True for platform_reports_gce() so get_data() was attempting to perform all those queries against metadata.google.internal which it never should have gotten to
<Putti> paste.ubuntu.com requires you to login (which doesn't work without javascript) if you want to see the plaintext without javascript. Another moment.
<blackboxsw> oops
<Putti> git doesn't seem to like the patch
 * Putti does the changes manually
<Putti> blackboxsw, the patch works, though no idea what it does :D
<Putti> aha, ok, I read your explanation now, and it makes sense
<Putti> thanks!
<blackboxsw> Putti: awesome. So that @mock.patch is me telling the specific unit test to mock the  DataSourceGCE.platform_reports_gce  within our unit test only. Then I set return_value = False there to the mocked method. This ensures the unit tests overrides the mock which was established in setUp()
<blackboxsw> yeah no prob
<blackboxsw> thanks for finding the bug and working through it w/ me
<Putti> :)
<blackboxsw> you can file a merge proposal if you want with that patch and I'll approve and land it if you'd like to go through the process. Or I can, either way
<Putti> blackboxsw, you should do it, you're the author
<blackboxsw> Putti could  you please file a bug against and link your initial problem paste then :) gotta get you some karma for finding this issue and bringing it to us. (and we need bugs associated w/ any cloud-init fix)  https://login.launchpad.net/08ubw7bY4m0DWmsi/+decide
<Putti> sure, bug report coming soon
<blackboxsw> https://bugs.launchpad.net/cloud-init/+filebug rather
<blackboxsw> thanks again
<powersj> blackboxsw: can you send me a pastebin of the mount image callback stuff
<Putti> blackboxsw, https://bugs.launchpad.net/cloud-init/+bug/1703935
<ubot5> Ubuntu bug 1703935 in cloud-init "GCE unit test tries to connect to the network" [Undecided,New]
<blackboxsw> awesome thank you again
<Putti> not very nice view: https://launchpadlibrarian.net/328486103/error .. Maybe I should have added .txt extension.
<nacc> Putti: is that for the bug attachment?
<nacc> Putti: edit it and mark it as text/plain instead?
<Putti> nacc, yes
<Putti> doign that :)
<nacc> Putti: it's currently text/html which is probably wrong :)
 * nacc has an open self-bug for lp-attach to fix that in that particular tool :)
<Putti> do you know if I can file somewhere a bug against paste.ubuntu.com ?
<Putti> nvm, https://launchpad.net/canonical-pastebin ... but it seems that it is not under a free license :(
<blackboxsw> smoser: still working on ipv6 support. scaly implementation looks easy on an already configured interface with a valid broadcast over UDP was easy with scaly http://paste.ubuntu.com/25064349/.
<smoser> blackboxsw, i think that the dhclient paht is probably sane as a first pass.
<smoser> freebsd is affected too though and need to think about that.
<blackboxsw> but, this requires that the interface is already up and configured. using dhclient  and cleaning up after seems the lowest hanging fruit as unicast packet sending on a raw socket requires a bit more reading for me (couple days probably()
<blackboxsw> smoser: +1 and centos dhclient supports a slightly different set of cmdline options that we'd have to account for (minimally "-I" is interpreted differently )
<dpb1> blackboxsw, smoser: first pass making dhclient-capable distros work seems sane.  This isn't working anywhere that I know of?
<dpb1> (on aws)
<smoser> blackboxsw, powersj i guess we could/should run c-i 'tox' at least with networking taken down.
<Putti> +1
<jhodapp> I'm curious if there's any special way of loading certain kernel modules via cloud-init or if I might simply use modprobe to do so in the free form command run section of the yaml config file?
<larsks> jhodapp: that should work (using runcmds, or just submitting a /bin/sh script in place of or in addition to the YAML file)
<jhodapp> larsks, ok, so there's really not any special provision for doing so
<larsks> Right.
<jhodapp> ok
<smoser> blackboxsw, https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/327325 you can push that if you want.
<blackboxsw> thx smoser, I have a working dhclient client run using your WIP branch on aws + a new dhcp_request_clean() function in cloudinit/net/_init_.py.
<smoser> nice
<blackboxsw> just polishing off a simple dhclient-script that could-init would deliver to ensure no other side-effects come from run-hookos
<blackboxsw> just polishing off a simple dhclient-script that could-init would deliver to ensure no other side-effects come from run-hooks
<blackboxsw> s/polish/pollish?
<smoser> blackboxsw, do you have something pushe d?
<blackboxsw> smoser will pull it down from my aws instance and push in a min
<smoser> yeah
<blackboxsw> smoser: a junker WIP for the moment https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+ref/aws-dhclient
<blackboxsw> I hacked up  the assign_ipv4_link_local context manager to call the dhcp_request_clean()
#cloud-init 2017-07-13
<niluje> smoser: do you think my updates are okay?
<Wulf> Hi, is there any way to rename network interface in ubuntu 16.04 on AWS back to eth0?
<ragechin> Wulf: cloud-init 0.7.9?
<ragechin> Wulf: PM me plz. I don't think it's a cloud-init issue
<smoser> niluje, i'll take a look now. i'm sorry.
<niluje> sorry to be oppressive :x
<smoser> niluje, no worries.
<smoser> blackboxsw, can you confirm that
<smoser>  4d9f24f5c385cb7fa21d87a097ccd9a297613a75
<smoser> is broken in the same way as
<smoser>  https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/327325
<smoser> was ? that path is just plain wrong?
<smoser> Wulf, https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/
<smoser> see
<smoser>  I don't like this, how do I disable this?
<smoser> niluje, you kind of have 3 unit tests in one
<blackboxsw> smoser: checking
<niluje> smoser: I can split them, but usually I prefer to test a feature in a single test rather than split it into smaller tests
<smoser> well, but you're really testing 3 different things.
<niluje> because split means having more tests, which take more time to run, and 99.9% of the time it doesn't make sense to test only a function when it works (for instance) and not when it fails
<smoser> a.) valid API / expected working case.
<smoser> b.) no user-data and vendor-data
<smoser> c.) local port retry
<smoser> blackboxsw, i'd also like your opninon on the use of HTTPConnection (via requests.packages.urllib3.connection) in niluje's MP
<smoser>  https://code.launchpad.net/%7Ejcastets/cloud-init/+git/cloud-init/+merge/325740/+index
<niluje> smoser: so you want me to split the unittests then?
<niluje> (won't do before monday)
<smoser> blackboxsw, . your thoughts on ^ too ? i think generally we do want them to be more granular
<larsks> smoser: possible 0.7.6 to 0.7.9 regression when using a config drive with a static network config; curious about your opinion: if after initial boot a config drive goes away, it looks like 0.7.6 will retain the network config while 0.7.9 will panic and replace the static network config with a dhcp config from the fallback data source.
<smoser> larsks, the config drive metadata service's check_instance_id() should end up saying that the cached object in /var/lib/cloud/instance/obj.pkl is still valid.
<niluje>                                                                               â logan-
<niluje> oops
<larsks> smoser:  Looks like not, but I will check.  It looks like with 0.7.6, cloud-init.service simply fails when there is no data source, so -config and -final never run.
<larsks> Let me take a closer look at my 0.7.9 setup.
<smoser> larsks, is this opssibly related to ovirt ?
<smoser> or is it really openstack
<larsks> Not openstack.  Right now, I'm testing locally with libvirt + config drive.
<smoser> (ovirt uses config drive, and i think doesn't set the dmi data to the instance id, which woudl make that fail)
<larsks> But I think the original problem is in RHEV, which I think means ovirt.
<niluje> i think generally we do want them to be more granular -> again I'll follow your recommendations, but for my personal projects I really feel being granular is cleaner, but it's often a pain to do (you need to do the test setup more than once) and doesn't actually help to test the code better
<niluje> but let me know :)
<blackboxsw> niluje: smoser yes please on the separation/simplification of unit tests to simply assert 1 or 2 things instead of compound set of assertions. We want each unit test to be representative of a single "thing" that it is testing, which makes for easier error tracking/resolution when the test fails in the future
<smoser> right. i think that is quite likely broken, but i'm not really sure how it would work previously.
<niluje> okay
<smoser> larsks, and i'm not sure how to make it work.
<niluje> blackboxsw: will do, thanks :)
<blackboxsw> I think we (cloud-init) should pull together a doc on unit test writing just to capture general intent/approach/style etc.
<smoser> hm..
<blackboxsw> thanks niluje, sorry if it feels a bit pedantic
<larsks> smoser: the bahvior in 0.7.6 seems fine (fail if no data source).  Why does 0.7.9 try to continue in that case?  Is there a way to disable the fallback datasource?
<blackboxsw> probably need a little addendum to the HACKING doc on cloudinit.readthedocs.io
<larsks> If not, should there be an option to do that?
<smoser> this path where it correctly identifies its the same instance id will only work on intel (as it uses dmi data, and although arm64 in theory could do dmi practice and kvm are different)
<niluje> if there's something else I'll fix it on monday morning, hoping we can merge the MR soon :)
<blackboxsw> +1 niluje I'll put some eyes on the branch too today to see if I can make more concise comments
<smoser> larsks, it re-writes netowrk data on 0.7.9 as it doesn't know that it is not a new instance, so it goes the path of "do a dhcp on eth0 so the network datasources can find network metadata"
<niluje> blackboxsw: if you are interested in having a node to do some testing, I can give you access to one
<niluje> smoser: same
<larsks> Right, I understand that.  But is there any way to disable that behavior if so desired?
<larsks> The problem is not just that it does a dhcp, but that it is actually writing a new network configuration to disk.
<smoser> larsks, right. but at the point where there is no config drive it can only really think that this is a new instance, or it has been snapshotted and moved to another cloud....
<smoser> its looking for a metadata source.
<larsks> No, I get that.  I am just arguing that it shouldn't be writing configs if ultimately it doesn't find a valid datasource.
<larsks> But I understand what you're saying.  I am trying to figure out if this whole "config drive goes away" thing is standard RHEV behavior...
<larsks> ...because if it's not, "don't do that" seems like the quickest fix.
<smoser> you can configure "manual_cache_clean" to true
<smoser> yeah..
<smoser> larsks,  so right now blackboxsw is working on moving the ec2 datasource to run at "local" time.
<smoser> if we replaced all network-time datasources to run at local, then we could essentially decide failure at the local time frame.
<smoser> and leave the networking in place
<smoser> the path would still result in cloud-init considering that a failure though
<smoser> as it would not have found a datasource (as there is none, and the previous one is not found to be valid)
<larsks> smoser: I wish that there was a shorter term fix when no network datasources are enabled (e.g., when datasource_list is [ConfigDrive, NoCloud]).
<smoser> larsks, well, 2 fixes to the platform
<smoser> a.) match the dmi data to the instance id
<smoser> b.) do not detach the drive ever
<smoser> both of these things qualify as "act more like the platform that you're imitating"
<smoser> larsks, i do agree it sucks. several months ago someone had pointed this out to me, and that was when i asked you what you knew of ovirt
<smoser> i considered raising an issue there.
<smoser> just kind of ran out of time/motivation
<smoser> :-(
<smoser> larsks, if there is other unice id in dmi information on that platform, we could adjust the check_instance_id() to have stored that bit too and ccompare that it is not new
<larsks> smoser: which dmi field are we checking?
<smoser> and another option that might make sense would be to allow vendor_data to declare manual_cache_clean
<smoser> if the cached obj.plk had manual_cache_clean=True, then we could trust it
<smoser> (rather than requiring that to come from system config)
<smoser> the field looked at is system-uuid
<smoser> i do recall that they had a unique id somewhere in their dmi data
<larsks> Thanks. Let me look into that a bit.
<blackboxsw> smoser: that 4d9f24f5c385cb7fa21d87a097ccd9a297613a75 is a completely different failure on my end than what was fixes in my gce mock branch. I'm seeing a magic number traceback in 4d9f24f5c385cb7fa21d87a097ccd9a297613a75. Will peek at that (as well as getting my gce-mock-fix branch pushed)
<blackboxsw> hmm PEBKAC. issue was on my side with a stale pyc file. checking now.
<smoser> :)
<blackboxsw> smoser: same failure mode which the branch I have fixes.
<smoser> you see failure ?
<smoser> because i do not
<smoser> (and neither does c-i)
<blackboxsw> smoser: not a failure, I add a pdb here
<blackboxsw> https://www.irccloud.com/pastebin/pP1aE6ZT/
<blackboxsw> .tox/py3/bin/python3 -m nose --tests tests/unittests/test_datasource/test_gce.py:TestDataSourceGCE.test_get_data_returns_false_if_not_on_gce -x -s
<blackboxsw> and this tox line gets to that pdb which it shouldn't
<blackboxsw> because platform_reports_gce should be mocked to return False in that case
<smoser> (fwiw, i dont think its valid to call python3 like that... you wont get the virtualenv installed things)
<smoser> (that is what ./tools/tox-venv does. ./tools/tox-venv py3 python3 -m nose --tests ...)
<blackboxsw> smoser: here too :) tox -e py3 -- --tests tests/unittests/test_datasource/test_gce.py:TestDataSourceGCE.test_get_data_returns_false_if_not_on_gce -s
<smoser> (and you can run that way by just tox -e py3 tests/unittests/test_datasource/test_gce.py:TestDataSourceGCE.test_get_data_returns_false_if_not_on_gce)
<smoser> but ok. let me look.
<blackboxsw> ahhh good good, I was getting tired of all the extra typing on those --
<smoser> but tox-venv is faster
<smoser> as it doesn't do the setup.py
<blackboxsw> smoser: today I'm testing dhclient in init-local for aws with centos
<blackboxsw> then freebsd (so I'm adapting that WIP branch and starting to make it actually work properly)
<blackboxsw> niluje: forgot to respond earlier about your offer to setup access to a test system for us. I think that is a good offer, we are trying to increase our test matrix coverage and this may assist in upcoming SRUs. While I don't think we have the bandwidth to integrate testing w/ your system, if it doesn't cost anything to allow us to access it. It would certainly assist us as we get a chance to login and validate
<blackboxsw> upcoming cloud-init changes.
<blackboxsw> let me try typing that with proper grammar and punctuation.    niluje: If you can setup system access for us and it doesn't cost anything it might help us in the future when we look at expanding out test matrix.
<smoser> niluje, i do have a scaleway account
<smoser> registered under smoser@brickies.net
<smoser> blackboxsw,
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+ref/bug/fix-gce-test
<smoser> that fixes the openstack metadata service to patch / use-mock correctly
<blackboxsw> smoser: correct and it's merged
<blackboxsw> oops
<blackboxsw> jussec
<blackboxsw> hrm smoser I landed that branch you approved in master a few mins ago
<blackboxsw> and your change is a rewrite. ok, checking it out now
<blackboxsw> smoser: ahh I see what you did to fix it. right the start method returns the actual mocked object so subsequent changes to return_value get honored
<blackboxsw> smoser: +1 on that change to avoid the additional decorators
<blackboxsw> you'll have a minor conflict w/ master I presume
<blackboxsw> as I landed that other fix
<smoser> blackboxsw, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327388
<smoser> you want to review that quick ?
<smoser> and ack and then i'll pul it
<blackboxsw> reading it smoser
<blackboxsw> smoser: approved https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/327388
<smoser> blackboxsw, thanks
<smoser> blackboxsw,
<smoser> =False)
<smoser> smoser@milhouse:/home/smoser-public/src/cloud-init/cloud-init$ tox-venv py3 python3 -m nose tests/unittests/test_runs/test_simple_run.py tests/unittests/test_distros/test_create_users.py
<smoser> that works.
<smoser> but dropping the test_simple_run before test_create_users will fail
<smoser> ie, its leaving some mocks or something in place making test_create_users not fail
<smoser> powersj, that is the only other non-jsonpatch failure in py36
<smoser> ie, it should be a failure in 3.5 but it was just hidden
<powersj> yeah
<powersj> test_create_users
<powersj> smoser: working on kvm backend for testing. I am trying to generate SSH keys on the image, which works, but then run into ssh_deletekeys, which deletes those keys. Have a best practice for doing this?
<powersj> should I generate and inject the key we want to use by hijacking the user_data/cloud-config and put the key in that way?
<smoser> so we can ssh into the system and know the keys, right?
<smoser> its kind of invasive and system ddpendent
<smoser> but maybe
<smoser> https://gist.github.com/smoser/b32bb1c33564d1d46971cd9ded2e8477
<smoser> we run our own ssh on port 9999 that reads its own keys and such
<powersj> smoser: hmm I was hoping to stick to port 22, so that when we extend to this to cloud providers I don't have to deal with firewall related issues
<smoser> powersj, well, that means that you can't really test any of the system ssh
<powersj> :\
#cloud-init 2017-07-14
<niluje> blackboxsw: I was suggesting to start an instance in case you wanted to try the connector
<niluje> but I'll ask to the team on monday if it is doable to give you an instance then, we do that for some open source projects
<blackboxsw> niluje. it'd be helpful for me. I'm actually reviewing your branch a bit right now and was talking to smoser about why that was necessary for ScaleWay (security)
<smoser> blackboxsw, niluje i can start one if its easier than niluje doing it as a one off.
<smoser> it'd be nice if we had a "comp"ed and limited account that we would play nice with too though.
<blackboxsw> s/ScaleWay/Scaleway/ rather
<blackboxsw> +1 on that. since we'd use the account(or instance) exclusively for cloud-init enablement/testing/certification.
<smoser> blackboxsw, niluje one thing that came out of my conversation with blackboxsw ... we'd like a comment in the code on why we're using the low source port number.
<smoser> blackboxsw, root@51.15.134.115
<blackboxsw> thx smoser
<blackboxsw> thanks I'm in the instance
<smoser> powersj, so for your ssh work, were you looking to do a separate ssh server or using the built in one ?
<powersj> Using built in one
<powersj> at this point I'd rather just hijack the user-data and inject my key for kvm backed testing
<powersj> where my key is a key we generate for each run
<smoser> i guess to get going that is acceptable, but we dont want to rely on nit as that means that
<smoser> a.) we always have to have user-data
<smoser> b.) we always have to have ssh server enabled
<smoser> c.) we can't test the ssh generation code
<powersj> so for A our tests are entirely built around user-data
<powersj> for now at least, so yes
<powersj> for kvm testing we have said in our spec that we would use SSH to communicate, so we have to have B
<smoser> but we also need to verify that systems work correctly without user-data.
<powersj> for C we can run those tests via lxd
<smoser> we have to have B because you just said we have to have B
<smoser> :)
<powersj> not trying with any user-data is a gap at the moment
<powersj> I'll throw up a card
<powersj> toss up* (throw up sounds odd)...
<smoser> looking at https://github.com/gravitational/teleport seems like that mnigiht give a fairly sane path
<powersj> smoser: while I have you, how does this look for a qemu cli http://paste.ubuntu.com/25089942/
<powersj> I'll take a look at that
<smoser> well, ultimately i think we really do want something that is part of our test that stands "outside" of the normal behavior of the image and doesn't affect it
<smoser> other wise what you're testing is not the thing you're intending to test.
<powersj> not sure I follow
<smoser> if we're trying to test running cloud-init from trunk on Ubuntu/17.04, then the goal is to test exactly that.   Just as if that version of cloud-init were already in it, with no other changes.
<smoser> ie, we dont want to install a bunch of extra packages on the image, or really do anything else.
<powersj> right
<smoser> every change that is required as part of the test harness should be made to have the least affects possible on the rest of the system.
<powersj> ^^ that's a good way of summarizing
<smoser> if we have to make changes, we want those changes to interact with the rest of the system as little as possible.  ideally no side affects and no dependencies outside of what cloud-init requires.
<powersj> right so using any odd SSH settings or new binaries should be avoided. Which is why I do like modifying user-data to provide us with a standard way of accessing the system.
<powersj> This way the only possible change I make to the images is injecting a new version of cloud-init sometimes. Otherwise, no changes should be made.
<smoser> powersj, yeah. i understand how that is "less" change, but in another way its more change.
<smoser> you're affecting the normal boot of the system
<smoser> i'm not suggesting that we change the port that the system ssh port runs on, i'm suggesting we run an additional ssh server on a random port.
<powersj> We should chat about this because I'm not sure we are on the same page
<smoser> :)
<powersj> or even talking about the same thing anymore ;)
<smoser> powersj, well http://c.brickies.net/hangout if you want. i'll hang there for a bit
<powersj> smoser: still meeting with boss man
<dpb1> smoser: and now he's out of battery
<dpb1> :)
<smoser> powersj, so you're not joining ?
<powersj> smoser: yeah sorry I'm getting lunch and will drive home
<powersj> I'll see if you are still around otherwise we can meet after stand up Monday
<smoser> powersj, no worry
<voja> Hi, is someone aware of a small / example implementation of a meta data server?
<smoser> voja, https://gist.github.com/smoser/1278651/
<voja> Thank you!
<smoser> voja, and https://git.launchpad.net/~cloud-init-dev/cloud-init/plain/tools/mock-meta.py
<voja> Is it possible to re-configure the network that way? DHCP to get a LAN IP and then fetch public IP via metaservice?
<smoser> voja, the only network based service that has that functionality is digital ocean
<voja> I see.
<smoser> so you'd have to mimic digital-ocean . i dont have an example of that server.. you can look in sources.
<smoser> it uses ipv4 link local
<smoser> but interestingly... what you asked for "DHCP to get a LAN IP and then fetch"....
<smoser> is exactly what blackboxsw is trying to turn the EC2 metadata service into
<smoser> but work would be required there anyway.
<voja> At least I got NoCloud with static dualstack configuration working for Ubuntu and Debian
<voja> I still have a problem with CentOS
<voja> Which appears to have an older cloud-init version. It does not load my NoCloud iso
<voja> What is blackboxsw btw?
<blackboxsw> yeah voja, work in progress branch that we're pulling together now for Ec2 I'm sorting some CentOs dhclient issues at the moment https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+ref/aws-dhclient
<blackboxsw> heh voja: short for blackbox software .. an allusion to https://en.wikipedia.org/wiki/Black_box
<blackboxsw> I lost a bet during the world cup that I needed to change my irc nick from csmith -> blackboxsw
<voja> Okay I see...
<voja> I am working on a project that should be able to boot KVM OS images without running the whole installer. They will need pointopoint IPv4 in my setup, which makes everything complicated.
<voja> I like to use cloud images for this
<blackboxsw> so that Work in progress branch should come together today once I get past a couple of additional hoops , but the theory is we'll use the OS's dhclient without side-effecs in the ephemeral environment in order to manually bring up one interface in order to talk to the metadata service to obtain the entire picture of network config for all interfaces.
<voja> That's indeed what I'd need
<voja> Was this part of the discussion how to get the latest cloud-init version into the image?
<blackboxsw> voja, nope it's a separate discussion for part of our work on IPv6 support in AWS.   so in cloud-init I'm hoping we can use the context manager cloudinit.net.dhcp_clean_discovery  to bring up a simple interface with details it gets from a dhcp server, then hit whatever
<blackboxsw> the approach we are taking in amazon to get ipv6 info would be from the metadataservice. But we can't get metadata until we have a valid interface up. So dhclient -4 , then hit metadata, then react to setup the rest of the ipV6 networking if needed
<voja> Okay, that would not bring static IPv4 to the cloud box as I would require
<blackboxsw> the first steps part of  the work should do that, cloudinit.net.dhcp_clean_discovery   would just obtain an ip offer from the dhcp server using discovery and bring up the interface with the dhcp ip address.   The rest of the ipv6 stuff would only happen if ipv6 network configuration were returned by the metadata service.
<niluje> smoser: blackboxsw: that's how our metadata API works. We decided to only allow reading/writing to the metadata API from a privileged port to limit access to the root user, since privileged ports can't be bound by non-roots
#cloud-init 2017-07-15
<Putti> Currently for example in DataSourceAltCloud.py some programs are called with their absolute path, e.g. /sbin/modprobe and /sbin/udevadm. Different operating systems have the udevadm command in different locations so could we use just the command name instead of its full path to sbin?
<Putti> ok, so I thought about this bit. In some operating systems like Debian the /sbin/ won't be included in the path unless the user is super user. So it all depends on how the program is run.
<Putti> I think it would be wiser to let the packagers do the PATH configurations
<Putti> so in my opinion /sbin/modprobe could be just modprobe
<Putti> In systemd the PATH variable is hardcoded to "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" (see man systemd.exec) and for sysvinit we have declared (in cloud-init/sysvinit) that /sbin should be in the PATH.
<Putti> the cloudinit.util module seems to have 'which' function also so that could be used, too. Which option is better?
#cloud-init 2018-07-09
<blackboxsw> smoser: pushed changes to https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/349081
<smoser> blackboxsw: it seems wrong to have sles and opensuse retain their old behavior though, doesnt it ?
<smoser> caribou: please ping me if you need any help for scaleway datasource stuff
<blackboxsw> smoser: probably, and I don't think that value in sles is actually used anywhere in cloud-init either so aligning that with other distros would be helpful more than harmful
<blackboxsw> I'm adding a comment to the branch right now
<blackboxsw> pushed that branch using load_shell_content, thanks for the review
<smoser> blackboxsw: c-i doenst like you
<powersj> c-i doens't like anyone right now
<smoser> oh?
<powersj> lxc seems to be hosed
<smoser> https://jenkins.ubuntu.com/server/job/cloud-init-ci/145/console i dont think was generic
<powersj> ah ok he hasn't gotten that far :D
<smoser> ugh. but also doesnt seem related to branch
<smoser> oh. yes it does
<blackboxsw> hab
<blackboxsw> bbah
<blackboxsw> got it
<smoser> blackboxsw: use of 'dedent' on OS_RELASE_DEBIAN is unnecessary
<smoser> probably want to indent that like the others tehre.
<blackboxsw> pushed fixes. will land it once ci passes again
<blackboxsw> powersj: smoser also for today (for cosmic plus SRU) doc changes https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/349209/+edit-description
<blackboxsw> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/349209 rather
<blackboxsw> powersj: so I can rely on https://jenkins.ubuntu.com/server/job/admin-lp-git-autoland/11/ to land the following right?  https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/349081
<powersj> yep
<powersj> blackboxsw: it is in testing now
<powersj> https://jenkins.ubuntu.com/server/job/cloud-init-autoland-test/4/console
<powersj> blackboxsw: landed: https://jenkins.ubuntu.com/server/job/admin-lp-git-autoland/12/console
<powersj> https://bugs.launchpad.net/cloud-init/+bug/1780481
<ubot5> Ubuntu bug 1780481 in cloud-init "ubuntu/centos/debian: get_linux_distro has different behavior than platform.dist" [High,Fix committed]
<powersj> bug marked fix committed :D
<blackboxsw> excellent
<blackboxsw> thanks for the review smoser: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/349209 is approved now
 * blackboxsw kicks the autoland-trigger job
<blackboxsw> smoser: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/349210 is for cosmic
<blackboxsw> and bionic https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/349212
<blackboxsw> bah dentist appt, back in an hour
<smoser> blackboxsw: i think i might miss you here..
<blackboxsw> No smoser wrapping up
<blackboxsw> No prob
<smoser> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/349212
<smoser> i'll come in and upload for you late.r
<blackboxsw> Thx will address it in 10 mins
<smoser> blackboxsw: add the bug reference
<blackboxsw> https://github.com/CanonicalLtd/uss-tableflip/pull/1
<smoser> http://paste.ubuntu.com/p/Q2ZPSkPxBV/
<blackboxsw> hrm smoser: shall I remove  it from the previous changelog?
<blackboxsw> ahh you mean geT_linux_distro
<blackboxsw> ok. not SRU bug
<smoser> right.
<smoser> make sense ?
<blackboxsw> was wondering again what we do about SRU bug# that was on previous changelog
<smoser> yeah, i think you have it right now
<blackboxsw> leave the previous changelog alone right?it alone right
<blackboxsw> leave the previous changelog alone right?
<smoser> yes. you leave the old in place
<smoser> it is "burned"
<blackboxsw> ok doing that now
<smoser> and then i have to remember when i upload to use '-v18.3-0ubuntu1~18.04.1' to debuild
<smoser> and /me has to run... back in 2 hours or so to upload for you.
<smoser> later
<blackboxsw> pushed xenial, artful, bionic
<blackboxsw> xenial: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/349223
<blackboxsw> artful: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/349222
<blackboxsw> bionic: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/349212
#cloud-init 2018-07-10
<blackboxsw> thx smoser
<smoser> artful on its way
<caribou> Hello ! @smoser & team just so you know, we've deployed cloud-init on our Scaleway infrastructure last week \o/
<smoser> caribou: hey. great.
<caribou> I'm backporting my MR to your latest release so our IPv6 config works
<smoser> ok. thanks.
<caribou> Question : what could cause di_report (in init.cfg) to be empty on Bionic and have {'datasource_list': ['Scaleway', 'None']} on Xenial (same 18.3 package from -proposed)
<caribou> matter of fact, on  Bionic I have init.cfg['datasource_list'] = ['Scaleway', 'None'] and on Xenial I have init.cfg['di_report'] = {'datasource_list': ['Scaleway', 'None']}
<caribou> Maybe I should explain what I'm after : the boot of a Bionic instance with cloud-init has a long (~2min) delay when 'cloud-init init' executes. Last line in the log is `main.py[DEBUG]: no di_report found in config.`
<caribou> oh, and it only happens on a 'real reboot' of an instance. Doing clean & init --local & init does not show that delay
<smoser> caribou: i'm confused.
<caribou> smoser: I am too ;-)
<smoser> can you  post log with the delay ?
<caribou> sure, hold on
<smoser> xenial runs in "report only" mode for ds-identify
<smoser> bionic runs in "enabled"
<smoser> you should not need to specify the datasource_list in either case
<smoser> i'm pretty sure.. i think it will identify scaleway correctly
<caribou> yeah, I noted the difference in /run/cloud/ds*
<caribou> smoser: http://paste.ubuntu.com/p/bkfjyfRQ5g/
<caribou> smoser: sorry wrong log
<caribou> di_report might just be a red herring
<smoser> di_report should not change behavior at all. but it also should not cause any sort of warning.
 * caribou is just waiting for the reboot to complete
<caribou> (I'll run a test with the vanilla cloud-init)
<caribou> smoser: http://paste.ubuntu.com/p/bC394TwdQP/, Line 245
<blackboxsw> hrm no di_report in config means no ds_identify ran is that right?
<blackboxsw> which makes me think disabled systemd units or something
<blackboxsw> ... but that's just me grasping at straws
<caribou> blackboxsw: that could be our funky  image building process getting in the way
<caribou> btw 18.3-0ubuntu1~18.04.1 from proposed does the same thing
<caribou> ok, don't spent time on that, I'll run more tests with something that resemble more to an ubuntu image
<caribou> well, I mean an ubuntu image built a different way
<caribou> blackboxsw: but ds-identity did run, /run/cloud-init/ds-identity.log    is there
<smoser> caribou: just to be perfectly clear... the goal for us is that you (scalway) does not have to know anything about cloud-init to get it to work right (other than how to install the package).
<caribou> smoser: indeed,  but so far I've seen a few occurences of some of our 'customization' getting in the way
<caribou> & I still need to have some level of knowledge to babysit DataSourceScaleway
<smoser> yeah. i undersand that... but the goal is to get rid of those things.
<caribou> ok, let me spend some more time on this & I'll let you know what I find
<smoser> if you want to let me in to look around... thats fine.
<caribou> oh sure hold on
<blackboxsw> smoser: are you repushing the artful upload?
<blackboxsw> the queue seems stale from last night
<blackboxsw> I was going to try pinging in ubuntu-release when ready
<smoser> blackboxsw: sorry. i get all those correct.
<smoser> blackboxsw: "queue seems stale" ?
<blackboxsw> smoser: I meant the timestamp on the upload was 13 hours ago
<blackboxsw> and I thought you mentioned you needed to reupload
<blackboxsw> https://launchpad.net/ubuntu/artful/+queue?queue_state=1&queue_text=cloud-init
<smoser> ah. all of them need re-upload
<smoser> because i did not pass '-v' to debuild
<blackboxsw> I'll add that to the note for https://github.com/CanonicalLtd/uss-tableflip/pull/1
<blackboxsw> on SRU regression re-uploads
<smoser> have we put build-package somwhere ?
<blackboxsw> hrm don't see it in the repos... looking around
<blackboxsw> https://github.com/cloud-init/qa-scripts/blob/master/scripts/build-package
<blackboxsw> hmm should that be in uss-tableflip
<blackboxsw> instead of qa-scripts
<smoser> blackboxsw: ok. i'll move it and add the feature to add -v based on rmadison
<blackboxsw> ok thanks
<smoser> blackboxsw: ok. uploaded, and moved build-package and updated build-package to pass '-v' based on rmadison
<blackboxsw> excellent
<smoser> caribou: i believe your bionic issue is related to bug 1780062
<ubot5> bug 1780062 in linux (Ubuntu Bionic) "Cloud-init causes potentially huge boot delays with 4.15 kernels" [High,Fix committed] https://launchpad.net/bugs/1780062
<smoser> the "smoking gun" there is
<smoser> Jul 10 10:19:34.562583 cloudinit-bionic systemd[1]: Starting Initial cloud-init job (metadata service crawler)...
<smoser> Jul 10 10:21:37.672833 cloudinit-bionic kernel: random: crng init done
<smoser> Jul 10 10:21:37.979128 cloudinit-bionic useradd[822]: new group: name=ubuntu, GID=1000
<smoser> in output of 'journalctl -o short-precise'
<smoser> blackboxsw: did mgerts have any MP up working off your update_events commit (https://git.launchpad.net/cloud-init/commit/?id=be9ecc12823) ?
<smoser> caribou's mp at https://code.launchpad.net/~louis/cloud-init/+git/cloud-init/+merge/347973 would want to use that
<smoser> also, and was looking for any work that already existed
<blackboxsw> smoser:  I pinged mgerts on that during the last week when it landed and he said he'd look at it. https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/343712
<blackboxsw> I see no updates to his branch though
<blackboxsw> but as an example branch that leverages the changes I have one for azure https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/348704
<smoser> thanks
<blackboxsw> about line 43 of the visual diff
<smoser> larsks: around ?
<larsks> Maaaaaaaaybe.
<smoser> outside of just continually throuwing stuff at copr
<smoser> how can i get a rawhide environment to debug
<smoser>  https://copr.fedorainfracloud.org/coprs/g/cloud-init/cloud-init-dev/build/775457/
<blackboxsw> smoser: do we want to re-upload cloud-init SRU without the additional bug references? slangasek is asking in #ubuntu-release
<blackboxsw> <slangasek> Steve Langasek blackboxsw: of course, now we have the situation that the cloud-init upload links to bugs that have no SRU test case: LP: #1780481 LP: #1770712 do you want to add SRU templates to those, or adjust the changelog to not reference them?
<ubot5> Launchpad bug 1780481 in cloud-init "ubuntu/centos/debian: get_linux_distro has different behavior than platform.dist" [High,Fix committed] https://launchpad.net/bugs/1780481
<ubot5> Launchpad bug 1770712 in cloud-init (Ubuntu Bionic) "It would be nice if cloud-init provides full version in logs" [Medium,Fix committed] https://launchpad.net/bugs/1770712
<larsks> smoser: Huh, I'm not sure. I don't think there exists anything like a rawhide image.
<larsks> From the build log it looks like /usr/bin/python doesn't exist.  Maybe it is only /usr/bin/python3?
<blackboxsw> 12:56 S<slangasek> Steve Langasek blackboxsw: ok.  Process on https://wiki.ubuntu.com/CloudinitUpdates dsoes say "single process bug, instead of individual bug reports"
<larsks> Let me ask around #fedora-cloud.
<smoser> larsks: yeah, i assume that is the cause
<smoser> but i want to be reasonably confident in a fix that works tehre and elsewhere
<larsks> Yup. Let me see what I can find out.
<larsks> smoser: https://fedoraproject.org/wiki/Changes/Move_usr_bin_python_into_separate_package
<larsks> So if you want /usr/bin/python to exist (and be python2) you can install python-unversioned-command, or you can just explicitly use /usr/bin/python2 or /usr/bin/python3.
<larsks> (And have a buildrequire on whichever one you use)
<blackboxsw> smoser: my response in #ubuntu-release was "I thought this was an exception to the exception due the the fact that we found a potential regression in the published -proposed version and we needed something to document/validate per a new delta.
<smoser> blackboxsw: sorry... reading
<blackboxsw> yeah I think the concern was that we were documenting more bugs than just the original SRU process bug (per this sru-test-regression fix)
<smoser> blackboxsw: we only added 1 bug
<smoser> number
<blackboxsw> I don't really feel like this warrants another upload attempt to fix it (stripping all bugs referenced) because we discovered new bugs that may be worth documenting, but it doesn't exactly adhere to the cloudinitupdates exception
<smoser> and i had chosen to add that number specificly because it was relevant
<blackboxsw> true and pasted comment over in ubuntu-release ... can continue conversation there
<smoser> i think we shoudl add a sru template to the version bug (1770712)
<blackboxsw> smoser: thanks for the sru template. just finished openstack, gce and azure manual tests.
<blackboxsw> pushing results to ubuntu-ru
<blackboxsw> pushing results to ubuntu-sru
#cloud-init 2018-07-11
<caribou> smoser: thank you very much, you've just saved me hours of investigationn
<caribou> so I went ahead & tested the kernel in -proposed which does fix the bug, so it's now verification-done
<caribou> I'll go back to working on adapting the MP
<smoser> caribou: \o/
<smoser> caribou: let me know if you need anything. its always ok to bother me.
<smoser> robjo: https://bugs.launchpad.net/cloud-init/+bug/1779139 ?
<ubot5> Ubuntu bug 1779139 in cloud-init "metadata api detection runs before network-online" [Undecided,Incomplete]
<smoser> you see that ?
<robjo> smoser: yes, there is also an euqivalent openSUSE bug
<robjo> I've not had time to poke at this.
<robjo> While the proposed patch of "After=network-online.target" supposedly avoids the problem I know there are other issues related to waiting for the network
<robjo> SO for now I removed the patch again from the SUSE package
<robjo> I think this may also be partially related to the dhcp client issue
<smoser> robjo: which is dhclient issue ?
<robjo> smoser: I couldn't find it
<smoser> i do remember discussing that. and ultimatelly i would like to have a sufficient python dhclient in cloud-init.
<robjo> but cloud-init uses dhclient to make a request and then looks at the lease file
<robjo> well that fails on SUSE because the interface is controled by wicked
<robjo> discussed the dhclient issue recently with blackboxsw and he said he'd put it on the agenda for the summit
<robjo> I added some notes to the bug, let me try again to find it
<smoser> but that proved to be more than a afternoon worth of work. all the example python dhclients oddly expect that your interface already has an address
<smoser> :
<smoser> :)
<robjo> Here it is lp#1733226
<robjo> https://bugs.launchpad.net/cloud-init/+bug/1733226
<ubot5> Ubuntu bug 1733226 in cloud-init "cloud-init-local service fails on SUSE distros" [Undecided,New]
<smoser> thanks robjo
<lorddoskias> hello, how does cloud-init detect first boot? if i have tested it locally ona an image, what do i have to do to ensure that the next time the image is run (in the cloud) first boot directives will also run
<smoser> lorddoskias: "per instance" things are done when the instance-id found changes.
<smoser> instance-id varies by "datasource"
<smoser> the goal is that you do not have to 'clean' anything.
<lorddoskias> smoser: so what i did is locally configure cloud-init and then execute systemctl start cloud-init-local systemctl start cloud-init etc
<lorddoskias> after i've verified that everything is working i uploaded this image to a private openstack-based cloud
<smoser> that should be sufficient, yes.
<lorddoskias> when i run a new instance, based on that image cloud-init will work as if it's the first time this image is booted?
<smoser> you could also test locally by attaching a config-drive (openstack) or a "NoCloud" data disk.
<smoser> but yes, it shoudl re-generate ssh keys, and other "per instance" things. as if it was new.
<smoser> if you want to clean state just to remove additional unused data, you can do that too... rm -Rf /var/lib/cloud/
<lorddoskias> right. Now another question regarding the users module i can see in the docs that users should be configured like so : users: - name: root
<lorddoskias> however, the default config has users: - root
<smoser> the default config from where?
<lorddoskias> from opensuse
<smoser> trunk does not have that... it uses a per-distro user.
<lorddoskias> my question is what's the differente between having - name: root and - root
<smoser> well, ... if it does work with just 'root' as teh value , then it will probaly just take defaults from system_info
<smoser> and you can speify moer values with the name: username
<smoser> and such
<lorddoskias> doesn't it work the same in the usptream cloud-init i.e. you can have "- blah" or "- admin" or whatever?
<lorddoskias> even here the documentation (https://cloudinit.readthedocs.io/en/latest/topics/examples.html?highlight=system_info#including-users-and-groups) gives examples as : users: - default -bob
<smoser> probably
<lorddoskias> but this syntax is never explained
<smoser> well, the syntax is explained in the link you provide d ('Valid Values')
<smoser> so if you say that
<smoser> - users: [root]
<seba> I have a VM where cloud init is run on every boot and I don't know where to start debugging. I just want it to be run once and not to regenerate ssh-keys on every boot.
<smoser> works, i believe you. it probalby just picks 'root' as 'name:' and takes defautls for other fields.
<seba> on first boot it finds its datasource (iso) and everything works fine. when I reboot it searches for different datasources (iso is not attached anymore)
<smoser> but i personally would take the morem verbose format.
<lorddoskias> smoser: yeah, makes sense
<smoser> seba: https://git.launchpad.net/cloud-init/tree/doc/examples/cloud-config.txt
<smoser> you want 'manual_cache_clean: True'
<smoser> or just
<seba> hm. okay, I can see why I want this parameter. I just wonder why I didn't encounter this problem earlier
<smoser> never mind 'or just'.  you can write that to a file in /etc/cloud/cloud.cfg.d/
<smoser> you had the cdrom there subsequently? or didnt notice it? if you're running bionic+ cloud-init should completely disable itself on subsequent runs.
<seba> debian stretch
<smoser> it depends on cloud-init version and such.
<seba> I've also run it on xenial and bionic
<smoser> on ubuntu, the cloud-init-generator will disable cloud-init if it does not find a datasource
<seba> currently cloud-init 0.7.9-2 on the host I encounter problems with
<smoser> thats older than i'd care to make guesses on
<smoser> manual_cache_clean is what you want though. and that goes back 0.7.9 for sure
<seba> noted and thx for the suggestion, will try it :)
<seba> smoser, works for me, thx again
<blackboxsw> hrm empty /etc/os-release on the copr build images. I'm adding a build to list /etc to see if we have other version files we can source or maybe use lsb_release?
<smoser> is that there?
<blackboxsw> yeah it looks there, just load_file looks like zero length
<blackboxsw> maybe check /etc/lsb-release
<blackboxsw> or lsb_release util as an option.
<blackboxsw> checking to see if that util is even there
<blackboxsw> but I have to wait 20 mins on each build/failure attempt
<blackboxsw> https://copr.fedorainfracloud.org/coprs/blackboxsw/cloud-init/build/776371/
<smoser> i doubt it has lsb_release.
<blackboxsw> more specifically https://copr-be.cloud.fedoraproject.org/results/blackboxsw/cloud-init/epel-6-x86_64/00776371-cloud-init/build.log.gz
<smoser> ugh
<smoser> powersj: why green here
<smoser>  https://jenkins.ubuntu.com/server/view/cloud-init,%20curtin,%20streams/job/cloud-init-integration-proposed-a/26/console
<powersj> because the last command was green
<powersj> ugh
<blackboxsw> found centos-release system-release redhat-release files in centos 6, might leverage one of them.
<blackboxsw> redhat-release is probably the most ubiquitous as I recall that from a loooong time ago in HP-land
<blackboxsw> this build, when run should give us all the content we need for any /etc/*release file in the failed centos6 env
<dpb1> blackboxsw: that's whats broken the build?
<blackboxsw> dpb1: there is an /etc/os-release in copr centos6 images (and various fedora rawhide images). but it is an empty file
<blackboxsw> so the build can't succeed in running get_linux_distro
<dpb1> blackboxsw: is it acceptable to just stub it out to get the build passing?
<blackboxsw> dpb1: we can stub it out, or make it also check for empty content in /etc/os-release.
<blackboxsw> I'll have a fix for this today no prob. but just waiting on a copr build run
 * dpb1 nods
<blackboxsw> https://copr.fedorainfracloud.org/coprs/blackboxsw/cloud-init/build/776389/ should have enough info for me to get a fix that will work for centos/redhat envs
<blackboxsw> easy to just source a different /etc/redhat-release file if available instead I think
<smoser> this garbage is why its deprecated in python :)
<powersj> smoser: pushed a fix for the proposed testing
<blackboxsw> yeah our get_linux_distro is basically going to re-write platform.dist() as we start supporting more oses
<smoser> for a more limited set of distros, yes.
<blackboxsw> yeah , it's not too bad
<blackboxsw> https://pastebin.ubuntu.com/p/7xVR3dQ2yv/.... ok. something to work with. but not optimal
 * smoser will check back in later. our launch-softlayer doesn't 'just work' for launching --proposed unfortunately. need to fix that somehow.
* blackboxsw changed the topic of #cloud-init to: Reviews: http://bit.ly/ci-reviews | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting: Monday 7/2 16:00 UTC | cloud-init 18.3 released (06/20/2018)
<blackboxsw> smoser: ok, build worked for centos6, not for rawhide https://copr-be.cloud.fedoraproject.org/results/blackboxsw/cloud-init/fedora-rawhide-i386/00776423-cloud-init/build.log.gz
<blackboxsw> BUILDSTDERR: /var/tmp/rpm-tmp.vOrrcj: line 31: /usr/bin/python: No such file or directory .... hrm
<blackboxsw> or rather:                mockbuild.exception.Error: Command failed:
<blackboxsw>  # /usr/bin/systemd-nspawn -q -M 39c142373fa449acb013a47ca64b0cf4 -D /var/lib/mock/775457-fedora-rawhide-i386-1531181651.602050/root -a --capability=cap_ipc_lock --bind=/tmp/mock-resolv.erqniz1j:/etc/resolv.conf --setenv=TERM=vt100 --setenv=SHELL=/bin/bash --setenv=HOME=/builddir --setenv=HOSTNAME=mock --setenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --setenv=PROMPT_COMMAND=printf "\033]0;<mock-chroot>\007"
<blackboxsw> --setenv=PS1=<mock-chroot> \s-\v\$  --setenv=LANG=en_US.UTF-8 -u mockbuild bash --login -c /usr/bin/rpmbuild -bb --target i686 --nodeps /builddir/build/SPECS/cloud-init.spec
<blackboxsw> ... bad paste sorry
<blackboxsw> yeah so the overall build from spec is failing, but no tracebacks about why in rawhide ... hrm
#cloud-init 2018-07-12
<smoser> blackboxsw: well, its that there is no python
<smoser> in rawhide
<smoser> well, we could buildpend on python2 and use python2.
<smoser> thats probably the easiest path out of it... to just build python2 for fedora, but we really want to be doing python3 there.
<blackboxsw> yet on successes I also see the missing python log
<blackboxsw> https://copr-be.cloud.fedoraproject.org/results/blackboxsw/cloud-init/fedora-rawhide-i386/00776423-cloud-init/build.log.gz
<smoser> all by myself tomorrow :)
<smoser> wll that is odd.
<blackboxsw> yet in this success case the diff is logged from sh instead of /var/tmp/rpm-tmp.v0 etc
<blackboxsw> BUILDSTDERR: sh: /usr/bin/python: No such file or directory
<smoser> i think that is just an error in their build system
<smoser> that it didn't catch failure
<blackboxsw> get a few of those and then it goes on building
<smoser> if you're saying that url is a success
<smoser> hm...
<blackboxsw> yeah the last one was a success and continued building perBuilding target platforms: i686
<blackboxsw>  "
<blackboxsw> it's strange, I'll try kick off a build myself from tip pre-get-linux-distro   to see if it passes. here's the proper "pass" linked logs from cloud-init-dev's run before my changeset landed in tip    https://copr-be.cloud.fedoraproject.org/results/@cloud-init/cloud-init-dev/fedora-rawhide-i386/00774784-cloud-init/build.log.gz
<blackboxsw> that differs from my local failed build attempt https://copr-be.cloud.fedoraproject.org/results/blackboxsw/cloud-init/fedora-rawhide-i386/00776423-cloud-init/build.log.gz
<smoser> how'd you local build ?
<blackboxsw> I local built a source rpm and uploaded it via copr UI in the "new Build" dialog ./tools/run-container  --source-package centos/7
<blackboxsw> ok gotta run
<smoser> oh. right. ok. i didn't understand that as "local"
<caribou> smoser: I just pushed modifications for my MR. For some reason Jenkins did not trigger after the failed build
<caribou> but I have one issue that I'd like to bounce by you people
<caribou> The network config gets generated on each boot, but, even though I see bring_up=True, the network config does not get applied (i.e. the IPv6 portion of it)
<caribou> I traced it down to _write_network_config that returns the result of _supported_write_network_config, which always happens to be an empty list
<caribou> Since _bring_up_interfaces() scans that list, it never brings up any device
<caribou> I hope I'm making some sense
<smoser> caribou: you have to be in a group to get it to autmatically run for you
<smoser> dont know if you're in it or not.
<caribou> well, usually it runs on each new commit
<caribou> the first one failed then I fixed & pushed again
<smoser> caribou: bring_up=True should actually be removed.  we dont do anything with it now, and i think its not really supposed to get called (but i'd have to dig to remember)
<smoser> it was kind of garbage from long time a go.
<smoser> what is supposed to happen is that cloud-init in local mod renders the networking configuration
<smoser> and then the operating system brings it all up as it would if that was writtenb efore the boot.
<caribou> hmm, that'll explain it, I moved it back to DEP_NETWORK
<caribou> hmm, then that means that I still need to rely on EphemeralDHCP since I need to fetch the metadata to the the IPv6 info
<smoser> caribou: yes. you do need to use ephemeraldhcp
<smoser> caribou: maybe some background info ...
<smoser>  - local runs before networking is brough up by the OS
<caribou> got that
<smoser>  - cloud-init.service runs *after* networking is brought up by the OS
<smoser>  - we want to guarnatee that things that need networking can use it when the OS says it is up.
<smoser>  ie, we do not want cloud-init to go bouncing the interfaces
<caribou> k
<smoser> so we write networking before the OS brings it up and let the OS bring it up
<smoser> with future events we may have cloud-initn bounsing networking
<caribou> So the use of EventType is to assure that the network_config in the ds will run at each boot and not on the initialone
<smoser> or otherwise telling the OS to apply new networking of some sort.
<smoser> right.
<caribou> ok got it
<caribou> I'll fix it up as such & push again
<caribou> thanks for the details
<smoser> users of scaleway will by default have cloud-init render networking on every boot
<smoser> they will be able to disable that
<smoser> but should they do that... then they wont get networking rendered on boot, and bad things might happen.
<smoser> the reason we allow them to disable that behavior.
<smoser> a.) cloud-init has not behaved like this in the past (backwards compat desire)
<smoser> b.) they may have other ways of building things on top . perhaps they bring up a vpn or something, and cloud-init would cause issues with that if it modified the networking configuraitoin on reboot.
<caribou> good, understood
<caribou> Grrr, I accidentaly delete my MR branch off LP :-(
<caribou> smoser: any chance of tying a new branch on the original MR ? Or I should just close this one & create a new one ?
<smoser> caribou: "tying a new branch?"
<smoser> i suspect if you just push the branch again with same name it might work
<caribou> smoser: doesn't look like it, maybe I try ro resubmit
<smoser> ok
<caribou> smoser: sorry, had to delete & resubmit it was referring to the dead branch
<smoser> caribou: i'll get a look at that today.
<powersj> smoser: are you using blackboxsw's gce instances?
<smoser> powersj: no.
 * powersj blows them away then
#cloud-init 2018-07-13
<smoser> paulmey: your change https://git.launchpad.net/cloud-init/commit/?id=aa4eeb80 is going into xenial and bionic, it'd be wonderful if you could excercise a path on that with a proposed cloud-init.
#cloud-init 2019-07-11
<infogulch> Hi!
<infogulch> I found that $USER is not set in a runcmd script, but whoami says root.  What kind of shell environment should I expect in a runcmd script?
<bitfehler> infogulch: no shell at all, I would say. the command gets run straight out of cloud-init
<bitfehler> you could of course run "bash -c 'command you actuallly want'"
<bitfehler> but that may or may not make sense. what are you trying to achieve?
<infogulch> Well i'm using the second mode.  From the docs: "If the item is a string, it will be written to a file and interpreted using sh."
<infogulch> I'm writing it as:   runcmd: |\n[shell commands....]
<infogulch> which i guess is basically equivalent to "sh -c 'commands...'"
<bitfehler> oh right. also, apparently $USER is not set by the shell, but even before it on regular login: https://unix.stackexchange.com/questions/76354/who-sets-user-and-username-environment-variables
<bitfehler> which you don't have (a login)
<infogulch> Ah ok that explains it
<infogulch> Hmm that makes me wonder what else I'm missing
<bitfehler> so i suppose everything you can find in the POSIX shell standard should work
<infogulch> Ok.  I guess I'm not intimately familiar with the posix shell standard
<infogulch> Thank you!
<Odd_Bloke> infogulch: I haven't done this myself, but if you run `env > /tmp/env` in a runcmd, you should get an idea of what you can expect.
<infogulch> That's a good idea thank you
#cloud-init 2019-07-12
<Goneri> hey! could you take a look at: https://code.launchpad.net/~goneri/cloud-init/+git/cloud-init/+merge/368507
<Goneri> and https://code.launchpad.net/~goneri/cloud-init/+git/cloud-init/+merge/365641 too
<FIux> they didn't have ahosman, call me flux'=D
<samgilson> hey blackboxsw , did any more conflicts come up in my MP? Just checking in cause I know the next release is coming on Monday.
<blackboxsw> samgilson: thanks for the ping. nothing in my mind, I think it looks good and works well. I'll click  approve here, I was waiting to chat with the other upstream devs on Monday to confirm we can pull that into 19.2 release
<samgilson> okay perfect! thanks so much for everything.
<blackboxsw> samgilson: that MP is marked work in progress. Do you have anything else that you have yet to push?
<blackboxsw> if not, I'll set it back to 'Needs review'
<samgilson> blackboxsw No I don't have anything left on my side to push.
<blackboxsw> ok marking it to make sure we review it before 19.2
<blackboxsw> Expectation is Tuesday EOD
 * blackboxsw needs to  fix the topic
* blackboxsw changed the topic of #cloud-init to:  Reviews: http://bit.ly/ci-reviews | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting July 8 16:15 UTC | cloud-init v 19.1 released (05/10) | pending 19.2 release (07/16)
<blackboxsw> marked approved on my side
<blackboxsw> will land the branch monday if no other concerns
<Odd_Bloke> blackboxsw: Are you not landing it because you want a second review?  If so, gimme a pointer. :)
<blackboxsw> Odd_Bloke: excellent, I just wanted a secondary review on that. I covered functionality pretty well there, and I feel like it is in pretty good shape for functionality that is not-risky to cloud-init default logic paths.https://code.launchpad.net/~samgilson/cloud-init/+git/cloud-init/+merge/368943
<blackboxsw> new subcommand == low risk in my mind, just wanted to run it by you gents too to see if there was something significant that I missed.
<Odd_Bloke> Yeah, I haven't reviewed it fully, but worst case scenario this new command doesn't work.
<blackboxsw> Odd_Bloke: right. leaving it unusable for getting 'metrics' for some corner cases.
<blackboxsw> but not related to cloud-init config,setup, deployment
<Odd_Bloke> But it wouldn't be a regression, right?
<blackboxsw> correct
<blackboxsw> no regression potentiall
<blackboxsw> completely new feature
<Odd_Bloke> samgilson: blackboxsw: I haven't done a full review (as my plane meal is being served right now), but I did have some comments that I've posted up.
<samgilson87> my client just crashed, but thank you Odd_Bloke, I'll go through those now.
<samgilson87> Odd_Bloke I agree with all your changes you put forth and pushed them up to the mp.
<Odd_Bloke> samgilson87: Thanks!  Apologies, I wasn't clear, but I did intend my docstring comment to apply to all of the docstrings that could use some extra whitespace.
<Odd_Bloke> But once that's done, I think we can land this.
<samgilson87> oh okay! I'll propagate that to all the docstrings then re-push
<Odd_Bloke> Thank you!
<samgilson87> alright, just pushed up those changes.
#cloud-init 2020-07-06
<langjv> Is anyone aware of any issues using cloud-init's #cloud-config YAML file format for a user-data file (from metadata service) on RHEL8? I have a simple cloud-config user-data that works fine in RHEL7 - but if i pass the same file in on RHEL8 - it doesn't read/parse it properly
<langjv> (website said asking questions her was OK - let me know if that isn't the case)
<langjv> https://pastebin.com/1igasWkr
<langjv> I opened a bug for my issue: https://bugs.launchpad.net/cloud-init/+bug/1886428 Hopefully i'm just doing something wrong/incorrect :(
<ubot5> Ubuntu bug 1886428 in cloud-init "Cloud-Init on RHEL8 cannot parse #cloud-config user-data" [Undecided,New]
<smoser> langjv: what cloud is this running on ?
<langjv> VMWare VIO
<langjv> I haven't yet reached out to them because i can "see" user-data, and it loads from metadata service fine/correctly, it seems more something on the cloud-init side simply not registering/parsing it correctly. I could be mis-interpreting what im seeing/troubleshooting though
<smoser> well. i'm surprised that it works on rhel7 but not rhel8.
<smoser> 2 things that'd be helpful:
<smoser> a.) collect the rhel7 (working) log also
<smoser> b.) try with newer rpms at https://copr.fedorainfracloud.org/groups/g/cloud-init/coprs/
<langjv> a was a no brainer - i should have done that initially. Working on "b" right now. will take a bit
<smoser> it looks to me like maybe you don't have vmware tools installed
<smoser> or maybe there is a systemd issue that prevents it from running.
<smoser> see the "Did not find VMware Customization Config File"
<smoser> https://bugzilla.redhat.com/show_bug.cgi?id=1660713 seems related.
<ubot5> bugzilla.redhat.com bug 1660713 in open-vm-tools "[ESXi][RHEL8.0]Enable cloud-init by default to change the systemd unit file vmtoolsd.service" [Medium,Closed: currentrelease]
<langjv> I can confirm my RHEL8 box has an 11.X version installed - and the "before" cloud-init line is appopriately in the systemd service file already.
<smoser> langjv: the key is DefaultDependencies=no
<smoser> it needs that
<langjv> it needs that removed?
<langjv> [root@rhel8-base log]# cat /usr/lib/systemd/system/vmtoolsd.service[Unit]Description=Service for virtual machines hosted on
<langjv> VMwareDocumentation=http://github.com/vmware/open-vm-toolsConditionVirtualization=vmwareRequires=vgauthd.serviceAfter=vgauthd.serviceDefaultDependencies=noBefore=cloud-init-local.service[Service]ExecStart=/usr/bin/vmtoolsdTimeoutStopSec=5[Install]WantedBy=multi-user.targetAlso=vgauthd.service
<langjv> https://pastebin.com/ESnih10p
<smoser> hm.. that looks sane.
<langjv> Yeah - the comments in the thread seem to indicate to me that DefaultDependencies there, or not there, shouldn't matter.
<langjv> I do have 20.2+90.gb923a9e3 downloded - working on adding that guys in and testing with him
<smoser> the thread is wrong
<smoser> you do need DefaultDependencies=no
<smoser> but you have it.
<langjv> Now wondering if my issue is related to one/both of these (seem to overlap a bit) https://bugs.launchpad.net/ubuntu/+source/open-vm-tools/+bug/1855458 or https://bugs.launchpad.net/cloud-init/+bug/1669875
<ubot5> Ubuntu bug 1855458 in cloud-init (Ubuntu) "cloud-init on Ubuntu 18.04 image does not run in VIO" [Undecided,Incomplete]
<ubot5> Ubuntu bug 1669875 in cloud-init "identify openstack vmware platform" [Medium,Fix released]
<Odd_Bloke> falcojr: Thinking about it a little more, I think the naming of that function is because I've been writing a lot of Rust recently, and using a _or suffix is commonly used to denote defaulting, e.g. https://doc.rust-lang.org/std/option/enum.Option.html#method.unwrap_or.
<falcojr> interesting, good to know
<Odd_Bloke> smoser: Last call to opine on https://github.com/canonical/cloud-init/pull/472 before I land it tomorrow. :)
<Alexey_TTT> Hi. I'm trying to use NoCloud provider, it successfully downloads user-data and meta-data from http server, saves user-data to /var/lib/cloud/instances/vm-debian/user-data.txt but user creation declarations in user-data doesn't seem to be respected. I'm unable to login via tty. There are no errors in log. What do I do wrong?
<blackboxsw> smoser: while in the business of last calls, I think the vmware  folks have a reasonable default behavior for vmware VMTools config vs vCloud config for post-script enablement. I'm not certain if we should push more on the cloud-config options to fully disable something that they have two switches for already. https://github.com/canonical/cloud-init/pull/441
<Odd_Bloke> Alexey_TTT: Can you pastebin your user-data?
<Alexey_TTT> Sorry, sorry for not attaching it before. Current config - https://pastebin.com/q8V8W03i and I also tried this - https://pastebin.com/KTaB9Kt6
<smoser> blackboxsw: i honeslty do not know if they do have a reasonable default behavior or not.
<smoser> either i'm to bone-headed to understand, don't have enough knowledge of vmware, or am to lazy to make an effort.
<smoser> but nothing said in that PR convinced me that it is behaving like i had hoped.
<smoser> (some nice documentation woudl go a long way toward that)
<smoser> If I register a stock ubuntu image on vmware cloud X, then I need to be able to provide it with some user-data that says "DO NOT RUN THE ADMIN'S SILLY ROOTKIT"
<smoser> Odd_Bloke: i responded there.
<Odd_Bloke> smoser: Thanks!
<Odd_Bloke> falcojr: You might be interested in reviewing https://github.com/canonical/cloud-init/pull/476, FYI.
<Odd_Bloke> (And if any committers are looking for a few small PRs: https://github.com/canonical/cloud-init/pull/475 https://github.com/canonical/cloud-init/pull/480 https://github.com/canonical/cloud-init/pull/481)
<AnhVoMSFT> hi folks, I'm trying to run cloud_tests on Azure platfrom from my local environment
<AnhVoMSFT> I ran into some error with lxd even when I already set enabled: False in the platforms.yaml. What did I miss?
<AnhVoMSFT> https://pastebin.com/EgrEXFAb
<meena> Odd_Bloke: what have i doneâ¦ https://github.com/canonical/cloud-init/pull/479/files
<Odd_Bloke> meena: ... hm.
<meena> what hath i wrought?
<Odd_Bloke> meena: My guess is that RbxCloud is not being found because of that hardcoded datasource_list and the change in question does fix that problem on BSD (but causes major issues on other platforms :p).  I'm commenting now.
<meena> Odd_Bloke: i meant the if (not) variant.endswith('bsd')
<meena> we have two different things in that same file, and it should only be one (if not?)
<meena> Odd_Bloke: thank you
<mruffell> Hello! Sorry to be a pain, but is there any ETA of the cloud-init SRU in Ubuntu being released to -updates?
#cloud-init 2020-07-07
<blackboxsw> mruffell: not a pain thanks for the query. We've finished all verification testing, I have sent an email to the SRU team to see if the final verification  by the solutions testing team was enough (bionic-only) to meet the merit of the SRU exception process we hold https://wiki.ubuntu.com/CloudinitUpdates#Solutions_Testing. I believe we are all good on testing and just awaiting a +1 from SRU team in
<blackboxsw> #ubuntu-release channel.
<blackboxsw> mruffell: watch bug https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1881018 for the updated Fix Released status and release comment from SRU team members. It should be this week in the next couple days.
<ubot5> Ubuntu bug 1881018 in cloud-init (Ubuntu) "sru cloud-init (19.4.33 to 20.2-45) Xenial, Bionic, Eoan and Focal" [Undecided,In progress]
<mruffell> blackboxsw: thanks Chad! I will be patiently waiting.
<blackboxsw> just added a comment for clarity and I'm marking SRU tags appropriately so it's not blocked on me per-se
<blackboxsw> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1881018/comments/16
<ubot5> Ubuntu bug 1881018 in cloud-init (Ubuntu) "sru cloud-init (19.4.33 to 20.2-45) Xenial, Bionic, Eoan and Focal" [Undecided,In progress]
<langjv> Hey all - im attempting to run cloud-init 20.2 on RHEL8 to rule out a possible "bug" in the provided releases from RHEL (18.x). It installs fine - but im having issues making it "work". After installing, i "systemctl enable cloud-init" to get it to run at boot, but upon boot, it doesn't actually run
<langjv> Is there some secret to making the latest RPM's provided on the cloud-init repos "work?
<langjv> https://pastebin.com/A26uGCWb
<otubo> Odd_Bloke, Hi! I'm still getting familiar with the tests, sorry for the long back and forth there. Quick question here, don't I still need to inherit test_helpers.FilesystemMockingTestCase in order to mock the filesystem to create a fake swap and further swapon?
<otubo> Odd_Bloke, or I'm missing something here?
<otubo> langjv, can you provide more logs?
<langjv> so - there "are no logs" because nothing runs - the logfiles dont even exist
<langjv> but as soon as i start the service - it runs and logfiles get created
<langjv> i am trying again enabling the following services (cloud-config, cloud-final, cloud-init-local, cloud-init(
<langjv> comparing the RHEL release (these 4 exist and are enabled by default) to the 20.2 release (these 4 exist but are not enabled by default) im hoping that enabling all 4 will cause it to "work" when i turn on the image
<otubo> langjv, 20.2 is not officially supported by rhel, so there might some pieces missing on the rpm. Did you built yourself?
<langjv> no i used the one released on the cloud-init repos. I was asked to try it to rule out issues im seeing in: https://bugs.launchpad.net/cloud-init/+bug/1886428 with the "supported" version
<ubot5> Ubuntu bug 1886428 in cloud-init "Cloud-Init on RHEL8 cannot parse #cloud-config user-data" [Undecided,New]
<otubo> langjv, right. I can take a look t that issue, since it's reproducible in 18.5.
<otubo> langjv, let me know if enabling all 4 services make it work
<langjv> otubo it did not work with all 4 services enabled either - weird: https://pastebin.com/j4VVLMss
<otubo> langjv, do you really need to run 20.2 or you're just investigating that bug?
<langjv> investigating that bug - someone here asked me to try it to see if it fixed my issue yesterday
<langjv> it does seem like "same issue" though in 20.2 - just manifesting differently
<langjv> https://pastebin.com/72YY6NqW
<langjv> cloud-init is enabled but no datasource found, disabling
<langjv> I had found this yesterday: https://kb.vmware.com/s/article/74597
<langjv> and set the needed smbios property on my image - but it "seems" to not be working ( i am on 5.1.0.3 of VIO so a "fixed" version)
<langjv> am using "OpenTelekomCloud" to allow me to test with both 18.5 and 20.2
<otubo> langjv, in which cloud are you running the vm on?
<langjv> VMWare VIO
<otubo> langjv, AFAIK there's another known issue with rhel and vmware, it changes its instance-id on every reboot making cloud-init think it's the first time it runs. Not sure if it's related. But worth looking into.
<langjv> wasnt aware of that - ill keep an eye out for it.
<langjv> i think my current issue is efinitely related to datsource though. im switching back to 18.5 to attempt to reproduce again
<langjv> https://pastebin.com/0YpFzT9b
<otubo> interesting
<Odd_Bloke> otubo: No worries, happy to help you get up to speed!  I'm not really around today, so we can either discuss how to proceed with doing this via pytest in the PR, or you can go ahead and subclass from FilesystemMockingTestCase and write unittest tests; I'm happy with either. :)
<otubo> Odd_Bloke, awesome, I'll try somethings here and update the PR. Thanks for the reviews!
<elsmorian> Hi, I'm trying to debug our cloud init config, its trying to run a bootcmd looks like and it's failing, and we are getting: https://pastebin.com/k17MtCkS
<elsmorian> But I can't see anything in that tmp directory so can't see what it is trying to run, traceback in the logs is also not super helpful. Is there anywhere else I can see more info, or run it so it will show what that tmp shell script contains? - logs: https://pastebin.com/q0nLdigr
<meena> elsmorian:   well, funny story, exit code 2 could very well mean ENOENT
<meena> or was that EACCESS?
<meena> Where's the OS source code (on your phone) when you need it?
<meena> https://docs.rs/libc/0.2.71/libc/constant.ENOENT.html thank you rust
<meena> elsmorian:  so, what's your config look like? maybe if we see that, we can get a better feel for why the file doesn't materialise
<elsmorian> meena: Ah, interesting, so maybe the missing entity is the cause..
<meena> that's what i just said
<elsmorian> meena: so I'm a bit new to this as a lot is templated by terraform by someone else and I'm just getting up to speed. Which config files would be the most useful
<elsmorian> meena: ð
<meena> or tried to say
<meena> let's look at the result that cloud-init sees then
<meena> cloud-init query
<elsmorian> cloud-init query --all? or something more specific?
<elsmorian> Argh, I have to disappear again, `cloud-init query --all` looks like it has some pointers to the things it's trying to run though, I will check that they they all point to things that are there, thanks for the pointer!
<meena> ð½
<AnhVoMSFT> @Odd_Bloke @blackboxsw how do I skip lxd when running tox integration test?
<powersj> AnhVoMSFT, you can use --platform to specify which platform you want to run on
<AnhVoMSFT> you would think setting enabled: false flag in the platforms.yaml file for lxd would have done it :)
<langjv> otubo thanks for your help earlier - my bug is this: https://kb.vmware.com/s/article/74597 - the reason i couldn't fix it is because the "flavor" arg fix is slightly different than the "image" arg fix. Which if course i didn't catch. Eventually got it though. RHEL requires "OpenTelekom" for now until upgrading to 19.2 but otherwise "working" now it
<langjv> seems.
<ahasenack> hi all, is there a way to set http_proxy globally via cloud-init? I'm thinking in /etc/environment
<ahasenack> a quick google pointed me at the proxy settings for apt, but I want then in the default environment
<ahasenack> hm, sounds like a long standing bug: https://bugs.launchpad.net/cloud-init/+bug/1089405
<ubot5> Ubuntu bug 1089405 in cloud-init "setting up system proxy via cloud-config might be nice" [Wishlist,Triaged]
<meena> that should be a module
<meena> cc_proxy
<ahasenack> write_files apparently can be used
<meena> yeah, but, it'd not be complete, given a specific distro and it's needs
<meena> especially with regards to package proxies
<meena> gem, pip, cargo, maven, apt, yum, pkg,
<ahasenack> sounds like a module for populating /etc/environment would be more appropriate
<ahasenack> which I think was smoser's suggestion in that bug
<ahasenack> write_module worked for me, btw
<meena> ahasenack:  different platforms have different ideas where to put the global environment into
<meena> which i think means, i should write a parser for login.conf for distros/freebsd.py
<meena> we could also do with an augeas lense
<smoser> i would hpe i never suggested putting http_proxy variables in /etc/environment
<smoser> i think generally thats a terrible idea
<smoser> ok. looks like in the final comment i came around to some sanity.
<smoser> cloud-init generically taking an environment dictionary
<smoser> i think i've posted such rants elsewhere
<smoser> but here is one
<smoser>  https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1671320/comments/1
<ubot5> Ubuntu bug 1671320 in maas (Ubuntu) "MaaS should set /etc/environment with appropriate proxy settings for rescue mode" [Wishlist,Triaged]
<meena> it's really unfair to demand our Unix such concepts as consistency
<meena> if you want to consistently configure a proxy on a system, you should do it on Windows, or else configure a transparent proxy
<meena> smoser:  so we last sentence sounds good, but i'm not sure if you're suggesting each service have a module with a proxy config, or if you'd have one proxy module where you list which services you'd like to be proxied
<smoser> i'd like nothing to do with the nonsense that is http proxy configuration on unix
<smoser> if we just set an 'environment' for cloud-init, and let the user set what they wanted, then programs it called would get that environment.
<smoser> and you'd get what your after without cloud-init having to know about proxy
#cloud-init 2020-07-08
<Rusty_Almighty> So, to define the question, how do I launch the installer from IPXE with an answer file that was generated and being served from a web address?
<Rusty_Almighty> Example: set params root=/dev/ram0 ramdisk_size=1500000 ip=dhcp url=http://${MYIP}/ubuntu-20.04-live-server-amd64.iso autoinstall ds=nocloud-net;s=http://${MYIP}/ubuntu-yaml.php
<Rusty_Almighty> the nocloud-net data source is expecting a folder and very specific file names to be in their respective file locations. Is there a way to provide the meta-data and the user-data files as a single .yaml file provided by a single web page?
<meena> Rusty_Almighty: ð¤·ð»ââï¸
<meena> that sounds like a NoCloud specific question, so maybe go looking there for answers
<meena> it does seem like NoCloud is one of the simpler datasourcres
<Rusty_Almighty> Is there a specific channel for only the nocloud data source?
<Rusty_Almighty> Also, is there another source that will allow me to doe what I am talking about?
<meena> https://pypi.org/project/cloud-init-seed/ this converts an existing image to one where the NoCloud seed is already mounted
<meena> https://github.com/rmb938/vmw-cloudinit-metadata/ this could be what you're looking for, Rusty_Almighty
<meena> Although it claims that it's VMware specific
<meena> you might be able to extract some essence
<meena> https://gist.github.com/smoser/635897f845f7cb56c0a7ac3018a4f476 this example by our own smoser  mounts the Filesystem
<meena> aaaaand it also says that NoCloud's "upstream" is *cloud-init*
<meena> now that's pretty embarrassing
<meena> we really should have a simple Metadaten
<meena> *metadata Server
<Rusty_Almighty> I would agree. Looking through the source code there doesn't seem to be any way to define just a single answer file in any of the source providers.
<Rusty_Almighty> https://github.com/canonical/cloud-init/blob/ubuntu/daily/focal/cloudinit/sources/DataSourceNoCloud.py
<otubo> langjv, interesting. I'll make sure we document this vmware_extra_config='{"smbios.assetTag":"OpenTelekomCloud"}' on our knowledge base. Or, if it doesn't interfere on other clouds we could push it upstream as well. What do you think?
<adgud> Hey, how do I regenerate network config (for netplan)? Let's say I changed IP address in cloud-init source file and I would like this change to be applied.
<toolz> I guess netplan apply ?
<adgud> But the netplan config file is not updated yet - I should mention that I'm using Proxmox and I changed cloud-init IP address in GUI
<paride> hey rharper, on https://github.com/canonical/cloud-init/pull/482: I also noticed that sometimes images are exported as a single .tar.gz file. I think it depends in how the images was imported in the first place, and that doesn't seem to happen with imaged imported using the ubuntu: or images: remotes
<paride> the cloud_tests were limited on squashfs + tar.xz exports even before
<paride> do we need to support "combined" .tar.gz exports in cloud_tests?
<paride> rharper, in any case not knowing what is going to be exported and guess is not nice
<paride> I'm asking the LXD people
<Odd_Bloke> meena: cloud-init doesn't, but Ubuntu ships all sorts of simple metadata servers: apache2 and nginx are the two most commonly used. ;)
<Odd_Bloke> Rusty_Almighty: This isn't exactly what you were asking for, but if you used something like `ds=nocloud-net;s=http://${MYIP}/ubuntu-yaml.php?document=` then you should get multiple requests to /ubuntu-yaml.php with different query data each time.
<Odd_Bloke> (I'm looking to see if we already have a bug for all-in-one.)
<Odd_Bloke> Rusty_Almighty: And I can't find such a bug, so please feel free to use the link in the /topic to file such a feature request. :)
<Odd_Bloke> otubo: We shouldn't be using `OpenTelekomCloud` anywhere other than https://open-telekom-cloud.com/en; can we identify why that string works and check if there's an alternative that isn't tied to a specific vendor's platform?
<Odd_Bloke> (Specifically, we could potentially land changes to implement behaviour specific to OTC, which might then start being applied to people's VMWare VMs; it's unlikely that this is their desired behaviour. :)
<rharper> paride: thanks;  yeah;  there's a real lack of formatting via the cli;
<smoser> Rusty_Almighty: well, the "seed" was essentially intended to be "a single answer file" per source
<smoser> oh. i see the question Rusty_Almighty was asking. there is no way to provide both user-data and meta-data as a single file. i can see that might be useful, but its not really a big limitation.  files/urls are "free". 2 is not really any harder to serve than 1.
<Rusty_Almighty> But I would at least need to be able to control the name for each file as a PHP with separate variables so that the file can be generated.
<smoser> https://github.com/rmb938/vmw-cloudinit-metadata/ is interesting.
<Rusty_Almighty> I would also need to make changes to my web server to forward certain url's to php files which breaks the paradigm a little bit.
<smoser> Rusty_Almighty: it isn't that hard.
<smoser> cloud-init 'seed' of some url. and then cloud-init just appends 'user-data' and 'meta-data' to that url
<smoser> php or any other web server can look at the url and use it.
<smoser> seed=http://my.address/instance/i-<your-id-here/
<smoser> i dont know in php how to have one php file take all requests "under" a certain path.
<smoser> but certainly you could do:
<smoser>  seed=http://my.address/instance/server.php?id=<your-id-here>&name=
<smoser> and cloud-init would just append 'user-data' and 'meta-data'
<smoser> then just look at the 'name' variable in php
<Rusty_Almighty> I'm not saying it couldn't. I'm just saying that the 53 other automatic installing operating systems that my ipxe server installs don't .  However, I'm liking this idea where we hack around the issue by providing a blank "document=" at the end of the get statement.
<Rusty_Almighty> 23*
<smoser> fwiw, its not a hack.
<smoser> cloud-init is doing string concatenation . urls are just strings.
<smoser> urls no longer have to map 1:1 to a file in a unix filesystem.  thats a good thing.
<Odd_Bloke> smoser: Having support for a single file would simplify things IMO; if people are just writing files to disk then then gain isn't massive, but if people are using some sort of framework (or PHP file) then _requiring_ two GETs makes things more complex than less.
<Odd_Bloke> Because you can't just implement a single endpoint which returns YAML, you have to either implement separate endpoints or implement logic based on query strings.
<Rusty_Almighty> Apologies. pidgin on Ubuntu keeps crashing.
<meena> Odd_Bloke:    did you know at i'm an apache httpd committer?
<Rusty_Almighty> What was that last comment about making a single request?
<Odd_Bloke> meena: I did not!
<meena> (i haven't contributed in ages, because the ASF is quite exhausting, but, yeah)
<Odd_Bloke> Rusty_Almighty: "Because you can't just implement a single endpoint which returns YAML, you have to either implement separate endpoints or implement logic based on query strings."
<Rusty_Almighty> Some one was saying that it would be easier to make a single web request.
<Odd_Bloke> Oh right, "Having support for a single file would simplify things IMO; if people are just writing files to disk then then gain isn't massive, but if people are using some sort of framework (or PHP file) then _requiring_ two GETs makes things more complex than less."
<Rusty_Almighty> Odd_Bloke: Yes. that would simplify things greatly in my specific case.
<smoser> Odd_Bloke: "single endpoint" don't know what you mean.
<smoser> this isn't 1999.
<smoser> s3 is a "single endpoint"
<Odd_Bloke> OK, well, I'm clearly not using it in that sense?
<Odd_Bloke> s/endpoint/route/ if that helps.
<smoser> baked into your thoughts is the idea that '/' means something.
<smoser> its jsut a character in a string.
<Odd_Bloke> What?
<Odd_Bloke> Does the current implementation require 2 GETs or can it only make 1 GET?
<smoser> you're right about 2 GETs.
<smoser> and that *is* simpler. and then also "atomic"
<smoser> so i'm not opposed to it.
<smoser> but, fwiw, no cloud that I know has chosen to implement things that way.
<smoser> ie, ec2 metadata, openstack metadata, hetzner ... they're all multiple GETs
<Odd_Bloke> Oracle has a separate GET for network metadata, but you can get metadata/user-data all as part of one GET from their new APIs.
<Odd_Bloke> But I agree that it isn't a general pattern.
<smoser> you'd have to send back in the resonpse some indication to cloud-init that this is a "nocloud combined" response.
<Odd_Bloke> But if I had to speculate, I'd say that isn't because that's what's best for the guest, it's because organisationally exposing all of the data from multiple different systems run by different orgs via a single GET is Complicated (TM).
<smoser> well.. i dont know.
<Odd_Bloke> Alternatively we could introduce a new `seedcombinedfrom=` or whatever, so you explicitly opt into the combined behaviour.
<smoser> ultimately its a "single endpoint"
<smoser> and you can either combine all the things when the client requests them individually or all at once.
<smoser> value in separate is less to deliver now.
<smoser> anyway
<smoser> so i'm not opposed to single
<smoser> i'd like for it to work with directory also. such that you could /var/lib/cloud/seed/nocloud/<single-file>
<smoser> your suggestion for combined 'seedfrom' means that we don't have to "go fishing"
<smoser> and see if there is a /<single-file> and then <user-data>/<meta-data>
<smoser> antoher option would be to justs have cloud-init check 'meta-data' for a given format.
<smoser> if it starts with '#nocloud-v2' or if yaml/json has 'type: nocloud' 'version: 2' ot something
<Rusty_Almighty> 2 requests are not impossible, maybe if we just had a way to provide what the file names were by allowing more keys to be passed to the data source.  eg. s=http://my-ip/some/folder/;m=meta-data;u=user-data
<meena> how did we go from providing a NoCloud dataserver that transforms one YAML file (per server) into the correct NoCloud formatâ¦to this?
<Rusty_Almighty> You would think that something like this would be required for localization.  I'm pretty sure that some of the poor Russians out there have no idea what meta-data means.
<meena> or did i misunderstand what Rusty_Almighty was talking about to begin with?
<Rusty_Almighty> Or maybe I'm just stating to speak gibberish because I have only slept 4 hours for the past 48.
<Rusty_Almighty> meena: Where is the best place to file a bug for the cloud-init project
<Rusty_Almighty> ?
<meena> Rusty_Almighty:    in the /topic
<meena> pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting July 14 16:15 UTC | 20.1 (Feb 18) | 20.2 (Apr 28) | https://bugs.launchpad.net/cloud-init/+filebug
<Rusty_Almighty> Thanks. I got it from the topic. Filing now.
<Rusty_Almighty> The way I see this is that this is 2 distinct problems. So maybe I should file 2 bugs.
<Rusty_Almighty> Problem 1. the file names "user-data" and "meta-data" are hard coded and cannot be changed.
<Rusty_Almighty> Problem 2. the installer is forced to make 2 seperate web requests which is not always very simple.
<smoser> "installer" is odd.
<smoser> that is confusing
<Rusty_Almighty> smoser: what would you call it?
<Odd_Bloke> Rusty_Almighty: cloud-init operates in already-installed generic images, to specialise them for this particular boot; something else has to do the "install" (e.g. subiquity, debootstrap, the Ubuntu cloud image build process) and include cloud-init before it can run.
<Rusty_Almighty> I filed it as a single bug, but would some one do me the pleasure of reading through my bug to make sure that we've documented everything we just discussed since I'm operating at compromised mental capacity at the moment?
<Rusty_Almighty> https://bugs.launchpad.net/cloud-init/+bug/1886846
<ubot5> Ubuntu bug 1886846 in cloud-init "The nocloud-net data source requires a folder forcing 2 seperate web requests to hard coded files" [Undecided,New]
<smoser> Rusty_Almighty: it doesn't "require a folder"
<smoser> it does require 2 GETs
<Rusty_Almighty> Well, it does require an open ended url which is folder like in nature.
<Rusty_Almighty> smoser: how would you word it?
<Rusty_Almighty> Also, if the data source is just plain old nocloud, then the URL is a folder.
<Odd_Bloke> Rusty_Almighty: It doesn't require a folder because the paths are determined by appending a string, not by path/URL "join"ing; so if the path you specify is /my/metadata- then cloud-init will read /my/metadata-user-data and /my/metadata-meta-data; you aren't forced to use a directory/folder (except insofar as most filesystems require files to be in directories ;).
<Odd_Bloke> Similarly, if you specified `http://example.com/metadata?type=` then your two GETs would be to `http://example.com/metadata?type=user-data` and `http://example.com/metadata?type=meta-data`.
<Odd_Bloke> (I may be misremembering and getting the hyphenation of user-data and meta-data here completely wrong; apologies if so!)
<Rusty_Almighty> Renaming folder to "folder like string" then?
<Rusty_Almighty> "The nocloud-net data source requires a folder like string forcing 2 seperate requests to hard coded locations"
<smoser> just drop "folder". requires 2 separate requests.
<smoser> http has no explicit relation to folders.
<smoser> this is surprising to me
<smoser> anyone else?
<smoser> $ python3 -c 'import io; buf = io.StringIO("he
<smoser> llo "); buf.write("world"); print(buf.getvalue())'
<smoser> bah. paste fail.
<smoser> $ python3 -c 'import io; buf = io.StringIO("hello "); buf.write("world"); print(buf.getvalue())'world
<smoser> again.
<smoser> https://paste.ubuntu.com/p/Sz6t7hNsHk/
<smoser> there. that seems more sane.
<smoser> sane paste. i still think that is very odd behavior.
<Odd_Bloke> The docs do say "The stream is positioned at the start of the buffer."
<Odd_Bloke> So I think you'd need to do `buf.seek(0, whence=os.SEEK_END)`?
<Odd_Bloke> (I agree it's unintuitive!)
<Rusty_Almighty> I believe the seek has to happen before you write.
<Rusty_Almighty> IE. you sert the buffer to hello and even though you did, the carrot is at 0 still. So when you write world, it overwrites hello.
<Rusty_Almighty> https://paste.ubuntu.com/p/MVD8VsvyD2/
<Rusty_Almighty> Odd_Bloke is correct.  "buf.seek(0, os.SEEK_END)" before writing will get you what you want.
<meena> smoser: what happens when you flush? (at the end? at the beginning?)
<Rusty_Almighty> meena: flushing at the end or at the beginning doesn't seem to affect the carrot.
<Rusty_Almighty> Carrot stays the same.
<smoser> it makes sense if you think of io.StringIO("foo") as open(file, "rw") where 'file' already had "foo" inside it.
<Rusty_Almighty> sigh.. more problems...
<Rusty_Almighty> https://paste.ubuntu.com/p/NM8z5KXjVW/
<Rusty_Almighty> There is also no logs from my web server that the URL was accessed.
<Odd_Bloke> Yeah, that message means that it didn't identify it as a valid URI to even attempt.
<Odd_Bloke> I'm not sure why.
<Odd_Bloke> Rusty_Almighty: What was your kernel cmdline?
<Odd_Bloke> Rusty_Almighty: I'm leaving for today (and off tomorrow), so I'm afraid this will be my last contribution: my only guess is that you've specified nocloud not nocloud-net, which changes the prefixes that it would consider valid.
<Odd_Bloke> (I think dsmode=net is a red herring there, but maybe it isn't and my guess is wrong! :)
<Rusty_Almighty> Sorry. Filing another bug with Debian. FIX ALL THE THINGS.
<Rusty_Almighty> set theKernelParams root=/dev/ram0 ramdisk_size=1500000 ip=dhcp url=${base}/ubuntu-20.04-live-server-amd64.iso autoinstall ds=nocloud-net;s=http://ipxe.localdomain/ubuntuks.php?ip=${ip}&version=${UBUNTUVERSION}&doctype=
#cloud-init 2020-07-09
<smoser> Rusty_Almighty: i wonder if you can describe what you're trying to do there.
<Rusty_Almighty> I want a web server to get information about the server, in this case the IP address as well as the version of ubuntu being installed, and then I want to generate the meta-data file as well as the user-data file on the fly when requested rather than having static files.
<smoser> Rusty_Almighty: fwiw, i might look at maas if i were you.
<Rusty_Almighty> LOL. Looks like I invented MAAS before MAAS was a thing back in 2012.
<rharper> Or cobbler
<rharper> https://github.com/canonical/cloud-init/pull/484  for https://bugs.launchpad.net/cloud-init/+bug/1886531
<ubot5> Ubuntu bug 1886531 in cloud-init "cloud-init status broken in groovy lxd containers" [Medium,In progress]
<rharper> ran into that working on a cloud_test for the chrony issue;
#cloud-init 2020-07-10
<otubo> smoser, hey, don't see anything regarding LVM on cloud-utils-growpart, would have plans to do it? I have a couple of issues with customers using RHEL requesting it.
<Odd_Bloke> We just landed a fix for an issue that affects groovy (thanks rharper!), so I'm going to upload a new cloud-init to groovy.
<meena> what's groovy?
<rharper> Odd_Bloke: nice!
<rharper> meena: new ubuntu development release name Groovy Gorilla
<meena> 20.10?
<meena> yes
<Odd_Bloke> rharper: If you have a minute to validate: https://github.com/canonical/cloud-init/pull/486
<smoser> https://bugs.launchpad.net/cloud-init/+bug/1799953
<ubot5> Ubuntu bug 1799953 in cloud-utils "growpart does not work for lvm" [Medium,Triaged]
<smoser> otubo: ^ . I have no plans. but ^ is the bug that was created for the issue.
<meena> works for zfs
<meena> but then again, zfs is lvm + $fs in one
<otubo> smoser, thanks
<otubo> Odd_Bloke, do you guys plans to migrate issues from launchpad to github as well in the future?
<Odd_Bloke> otubo: Nope, we have to use Launchpad for Ubuntu issues/process regardless, so migrating "upstream" issues to GitHub would just leave us handling issues across two platforms.
<smoser> otubo: https://bugs.launchpad.net/cloud-init/+bug/1799953
<ubot5> Ubuntu bug 1799953 in cloud-utils "growpart does not work for lvm" [Medium,Triaged]
<smoser> comment added there. it seems pretty easy. if you wanted to try
<meena> smoser: the only issue i see is using the device uuid, or some other link to it, instead of the device path
<smoser> meena: ?
<smoser> how so.
<meena> i dunno, hmm, maybe that just means more steps in the resolution of the name / path
<smoser> you mea if the pvcreate had been done with a different path ? hm...
<smoser> i dont know how names for pvs are done.
<smoser> so yeah, th3e check that is uggested (pvs | grep) could be an issue. and result in not running pvresize.
<smoser> but it seems the other pv commands when given a block device path probably open it, read the PV UUID and use *that*
<smoser> just playing... i did a 'pvcreate' with the by-partuuid path:
<smoser> pvcreate /dev/disk/by-partuuid/2079bb1d-d7ca-4b4f-a1d1-a62c06b7c570
<smoser> and then 'pvs' will show the pv with the 'loop5p1' name. I'm guessing that it just does effectively 'readlink -f' on whatever input for that.
<smoser> thanks alot meena
<smoser> now i'm back to being depressed about linux storage
<rharper> s/linux storage/linux-lvm2
<meena> i wish zfs would work somehow across computers,so you could build cloud storage
<meena> but chances are that then it simply wouldn't work
<smoser> rharper: wrt sucking, i mean that wrt almost all of linux storage. its not limited to lvm.
<meena> imagine if ceph was easy!
<smoser> it woudl still suck.
<smoser> sorry.... rharper and I submitted a talk to open source summit, but it didn't get accepted.
<smoser> things we learned and got burnedon in curtin and other places.
<meena> smoser: about storage on Linux?
<meena> did you still write it down?
<smoser> https://hackmd.io/eMOklfSvS6aFCcmu_1h79Q
<meena> wow, this is literally the worst web page i've visited all day: http://www.golismero.com/
<meena> smoser: cool! i â¦ didn't expect that, but wow, cool!
<meena> uuids are not unique when you snapshot and re-attach. this goes for partition uuids, and fs uuids. What is a good way to find the right path to this thing. fstab â¬ï¸ wat
<rharper> smoser: heh;  lvm2 has a special place on my crap list ...
<rharper> meena:  https://wiki.ubuntu.com/FSTAB
<rharper> consolidated list of best persistent links;   we generally prefer to avoid FS_UUID if possible, as it;'s mutable;  but in the absence of any other one, we use that
<smoser> all that magic (like LABEL= or UUID=) it all works most of the time.
<smoser> and when a human is typing stuff, the times ti doesn't work, they just live with and reboot or whatever. they say "oh well, that doesn't work because x y z"
<meena> i can copy/paste, or i can use human readable / writable paths.
#cloud-init 2020-07-11
<MICROburst> Anyone got a snippet that shows the use of "renderer: network-manager"? Neither "network-manager"(https://netplan.io/design) nor "NetworkManager" (https://netplan.io/) is working
