[11:25] <bob_cheesey> morning all - I've been having some issues setting the hostname on debian jessie using 0.7.6 (and 0.7.7) - is there somewhere I can raise issues like this? I'm not sure whether it's a bug or my implementation
[13:31] <bob_cheesey> well clearly this isn't the place to ask!
[13:41] <rharper> bob_cheesey: it's mostly quiet here except during US Central hours;  ask away and we usually get back to you if your still in-channel ;  you can report bugs here: https://bugs.launchpad.net/cloud-init
[13:41]  * rharper has to take kiddos to school, bbiab 
[13:42] <bob_cheesey> rharper: ah thanks, that's useful to know!
[13:42] <bob_cheesey> i'll try again later
[16:07] <smoser> rharper, http://paste.ubuntu.com/23911739/
[18:43] <magicalChicken> bob_cheesey: are you getting a timeout from hostnamectl in cc_set_hostname?
[18:44] <magicalChicken> I've had similar issues on centos but haven't had time to debug
[19:27] <burgerk> smoser, sorry I was out of the office yesterday...  to followup on the issue I was having with rebooting without the ConfigDrive attached.   I captured the logs you requested.   I believe this is likely the section of logs of interest.  http://paste.ubuntu.com/23912850/
[19:29] <burgerk> I can attach a tar of all the logs, just tell me where
[19:30] <smoser> burgerk, yeah... where is this running ?
[19:30] <smoser> what os are you on ?
[19:30] <smoser> and what openstack ?
[19:30] <smoser> xen, isnt it.
[19:32] <burgerk> Ubuntu 16.04.1
[19:32] <burgerk> or host hardware ?
[19:32] <smoser> your guest is 16.04
[19:32] <burgerk> yes
[19:32] <smoser> do you  know what openstack version is under you ?
[19:33] <burgerk> Newton ... or do you want a specific driver #
[19:39] <smoser> burgerk, is it xen ?
[19:39] <burgerk> ppc64le
[19:39] <smoser> rharper, ^
[19:39] <smoser> wow.
[19:40] <rharper> \o/
[19:40] <smoser> this very thing rharper and i were wondering about
[19:40] <smoser> :)
[19:40] <rharper> virsh dumpxml need
[19:40] <smoser> burgerk, can you do that ^ ? can you get at the host at all ?
[19:41] <rharper> smoser: IIRC ppc64le has a uuid out in the device tree sys path
[19:42] <smoser> so in your guest, can you do this, burgerk
[19:42] <smoser>  sudo grep . -r /sys/firmware/devicetree/ | grep -v Binary
[19:42] <smoser>  sudo grep . -r /sys/firmware/devicetree/ | grep -v Binary | pastebinit
[19:43] <rharper>  /sys/firmware/devicetree/base/vm,uuid
[19:44] <rharper>  but yeah, a full collection will be super useful
[19:44] <smoser> that is crazy helpful, burgerk
[19:45] <burgerk> http://paste.ubuntu.com/23912946/
[19:45] <smoser> booo
[19:45] <burgerk> not much there
[19:46] <smoser> can just
[19:46] <smoser>  cat /sys/firmware/devicetree/base/vm,uuid
[19:46] <smoser> shows no file ?
[19:46] <smoser> (ie,m wondered if grep -r didn't follow a symlink maybe)
[19:47] <burgerk> yes
[19:47] <burgerk> no such file
[19:50] <burgerk> contents of /sys/firmware/devicetree/base  http://paste.ubuntu.com/23912964/
[19:51] <smoser>  ibm,partition-uuid ?
[19:51] <smoser> and artition-name
[19:51] <smoser> anything in those ?
[19:51] <smoser> is this kvm ?
[19:51] <smoser> or is it powervm
[19:51] <smoser> oh. wait. i dont knwo if there is ppc64el powervm
[19:51] <smoser> and system-id
[19:51] <smoser> theres just binary stuff in there ?
[19:52] <burgerk> powervm
[19:53] <burgerk> ibm,partition-uuid 7ab415a2-30e7-482f-8806-e9ec4197b7ba
[19:54] <smoser> that by chance match the uuid ?
[19:54] <smoser> from the openstack c onfig drive?
[19:55] <burgerk> instance -> /var/lib/cloud/instances/fab415a2-30e7-482f-8806-e9ec4197b7ba
[19:56] <burgerk> same except 1st character
[19:58] <rharper> eww
[19:58] <burgerk> I don't have config drive anymore, removed to repeat test
[19:58] <rharper> that looks bugy
[19:58] <smoser> wow
[19:59] <smoser> that would be a massive coincidence
[19:59] <smoser> :)
[20:00] <rharper> https://review.openstack.org/#/c/200879/
[20:00] <rharper> but doesn't show where it would appear
[20:00] <smoser> burgerk, so why didnt our 'grep ." above show us that file ?
[20:00] <burgerk> powervm
[20:00] <burgerk> sorry bad paste
[20:01] <rharper> https://github.com/openstack/nova-powervm/blob/master/nova_powervm/virt/powervm/vm.py#L310
[20:17] <burgerk> so that covert method is changing the system UUID byte 0, bit 0 to 0
[20:22] <rharper> eww
[20:23] <rharper> certainly then the OpenStack instance UUID ought to match the UUID inside the vm
[20:23] <rharper> even if they need to drop a bit to avoid collisions (or for whatever reason)
[20:23] <rharper> nova list $UUID should match inside the instance ; I would think
[20:26] <burgerk> I believe the converted UUID is being sent as system UUID when the VM is created and the config drive contains the original openstack UUID
[20:27] <rharper> but your instance symlink says otherwise
[20:27] <rharper> vs. the device-tree value which is created by the hypervisor
[20:27] <burgerk> instance symlink matches config drive value
[20:27] <rharper> but not the vm id
[20:27] <rharper> which is what the hypervisor got
[20:28] <rharper>  ibm,partition-uuid ;  maybe that's by design
[20:28] <burgerk> yes I believe it is some sort of requirement that the system UUID bit 0 be 0
[20:29] <rharper> what we were hoping for was something that indicated this was an openstack instance;  on KVM and Xen, for example the SystemVendor is set to Openstack
[20:31] <rharper> that's mostly for the network-facing metadata service; config-drives indicate their presence via filesystem attributes;  however; IIRC, your issue was that you removed the configdrive, which also set the UUID, then when cloud-init runs a second time somewhere else, its looking for a datasource, it's going to search the system for a uuid to see if that matches existing instance directories
[20:31] <burgerk> yes that appears to be the issue
[20:32] <rharper> that seems to have gone awry in that we're poking for DMI data (which we won't see on power); also, even if we did look at the partition,uuid, we'd need to apply a 1 to bit-0 of byte-0 to match if we're on powervm
[20:33]  * rharper will bbiab, kiddo pick up time
[20:58] <rharper> burgerk: in your situation, did you set manual_cache_clean: True ? ISTR something (or someone) not wanting to set that; though for configdrive it's required if you expect it to boot with the same instance data;  if you do have that set, and it's failing (due to the search for dmi data on power) then definitely file a bug, https://bugs.launchpad.net/cloud-init
[21:00] <burgerk> The drawback of that was that I can't just snapshot the VM and deploy that image then, right ?
[21:01] <burgerk> there would be manual steps before snapshoting?
[21:04] <rharper> I think smoser pointed out that if you snapshot and boot somewhere else it _is_ a new instance; so you shouldn't deploy the snapshot without re-running cloud init
[21:04] <rharper> you don't want nodes with the same ip, the same sshd host keys, etc
[21:06] <burgerk> right, I want cloud-init to run again if it is a new instance, i.e. new config drive and new UUID on it
[21:07] <burgerk> what I don't want is to have to manually delete something before I snapshot
[21:08] <rharper> how are you deploying your snapshot? with openstack still, so nova boot points to an existing snapshot image?
[21:09] <burgerk> yes, snapshot the current VM with openstack, then nova boot pointing to that snapshot
[21:09] <rharper> IIUC, when you boot the snapshot again, cloud-init will run again, as it needs to since you're on a new instance;  but I'm guessing there is something that you don't want to run again as you're on this snapshot?
[21:11] <burgerk> I will give that a value a try and see what happens
[21:12] <smoser> rharper is right, it should all "just work"
[21:59] <burgerk> smoser, after snapshot and deploy of the snapshot, it is not writing network info from the config drive on first boot using manual_cache_cleanup: True
[22:01] <smoser> burgerk, i'm confused.
[22:01] <smoser> why are you setting manual_cache_cleanup=True ?
[22:02] <smoser> (and its manual_cache_clean (no "up").
[22:02] <smoser> i thoguht your problem before was that cloud-init *was* re-writing network configuration
[22:03] <rharper> smoser: my fault
[22:03] <burgerk> rharper suggested using that property to fix the situation for removing the config drive
[22:03] <rharper> smoser: when using a config drive, and then removing it, we have to use that , right?
[22:05] <rharper> it sounds like in manual_cache_clean mode, we don't re-render network-config?  but generate fall-back config (dhcp on first interface)
[22:05] <rharper> burgerk: in general the static network config is going to break if someone else also boots that static network config in the same tenant network
[22:06] <smoser> manual cache clean should basically mean that cloud-init always re-uses the stale data in /var/lib/cloud/instance/
[22:07] <smoser> and thus, the second time or any time after (even after snapshotting and re-deploying) it boots will think it is not first boot
[22:07] <smoser> basically its *supposed* to just never go looking for a datasource again
[22:07] <smoser> as long as there is one in /var/lib/cloud/instance/
[22:08] <rharper> right; how does that intersect network config
[22:08] <burgerk> ok, that was my understanding of the property and why I didn't want to use it ...   I need something like use datasource if it exists, otherwise stale data in /var/lib/cloud/instance/  :)
[22:08] <rharper> I've not seen the config-drive, but apparently it's including some static eni configuration which gets written to /etc/network/interfaces.d/50-cloud-init.cfg
[22:09] <burgerk> correct
[22:09] <smoser> burgerk, well, that i think is the behavior you'd get on intel
[22:09] <smoser> in a vm on openstack.
[22:10] <rharper> if we set manual_cache,  it appears that we're still doing fallback networking (re-writing the 50-cloud-init.cfg file) but not with the network info which should be embedded in the instance.obj right?
[22:10] <smoser> because on intel, cloud-init comes up and checks the product uuid and says "oh, its the same as the one in /ar/lib/cloud/instance. no need to look again".
[22:10] <smoser> i think.
[22:10] <rharper> burgerk: is it possible to share your user-data/meta-data files in your config-drive ?? (or a variant of it) ?
[22:10] <smoser> it shouldnt do that
[22:10] <rharper> but burgerk is not using the same instance
[22:11] <rharper> snapshot to cinder, nova boot -from-snapshot
[22:11] <smoser> for sure its possible it is,b ut it shoudl only render networkign on first instance boot
[22:11] <rharper> it *is* a new instance
[22:11] <rharper> but we're saying, use stale data (on purpose)
[22:11] <burgerk> with the property set to true it is using stale data, not fallback data
[22:19] <burgerk> rharper, here is network config drive files https://paste.ubuntu.com/23913828/
[22:20] <rharper> burgerk: thanks, I'll recreate; I suspect you're still going to have issues with using the snapshot with stale-data; but if we say to use stale-data, then it shouldn't wipe the network config
[22:21] <burgerk> correct it appears to be using stale-data on all boots ( including first boot )
[22:42] <burgerk> rharper, smoser  I will also see if something can be done on the powervm side to get matching UUIDs for system UUID and ConfigDrive UUID value
[22:42] <rharper> ok
[22:43] <rharper> if you asking, we're very interested in getting the Openstack Nova string into the devicetree output so we can detect that we're Openstack on ppc64le platforms
[22:44] <burgerk> ok
[22:50] <rharper> or, somewhere else if devicetree isn't possible