[05:34] <goten> script placed in /etc/cloud/cloud.cfg.d/my_script.cfg is not executed during boot up. Did not find any useful info in /var/log/cloud-init.log, is other places I can look for more info? Or what is the best way to troubleshoot this?
[15:17] <powersj> just to have them in here: these are my two MPs for KVM
[15:17] <powersj> commands as strings: https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/330535
[15:17] <powersj> add xkvm: https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/330536
[15:32] <dpb1> xkvm?
[15:44] <powersj> wrapper around qemu command
[15:44] <powersj> rharper/smoser created for curtin's use
[15:45] <nacc> since qemu's parameters make little/no sense to most people, dpb1 :)
[15:45] <dpb1> people who aren't cpaelzer
[15:46] <dpb1> powersj: TIL, thx
[16:05] <smoser> rharper, https://blueprints.launchpad.net/nova/+spec/virt-device-tagged-attach-detach
[16:06] <smoser> that is what i think i remember.
[16:06] <rharper> \o/
[17:33] <rharper> smoser: in that spec, it mentions the qemu firmware exposing metadata path;  we have support for that now, right?  the nocloud from smbios?  do you think that's the same?
[17:34] <rharper> "Outside the scope of the Nova work, a simple tool will be created that can parse this metadata file and set tags against devices in the udev database. It is anticipated that cloud-init would trigger this tool. "
[17:49] <smoser> rharper, ?
[17:49] <smoser> i dont follow
[17:50] <rharper> I know we added some dmi path to getting datasource, or user-data, right ?
[17:50] <rharper> the DMI NoCloud mode ?
[17:51] <rharper> I was wondering if that was what the spec was referring to w.r.t qemu exposing metadata via firmware to instances;    the spec has a general concern about providing network-based metadata to instances which may not have network configuration (the writers were unaware of how cloud-init handles this in other network-based datasources)
[17:52] <rharper> and the second quote was just an interesting tidbit;  the use-case of 'tagging' devices for service discovery (sdc is for 'oracledb') and wanted to provide a more "General" way for applications to discover the tagging without forcing the app to learn "openstack device tagging json schema"; and suggested udev tags/attrs as such a method
[17:53] <smoser> well, the dmi nocloud stuff is not related to the openstack stuff for sure.
[17:53] <rharper> but it
[17:54] <rharper> but it was provided by qemu (aka firmware)
[17:54] <smoser> we just now support reading nocloud info from dmi
[17:54] <smoser> yeah
[17:54] <rharper> I wonder if it could be a ConfigDrive style support if they could stuff enough info in there
[17:55] <rharper> I guess it's only useful during boot;
[17:55] <rharper> hence NoCloud;  I don't suppose qemu has an interface to update values in the dmi table dynamically
[17:56] <smoser> powersj, you just replaced my mp ?
[17:57] <smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/330459 -> https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/330535
[17:57] <powersj> Based on your comment it seemed like you wanted a seperate MP. So we can merge yours and then put mine on top to change the other strings.
[17:57] <smoser> oh i see you added some of the string changes too
[17:57] <powersj> yeah
[18:34] <smoser> powersj, i'm sorry.
[18:34] <powersj> ?
[18:38] <powersj> smoser: I assume the sorry is for the review?
[18:42] <smoser> yeah
[18:50] <powersj> smoser: for KVM when we execute SSH expects a string, and mount-image-callback expect a string
[18:50] <powersj> should I then do the opposite as LXD, in that if it detects it is a of type array, convert it to a string?
[19:08] <smoser> powersj, mount-image-callback doesn't expect a string. it has a sane interface.
[19:08] <smoser> ssh interface sucks
[19:09] <smoser> it shoves you through sh (or rather the user's shell) more for security reasons than anything else.
[19:09] <smoser> that way the platform can manage what your user can do by setting the shell differently.
[19:10] <powersj> smoser: the way I was calling MIC https://paste.ubuntu.com/25522742/
[19:10] <smoser> to fix ssh, you can do this:
[19:10] <smoser>  https://gist.github.com/smoser/88a5a77ab0debf268b945d46314ea447
[19:10] <smoser> you have to be creative.
[19:10] <smoser> (that does insert a dependency on 'base64', which i dont have a good way to work around)
[19:12] <powersj> sigh
[19:13] <powersj> smoser: I guess I'm not sure I understand why the usage of the python library doesn't work
[19:14] <smoser> that is basically what we use in tools/run-centos : inside_as
[19:14] <smoser> what python library ? i'ms orry.
[19:14] <powersj> the integration tests are in python
[19:14] <powersj> I am using paramiko to call ssh
[19:14] <powersj> from what you are saying you want me to get rid of that to call your ussh shell script
[19:24] <smoser> powersj, no. using paramiko is fine
[19:24] <smoser> we just wrap it sanely.
[19:26] <powersj> smoser: define wrapping sanely :) outside of your base64 encoded strings this is a new to me
[19:26] <smoser> :)
[19:26] <smoser> i want to provide an interface '.execute()' that takes a list. not a string.
[19:27] <smoser> (just like subprocess.call() does... its not a crazy idea)
[19:27] <dpb1> powersj: you are familiar with pep8, right?
[19:27] <dpb1> the pep, not the tool
[19:27] <powersj> dpb1: of course
[19:27] <dpb1> ok
[19:31] <powersj> smoser: you are choosing to pass a list to replicate the behavior or because you feel passing the list is a more design decision?
[19:31] <powersj> is a better design decision*
[19:45] <smoser> i think it is a better design decision. because in the end for doing anything complex you fight the shell more than it helps you.
[19:45] <smoser> and... as shown if you want to invoke a shell, that is easy.
[19:47] <powersj> smoser: ok that is fair and makes sense when using subprocess calls, what I am trying to grawk is how to take that decision and apply it to my use cases.
[19:48] <powersj> 1) the use of the python ssh library which expects a string
[19:48] <powersj> 2) how I call MIC: https://paste.ubuntu.com/25522742/
[19:49] <smoser> well you can just call mic with a list
[19:49] <smoser> mount-image-callback image -- command
[19:49] <smoser> mount-image-callback image -- command arg1 arg2
[19:49] <smoser> so 2 is pretty easy
[19:50] <powersj> well, that's not so clear to me
[19:50] <powersj> I can't append a list to it, so I have to process the list more than simply ' '.join(list) right?
[19:51] <powersj> or append the list to the whole command, meaning something like: ['mic', 'image', '--'] + command
[20:05] <smoser> powersj, right.
[20:08] <powersj> smoser: https://paste.ubuntu.com/25523087/ that is where I get stuck
[20:10] <smoser> your /bin/sh is the problem there
[20:10] <smoser> chroot _MOUNTPOINT_ /bin/sh dpkg-query
[20:10] <smoser> you're telling /bin/sh to execute something named 'dpkg-query'
[20:10] <smoser> which is not a file
[20:11] <powersj> and sure enough that works
[20:11] <smoser> so you just have to say:
[20:11] <smoser>  if command_is_a_string:
[20:12] <smoser>      command = ["sh", "-c", command]
[20:12] <smoser> where 'command_is_a_string' is syntax to do that idea
[20:12] <powersj> ok, do you even want to allow strings at this point then?
[20:13] <smoser> we can allow them.
[20:13] <powersj> you mentioned in only certain places in the last review
[20:13] <smoser> its just that they're scary to me
[20:13] <powersj> "lets only fix strings where we were already using sh (via sh -c)."
[20:13] <smoser> right
[20:13] <powersj> so if using sh -c --> convert to string and remove sh -c from it
[20:13] <smoser> so the places where we *were* doing 'sh', '-c'
[20:14] <smoser> it makes sense to just drop the sh -c
[20:14] <smoser> and pass the thing in as a string
[20:14] <smoser> and let the "do the right thing for strings" logic take over
[20:14] <powersj> if we were not doing that, then use as a list
[20:14] <smoser> yeah.
[20:14] <smoser> the one you have theere in that paste has a bug.
[20:14] <smoser> --showformat='${Version}'
[20:14] <smoser> we really wanted
[20:14] <smoser>  --showformat=${Version}
[20:15] <powersj> I'm not so sure those quotes are a bug ;)
[20:15] <powersj> that is getting the version to print it in the logs
[20:15] <smoser> its getting the version with '' around it
[20:15] <powersj> ok so yes not part of the version
[20:15] <powersj> but made the output look nice ;)
[20:16] <powersj> ok one more question to let me get going on this
[20:16] <powersj> you said this command is easily fixed into sh
[20:16] <powersj> cmd = ('for ((i=0;i<{time};i++)); do {test} && exit 0; sleep 1; done; exit 1;'
[20:20] <smoser> the for() is bash syntax.
[20:21] <smoser> cmd = ('i=0; while [ $i -lt {time} ] && i=$(($i+1)); do {test} && exit 0; sleep 1; done')
[20:21] <powersj> ah
[20:21] <smoser> exit 1 at the end.
[20:21] <powersj> ok I'll get these into a new merge for you to look at in the morning
[20:21] <powersj> then we need to talk about SSH
[20:22] <smoser> yeah.
[20:22] <smoser> it is doable
[20:23] <powersj> smoser: thx
[20:45] <smoser> powersj, http://paste.ubuntu.com/25523267/
[20:50] <powersj> smoser: interesting
[20:51] <smoser> http://paste.ubuntu.com/25523299/
[20:51] <smoser> powersj, so whatever you were going to pass to paramiko to execute
[20:52] <smoser> just pass paramiko.execute(shell_pack(cmd))
[20:52] <powersj> smoser: but I can't pass a list
[20:52] <powersj> ;) so convert list to string?
[20:53] <powersj> or add another isinstance(cmd, list) and run that
[20:53] <smoser> you can pass shell_pack() a list or a string. or you can change it to only take a list
[20:53] <smoser> and then do the isinstance business elsewhere.
[20:56] <smoser> to be nice, we can wrap paramiko.execute() with a try/except and then lie about the commadn that we executed
[20:56] <smoser> so that the command that is printed looks like what was executed, rather than
[20:57] <smoser> eval set -- "$(echo J2VjaG8nICcoIGhpIHBvd2Vyc2ogKSc= | base64 --decode)" && exec "$@"
[20:59] <powersj> heh yeah...
[21:01] <dpb1> bad memories there
[21:01] <powersj> one of my biggest complaints is debugging failed commands, especially when they look like lists
[21:01] <dpb1> in my last last job, we had a windows service that did something like this on every incoming ssh attempt.
[21:01] <powersj> so to get a base64 encoding string is even worse
[21:02] <powersj> haha
[21:02] <dpb1> in our case, it was an improvment
[21:02] <dpb1> at least then you could cut and paste a string out
[21:02] <dpb1> instead of a random mix of ASCII and non-ascii characters that windows programs would decide to show you
[21:02] <dpb1> especially quotes
[21:04] <smoser> http://docs.paramiko.org/en/2.2/api/client.html
[21:04] <smoser> that is odd
[21:04] <smoser> it returns a file-like object representing stdin, stdout, stderr
[21:04] <smoser> how does it return stdin ?
[21:08] <rharper> smoser:  https://bugs.launchpad.net/cloud-init/+bug/1716773
[21:24]  * powersj loves the faster unit tests
[21:24] <powersj> thx blackboxsw
[21:25] <blackboxsw> hopefully as we get more that continues to be the case
[21:44] <powersj> smoser: should the shell pack be a separate merge?
[21:44] <powersj> working quite nice so far
[21:51] <powersj> smoser: string merge is ready https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/330535
[22:25] <blackboxsw> grr smoser https://bugs.launchpad.net/cloud-init/+bug/1691489 I just saw this on zesty on my nth redeployment iteration with our SRU candidate
[22:25] <blackboxsw> ubuntu@zesty1:~$ grep reformat /var/log/cloud-init.log 2017-09-12 22:23:52,316 - DataSourceAzure.py[DEBUG]: reformattable=True: partition 1 (/dev/sdb1) on device /dev/disk/cloud/azure_resource was ntfs formatted and had no important files. Safe for reformatting.
[22:25] <blackboxsw> this message came up with a reformated disk on an instance that I had already created
[22:25] <blackboxsw> cloud-init	0.7.9-233-ge586fe35-0ubuntu1~17.04.1
[22:34] <blackboxsw> hrm reproduced multiple times on zesty. not sure what we do w/ that bug then. the "fix" didn't fully resolve  it, but it didn't break cloud-init either.
[22:41] <blackboxsw> ahh nevermind. even though reformatting happens /mnt is still formatted an no errors in cloud-init.log (per the original bug)
[22:42] <blackboxsw> I was taking the ephemeral reformat log as a symptom of failure, but it's an ephemeral disk, reformatting it is okay. Not mounting it is not okay
[22:42] <blackboxsw> ok gotta run