[05:34] script placed in /etc/cloud/cloud.cfg.d/my_script.cfg is not executed during boot up. Did not find any useful info in /var/log/cloud-init.log, is other places I can look for more info? Or what is the best way to troubleshoot this? === smoser changed the topic of #cloud-init to: reviews: http://bit.ly/ci-reviews | mail: http://bit.ly/ci-mail-archive | docs: http://cloudinit.readthedocs.io/en/latest/ | next meeting Tue, 12 Sep 2017 16:30:00 +0000 [15:17] just to have them in here: these are my two MPs for KVM [15:17] commands as strings: https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/330535 [15:17] add xkvm: https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/330536 [15:32] xkvm? [15:44] wrapper around qemu command [15:44] rharper/smoser created for curtin's use [15:45] since qemu's parameters make little/no sense to most people, dpb1 :) [15:45] people who aren't cpaelzer [15:46] powersj: TIL, thx [16:05] rharper, https://blueprints.launchpad.net/nova/+spec/virt-device-tagged-attach-detach [16:06] that is what i think i remember. [16:06] \o/ === smoser changed the topic of #cloud-init to: reviews: http://bit.ly/ci-reviews | mail: http://bit.ly/ci-mail-archive | docs: http://cloudinit.readthedocs.io/en/latest/ | next meeting Mon, 18 Sep 2017 16:00:00 +0000 [17:33] smoser: in that spec, it mentions the qemu firmware exposing metadata path; we have support for that now, right? the nocloud from smbios? do you think that's the same? [17:34] "Outside the scope of the Nova work, a simple tool will be created that can parse this metadata file and set tags against devices in the udev database. It is anticipated that cloud-init would trigger this tool. " [17:49] rharper, ? [17:49] i dont follow [17:50] I know we added some dmi path to getting datasource, or user-data, right ? [17:50] the DMI NoCloud mode ? [17:51] I was wondering if that was what the spec was referring to w.r.t qemu exposing metadata via firmware to instances; the spec has a general concern about providing network-based metadata to instances which may not have network configuration (the writers were unaware of how cloud-init handles this in other network-based datasources) [17:52] and the second quote was just an interesting tidbit; the use-case of 'tagging' devices for service discovery (sdc is for 'oracledb') and wanted to provide a more "General" way for applications to discover the tagging without forcing the app to learn "openstack device tagging json schema"; and suggested udev tags/attrs as such a method [17:53] well, the dmi nocloud stuff is not related to the openstack stuff for sure. [17:53] but it [17:54] but it was provided by qemu (aka firmware) [17:54] we just now support reading nocloud info from dmi [17:54] yeah [17:54] I wonder if it could be a ConfigDrive style support if they could stuff enough info in there [17:55] I guess it's only useful during boot; [17:55] hence NoCloud; I don't suppose qemu has an interface to update values in the dmi table dynamically [17:56] powersj, you just replaced my mp ? [17:57] https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/330459 -> https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/330535 [17:57] Based on your comment it seemed like you wanted a seperate MP. So we can merge yours and then put mine on top to change the other strings. [17:57] oh i see you added some of the string changes too [17:57] yeah [18:34] powersj, i'm sorry. [18:34] ? [18:38] smoser: I assume the sorry is for the review? [18:42] yeah [18:50] smoser: for KVM when we execute SSH expects a string, and mount-image-callback expect a string [18:50] should I then do the opposite as LXD, in that if it detects it is a of type array, convert it to a string? [19:08] powersj, mount-image-callback doesn't expect a string. it has a sane interface. [19:08] ssh interface sucks [19:09] it shoves you through sh (or rather the user's shell) more for security reasons than anything else. [19:09] that way the platform can manage what your user can do by setting the shell differently. [19:10] smoser: the way I was calling MIC https://paste.ubuntu.com/25522742/ [19:10] to fix ssh, you can do this: [19:10] https://gist.github.com/smoser/88a5a77ab0debf268b945d46314ea447 [19:10] you have to be creative. [19:10] (that does insert a dependency on 'base64', which i dont have a good way to work around) [19:12] sigh [19:13] smoser: I guess I'm not sure I understand why the usage of the python library doesn't work [19:14] that is basically what we use in tools/run-centos : inside_as [19:14] what python library ? i'ms orry. [19:14] the integration tests are in python [19:14] I am using paramiko to call ssh [19:14] from what you are saying you want me to get rid of that to call your ussh shell script [19:24] powersj, no. using paramiko is fine [19:24] we just wrap it sanely. [19:26] smoser: define wrapping sanely :) outside of your base64 encoded strings this is a new to me [19:26] :) [19:26] i want to provide an interface '.execute()' that takes a list. not a string. [19:27] (just like subprocess.call() does... its not a crazy idea) [19:27] powersj: you are familiar with pep8, right? [19:27] the pep, not the tool [19:27] dpb1: of course [19:27] ok [19:31] smoser: you are choosing to pass a list to replicate the behavior or because you feel passing the list is a more design decision? [19:31] is a better design decision* [19:45] i think it is a better design decision. because in the end for doing anything complex you fight the shell more than it helps you. [19:45] and... as shown if you want to invoke a shell, that is easy. [19:47] smoser: ok that is fair and makes sense when using subprocess calls, what I am trying to grawk is how to take that decision and apply it to my use cases. [19:48] 1) the use of the python ssh library which expects a string [19:48] 2) how I call MIC: https://paste.ubuntu.com/25522742/ [19:49] well you can just call mic with a list [19:49] mount-image-callback image -- command [19:49] mount-image-callback image -- command arg1 arg2 [19:49] so 2 is pretty easy [19:50] well, that's not so clear to me [19:50] I can't append a list to it, so I have to process the list more than simply ' '.join(list) right? [19:51] or append the list to the whole command, meaning something like: ['mic', 'image', '--'] + command [20:05] powersj, right. [20:08] smoser: https://paste.ubuntu.com/25523087/ that is where I get stuck [20:10] your /bin/sh is the problem there [20:10] chroot _MOUNTPOINT_ /bin/sh dpkg-query [20:10] you're telling /bin/sh to execute something named 'dpkg-query' [20:10] which is not a file [20:11] and sure enough that works [20:11] so you just have to say: [20:11] if command_is_a_string: [20:12] command = ["sh", "-c", command] [20:12] where 'command_is_a_string' is syntax to do that idea [20:12] ok, do you even want to allow strings at this point then? [20:13] we can allow them. [20:13] you mentioned in only certain places in the last review [20:13] its just that they're scary to me [20:13] "lets only fix strings where we were already using sh (via sh -c)." [20:13] right [20:13] so if using sh -c --> convert to string and remove sh -c from it [20:13] so the places where we *were* doing 'sh', '-c' [20:14] it makes sense to just drop the sh -c [20:14] and pass the thing in as a string [20:14] and let the "do the right thing for strings" logic take over [20:14] if we were not doing that, then use as a list [20:14] yeah. [20:14] the one you have theere in that paste has a bug. [20:14] --showformat='${Version}' [20:14] we really wanted [20:14] --showformat=${Version} [20:15] I'm not so sure those quotes are a bug ;) [20:15] that is getting the version to print it in the logs [20:15] its getting the version with '' around it [20:15] ok so yes not part of the version [20:15] but made the output look nice ;) [20:16] ok one more question to let me get going on this [20:16] you said this command is easily fixed into sh [20:16] cmd = ('for ((i=0;i<{time};i++)); do {test} && exit 0; sleep 1; done; exit 1;' [20:20] the for() is bash syntax. [20:21] cmd = ('i=0; while [ $i -lt {time} ] && i=$(($i+1)); do {test} && exit 0; sleep 1; done') [20:21] ah [20:21] exit 1 at the end. [20:21] ok I'll get these into a new merge for you to look at in the morning [20:21] then we need to talk about SSH [20:22] yeah. [20:22] it is doable [20:23] smoser: thx [20:45] powersj, http://paste.ubuntu.com/25523267/ [20:50] smoser: interesting [20:51] http://paste.ubuntu.com/25523299/ [20:51] powersj, so whatever you were going to pass to paramiko to execute [20:52] just pass paramiko.execute(shell_pack(cmd)) [20:52] smoser: but I can't pass a list [20:52] ;) so convert list to string? [20:53] or add another isinstance(cmd, list) and run that [20:53] you can pass shell_pack() a list or a string. or you can change it to only take a list [20:53] and then do the isinstance business elsewhere. [20:56] to be nice, we can wrap paramiko.execute() with a try/except and then lie about the commadn that we executed [20:56] so that the command that is printed looks like what was executed, rather than [20:57] eval set -- "$(echo J2VjaG8nICcoIGhpIHBvd2Vyc2ogKSc= | base64 --decode)" && exec "$@" [20:59] heh yeah... [21:01] bad memories there [21:01] one of my biggest complaints is debugging failed commands, especially when they look like lists [21:01] in my last last job, we had a windows service that did something like this on every incoming ssh attempt. [21:01] so to get a base64 encoding string is even worse [21:02] haha [21:02] in our case, it was an improvment [21:02] at least then you could cut and paste a string out [21:02] instead of a random mix of ASCII and non-ascii characters that windows programs would decide to show you [21:02] especially quotes [21:04] http://docs.paramiko.org/en/2.2/api/client.html [21:04] that is odd [21:04] it returns a file-like object representing stdin, stdout, stderr [21:04] how does it return stdin ? [21:08] smoser: https://bugs.launchpad.net/cloud-init/+bug/1716773 [21:08] Ubuntu bug 1716773 in cloud-init "cloud-init doesn't cache network_config property in cache" [Undecided,New] [21:24] * powersj loves the faster unit tests [21:24] thx blackboxsw [21:25] hopefully as we get more that continues to be the case [21:44] smoser: should the shell pack be a separate merge? [21:44] working quite nice so far [21:51] smoser: string merge is ready https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/330535 === dhill__ is now known as dhill_ [22:25] grr smoser https://bugs.launchpad.net/cloud-init/+bug/1691489 I just saw this on zesty on my nth redeployment iteration with our SRU candidate [22:25] Ubuntu bug 1691489 in cloud-init (Ubuntu Yakkety) "fstab entries written by cloud-config may not be mounted" [Medium,Confirmed] [22:25] ubuntu@zesty1:~$ grep reformat /var/log/cloud-init.log 2017-09-12 22:23:52,316 - DataSourceAzure.py[DEBUG]: reformattable=True: partition 1 (/dev/sdb1) on device /dev/disk/cloud/azure_resource was ntfs formatted and had no important files. Safe for reformatting. [22:25] this message came up with a reformated disk on an instance that I had already created [22:25] cloud-init 0.7.9-233-ge586fe35-0ubuntu1~17.04.1 [22:34] hrm reproduced multiple times on zesty. not sure what we do w/ that bug then. the "fix" didn't fully resolve it, but it didn't break cloud-init either. [22:41] ahh nevermind. even though reformatting happens /mnt is still formatted an no errors in cloud-init.log (per the original bug) [22:42] I was taking the ephemeral reformat log as a symptom of failure, but it's an ephemeral disk, reformatting it is okay. Not mounting it is not okay [22:42] ok gotta run === caphrim007 is now known as caphrim007_ === caphrim007_ is now known as caphrim00_ === caphrim00_ is now known as caphrim007