[17:39] so once upon a time, I had a multipart mime cloudinit script in ec2 userdata that worked just fine on both lucid and precise: http://pastebin.com/8WPmCmAQ [17:40] on saucy, however, it appears to do... nothing. [17:41] the log output on saucy: http://pastebin.com/PY2jeRSk [17:43] the text/cloud-init section of the mime blob does not seem to have been written out anywhere under /var/lib/cloud, which I'm guessing is the problem? But it's unclear to me why. [17:44] any pointers, even if to the relevant sections of TFM, would be appreciated. :) [18:02] * Dr_Memory drops a pin :) [18:15] sssht! in the changelog i see one -maybe- relevant piece touched between precise and saucy, might want to have a closer look at https://bugs.launchpad.net/cloud-init/+bug/1065116 [18:16] this channel isn't very active, btw, patience helps. or a bug, mail, .. [18:25] kwadronaut: thanks. I realized after pin-dropping that most ppl here were on the plus side of GMT, so I will be patient. :) [18:29] (also not sure 1065116 is relevant on first blush -- I don't see a KeyError anywhere in cloud-init.log) [18:32] I took that pin and threw it in a haystack. but you're probably right, wrong guess of me. [19:43] also i think that smoser and i will be travelling soon to, so it might get even more quiet [19:45] Dr_Memory i'd be useful to know whats in /var/lib/cloud/instances/i-fa2e879e/user-data.txt [19:45] if u want to share [19:45] harlowja: just a sec [19:46] the debug logs look ok, just as u said, not seeing any mime writing really [19:46] or much other file writing [19:46] which seems off [19:47] Dr_Memory, it doen't look terribly unreasonable. [19:47] harlowja: just a sec [19:47] http://pastebin.com/WwDY5MTZ [19:48] (I've been fiddling a bit and this is a new instance from the last one I tried, but the behavior has been the same. [19:48] ) [19:48] user-data.txt.i [19:48] (possibly of note: there is no newline at the end of the last mime header there?) [19:48] ls: cannot access /var/lib/cloud/instance/user-data.txt.i: No such file or directory [19:50] Dr_Memory, /var/log/cloud-init.klog [19:50] s/klog/log [19:51] ya, the full contents of that log file should give us the dirty details [19:51] *or not so dirty [19:52] just a sec :) [19:53] the log: http://pastebin.com/BTjQw2Vg [19:53] or rather, that's the log from the last reboot -- if you like I can spawn a new instance and get a clean log [19:55] I tried, in desperation, putting the "cloud_final_modules:" stanza into /etc/cloud/cloud.cfg.d/99_final.cfg (and then rm -rf /var/lib/cloud; reboot), which is why you see it referenced in the logs there. [19:56] smoser: just pinged you privately [19:57] np dr. a clean instance log might be the best i think [19:57] harlowja: figured. on moment, por favor :) [19:58] (cloudformation is thinking... thinking... thinking...) [19:59] more thinkin [19:59] (AWS gears grind slowly, but they grind exceedingly... slow.) [20:01] okay, virgin log: http://pastebin.com/FrpZp8ir [20:02] user-data.txt still the same. user-data.txt.i still nonexistent. [20:05] damn, who put all this merging logs, hahaha [20:06] :-/ [20:07] like it seems to be doing stuff [20:08] is https://org.blank.bootstrap.s3.amazonaws.co ok? [20:08] not sure what thats supposed to return for u [20:08] yeah, I can manually grab that URL with curl/wget [20:08] it returns a #!/bin/sh file [20:08] k [20:11] * harlowja scanning codes [20:12] like so: [20:12] http://pastebin.com/kififAdN [20:14] sure [20:14] huh. I wonder if some library got more strict about certificate alt name validation between precise and saucy? [20:15] although there's nothing in the cloudinit log to suggest that fetching that url failed [20:16] ya [20:16] it seems to have worked, although it didn't write out the right file [20:17] what does 'ls /var/lib/cloud/instance/' show? [20:17] or 'find /var/lib/cloud/instance/' [20:17] holy shit [20:17] I just s/https/http/ [20:17] ...and it worked [20:18] hmmmm [20:18] *facepalm* [20:18] if u also try #include-once this will write out a file [20:18] that file writing should showup in the log [20:19] just #include just loads it into memory [20:19] *no log [20:19] ah, interesting [20:19] although ssh/not ssh is odd [20:19] *https/not https [20:20] (brainfart) [20:20] well, S3 signed urls are kinda special [20:20] they end up being https://bucketname.s3.amazonaws.com/ [20:20] hmmm [20:20] and if your bucketname has dots in it: https://bucket.name.s3.amazonaws.com/ [20:20] which doesn't match their wildcard cert [20:21] hehe, u out of my area of expertise :-P openstack doesn't have these ;) [20:22] is this worth a bug report? [20:23] smoser what do u think [20:23] arguably "don't provide https include urls to sites that don't provide a correct certificate" is a "doctor it hurts when I do this" issue [20:23] it does seem like odd behavior [20:23] but the logging could definitely have been better in this case :( [20:23] def [20:23] and the behavior is, if not a regression, certainly a change [20:24] ya, we switched to using the requests library, which is more strict about this stuff [20:24] although i would have expected some log to [20:25] i'd be fine with a bug, but smoser might think otherwise [20:25] http://www.python-requests.org/en/latest/ [20:39] what was the issue there ? [20:39] you effectively had an invalid cert ? [20:40] smoser: not 100% sure, but when I changed the #include url from https://org.blank.bootstrap.s3.amazon.com/ to http:// suddenly everything just worked. [20:40] and yes: amazon has a wildcard cert for *.s3.amazonaws.com but wildcard certs AFAIK can't cover sub-subdomains. [20:49] the can i think [20:49] it'd seem like possibly a bug in python-requests [20:49] or possibly its doing the right thing [20:49] i'm not sure. [20:51] yeah [20:52] if nothing else, I feel like the log should have told us a bit more about what was going on