| Dr_Memory | so once upon a time, I had a multipart mime cloudinit script in ec2 userdata that worked just fine on both lucid and precise: http://pastebin.com/8WPmCmAQ | 17:39 |
|---|---|---|
| Dr_Memory | on saucy, however, it appears to do... nothing. | 17:40 |
| Dr_Memory | the log output on saucy: http://pastebin.com/PY2jeRSk | 17:41 |
| Dr_Memory | the text/cloud-init section of the mime blob does not seem to have been written out anywhere under /var/lib/cloud, which I'm guessing is the problem? But it's unclear to me why. | 17:43 |
| Dr_Memory | any pointers, even if to the relevant sections of TFM, would be appreciated. :) | 17:44 |
| * Dr_Memory drops a pin :) | 18:02 | |
| kwadronaut | sssht! in the changelog i see one -maybe- relevant piece touched between precise and saucy, might want to have a closer look at https://bugs.launchpad.net/cloud-init/+bug/1065116 | 18:15 |
| kwadronaut | this channel isn't very active, btw, patience helps. or a bug, mail, .. | 18:16 |
| Dr_Memory | kwadronaut: thanks. I realized after pin-dropping that most ppl here were on the plus side of GMT, so I will be patient. :) | 18:25 |
| Dr_Memory | (also not sure 1065116 is relevant on first blush -- I don't see a KeyError anywhere in cloud-init.log) | 18:29 |
| kwadronaut | I took that pin and threw it in a haystack. but you're probably right, wrong guess of me. | 18:32 |
| harlowja | also i think that smoser and i will be travelling soon to, so it might get even more quiet | 19:43 |
| harlowja | Dr_Memory i'd be useful to know whats in /var/lib/cloud/instances/i-fa2e879e/user-data.txt | 19:45 |
| harlowja | if u want to share | 19:45 |
| Dr_Memory | harlowja: just a sec | 19:45 |
| harlowja | the debug logs look ok, just as u said, not seeing any mime writing really | 19:46 |
| harlowja | or much other file writing | 19:46 |
| harlowja | which seems off | 19:46 |
| smoser | Dr_Memory, it doen't look terribly unreasonable. | 19:47 |
| Dr_Memory | harlowja: just a sec | 19:47 |
| Dr_Memory | http://pastebin.com/WwDY5MTZ | 19:47 |
| Dr_Memory | (I've been fiddling a bit and this is a new instance from the last one I tried, but the behavior has been the same. | 19:48 |
| Dr_Memory | ) | 19:48 |
| smoser | user-data.txt.i | 19:48 |
| Dr_Memory | (possibly of note: there is no newline at the end of the last mime header there?) | 19:48 |
| Dr_Memory | ls: cannot access /var/lib/cloud/instance/user-data.txt.i: No such file or directory | 19:48 |
| smoser | Dr_Memory, /var/log/cloud-init.klog | 19:50 |
| smoser | s/klog/log | 19:50 |
| harlowja | ya, the full contents of that log file should give us the dirty details | 19:51 |
| harlowja | *or not so dirty | 19:51 |
| Dr_Memory | just a sec :) | 19:52 |
| Dr_Memory | the log: http://pastebin.com/BTjQw2Vg | 19:53 |
| Dr_Memory | or rather, that's the log from the last reboot -- if you like I can spawn a new instance and get a clean log | 19:53 |
| Dr_Memory | I tried, in desperation, putting the "cloud_final_modules:" stanza into /etc/cloud/cloud.cfg.d/99_final.cfg (and then rm -rf /var/lib/cloud; reboot), which is why you see it referenced in the logs there. | 19:55 |
| utlemming | smoser: just pinged you privately | 19:56 |
| harlowja | np dr. a clean instance log might be the best i think | 19:57 |
| Dr_Memory | harlowja: figured. on moment, por favor :) | 19:57 |
| Dr_Memory | (cloudformation is thinking... thinking... thinking...) | 19:58 |
| harlowja | more thinkin | 19:59 |
| Dr_Memory | (AWS gears grind slowly, but they grind exceedingly... slow.) | 19:59 |
| Dr_Memory | okay, virgin log: http://pastebin.com/FrpZp8ir | 20:01 |
| Dr_Memory | user-data.txt still the same. user-data.txt.i still nonexistent. | 20:02 |
| harlowja | damn, who put all this merging logs, hahaha | 20:05 |
| harlowja | :-/ | 20:06 |
| harlowja | like it seems to be doing stuff | 20:07 |
| harlowja | is https://org.blank.bootstrap.s3.amazonaws.co ok? | 20:08 |
| harlowja | not sure what thats supposed to return for u | 20:08 |
| Dr_Memory | yeah, I can manually grab that URL with curl/wget | 20:08 |
| Dr_Memory | it returns a #!/bin/sh file | 20:08 |
| harlowja | k | 20:08 |
| * harlowja scanning codes | 20:11 | |
| Dr_Memory | like so: | 20:12 |
| Dr_Memory | http://pastebin.com/kififAdN | 20:12 |
| harlowja | sure | 20:14 |
| Dr_Memory | huh. I wonder if some library got more strict about certificate alt name validation between precise and saucy? | 20:14 |
| Dr_Memory | although there's nothing in the cloudinit log to suggest that fetching that url failed | 20:15 |
| harlowja | ya | 20:16 |
| harlowja | it seems to have worked, although it didn't write out the right file | 20:16 |
| harlowja | what does 'ls /var/lib/cloud/instance/' show? | 20:17 |
| harlowja | or 'find /var/lib/cloud/instance/' | 20:17 |
| Dr_Memory | holy shit | 20:17 |
| Dr_Memory | I just s/https/http/ | 20:17 |
| Dr_Memory | ...and it worked | 20:17 |
| harlowja | hmmmm | 20:18 |
| Dr_Memory | *facepalm* | 20:18 |
| harlowja | if u also try #include-once this will write out a file | 20:18 |
| harlowja | that file writing should showup in the log | 20:18 |
| harlowja | just #include just loads it into memory | 20:19 |
| harlowja | *no log | 20:19 |
| Dr_Memory | ah, interesting | 20:19 |
| harlowja | although ssh/not ssh is odd | 20:19 |
| harlowja | *https/not https | 20:19 |
| harlowja | (brainfart) | 20:20 |
| Dr_Memory | well, S3 signed urls are kinda special | 20:20 |
| Dr_Memory | they end up being https://bucketname.s3.amazonaws.com/ | 20:20 |
| harlowja | hmmm | 20:20 |
| Dr_Memory | and if your bucketname has dots in it: https://bucket.name.s3.amazonaws.com/ | 20:20 |
| Dr_Memory | which doesn't match their wildcard cert | 20:20 |
| harlowja | hehe, u out of my area of expertise :-P openstack doesn't have these ;) | 20:21 |
| Dr_Memory | is this worth a bug report? | 20:22 |
| harlowja | smoser what do u think | 20:23 |
| Dr_Memory | arguably "don't provide https include urls to sites that don't provide a correct certificate" is a "doctor it hurts when I do this" issue | 20:23 |
| harlowja | it does seem like odd behavior | 20:23 |
| Dr_Memory | but the logging could definitely have been better in this case :( | 20:23 |
| harlowja | def | 20:23 |
| Dr_Memory | and the behavior is, if not a regression, certainly a change | 20:23 |
| harlowja | ya, we switched to using the requests library, which is more strict about this stuff | 20:24 |
| harlowja | although i would have expected some log to | 20:24 |
| harlowja | i'd be fine with a bug, but smoser might think otherwise | 20:25 |
| harlowja | http://www.python-requests.org/en/latest/ | 20:25 |
| smoser | what was the issue there ? | 20:39 |
| smoser | you effectively had an invalid cert ? | 20:39 |
| Dr_Memory | smoser: not 100% sure, but when I changed the #include url from https://org.blank.bootstrap.s3.amazon.com/ to http:// suddenly everything just worked. | 20:40 |
| Dr_Memory | and yes: amazon has a wildcard cert for *.s3.amazonaws.com but wildcard certs AFAIK can't cover sub-subdomains. | 20:40 |
| smoser | the can i think | 20:49 |
| smoser | it'd seem like possibly a bug in python-requests | 20:49 |
| smoser | or possibly its doing the right thing | 20:49 |
| smoser | i'm not sure. | 20:49 |
| Dr_Memory | yeah | 20:51 |
| Dr_Memory | if nothing else, I feel like the log should have told us a bit more about what was going on | 20:52 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!