[17:39] <Dr_Memory> so once upon a time, I had a multipart mime cloudinit script in ec2 userdata that worked just fine on both lucid and precise: http://pastebin.com/8WPmCmAQ
[17:40] <Dr_Memory> on saucy, however, it appears to do... nothing.
[17:41] <Dr_Memory> the log output on saucy: http://pastebin.com/PY2jeRSk
[17:43] <Dr_Memory> the text/cloud-init section of the mime blob does not seem to have been written out anywhere under /var/lib/cloud, which I'm guessing is the problem?  But it's unclear to me why.
[17:44] <Dr_Memory> any pointers, even if to the relevant sections of TFM, would be appreciated. :)
[18:02]  * Dr_Memory drops a pin :)
[18:15] <kwadronaut> sssht! in the changelog i see one -maybe- relevant piece touched between precise and saucy, might want to have a closer look at https://bugs.launchpad.net/cloud-init/+bug/1065116
[18:16] <kwadronaut> this channel isn't very active, btw, patience helps. or a bug, mail, ..
[18:25] <Dr_Memory> kwadronaut: thanks.  I realized after pin-dropping that most ppl here were on the plus side of GMT, so I will be patient. :)
[18:29] <Dr_Memory> (also not sure 1065116 is relevant on first blush -- I don't see a KeyError anywhere in cloud-init.log)
[18:32] <kwadronaut> I took that pin and threw it in a haystack. but you're probably right, wrong guess of me.
[19:43] <harlowja> also i think that smoser and i will be travelling soon to, so it might get even more quiet
[19:45] <harlowja> Dr_Memory i'd be useful to know whats in /var/lib/cloud/instances/i-fa2e879e/user-data.txt
[19:45] <harlowja> if u want to share
[19:45] <Dr_Memory> harlowja: just a sec
[19:46] <harlowja> the debug logs look ok, just as u said, not seeing any mime writing really
[19:46] <harlowja> or much other file writing
[19:46] <harlowja> which seems off
[19:47] <smoser> Dr_Memory, it doen't look terribly unreasonable.
[19:47] <Dr_Memory> harlowja: just a sec
[19:47] <Dr_Memory> http://pastebin.com/WwDY5MTZ
[19:48] <Dr_Memory> (I've been fiddling a bit and this is a new instance from the last one I tried, but the behavior has been the same.
[19:48] <Dr_Memory> )
[19:48] <smoser> user-data.txt.i
[19:48] <Dr_Memory> (possibly of note: there is no newline at the end of the last mime header there?)
[19:48] <Dr_Memory> ls: cannot access /var/lib/cloud/instance/user-data.txt.i: No such file or directory
[19:50] <smoser> Dr_Memory, /var/log/cloud-init.klog
[19:50] <smoser> s/klog/log
[19:51] <harlowja> ya, the full contents of that log file should give us the dirty details
[19:51] <harlowja> *or not so dirty
[19:52] <Dr_Memory> just a sec :)
[19:53] <Dr_Memory> the log: http://pastebin.com/BTjQw2Vg
[19:53] <Dr_Memory> or rather, that's the log from the last reboot -- if you like I can spawn a new instance and get a clean log
[19:55] <Dr_Memory> I tried, in desperation, putting the "cloud_final_modules:" stanza into /etc/cloud/cloud.cfg.d/99_final.cfg (and then rm -rf /var/lib/cloud; reboot), which is why you see it referenced in the logs there.
[19:56] <utlemming> smoser: just pinged you privately
[19:57] <harlowja> np dr. a clean instance log might be the best i think
[19:57] <Dr_Memory> harlowja: figured.  on moment, por favor :)
[19:58] <Dr_Memory> (cloudformation is thinking... thinking... thinking...)
[19:59] <harlowja> more thinkin
[19:59] <Dr_Memory> (AWS gears grind slowly, but they grind exceedingly... slow.)
[20:01] <Dr_Memory> okay, virgin log: http://pastebin.com/FrpZp8ir
[20:02] <Dr_Memory> user-data.txt still the same.  user-data.txt.i still nonexistent.
[20:05] <harlowja> damn, who put all this merging logs, hahaha
[20:06] <harlowja> :-/
[20:07] <harlowja> like it seems to be doing stuff
[20:08] <harlowja> is https://org.blank.bootstrap.s3.amazonaws.co ok?
[20:08] <harlowja> not sure what thats supposed to return for u
[20:08] <Dr_Memory> yeah, I can manually grab that URL with curl/wget
[20:08] <Dr_Memory> it returns a #!/bin/sh file
[20:08] <harlowja> k
[20:11]  * harlowja scanning codes
[20:12] <Dr_Memory> like so: 
[20:12] <Dr_Memory> http://pastebin.com/kififAdN
[20:14] <harlowja> sure
[20:14] <Dr_Memory> huh.  I wonder if some library got more strict about certificate alt name validation between precise and saucy?
[20:15] <Dr_Memory> although there's nothing in the cloudinit log to suggest that fetching that url failed
[20:16] <harlowja> ya
[20:16] <harlowja> it seems to have worked, although it didn't write out the right file
[20:17] <harlowja> what does 'ls /var/lib/cloud/instance/' show?
[20:17] <harlowja> or 'find  /var/lib/cloud/instance/'
[20:17] <Dr_Memory> holy shit
[20:17] <Dr_Memory> I just s/https/http/
[20:17] <Dr_Memory> ...and it worked
[20:18] <harlowja> hmmmm
[20:18] <Dr_Memory> *facepalm*
[20:18] <harlowja> if u also try #include-once this will write out a file
[20:18] <harlowja> that file writing should showup in the log
[20:19] <harlowja> just #include just loads it into memory
[20:19] <harlowja> *no log
[20:19] <Dr_Memory> ah, interesting
[20:19] <harlowja> although ssh/not ssh is odd
[20:19] <harlowja> *https/not https
[20:20] <harlowja> (brainfart)
[20:20] <Dr_Memory> well, S3 signed urls are kinda special
[20:20] <Dr_Memory> they end up being https://bucketname.s3.amazonaws.com/
[20:20] <harlowja> hmmm
[20:20] <Dr_Memory> and if your bucketname has dots in it: https://bucket.name.s3.amazonaws.com/
[20:20] <Dr_Memory> which doesn't match their wildcard cert
[20:21] <harlowja> hehe, u out of my area of expertise :-P openstack doesn't have these ;)
[20:22] <Dr_Memory> is this worth a bug report?
[20:23] <harlowja> smoser what do u think
[20:23] <Dr_Memory> arguably "don't provide https include urls to sites that don't provide a correct certificate" is a "doctor it hurts when I do this" issue 
[20:23] <harlowja> it does seem like odd behavior
[20:23] <Dr_Memory> but the logging could definitely have been better in this case :(
[20:23] <harlowja> def
[20:23] <Dr_Memory> and the behavior is, if not a regression, certainly a change
[20:24] <harlowja> ya, we switched to using the requests library, which is more strict about this stuff
[20:24] <harlowja> although i would have expected some log to
[20:25] <harlowja> i'd be fine with a bug, but smoser might think otherwise
[20:25] <harlowja> http://www.python-requests.org/en/latest/
[20:39] <smoser> what was the issue there ?
[20:39] <smoser> you effectively had an invalid cert ?
[20:40] <Dr_Memory> smoser: not 100% sure, but when I changed the #include url from https://org.blank.bootstrap.s3.amazon.com/ to http:// suddenly everything just worked.
[20:40] <Dr_Memory> and yes: amazon has a wildcard cert for *.s3.amazonaws.com but wildcard certs AFAIK can't cover sub-subdomains.
[20:49] <smoser> the can i think
[20:49] <smoser> it'd seem like possibly a bug in python-requests
[20:49] <smoser> or possibly its doing the right thing
[20:49] <smoser> i'm not sure.
[20:51] <Dr_Memory> yeah
[20:52] <Dr_Memory> if nothing else, I feel like the log should have told us a bit more about what was going on