[10:44] <fish_> hi
[10:45] <fish_> I'm building a ubuntu AMI from scratch and use cloud-init. when finished building the AMI, /var/lib/upstart is there but after I boot it's gone and I don't have any upstat logs. wondering if this might be related to cloud-init
[10:46] <Odd_Bloke> fish_: Do you mean /var/log?
[10:48] <fish_> Odd_Bloke: ehh sorry, yes sure /var/log/upstart
[10:49] <Odd_Bloke> fish_: I would be surprised if cloud-init were doing anything to /var/log/upstart, but smoser could probably tell you more.
[10:50] <fish_> Odd_Bloke: hrm okay, any ideas what might cause it beside cloud-init?
[10:50] <fish_> I found this: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/990102 - unfortunately (and a great example why important) it doesn't mention the commit this was fixed
[10:52] <Odd_Bloke> fish_: Well, that wasn't a cloud-init bug, it was a problem with the cloud images.
[10:53] <Odd_Bloke> fish_: Which was fixed ~3 years ago. :)
[10:53] <fish_> Odd_Bloke: I know, I just ask here because cloud-init is the only thing I *expect* to change stuff on the system on first boot :)
[10:53] <fish_> but yes, there is probably something else tampering with /var/log on first? boot
[10:54] <Odd_Bloke> fish_: Are you sure /var/log/upstart is actually included in your image?
[10:54] <fish_> Odd_Bloke: I know the bug is old, but I suspect it's the same root cause as in my cause (they tried the same - building AMIs)
[10:54] <fish_> Odd_Bloke: yes, I'm I double checked that
[10:55] <Odd_Bloke> fish_: Out of interest, why not build your image based on the images at cloud-images.ubuntu.com?
[10:56] <fish_> Odd_Bloke: well, I want to actually build the images (vs starting a instance and snapshotting) - I actually wrote a few words about why (+how) here: http://5pi.de/2015/03/13/building-aws-amis-from-scratch/ but tl;dr
[10:57] <fish_> I want the AMIs small and a clean separation between run and built-time
[10:57] <Odd_Bloke> fish_: Right, but why not take the images from there and modify them by mounting them?
[10:59] <fish_> Odd_Bloke: that's an option.. deboostrap seemed more straight forward. do you think there are advantages in using the cloud-images directly? also worried about security updates because I want the amis to be pretty much immutable (well, at least I don't want to run config management etc on top to keep things up to date)
[11:00] <fish_> but I just realize that I'm wrong, the resulting image indeed has no /var/log/upstart.. now I'm confused since I added a 'mkdir /var/log/upstart' to my build process which failed because it was there already.. looks like something during the build removes it
[11:01] <Odd_Bloke> fish_: Well, using the Ubuntu images would (probably) have saved you from this bug, for example.
[11:01] <Odd_Bloke> And you presumably have the same problem with security updates whoever is building the images?
[11:02] <Odd_Bloke> In your modification process, an "apt-get -y update; apt-get -y upgrade" would pull in any security updates the same as installing from scratch.
[11:03] <fish_> hrm yes that's true.. well, I simply used debootstrap because it seemed like the right tool to use. but yeah, it's definitely a good option to use the cloud-images, will consider that
[12:57] <smoser> fish_, generally speaking, i really dont think you should build your own images.
[12:58] <smoser> any more than i think you should build your own kernel, or your own python or elibc.
[12:58] <smoser> you're certainly welcome to do it, and tools are availalbe to do so.  but doing so means you get to re-discover bugs which are fixed.
[12:59] <smoser> the process that I would recommend, is
[12:59] <smoser>  * download cloud image
[13:00] <smoser>  * mount-image-callback --system-mounts --system-resolvconf $IMAGE -- chroot _MOUNTPOINT bash -s < your-update-script
[13:00] <smoser> and 'your-update-script' does things like: 
[13:00] <smoser>  apt-get update
[13:00] <smoser>  apt-get install foo
[13:00] <smoser>  apt-get clean
[13:01] <smoser> the above does require root, which is less than ideal, but 
[13:01] <smoser> a.) if you don't trust the ubuntu images, you're kind of SOL anyway
[13:01] <smoser> b.) you can just run that in a VM to alleviate potential mount based attacks.
[13:15] <fish_> smoser: yes, the more I think about that the more it makes sense. for now, things are working and I'm about to roll that out, but in the next iteration I'll definitely look into that
[13:16] <fish_> I use a intermediate "base" image to avoid running a fresh deboostrap for every update anyways, so it should be fairely easy to use a cloud-image instead
[13:20] <smoser> fwiw, the maas-images build process does essentially the above.
[16:46] <harlowja_> claudiupopa i think we might have to remove the channel from https://review.openstack.org/#/c/169293/
[16:46] <harlowja_> its gonna be hard to get an operator in this channel without kicking everyone out (which nobody has the permission to do)
[16:46] <harlowja_> so might have to just skip that part
[16:47] <harlowja_> years ago i think we all forgot to setup this channel with an operator and its pretty hard to do it post-creation
[16:49] <harlowja_> *afaik*
[16:52] <smoser> harlowja_, why didn't you ever add 'tar' to the write_files. and http:// 
[16:52] <smoser> s/you/me/
[16:52] <harlowja_> hmmm
[16:52] <harlowja_> or was that me?
[16:52] <harlowja_> idk
[16:52] <harlowja_> haha
[16:52] <smoser> ie, would be nice to have write_files either read content from a url
[16:53] <smoser> or read a tarball from url and extract it into a target dir.
[16:53]  * harlowja_ looking
[16:53] <smoser> http://paste.ubuntu.com/10691441/
[16:54] <smoser> that is what i have, but having those 2 big blobs as 'path: http://' would have been nice.
[16:54] <smoser> and then also:
[16:54] <smoser>  path: http://
[16:54] <smoser>  format: tar
[16:54] <smoser>  extract-dir: /writable/user-data/cloud-init
[16:54] <smoser> or something
[16:54] <harlowja_> hmmm, ya why did we do that, ha
[16:56] <harlowja_> msg: ":::::: Hi Mom :::::::"
[16:56] <harlowja_> lol
[16:56] <harlowja_> hmmmm
[16:57] <harlowja_> seems like we should just do that... (allow url stufF)
[16:57] <harlowja_> and tar
[16:57] <harlowja_> guess maybe we just didn't think of it ?
[17:09] <smoser> yeah, i think we just didnt. 
[17:09] <smoser> the inline makes sense.
[17:09] <smoser> as you may not have networking at that point.
[17:09] <smoser> but if you do have networking, then http:// makes good sense.
[17:09] <smoser> and tar is just a nice archive format :)
[17:17] <harlowja_> :-P
[17:17] <harlowja_> shall i code that up or u want to?
[17:17] <harlowja_> pop out the codes
[17:17] <harlowja_> lol
[17:37] <harlowja_> smoser ^
[19:12] <harlowja_> smoser https://code.launchpad.net/~harlowja/cloud-init/write-files-fetch-from-somewhere/+merge/254816
[19:12] <harlowja_> ok thats part 1 (no tar)
[19:12] <harlowja_> should be mostly ok, ha
[19:13] <smoser> python3... 
[19:13] <harlowja_> ya, durn it
[19:13] <smoser> pre_content comes as binary... pretty sure (or we ant to make sure it is)
[19:13] <smoser> and then we want to write content without conversion
[19:14] <harlowja_> will make sure
[19:14] <harlowja_> load_tfile_or_url does that i guess
[19:14] <smoser> tfile loads text
[19:14] <smoser> ew want blobs of unadulterated raw data
[19:14] <harlowja_> kk, let me check here
[19:14] <harlowja_> stupid stff
[19:14] <harlowja_> lol
[19:14] <harlowja_> *stuff
[19:15]  * harlowja_ goes and builds up my 3.4 venv
[19:19] <smoser> harlowja_, see why i said i hate pypi
[19:19] <harlowja_> lol
[19:19] <harlowja_> ya
[19:19] <smoser> oh darn, some dude's cable modem is out
[19:19] <smoser> pypi fail
[19:19] <harlowja_> lol
[19:47] <harlowja_> ok dokie; added some tests and stuff; seems to work as expected
[19:48] <harlowja_> at least under basic tests
[19:51] <harlowja_> smoser check that out if u want
[19:57] <smoser> harlowja_, thanks.
[19:58] <harlowja_> np
[20:19] <Odd_Bloke> smoser: Am I right in thinking that vendor-data simply isn't a thing in the version of cloud-init in precise?
[20:28] <smoser> right . not at all.
[20:28] <smoser> Odd_Bloke, i'd sponsor an sru
[20:28] <harlowja_> precise is 12.04 right?
[20:28] <Odd_Bloke> harlowja_: Yeah.
[20:28] <harlowja_> k, can't ever remember the codenames, lol
[22:07] <tmclaugh[work]> I want to set hostname to the aws instance ID.  Is there a var available for doing that ot do I have to use some commands during bootcmd to handle this?