=== harlowja_ is now known as harlowja_away [10:44] hi [10:45] I'm building a ubuntu AMI from scratch and use cloud-init. when finished building the AMI, /var/lib/upstart is there but after I boot it's gone and I don't have any upstat logs. wondering if this might be related to cloud-init [10:46] fish_: Do you mean /var/log? [10:48] Odd_Bloke: ehh sorry, yes sure /var/log/upstart [10:49] fish_: I would be surprised if cloud-init were doing anything to /var/log/upstart, but smoser could probably tell you more. [10:50] Odd_Bloke: hrm okay, any ideas what might cause it beside cloud-init? [10:50] I found this: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/990102 - unfortunately (and a great example why important) it doesn't mention the commit this was fixed [10:52] fish_: Well, that wasn't a cloud-init bug, it was a problem with the cloud images. [10:53] fish_: Which was fixed ~3 years ago. :) [10:53] Odd_Bloke: I know, I just ask here because cloud-init is the only thing I *expect* to change stuff on the system on first boot :) [10:53] but yes, there is probably something else tampering with /var/log on first? boot [10:54] fish_: Are you sure /var/log/upstart is actually included in your image? [10:54] Odd_Bloke: I know the bug is old, but I suspect it's the same root cause as in my cause (they tried the same - building AMIs) [10:54] Odd_Bloke: yes, I'm I double checked that [10:55] fish_: Out of interest, why not build your image based on the images at cloud-images.ubuntu.com? [10:56] Odd_Bloke: well, I want to actually build the images (vs starting a instance and snapshotting) - I actually wrote a few words about why (+how) here: http://5pi.de/2015/03/13/building-aws-amis-from-scratch/ but tl;dr [10:57] I want the AMIs small and a clean separation between run and built-time [10:57] fish_: Right, but why not take the images from there and modify them by mounting them? [10:59] Odd_Bloke: that's an option.. deboostrap seemed more straight forward. do you think there are advantages in using the cloud-images directly? also worried about security updates because I want the amis to be pretty much immutable (well, at least I don't want to run config management etc on top to keep things up to date) [11:00] but I just realize that I'm wrong, the resulting image indeed has no /var/log/upstart.. now I'm confused since I added a 'mkdir /var/log/upstart' to my build process which failed because it was there already.. looks like something during the build removes it [11:01] fish_: Well, using the Ubuntu images would (probably) have saved you from this bug, for example. [11:01] And you presumably have the same problem with security updates whoever is building the images? [11:02] In your modification process, an "apt-get -y update; apt-get -y upgrade" would pull in any security updates the same as installing from scratch. [11:03] hrm yes that's true.. well, I simply used debootstrap because it seemed like the right tool to use. but yeah, it's definitely a good option to use the cloud-images, will consider that [12:57] fish_, generally speaking, i really dont think you should build your own images. [12:58] any more than i think you should build your own kernel, or your own python or elibc. [12:58] you're certainly welcome to do it, and tools are availalbe to do so. but doing so means you get to re-discover bugs which are fixed. [12:59] the process that I would recommend, is [12:59] * download cloud image [13:00] * mount-image-callback --system-mounts --system-resolvconf $IMAGE -- chroot _MOUNTPOINT bash -s < your-update-script [13:00] and 'your-update-script' does things like: [13:00] apt-get update [13:00] apt-get install foo [13:00] apt-get clean [13:01] the above does require root, which is less than ideal, but [13:01] a.) if you don't trust the ubuntu images, you're kind of SOL anyway [13:01] b.) you can just run that in a VM to alleviate potential mount based attacks. [13:15] smoser: yes, the more I think about that the more it makes sense. for now, things are working and I'm about to roll that out, but in the next iteration I'll definitely look into that [13:16] I use a intermediate "base" image to avoid running a fresh deboostrap for every update anyways, so it should be fairely easy to use a cloud-image instead [13:20] fwiw, the maas-images build process does essentially the above. === zz_gondoi is now known as gondoi === gondoi is now known as zz_gondoi === harlowja_away is now known as harlowja_ [16:46] claudiupopa i think we might have to remove the channel from https://review.openstack.org/#/c/169293/ [16:46] its gonna be hard to get an operator in this channel without kicking everyone out (which nobody has the permission to do) [16:46] so might have to just skip that part [16:47] years ago i think we all forgot to setup this channel with an operator and its pretty hard to do it post-creation [16:49] *afaik* [16:52] harlowja_, why didn't you ever add 'tar' to the write_files. and http:// [16:52] s/you/me/ [16:52] hmmm [16:52] or was that me? [16:52] idk [16:52] haha [16:52] ie, would be nice to have write_files either read content from a url [16:53] or read a tarball from url and extract it into a target dir. [16:53] * harlowja_ looking [16:53] http://paste.ubuntu.com/10691441/ [16:54] that is what i have, but having those 2 big blobs as 'path: http://' would have been nice. [16:54] and then also: [16:54] path: http:// [16:54] format: tar [16:54] extract-dir: /writable/user-data/cloud-init [16:54] or something [16:54] hmmm, ya why did we do that, ha [16:56] msg: ":::::: Hi Mom :::::::" [16:56] lol [16:56] hmmmm [16:57] seems like we should just do that... (allow url stufF) [16:57] and tar [16:57] guess maybe we just didn't think of it ? [17:09] yeah, i think we just didnt. [17:09] the inline makes sense. [17:09] as you may not have networking at that point. [17:09] but if you do have networking, then http:// makes good sense. [17:09] and tar is just a nice archive format :) [17:17] :-P [17:17] shall i code that up or u want to? [17:17] pop out the codes [17:17] lol [17:37] smoser ^ [19:12] smoser https://code.launchpad.net/~harlowja/cloud-init/write-files-fetch-from-somewhere/+merge/254816 [19:12] ok thats part 1 (no tar) [19:12] should be mostly ok, ha [19:13] python3... [19:13] ya, durn it [19:13] pre_content comes as binary... pretty sure (or we ant to make sure it is) [19:13] and then we want to write content without conversion [19:14] will make sure [19:14] load_tfile_or_url does that i guess [19:14] tfile loads text [19:14] ew want blobs of unadulterated raw data [19:14] kk, let me check here [19:14] stupid stff [19:14] lol [19:14] *stuff [19:15] * harlowja_ goes and builds up my 3.4 venv [19:19] harlowja_, see why i said i hate pypi [19:19] lol [19:19] ya [19:19] oh darn, some dude's cable modem is out [19:19] pypi fail [19:19] lol [19:47] ok dokie; added some tests and stuff; seems to work as expected [19:48] at least under basic tests [19:51] smoser check that out if u want [19:57] harlowja_, thanks. [19:58] np [20:19] smoser: Am I right in thinking that vendor-data simply isn't a thing in the version of cloud-init in precise? [20:28] right . not at all. [20:28] Odd_Bloke, i'd sponsor an sru [20:28] precise is 12.04 right? [20:28] harlowja_: Yeah. [20:28] k, can't ever remember the codenames, lol === harlowja_ is now known as harlowja_away [22:07] I want to set hostname to the aws instance ID. Is there a var available for doing that ot do I have to use some commands during bootcmd to handle this? === harlowja_away is now known as harlowja_