[15:02] <tz__> Good morning, I’m having trouble debugging my cloud-config script.  I’m on ubuntu 16.04.  I edit /var/lib/cloud/instance/user-data.txt and run cloud-init --debug init, but i get the same error message of “Unhandled non-multipart (text/x-not-multipart) userdata: ‘b’{‘…’” every time i run it, regardless of changes to user-data.txt
[15:03] <smoser> tz__, cloud-init does not re-read that file.
[15:03] <smoser> it reads form the datasource.
[15:04] <tz__> can i edit the data source on digital ocean after the VPS has been launched?
[15:04] <smoser> not terribly easily.
[15:04] <smoser> for debugging quick launches i suggest using lxc
[15:04] <tz__> how can i iterate quickly on the cloud-config file then without having to destroy, create a droplet?
[15:05] <smoser> since you already have one up, try this
[15:05] <smoser> sudo lxd init
[15:05] <smoser> ancer its questions
[15:05] <smoser> then as non-root
[15:05] <smoser> s/ancer/answer/
[15:05] <smoser> then as non-root
[15:07] <smoser> lxc image copy ubuntu-daily:xenial local: --alias xenial --auto-update
[15:07] <smoser> actually, i woudl install zfs first (apt-get install zfs)
[15:07] <smoser> then you can use zfs with lxc which is much faster.
[15:07] <smoser> then you launch new instances with
[15:07] <smoser>  lxc image copy ubuntu-daily:xenial local: --alias xenial --auto-update
[15:07] <smoser> fudge
[15:07] <smoser> sorr.
[15:08] <smoser> launch a new instance with user data in file 'user-data' like:
[15:08] <smoser> lxc launch ubuntu-daily:xenial my-name "--config=user.user-data=$(cat ./user-data)"
[15:09] <tz__> okay cool, trying it out now
[15:09] <tz__> thanks smoser!
[15:11] <tz__> is there anyway to see cloud init output inline when i launch?
[15:11] <tz__> or whats the best way to see the cloud init output?
[15:12] <smoser> no. actually.
[15:12] <smoser> that sucks.
[15:13] <smoser> at least not easily
[15:13] <smoser> you probably can
[15:13] <smoser> but you have to fiddle with the config options of the container
[15:13] <smoser> lxc used to put console output right in front of you
[15:13] <smoser> but lxc2.0 does not.
[15:13] <tz__> hmmm so how do people build the cloud config the first time? im probably gonna have to edit it 100 times before i get it right
[15:13] <smoser> tz__ but you can easily enough do things like:
[15:14] <smoser>  lxc exec NAME tail -f /var/log/cloud-init.log
[15:14] <smoser> tz__, if you're passing cloud-config, the first thing is that it needs to be valid yaml
[15:14] <smoser> so checking that is as simple as:
[15:14] <smoser>  python -c 'import yaml; yaml.load(open("your-user-data"))'
[15:16] <tz__> okay, that passes at least
[15:18] <tz__> got “Failed to shellify” then a dumb of all the runcmds
[15:20] <tz__> okay, i think it’s because i have blank lines (trying to split up the runcmds into sensible chunks)
[15:23] <tz__> smoser: thanks for your help, i’m now seeing cloud-init debug information
[15:25] <smoser> tz__, couple things...
[15:25] <smoser> you can do a print() of the loaded yaml to make sure it looks like you epxect
[15:25] <smoser> and also yaml "anchors" are useful for making things readable
[15:26] <smoser> look at http://paste.ubuntu.com/22322345/
[15:29] <tz__> okay, i’ll check that out
[15:29] <tz__> thanks!
[18:01] <tz__> is there such a thing as cloud config snippets? a repo of best installations of, for example, redis?
[18:01] <tz__> node, databases, node app
[18:02] <tz__> sshd_config
[18:02] <smoser> tz__, no.
[18:02] <tz__> sad.
[18:03] <tz__> seems like it would be super valuable since cloud configs are cross platform(ish)
[18:04] <tz__> check out what you want on the box, and it generates the cloud config for you
[18:04] <smoser> harlowja, around ?
[18:05] <harlowja> sup
[18:05] <smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+ref/version-info
[18:05] <smoser> actually..
[18:05] <smoser> let me propose for merge to make it easier for you
[18:05] <harlowja> thx :-P
[18:06] <harlowja> do we need to start using git rebase more also?
[18:06] <harlowja> to avoid crap histories?
[18:06] <harlowja> rebase then push --force
[18:08] <smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/302176
[18:08] <smoser> harlowja, are you trolling ?
[18:08] <harlowja> not really :-P
[18:08] <smoser> thats been my biggest pain ever.
[18:09] <smoser> 2 things that bzr did better than git
[18:09] <smoser> a.) bzr commit -p
[18:09] <harlowja> u just gotta know how to use git right ;)
[18:09] <smoser>   (this just shows diff in the EDITOR window, so you can write your message and look at the diff)
[18:09] <harlowja> http://blog.adamspiers.org/2015/03/24/why-and-how-to-correctly-amend-github-pull-requests/#sec-3
[18:09] <smoser> b.) uses '--first-parent' by default behaviori for git log
[18:09] <harlowja> its the same crap for github PRs that most people don't know how to do :-P
[18:10] <harlowja> http://blog.adamspiers.org/2015/03/24/why-and-how-to-correctly-amend-github-pull-requests/
[18:10] <smoser> using first-parent by default is genious
[18:10] <smoser> genius
[18:10] <harlowja> ya, bzr prob did that nicer
[18:10] <harlowja> gerrit forces it
[18:10] <harlowja> but github doesn't
[18:10] <smoser> it means that people dont have to squash and throw away data
[18:10] <smoser> in order to avoid crap history
[18:10] <harlowja> ya, fair
[18:10] <smoser> i almost consider just getting crap history
[18:11] <smoser> and saying "please use --first-parent"
[18:11] <smoser> so there... at that mp, i tested building of rpm.
[18:12] <harlowja> cool
[18:12] <harlowja> how'd that go
[18:12] <harlowja> ha
[18:12] <smoser>  http://paste.ubuntu.com/22339840/
[18:13] <smoser> the way to build is this:
[18:13] <smoser>  http://paste.ubuntu.com/22339872/
[18:13] <harlowja> cools
[18:13] <smoser> that does it all in an lxc container, but basic path is there.
[18:13] <harlowja> will use that :-P
[18:13] <smoser> right. no one. although maybe sometime soon honestly as we want to get c-i on centos
[18:14] <smoser> larsks, you probably have thoughts on my brpm stuff
[18:14] <smoser>  spec looks like thsi: http://paste.ubuntu.com/22339840/
[18:14] <harlowja> so only other thing i'm not sure about is
[18:14] <harlowja> say i have a addon that doesn't really belong in cloud-init, but i need to add it into cloud-init
[18:15] <harlowja> i can show u this addon if u really care
[18:15] <harlowja> how do i make sure that the version number won't mess with the upstream one
[18:15] <smoser> by default, we dont bump the 'Release' as iunderstand thats like "packaging release" in debian and in order to get a new version of that, you  kind of had to bump the source release so 'Version' would go up.
[18:15] <harlowja> k
[18:15] <smoser> example ?
[18:16] <harlowja> a godaddy module addon
[18:16] <harlowja> config module
[18:16] <smoser> ah.
[18:16] <smoser> and you have patches/ that you use --patches ?
[18:16] <harlowja> this one its not yet a patch, but ya, its equivalent to
[18:17] <larsks> This is where I wish cloud-init using something like stevedore for discovering out-of-tree modules...
[18:17] <harlowja> right
[18:17] <harlowja> https://gist.github.com/harlowja/e6d79d1de62e0edd821bedf76bfd5a20
[18:17] <harlowja> smoser thats part of that module
[18:17] <harlowja> bunch of crap that getting out of puppet (doing it in puppet and half of it in cloud-init was causing stupid issues)
[18:17] <harlowja> so fixing that by just moving some of it to a cloud-nit module
[18:17] <smoser> right.
[18:17] <smoser> so how should we do that?
[18:18] <harlowja> so stevedore is one option
[18:18] <smoser> you want yours to be bigger than trunk or something.
[18:18] <smoser> the version ?
[18:18] <smoser> is that what we're talking about ?
[18:18] <harlowja> well i either need the release to be bigger or smaller :-P
[18:18] <smoser> yeah, larsks i do want to support out-of-tree moduels.
[18:18] <smoser> and datasources.
[18:18] <harlowja> oorrr go the path of larsks with stevedore
[18:19] <harlowja> i can cook up osmething with stevedore
[18:19] <smoser> we already have enough dynamic loading
[18:19] <harlowja> (yet another runtime dep...)
[18:19] <smoser> we dont need to add stevedoor for it.
[18:19] <harlowja> stevedore is just a tiny wrapper over entrypoints
[18:19] <harlowja> which is a python thing
[18:19] <smoser> and /me cries everytime he runs a stevedore-using executable
[18:19] <harlowja> stevedore shouldn't make u cry
[18:19] <smoser> stat stat stat stat stat disk io disk io
[18:19] <harlowja> its pretty dinky in all reality
[18:20]  * larsks cries every time he sees another project re-implement the same basic features.
[18:20] <harlowja> stevedore is a light layer over the python entrypoints stuff
[18:20] <harlowja> so either use stevedore, or i just use entrypoints like stevedore is doing
[18:20] <harlowja> cloud-init right now has its custom thing
[18:20] <smoser> i dont know.
[18:20] <harlowja> https://github.com/openstack/stevedore/blob/master/stevedore/extension.py#L147
[18:20] <harlowja> thats really the 'thing' that does this, lol
[18:20] <smoser> i'm very much feeling pain of startup performance right now.
[18:20] <harlowja> 'eps = list(pkg_resources.iter_entry_points(namespace))'
[18:21] <harlowja> mainly that, ha
[18:21] <smoser> and also feel pain of external dependencies.
[18:21] <harlowja> the rest of stevedore is just lue
[18:21] <harlowja> *glue
[18:21] <harlowja> so i'm in the middle between larsks and smoser
[18:21] <harlowja> in that if i could use something like entrypoints
[18:21] <larsks> An entrypoints-based internal implementation would I guess be fine.
[18:21] <harlowja> then i can get rid of this complexity :-P
[18:22] <harlowja> larsks ya, i can easily make something dinky using entrypoints
[18:22] <harlowja> its not really that hard, ha
[18:22] <harlowja> https://github.com/openstack/stevedore/blob/master/stevedore/extension.py#L153 is the loop, and the other code is the finding
[18:22] <harlowja> not magical crazy stuff, ha
[18:22] <harlowja> which is why stevedore is a dinky layer
[18:23] <larsks> Yeah.  also, pkg_resources.iter_entry_points may have most of what we need...
[18:23] <harlowja> yup
[18:25] <smoser> so 2 thigns i'd like to support out of tree
[18:25] <smoser> a.) config modules
[18:25] <smoser> b.) datasources
[18:25] <smoser> and thats easy enough to "find" them.
[18:25] <smoser> but getting them into the right order is somethign we dont have. no way to declare a dependency or something.
[18:25] <smoser> so at the momemnt a list of datasources and a list of modules at each stage.
[18:26] <smoser> harlowja, so back to your version / thing...
[18:26] <smoser> what were you saying ?
[18:26] <larsks> smoser: well, config modules get an ordering from cloud.cfg.
[18:26] <smoser> does using that version with a git cause issues for you?
[18:26] <harlowja> smoser i forget
[18:26] <smoser> larsks, ah. yeah, but not if they're just laid down on disk.
[18:26] <smoser> ie, ideally you just put new modules in some directory and they get found
[18:26] <smoser> and datasources too
[18:26] <harlowja> someway of making sure that a godaddy built cloud-init.rpm doesn't use the same stuff (Version...) as a public cloud-init.rpm
[18:27] <smoser> harlowja, we could expose branch
[18:27] <larsks> harlowja: add a suffix to the package name (cloud-init-godaddy) and make it obsolete the other one?
[18:27] <smoser> in the spec
[18:27] <harlowja> larsks ah, that might work to
[18:27] <harlowja> until we have this stevedore and ... stuff
[18:28] <harlowja> then i can get out of this :-P
[18:29] <larsks> smoser: looked briefly at your brpm changes, which seem fine, but there is a weird try/except block at the top of brpm that seems to be a noop (it catches any exception...and raises it)
[18:35] <smoser> larsks, yeah its stupid
[18:35] <smoser> for flake8
[18:35] <smoser> it complains about module level imports not at top of filel
[18:37] <smoser> harlowja, you know of a nicer way to work around that ?
[18:37] <harlowja> not sure
[18:38] <larsks> smoser, I don't see that if I remove the try/except block.  no complaints from flake8 (https://asciinema.org/a/bbqseaf84npuoon71hciwvc99)
[18:38] <larsks> (where flake8 == 2.4.1)
[18:39] <smoser> how'd you do that larsks ?
[18:40] <larsks> do which?
[18:40] <smoser> the asciicinima thing
[18:40] <larsks> https://asciinema.org/
[18:40] <larsks> There is a simple 'asciinema' cli tool
[18:41] <larsks> Looks like you need a ppa for ubuntu.  Seems to be packaged for fedora and debian...
[18:41] <smoser> well, its really pep8 . pep8 1.7.0 here does not like it.
[18:41] <larsks> Interesting.  I seem to have 1.5.7.  I guess I would add a comment there, otherwise everyone who looks at the code is going to ask the same question :)
[18:41] <smoser> yeah
[18:41] <smoser> deal
[18:42] <harlowja> hmmm asciinema neat
[18:45] <smoser> if "avoid-pep8-E402-import-not-top-of-file":
[18:46] <smoser>   import ...
[18:46] <smoser> committing that
[19:16] <smoser> harlowja,
[19:16] <smoser> https://bugs.launchpad.net/cloud-init/+bug/1603533
[19:16] <smoser> is fix committed ?
[19:22] <harlowja> i think so
[19:23] <harlowja> assuming its still carried forward with the git stuff (and not lost)
[21:46] <smoser> hey all. i have some changes here locally.
[21:47] <smoser> i'm going to mark a 0.7.7 release soon. tonight or tomorrow.
[21:47] <smoser> and call it released.
[21:47] <smoser> and then we can work towards a quick 0.7.8 iteration on that.
[21:47] <smoser> the git stuff now in i think.
[22:03] <JayF> smoser: http://lists.openstack.org/pipermail/openstack-dev/2016-August/101007.html is something you might find interesting
[22:37] <harlowja> cool
[23:29] <tz__> anyone have a snippet that shows how to pull a repo from github?  I have http://pastie.org/private/kpnunfkgktxxse2m4xwsga , but problems ensue