[08:49] If I want to use the ca-certs module (as here: https://cloudinit.readthedocs.io/en/latest/topics/examples.html#configure-an-instances-trusted-ca-certificates), what do I need to add to /etc/cloud/cloud.cfg? [08:52] I can see /usr/lib/python2.7/site-packages/cloudinit/config/cc_ca_certs.py, so do I add "ca_certs" to cloud_config_modules? Or "ca-certs"? Is the difference between "-" and "_" significant? [09:27] ccard: the config value should be behind ca-certs: [09:27] ccard: ca_cert_cfg = cfg['ca-certs'] - that line gets the "section" (if you want to call it that way - behind ca-cert [09:28] ccard: all following uses ca_cert_cfg to read subelements [09:31] ccard: http://cloudinit.readthedocs.io/en/latest/topics/examples.html#configure-an-instances-trusted-ca-certificates [09:32] cpaelzer: thanks. What do I need in /etc/cloud/cloud.cfg in the cloud_config_modules section? "ca-certs" or "ca_certs"? [09:33] the former, as listed in the example link [09:33] ccard: ^^ [09:34] cpaelzer: I don't see anything in that link about /etc/cloud/cloud.cfg [09:34] ccard: right I just found the config syntax for it - not a particular statement about cloud.cfg [09:37] ccard: usually you put that as a submodule and include it from cloud.cfg [09:37] ccard: but IIRC you could add it directly there as well if you want [09:37] let me take a look [09:39] Currently, the /etc/cloud/cloud.cfg I get by default doesn't have ca-certs module listed (similarly for the resolv_conf module). I have cloud-init-0.7.5-10.el7.centos.1.x86_64 on CentOS 7 which lays down /etc/cloud/cloud.cfg. The ca-certs module exists, but is not configured into cloud-init as far as I can see. [09:40] ccard: ah - now I get you what you want to put into cloud.cfg [09:40] ccard: looking at my 0.7.7~bzr1212-0ubuntu1 I see it listed in modules [09:40] ccard: http://paste.ubuntu.com/16373639/ [09:41] ccard: I don't know if this is not yet fully exploited on your version or just disabled by default - sorry [09:42] cpaelzer: so it should be "ca-certs" in the cloud_init_modules (not cloud_config_modules)? [09:43] ccard: that is what is in my more recent version, but it leaves my comfort zone to hard-confirm it for your case [09:44] cpaelzer: ok, thanks for your help [09:44] ccard: I'll look in the repo and check when it was added there [09:44] ccard: maybe it sheds some more light [09:44] cpaelzer: thanks [09:48] ccard: it was just adding it there (no moving) back in 2012 with the comment "Add ca-certs into the main config to run just before rsyslog." [09:49] ccard: but then the code handling that got merged along that - so I'd find it quite weird if you had the code but not this change in the config [09:50] cpaelzer: probably a RedHat thing [09:51] ccard: you can find a bit of the devl history n just this in https://bugs.launchpad.net/cloud-init/+bug/915232 [09:52] cpaelzer: thanks [11:00] smoser: for the testing with mock I wonder if/how to access files usually deployed by the package itself [11:00] smoser: I think I got the concept of self.patchUtils and such to rewrite the basic file access methods [11:00] smoser: and that works fine for stuff that I write to stay in e.g. a tmp dir [11:01] smoser: but my tests now pass code that e.g. checks for /etc/cloud/templates/sources.list.ubuntu.tmpl [11:01] smoser: and this is not "there" in the test environment - no matter if I rewrite access or not [11:01] smoser: I wonder how those files - that would be provided by the package install - would get into the test environment [11:02] smoser: I didn't find a test that cared about that yet - is there one which I could use to learn from how to get those files in the test environment? [11:03] smoser: I might just mock around the clock to get it for the unittests, but I wonder if it could be done [12:02] smoser: would you expect urlparse to return for hostname on 'ftp.us.debian.org'? [12:02] I need to read more of the specs or write a small test app ... [13:08] smoser: it is so sad, I added all kind of nice stuff like self.patchOS(self.new_root) once I understood what it did [13:08] smoser: but later I understood it even more and removed oh so much [13:09] smoser: final changes look silly when I think how much I wrote to get there :-) [13:09] smoser: at least - nice learning experience [13:33] cpaelzer, did you get all you needed ? [13:37] smoser: other than the theoretical question about /etc/cloud/templates/sources.list.ubuntu.tmpl above yes [13:37] smoser: and even that I got working via mocking, but I was eager to find out if it would be possible at all [15:19] smoser: I've sent you a MP for the cloud init part what you asked for [15:19] smoser: currently down in calls and soon in the same as you :-) [15:19] smoser: after that is family handling time, but in like 3 hours I'd be back [15:20] smoser: if you have any complex feedback til then we coudl discuss it - otherwise I'd go on on the curtin part of it tomorrow [15:38] cpaelzer, just hit 'reply' [15:38] or whateer that button was labeleed [15:38] https://code.launchpad.net/~paelzer/cloud-init/test-apt-source/+merge/294521 [15:39] smoser: thanks *reading* [15:40] smoser: ack to all, with some vague yes to the style things [15:41] smoser: I'll modify once everybody around is sleeping [15:41] smoser: I'm sitll on 50% Austin tz anyway [15:41] smoser: so you should have a new MP to look at over my night later on [15:43] k. [17:15] smoser i'm also going to go through and fix up a bunch of flake8 issues [17:15] https://gist.github.com/harlowja/3ba219c41f1e5dd3cd596b4795f8abe8 [17:15] nothing major, just some minor stuff [17:16] https://gist.github.com/harlowja/3ba219c41f1e5dd3cd596b4795f8abe8#file-gistfile1-txt-L119 is sorta major i guess though, lol [17:16] probably won't work in py3 [17:17] hmm [17:19] harlowja, [17:19] flake8 2.5.4-2 [17:19] that does not complain about it [17:20] odd [17:20] what version do you have ? [17:20] 2.2.4 [17:20] still shows it for me if i upgrade [17:21] cloudinit/sources/helpers/openstack.py:149:5: H236 Python 3.x incompatible __metaclass__, use six.add_metaclass() [17:21] cloudinit/sources/__init__.py:50:5: H236 Python 3.x incompatible __metaclass__, use six.add_metaclass() [17:21] that one also [17:21] i'll fix it up sir [17:22] http://paste.ubuntu.com/16378958/ [17:23] ya, i'm pretty sure u are just weird [17:23] lol [17:25] $ pip freeze | grep "pep8\|pyflakes\|flake8\|pylint" [17:25] flake8==2.5.4 [17:25] pep8==1.7.0 [17:25] pyflakes==1.0.0 [17:25] pylint==1.5.5 [17:25] thats my stuffs [17:27] then [17:27] flake8 cloudinit/sources/helpers/openstack.py [17:27] cloudinit/sources/helpers/openstack.py:149:5: H236 Python 3.x incompatible __metaclass__, use six.add_metaclass() [17:27] cloudinit/sources/helpers/openstack.py:487:1: H401 docstring should not start with a space [17:31] harlowja, that warning comes from pyflakes, right ? [17:31] and mine is newer than yours [17:31] and newer is always better [17:31] always [17:31] lol === cbolt- is now known as cbolt [18:00] rharper, (or harlowja ) there is no way to go from ENI to network-config, right ? [18:00] i see parse_deb_config() but that returns some other 'ifaces' dictionary [18:00] i didn't see anything that does that parsing [18:00] just network-config -> network-state -> eni [18:01] http://paste.ubuntu.com/16379781/ [18:01] so my current path of attack for all this network config stuff is to drop 'dsmode' [18:01] ya, i think thats this other thing [18:01] there are a few formats at play here in the net stuff :-P [18:02] network-config isn't the internally used format [18:02] its converted into something else [18:02] (for some reason) [18:03] hm.. ok. [18:03] ya [18:04] https://code.launchpad.net/~harlowja/cloud-init/cloud-init-net-refactor has some examples/tests that i added for this [18:04] http://bazaar.launchpad.net/~harlowja/cloud-init/cloud-init-net-refactor/revision/1227 (one test) [18:05] but ya, smoser there is a internal format that i'm not sure why it was created :-P [18:06] well, rharper did that. and i dont really care if theres an internal format as long as you can got fmt1 <-> Internal <-> fmt2 [18:06] and ideally losslessly [18:06] ya, i'm not sure its loseless [18:06] right. well, /etc/network/interfaces as a format is lossy in and of itself. [18:07] because it does not declare the mac or of the nic. just its name. [18:07] so at least you have to read from the /sys and hope for the best state :) [18:07] http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/sources/DataSourceConfigDrive.py#L309 [18:07] the converter ^ [18:08] bazaar.launchpad.net/~harlowja/cloud-init/cloud-init-net-refactor/ i think helps here, so ya, merge it, lol [18:09] https://code.launchpad.net/~harlowja/cloud-init/cloud-init-flake8-fixups/+merge/294548 also [18:15] harlowja, i'm generally ok with fixing up lint things (and pyflakes things) [18:15] but its obnoxious when that is a never ending task, and you end up chasing your tail to work in all different versions [18:16] for example, pyflakes for me is perfectly happy. [18:16] but for you it is not [18:18] well some of these are somewhat basic issues [18:18] yeah, but why does my pyflakes not care [18:18] __metaclass__ afaik is gone in py3 :-/ [18:18] when mine is >= to yours for all of pep8, flake8 and pyflakes [18:18] and stuff like class A: is different on py3 vs p2 where u need to do class A(object) [18:19] well, sure. and that is all ok. [18:19] but why does my system think its ok. [18:19] and yours not [18:20] i'm not ok randomly fixing crap like that [18:21] https://gist.github.com/harlowja/fb8a1c2c27ecf433daf71524f150bc18 [18:21] i just installed all the same versions :-P [18:21] your system is weird, lol [18:21] ohhh i think i know [18:22] can u install hacking==0.10.2 [18:24] i'm using this thing called a distribution [18:24] i get stuff from it. [18:24] :-P [18:25] ya, it should be packaged under some name [18:25] https://github.com/openstack-dev/hacking is the project [18:27] wow [18:27] that is quite helpful (/sarcasm) [18:27] apt-get install python3-hacking and all of a sudden flake8 decides to act differently. [18:27] ya.... [18:28] action at a distance :-P [18:28] neither of these packages indicate interaction with one another. [18:28] ya, i know, i forgot about it [18:28] but they do :-/ [18:29] see the 'chasing your tail' comment above ? [18:29] but the issues its raising are still useful ;) [18:29] so what do you do about this sort of thing. [18:29] https://code.launchpad.net/~harlowja/cloud-init/cloud-init-flake8-fixups/+merge/294548 (imho fixes a bunch that i think are useful) [18:30] i really dont like it when random dude on the internet uploads a new python- package [18:30] lol [18:30] and then my c-i (which is not present for cloud-init, but should be) starts to fail [18:30] i dont find that terribly useful. [18:30] yes sir, u are from a distro company, i can tell :-P [18:30] u must work at canonical [18:30] lol [18:30] if everything always got better, that'd be one thing. [18:31] but i have evidence otherwise. [18:31] https://bugs.launchpad.net/pyflakes/+bug/1560134 [18:31] on your python packages, what do you do ? do you pin versions of stuff in tox ? [18:31] yup [18:31] or do you just love the daily activity of changing source code so you can fit joe-random-dude's daily gut feeling on what is the best way to write code. [18:32] as long as joe random dude == josh random dude [18:32] where josh random dude == me [18:32] lol [18:32] alright. so lets pin versions then. [18:33] k [18:33] for curtin i added a 'trusty' tox target [18:33] as you can write code that is happy on today's version of "good" but not april 2014's version of good. [18:34] i trusty u [18:34] lol [18:35] ok in https://code.launchpad.net/~harlowja/cloud-init/cloud-init-flake8-fixups/+merge/294548 i froze the lint requirements [18:38] i do like the alphabetical order on imports [18:47] harlowja, i'll merge that if you read and offer thoughts on http://paste.ubuntu.com/16380605/ [18:50] yes boss [19:01] smoser looks ok, i think i'd want each datasource to provide a setup_network (that most can just not implement) and cloud-init will call this on first boot (if there are multiple datasources that impl this that are active, we can i guess pick the first) [19:02] that will get rid of some of the weird networking crap in config drive [19:03] harlowja, well, as it is right nwo in trunk the datasource may provide a network_config [19:04] and if it does, then cloud-init (in stages.py) will aply networkign as described there. [19:04] smoser: did you know that long key fingerprints where broken? [19:04] oh ya, right [19:05] smoser: next MP will contain a fix but that was a good idea to test the long fingerprints [19:05] smoser http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/sources/DataSourceConfigDrive.py#L114 still exists though :- [19:05] smoser: the embedded shell script lackes a few " and due to that it was broken for long fingerprints with spaces [19:05] :-/ [19:06] cpaelzer, :). how are they broken ? [19:06] harlowja, yeah, that is yucky. [19:06] thats why i was asking about converting network-interfaces into network-config [19:06] as ideally if there was an interfaces style file there, then we'd just convert it to a network-config and use it there. [19:07] smoser: it is well hidden and just appears to not work, but at the root cause changing gpg --keyserver ${ks} --recv $k to use "${k}" does the job [19:07] smoser hmmmm [19:07] i mean, i can make a shitty converter [19:07] :) [19:07] one already exists (but it doens't go to network-config) [19:07] smoser: I reorganized the code a bit to allow testing down to the last level so it is not only fixed but should be catched next time [19:08] smoser http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/distros/net_util.py [19:08] ^ shitty converter [19:08] u can just take the output format of that and turn it into network-config [19:09] or whatever format u want, (of the 3+ that exist in cloud-init to represent network stuff, lol) [19:09] yeah :-( [19:09] we really need to have one internal representation [19:09] yup [19:09] i guess i'm ok with that being 'network state' [19:09] ya, that format isn't to well defined [19:10] i tried deciphering it, lol [19:11] cause network state is really just a object with a version and a dict (the dict is the actual info) [19:12] http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/sources/DataSourceConfigDrive.py#L308 (network_config there seems to be the actual format) [19:12] and i sorta tried to figure out what network_config is, lol [19:12] because network_config seems to be the dict that network state actually uses [19:12] so ya, it'd be nice to well have that more well ummmm, defined, lol [19:14] cause http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/net/network_state.py#L72 sorta just sets some random attributes, hard to tell what the real format is :-P [19:14] harlowja, curtin has the best examples of it. [19:14] we do need some much more formal documentation on that. [19:14] def [19:15] http://bazaar.launchpad.net/~curtin-dev/curtin/trunk/files/head:/examples/ [19:15] yup yup [19:15] sooo i'd be ok with that being the internal cloud-init format [19:15] and then adjusting other places [19:15] and having a loseless(?) openstack -> that format converter [19:16] * harlowja won't ask why the openstack format wasn't just used :-P [19:16] bb, food [19:19] cpaelzer, hm. [19:19] i'm looking at that export key thing. and it is crap. [19:19] which is probably not a surprise to you :) [19:20] but it does seem to work for long fingerprint also [19:22] with " it works for long [19:22] without bash might break it [19:22] and the gpg tool considers it a list of short id's [19:23] http://paste.ubuntu.com/16381150/ [19:23] it works for me for long and short as is. [19:23] haha [19:23] smoser: wait for my example [19:24] smoser: use this as the long key [19:24] smoser: "B59D 5F15 97A5 04B7 E230 6DCA 0620 BBCF 0368 3F77" [19:24] smoser: which is how it is listed by some tools, so people filling the config file might copy and paste it with spaces [19:25] smoser: dpending on how things are connected it wither selects 5F15 as keyserver or tries to fetch 10 shortkeys with their ids [19:25] smoser: both wrong [19:26] ah. yeah. [19:26] so yeah, that shoudl be quoted for sure. [19:27] its really crap that it has to recieve them [19:27] it tries to clean up after itself, and delete the thing (and not delete it if it found it locally already). [19:27] smoser: I saw it [19:27] smoser: all kind of messy if things get interrupted [19:28] smoser: yet I didn't want to fix the whole world in one day [19:28] smoser: I updated the MP and the commit message [19:28] well, its not really *too* messy. [19:28] root's local .gpg will just know of that fingerprint [19:28] its not really the end of the world [19:28] ah is that local - yeah it is just gpg and not apt-key [19:29] I'll call it EOD then [19:29] smoser what about extracting the curtin and net stuff into a curtin_net or something package? [19:29] smoser: is there a good way of running curtin e.g. vmtests bundled with that experimental cloud-init? [19:29] this would just have the format, the converters and renderers and that's about it [19:29] smoser: if there is please drop me a mail, so I can use it tomorrow [19:29] instead of being in cloudinit or being in curtin [19:30] (or make other fancy name besides curtin_net) [19:30] lol [19:30] originally i intended to have curtin's net and block modules move externally to something else. [19:31] but then rharper and i kind of decided that we would like for it to be in cloudd-init. [19:33] hmmm [19:50] Is it possible to read mdata from smartos within cloud-init? [19:51] dcrouch, what is mdata ? [19:52] ah. i see. [19:52] dcrouch, http://bazaar.launchpad.net/~smoser/cloud-init/trunk.joyent-cleanup/revision/1216 [19:53] smoser, Basically I can provide metadata to the server from smartos. mdata is that metadata, for example. https://dpaste.de/sDKS [19:53] i started a re-work of the smartos datasource that has a much simpler to use client [19:55] http://paste.ubuntu.com/16381487/ [19:55] I'm thinking maybe I can write a python module, pull that data, and have it loaded on firstboot for items such as root password. [19:56] oh. well, the answer then is yes [19:56] and cloud-init already has a datasource for smartos [19:56] and it reads that stuff. [19:56] you give cloud-init config and it does what you tell it [19:57] I understand that, but when we create our images in SmartOS, I need a way to send data to the vms for firstboot, so was looking how to get this for ubuntu. [19:58] https://dpaste.de/TFYr [19:58] dcrouch, that client is only in the branch i pointed you too above. [19:59] dcrouch, you're the cloud ? [19:59] ie, you're smartos ? [20:00] Correct smoser! [20:00] vendor-data [20:00] thats what you want. [20:01] thats what it was designed for basically. vendor-data is user-data for the vendor [20:01] look at DataSourceSmartOS and read the comments around 'vendor-data' there :) [20:02] harlowja, you added 'igmore' of all those flake8 stuff [20:02] H404,H405 [20:02] smoser, I've been looking up cloud-init for the last few hours and a few topics around there, I thought I recognized your name! https://github.com/number5/cloud-init [20:02] but why ? [20:02] smoser less things for me to fix in 1 merge, lol [20:03] but they're all fixed [20:03] u can fix all the 404 and 405 ;) [20:03] oh [20:03] welllll then (makes up other reason) [20:03] at least i just comment that line out [20:03] and type 'tox -e pyflakes' [20:03] and ENOERROR [20:04] hmmm, run flake8 not pyflakes [20:04] * smoser shames fist at his feeble brain [20:04] i confuse those two things all the time [20:04] https://gist.github.com/harlowja/32845acbd0301fc2132fbd9255bddd1d [20:04] 405 and 404 [20:04] and i'm like screw that, its a dumb warning, lol [20:04] i assumed you had added a tox [20:05] nah, i didn't want to do everything, gotta leave u some work to [20:05] yeah. thanks. [20:05] i'm nice like that [20:09] harlowja, so i'm gonna change pyflakes to flake8 [20:09] which i think is a strict superset of pyflakes [20:09] right? [20:09] and then i'm going to fix all the tests/ so that it wont complain about those either [20:10] cool [20:11] sound good to me [20:11] (yes afaik its a superset of) [20:48] Thanks for your help smoser I think I'll be back still confused but need to take off for a bit.