/srv/irclogs.ubuntu.com/2016/05/12/#cloud-init.txt

ccardIf I want to use the ca-certs module (as here: https://cloudinit.readthedocs.io/en/latest/topics/examples.html#configure-an-instances-trusted-ca-certificates), what do I need to add to /etc/cloud/cloud.cfg?08:49
ccardI can see /usr/lib/python2.7/site-packages/cloudinit/config/cc_ca_certs.py, so do I add "ca_certs" to cloud_config_modules? Or "ca-certs"? Is the difference between "-" and "_" significant?08:52
cpaelzerccard: the config value should be behind ca-certs:09:27
cpaelzerccard: ca_cert_cfg = cfg['ca-certs'] - that line gets the "section" (if you want to call it that way - behind ca-cert09:27
cpaelzerccard: all following uses ca_cert_cfg to read subelements09:28
cpaelzerccard: http://cloudinit.readthedocs.io/en/latest/topics/examples.html#configure-an-instances-trusted-ca-certificates09:31
ccardcpaelzer: thanks. What do I need in /etc/cloud/cloud.cfg in the cloud_config_modules section? "ca-certs" or "ca_certs"?09:32
cpaelzerthe former, as listed in the example link09:33
cpaelzerccard: ^^09:33
ccardcpaelzer: I don't see anything in that link about /etc/cloud/cloud.cfg09:34
cpaelzerccard: right I just found the config syntax for it - not a particular statement about cloud.cfg09:34
cpaelzerccard: usually you put that as a submodule and include it from cloud.cfg09:37
cpaelzerccard: but IIRC you could add it directly there as well if you want09:37
cpaelzerlet me take a look09:37
ccardCurrently, the /etc/cloud/cloud.cfg I get by default doesn't have ca-certs module listed (similarly for the resolv_conf module). I have cloud-init-0.7.5-10.el7.centos.1.x86_64 on CentOS 7 which lays down /etc/cloud/cloud.cfg. The ca-certs module exists, but is not configured into cloud-init as far as I can see.09:39
cpaelzerccard: ah - now I get you what you want to put into cloud.cfg09:40
cpaelzerccard: looking at my 0.7.7~bzr1212-0ubuntu1 I see it listed in modules09:40
cpaelzerccard: http://paste.ubuntu.com/16373639/09:40
cpaelzerccard: I don't know if this is not yet fully exploited on your version or just disabled by default - sorry09:41
ccardcpaelzer: so it should be "ca-certs" in the cloud_init_modules (not cloud_config_modules)?09:42
cpaelzerccard: that is what is in my more recent version, but it leaves my comfort zone to hard-confirm it for your case09:43
ccardcpaelzer: ok, thanks for your help09:44
cpaelzerccard: I'll look in the repo and check when it was added there09:44
cpaelzerccard: maybe it sheds some more light09:44
ccardcpaelzer: thanks09:44
cpaelzerccard: it was just adding it there (no moving) back in 2012 with the comment "Add ca-certs into the main config to run just before rsyslog."09:48
cpaelzerccard: but then the code handling that got merged along that - so I'd find it quite weird if you had the code but not this change in the config09:49
ccardcpaelzer: probably a RedHat thing09:50
cpaelzerccard: you can find a bit of the devl history n just this in https://bugs.launchpad.net/cloud-init/+bug/91523209:51
ccardcpaelzer: thanks09:52
cpaelzersmoser: for the testing with mock I wonder if/how to access files usually deployed by the package itself11:00
cpaelzersmoser: I think I got the concept of self.patchUtils and such to rewrite the basic file access methods11:00
cpaelzersmoser: and that works fine for stuff that I write to stay in e.g. a tmp dir11:00
cpaelzersmoser: but my tests now pass code that e.g. checks for /etc/cloud/templates/sources.list.ubuntu.tmpl11:01
cpaelzersmoser: and this is not "there" in the test environment - no matter if I rewrite access or not11:01
cpaelzersmoser: I wonder how those files - that would be provided by the package install - would get into the test environment11:01
cpaelzersmoser: I didn't find a test that cared about that yet - is there one which I could use to learn from how to get those files in the test environment?11:02
cpaelzersmoser: I might just mock around the clock to get it for the unittests, but I wonder if it could be done11:03
cpaelzersmoser: would you expect urlparse to return for hostname on 'ftp.us.debian.org'?12:02
cpaelzerI need to read more of the specs or write a small test app ...12:02
cpaelzersmoser: it is so sad, I added all kind of nice stuff like self.patchOS(self.new_root) once I understood what it did13:08
cpaelzersmoser: but later I understood it even more and removed oh so much13:08
cpaelzersmoser: final changes look silly when I think how much I wrote to get there :-)13:09
cpaelzersmoser: at least - nice learning experience13:09
smosercpaelzer, did you get all you needed ?13:33
cpaelzersmoser: other than the theoretical question about /etc/cloud/templates/sources.list.ubuntu.tmpl above yes13:37
cpaelzersmoser: and even that I got working via mocking, but I was eager to find out if it would be possible at all13:37
cpaelzersmoser: I've sent you a MP for the cloud init part what you asked for15:19
cpaelzersmoser: currently down in calls and soon in the same as you :-)15:19
cpaelzersmoser: after that is family handling time, but in like 3 hours I'd be back15:19
cpaelzersmoser: if you have any complex feedback til then we coudl discuss it - otherwise I'd go on on the curtin part of it tomorrow15:20
smosercpaelzer, just hit 'reply'15:38
smoseror whateer that button was labeleed15:38
smoserhttps://code.launchpad.net/~paelzer/cloud-init/test-apt-source/+merge/29452115:38
cpaelzersmoser: thanks *reading*15:39
cpaelzersmoser: ack to all, with some vague yes to the style things15:40
cpaelzersmoser: I'll modify once everybody around is sleeping15:41
cpaelzersmoser: I'm sitll on 50% Austin tz anyway15:41
cpaelzersmoser: so you should have a new MP to look at over my night later on15:41
smoserk.15:43
harlowjasmoser i'm also going to go through and fix up a bunch of flake8 issues17:15
harlowjahttps://gist.github.com/harlowja/3ba219c41f1e5dd3cd596b4795f8abe817:15
harlowjanothing major, just some minor stuff17:15
harlowjahttps://gist.github.com/harlowja/3ba219c41f1e5dd3cd596b4795f8abe8#file-gistfile1-txt-L119 is sorta major i guess though, lol17:16
harlowjaprobably won't work in py317:16
smoserhmm17:17
smoserharlowja,17:19
smoserflake82.5.4-217:19
smoserthat does not complain about it17:19
harlowjaodd17:20
smoserwhat version do you have ?17:20
harlowja2.2.417:20
harlowjastill shows it for me if i upgrade17:20
harlowjacloudinit/sources/helpers/openstack.py:149:5: H236  Python 3.x incompatible __metaclass__, use six.add_metaclass()17:21
harlowjacloudinit/sources/__init__.py:50:5: H236  Python 3.x incompatible __metaclass__, use six.add_metaclass()17:21
harlowjathat one also17:21
harlowjai'll fix it up sir17:21
smoserhttp://paste.ubuntu.com/16378958/17:22
harlowjaya, i'm pretty sure u are just weird17:23
harlowjalol17:23
harlowja$ pip freeze | grep   "pep8\|pyflakes\|flake8\|pylint"17:25
harlowjaflake8==2.5.417:25
harlowjapep8==1.7.017:25
harlowjapyflakes==1.0.017:25
harlowjapylint==1.5.517:25
harlowjathats my stuffs17:25
harlowjathen17:27
harlowjaflake8 cloudinit/sources/helpers/openstack.py17:27
harlowjacloudinit/sources/helpers/openstack.py:149:5: H236  Python 3.x incompatible __metaclass__, use six.add_metaclass()17:27
harlowjacloudinit/sources/helpers/openstack.py:487:1: H401  docstring should not start with a space17:27
smoserharlowja, that warning comes from pyflakes, right ?17:31
smoserand mine is newer than yours17:31
smoserand newer is always better17:31
smoseralways17:31
harlowjalol17:31
=== cbolt- is now known as cbolt
smoserrharper, (or harlowja ) there is no way to go from ENI to network-config, right ?18:00
smoseri see parse_deb_config() but that returns some other 'ifaces' dictionary18:00
harlowjai didn't see anything that does that parsing18:00
harlowjajust network-config -> network-state -> eni18:00
smoserhttp://paste.ubuntu.com/16379781/18:01
smoserso my current path of attack for all this network config stuff is to drop 'dsmode'18:01
harlowjaya, i think thats this other thing18:01
harlowjathere are a few formats at play here in the net stuff :-P18:01
harlowjanetwork-config isn't the internally used format18:02
harlowjaits converted into something else18:02
harlowja(for some reason)18:02
smoserhm.. ok.18:03
harlowjaya18:03
harlowjahttps://code.launchpad.net/~harlowja/cloud-init/cloud-init-net-refactor has some examples/tests that i added for this18:04
harlowjahttp://bazaar.launchpad.net/~harlowja/cloud-init/cloud-init-net-refactor/revision/1227 (one test)18:04
harlowjabut ya, smoser there is a internal format that i'm not sure why it was created :-P18:05
smoserwell, rharper did that. and i dont really care if theres an internal format as long as you can got fmt1 <-> Internal <-> fmt218:06
smoserand ideally losslessly18:06
harlowjaya, i'm not sure its loseless18:06
smoserright. well, /etc/network/interfaces as a format is lossy in and of itself.18:06
smoserbecause it does not declare the mac or of the nic. just its name.18:07
smoserso at least you have to read from the /sys and hope for the best state :)18:07
harlowjahttp://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/sources/DataSourceConfigDrive.py#L30918:07
harlowjathe converter ^18:07
harlowjabazaar.launchpad.net/~harlowja/cloud-init/cloud-init-net-refactor/  i think helps here, so ya, merge it, lol18:08
harlowjahttps://code.launchpad.net/~harlowja/cloud-init/cloud-init-flake8-fixups/+merge/294548 also18:09
smoserharlowja, i'm generally ok with fixing up lint things (and pyflakes things)18:15
smoserbut its obnoxious when that is a never ending task, and you end up chasing your tail to work in all different versions18:15
smoserfor example, pyflakes for me is perfectly happy.18:16
smoserbut for you it is not18:16
harlowjawell some of these are somewhat basic issues18:18
smoseryeah, but why does my pyflakes not care18:18
harlowja__metaclass__ afaik is gone in py3 :-/18:18
smoserwhen mine is >= to yours for all of pep8, flake8 and pyflakes18:18
harlowjaand stuff like class A: is different on py3 vs p2 where u need to do class A(object)18:18
smoserwell, sure. and that is all ok.18:19
smoserbut why does my system think its ok.18:19
smoserand yours not18:19
smoseri'm not ok randomly fixing crap like that18:20
harlowjahttps://gist.github.com/harlowja/fb8a1c2c27ecf433daf71524f150bc1818:21
harlowjai just installed all the same versions :-P18:21
harlowjayour system is weird, lol18:21
harlowjaohhh i think i know18:21
harlowjacan u install hacking==0.10.218:22
smoseri'm using this thing called a distribution18:24
smoseri get stuff from it.18:24
harlowja:-P18:24
harlowjaya, it should be packaged under some name18:25
harlowjahttps://github.com/openstack-dev/hacking is the project18:25
smoserwow18:27
smoserthat is quite helpful (/sarcasm)18:27
smoserapt-get install python3-hacking and all of a sudden flake8 decides to act differently.18:27
harlowjaya....18:27
harlowjaaction at a distance :-P18:28
smoserneither of these packages indicate interaction with one another.18:28
harlowjaya, i know, i forgot about it18:28
harlowjabut they do :-/18:28
smosersee the 'chasing your tail' comment above ?18:29
harlowjabut the issues its raising are still useful ;)18:29
smoserso what do you do about this sort of thing.18:29
harlowjahttps://code.launchpad.net/~harlowja/cloud-init/cloud-init-flake8-fixups/+merge/294548 (imho fixes a bunch that i think are useful)18:29
smoseri really dont like it when random dude on the internet uploads a new python-<foo> package18:30
harlowjalol18:30
smoserand then my c-i (which is not present for cloud-init, but should be) starts to fail18:30
smoseri dont find that terribly useful.18:30
harlowjayes sir, u are from a distro company, i can tell :-P18:30
harlowjau must work at canonical18:30
harlowjalol18:30
smoserif everything always got better, that'd be one thing.18:30
smoserbut i have evidence otherwise.18:31
smoserhttps://bugs.launchpad.net/pyflakes/+bug/156013418:31
smoseron your python packages, what do you do ? do you pin versions of stuff in tox ?18:31
harlowjayup18:31
smoseror do you just love the daily activity of changing source code so you can fit joe-random-dude's daily gut feeling on what is the best way to write code.18:31
harlowjaas long as joe random dude == josh random dude18:32
harlowjawhere josh random dude == me18:32
harlowjalol18:32
smoseralright. so lets pin versions then.18:32
harlowjak18:33
smoserfor curtin i added a 'trusty' tox target18:33
smoseras you can write code that is happy on today's version of "good" but not april 2014's version of good.18:33
harlowjai trusty u18:34
harlowjalol18:34
harlowjaok in https://code.launchpad.net/~harlowja/cloud-init/cloud-init-flake8-fixups/+merge/294548  i froze the lint requirements18:35
smoseri do like the alphabetical order on imports18:38
smoserharlowja, i'll merge that if you read and offer thoughts on http://paste.ubuntu.com/16380605/18:47
harlowjayes boss18:50
harlowjasmoser looks ok, i think i'd want each datasource to provide a setup_network (that most can just not implement) and cloud-init will call this on first boot (if there are multiple datasources that impl this that are active, we can i guess pick the first)19:01
harlowjathat will get rid of some of the weird networking crap in config drive19:02
smoserharlowja, well, as it is right nwo in trunk the datasource may provide a network_config19:03
smoserand if it does, then cloud-init (in stages.py) will aply networkign as described there.19:04
cpaelzersmoser: did you know that long key fingerprints where broken?19:04
harlowjaoh ya, right19:04
cpaelzersmoser: next MP will contain a fix but that was a good idea to test the long fingerprints19:05
harlowjasmoser  http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/sources/DataSourceConfigDrive.py#L114 still exists though :-19:05
cpaelzersmoser: the embedded shell script lackes a few " and due to that it was broken for long fingerprints with spaces19:05
harlowja:-/19:05
smosercpaelzer, :). how are they broken ?19:06
smoserharlowja, yeah, that is yucky.19:06
smoserthats why i was asking about converting network-interfaces into network-config19:06
smoseras ideally if there was an interfaces style file there, then we'd just convert it to a network-config and use it there.19:06
cpaelzersmoser: it is well hidden and just appears to not work, but at the root cause changing gpg --keyserver ${ks} --recv $k to use "${k}" does the job19:07
harlowjasmoser hmmmm19:07
harlowjai mean, i can make a shitty converter19:07
smoser:)19:07
harlowjaone already exists (but it doens't go to network-config)19:07
cpaelzersmoser: I reorganized the code a bit to allow testing down to the last level so it is not only fixed but should be catched next time19:07
harlowjasmoser http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/distros/net_util.py19:08
harlowja^ shitty converter19:08
harlowjau can just take the output format of that and turn it into network-config19:08
harlowjaor whatever format u want, (of the 3+ that exist in cloud-init to represent network stuff, lol)19:09
smoseryeah :-(19:09
smoserwe really need to have one internal representation19:09
harlowjayup19:09
smoseri guess i'm ok with that being 'network state'19:09
harlowjaya, that format isn't to well defined19:09
harlowjai tried deciphering it, lol19:10
harlowja cause network state is really just a object with a version and a dict (the dict is the actual info)19:11
harlowjahttp://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/sources/DataSourceConfigDrive.py#L308 (network_config there seems to be the actual format)19:12
harlowjaand i sorta tried to figure out what network_config is, lol19:12
harlowjabecause network_config seems to be the dict that network state actually uses19:12
harlowjaso ya, it'd be nice to well have that more well ummmm, defined, lol19:12
harlowjacause http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/net/network_state.py#L72 sorta just sets some random attributes, hard to tell what the real format is :-P19:14
smoserharlowja, curtin has the best examples of it.19:14
smoserwe do need some much more formal documentation on that.19:14
harlowjadef19:14
smoserhttp://bazaar.launchpad.net/~curtin-dev/curtin/trunk/files/head:/examples/19:15
harlowjayup yup19:15
harlowjasooo i'd be ok with that being the internal cloud-init format19:15
harlowjaand then adjusting other places19:15
harlowjaand having a loseless(?) openstack -> that format converter19:15
* harlowja won't ask why the openstack format wasn't just used :-P19:16
harlowjabb, food19:16
smosercpaelzer, hm.19:19
smoseri'm looking at that export key thing. and it is crap.19:19
smoserwhich is probably not a surprise to you :)19:19
smoserbut it does seem to work for long fingerprint also19:20
cpaelzerwith " it works for long19:22
cpaelzerwithout bash might break it19:22
cpaelzerand the gpg tool considers it a list of short id's19:22
smoserhttp://paste.ubuntu.com/16381150/19:23
smoserit works for me for long and short as is.19:23
cpaelzerhaha19:23
cpaelzersmoser: wait for my example19:23
cpaelzersmoser: use this as the long key19:24
cpaelzersmoser: "B59D 5F15 97A5 04B7 E230  6DCA 0620 BBCF 0368 3F77"19:24
cpaelzersmoser: which is how it is listed by some tools, so people filling the config file might copy and paste it with spaces19:24
cpaelzersmoser: dpending on how things are connected it wither selects 5F15 as keyserver or tries to fetch 10 shortkeys with their ids19:25
cpaelzersmoser: both wrong19:25
smoserah. yeah.19:26
smoserso yeah, that shoudl be quoted for sure.19:26
smoserits really crap that it has to recieve them19:27
smoserit tries to clean up after itself, and delete the thing (and not delete it if it found it locally already).19:27
cpaelzersmoser: I saw it19:27
cpaelzersmoser: all kind of messy if things get interrupted19:27
cpaelzersmoser: yet I didn't want to fix the whole world in one day19:28
cpaelzersmoser: I updated the MP and the commit message19:28
smoserwell, its not really *too* messy.19:28
smoserroot's local .gpg will just know of that fingerprint19:28
smoserits not really the end of the world19:28
cpaelzerah is that local - yeah it is just gpg and not apt-key19:28
cpaelzerI'll call it EOD then19:29
harlowjasmoser what about extracting the curtin and net stuff into a curtin_net or something package?19:29
cpaelzersmoser: is there a good way of running curtin e.g. vmtests bundled with that experimental cloud-init?19:29
harlowjathis would just have the format, the converters and renderers and that's about it19:29
cpaelzersmoser: if there is please drop me a mail, so I can use it tomorrow19:29
harlowjainstead of being in cloudinit or being in curtin19:29
harlowja(or make other fancy name besides curtin_net)19:30
harlowjalol19:30
smoseroriginally i intended to have curtin's net and block modules move externally to something else.19:30
smoserbut then rharper and i kind of decided that we would like for it to be in cloudd-init.19:31
harlowjahmmm19:33
dcrouchIs it possible to read mdata from smartos within cloud-init?19:50
smoserdcrouch, what is mdata ?19:51
smoserah. i see.19:52
smoserdcrouch, http://bazaar.launchpad.net/~smoser/cloud-init/trunk.joyent-cleanup/revision/121619:52
dcrouchsmoser, Basically I can provide metadata to the server from smartos.  mdata is that metadata, for example.  https://dpaste.de/sDKS19:53
smoseri started a re-work of the smartos datasource that has a much simpler to use client19:53
smoserhttp://paste.ubuntu.com/16381487/19:55
dcrouchI'm thinking maybe I can write a python module, pull that data, and have it loaded on firstboot for items such as root password.19:55
smoseroh. well, the answer then is yes19:56
smoserand cloud-init already has a datasource for smartos19:56
smoserand it reads that stuff.19:56
smoseryou give cloud-init config and it does what you tell it19:56
dcrouchI understand that, but when we create our images in SmartOS, I need a way to send data to the vms  for firstboot, so was looking how to get this for ubuntu.19:57
dcrouchhttps://dpaste.de/TFYr19:58
smoserdcrouch, that client is only in the branch i pointed you too above.19:58
smoserdcrouch, you're the cloud ?19:59
smoserie, you're smartos ?19:59
dcrouchCorrect smoser!20:00
smoservendor-data20:00
smoserthats what you want.20:00
smoserthats what it was designed for basically.  vendor-data is user-data for the vendor20:01
smoserlook at DataSourceSmartOS and read the comments around 'vendor-data' there :)20:01
smoserharlowja, you added 'igmore' of all those flake8 stuff20:02
smoserH404,H40520:02
dcrouchsmoser, I've been looking up cloud-init for the last few hours and a few topics around there, I thought I recognized your name!    https://github.com/number5/cloud-init20:02
smoserbut why ?20:02
harlowjasmoser less things for me to fix in 1 merge, lol20:02
smoserbut they're all fixed20:03
harlowjau can fix all the 404 and 405 ;)20:03
harlowjaoh20:03
harlowjawelllll then (makes up other reason)20:03
smoserat least i just comment that line out20:03
smoserand type 'tox -e pyflakes'20:03
smoserand ENOERROR20:03
harlowjahmmm, run flake8 not pyflakes20:04
* smoser shames fist at his feeble brain20:04
smoseri confuse those two things all the time20:04
harlowjahttps://gist.github.com/harlowja/32845acbd0301fc2132fbd9255bddd1d20:04
harlowja405 and 40420:04
harlowjaand i'm like screw that, its a dumb warning, lol20:04
smoseri assumed you had added a tox20:04
harlowjanah, i didn't want to do everything, gotta leave u some work to20:05
smoseryeah. thanks.20:05
harlowjai'm nice like that20:05
smoserharlowja, so i'm gonna change pyflakes to flake820:09
smoserwhich i think is a strict superset of pyflakes20:09
smoserright?20:09
smoserand then i'm going to fix all the tests/ so that it wont complain about those either20:09
harlowjacool20:10
harlowjasound good to me20:11
harlowja(yes afaik its a superset of)20:11
dcrouchThanks for your help smoser I think I'll be back still confused but need to take off for a bit.20:48

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!