| === rangerpb is now known as rangerpbzzzz | ||
| harlowja | :( | 00:29 |
|---|---|---|
| harlowja | "WTH SMOSER!" | 00:29 |
| harlowja | lol | 00:29 |
| harlowja | ;) | 00:29 |
| smoser | mgagne, this is same place that you had run and copied that config drive out ? | 02:01 |
| smoser | mgagne, i'd really appreciate some help working this out tomorrow. i think you're the only thing i have left on https://code.launchpad.net/~smoser/cloud-init/trunk.fix-networking/+merge/296272 to get right. it seems working well other places. | 03:14 |
| smoser | the latest commit there (available in the ppa - 1251) will name the devices whatever they happened to be named. | 04:27 |
| smoser | but i've also tested in a container and modifying the config drive data to match my MAC addresses that it works. | 04:28 |
| mgagne | smoser: I found the issue | 16:10 |
| mgagne | there was interface config left in /etc/network/interfaces which conflicted with cloud-init.cfg. The interface found in /etc/network/interfaces looks to be the one before rename | 16:11 |
| mgagne | but it still doesn't ping | 16:12 |
| mgagne | gateway is not configured at all in 50-cloud-init.cfg | 16:12 |
| smoser | mgagne, ok. great. so cloud-init will not write /etc/network/interfaces | 16:13 |
| smoser | if you have an image with that in it, then i consider the image broken... | 16:13 |
| mgagne | yea, I used to delete interfaces in there. I removed it in Xenial in hope it got fixed since but it didn't so I restored it | 16:14 |
| mgagne | smoser: it is an image built from an iso | 16:14 |
| mgagne | and I install cloud-init in there | 16:14 |
| smoser | well, you shoudl use our cluod images :) | 16:14 |
| mgagne | it worked before =) | 16:14 |
| mgagne | anyone building an image from iso has a broken installation and isn't compatible with cloud-init | 16:15 |
| smoser | but basically ENI needs to be writen like http://paste.ubuntu.com/16951086/ | 16:15 |
| mgagne | ok, I ended up with something similar, all but lo interface | 16:15 |
| smoser | mgagne, well you had static configuration in a image that waas to be used in a dynamic (multiple instances) thing. | 16:15 |
| smoser | that clearly can't work | 16:15 |
| mgagne | smoser: original image is built with dhcp | 16:15 |
| smoser | it used to work only because | 16:15 |
| smoser | a.) 'eth0' was static | 16:16 |
| smoser | b.) we only supported massively simple networking | 16:16 |
| smoser | now with fancier networking supported, that can't work. | 16:16 |
| mgagne | ok, will continue to clear interfaces like before | 16:16 |
| smoser | i'd really rather not have cloud-init open that file and rip out stanzas | 16:16 |
| mgagne | now for the gateway =) | 16:16 |
| smoser | yeah. that is a issue. | 16:16 |
| mgagne | smoser: right, fine with me regarding ENI clearing | 16:17 |
| smoser | whats the network_json ? | 16:17 |
| mgagne | one with default route in routes =) | 16:17 |
| mgagne | let me setup network so I can scp that thing | 16:17 |
| smoser | :) | 16:17 |
| smoser | if only something could set up networking for you :) | 16:17 |
| mgagne | grep WARN returns nothing | 16:17 |
| mgagne | what I found weird though is that | 16:17 |
| mgagne | if network fails for whatever reasons | 16:18 |
| mgagne | EVERYTHING stops | 16:18 |
| mgagne | no cloud-config, hostname, etc. | 16:18 |
| mgagne | we could argue that | 16:18 |
| mgagne | cloud-config might need network to work properly | 16:18 |
| smoser | well, "everything stops"... | 16:18 |
| mgagne | but that's a huge inconvenience when debugging network issue, have to reboot in single mode | 16:18 |
| smoser | what happens is if cloud-inti writes busted networking configuration (or busted networking configuration somehow comes into being) | 16:19 |
| mgagne | well, my hostname is localhost and password doesn't work =) | 16:19 |
| smoser | then boot will just never get to the point where networking is available | 16:19 |
| smoser | so things that depend upon it (such as cloud-config and cloud-final) can't run. | 16:19 |
| mgagne | hmm so boot will stall forever? (or close to eternity) | 16:19 |
| smoser | i'm not sure exactly... it systemd timeouts ont hte stuff. | 16:19 |
| smoser | but since cluod-init-final and cloud-config "require" those things, they wont run | 16:20 |
| smoser | because those were never there. | 16:20 |
| mgagne | boot is too fast and I don't know systemd that much to debug what happened at boot time =) | 16:20 |
| mgagne | I sure did glanced at a "failed" at boot time | 16:20 |
| mgagne | smoser: http://paste.openstack.org/show/Q3N1BWnP2ecC9yoYcHpk/ | 16:23 |
| mgagne | with actual IPs first 3 bytes changed | 16:24 |
| mgagne | I do not own 1.1.1.0/24 =) | 16:24 |
| smoser | k. thanks. | 16:24 |
| mgagne | is there any log outputted when default gw is configured? | 16:39 |
| mgagne | smoser: will need to rebuild image from latest cloud-init I guess, looks like you made a change 12h ago | 16:44 |
| smoser | mgagne, yeah. that was to not name tap-ASDFASDF | 16:45 |
| smoser | mgagne, i'll get that fixed. | 16:45 |
| mgagne | right, will rebuild and see if it changes anything | 16:45 |
| smoser | mgagne, can you just easily mount the image and dpkg -i it ? | 16:48 |
| smoser | for quicker iteration | 16:48 |
| mgagne | well, I guess that would be possible. Need to clean cloud-init state right? | 16:49 |
| mgagne | still no routes but interface names are preserved now | 16:50 |
| mgagne | in fact, I will test with additional routes and see if those are configured | 16:50 |
| mgagne | ok, additional routes are not configured | 16:56 |
| mgagne | so I feel like routes field isn't even parsed | 16:56 |
| mgagne | is there a way I can test the network config parser manually? | 17:05 |
| mgagne | I see a lot of print debugging and they aren't logged at boot time | 17:06 |
| smoser | mgagne, give me a bit, i'm on a call. you can definitely test. | 17:07 |
| smoser | mgagne, really appreciate your time, thank you | 17:07 |
| mgagne | =) | 17:07 |
| mgagne | thanks for help ;) | 17:07 |
| === JayF is now known as notJayF | ||
| === notJayF is now known as JasonF | ||
| smoser | mgagne, so to summarize, gateway and static routes ? | 18:25 |
| smoser | is that what we're missing | 18:25 |
| mgagne | yes | 18:25 |
| smoser | k. | 18:25 |
| smoser | i'll take a look (long call) | 18:25 |
| mgagne | I switched to other tasks but I'm back now. is there a way I can test the network config/parsing? at first glance, I don't see any logic around routes parsing which looks for the "0.0.0.0" destination. "0.0.0.0" destination would be the default gateway. But I might have missed it. | 18:27 |
| smoser | mgagne, i just took your paste and am trying to put together a test | 18:31 |
| smoser | there is a test in tests/unittests/test_datasource/test_configdrive.py that does the convert | 18:31 |
| mgagne | ok, I was more or less hoping for a way to run test after the fact against configdrive to see what cloud-init "thinks" about it | 18:32 |
| mgagne | but I guess it's a feature that would be of limited use and for dev only | 18:33 |
| smoser | rharper, http://paste.ubuntu.com/16956870/ | 18:36 |
| smoser | that is me taking mgagne's paste and putting it into a test to run conversion to network config on . | 18:36 |
| smoser | and then we'll see | 18:36 |
| rharper | y | 18:36 |
| smoser | http://paste.ubuntu.com/16956919/ | 18:37 |
| smoser | thats what it prints otu. | 18:37 |
| mgagne | does this format expect the field gateway to be defined? | 18:38 |
| smoser | rharper, http://paste.ubuntu.com/16957033/ <-- my current diff | 18:40 |
| smoser | produces http://paste.ubuntu.com/16957018/ | 18:40 |
| smoser | so yeah, we dont get the stuff rendered at the moment. | 18:40 |
| rharper | yeah, there's a gateway in the routes; the render_routes() doesn't seem to be getting called on a per-subnet basis | 18:40 |
| rharper | it should | 18:40 |
| rharper | basicall under the for subnet in subnets, there should be an if 'route' in subnet, etc... IIRC; so somethings not right; | 18:41 |
| rharper | something like this: http://paste.ubuntu.com/16957110/ | 18:41 |
| smoser | http://paste.ubuntu.com/16957213/ | 18:44 |
| rharper | almost, need to pass in an indent=" " | 18:45 |
| rharper | smoser: lp:~raharper/cloud-init/trunk.fix-networking-subnet-routes | 18:47 |
| smoser | rharper, right. | 18:48 |
| smoser | http://paste.ubuntu.com/16957377/ | 18:48 |
| smoser | does that look right .. | 18:48 |
| rharper | yes, the pre post | 18:49 |
| smoser | seems sane | 18:49 |
| smoser | how do you suggest we test this... | 18:49 |
| smoser | comparing the rendered eni is a pita | 18:49 |
| rharper | I'm just about to run through cloud-init-test | 18:49 |
| rharper | VM booting and validating the values get set, ala vmtest | 18:50 |
| rharper | we at least get feedback that route_n, ifconfig_a matches what's in eni that got rendered | 18:50 |
| smoser | well, right. but we really need a way to unit test compare more easily | 18:50 |
| smoser | i'lll put a crappy test in | 18:50 |
| rharper | we have a compare rendered eni test in place | 18:51 |
| rharper | it's just inside the vmtest, it includes parsing ifconfig_a output and the eni and such; you can lift that, and we just need to presupply ifconfig_a and route_n output | 18:51 |
| rharper | that's expected | 18:51 |
| rharper | which shouldn't be too difficult to construct; it could be templatized w.r.t specifics in the source network_data.json (like mac and ip and netmask and such) | 18:52 |
| rharper | smoser: see curtin's tests/vmtests/test_network.py for the basics; I have an updated version for cloud-init-test here: https://git.launchpad.net/~raharper/+git/cloud-init-test/tree/tests/vmtests/test_network.py | 18:53 |
| smoser | rharper, http://paste.ubuntu.com/16957806/ | 18:58 |
| rharper | smoser: y | 18:58 |
| rharper | smoser: and ideally you extend and add a second non-gateway route to the routes array, and check that we do non 'default' key | 18:59 |
| rharper | and an ipv6 one, that'll fully exercise the render_routes() method | 19:00 |
| smoser | ok. rharper what am i missign from you ? | 19:01 |
| smoser | you had one fix for me. | 19:02 |
| smoser | right? | 19:02 |
| rharper | lemme get you branch | 19:02 |
| rharper | http://paste.ubuntu.com/16958041/ | 19:03 |
| rharper | that'll let the LinuxBridge network_data.json work | 19:04 |
| rharper | I sent you the configdrive data for that earlier today | 19:04 |
| smoser | andthere is a fix for mtu there ? | 19:05 |
| smoser | rharper, ^ | 19:05 |
| smoser | yeah. ok . i see | 19:06 |
| rharper | smoser: yes, it's a physical/link property, not subnet property | 19:06 |
| rharper | that said, if we bring it in, then we likely need a bit more to convert it | 19:07 |
| rharper | I have some data with mtu: null, | 19:07 |
| smoser | committed and pushed | 19:07 |
| rharper | so, it needs some care to drop the key, or if it's not null, apply int() on it | 19:07 |
| smoser | ? | 19:07 |
| rharper | { | 19:08 |
| rharper | "links": [ | 19:08 |
| rharper | { | 19:08 |
| rharper | "ethernet_mac_address": "fa:16:3e:ed:9a:59", | 19:08 |
| rharper | "mtu": null, | 19:08 |
| rharper | if that's not null, but a number, like 1500 | 19:08 |
| rharper | we want to convert to int | 19:08 |
| smoser | http://paste.ubuntu.com/16958299/ | 19:09 |
| smoser | it does get rendered there. | 19:09 |
| smoser | ie, its safe to treat it as a string. | 19:09 |
| rharper | ok, then maybe just handle the null case | 19:09 |
| smoser | seems to work. | 19:10 |
| smoser | http://paste.ubuntu.com/16958359/ | 19:10 |
| smoser | that showed it with None | 19:10 |
| smoser | and also tested with it as not present | 19:11 |
| smoser | and it just doesnt appear | 19:11 |
| smoser | so good work :) | 19:11 |
| rharper | smoser: yeah, working here on cloud-init test and latest fix-networking branch | 19:13 |
| smoser | mgagne, fwiw, why do you not use our images ? | 20:11 |
| mgagne | smoser: it's not the flow we adopted a couple of years ago. The tool we use (Oz) doesn't deal with .qcow2 but iso | 20:11 |
| mgagne | and the main thing we change in the image is the cloud-init package + config | 20:12 |
| smoser | what do you change ? | 20:12 |
| mgagne | there never was a time in our history where cloud-init works as-is in our cloud | 20:12 |
| smoser | (and the argument of "We use Oz" is like if i asked you 'why do you compile glibc' and you said "we use -O2") | 20:13 |
| mgagne | + we need to backport cloud-init for older releases | 20:13 |
| mgagne | smoser: we have a unified build system for all OS, including Windows | 20:13 |
| mgagne | this is a compromised we made to make it easy for us to enroll now dev so they don't have to learn new tools for each OS | 20:13 |
| smoser | again... yoou're arguing "we use compiler foo". i'm saying "don't compile" | 20:14 |
| mgagne | we aren't closed to the idea of switching workflow/tool, it just happened to not be our priority for now. priority being infra ops | 20:14 |
| smoser | we give you somethign that is supposed to "just work" | 20:14 |
| mgagne | well I don't think this discussion will be productive tbh | 20:14 |
| smoser | (i understand your argument and don't mean to sound like i'm fighting... we just work to make our downloadable images "just work". | 20:15 |
| mgagne | they never worked | 20:15 |
| smoser | the same way we work to make our compiled kernel or python or glibc "just work" | 20:15 |
| mgagne | cloud-init used to not support cdrom configdrive. we had to patch it (or backport) | 20:16 |
| smoser | definitely. there are bugs, and our sru process is slow. | 20:16 |
| smoser | (i really *really* appreciate your help) | 20:16 |
| mgagne | we inject password in admin_pass field, cloud-init doesn't support that feature https://bugs.launchpad.net/cloud-init/+bug/1236883 | 20:16 |
| smoser | we can work to make our images "just work" for you. which will make them "just work" for our customers who use RDO . which is a win for everyone. | 20:16 |
| mgagne | smoser: tbh, this would thing is *very* frustrating | 20:16 |
| smoser | oh , well thats just silly. | 20:17 |
| mgagne | it just happened that we don't feel there is any progress despite opening bugs | 20:17 |
| smoser | use ssh keys :) | 20:17 |
| mgagne | ... | 20:17 |
| mgagne | we very well know that ssh keys exist | 20:18 |
| mgagne | that's not the UX we are selling our users | 20:18 |
| mgagne | and they expect | 20:18 |
| mgagne | I respond to our users expectations in a reasonable fashion. I sure won't install golang by default if they need it. it's their problem. | 20:19 |
| mgagne | I can come up with a list of "features" we provide and see where cloud-init lacks | 20:20 |
| smoser | lets work to make this better. i agree there is lots of room for improvement. | 20:20 |
| mgagne | smoser: I'm glad we can work this out. | 20:20 |
| mgagne | smoser: the most best ever feature that could come up from this is backporting of cloud-init to old releases. | 20:21 |
| mgagne | this is really the thing that causes the most pain | 20:21 |
| smoser | to old ubuntu ? | 20:22 |
| mgagne | all those fixes you are doing to cloud-init (thanks again btw), trusty might never see the light of it due to SRU policies. So we end up needing to backport it ourselves and then rebuild the image ourselves. | 20:22 |
| smoser | at this point trusty has issues also because of systemd | 20:23 |
| smoser | err.. lack of. | 20:23 |
| mgagne | I can understand that it won't be backported to 12.10. I agree with don't care about EOL releases. | 20:23 |
| smoser | doing the work to get this well functional again in upstart will require some work. | 20:23 |
| mgagne | but trusty... it needs to work | 20:23 |
| smoser | yeah. | 20:23 |
| smoser | its ok for you to be blunt with me. i promise :) | 20:23 |
| mgagne | smoser: it used to work, thanks for jayofdoom patch. but we want to get ride of that. and so far, we don't know how to make that happen without canonical contribution. | 20:24 |
| mgagne | a couple of walls have been punched lately tbh | 20:24 |
| smoser | ah. the networkign_json for trusty ? | 20:24 |
| mgagne | yes | 20:24 |
| smoser | it will ble non-trivial :-( | 20:24 |
| mgagne | it used to work for 3rd party patch =) | 20:25 |
| smoser | this merge just did is quite non-trivial | 20:25 |
| mgagne | so there is a way | 20:25 |
| mgagne | I wouldn't mind contributing code. what I'm lacking is experience with bzr (and knowledges of cloud-init inner working). Unfortunately, not a lot of people like it in our team | 20:27 |
| rharper | heh, now that it's merged, maybe smoser will push to +git on launchpad | 20:30 |
| harlowja | git git gi | 20:47 |
| harlowja | git | 20:47 |
| harlowja | git | 20:47 |
| harlowja | lol | 20:47 |
| waldi | igit? | 20:55 |
| harlowja | ugit | 20:57 |
| harlowja | weallgit | 20:57 |
| rharper | harlowja: nice | 20:57 |
| harlowja | make git great again | 20:58 |
| harlowja | lol | 20:58 |
| === JasonF is now known as JayF | ||
| ajorg | cloud-init is on github now, but I don't see any pull requests there. are contributions still to be directed at bzr / launchpad? | 22:15 |
| ajorg | would patches to make cloud-init compatible with python 2.6 be welcomed? | 22:59 |
| harlowja | which cloud-init btw? | 22:59 |
| harlowja | just want to make sure the source repo u are working on is the right one | 23:00 |
| ajorg | 0.7.x | 23:00 |
| harlowja | ya, i fixed a bunch in a merge i have pending | 23:00 |
| harlowja | prob similar to ones u fixed | 23:00 |
| ajorg | ah, good | 23:00 |
| ajorg | os.uname in cloudinit/distros/__init__.py is one of them | 23:02 |
| ajorg | was not a propery in 2.6 | 23:02 |
| harlowja | hmmmm | 23:02 |
| harlowja | unsure if i fixed that one, doesn't sound familar | 23:02 |
| harlowja | do u have a set of patches with the ones u fixed | 23:03 |
| harlowja | i can make sure i incorporate (or already might have) | 23:03 |
| ajorg | I do, but I have a lot of patches and need to sort them out now that I'm able to contribute them. | 23:03 |
| ajorg | many of them not for 2.6 | 23:03 |
| harlowja | ok, anyway u can figure out the 2.6 ones | 23:04 |
| ajorg | I have been maintaining cloud-init for the Amazon Linux AMI, so I have a lot of RPM / Yum distro patches. | 23:04 |
| ajorg | On monday I'll try to post a branch with my 2.6 related patches for you. | 23:05 |
| ajorg | We're on 2.7 now, but the patches are still useful for CentOS / Red Hat 6 | 23:05 |
| ajorg | harlowja: what's the best time of day for me to find you and smoser? | 23:06 |
| harlowja | so scott moser is on US Central time zone | 23:06 |
| harlowja | i'm on US pacific | 23:06 |
| harlowja | so working hours those time zones is usually when we around | 23:07 |
| ajorg | i'm also on pacific, so that should work well for me. | 23:07 |
| harlowja | cool | 23:07 |
| ajorg | Some of our patches will likely require some discussion. | 23:07 |
| harlowja | but i bounce around alot, but sure let's discuss | 23:07 |
| harlowja | i'm all over the place in cloud-init and ... openstack and ... | 23:07 |
| harlowja | lol | 23:07 |
| ajorg | I'm actually headed home soon, so I'll have to catch you on Monday. | 23:07 |
| harlowja | k | 23:07 |
| ajorg | I'll try to be prepared. | 23:07 |
| harlowja | np | 23:08 |
| ajorg | is it best to put each patch on a feature branch? | 23:08 |
| ajorg | or can some of them that are closely related land together? | 23:08 |
| harlowja | closely related land together i think is ok | 23:09 |
| harlowja | (with-in reason, ha) | 23:09 |
| ajorg | k | 23:09 |
| ajorg | and several of our patches provide backward compatibility for our 0.5.x features (before any Yum / RPM support was added to cloud-init) | 23:10 |
| ajorg | oops, there he went. | 23:10 |
| ajorg | welcome back :-) | 23:10 |
| harlowja | ha | 23:10 |
| harlowja | ya, wireless -> wired | 23:10 |
| ajorg | was just saying that several of our patches are for backward compatibility with our old 0.5 fork. are those likely to be accepted? | 23:11 |
| harlowja | unsure | 23:11 |
| harlowja | are u from amazon?? :-P | 23:11 |
| ajorg | yes. | 23:11 |
| harlowja | ah | 23:11 |
| ajorg | I'm the cloud-init maintainer here. | 23:11 |
| harlowja | lucky guess, i didn't know anyone else running 0.5 :-P | 23:11 |
| ajorg | haha | 23:11 |
| ajorg | we haven't been for a long time | 23:11 |
| harlowja | ah, maybe guess was more about who has been running cloud-init for that long then | 23:12 |
| ajorg | but we landed a bunch of yum / rpm features and our customers used them | 23:12 |
| harlowja | in fact it was mostly just lucky, lol | 23:12 |
| harlowja | how's seattle :-P | 23:12 |
| ajorg | great actually | 23:13 |
| ajorg | love it up here | 23:13 |
| harlowja | cool | 23:14 |
| harlowja | but ya, can't hurt to propose them imho | 23:15 |
| ajorg | i've recently become able to contribute any of our current patches, of which we have a variety | 23:15 |
| harlowja | ya, no doubt | 23:15 |
| ajorg | 29 patches at the moment, some of them features, some bugfixes, etc. | 23:16 |
| harlowja | i remember a cloud-init bug many years old that had a amazon person on it, trying to give a patch file and i think smoser couldn't touch it (legal reasons idk?) | 23:16 |
| harlowja | i wonder if that bug is still there somewhere, ha | 23:16 |
| ajorg | if it had something to do with region names that was probably me. | 23:16 |
| ajorg | the regex was (is?) too strict. | 23:17 |
| harlowja | unsure, lol | 23:17 |
| ajorg | i'll be more open this year, i promise. | 23:18 |
| harlowja | woot | 23:18 |
| ajorg | anyway, i need to find a bus. i'll catch up on monday, hopefully | 23:18 |
| harlowja | cool | 23:18 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!