[03:44] http://termbin.com/16tk [03:44] why am I getting this error? [05:02] is ANYONE around? [05:02] ailed to start Initial cloud-init job (pre-networking). [05:26] http://termbin.com/8uxe [05:26] this is on arch [06:49] . [06:49] anyone? === sterfield_ is now known as sterfield === shardy is now known as shardy_lunch === shardy_lunch is now known as shardy [13:22] jwilx, well, you're probalby missing python3 package for pkg_resources [15:11] smoser: very helpful... I already corrected that error, only to get this one http://pastebin.com/raw/nS6KXz0w [15:12] http://termbin.com/8uxe [15:12] it was working fine on arch [15:12] I did a clean install, and did nothing but pacman -Syu [15:12] and it screwed up so I had to install a bunch of python packages [15:12] I think it's cloud-init bugging out [15:12] somebody should fix it [15:15] come on surely somebody can help... [15:15] jwilx, are you on the system ? [15:15] could you try: [15:15] i'm on it [15:15] yes [15:15] python3 -c 'from cloudinit import util; import sys; print(util.load_file(sys.argv[1]))' /sys/class/net/eth0/carrier [15:16] i think harlowja was seeing that also [15:16] it's using python3 [15:16] 2 * [15:16] not 3 [15:16] [arch@arch ~]$ sudo python2 -c 'from cloudinit import util; import sys; print(util.load_file(sys.argv[1]))' /sys/class/net/eth0/carrier [15:16] 1 [15:17] hm. [15:18] fyi i'm using systemd networking here [15:20] that should not cause this error, but cloud-init does not have support at this time for rendering systemd-networkd networking configuration. [15:20] can you run [15:21] sudo python2 -c 'from cloudinit.net import generate_fallback_config; print(generate_fallback_config())' [15:21] {'version': 1, 'config': [{'subnets': [{'type': 'dhcp'}], 'type': 'physical', 'name': 'eth0', 'mac_address': u'fa:16:3e:37:ae:97'}]} [15:23] well that is odd. [15:23] thats where it stacktraced. [15:23] hm [15:23] smoser: I remember this from when I was trying to get fallback_net_config working I think [15:24] Depending on the state that the net interface is in, reading from the sys info is an IOError with errno 22 [15:24] But later if that net dev is in a clean state it can read [15:25] I think the issue is in python2 IOError does not inherit from OSError [15:25] magicalChicken, yeah, i just checked that it doesnt in py3 either. [15:25] but the catch works in py3 [15:25] i suspect it raises IOError and we're just never catching it [15:25] yeah [15:25] but even if we did catch it, i'm not sure what to do [15:26] the code handles that already [15:26] i dont think its py3 v py2 [15:26] hang on 1 sec [15:27] http://paste.ubuntu.com/23207506/ [15:28] also: http://paste.ubuntu.com/23207515/ [15:29] yeah, you're right [15:29] $ python -c 'print(isinstance(IOError(), OSError))' [15:29] False [15:29] $ python3 -c 'print(isinstance(IOError(), OSError))' [15:29] True [15:30] I just remember that cause it caused curtin but #1562003 [15:30] which started the clear_holders refactor [15:35] jwilx, are you able to try this: [15:36] http://paste.ubuntu.com/23207571/ [15:36] bug 1562003 [15:52] does the minus mean to remove? [16:01] jwilx: you can just use patch with that [16:02] so clone cloud-init, paste that into a file and do 'patch -p0 < $filename' [16:03] jwilx, yes. '-' means remove [16:03] and + add [16:03] or probably something like: [16:04] ( cd /usr/lib/python2.7/site-packages/ && patch --dry-run -p0 ) < that-content [16:09] http://termbin.com/zkqf [16:09] I get this [16:16] shoot.s orry, jwilx [16:16] jwilx, kill the final ')' off that line i think is all you need. [16:17] msg = "%s: load file failed with %s" % (fname, e) [16:17] this one? [16:18] i did and it is giving same error [16:18] as above [16:23] http://termbin/zoeb this smoser [16:23] http://termbin.com/zoeb [16:25] hm.. jwilx pastebin /var/lib/cloud-init.log ? [16:26] nothing there [16:30] smoser: http://termbin.com/93eq [16:30] that's output log [16:32] jwilx, someone was here working on arch support recently... [16:33] prometheanfire, ^ [16:33] maybe summoning him will make magic occur [16:55] hm [16:59] prometheanfire: any idea? [17:03] Hey guys! [17:05] Listen, I launching Ubuntu 16.04 on OpenStack, wired to 3 different subnets: subnet 1 is the default, have dhcp=True, subnet 2 is secondary (no dhcp), subnet 3 is the third (no dhcp). However if I disable the DHCP for a specific subnet (like I just said), Cloud Init still configured the IP as static!!! But I do not want any setup for the secondarie networks. As a workaroung, I'm doing this on cloud init script: "ifdown ens4 ; ifdown ens5" (while en [17:05] s3 is fine, first one). [17:05] so, how to avoid this workaround and tell cloud-init to configure only the 1 subnet (ens3 vNIC)? [18:05] smoser: I was working on gentoo [18:06] I need to get back on working with it, but with the openstack release coming up... [18:07] prometheanfire, aren't those the same thing [18:07] * smoser ducks [18:07] sorry for the bad ping. [18:08] :P [18:08] all the ricers left gentoo for arch ages ago [18:08] gentoo is much better off now :P [18:16] smoser: so any idea? [18:16] :/ [18:17] jwilx, well, the general 'None' bug is probably just not considering no dns information. [18:17] thats probalby fairly easily fixable. [18:17] can you file a bug against cloud-init ? [18:17] no idea how [18:17] and state where we've been and where you are now ? [18:17] https://bugs.launchpad.net/cloud-init/+filebug [18:18] jsut recap what is here. you can even reference here [18:18] https://irclogs.ubuntu.com/2016/09/20/%23cloud-init.html [18:18] k [18:29] https://bugs.launchpad.net/cloud-init/+bug/1625766 [18:43] smoser jwilx i've def seen the invalid argument one :-P [18:43] weird that it is [18:44] harlowja, magicalChicken suggests that that happens before nic is really initialized or brought up. and that we're just missing a exception [18:45] @smoser talking with nrezinorn and we were thinking about how we could get image building for cent7 into some public image publicing/testing pipeline [18:45] cause there be some weirdness with networking starting with DataSourceNoCloud [18:45] that may just be some kind of cent7 issue, but is still weird (same with the argparse crap) [18:46] harlowja, i have to run ... lke right now. [18:46] but i have been thinking about cloud-init integration test [18:47] i'll try to put something together on my thoughts and send to list tomorrow. [18:47] we'lls ee. [18:47] later. [18:59] whattt [18:59] lol [18:59] a chicken? [18:59] i caught the IOError [18:59] is this for the /sys/ question [18:59] ha [18:59] where u running [18:59] come back [19:00] lol [19:00] k [19:02] any suggestions for setting a http proxy with cloud-init? I've been trying to do so by exporting the http_proxy env variable via a runcmd: but it doesn't seem to take (i run a yum update several commands later and it fails because the proxy isn't set) [19:27] cwa, well, you could probabl do a bootcmd to write /etc/environment . cloud-init does not have any way to affect its environment. [19:28] smoser know how long it usually takes? [19:28] to get a response [19:28] here is something i do: http://bazaar.launchpad.net/~smoser/+junk/sstack-proxy/view/head:/sstack-proxy [19:28] jwilx, to get a response to the bug ? [19:28] yeah [19:28] thanks all, i'll take a look at that [19:29] well a while. [19:29] cwa, generally speaking http_proxy globally is a pita [19:29] meh guess I can try installing gentoo [19:29] while waiting [19:29] as http_proxy=http://your.proxy/ wget http://localhost:9999/ [19:29] and have that fail [19:30] really wanted to use arch but w/e [19:30] jwilx, what is the provider you ran on ? [19:30] lunanode [19:30] linode ?? [19:30] luna [19:30] https://www.lunanode.com/ [19:30] ah. k [19:33] smoser why do you need to know [19:39] jwilx, did you just pick from the template ? [19:39] smoser [19:39] yes. [19:39] it worked fine. until I did an update [19:39] ok. i'll maybe take a look tomorrow. [19:39] just signed up there, gave them $10 [19:40] ah [20:28] Hey guys, can someone help me? [20:42] do u require emergency assistance, if so please dial 911 [20:55] Are there any active users of cloud-init on FreeBSD? Anyone know if there is active development for FreeBSD? [20:57] harmw might know that x58 [20:57] i think he did a bunch of the cloud-init stuff for freebsd [20:58] though idk if it actually works :-P [20:58] Yeah, I have been talking with him. [20:58] He is no longer doing OpenStack [20:58] So he is no longer able or interested in maintaining cloud-init on FreeBSD. [20:58] harlowja: ^ [20:58] ok [20:59] u can be the replacement :-P [20:59] Yeah, he suggested that too ;-) I was updating the FreeBSD port to 0.7.8 but it's broken due to linuxisms that are not an easy fix. [20:59] ?? [20:59] https://bugs.launchpad.net/cloud-init/+bug/1625802 [20:59] it tries to read from /sys/class/net which doesn't exist on FreeBSD. [21:00] ya, that [21:00] poop [21:00] I just filed that bug. [21:00] k [21:00] There has been a patch for another issue: https://bugs.launchpad.net/cloud-init/+bug/1404745 [21:01] ya, perhaps we should bundle this into https://code.launchpad.net/~harlowja/cloud-init/+git/cloud-init/+ref/sys-io-errors [21:01] which was also simialr [21:01] that has been languishing in review... and tools/freebsd-build* hasn't been updated with appropriate depdendencies either. [21:01] ya, never enough people [21:01] nrezinorn and i have been working through some cent7 issues [21:01] some weirdness there [21:01] I had a patch for CentOS 7 issues, but don't want to go through hassle of signing CLA... [21:02] So I've got a fork running locally. [21:02] (I'm Bert on launchpad) [21:03] hassle [21:03] hmmm [21:03] idk how much of a hassle it is [21:03] hard to accept patches afaik without that :-/ [21:03] I'd have to get corp legal involved... [21:03] im pretty hassled harlowja :P [21:03] Only because of Canonical's policy of not accepting stuff under the GPLv3... [21:04] nrezinorn u should be able to contribute to cloud-init without problem [21:04] and wanting what is basically the right to relicense any and all contributions under whatever license. [21:04] no hassle for u [21:04] smoser and I had a discussion on the matter already. [21:04] k [21:04] ya, IANAL [21:04] to me the whole license blah blah, is more or/less blah blah [21:04] Hire me, then it won't be a problem ;-) [21:05] where are your cent7 patches? ;0 [21:05] :) [21:05] harlowja: https://code.launchpad.net/~bregeer-ctl/cloud-init/+git/cloud-init/+merge/305058 [21:06] kk [21:06] ya, nrezinorn was seeing something else [21:07] my "something else" is related to starting a VM locally in libvirt and feeding it a cdrom config drive with some data :P - pre-cloud QA tests if you will [21:07] precloud cloud [21:07] nrezinorn: I've been testing by uploading to glance and starting the image ;-) [21:07] Thankfully my cloud is local, and 1 Gbit/sec uploads into glance, so it doesn't take a long time to do testing. [21:07] ya, what nrezinorn say in 0.7.7 is that networking seems to not start initially (when using DataSourceNoCloud) but after reboot it does [21:08] which then made me wonder about some thing not recgonizing networking was changed (until after reboot) [21:08] * harlowja not sure if /etc/udev/rules.d/70-persistent-net.rules does that [21:08] or could cause that [21:08] or something else [21:09] Ah, that was another issue... I had to set cloud-init-local to start Before=NetworkManager.service [21:09] all the systemd dependencies (cloud-init-local before networking) seem ok [21:09] https://bugs.launchpad.net/cloud-init/+bug/1620845 [21:09] well damn [21:09] that would probably be it [21:09] lol [21:09] cloud-init-local and NetworkManager would start together... and race each other. [21:09] It was glorious (not, I wanted to shoot someone) [21:09] ya, sounds about right [21:09] nrezinorn may be about to shoot someone [21:10] not really sure [21:10] lol [21:10] My issue was related to /etc/resolv.conf but it also causes issues with ifcg-* files [21:10] and whatnot [21:10] i thought cloud-init-local had before=network-pre [21:10] guess that doesn't do it [21:11] https://git.launchpad.net/cloud-init/tree/systemd/cloud-init-local.service#n8 [21:11] assuming thats not enough [21:11] Nope [21:11] not enough. [21:11] stupid [21:11] lol [21:12] well, NetworkManager doens't have any require=network-pre in it... [21:12] gotcha [21:12] so NetworkManager will start even though network-pre is not yet "complete" [21:12] It's a dependency ordering issue. [21:12] def [21:13] Changing system shipped service files is not my thing though, so I forced cloud-init-local first by making it Before [21:13] lets just blame systemd? thats the cool thing to do right? [21:13] ya+1 [21:13] lol [21:13] torches ready [21:14] getting my pitch fork [21:14] Pitchforks for sale here! [21:14] ill locally make the changes for netmanager tho and see if that helps! [21:14] nrezinorn: no promises ;-) [21:14] "set cloud-init-local to start Before=NetworkManager.service" <-- this [21:14] lol [21:15] https://code.launchpad.net/~harlowja/cloud-init/+git/cloud-init/+merge/306283 [21:15] ya, that should do it ^ [21:15] i was handed down making golden images for the cloud. i didnt want to push junk into the cloud if it was totally broken, so i import it into libvirt locally and QA a bunch of crap then it ships to cloud (staging) [21:15] Yeah, I am currently trying to build an golden image for FreeBSD 10.3 but cloud-init is broken. [21:16] Now I need to go see how much time I can spend on this, and patch it just enough to work. [21:16] i have been mucking with this centos7 jazz for ...3 days? 4 days? i dont even know. [21:16] My local git server now has a fork of cloud-init with a bunch of branches fixing random stuff. [21:16] i hope that this also fixes the hostname issue i am seeing lol [21:17] nrezinorn: You'll want to look at the patches for /etc/resolv.conf as well (make it a symlink to something else so that NetworkManager leaves it alone) [21:17] Otherwise NetworkManager will happily overwrite your changes to /etc/resolv.conf [21:17] x58 can u please sign the CLA [21:17] with sugar on top [21:18] don't like out of tree fixes that someone has to go discover and recreate [21:19] harlowja: Patches are under GPLv3. Feel free to pull em in. At least then I can be assured my code won't suddenly go closed source (happened with maas-image-builder... for which I had not signed a CLA!) [21:19] ya, this is a weird gray area that idk about [21:19] like if i just submit your patches under my name [21:19] lol [21:19] Sorry, gonna be a stickler about it. I'll reach out to my boss to see if I can sign a CLA for stuff I did on work time. [21:20] harlowja: You'd be infringing my copyright... you can't agree to have canonical relicense something you didn't write. [21:20] Just saying. [21:20] but u just said ' Feel free to pull em in.' [21:20] lol [21:20] Yeah, GPLv3 allows you do that. [21:20] Just means you can't claim you wrote it :P [21:20] gotcha [21:20] and give someone permission to relicense. [21:21] That's what the CLA requires. [21:21] but canonical will if i submit them [21:21] so if i take them, make new pull request with them [21:21] and don't say that i wrote them [21:21] someone else will have permission to relicense right [21:22] If someone were to merge those into the tree, and Canonical ata later point in time were to relicense the entire codebase under an alternate license, that would be copyright infringement. [21:22] right [21:22] Standard copyright still applies even with GPLv3 [21:23] That being said. Let's just say I am Sergeant Schulz. I didn't see nothing. I didn't hear nothing. I don't know nothing. [21:23] so pulling them in i guess u are meaning, not into canonical cloud-init code-base [21:23] harlowja: You can pull them in, that would just preclude Canonical from ever relicensing the code base without my explicit approval first. [21:24] ya, to much blah blah [21:24] lol [21:24] haha [21:26] can i sell u some kool-aid [21:26] leol [21:26] *lol [21:28] harlowja: work with smoser and get cloud-init changed so that it doesn't require the CLA ;-) [21:30] ya, i'm not at canonical though :-P [21:30] only smoser holds those keys [21:32] Ah, thought maybe you were. [21:32] With the @ and all ;-) [21:33] nah, i'm just super-cool [21:33] lol [21:33] and been doing cloud-init for to long [21:33] lol [21:33] super cool? that's debatable lol [21:33] :) [21:33] super-awesome-cool [21:33] lol [21:35] lol [21:53] x58: your patch didnt fix the issue i was seeing, but maybe you know what i can look through [21:53] our gold images specifically delete the ifcfg-eth0 file [21:54] running the 0.7.7 i have to reboot after creating the vm with cloud-init and then my networking works. [22:00] x58 ^ any ideas? [22:00] nrezinorn afaik the delete happens way before cloud-init (or even the operating system) starts [22:00] in kickstart land [22:00] so from the image/operating system its like it never existed [22:09] nrezinorn: does cloud-init mention it creating ifcfg-eth0? [22:09] It should show some debug output for that. [22:09] Can you pastebin your cloud-init output somewhere? [22:09] journalctl -a | grep cloud-init [22:13] Ohhhhh [22:13] cloud-init-local.service, does it have Before=networking.target or something similar? [22:14] pretty sure nrezinorn added in the before=NetworkManager and think it has that also, will let nrezinorn respond [22:14] Hmmm, that's cloud-init.service [22:14] there is no "networking.service" in CentOS [22:14] it's "network.service" [22:14] so cloud-init.service depends on the wrong thing, [22:15] lol [22:15] Looking at my notes for building this CentOS image. [22:15] kk [22:15] man this systemd stuff [22:15] * harlowja gets my pitchfork [22:15] lol [22:15] networking.service is a Ubuntuism. [22:16] /etc/init.d/network is CentOS's [22:16] also make sure that network is enabled, especially if NM_MANAGED is set to NO in /etc/ifcfg-eth0 [22:16] systemctl enable network [22:16] (It should be by default) [22:16] otherwise a reboot wouldn't fix it. [22:17] You have an ordering issue somewhere... cloud-init-local is not running early enough (before network.service) [22:19] so nrezinorn when u get back ^ [22:19] nrezinorn: http://pastebin.ubuntu.com/23209143/ is a successful run. [22:21] https://gist.github.com/bertjwregeer/61fede0609fff681fb786bd181a19c8b [22:21] there are my .service files. [22:26] This one is a better journalctl output: http://pastebin.ubuntu.com/23209158/ [22:26] includes all the other processes that are started interspersed. [22:29] nrezinorn: here's the systemd critical chain: http://paste.ubuntu.com/23209165/ [22:29] (ignore cleanup_defgw.service, it's a work-around for an issue with our private cloud, one I am hoping I can remove in the next month) [22:31] nrezinorn: If NoCloud datasource works like the ConfigDrive datasource, then as long as cloud-init-local runs before NetworkManager and network.service any changes to ifcg-eth0 should be picked up on the first run. [22:32] hostname should be getting changed too using the hostname service, you should see something about dbus activating the org.freedesktop.org.hostname1 unit [22:33] x58: yes in the intial boot cloud-init makes a new ifcfg-eth0 file [22:34] but the issue is it requires a reboot for networking to functon - it works fine as-is on 0.7.5 so im trying to track what specifically may be causing the issue [22:35] nrezinorn: I understand that, check the systemd critical-chain and make sure that network.service is running after cloud-init-local.service [22:35] centos uses systemd too, fwiw, sonds like maybe the names of things need to get fixed up in the systemd service files - sorry i sorta stepped away from this 'fun" lol [22:35] Look at the pastebin I posted above, it should look similar. [22:36] I have only been talking about CentOS... [22:36] nrezinorn: systemd-analyze critical-chain network.service [22:36] does that show cloud-init-local.service as coming before it? [22:36] yeah, so when the file is NOT there (ifcfg-eth0) the "network info+++" stuff says eth0 False until its rebooted [22:37] 1s let me get back to teh vm! [22:37] network info +++ is printed by cloud-init [22:37] that's after networking is completed [22:37] that's too late for anything to still be done. [22:37] yesh im just saying that when it doesnt work, it lists as false for eth0 :) [22:37] ill pastebin the stuff in a sec [22:39] For what it's worth, I am running 0.7.7 on my CentOS images. [22:40] systemd-analyze critical-chain network.service <-- does not show anything cloud-init related in the chain [22:40] That's bad. [22:40] That means network.service is allowed to start before cloud-init-local.service has done it's thing. [22:40] man who named these things all differently [22:40] lol [22:40] wtf people [22:41] lol [22:41] i thought systemd was suppoed to make everything common [22:41] but people name the freaking files differently [22:41] lol [22:41] you've seen the distro rosetta stone right harlowja ? ;P [22:41] nrezinorn: Make sure you have a Before=NetworkManager.service in cloud-init-local.service [22:41] is that before or after the crack? [22:41] network.service requires NetworkManager.service [22:41] nrezinorn: IF you modify the file, make sure to call systemctl reload-daemon or whatever that command is. [22:41] i did put that in there [22:42] Did you reload the daemon files? [22:42] i did hte netmanager fix up in my golden image and when i boot it -its still misbehaving [22:42] nrezinorn: What's the critical chain for NetworkManager say? [22:42] i used guestmount for testing stuff :) [22:42] 1s [22:42] does the critical chain for network.service show NetworkManager.service? [22:43] for networkmanager crit - it lists zero cloud-init related services lol [22:43] * nrezinorn blames ubuntu! [22:43] no no systemd [22:43] nrezinorn: grab my .service file here: https://gist.github.com/bertjwregeer/61fede0609fff681fb786bd181a19c8b [22:44] What version of CentOS? [22:44] modify both service files? theres 2 [22:44] centos 7.2 [22:44] Yes. [22:44] its basically the latest centos 7 (from 9/14 lol) because i dont wnat to wait 30 min for a new image every time [22:44] cloud-init.service should be after network.service and requires=network.service [22:44]