=== blaisebool is now known as Guest67678 [08:23] blackboxsw: hi, any idea when smoser comes back? === blaisebool is now known as Guest55861 [10:32] 19:28:14 <@smoser> blackboxsw, i'm out the rest of the week thoguh [10:32] That was quite some time ago [10:33] Jul 03 [10:33] niluje: My best guess is 5 hours from now. [12:01] Wulf: yeah he told me he was leaving for a week :p [12:01] I wanted to know if he could give a look to the PR introducing the scaleway datasource [14:52] niluje: yeah should be in this week. [14:53] niluje, was that a ping for me ? /me is here now [14:54] (we have a 'cloud-init status meeting' in 67 minutes.. 4:00 GMT) [14:54] :D [14:55] hi smoser :) yeah, I was just wondering if you had the time to get a look to my mr [14:55] smoser [14:56] welcome back. [14:56] ..."I wanted to know if he could give a look to the PR introducing the scaleway datasource" [14:56] niluje, i will today. [14:57] smoser: oh, thanks a lot :) [16:00] hi... i'm going to wait about 2 more minutes before starting a meeting here just to not have it finished before a late comer arrived. [16:01] i've written down a agenda like thing at https://public.etherpad-mozilla.org/p/cloud-init-meeting [16:01] hi smoser, here [16:04] ok... 4 minutes late. [16:04] Hello everyone. [16:04] This is the first 'cloud-init status meeting' (for lack of a better name) [16:05] hello hello [16:05] this was announced / asked for feedback on time at https://lists.launchpad.net/cloud-init/msg00091.html [16:05] o/ [16:05] and this is what we came up with. [16:05] I'm working off a very informal agenda at https://public.etherpad-mozilla.org/p/cloud-init-meeting [16:06] # Introduction [16:06] ^ see above... very informal, let people raise issues and discuss things [16:06] Goals for the meeting [16:06] sounds good. we can see what themes come up and build from there [16:06] * Not waste (much of) people's time [16:07] * Function as 'office hours' for 30 minutes after... smoser and others will be present to talk on irc. [16:07] blackboxsw, yeah.. i am hoping that the 'office hours' portion will be used by people to help communication. [16:08] +1 [16:08] # Recent Changes / Highlights [16:08] here... i should have a list of "highlights". but I do not have htat at this point. [16:09] * smoser looks for a link [16:09] 1. https://git.launchpad.net/cloud-init/commit/ [16:10] You can always look there and see recent commits to trunk. We always try to write readable git-style commit messages with links to bugs for more information. [16:10] 2. https://lists.ubuntu.com/archives/ubuntu-server/2017-July/007559.html [16:10] Any recent SRU lands that we want to highlight (per 1 week ago) [16:11] Josh (powersj) has been writing some of these things and sending to the ubuntu server mailing list as above. [16:11] I guess it would be good to take the cloud-init portion and send to the cloud-init list also. [16:11] smoser: +1 [16:11] smoser: I'll do that with the next one [16:12] a highlight there... powersj will be presenting at Debconf https://debconf17.debconf.org/talks/164/ [16:12] [ACTION -smoser] Send cloud-init portion of the ubuntu-server updates to the cloud-init list. [16:12] next topic [16:12] # In Progress Development / Highlights [16:13] * Merge Proposals: https://code.launchpad.net/%7Ecloud-init-dev/cloud-init/+git/cloud-init/+ref/master/+activereviews [16:13] * Trello Board: [https://trello.com/b/hFtWKUn3/daily-cloud-init-curtin] [16:13] * Bugs: https://bugs.launchpad.net/cloud-init [16:13] * AWS ipv6 support: http://pad.lv/1639030 [16:13] Launchpad bug 1639030 in cloud-init (Ubuntu) "Configure networking based on EC2 metadata source" [Medium,Confirmed] [16:13] The last there is what blackboxsw is looking at right now.. [16:14] we're wanting to make cloud-init able to configure ipv6 networking on AWS. [16:14] (aws has ipv6 support for a while now, but cloud-init hasn't taken advantage of it) [16:14] Also, it looks like powersj added [16:14] Opensuse builds: https://progress.opensuse.org/issues/19712 [16:14] that's me [16:14] ah. [16:15] was going to start adding a doc here https://docs.google.com/document/d/1VWnp-29_UM_LGr1h_CTu14UmLJFu_Vu6yzZTne5x-Qw/edit?usp=sharing [16:15] Bew aware the AWS DHCPv6 server implementation is "broken" [16:15] s/Bew/Be/ [16:15] I'll make sure the doc is publicly readable. as I think it might not be currently [16:15] can you elaborate? [16:16] when the server receives options it does not understand in the original request for a lease it will drop the request on the floor and you'll never get an answer [16:16] robjo, ? in my limited knowledge it seemed to work at least enough for 'dhclient -6' to work [16:16] robjo, oh. thats good to be aware of. [16:16] ahh interesting. [16:16] thanks! will test out what options we intend on sending to ensure we aren't falling on our face [16:17] so, yeah, we're working on getting cloud-init repos up for open suse (as dtb pointed out above) and have also recently got builds in fedora's copr [16:17] yes, dhclient -6 works, dhclient only send the minimal set of info to get the lease. It works because that's the client that's in amazon-linux ;) [16:17] robjo: is there a know "good set" of options to send on the request? [16:17] the goal for both will be to have "trunk" builds to more easily try for users. https://copr.fedorainfracloud.org/coprs/g/cloud-init/cloud-init-dev/ [16:18] we only figured out what we cannot sent ;) [16:18] robjo, thats really helpful. thank you. [16:18] robjo: thx [16:19] Anyone have anything else here ? [16:19] if not we'll move onto 'open discussion', and then 'office hours' [16:20] Well, thanks for setting this up and getting started with OBS, I no longer feel so alone :D [16:20] # open discussion [16:21] us too :) [16:21] is open discussion and office hours the same thing? [16:21] :) [16:21] yeah, i was goin gto comment to that affec.t [16:21] so at this point lets consider this meeting done. [16:22] if you have any questions, please feel free to ask. [16:22] and more generically, always feel free to ask. [16:22] do you think we should invite meetingbot or some other bot to this so we record previous cloud-init meetings in the future? [16:23] another link for people's usage [16:23] this channel is logged vi irclogs.ubuntu.com [16:23] https://irclogs.ubuntu.com/2017/07/10/%23cloud-init.html [16:23] discussion on the finalpoints here? https://code.launchpad.net/~ajorgens/cloud-init/+git/cloud-init/+merge/325857 [16:23] so you can look back if need be. I often use that to post links in bugs. [16:23] blackboxsw, yeah, we could/shoudl have a meetingbot for this. [16:24] there we go thx smoser [16:24] blackboxsw: +1, want to figure out how to do that? [16:24] dpb1: I don't mind, I wanted to get another bot in here for cloud-init branch lands, new merge proposals [16:24] etc. [16:25] just to raise the communication about what's happening in cloud-init [16:25] blackboxsw: ok, meetingbot should be easy. probably a file with channels it joins somewhere [16:25] [ACTION - bbsw] get a bot running in channel [16:25] ajorg, on that .. i can comment there. i suspect it did "just work" in python 2. but busted in python3. [16:26] i *think* writing in binary mode should be ok to change to. [16:26] because of the utf8 stuff? [16:26] as the input i *think* will be crlf adjusted..... [16:27] should be, by the mime decoder, right? [16:27] i dont want to break something where a user fed 'crlf' end lines and that worked before due to writing in text mode [16:27] ie, windows user put something in and python 'did the right thing' before but would break if we had written in binary mode. [16:27] i'm not sure. [16:28] but yeah, as you said above. yaml would handle it to i think. [16:30] ajorg, ... in the aws ipv6 path, blackboxsw had some questions maybe you could help us out with [16:30] https://docs.google.com/document/d/1VWnp-29_UM_LGr1h_CTu14UmLJFu_Vu6yzZTne5x-Qw/edit?usp=sharing [16:30] hmm. for it to be executed it has to have a text/ mime type, so maybe allowing binaries there is a bad idea anyway (can't actually be used correctly) [16:30] i had always assumed that the metdata service was put on a link local address (169.254.169.254) to in fact be "link local" [16:31] * ajorg reads [16:31] I see folks can read this. i'm mid typing it up but you have all the context below in the doc [16:32] interesting [16:33] I can go back to the VPC team and see if we can get good answers for that. better would be to cut as a ticket via the canonical internal channels so David can track it. [16:33] basically it seems AWS network timesout when trying to reference the link local 169.254.169.254 metadata service. We're thinking it's because the either that the metadata service rejects requests from an IP not allocated to the specific instance (like a static 169.254.169.200 address being dropped). Or potentially because the metadata service response might be coming from the router/gateway address. [16:33] In Amazon Linux I think we create a static route on the first network device, so that whatever else happens the 169.254 address should work [16:33] but that's just a hypothesis from our side. :) [16:34] thanks ajorg for the help [16:34] sounds like a good hypothesis to me. [16:34] I'll reach out to someone here. Please let me know if you cut a ticket about it so I can subscribe to it. [16:35] my next pass just after this meeting will be allocating the link local 169.254.169.10 to eth0 and setting up a route that'll point at the default known gateway which was given via dhcp and we'll see if we get a response. [16:35] +1 ajorg will do [16:36] ajorg, a static route to where ? [16:36] * ajorg checks code to be sure === shardy is now known as shardy_afk [16:37] correction... [16:37] * ajorg finds the hardware token that will let him check the code... [16:37] * smoser launches a amazon linux ami [16:38] Amazon-linux has a bunch of scripts that handle all the route setup etc. Unfortunately the scripts are not consumable by other distros as not published via GitHub or other repo. In an effort to get multiple interface support working properly across clouds we are working on networking scripts that will eventually show up on GitHub, hoping by the end of the week [16:38] hopefully that will be useful to others [16:38] 169.254.169.254 0.0.0.0 255.255.255.255 UH 0 0 0 eth0 [16:38] ooh good info robjo. Is there an existing amazon github repo somewhere that we should subscribe to? [16:38] hm... [16:38] $ ip route [16:38] default via 172.31.0.1 dev eth0 [16:38] 169.254.169.254 dev eth0 [16:38] 172.31.0.0/20 dev eth0 proto kernel scope link src 172.31.2.240 [16:38] Just a static route straight to that one host. [16:39] (from the IP given by DHCP) [16:39] yeah, like that [16:39] i dont *thinK* htat should fix it for us. [16:40] blackboxsw: not an official one, yet, and unfortunately it's specific to Red Hat / Fedora style network scripts. [16:40] no, I have asked repeatedly and Dave has been trying to move the needle at AWS but there appears to be no interest for AWS to maintain their scripts in GitHub thus we are doing our own thing now, I ran out of patience [16:40] currently we do basically: [16:41] ip addr add 169.254../16 dev eth0 [16:41] ip link set dev eth0 up [16:41] (where 'rand' is random.randint(1, 168) and random.randint(0, 255)) [16:41] understood ajorg & robjo [16:42] config for that route isn't in those scripts, it's in a flat file: [16:43] http://paste.ubuntu.com/25062274/ [16:44] smoser: that's probably not going to work, i suspect you're right that it's checking the source IP and dropping anything not expected [16:44] ajorg, so your route above... that is *really* wierd [16:44] IIRC this is actually why we use a static route, because if it comes from eth1 instead of eth0 it will also be an unexpected address. [16:44] in that it says "don't send this to the gateway" [16:44] right? [16:45] which i'd have thought would break stuff. [16:45] correct that it says that, less correct that it's weird [16:45] well, its wierd because we have no such route [16:45] it says that packets to that host should just go straight there [16:46] right, that is a bit weird [16:46] fair enough [16:46] it seems wierd to me though. because i think i have the following scenarios [16:46] packets headed to the gateway probably get captured, but if i knew for sure i'd still be bound by the "we don't talk about internal implementation details" thing [16:47] a.) ubuntu current default behavior.... requests to MD address get sent to the default gateway [16:47] by captured i mean they never actually reach the gateway [16:47] b.) amazon linux default behavior... requests to MD explicitly do not go to gateway [16:48] but have a 172.X.X.X (or 10.0.X.X) source address, as provided by dhcp [16:48] IIUC all that's needed is for the source address to be correct and possibly for the packet to be on the correct interface [16:48] c.) our attempt at ipv4 link local... requests to MD not through the gateway, but with a 168.254.X.X source address. [16:50] so if the source address is not what's expected, it doesn't get caught by whatever is routing it to the instance metadata service [16:50] if it is what's expected it does get routed [16:51] we added the static host route so that the source address will always be correct [16:51] do you know if you run into problems if you have multiple interfaces? (ipv4) [16:52] (in 'b', it seems somewhat wrong.. as those are "martian" packets i think... per linux rp_filter) [16:52] (static route forces it onto eth0, which has the correct address) [16:52] ajorg, yeah, multiple nics arent really addressed yet. and it wouldnt surprise me if we hit things like that. [16:53] someone ported our ec2-net-utils scripts to ubuntu and posted them to github... lemme see if i can find that [16:53] ajorg, well... [16:53] https://github.com/ademaria/ubuntu-ec2net [16:54] i think what we're agreeing on is that the source address has to be the ipv4 that is handed out by the dhcp server on "eth0". [16:54] correct [16:54] and (to my knowledge) the only way to know that is in fact dhcp [16:54] which we were hoping to avoid. [16:54] ah [16:54] reason to avoid? [16:54] (as with dhclient... it will have side affects) [16:55] I can imagine reasons, but I might be wrong [16:55] the goal is to bring up the networking required to see the metadata service, read the configuration there, render /etc/network/interfaces (or appropriate configuraiton per distro) and then let the distro bring up the interfaces. [16:55] ahhh... that makes sense. [16:56] dhclient runs scripts and such on 'up', and leaves state in /var/lib/ and other things... [16:56] I totally get how that's desirable. [16:56] And that works on other clouds? [16:56] we may end up having to resort to that, but, its less than ideal. [16:56] well on digital ocean, they have ipv4 link local working correctly :) [16:57] and it *feels* to me like a platform change coudl occur on amazon to support doing the right thing if the source address is a 169.254.X.X address [16:57] as in, no default gw needed, just the static link loval IPv4 addr and metadata responds. [16:57] it's worth pushing for, i think. [16:58] blackboxsw, yeah... so ajorg says we dont need the gateway [16:58] we need the source address. [16:58] which i said was "wierd" :) [16:58] ahh right sorry crossed my wires with his above [16:58] i'm pretty sure that is a "martian" packet [16:59] bbiab, trying to find the right team to ask about this internally [16:59] ajorg, thank you for the discussion. [16:59] thx ajorg . so time check smoser? [16:59] robjo, thanks for being here. [16:59] shall this wrap up the meeting? [17:00] lets say meeting ended. [17:00] great discussion folks. [17:00] smoser: Thanks again for getting this started [17:00] i think the difference i had between "open discussion" and "office hours" was just to have one be part of "meeting" and one just "people are around" [17:00] (in my head at least) [17:01] ie... we wrap up meeting in ~ 20 minutes or something, but office hours remain... someone can know that they can come and just ask questions and have some expectation that someone will respond. [17:01] anyway. [17:01] #end meeting [17:01] thanks everyone [17:01] the next meeting will be on the 24th. [17:01] yes that makes sense smoser [17:01] thanks again [17:11] smoser: I just submitted pylxd update merge and I did submit a fix for artful tests last week [17:13] * ajorg returns [17:14] so new theory on the link-local thing (without having found the right sources internally yet)... [17:14] ajorg: (btw) will you be there on wednesday? [17:14] yup [17:15] great [17:15] it may be that the problem is simply that it's too soon. it takes a little while for the network stack (on the cloud side) to be fully ready, and your packets might be dropped in the mean time [17:15] so if you want to try again w/ link-local but add a big sleep before it, you might find that that works [17:15] obviously that wouldn't be desirable, but it would help explain things [17:37] hmm ajorg how big a sleep, my wget was retrying for 10 mins [17:38] gah, well that's plenty big [17:38] was this with other networking unconfigured? [17:38] heh [17:39] this was a single eth0 static config @ 169.254.71.10 or something close to that IP. [17:40] k [17:40] around line 150 http://paste.ubuntu.com/25036317/ any lines prefaced with Chad were my additional network setup ( I tuned the retries down to 2 on the wget in this run though. [17:42] as the 20 retries and 15 min default timeout were too larege [17:42] *too large [18:07] blackboxsw: I've added more details on the ticket here and CC'd David Duncan on it. We'll try to get you some answers. Meanwhile you might be stuck with having to rely on DHCP. [18:09] blackboxsw, http://paste.ubuntu.com/25062706/ [18:10] results of: [18:10] addr: [18:10] addr: http://paste.ubuntu.com/25062713/ [18:11] link-local: http://paste.ubuntu.com/25062728/ [18:11] in my mind that basically proves ajorg right. you have to have the "correct" source address. [18:12] I like being right, except when it makes someone sad. [18:13] well, in blackboxsw and my assesment before talking to you, we thought we'd need to send through the gateway, not have the right source address. [18:13] so you helped diagnose quicker. [18:13] glad to be of service then [18:16] ajorg, so. https://code.launchpad.net/~ajorgens/cloud-init/+git/cloud-init/+merge/325857 [18:16] current behavior in cloud-init is t fail if no '#!' at the front. [18:17] changing to your suggestion "defaults" to using /bin/sh if there is no shebang (per the shells' implementation of "read the first 2 bytes") [18:17] right [18:18] i really think it better to make convince / teach people to use '#!' [18:18] okay [18:18] i'm actually totally okay with dropping this one [18:18] (and i'd like to support binaries... write_files definitely should support binaries, and almost certainly did in python2 ... thats a bug we need to fix) [18:18] (or carrying a local patch if our customers don't want to learn [18:19] just failing with no idea why isn't great... [18:19] so we could try without shell=True [18:19] and on exception look at the file, and if it did not start with '#!' warn [18:20] or rather just adjust the warning message to say 'did you mean to start your data with "#!/bin/sh"?' [18:20] i realize that cloud-init warnings are probably not read by most people, but at least that gives a hint [18:21] I'm more inclined to giving the warning. It requires peeking, but that's not exactly terrible. [18:21] and only shows a small bit of noise when in the case that the user gave a binary that just exited non-zero [18:21] right [18:21] the additional noise just in the message above, as if you sent '/bin/false' it would still WARN that it exited non-zero [18:22] or isn't there a specific exit code for "I couldn't even try to execute that"? [18:24] 126? [18:25] http://paste.ubuntu.com/25062793/ [18:25] you were right.. it should raise a OSError errno 8 [18:25] cool [18:26] you want to re-work this to handle that ? [18:26] I'll have to write a note on the PR for now, have to do my "real" job for a while today. [18:26] but yes [18:26] yeah. thanks. i'll comment in the pr [18:27] ah, great. thank you! [18:41] smoser: nice work on the validation there w/ the route added [18:44] ok. so you validated w/ the small script that link-local fails. ok [18:45] I had looked at the wrong paste for a second and thought you had it succeeding [18:47] right. fails with ipv4ll address [18:47] passes with the "right" source address. [18:53] yeah so smoser, where shall we go from here for today? You going to continue working on DataSourceAWS ipv6 support using dhclient while we determine if aws can suppport link-local source addresses? [18:55] *while we wait to see if AWS can add support for link-local source addresses [18:56] blackboxsw, you can take a look at the dhclient path. [18:57] blackboxsw, but if the thing actually works.. [18:57] http://code.activestate.com/recipes/577649-dhcp-query/ [18:58] does seem consumable === shardy_afk is now known as shardy [19:03] right smoser and the primary reason we'd use the python client instead of dhclient in ephemeral is to avoid the side-effects right [19:03] correct [19:03] and to get cross-distro [19:04] basically it would be (it seems) more distro agnostic [19:04] makes sense to me [22:03] ok figured out how to do a simple dhcp discover using scapy. not sure if that dependency is too large for cloud-init though [22:05] smoser ^ [22:05] oh [22:05] he gone [22:10] I'll see what it pulls in. but it's a fairly simple python security library that only strictly depends on python. lots of suggests for the package. but anyway. I'll have about a 20 liner that should give is a tiny python dhclient which can perform our dhcp discovery and without any side-effects [22:12] just tested locally and had no problem identifying my dhcp server as well as using a obtaining a new dhcp ip. problem I [22:13] blackboxsw: sounds nice [22:13] see with this and with the link smoser sent was that it requires an IP configured on the interface to start with in order to use the socket to broadcast the dhcp discover packaets. feels like we are putting the cart before the horse as we don't truly know what I' [22:13] see with this and with the link smoser sent was that it requires an IP configured on the interface to start with in order to use the socket to broadcast the dhcp discover packaets. feels like we are putting the cart before the horse as we don't truly know what IP to give the instance on a given aws private net [22:13] better to not maintain our own code too [22:13] right [22:38] make it pretty much a 5-liner for just a simple dhcp client request http://paste.ubuntu.com/25064349/ [22:39] but I'm still missing how we bring up the initial interface so scapy can talk on the socket as the udp broadcast message isn't permitted w/out some viable(up) interface. will have to think on this over dinner [23:11] Is anyone here from canonical that could check where my contributor agreement has lost? Launchpad link to my account: https://launchpad.net/~j.kylmala [23:56] Putti: I suspect most of the canonical folks are going to be here us business hours (central/mountain/pacific time, I think).