/srv/irclogs.ubuntu.com/2017/07/10/#cloud-init.txt

=== blaisebool is now known as Guest67678
nilujeblackboxsw: hi, any idea when smoser comes back?08:23
=== blaisebool is now known as Guest55861
Wulf19:28:14 <@smoser> blackboxsw, i'm out the rest of the week thoguh10:32
WulfThat was quite some time ago10:32
WulfJul 0310:33
Wulfniluje: My best guess is 5 hours from now.10:33
nilujeWulf: yeah he told me he was leaving for a week :p12:01
nilujeI wanted to know if he could give a look to the PR introducing the scaleway datasource12:01
blackboxswniluje: yeah should be in this week.14:52
smoserniluje, was that a ping for me ? /me is here now14:53
smoser(we have a 'cloud-init status meeting' in 67 minutes.. 4:00 GMT)14:54
niluje:D14:54
nilujehi smoser :) yeah, I was just wondering if you had the time to get a look to my mr14:55
blackboxswsmoser14:55
blackboxswwelcome back.14:56
blackboxsw..."I wanted to know if he could give a look to the PR introducing the scaleway datasource"14:56
smoserniluje, i will today.14:56
nilujesmoser: oh, thanks a lot :)14:57
smoserhi... i'm going to wait about 2 more minutes before starting a meeting here just to not have it finished before a late comer arrived.16:00
smoseri've written down a agenda like thing at https://public.etherpad-mozilla.org/p/cloud-init-meeting16:01
dpb1hi smoser, here16:01
smoserok... 4 minutes late.16:04
smoserHello everyone.16:04
smoserThis is the first 'cloud-init status meeting' (for lack of a better name)16:04
blackboxswhello hello16:05
smoserthis was announced / asked for feedback on time at https://lists.launchpad.net/cloud-init/msg00091.html16:05
powersjo/16:05
smoserand this is what we came up with.16:05
smoserI'm working off a very informal agenda at https://public.etherpad-mozilla.org/p/cloud-init-meeting16:05
smoser# Introduction16:06
smoser^ see above... very informal, let people raise issues and discuss things16:06
smoserGoals for the meeting16:06
blackboxswsounds good. we can see what themes come up and build from there16:06
smoser * Not waste (much of) people's time16:06
smoser * Function as 'office hours' for 30 minutes after... smoser and others will be present to talk on irc.16:07
smoserblackboxsw, yeah.. i am hoping that the 'office hours' portion will be used by people to help communication.16:07
blackboxsw+116:08
smoser# Recent Changes / Highlights16:08
smoserhere... i should have a list of "highlights". but I do not have htat at this point.16:08
* smoser looks for a link16:09
smoser 1. https://git.launchpad.net/cloud-init/commit/16:09
smoser   You can always look there and see recent commits to trunk.  We always try to write readable git-style commit messages with links to bugs for more information.16:10
smoser 2. https://lists.ubuntu.com/archives/ubuntu-server/2017-July/007559.html16:10
blackboxswAny recent SRU lands that we want to highlight (per 1 week ago)16:10
smoser  Josh (powersj) has been writing some of these things and sending to the ubuntu server mailing list as above.16:11
smoser  I guess it would be good to take the cloud-init portion and send to the cloud-init list also.16:11
dpb1smoser: +116:11
dpb1smoser: I'll do that with the next one16:11
smosera highlight there... powersj will be presenting at Debconf https://debconf17.debconf.org/talks/164/16:12
blackboxsw[ACTION  -smoser] Send cloud-init portion of the ubuntu-server updates to the cloud-init list.16:12
smosernext topic16:12
smoser# In Progress Development / Highlights16:12
smoser * Merge Proposals: https://code.launchpad.net/%7Ecloud-init-dev/cloud-init/+git/cloud-init/+ref/master/+activereviews16:13
smoser * Trello Board: [https://trello.com/b/hFtWKUn3/daily-cloud-init-curtin]16:13
smoser * Bugs: https://bugs.launchpad.net/cloud-init16:13
smoser * AWS ipv6 support: http://pad.lv/163903016:13
ubot5Launchpad bug 1639030 in cloud-init (Ubuntu) "Configure networking based on EC2 metadata source" [Medium,Confirmed]16:13
smoserThe last there is what blackboxsw is looking at right now..16:13
smoserwe're wanting to make cloud-init able to configure ipv6 networking on AWS.16:14
smoser(aws has ipv6 support for a while now, but cloud-init hasn't taken advantage of it)16:14
smoserAlso, it looks like powersj added16:14
smoser     Opensuse builds: https://progress.opensuse.org/issues/1971216:14
dpb1that's me16:14
smoserah.16:14
blackboxswwas going to start adding a doc here https://docs.google.com/document/d/1VWnp-29_UM_LGr1h_CTu14UmLJFu_Vu6yzZTne5x-Qw/edit?usp=sharing16:15
robjoBew aware the AWS DHCPv6 server implementation is "broken"16:15
robjos/Bew/Be/16:15
blackboxswI'll make sure the doc is publicly readable. as I think it might not be currently16:15
smosercan you elaborate?16:15
robjowhen the server receives options it does not understand in the original request for a lease it will drop the request on the floor and you'll never get an answer16:16
smoserrobjo, ? in my limited knowledge it seemed to work at least enough for 'dhclient -6' to work16:16
smoserrobjo, oh. thats good to be aware of.16:16
blackboxswahh interesting.16:16
blackboxswthanks! will test out what options we intend on sending to ensure we aren't falling on our face16:16
smoserso, yeah, we're working on getting cloud-init repos up for open suse (as dtb pointed out above) and have also recently got builds in fedora's copr16:17
robjoyes, dhclient -6 works, dhclient only send the minimal set of info to get the lease. It works because that's the client that's in amazon-linux ;)16:17
dpb1robjo: is there a know "good set" of options to send on the request?16:17
smoserthe goal for both will be to have "trunk" builds to more easily try for users.  https://copr.fedorainfracloud.org/coprs/g/cloud-init/cloud-init-dev/16:17
robjowe only figured out what we cannot sent ;)16:18
smoserrobjo, thats really helpful. thank you.16:18
dpb1robjo: thx16:18
smoserAnyone have anything else here ?16:19
smoserif not we'll move onto 'open discussion', and then 'office hours'16:19
robjoWell, thanks for setting this up and getting started with OBS, I no longer feel so alone :D16:20
smoser# open discussion16:20
blackboxswus too :)16:21
dpb1is open discussion and office hours the same thing?16:21
dpb1:)16:21
smoseryeah, i was goin gto comment to that affec.t16:21
smoserso at this point lets consider this meeting done.16:21
smoserif you have any questions, please feel free to ask.16:22
smoserand more generically, always feel free to ask.16:22
blackboxswdo you think we should invite meetingbot or some other bot to this so we record previous cloud-init meetings in the future?16:22
smoseranother link for people's usage16:23
smoser this channel is logged vi irclogs.ubuntu.com16:23
smoser https://irclogs.ubuntu.com/2017/07/10/%23cloud-init.html16:23
ajorgdiscussion on the finalpoints here? https://code.launchpad.net/~ajorgens/cloud-init/+git/cloud-init/+merge/32585716:23
smoserso you can look back if need be.  I often use that to post links in bugs.16:23
smoserblackboxsw, yeah, we could/shoudl have a meetingbot for this.16:23
blackboxswthere we go thx smoser16:24
dpb1blackboxsw: +1, want to figure out how to do that?16:24
blackboxswdpb1: I don't mind, I wanted to get another bot in here for cloud-init branch lands, new merge proposals16:24
blackboxswetc.16:24
blackboxswjust to raise the communication about what's happening in cloud-init16:25
dpb1blackboxsw: ok, meetingbot should be easy.  probably a file with channels it joins somewhere16:25
blackboxsw[ACTION - bbsw] get a bot running in channel16:25
smoserajorg, on that .. i can comment there. i suspect it did "just work" in python 2. but busted in python3.16:25
smoseri *think* writing in binary mode should be ok to change to.16:26
ajorgbecause of the utf8 stuff?16:26
smoseras the input i *think* will be crlf adjusted.....16:26
ajorgshould be, by the mime decoder, right?16:27
smoseri dont want to break something where a user fed 'crlf' end lines and that worked before due to writing in text mode16:27
smoserie, windows user put something in and python 'did the right thing' before but would break if we had written in binary mode.16:27
smoseri'm not sure.16:27
smoserbut yeah, as you said above. yaml would handle it to i think.16:28
smoserajorg, ... in the aws ipv6 path, blackboxsw had some questions maybe you could help us out with16:30
smoser https://docs.google.com/document/d/1VWnp-29_UM_LGr1h_CTu14UmLJFu_Vu6yzZTne5x-Qw/edit?usp=sharing16:30
ajorghmm. for it to be executed it has to have a text/ mime type, so maybe allowing binaries there is a bad idea anyway (can't actually be used correctly)16:30
smoseri had always assumed that the metdata service was put on a link local address (169.254.169.254) to in fact be "link local"16:30
* ajorg reads16:31
blackboxswI see folks can read this. i'm mid typing it up but you have all the context below in the doc16:31
ajorginteresting16:32
ajorgI can go back to the VPC team and see if we can get good answers for that. better would be to cut as a ticket via the canonical internal channels so David can track it.16:33
blackboxswbasically it seems AWS network timesout when trying to reference the link local 169.254.169.254 metadata service. We're thinking it's because the either that the metadata service rejects requests from an IP not allocated to the specific instance (like a static 169.254.169.200 address being dropped).  Or potentially because the metadata service response might be coming from the router/gateway address.16:33
ajorgIn Amazon Linux I think we create a static route on the first network device, so that whatever else happens the 169.254 address should work16:33
blackboxswbut that's just a hypothesis from our side. :)16:33
blackboxswthanks ajorg for the help16:34
ajorgsounds like a good hypothesis to me.16:34
ajorgI'll reach out to someone here. Please let me know if you cut a ticket about it so I can subscribe to it.16:34
blackboxswmy next pass just after this meeting will be allocating the link local 169.254.169.10 to eth0 and setting up a route that'll point at the default known gateway which was given via dhcp  and we'll see if we get a response.16:35
blackboxsw+1 ajorg will do16:35
smoserajorg, a static route to where ?16:36
* ajorg checks code to be sure16:36
=== shardy is now known as shardy_afk
ajorgcorrection...16:37
* ajorg finds the hardware token that will let him check the code...16:37
* smoser launches a amazon linux ami16:37
robjoAmazon-linux has a bunch of scripts that handle all the route setup etc. Unfortunately the scripts are not consumable by other distros as not published via GitHub or other repo. In an effort to get multiple interface support working properly across clouds we are working on networking scripts that will eventually show up on GitHub, hoping by the end of the week16:38
robjohopefully that will be useful to others16:38
ajorg169.254.169.254 0.0.0.0         255.255.255.255 UH    0      0        0 eth016:38
blackboxswooh good info robjo. Is there an existing amazon github repo somewhere that we should subscribe to?16:38
smoserhm...16:38
smoser$ ip route16:38
smoserdefault via 172.31.0.1 dev eth016:38
smoser169.254.169.254 dev eth016:38
smoser172.31.0.0/20 dev eth0  proto kernel  scope link  src 172.31.2.24016:38
ajorgJust a static route straight to that one host.16:38
ajorg(from the IP given by DHCP)16:39
ajorgyeah, like that16:39
smoseri dont *thinK* htat should fix it for us.16:39
ajorgblackboxsw: not an official one, yet, and unfortunately it's specific to Red Hat / Fedora style network scripts.16:40
robjono, I have asked repeatedly and Dave has been trying to move the needle at AWS but there appears to be no interest for AWS to maintain their scripts in GitHub thus we are doing our own thing now, I ran out of patience16:40
smosercurrently we do basically:16:40
smoser ip addr add 169.254.<rand>.<rand>/16 dev eth016:41
smoser ip link set dev eth0 up16:41
smoser(where 'rand' is random.randint(1, 168) and random.randint(0, 255))16:41
blackboxswunderstood ajorg  & robjo16:41
ajorgconfig for that route isn't in those scripts, it's in a flat file:16:42
ajorghttp://paste.ubuntu.com/25062274/16:43
ajorgsmoser: that's probably not going to work, i suspect you're right that it's checking the source IP and dropping anything not expected16:44
smoserajorg, so your route above... that is *really* wierd16:44
ajorgIIRC this is actually why we use a static route, because if it comes from eth1 instead of eth0 it will also be an unexpected address.16:44
smoserin that it says "don't send this to the gateway"16:44
smoserright?16:44
smoserwhich i'd have thought would break stuff.16:45
ajorgcorrect that it says that, less correct that it's weird16:45
smoserwell, its wierd because we have no such route16:45
ajorgit says that packets to that host should just go straight there16:45
ajorgright, that is a bit weird16:46
ajorgfair enough16:46
smoserit seems wierd to me though. because i think i have the following scenarios16:46
ajorgpackets headed to the gateway probably get captured, but if i knew for sure i'd still be bound by the "we don't talk about internal implementation details" thing16:46
smosera.) ubuntu current default behavior.... requests to MD address get sent to the default gateway16:47
ajorgby captured i mean they never actually reach the gateway16:47
smoserb.) amazon linux default behavior... requests to MD explicitly do not go to gateway16:47
smoser  but have a 172.X.X.X (or 10.0.X.X) source address, as provided by dhcp16:48
ajorgIIUC all that's needed is for the source address to be correct and possibly for the packet to be on the correct interface16:48
smoserc.) our attempt at ipv4 link local... requests to MD not through the gateway, but with a 168.254.X.X source address.16:48
ajorgso if the source address is not what's expected, it doesn't get caught by whatever is routing it to the instance metadata service16:50
ajorgif it is what's expected it does get routed16:50
ajorgwe added the static host route so that the source address will always be correct16:51
ajorgdo you know if you run into problems if you have multiple interfaces? (ipv4)16:51
smoser(in 'b', it seems somewhat wrong.. as those are "martian" packets i think... per linux rp_filter)16:52
ajorg(static route forces it onto eth0, which has the correct address)16:52
smoserajorg, yeah, multiple nics arent really addressed yet. and it wouldnt surprise me if we hit things like that.16:52
ajorgsomeone ported our ec2-net-utils scripts to ubuntu and posted them to github... lemme see if i can find that16:53
smoserajorg, well...16:53
ajorghttps://github.com/ademaria/ubuntu-ec2net16:53
smoseri think what we're agreeing on is that the source address has to be the ipv4 that is handed out by the dhcp server on "eth0".16:54
ajorgcorrect16:54
smoserand (to my knowledge) the only way to know that is in fact dhcp16:54
smoserwhich we were hoping to avoid.16:54
ajorgah16:54
ajorgreason to avoid?16:54
smoser(as with dhclient... it will have side affects)16:54
ajorgI can imagine reasons, but I might be wrong16:55
smoserthe goal is to bring up the networking required to see the metadata service, read the configuration there, render /etc/network/interfaces (or appropriate configuraiton per distro) and then let the distro bring up the interfaces.16:55
ajorgahhh... that makes sense.16:55
smoserdhclient runs scripts and such on 'up', and leaves state in /var/lib/ and other things...16:56
ajorgI totally get how that's desirable.16:56
ajorgAnd that works on other clouds?16:56
smoserwe may end up having to resort to that, but, its less than ideal.16:56
smoserwell on digital ocean, they have ipv4 link local working correctly :)16:56
smoserand it *feels* to me like a platform change coudl occur on amazon to support doing the right thing if the source address is a 169.254.X.X address16:57
blackboxswas in, no default gw needed, just the static link loval IPv4 addr and metadata responds.16:57
ajorgit's worth pushing for, i think.16:57
smoserblackboxsw, yeah... so ajorg says we dont need the gateway16:58
smoserwe need the source address.16:58
smoserwhich i said was "wierd" :)16:58
blackboxswahh right sorry crossed my wires with his above16:58
smoseri'm pretty sure that is a "martian" packet16:58
ajorgbbiab, trying to find the right team to ask about this internally16:59
smoserajorg, thank you for the discussion.16:59
blackboxswthx ajorg . so time check smoser?16:59
smoserrobjo, thanks for being here.16:59
blackboxswshall this wrap up the meeting?16:59
smoserlets say meeting ended.17:00
blackboxswgreat discussion folks.17:00
robjosmoser: Thanks again for getting this started17:00
smoseri think the difference i had between "open discussion" and "office hours" was just to have one be part of "meeting" and one just "people are around"17:00
smoser(in my head at least)17:00
smoserie... we wrap up meeting in ~ 20 minutes or something, but office hours remain... someone can know that they can come and just ask questions and have some expectation  that someone will respond.17:01
smoseranyway.17:01
smoser#end meeting17:01
smoserthanks everyone17:01
smoserthe next meeting will be on the 24th.17:01
blackboxswyes that makes sense smoser17:01
blackboxswthanks again17:01
powersjsmoser: I just submitted pylxd update merge and I did submit a fix for artful tests last week17:11
* ajorg returns17:13
ajorgso new theory on the link-local thing (without having found the right sources internally yet)...17:14
dpb1ajorg: (btw) will you be there on wednesday?17:14
ajorgyup17:14
dpb1great17:15
ajorgit may be that the problem is simply that it's too soon. it takes a little while for the network stack (on the cloud side) to be fully ready, and your packets might be dropped in the mean time17:15
ajorgso if you want to try again w/ link-local but add a big sleep before it, you might find that that works17:15
ajorgobviously that wouldn't be desirable, but it would help explain things17:15
blackboxswhmm ajorg how big a sleep, my wget was retrying for 10 mins17:37
ajorggah, well that's plenty big17:38
ajorgwas this with other networking unconfigured?17:38
blackboxswheh17:38
blackboxswthis was a single eth0 static config @ 169.254.71.10 or something close to that IP.17:39
ajorgk17:40
blackboxswaround line 150 http://paste.ubuntu.com/25036317/ any lines prefaced with Chad were my additional network setup ( I tuned the retries down to 2 on the wget in this run though.17:40
blackboxswas the 20 retries and 15 min default timeout were too larege17:42
blackboxsw*too large17:42
ajorgblackboxsw: I've added more details on the ticket here and CC'd David Duncan on it. We'll try to get you some answers. Meanwhile you might be stuck with having to rely on DHCP.18:07
smoserblackboxsw, http://paste.ubuntu.com/25062706/18:09
smoserresults of:18:10
smoser addr:18:10
smoser addr: http://paste.ubuntu.com/25062713/18:10
smoser link-local: http://paste.ubuntu.com/25062728/18:11
smoserin my mind that basically proves ajorg right. you have to have the "correct" source address.18:11
ajorgI like being right, except when it makes someone sad.18:12
smoserwell, in blackboxsw and my assesment before talking to you, we thought we'd need to send through the gateway, not have the right source address.18:13
smoserso you helped diagnose quicker.18:13
ajorgglad to be of service then18:13
smoserajorg, so. https://code.launchpad.net/~ajorgens/cloud-init/+git/cloud-init/+merge/32585718:16
smosercurrent behavior in cloud-init is t fail if no '#!' at the front.18:16
smoserchanging to your suggestion "defaults" to using /bin/sh if there is no shebang (per the shells' implementation of "read the first 2 bytes")18:17
ajorgright18:17
smoseri really think it better to make convince / teach people to use '#!'18:18
ajorgokay18:18
ajorgi'm actually totally okay with dropping this one18:18
smoser(and i'd like to support binaries... write_files definitely should support binaries, and almost certainly did in python2 ... thats a bug we need to fix)18:18
ajorg(or carrying a local patch if our customers don't want to learn18:18
smoserjust failing with no idea why isn't great...18:19
smoserso we could try without shell=True18:19
smoserand on exception look at the file, and if it did not start with '#!' warn18:19
smoseror rather just adjust the warning message to say 'did you mean to start your data with "#!/bin/sh"?'18:20
smoseri realize that cloud-init warnings are probably not read by most people, but at least that gives a hint18:20
ajorgI'm more inclined to giving the warning. It requires peeking, but that's not exactly terrible.18:21
smoserand only shows a small bit of noise when in the case that the user gave a binary that just exited non-zero18:21
ajorgright18:21
smoserthe additional noise just in the message above, as if you sent '/bin/false' it would still WARN that it exited non-zero18:21
ajorgor isn't there a specific exit code for "I couldn't even try to execute that"?18:22
ajorg126?18:24
smoserhttp://paste.ubuntu.com/25062793/18:25
smoseryou were right.. it should raise a OSError errno 818:25
ajorgcool18:25
smoseryou want to re-work this to handle that ?18:26
ajorgI'll have to write a note on the PR for now, have to do my "real" job for a while today.18:26
ajorgbut yes18:26
smoseryeah. thanks. i'll comment in the pr18:26
ajorgah, great. thank you!18:27
blackboxswsmoser: nice work on the validation there w/ the route added18:41
blackboxswok. so you validated w/ the small script that link-local fails. ok18:44
blackboxswI had looked at the wrong paste for a second and thought you had it succeeding18:45
smoserright. fails with ipv4ll address18:47
smoserpasses with the "right" source address.18:47
blackboxswyeah so smoser, where shall we go from here for today?  You going to continue working on DataSourceAWS ipv6 support using dhclient while we determine if aws can suppport link-local source addresses?18:53
blackboxsw*while we wait to see if AWS can add support for link-local source addresses18:55
smoserblackboxsw, you can take a look at the dhclient path.18:56
smoserblackboxsw, but if the thing actually works..18:57
smoser http://code.activestate.com/recipes/577649-dhcp-query/18:57
smoserdoes seem consumable18:58
=== shardy_afk is now known as shardy
blackboxswright smoser and the primary reason we'd use the python client instead of dhclient in ephemeral is to avoid the side-effects right19:03
smosercorrect19:03
smoserand to get cross-distro19:03
smoserbasically it would be (it seems) more distro agnostic19:04
blackboxswmakes sense to me19:04
blackboxswok figured out how to do a simple dhcp discover using scapy. not sure if that dependency is too large for cloud-init though22:03
dpb1smoser ^22:05
dpb1oh22:05
dpb1he gone22:05
blackboxswI'll see what it pulls in. but it's a fairly simple python security library that only strictly depends on python.  lots of suggests for the package. but anyway. I'll have about a 20 liner that should give is a tiny python dhclient which can perform our dhcp discovery and without any side-effects22:10
blackboxswjust tested locally and had no problem identifying my dhcp server as well as using a obtaining a new dhcp ip. problem I22:12
dpb1blackboxsw: sounds nice22:13
blackboxswsee with this and with the link smoser sent was that it requires an IP configured on the interface to start with in order to use the socket to broadcast the dhcp discover packaets. feels like we are putting the cart before the horse as we don't truly know what I'22:13
blackboxswsee with this and with the link smoser sent was that it requires an IP configured on the interface to start with in order to use the socket to broadcast the dhcp discover packaets. feels like we are putting the cart before the horse as we don't truly know what IP to give the instance on a given aws private net22:13
dpb1better to not maintain our own code too22:13
blackboxswright22:13
blackboxswmake it pretty much a 5-liner for just a simple dhcp client request http://paste.ubuntu.com/25064349/22:38
blackboxswbut I'm still missing how we bring up the initial  interface so scapy can talk on the socket as the udp broadcast message isn't permitted w/out some viable(up) interface. will have to think on this over dinner22:39
PuttiIs anyone here from canonical that could check where my contributor agreement has lost? Launchpad link to my account: https://launchpad.net/~j.kylmala23:11
larsksPutti: I suspect most of the canonical folks are going to be here us business hours (central/mountain/pacific time, I think).23:56

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!