| BrunoADuarte | Hello | 10:43 |
|---|---|---|
| BrunoADuarte | [ 13.512947] cloud-init[477]: Cloud-init v. 23.2.1-0ubuntu0~20.04.2 running 'init-local' at Fri, 04 Aug 2023 09:56:19 +0000. Up 13.23 seconds. | 10:44 |
| BrunoADuarte | [ 13.515664] cloud-init[477]: 2023-08-04 09:56:20,232 - url_helper.py[WARNING]: Exception(s) [UrlError('Failed to parse: http://169.254.169.254/latest/api/token'), UrlError('Failed to parse: http://[fd00:ec2::254]/latest/api/token')] during request to http://[fd00:ec2::254]/latest/api/token, raising last exception | 10:44 |
| BrunoADuarte | How can I fix this error? | 10:44 |
| BrunoADuarte | It happens everytime I restart my Ubuntu AWS virtual machine. | 10:44 |
| meena | that looks like a perfectly good URL to me | 10:45 |
| BrunoADuarte | Yes, but it seems my vm has problems connecting to it at startup | 11:00 |
| BrunoADuarte | datasource: | 11:00 |
| BrunoADuarte | Ec2: | 11:00 |
| BrunoADuarte | metadata_urls: [ 'http://169.254.169.254:80', 'http://instance-data:8773' ] | 11:00 |
| BrunoADuarte | timeout: 5 # (defaults to 50 seconds) | 11:00 |
| BrunoADuarte | max_wait: 10 # (defaults to 120 seconds) | 11:00 |
| BrunoADuarte | Can I add this to the /etc/cloud/cloud.cfg file to skip this huge default timeout of 120s ? | 11:01 |
| BrunoADuarte | [ 5.461936] cloud-init[869]: 2022-04-09 03:53:54,863 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/latest/api/token' failed [0/120s]: request error [Failed to parse: http://169.254.169.254/latest/api/token] | 11:03 |
| BrunoADuarte | meena | 11:03 |
| meena | why is it failing to parse these? that doesn't make sense | 11:07 |
| BrunoADuarte | [ OK ] Reached target System Time Synchronized. | 11:15 |
| BrunoADuarte | [ OK ] Finished Load AppArmor proâ¦s managed internally by snapd. | 11:15 |
| BrunoADuarte | [ 6.557320] cloud-init[479]: Cloud-init v. 23.1.2-0ubuntu0~20.04.2 running 'init-local' at Fri, 04 Aug 2023 09:46:41 +0000. Up 6.27 seconds. | 11:15 |
| BrunoADuarte | [ 6.560078] cloud-init[479]: 2023-08-04 09:46:42,205 - url_helper.py[WARNING]: Exception(s) [UrlError('Failed to parse: http://169.254.169.254/latest/api/token'), UrlError('Failed to parse: http://[fd00:ec2::254]/latest/api/token')] during request to http://[fd00:ec2::254]/latest/api/token, raising last exception | 11:15 |
| BrunoADuarte | [ 6.565378] cloud-init[479]: 2023-08-04 09:46:42,205 - url_helper.py[WARNING]: Calling 'None' failed [0/120s]: request error [Failed to parse: http://[fd00:ec2::254]/latest/api/token] | 11:15 |
| BrunoADuarte | [ 7.709959] cloud-init[479]: 2023-08-04 09:46:43,358 - url_helper.py[WARNING]: Exception(s) [UrlError('Failed to parse: http://169.254.169.254/latest/api/token'), UrlError('Failed to parse: http://[fd00:ec2::254]/latest/api/token')] during request to http://[fd00:ec2::254]/latest/api/token, raising last exception | 11:15 |
| BrunoADuarte | The URL is ok, there's a parsing error, but there are not much details about it | 11:16 |
| BrunoADuarte | Maybe something related to IPv6 ? | 11:16 |
| BrunoADuarte | And it keeps retrying for 2 x 120 seconds | 11:17 |
| minimal | BrunoADuarte: please do not paste logs in the channel, use something like pastebin | 11:17 |
| minimal | BrunoDuarte: is this running on AWS? | 11:17 |
| BrunoADuarte | ok, sorry | 11:17 |
| BrunoADuarte | yes | 11:18 |
| minimal | why are you specifying the urls in the cloud-init config? the EC2 datasource should have the metadata urls builtin | 11:19 |
| minimal | do you have disable_ec2_metadata set in your user-data? | 11:26 |
| BrunoADuarte | minimal so ealier you asked if my VM was on AWS. Yes it is... here's the last cloud-init startup log I got from it, the error ocurred several times | 13:03 |
| BrunoADuarte | https://pastebin.com/UyN27ytg | 13:03 |
| BrunoADuarte | The first error appears on line #65 | 13:04 |
| minimal | the fetches for [fd00:ec2::254] will fail as no IPv6 is setup | 13:09 |
| minimal | I assume these logs/errors are NOT from the 1st boot after this VM has been created. Is this correct? | 13:09 |
| BrunoADuarte | correct. | 13:11 |
| BrunoADuarte | This is 3 year old AWS Ubuntu instance that I always upgrade when there are new updates to it. | 13:11 |
| minimal | did you notice this in the logs: 2023-08-04 09:48:46,043 - DataSourceEc2.py[WARNING]: IMDS's HTTP endpoint is probably disabled | 13:11 |
| BrunoADuarte | no... I never messed with the IMDS | 13:15 |
| minimal | have you tried wget/curl to it from the CLI? | 13:16 |
| BrunoADuarte | no... | 13:16 |
| minimal | then try that to see what happens? | 13:17 |
| BrunoADuarte | wget http://169.254.169.254/latest/meta-data/ | 13:22 |
| BrunoADuarte | --2023-08-04 13:19:24-- http://169.254.169.254/latest/meta-data/ | 13:22 |
| BrunoADuarte | Connecting to 169.254.169.254:80... connected. | 13:22 |
| BrunoADuarte | .... | 13:22 |
| BrunoADuarte | Saving to: ‘index.html’ | 13:22 |
| BrunoADuarte | .... | 13:22 |
| BrunoADuarte | 2023-08-04 13:19:24 (45.3 MB/s) - ‘index.html’ saved [320/320] | 13:22 |
| minimal | ok, and what's the current IP address of the VM? | 13:22 |
| BrunoADuarte | but if fails for IPv6 | 13:23 |
| BrunoADuarte | Connecting to [fd00:ec2::254]:80... failed: Network is unreachable. | 13:23 |
| BrunoADuarte | inet 172.31.72.80 netmask 255.255.240.0 broadcast 172.31.79.255 | 13:24 |
| BrunoADuarte | inet6 fe80::1486:10ff:fe06:d17b prefixlen 64 scopeid 0x20<link> | 13:24 |
| minimal | if IPv6 actually configured for the VM? I believe not | 13:24 |
| BrunoADuarte | inet6 fe80::1486:10ff:fe06:d17b prefixlen 64 scopeid 0x20<link> | 13:24 |
| minimal | ok, that's a link-local IPv6 address, not sure if metadata server is reacable via that | 13:24 |
| BrunoADuarte | on the AWS page there's no IPv6 assigned | 13:26 |
| BrunoADuarte | *aws console page | 13:26 |
| minimal | not relevant, cloud-init only uses temp IPv4 (no IPv6) config to contact metadata server, and as cloud-init was unable to contact metadata server (for network config) it therefore only setup the IPv4 fallback config - IPv4 via DHCP, no IPv6 config | 13:28 |
| minimal | s/uses/used/ | 13:28 |
| minimal | line 40 | 13:28 |
| minimal | the version of cloud-init you're using only uses dhclient/IPv4 (newer releases use dhcpc for both IPv4 & IPv6) | 13:29 |
| minimal | can you run an "ip r" to show the routing table | 13:32 |
| BrunoADuarte | default via 172.31.64.1 dev ens5 proto dhcp src 172.31.72.80 metric 100 | 13:34 |
| BrunoADuarte | 172.31.64.0/20 dev ens5 proto kernel scope link src 172.31.72.80 | 13:34 |
| BrunoADuarte | 172.31.64.1 dev ens5 proto dhcp scope link src 172.31.72.80 metric 100 | 13:34 |
| minimal | ok, that basically matches the routes cloud-init "manually" added before trying to contact metadata server, lines 54 & 55 | 13:40 |
| BrunoADuarte | correct | 13:41 |
| minimal | you have the datasource urls defined in /etc/cloud.cfg or a file in /etc/cloud.cfg.d/ ? | 13:41 |
| BrunoADuarte | no... the lines are commented | 13:41 |
| BrunoADuarte | datasource: | 13:42 |
| BrunoADuarte | Ec2: | 13:42 |
| BrunoADuarte | metadata_urls: [ 'http://169.254.169.254:80', 'http://instance-data:8773' ] | 13:42 |
| BrunoADuarte | timeout: 5 | 13:42 |
| BrunoADuarte | max_wait: 10 | 13:42 |
| BrunoADuarte | i though of adding like this, to it gives up after 10s | 13:42 |
| BrunoADuarte | *so | 13:43 |
| minimal | they don't appear commented out there | 13:43 |
| BrunoADuarte | this is what I want to add... | 13:44 |
| BrunoADuarte | the actual lines are | 13:44 |
| BrunoADuarte | # Ec2: | 13:44 |
| BrunoADuarte | # metadata_urls: [ 'blah.com' ] | 13:44 |
| BrunoADuarte | # timeout: 5 # (defaults to 50 seconds) | 13:44 |
| BrunoADuarte | # max_wait: 10 # (defaults to 120 seconds) | 13:45 |
| BrunoADuarte | *the default lines | 13:45 |
| minimal | this is a VPC instance rather than a "Classic" (non-VPC) instance, right? | 13:47 |
| BrunoADuarte | correct | 13:49 |
| minimal | the "Failed to parse" error is coming from the urllib python library I think | 14:00 |
| minimal | which cloud-init uses | 14:00 |
| BrunoADuarte | yes, from "url_helper.py" it seems | 14:06 |
| minimal | I mean it almost seems like urllib is where the issue is. The urls look ok to me from the logs | 14:06 |
| minimal | you said you upgraded, could something have gone wrong with that? | 14:07 |
| BrunoADuarte | I'm not sure, the machine works fine I don't remember of any broken updates... maybe I can force the update of this library? | 14:12 |
| BrunoADuarte | TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 'REDACTED'"` && curl -H " | 14:12 |
| BrunoADuarte | X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/ | 14:12 |
| BrunoADuarte | this command works perfectly from the cli | 14:12 |
| BrunoADuarte | is the same request that fails with the url_helper lib | 14:13 |
| minimal | yeah, it looks to me like cloud-init never actually requests the url, the "parse" error happens just before then | 14:13 |
| minimal | I'm out of ideas, perhaps someone else in channel has some? | 14:13 |
| meena | I ran out of ideas hours ago | 14:23 |
| meena | I would start an iphython shell and look what the url_helper does step by step | 14:23 |
| meena | only do this if you're desperate | 14:24 |
| BrunoADuarte | meena , can I just run "sudo dpkg-reconfigure cloud-init" and leave only "NoCloud" enabled ? | 14:38 |
| meena | BrunoADuarte: if you're in AWS, why would you have anything other than AWS enabled as source? | 15:06 |
| minimal | looking at the config schema I see various entries marked as "required: [ "abc" ]" but there is nothing on the docs website to highlight those things that are required | 16:21 |
| BrunoADuarte | meena because the AWS EC2 source isn't working. | 17:47 |
| minimal | BrunoADuarte: have you tried deploying a new VM instance to see if that works? | 17:51 |
| BrunoADuarte | It will probably work, otherwise I believe there would be a lot of issues reported, and I could not find anything related. It's a specific problem with my machine. It makes the boot take up to 5 minutes. | 17:58 |
| minimal | yeah that was what I was thinking to narrow down, that in general it works and that is is something specific to that VM instance | 17:59 |
| minimal | it would also givew you something to compare with, i.e. the contents of /etc/cloud/ with a working machine on the same OS version, same user-data etc | 18:00 |
| blackboxsw | trying to catch up on BrunoADuarte's conversation here. the first strange thing with the logs is that line 11 telling us that the DataSourceNone was detected on this machine prior to the most recent boot logs from 23.1.2. I'd be curious what was failing before this boot of 23.1.2 as it may shed light on the problem. | 20:04 |
| blackboxsw | The 2nd thing strange is the log "Calling None failed..." which tell us that at the point of failure implying that the url_reader_cb never successfully returned a valid url (due to the urlparse error).. yet urllib.urlparse is happy with that url as is. | 20:07 |
| blackboxsw | python3 -c 'from urllib.parse import quote, urlparse, urlunparse; print(urlparse("http://[fd00:ec2::254]/latest/api/token"))' | 20:07 |
| blackboxsw | ParseResult(scheme='http', netloc='[fd00:ec2::254]', path='/latest/api/token', params='', query='', fragment='') | 20:07 |
| minimal | blackboxsw: yeah it's all very strange | 20:25 |
| blackboxsw | BrunoADuarte: if you get a chance to file an issue with cloud-init and attach your full cloud-init logs (obtained via cloud-init collect-logs) plus your the output of your configured network (ip r; ip -6 r; ip addr) that'd help debug a bit async on this, and with a bit more data. I'm wondering how the upgrade path could hit this flavor issue. | 20:26 |
| minimal | blackboxsw: he filed #4316 but didn't attach logs to it | 20:28 |
| blackboxsw | But, there is obviously something else going on here as the DataSourceEc2 was not previously detected. And, it does look like the `url_helper.dual_stack` implementation may need to better handle exceptions in ipv4 vs ipv6 stacks when errors occur as it looks like it loops indefinitely. | 20:28 |
| blackboxsw | ahh thx. | 20:28 |
| minimal | IPv6 uses link-local address to reach metadata server? | 20:28 |
| blackboxsw | yep it does https://aws.amazon.com/blogs/networking-and-content-delivery/introducing-ipv6-only-subnets-and-ec2-instances/ | 20:30 |
| blackboxsw | and same with the typical openstack/ec2 ivp4 too . 169.254.169.254 | 20:30 |
| blackboxsw | theory being that the IMDS is on the private subnet locally accessible to the vm/instance being launched and won't need external routing to get to that addr | 20:31 |
| minimal | yupe. Of course some other providers in future could implement IPv6 access in a completely different way :-( | 20:32 |
| blackboxsw | LL should allow cloud-init to avoid costly ephemeral dhcp client setup in init-local boot stage if we have enough data to understand a unique link local address we can assign pre-network setup. | 20:33 |
| BrunoADuarte | Sure, I'll export the cloud-init logs and attach there. | 20:33 |
| blackboxsw | BrunoADuarte: many thanks I was commenting on that issue with that request now. | 20:33 |
| minimal | blackboxsw: hmm, the IPv6 metadata server address is a ULA address, so wondering if AWS's RA announces a default route to that via the link-local gateway fe80:ec2::1, otherwise how would VM get to it | 20:40 |
| blackboxsw | yeah, not certain offhand how that's sorted. | 20:57 |
| minimal | blackboxsw: actually I'm wrong, that AWS article doesn't make sense as fe80:ec2::1 is *not* a link-local address (as LL are fe80 followed by 56 "0" bits) and LL addresses are /64 so I'm confused how with LL only a machine can get to outside the LL /64 | 21:07 |
| blackboxsw | Sorry, I think I got my wires are crossed here but EC2 ipv6 IMDS looks like it is **not** a link local address yet the default ipv6 config on aws instances is that systemd.networkd is configured LinkLocalAddressing=ipv6... Which gives typical instances LL ipv6 addresses `$ python3 -c 'from ipaddress import IPv6Address; print(IPv6Address("fe80::14:2aff:fe09:57b9").is_link_local); print(IPv6Address("fd00:ec2::254").is_link_local)' | 21:27 |
| blackboxsw | True | 21:27 |
| blackboxsw | False | 21:27 |
| blackboxsw | ` | 21:27 |
| minimal | blackboxsw: that was my point, how to get to a ULA metadata server from a LL address without a route? (which would require an RA to announcing such a route and the distro to be configured to see RA announcements) | 21:37 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!