| cpaelzer | bjonnh: If you don't insist on custom install options and instead would be ok to customize an instance at deploy time then I'd recommend taking a look at either multipass (has a libvirt option that is not the deault) or uvtool | 05:19 |
|---|---|---|
| cpaelzer | so much less complexity and much faster than virt-install | 05:20 |
| lordievader | Good morning | 07:27 |
| === mIk3_09 is now known as mIk3_08 | ||
| j_c | Hi anyone using , Ubuntu server 18 with more than 1 NIC cards? I am not able to bring network up on the second NIC. Same issue on Ubuntu server 16.04 as well | 09:21 |
| cpaelzer | j_c: I have multiple nics, but they work for me | 09:30 |
| cpaelzer | and with a VM guest with mutliple cards everyone can easily try, what happens on your system when you say "not able to bring network up on the second NIC" ? | 09:31 |
| cpaelzer | by default the further devices are offline in my case | 09:33 |
| cpaelzer | but you can follow e.g. https://netplan.io/examples#connecting-multiple-interfaces-with-dhcp (whatever matches your config needs) | 09:33 |
| cpaelzer | with that I get all three devices up just fine, so please outline what your issue exactly is | 09:34 |
| cpaelzer | maybe start with an output of "ip link" "ip addr" and your netplan config in /etc/netplan/ | 09:35 |
| j_c | @cpaelzer: Ip link and ip addr everything works fine. I am not able to connect to the server using the second NIC | 09:42 |
| cpaelzer | j_c: was so kind to keep the data to a query - thanks | 09:46 |
| cpaelzer | j_c: so you have setup two distinct networks | 09:48 |
| cpaelzer | 10.1.14.x which you can reach via the first NIC | 09:48 |
| cpaelzer | and 10.2.14.x which is the second NIC | 09:48 |
| j_c | Yes, two different networks. I am able to connect using 10.1.14.x, but not able to connect to 10.2.14.x | 09:48 |
| cpaelzer | do I understand correctly that on the second NIC there is a network 10.2.14.x and you can't reach any of those 10.2.14.x systems? | 09:49 |
| j_c | Yes, you are correct. I am not able to connect to the server using 10.2.14.x | 09:49 |
| cpaelzer | can you send me (in the query) the output of "ip route" | 09:49 |
| j_c | sent | 09:50 |
| cpaelzer | ok, LGTM 10.2.14.x should go out of eno2 in your case | 09:51 |
| cpaelzer | what command exactly does "try to connect" mean? | 09:51 |
| lordievader | j_c: What do you see on a tcpdump on that interface? For example with a ping, do you see the host responding? | 09:52 |
| j_c | I tried to ping , ssh | 09:52 |
| cpaelzer | yep, good next questions lordievader | 09:52 |
| cpaelzer | and to be sure on the routing | 09:52 |
| cpaelzer | what does "ip route get <targetip>" give? | 09:53 |
| cpaelzer | should be ... via eno2 | 09:53 |
| j_c | i don't see that, sent you the output | 09:53 |
| cpaelzer | ok so the routing is broken | 09:55 |
| cpaelzer | here a valid example: | 09:55 |
| j_c | if add the routing manually, after the restart the routing information is lost. | 09:56 |
| cpaelzer | 10.245.237.5 via 10.172.192.1 dev ens8 src 10.172.196.173 uid 1000 | 09:56 |
| cpaelzer | yeah, you want to find why it is broken | 09:56 |
| cpaelzer | not manually add routes | 09:56 |
| lordievader | Systemd-networkd, netplan, networkmanager adding routes? | 09:56 |
| j_c | cpaelzer: Okey. I will check that. | 09:59 |
| j_c | @<lordievader> : i sent tcpdump to you | 10:00 |
| lordievader | You did? I'm on matrix, maybe the PM invite hasn't arrived yet. | 10:01 |
| j_c | root@h019:~# tcpdump -i eno2 | 10:02 |
| j_c | tcpdump: verbose output suppressed, use -v or -vv for full protocol decode | 10:02 |
| j_c | listening on eno2, link-type EN10MB (Ethernet), capture size 262144 bytes | 10:02 |
| j_c | 10:00:21.615095 ARP, Request who-has 10.2.14.116 tell 10.2.14.111, length 46 | 10:02 |
| j_c | 10:00:22.072618 ARP, Request who-has 10.2.14.116 tell 10.2.14.115, length 46 | 10:02 |
| j_c | 10:00:22.134786 ARP, Request who-has 10.2.14.116 tell 10.2.14.113, length 46 | 10:02 |
| j_c | 10:00:22.351513 ARP, Request who-has 10.2.14.116 tell 10.2.14.117, length 46 | 10:02 |
| j_c | 10:00:22.615112 ARP, Request who-has 10.2.14.116 tell 10.2.14.111, length 46 | 10:02 |
| j_c | 10:00:23.072670 ARP, Request who-has 10.2.14.116 tell 10.2.14.115, length 46 | 10:02 |
| j_c | 10:00:23.134753 ARP, Request who-has 10.2.14.116 tell 10.2.14.113, length 46 | 10:02 |
| j_c | 10:00:23.351472 ARP, Request who-has 10.2.14.116 tell 10.2.14.117, length 46 | 10:02 |
| j_c | 10:00:23.615154 ARP, Request who-has 10.2.14.116 tell 10.2.14.111, length 46 | 10:02 |
| j_c | 10:00:24.073711 ARP, Request who-has 10.2.14.116 tell 10.2.14.115, length 46 | 10:02 |
| j_c | 10:00:24.178287 ARP, Request who-has 10.2.14.116 tell 10.2.14.113, length 46 | 10:02 |
| j_c | 10:00:24.354421 ARP, Request who-has 10.2.14.116 tell 10.2.14.117, length 46 | 10:02 |
| j_c | 10:00:24.371219 ARP, Request who-has 10.2.14.116 tell 10.2.14.112, length 46 | 10:02 |
| j_c | 10:00:24.615194 ARP, Request who-has 10.2.14.116 tell 10.2.14.111, length 46 | 10:02 |
| j_c | 10:00:25.076653 ARP, Request who-has 10.2.14.116 tell 10.2.14.115, length 46 | 10:02 |
| j_c | 10:00:25.174792 ARP, Request who-has 10.2.14.116 tell 10.2.14.113, length 46 | 10:02 |
| j_c | 10:00:25.351543 ARP, Request who-has 10.2.14.116 tell 10.2.14.117, length 46 | 10:02 |
| lordievader | !paste | 10:02 |
| ubottu | For posting multi-line texts into the channel, please use https://paste.ubuntu.com | To post !screenshots use https://imgur.com/ !pastebinit to paste directly from command line | Make sure you give us the URL for your paste - see also the channel topic. | 10:02 |
| j_c | 10:00:25.367581 ARP, Request who-has 10.2.14.116 tell 10.2.14.112, length 46 | 10:02 |
| j_c | 10:00:25.614942 ARP, Request who-has 10.2.14.116 tell 10.2.14.111, length 46 | 10:02 |
| j_c | 10:00:26.076689 ARP, Request who-has 10.2.14.116 tell 10.2.14.115, length 46 | 10:02 |
| j_c | 10:00:26.174812 ARP, Request who-has 10.2.14.116 tell 10.2.14.113, length 46 | 10:03 |
| j_c | 10:00:26.351530 ARP, Request who-has 10.2.14.116 tell 10.2.14.117, length 46 | 10:03 |
| lordievader | Hrm, no bot? | 10:03 |
| j_c | 10:00:26.367552 ARP, Request who-has 10.2.14.116 tell 10.2.14.112, length 46 | 10:03 |
| j_c | 10:00:26.614897 ARP, Request who-has 10.2.14.116 tell 10.2.14.111, length 46 | 10:03 |
| j_c | 10:00:27.096251 ARP, Request who-has 10.2.14.116 tell 10.2.14.115, length 46 | 10:03 |
| j_c | 10:00:27.176037 ARP, Request who-has 10.2.14.116 tell 10.2.14.113, length 46 | 10:03 |
| j_c | 10:00:27.368756 ARP, Request who-has 10.2.14.116 tell 10.2.14.112, length 46 | 10:03 |
| j_c | 10:00:27.622547 ARP, Request who-has 10.2.14.116 tell 10.2.14.117, length 46 | 10:03 |
| j_c | 10:00:27.631496 ARP, Request who-has 10.2.14.116 tell 10.2.14.111, length 46 | 10:03 |
| j_c | 10:00:28.092703 ARP, Request who-has 10.2.14.116 tell 10.2.14.115, length 46 | 10:03 |
| j_c | 10:00:28.174807 ARP, Request who-has 10.2.14.116 tell 10.2.14.113, length 46 | 10:03 |
| j_c | 10:00:28.367566 ARP, Request who-has 10.2.14.116 tell 10.2.14.112, length 46 | 10:03 |
| j_c | 10:00:28.619542 ARP, Request who-has 10.2.14.116 tell 10.2.14.117, length 46 | 10:03 |
| j_c | 10:00:28.630902 ARP, Request who-has 10.2.14.116 tell 10.2.14.111, length 46 | 10:03 |
| j_c | 10:00:29.043471 LLDP, length 46 | 10:03 |
| lordievader | Anyhow, you can use a pastebin service, rather than pasting directly into here. | 10:03 |
| j_c | 10:00:29.092648 ARP, Request who-has 10.2.14.116 tell 10.2.14.115, length 46 | 10:03 |
| j_c | 10:00:29.174832 ARP, Request who-has 10.2.14.116 tell 10.2.14.113, length 46 | 10:03 |
| j_c | sorry, next time. I will not post | 10:03 |
| lordievader | Is .116 localhost? If so, you are not responding to ARP it seems. | 10:03 |
| j_c | sure, I will use it from next time. | 10:03 |
| j_c | https://paste.ubuntu.com/p/m4dBNpNDkM/ | 10:04 |
| cpaelzer | lordievader: he is 10.2.14.119 | 10:06 |
| j_c | yes, my second NIC has 10.2.14.119 | 10:06 |
| cpaelzer | j_c: you checked the route for your own ip "ip route get 10.2.14.119" | 10:07 |
| cpaelzer | what happens if you use the actual target IP | 10:07 |
| j_c | I am not able to ping or ssh the server. | 10:08 |
| j_c | root@h019:~# ip route get 10.2.14.119 | 10:08 |
| j_c | local 10.2.14.119 dev lo src 10.2.14.119 uid 0 | 10:08 |
| j_c | cache <local> | 10:08 |
| cpaelzer | but .119 is you | 10:08 |
| cpaelzer | if you happen to try to reach .120 then use that in the "ip route get" command | 10:09 |
| j_c | yes, .119 is the current server. I am able to ssh from 10.1.14.119 not using the 10.2.14.119 | 10:09 |
| j_c | even that server has same issue, I configured it. I will check some other server which is working | 10:10 |
| j_c | root@h019:~# ip route get 10.2.14.117 | 10:11 |
| j_c | 10.2.14.117 dev eno2 src 10.2.14.119 uid 0 | 10:11 |
| j_c | .117 is another server which I am able to use with second NIC. above output is for that server | 10:11 |
| cpaelzer | ok so it tries to leave on eno2 | 10:18 |
| cpaelzer | now as lordievader asked - when you try like ssh ubuntu@10.2.14.117 while being on 10.2.14.119 - what does tcpdump (just for eno2) show on both systems? | 10:19 |
| TJ- | j_c: are these systems connected to a common switch or a router? | 10:22 |
| j_c | Tj-: it is connected to a switch. | 11:02 |
| TJ- | j_c: I've had recent exerperience of a switch that does offloading 'eating' ARP replies and the symptoms appear the same as yours | 11:03 |
| j_c | <cpaelzer> and <lordievader> tcpdump while showd an extra message | 11:03 |
| j_c | 10:53:16.488770 IP 192.168.0.103.32176 > h019.ssh: Flags [S], seq 3422299043, win 16384, options [mss 1460,nop,nop,sackOK,nop,wscale 3,nop,nop,TS val 3662167688 ecr 0], length 0 | 11:03 |
| j_c | Tj-: I think , I need to learn many of basic things in networking. But I will find more details, about the switch first. Thanks for the info | 11:04 |
| lordievader | On what side is that? Client or server? | 11:06 |
| TJ- | j_c: a diagnostic method I used to determine that was to run tcpdump on both hosts; I then saw: trasmitter sends Who-Has, receiver receives Who-Has, sends I-Have, transmitter never sees the I-Have | 11:06 |
| j_c | <TJ-> : are you talking about -- 10:00:28.092703 ARP, Request who-has 10.2.14.116 tell 10.2.14.115, length 46 -- These messages are due to the server which is done. | 11:07 |
| TJ- | j_c: all done whilst ensuring neither host had any firewall rules in place and no weird routes | 11:07 |
| j_c | .116 is down. I am more concern about .117, which is not able to connect to network | 11:08 |
| j_c | sorry ... .119 *** | 11:08 |
| TJ- | j_c: ahhh, I must have come in after you reported that, I arrived when you were flooding the channel! | 11:09 |
| TJ- | j_c: you have local console access to .119 ? | 11:09 |
| j_c | sorry for that--- yes, my second NIC has 10.2.14.119, which I am not able to ping or ssh | 11:09 |
| TJ- | j_c: right, so, first prove that NIC can *receive* any packets at all (This is to prove the receivce side of the phy layer is not asleep/low-powered) "tpcdump -ni <ifname>" -- you ought to see IPv6 RAs, possible STP and other broadcasts from the network. If you see nothing after a minute or so its worth investigating if the NIC hardware is 'asleep' | 11:11 |
| TJ- | j_c: secondly, are VLANs in use on the switch or this host? | 11:13 |
| j_c | root@h019:~# tcpdump -ni eno2 | 11:13 |
| j_c | 11:12:08.808791 IP 192.168.0.103.17164 > 10.2.14.119.22: Flags [S], seq 1751977968, win 16384, options [mss 1460,nop,nop,sackOK,nop,wscale 3,nop,nop,TS val 1629842332 ecr 0], length 0 | 11:13 |
| j_c | I am not sure , why length is 0 , which trying to ssh | 11:14 |
| TJ- | j_c: you're listening on 10.2.14.119 ? | 11:14 |
| j_c | No, there are no vlan. | 11:14 |
| j_c | Yes, I am listening on eno2 (10.2.14.119) | 11:15 |
| j_c | If I change the netplan settings. second NIC is coming up, in that case first NIC is going down | 11:16 |
| j_c | I think, this is more of a routing issue. But I am not able to find solution. | 11:17 |
| TJ- | j_c: so you've got 2 NICs on the .119, both connected to the same switch? | 11:17 |
| j_c | Yes. but they are in two different networks. 1 nic is 10.1.14.119. second nic is 10.2.14.119. | 11:18 |
| TJ- | j_c: right, but they're connected to the same switch? | 11:19 |
| j_c | Yes, they are connected to the same switch | 11:20 |
| TJ- | j_c: I'm trying to build a picture of the layout. What is/are the issues? You've just said when 1 NIC is brought up the other goes down, but you've also been talking about a SSH packet being 0 length | 11:21 |
| TJ- | j_c: can you show "pastebinit <( ip link; ip route; ip addr; sudo iptables -S )" | 11:23 |
| j_c | Network is the main issue. <cpaelzer> and <lordievader> asked me to do tcpdump while connecting to the server using ssh. the above log shows, when I am trying to do ssh to the server using second NIC. | 11:24 |
| TJ- | j_c: what is the IP address of the host you're SSH-ing from? is it 192.168.0.103 ? | 11:25 |
| j_c | I am trying ssh from firewall. Firewall has public IP. | 11:27 |
| j_c | ( ip link; ip route; ip addr; sudo iptables -S ) https://paste.ubuntu.com/p/4G5KyxJbBT/ | 11:27 |
| j_c | <lordievader> : that is on server side. I connected to server using first NIC card and running tcpdump on the second NIC. While trying to ssh using secind NIC ip, I am getting that line. | 11:36 |
| TJ- | j_c: aha, the issue will be your default route is taking traffic for 192.168.0.103 which is going to have a src address of 10.1.14.119 | 11:39 |
| TJ- | j_c: if you want 192.168.0.183 to be using eno2 then you need an additional route of the form "ip route add 192.168.0.0/24 dev eno2 src 10.2.14.119" | 11:40 |
| j_c | Tj- if add that, if it works. Does it work after reboot? | 11:45 |
| TJ- | j_c: you'd have to add the route into your network configuration | 11:45 |
| j_c | On ubuntu 16: it will be /etc/network/interfaces but on Ubuntu 18, should i add that to same path or netplan? | 11:47 |
| TJ- | j_c: you can add additional routes in the netplan config | 11:48 |
| j_c | Tj- same, after adding the command. Second NIC is working, but I am not able to ssh to server using first NIC | 11:49 |
| TJ- | j_c: from 192.168.0.183? | 11:49 |
| TJ- | j_c: that is expected - as I said earlier, you've connected both NICs to the same Ethernet switch, and you've got 192.168.0.182 connected to that switch, so unless you partition things with VLANs .119 is only reachable via one or the other NICS, not both | 11:52 |
| j_c | Hoo ok. Thanks TJ-. I will check about more details. I don't have complete information about the switch. | 11:54 |
| TJ- | j_c: if you have 192.168.0.183 another IP address, it would be possible on .119 to have two rules that route via .2 or .1 based on source IP from the remote SSH host. on the SSH remote host you'd have to ensure you could force the source-IP too, which may rquire policy routing table | 11:56 |
| j_c | TJ-: I have one more question, but on other servers, where it is working. They have some different routing rules | 12:00 |
| TJ- | j_c: do they have multiple NICs connected to the same switch in the same way? | 12:01 |
| TJ- | j_c: maybe they also have some policy routing tables? | 12:01 |
| j_c | yes. they are also connected to same switch. two NICs | 12:02 |
| j_c | TJ- can i add something like this and make it work. https://paste.ubuntu.com/p/rqKvS6rzS4/ | 12:05 |
| TJ- | j_c: aha, see that has the default with "onlink" | 12:10 |
| lordievader | j_c: Rather than testing ssh, I'd start with icmp. See if echo requests are coming in on the other point, replies going out and being received. | 12:12 |
| lordievader | I.e. tcpdump with a filter on icmp on both ends. | 12:12 |
| TJ- | j_c: "onlink pretend that the nexthop is directly attached to this link, even if it does not match any interface prefix" | 12:12 |
| coreycb | jamespage: python-oslo.log was removed from proposed yesterday fyi | 13:21 |
| coreycb | jamespage: nm i see you have your new version in proposed now | 13:21 |
| coreycb | jamespage: seems your new upload is still picking up the bad proposed package that was removed | 13:24 |
| jamespage | hmm | 13:24 |
| coreycb | jamespage: probably can force oslo.config to depend on >= new oslo.log version | 13:27 |
| jamespage | coreycb: not until its built... | 13:29 |
| jamespage | I suspect the binary package needs removing as well | 13:29 |
| coreycb | jamespage: yeah ok. let me ping in #ubuntu-release. | 13:29 |
| Ussat | OK, so I now have DLPAR working w/Ubuntu on PPCLE :) | 13:45 |
| Ussat | \o/ | 13:45 |
| jamespage | coreycb: sorry I'd not realized that we'd not rebuilt | 13:47 |
| coreycb | jamespage: np i didn't realize the order of events | 13:49 |
| coreycb | sahid: python-cinderclient merged/uploaded for eoan, thanks | 14:23 |
| sahid | coreycb: thanks for the review and fixes you made | 14:28 |
| bjonnh | hi cpaelzer I ended up doing a iso with cloud-init config and using the cloudimg. Works beautifuly. | 14:43 |
| bjonnh | I'm going to try with minimal image now | 14:43 |
| bjonnh | hmmm minimal doesn't autoload cloud-init… | 14:48 |
| bjonnh | too bad | 14:48 |
| bjonnh | weird the minimal image doesn't seem to even have any network so I can't even debug it… | 15:53 |
| bryce | given an installed snap package, is there a way to get snap to tell me the git repo (if any) for that package? | 16:01 |
| sdeziel | bryce: snap info $package might give you a link in the description | 16:13 |
| coreycb | jamespage: sahid: nova snapshot is uploaded for train - note I had to deal with CRLFs with core.autocrlf before importing the tarball. | 16:16 |
| bryce | sdeziel, thanks, but yeah that was first place I looked. guessing it's not being tracked then | 16:18 |
| sdeziel | bryce: I'd take a look at the web page on the snapstore then | 16:18 |
| coreycb | jamespage: sahid: i think i have the current train backport issues sorted out too | 16:20 |
| bryce | sdeziel, ok thanks | 16:30 |
| bjonnh | finally | 16:33 |
| bjonnh | so the minimal image didn't work with cloud-init data on ext4, didn't work on iso, but it works with vfat… | 16:33 |
| === jelly is now known as \driver | ||
| === \driver is now known as jelly | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!