[06:08] <sergeant> while connecting to the mongo shell on an ubuntu server. I am getting this error http://paste.ubuntu.com/p/SS7VBxFdck/. Please help!!!!!!!
[06:10] <lotuspsychje> sergeant: please idle a bit here, volunteers might be still waking up ok
[06:12] <sergeant> yeah sorry
[06:17] <igordc> or going to sleep :(
[06:18] <igordc> sergeant, I haven't particularly dealt with mongo much but it looks like it isn't running on that specified port
[06:18] <igordc> sergeant, double check the server
[06:19] <sergeant> how do i check that ?
=== cpaelzer__ is now known as cpaelzer
[08:20] <eject_ck> After installing packages updates on 5 Ubuntu 16.04 servers one wont start (just stuck during kernel boot), https://imgur.com/a/Nukuvk9
[08:20] <eject_ck> Anybody had such an issue ?
[08:22] <eject_ck> how can I collect details for such probelm?
[08:24] <eject_ck> kernel 4.4.0.141
[08:31] <lotuspsychje> !info linux-image-generic xenial
[08:31] <ubottu> linux-image-generic (source: linux-meta): Generic Linux kernel image. In component main, is optional. Version 4.4.0.141.147 (xenial), package size 2 kB, installed size 14 kB
[08:31] <lotuspsychje> eject_ck: did you try booting a previous kernel yet?
[08:32] <eject_ck> yes, no luck
[08:33] <eject_ck> I tried to boot with dis_ucode_ldr   [X86] Disable the microcode loader. and it worked
[08:33] <eject_ck> interesting now why it caused problems
[08:34] <lotuspsychje> eject_ck: maybe provide us some dpkg logs from the installed updates recently, maybe volunteers can find a link
[08:35] <eject_ck> ok
[08:36] <lordievader> Good morning
[08:55] <eject_ck> lotuspsychje: where to send ?
[08:55] <lotuspsychje> !paste | eject_ck
[08:55] <ubottu> eject_ck: For posting multi-line texts into the channel, please use https://paste.ubuntu.com | To post !screenshots use https://imgur.com/ !pastebinit to paste directly from command line | Make sure you give us the URL for your paste - see also the channel topic.
[08:57] <eject_ck> https://paste.ubuntu.com/p/4f4rfg2Xvk/
[08:59] <lotuspsychje> eject_ck: that seems like a big dpkg list, is that a normal update or did you upgrade or wait long?
[09:01] <eject_ck> normal update
[09:01] <eject_ck> after long break
[09:01] <lordievader> You want the apt history log 😉
[09:02] <lotuspsychje> lordievader: he got 5 xenial servers not booting anymore
[09:02] <eject_ck> no no
[09:02] <lordievader> `/var/log/apt/history.log` to be precise.
[09:02] <eject_ck> I got 1 server out of 5 not bootable
[09:02] <lotuspsychje> ah
[09:03] <lordievader> `dpkg -l` just shows what is installed. The apt history log shows what it did, what it updated, installed, etc.
[09:03] <eject_ck> adding dis_ucode_ldr to boot options helped to start server
[09:03] <eject_ck> then I ran apt update && initramfs -u
[09:03] <Mudchains> Good morning all. I have a old 8.04 ubuntu server with mysql databases on it. I want to setup a new 18.04 server and migrate the databases. Is there any guide lines about scsci controllers and disk design ? I am running VMware Vcenter/ESX 6.5
[09:04] <Mudchains> hi lotuspsychje ;)
[09:04] <eject_ck> then I downloaded latest microcode from Intel and put it into /var/firmware and restarted
[09:04] <eject_ck> it started with no problems
[09:10] <lordievader> Mudchains: What you could do is boot a live-usb/iso of 18.04 and check if everything works.
[09:12] <Mudchains> lordievader: I am running 18.04 on multiple VM's and they are working fine. They have only a 'old' disk design and scsci controller attached.
[09:13] <Mudchains> For the next VM's I want to make a template :)
[09:13] <lordievader> If you only have databases on there, wouldn't it be a better idea to setup a new VM with 18.04 install maria-db and transfer the databases to the new VM?
[09:14] <Mudchains> lordievader: thats my idea also
[09:17] <Mudchains> lordievader : do you have experience with the vmware paravirtual scsci controller and performance?
[09:17] <lordievader> No, I use kvm/qemu.
[09:17] <Mudchains> ah ok :)
[09:17] <lordievader> I try to stay away from vmware 😉
[09:18] <Mudchains> we are running 310+ machines on vmware atm :)
[09:19] <Mudchains> google doesnt say anything about optimized ubuntu templates for vmware unfortunally
=== jelly-home is now known as jelly
=== lotuspsychje_ is now known as lotuspsychje
[09:59] <Mudchains> lordievader: the new server is up and running, now the most time taking job..mysql, optimalisation and db migration.. :)
[10:17] <Mudchains> lordievader: why choosing maria-db btw?
[10:18] <lordievader> Because mysql is Oracle now.
[10:18] <lordievader> Maria-db is drop-in replacement.
[10:19] <Mudchains> lordievader: thats the only reason? :)
[10:20] <lordievader> For me it is, but I've moved away from mysql alltogether.
[10:21] <lordievader> A more indepth comparison: https://blog.panoply.io/a-comparative-vmariadb-vs-mysql
[10:23] <Mudchains> lordievader: I just readed it haha :D
[11:03] <ahasenack> good morning
[11:04] <rbasak> o/
[11:05] <ahasenack> hi rbasak
[11:07] <ahasenack> kstenerud: did you see my notes about schleuder?
[11:07] <ahasenack> I added a bug to the exim4 card comments
[11:09] <kstenerud> Yes, so we need to fix schleuder to unblock exim4 right?
[11:10] <ahasenack> yep, for a loose definition of "fix"
[11:10] <ahasenack> might need kicking out too, I asked in #ubuntu-release yesterday, didn't get a response
[11:18] <Mudchains> lordievader: pff what a job, migrating the databases xD
[11:19] <lordievader> Is it?
[11:19] <lordievader> Dump, scp, import. Right?
[11:19] <Mudchains> at least the new ubuntu 18.04 server is up and running :)
[11:20] <Mudchains> lordievader: yes thats correct, also found a query to copy the mysql users
[11:22] <Mudchains> the most annoying part is all the application/odbc connections
[14:41] <awalende> Hi there, is it possible to write iptable rules for vlans?
[14:42] <awalende> Id like to block all incoming on a vlan interface of mine
[14:42] <sdeziel> awalende: yes, -i and -o support any interface name
[14:43] <awalende> sooo "iptables -P INPUT DROP -i vlan118" should do the trick
[14:44] <sdeziel> awalende: well, -P doesn't accept -i
[14:45] <sdeziel> awalende: -P is to set the chain policy (aka default faith of a packet reaching the end of the chain)
[14:45] <sdeziel> awalende: but any -I/-A rules that you have can use -i vlan118
[14:46] <sdeziel> awalende: ex: "iptables -A INPUT -i vlan118 -j DROP"
[14:46] <awalende> ah okay, I'll try this. Thanks!
[14:51] <awalende> mhh weird, "iptables -L" shows me that I have a new DROP rule. However this list does not show my any information on which vlan this rule is enforced.
[14:52] <awalende> "DROP       all  --  anywhere             anywhere  "
[14:52] <sdeziel> awalende: could you pastebin "iptables-save" ?
[14:52] <TJ-> awalende: "iptables -nvL"
[14:54] <awalende> https://pastebin.com/k5YhK1RZ
[14:55] <awalende> ah I believe "iptables -nvL" did the trick, I see the rule for vlan118 now
[14:56] <awalende> -nvl - > https://pastebin.com/WViuyD5G
[14:57] <awalende> thanks for your help folks :)
[14:59] <sdeziel> awalende: np. FYI, you can use prefix matching for input/output devices like this "-i vlan+"
[14:59] <sdeziel> I find this quite useful at times so I thought I'd mention it ;)
[15:00] <awalende> :)
=== ossurayynot is now known as tonyyarusso
[16:15] <herald85> hi, i keep having issues during updates with downloading the required version of linux-headers. When I manually browse to http://security.ubuntu.com/ubuntu/pool/main/l/linux/  and click on linux-headers-4.4.0-141_4.4.0-141.167_all.deb it also fails to download. Anyone know how I can work around this?
[16:16] <ansyeb> hello. how it that possible? https://pastebin.com/YgJttRpw
[16:16] <ansyeb> what is on 22001?
[16:16] <ansyeb> <SerajewelKS> ansyeb: almost certainly a -R forward from one of your users
[16:16] <ansyeb> could someone provide a link to the corresponding manual page?
[16:20] <jelly> ansyeb: man ssh_config, search for RemoteForward
[16:20] <ansyeb> I found this: http://www.unixwiz.net/techtips/ssh-agent-forwarding.html#fwd
[16:21] <jelly> ssh has about 4-5 different forwardings and agent forwarding or X11 forwarding is not relevant for -R
[16:22] <jelly> -R is a ssh client command line option that opens a listener on the remote side and tunnels tcp connections somewhere visible to the client side
[16:22] <ansyeb> oh man..
[16:23] <jelly> serajewelks suggested you were seeing a remote listener side of such a setup
[16:23] <jelly> the* remote listener side
[16:24] <jelly> if you want web search keywords: ssh remote forwarding
[16:24] <ansyeb> щл ен
[16:24] <ansyeb> ok ty
[16:26] <jelly> it's a way to enable access to a service that ssh server you connected to can't otherwise reach; somethimes used as a workaround instead of having to punch holes in firewalls
[16:32] <jelly> ansyeb: what does "ps -fp 1973" say? it might be interesting to see what is its parent process, that might confirm the -R theory.  However,but it is somewhat unusual for a sshd process with a -R listener socket open to be running as root
[17:29] <baffle> I have a server with 2*L3 uplinks; The uplinks has a /31 for basic connectivity, and the same /32 on both interfaces+loopback. 0.0.0.0/0 is routed via the /31 on both interfaces, and src set to the /32. Packets gets sent randomly out via both interfaces, but there is asymmetrical routing so replies might come to the other interfaces; If this happens, the package seems to just disappear. I have
[17:29] <baffle> rp_filter set to 0, what else have I forgotten? Iptables INPUT/FORWARD is set to ACCEPT...
[17:29] <baffle> Anyone have any ideas what it could be?
[19:32] <sarnold> baffle: rp_filter on *all* interfaces? or just the global config?
[19:50] <baffle> sarnold: It's set to 2 on all interfaces..
[19:52] <sarnold> baffle: this guide suggests assymetric routing uses would benefit from '1' https://www.tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.kernel.rpf.html
[19:58] <baffle> sarnold: That's weird, this does not match documentation in https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt <- 1 is strict, 2 is loose. 0 is disabled.
[19:58] <baffle> (I have it set to 2, not 0, so that was wrong on my part)
[20:00] <baffle> Set to 0 on all/default/<interfaces> now, same behaviour.
[20:00] <baffle> And I have enabled log_martians too, and there are no entries in any logs. :-/
[20:02] <sarnold> baffle: huhn. that is more or less exactly opposite of the what the guide said :(
[20:14] <baffle> sarnold: Any tips on how to debug further?
[20:15] <sarnold> baffle: nothing good, I'm insanely rusty on router kinds of things :/ .. maybe firewall packet counts? perhaps they're being blocked by rules?
[20:17] <TJ-> baffle: I may be confused but your original question seems to be talking about *replies* coming into this server on a different interface - rp_filter is about sending replies *out* from this system
[20:19] <sdeziel> rp_filter is a source validation mechanism so should apply to inbound traffic I think
[20:19] <baffle> sdeziel: But maybe in the forwarding path?
[20:19] <baffle> I.e. for packets passing.
[20:19] <baffle> (Typically a router)
[20:20] <sdeziel> baffle: AFAIK, rp_filter is applied on traffic reception even in the forwarding case
[20:21] <sdeziel> baffle: I don't have the time to look myself but surely sharing "ip a; ip ro" would help folks have a better idea of your setup
[20:22] <baffle> sdeziel: Oh. I forgot the paste-link I prepared... https://paste2.org/FLH32Z5F
[20:22] <baffle> (rp_filter is now set to 0, not 2 as in the paste)
[20:22] <sdeziel> baffle: is 10.200.0.1/32 supposed to be some kind of HA IP or Virtual IP?
[20:23] <baffle> sdeziel: Yes, it is supposed to be a HA IP.
[20:23] <sdeziel> baffle: then I wouldn't expect to see it configured on the 2 NICs at the same time.
[20:25] <baffle> sdeziel: Well, I only had it on loopback before, then packets could be sent to it from an external host.. I.e. to reach a service bound to that IP. But for the host to source packets to that IP, and not the /31 linknets, one both needs to set the src in ip route + set IP on the interfaces. If IP is not set on the interfaces, packets will not be sent out by the kernel..
[20:25] <sdeziel> baffle: looks like you are running bgp which is unexplored territory for me, sorry ;)
[20:26] <sdeziel> baffle: I would have think that you could set just an ip route with source specification without needing to actually have the IP configured on the real outbound NIC (just lo0)
[20:27] <sarnold> btw how does a /31 work? there's two addresses, and the all-ones-equiv would be broadcast.. leaving the all-zeros for the one host?
[20:28] <baffle> sdeziel: Yes, it's BGP, but all it does it populate the routing table, and announce the /32 to the switches.
[20:29] <baffle> sdeziel: If I remove the IP from the outbound NICs, and just put it in lo0, and have src set to the IP, no packets go out..
[20:30] <sdeziel> baffle: your default route with 2 nexthops looks good to me. Have you confirmed with tcpdump what's going on?
[20:31] <sdeziel> sarnold: I'm suspecting some kind of p2p setup
[20:32] <sdeziel> ipcalc says there are no broadcasts for /31: https://paste.ubuntu.com/p/yhR2dkxS8B/
[20:33] <sdeziel> ipcalc also mentions https://tools.ietf.org/html/rfc3021 for /31 ranges :)
[20:34] <sarnold> ha! of course there's an rfc to answer my exact question :)
[20:34] <sarnold> thanks sdeziel
[20:37] <baffle> sarnold: It works great for linknets between routers, and there is no network/broadcast address.
[20:40] <baffle> sdeziel: Yeah, if I source icmp packets from interface ens1f0 I get echo+reply back on ens1f0. If I source icmp packets from interface ens1f1 the echo is sent out from ens1f1, but the reply comes back on ens1f0 (due to what I ping being a few hops away, and having a best path via the switch ens1f0 is connected to)
[20:43] <sdeziel> baffle: have you tried "ping -I 10.200.0.1 10.100.1.5" ?
[20:44] <sdeziel> baffle: I'd expect it to work and load balance the echo requests evenly between the 2 NICs since they have the same weight
[20:45] <sdeziel> baffle: the echo replies might all come via ens1f0 though if the switch behaves that way
[20:46] <sdeziel> baffle: out of curiosity, why deal with this at L3 instead of L2 (LACP, bonds, etc)?
[20:51] <baffle> sdeziel: Uh, that worked. One minute, I'll check something....
[20:54] <baffle> sdeziel: Whaddayaknow. Facepalm time. It works fine, and probaly has been all along, I think I was looking the wrong place all along. All day.
[20:54] <sdeziel> baffle: hehe
[20:56] <sarnold> uhoh
[20:56] <sarnold> whatr exactly was the wrong thing in question? :)
[20:59] <baffle> sdeziel: The reason for going with L3 instead of L2 is to avoid having MLAG on the switches, I've seen that (and stacking) fail too many times..
[20:59] <blackflow> sarnold: oh hey, you're a ZFS fan amirite?
[20:59] <sdeziel> baffle: OK
[20:59] <sarnold> blackflow: yeah
[21:00] <blackflow> sarnold: is the ZFS wiki page editable only by ubuntu devs, or community? because the uses cases are blatantly lying :)  https://wiki.ubuntu.com/ZFS
[21:00] <baffle> sdeziel: So, instead of using L2 that we all know and love, I'm introducing more complexity with routing instead.. It is probably a bad idea.. But at least it is standardized, and you can use whatever vendor..
[21:00] <blackflow> that really needs some correction, because it's very much false.
[21:00] <sarnold> blackflow: I'd expect anyone in the right launchpad group would be able to edit it
[21:01] <blackflow> should I open a bug report then?
[21:01] <blackflow> eg. Jack's use case is fiction. ZFS does no such thing.
[21:01] <sdeziel> baffle: I've heard good things about L3 redundant setups so I guess it's just a matter of fully understanding this new paradigm
[21:01] <sarnold> blackflow: hah yeah that looks way wrong
[21:01] <blackflow> so is Ari's use case, ZFS does not do that :)
[21:02] <sarnold> ha
[21:02] <baffle> sarnold: I think what I originally observed, but failed to catch, was that outgoing connections from a container got masqueraded (randomly) to linknet IPs on interfaces + the "HA" IP.
[21:03] <lordcirth_> Yeah, Jack's is handled by btrfs, I'm not aware of any filesystem that just grabs storage devices lol
[21:04] <sarnold> blackflow: if you want to edit the wiki, this is the group to join https://launchpad.net/~ubuntu-wiki-editors -- many other groups are already included on the thing, so maybe it'd make sense to join one of the other groups instead of this one
[21:04] <sarnold> blackflow: I've got to run for lunch.. if you'd rather not bother, just let me know and I'll happily delete those usecases :)
[21:04] <blackflow> sarnold: thanks, I'll see what I can do first.
[21:05] <sdeziel> baffle: the masquerading shouldn't be random since your default route says to go out with 10.200.0.1, no?
[21:05] <sarnold> blackflow: thanks
[21:05] <blackflow> sarnold: bon apetit!
[21:05] <sarnold> :D
[21:05] <baffle> sdeziel: Yes, but it is still very hard to know what is the correct way to design a spine/leaf design with full redundancy on hosts.. Some designs seems to think that spine/leaf should be core, with ToR switches connected to the leafs, and host using L2 to one ToR switch. Or LACP/MLAG to two ToR switchces. Some designs use ToR switches as leafs (as I do).. But that both in a rack should use iBGP and
[21:05] <baffle> bgp and be in the same AS.. Some have the same AS on spines.. It's very confusing..
[21:05] <baffle> sdeziel: That's what I tought...
[21:06] <sdeziel> baffle: the only semi-random (round robin I think) portion would be the outbound NIC the kernel picks
[21:06] <baffle> sdeziel: But I'll modify the rule to have --to-source..
[21:07] <sdeziel> baffle: out of curiosity, if you run this multiple time, do you see the kernel alternating the outbound NIC: ip ro get 1.1.1.1
[21:11] <baffle> sdeziel: No, that returns same IP consistently. And I've set sys/net/ipv4/fib_multipath_hash_policy to 1 (L4).. But 1 sec, I'll see what happens.
[21:11] <sdeziel> baffle: yeah, same source IP but what about the dev?
[21:12] <baffle> sdeziel: Same device, same link IP, same source IP. I.e. -> 1.1.1.1 via 10.20.128.32 dev ens1f0 src 10.200.0.1 uid 1000
[21:13] <baffle> sdeziel: But I assume that is just cached. If I actually generate TCP traffic to the same host now, the flows round-robin.
[21:13] <baffle> sdeziel: I'll modify the masquerade rule and test now..
[21:13] <sdeziel> baffle: probably but I would have appreciate the kernel telling you about the round robin thing
[21:15] <sdeziel> "ip route get fibmatch 1.1.1.1" maybe?
[21:20] <baffle> sdeziel: That works, returns both path.
[21:20] <baffle> s
[21:21] <baffle> sdeziel: Also, manually replacing MASQUERADE with -J SNAT --to-source works a treat.
[21:21] <sdeziel> baffle: thanks good to know
[21:22] <sdeziel> baffle: I don't understand why MASQUERADE would do the wrong thing though
[21:23] <teward> masquerade uses the primary IP address on the system, if I'm not mistaken
[21:23] <teward> and not "alternative IPs" (secondary, tertiary, extra, etc.)
[21:25] <sdeziel> teward: it should make a decision based on the info from routing table, or at least that would be a logical (to me) way of doing it
[21:25] <teward> while I agree with you, i'm also coming in late.
[21:25] <teward> so I'm not up to speed :P
[21:27] <baffle> teward: What is the "primary" IP anyway?
[21:27] <sdeziel> that's what the routing table tells you it is
[21:28] <teward> unless your routing tables are screwed, the 'default route' according to the routing table typically
[21:28] <teward> usually the first IP address on an interface if you don't have any custom routing tables in play
[21:29] <baffle> Hmm, wonder what happens if I reorder IP addresses in netplan.
[21:30] <teward> just as an FYI I came in late, did you share your configuration?  Do you have custom policy-based route rules set up?
[21:30] <teward> (which would therefore alter the 'default routes')
[21:30] <sdeziel> teward: 2 nexthops with same weight as default gw
[21:30] <sdeziel> https://paste2.org/FLH32Z5F
[21:31] <teward> hah BGP is at play I see
[21:32] <teward> sdeziel: I usually consider in a Multi IP scenario SNAT/DNAT is better than the MASQUERADE functionality in iptables
[21:32] <teward> just from experience
[21:32] <baffle> sdeziel: I totally agree.
[21:32] <sdeziel> teward: agreed but I would still expect MASQUERADE to do the right thing in such scenario
[21:33] <teward> sdeziel: my two cents is I call masquerade a 'hackish' way to SNAT/DNAT automagically.
[21:33] <teward> just my thoughts on it :P
[21:34] <sdeziel> this automagic should be reliable ;)
[21:34] <teward> sdeziel: when is anything networking related EVER reliable :p
[21:36] <sdeziel> teward: lo is pretty reliable but that's the exception
[21:37] <baffle> Now the /32 is the first IP on both interfaces, but MASQUERADE still chooses the link-net as NAT source. ¯\_(ツ)_/¯
[21:38] <baffle> Guess I'll have to disable the automatic creation of NAT rules in Docker. Maybe it's time to check if they've added more functionality..
[21:38] <teward> sdeziel: well other than that lol
[21:39] <sdeziel> baffle: oh well, I was wrong (again) :P
[21:39] <teward> baffle: to be fair in my containerized environments (EXCEPT for this laptop, because it has only 1 IP lol), i never trust MASQUERADE to do what I want lol
[21:39] <teward> always SNAT everything :P
[21:39] <teward> my two cents.
[21:42] <baffle> teward: I don't think I have a choice.
[21:42] <Mudchains> i love it when a old optimized my.cnf of mysql5.0 fixes the new slow installed mysql5.7 server
[21:43] <Mudchains> first i changed the scsi controllers, but then ubuntu didnt start up anymore haha
[21:44] <baffle> sdeziel/sarnold/teward++: Thanks for all the help!
[21:46] <sdeziel> baffle: yw
=== lifeless_ is now known as lifeless