| apawl | My home server (Ubuntu 18.04.2 LTS -- GNU/Linux 4.15.0-43-generic x86_64) is unresponsive when downloading large files. Netdata is alarming on the following "number of times, during the last 10min, ksoftirq ran out of sysctl net.core.netdev_budget or time slice, with work remaining" | 00:34 |
|---|---|---|
| apawl | It’s a 10GbE interface though, with lots of CPU / mem. | 00:34 |
| apawl | I'm wondering if there's additional tuning I need to do? This is just a home server, but beefier hardware than I've used with Ubuntu before.. | 00:34 |
| apawl | The default netdev_budget value is 300. I'm in the process of testing that value of 600. Part of me wants to jack it up way higher, but I'm unsure of the correct practice | 00:36 |
| sarnold | apawl: I think ethtool -S <iface> will show some stats on dropped packets.. | 00:44 |
| sarnold | apawl: I think your missing packets may show up in this output too | 00:44 |
| apawl | I just rebooted, so am going to run a new test. Hopefully I'll see missing packets with that command | 00:47 |
| apawl | So I'm not seeing actual packets dropped, but I am seeing this number, which matches very closely to the netdata alarm | 01:40 |
| apawl | rx_flow_control_xon: 18513 | 01:40 |
| apawl | Could that explain an unresponsive ssh session during heavy network downloads? | 01:41 |
| sarnold | flashbacks to RTS/CTS modem days.. | 01:41 |
| sarnold | a friend put this together back in 2002 .. https://lartc.org/wondershaper/ | 01:42 |
| sarnold | probably it'd be better to use fqcodel flow control things today .. | 01:42 |
| sarnold | but I wonder if this is related. (I still haven't gotten networking gear like yours, sorry it's all speculation on my part) | 01:42 |
| apawl | Honestly, I'm at the edge of my networking knowledge. Could be related? I was really surprised to see packets aren't being reported as dropped, but that I ran out of netdev_budget like 18k times | 01:45 |
| sarnold | this is wild speculation, but perhaps you did run out the budget, but the packets weren't dropped, and were handled by the next workqueue run? | 01:46 |
| apawl | I _think_ that would suggest that the behavior would be a delayed SSH session, but from what I can tell that doesn't happen. | 01:47 |
| sarnold | have you had any success with the 300 -> 600 test? | 01:48 |
| apawl | Nope. I'm going to try jacking it up to something silly. | 01:48 |
| sarnold | if that doesn't give results, then I'd think it's time for wider investigation.. maybe http://www.brendangregg.com/flamegraphs.html or http://www.brendangregg.com/usemethod.html could be good starting points | 01:50 |
| cpaelzer | jamespage: the OVS config patch for DPDK 18.11 as we have it was accepted | 06:46 |
| cpaelzer | jamespage: I updated the OVS packaging git with that update | 06:46 |
| cpaelzer | jamespage: but I didn't upload anything as that can wait for OVS 2.11 being released | 06:46 |
| cpaelzer | jamespage: but I wanted to make sure that updated patch is ready and waiting for you then :-) | 06:46 |
| lordievader | Good morning | 07:19 |
| === Bodenhaltung_ is now known as Bodenhaltung | ||
| siavoshkc | OK. What is the best method to create backups on ubuntu server | 07:53 |
| lordievader | There is no "best" method... I use dirvish. | 08:11 |
| === cpaelzer__ is now known as cpaelzer | ||
| zzarr | hello! | 08:56 |
| zzarr | I have a problem | 08:56 |
| zzarr | I get "sudo: no tty present and no askpass program specified" when I try to run a command | 08:56 |
| zzarr | (as root) | 08:56 |
| zzarr | I don't know what I did to cause this | 08:57 |
| zzarr | I would restart the machine if I could | 08:59 |
| zzarr | it's only sudo that's broken | 09:00 |
| zzarr | and su | 09:01 |
| lordievader | What are you trying to run precisely? | 09:02 |
| zzarr | I'm stuck | 09:02 |
| zzarr | sudo reboot | 09:02 |
| zzarr | it's a cloud server and I don't have access to the management environment | 09:03 |
| zzarr | when I run su it says "su: must be run from a terminal" | 09:04 |
| zzarr | the server is running 18.04 | 09:05 |
| lordievader | Almost sounds like you do not have a real terminal. | 09:05 |
| zzarr | I only have ssh | 09:06 |
| zzarr | I have tried to start a new ssh session but that did not help | 09:06 |
| zzarr | I have 2 screen sessions running and 1 tmux | 09:07 |
| lordievader | Close all ssh connections and run `sudo -t <host> "sudo ls"`, what is the output? The same error? | 09:07 |
| lordievader | Err `ssh -t <host> "sudo ls"`. | 09:07 |
| zzarr | it says "PTY allocation request failed on channel 0" | 09:09 |
| zzarr | I have bind mounted /dev /dev/pts /sys and /proc trying to run a chroot, but that should not be a problem | 09:10 |
| zzarr | I have root access now :) | 09:11 |
| jamespage | cpaelzer: tbh I'll push a new snapshot this week | 09:11 |
| lordievader | OK, good | 09:12 |
| zzarr | up and running again :D | 09:17 |
| frickler | jamespage: did you look into building ceph 12.2.11 yet? I'm getting issues with symbol changes, some additions can probably be amended, but this deletion looks bad https://paste.ubuntu.com/p/87JMf8fppV/ | 09:36 |
| jamespage | frickler: yes had the same thing - not to worried about the deletion - https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3534 | 09:38 |
| jamespage | frickler: my build/test loop is quite slow with LP builders as they don't have huge amounts of RAM | 09:38 |
| frickler | jamespage: oh, great, I'll try to build that locally, thx | 09:41 |
| jamespage | frickler: pushed all to https://code.launchpad.net/~ubuntu-server-dev/ubuntu/+source/ceph | 09:46 |
| jamespage | frickler: that PPA is the tip of my work for bionic | 09:47 |
| jamespage | frickler: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3535 has similar but for xenial | 09:48 |
| jamespage | frickler: that should have been https://code.launchpad.net/~ubuntu-server-dev/ubuntu/+source/ceph/+git/ceph I think | 09:52 |
| frickler | jamespage: seems it defaults to git already | 09:55 |
| jamespage | apparently so | 09:56 |
| * jamespage needs more coffee | 09:56 | |
| frickler | thanks to smoser I also have a bit of experience with bzr, though ;) | 09:56 |
| rbasak | kstenerud: so is bug 1812387 actually in the tmux-plugin-manager package? Because that's in universe. | 10:11 |
| ubottu | bug 1812387 in tmux (Ubuntu) "tmux crashes on tpm init" [Medium,Confirmed] https://launchpad.net/bugs/1812387 | 10:11 |
| kstenerud | oh ok, so that means we don't bugfix universe? | 10:28 |
| blackflow | "we"? | 10:34 |
| rbasak | kstenerud: Canonical's commitment is in main only. That's effectively what main means. | 11:04 |
| zenirc369 | command| grep -w heat | 12:46 |
| zenirc369 | includes output with both heat and heat-cfn | 12:46 |
| zenirc369 | how can i exclude heat-cfn from output to get the output with only the word 'heat' | 12:46 |
| blackflow | zenirc369: ... | grep -v "heat-cfn" | 12:47 |
| frickler | zenirc369: or use negative lookhead: grep -P 'heat(?!-cfn)' | 13:09 |
| dpawlik | coreycb: hi, don't you have any error on queens/rocky | 13:23 |
| dpawlik | like: http://paste.openstack.org/show/ppKG3IWpaBvuaiEnyf7h/ | 13:23 |
| dpawlik | libvirt version: 4.0.0-1ubuntu8.6 | 13:23 |
| dpawlik | all is installed from bionic repo | 13:24 |
| dpawlik | coreycb: I report a bug: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1815910 | 13:33 |
| ubottu | Launchpad bug 1815910 in libvirt (Ubuntu) "Apparmor blocks access to /dev/vhost-net" [Undecided,New] | 13:33 |
| coreycb | cpaelzer: is that a known issue? ^ | 13:37 |
| cpaelzer | coreycb: reading | 13:45 |
| cpaelzer | dpawlik: let me guess - your guest initially does not have any vhost-net devices and you then hotplug one right? | 13:46 |
| cpaelzer | the rule for /dev/vhost-net i added on guest definition if a network device has VIR_DOMAIN_NET_BACKEND_TYPE_QEMU and virDomainNetIsVirtioModel | 13:49 |
| cpaelzer | that means if you start withotu it isn't added | 13:49 |
| cpaelzer | there might be no security relabeling call that we could hook onto for adding the network device | 13:49 |
| cpaelzer | let me check that | 13:49 |
| cpaelzer | dpawlik: please confirm (or correct) my assumptions if you are still around | 13:49 |
| cpaelzer | I replied on the bug, let me know there once you have time | 13:56 |
| cpaelzer | coreycb: I assume you usually start with at least some vhost-net devices right away right? | 13:56 |
| coreycb | cpaelzer: i would assume so too but would have to check a deployed instance to be sure | 13:57 |
| cpaelzer | coreycb: well yours work, which is a hard argument that it is that way :-) | 13:57 |
| coreycb | cpaelzer: sahid and i are working through enabling py37 unit tests for openstack. how's libvirt-python wrt py37? | 14:05 |
| dpawlik | cpaelzer: w8, let me finish eat lunch :) | 14:05 |
| coreycb | cpaelzer: 14 months until likely having to support py37 too in 20.04 | 14:06 |
| coreycb | i mean py38 too | 14:06 |
| cpaelzer | dpawlik: take yur time with lunch, no rush | 14:06 |
| dpawlik | cpaelzer: yes, I would like to add a vm which doesn't have any nic, attach a new interface so later it raises an error | 14:06 |
| cpaelzer | coreycb: TBH - I have exactly zero idea about the py3 support state of python-libvirt | 14:07 |
| cpaelzer | dpawlik: ok, then all my assumptions are right | 14:07 |
| cpaelzer | dpawlik: just state so on the bug where I asked | 14:07 |
| cpaelzer | I need to debug the labeling calls that go on while hot adding a further network device | 14:07 |
| cpaelzer | depending on that result this is annoying or really hard to implement, so no commitment before I have that result | 14:08 |
| cpaelzer | :-) | 14:08 |
| dpawlik | cpaelzer: you are genius. Now when I read bug on launchpad I see that I miss important information | 14:09 |
| cpaelzer | really, then add it please :-) | 14:09 |
| coreycb | cpaelzer: would you be able to push on py37 support by any chance? doko will be very happy if you do. :) 20.04 will likely be python 3.8 by default but minimally will be python 3.7. also thanks for looking at the vhost-net issue. | 14:09 |
| dpawlik | but you find out that :D | 14:09 |
| dpawlik | cpaelzer: ack | 14:10 |
| cpaelzer | it was enough to get me thinking, which to admit is more than what you usually find on the average bug report, so you did well already | 14:10 |
| cpaelzer | coreycb: I really have had no business with python-libvirt so far | 14:10 |
| cpaelzer | coreycb: your push a.k.a email will be just as effective (or not) as mine | 14:11 |
| coreycb | cpaelzer: ok looks like we generally just sync that one from debian | 14:12 |
| cpaelzer | yep | 14:13 |
| cpaelzer | coreycb: I appreciate that you think I'm involved everywhere, but no :-) | 14:14 |
| coreycb | cpaelzer: but you are Mr Libvirt :) | 14:14 |
| dpawlik | coreycb: +1 | 14:17 |
| dpawlik | he is a Mr Libvirt ;D | 14:17 |
| coreycb | cpaelzer: sahid: fyi https://bugzilla.redhat.com/show_bug.cgi?id=1677321 | 14:32 |
| ubottu | bugzilla.redhat.com bug 1677321 in libvirt "libvirt-python missing support for modern python versions (python 3.7)" [Unspecified,New] | 14:32 |
| smoser | frickler: if you don't have anything nice to say, don't say anything at all ;) | 14:34 |
| frickler | smoser: hmm, the nice thing I'd have to say about that is that you responded to my request to move to git pretty fast :) | 14:39 |
| smoser | frickler: thanks! | 14:54 |
| cpaelzer | coreycb: 1815917 is a tracker onto that, thanks for filing | 15:05 |
| coreycb | cpaelzer: ok thx | 15:07 |
| kstenerud | ahasenack: In reference to https://bugs.launchpad.net/ubuntu/+source/freeipmi/+bug/1808637/comments/3 | 16:01 |
| ubottu | Launchpad bug 1808637 in freeipmi (Ubuntu) "freeipmi-ipmidetect postinst fails on default install" [Undecided,Triaged] | 16:01 |
| kstenerud | How would one check the RUN variable in systemd? | 16:01 |
| sdeziel | kstenerud: probably not the most elegant thing but you could add ExecStartPre=/bin/echo $RUN or check ipmidetectd's /proc/$pid/environment | 16:04 |
| sdeziel | kstenerud: maybe adding "export SYSTEMD_LOG_LEVEL=debug" before "systemctl start" would also provide what env vars it is going inject to the ExecStart process | 16:08 |
| kstenerud | sdeziel: That would definitely show what's going on, but I need to always allow the postinstall to succeed, but print out a warning message when $RUN is no. | 16:12 |
| kstenerud | basically the package can't run with no config, but there's no default config. So it needs to say something to the user, but also allow the package install to succeed via apt | 16:12 |
| sdeziel | kstenerud: ExecStartPre=/usr/bin/test "$RUN" != "no" | 16:14 |
| sdeziel | untested of course | 16:17 |
| kstenerud | ok thanks I'll play around with it | 16:18 |
| sdeziel | but that sounds like a kludge. It would be better to instead add a ConditionPathExist=/path/to/config IMHO | 16:24 |
| sdeziel | I'm spinning a disco container to take a closer look | 16:25 |
| sdeziel | ipmidetectd.conf is provided by the package but in a unusable state so I think the ExecStartPre test is the way to go then | 16:31 |
| === Ussat-1 is now known as Ussat | ||
| kinghat | if you disable ssh access for root and try to login via root does it just say no or give you password attempts even though you will never be able to login? | 18:41 |
| kinghat | for me its letting me try to login 3 times with password but getting permission denied. | 18:41 |
| tomreyn | if password authentication is enabled and preferred / no preffered authentications succeed, i guess it'll show password prompts, otherwise it'd allow for user enumeration. | 18:43 |
| kinghat | tomreyn: is password login default? i dont have that enabled in sshd_config | 18:48 |
| kinghat | i dont even see it in there actually. | 18:48 |
| kinghat | or is that what `PAM` is? | 18:49 |
| kinghat | ok so password auth config is for both root and users? | 18:52 |
| kinghat | i guess i dont full understand what all im disabling if i disable PAM. | 18:53 |
| kinghat | fully* | 18:53 |
| tomreyn | kinghat: "PasswordAuthentication yes" is default, i think, also "PermitRootLogin prohibit-password" | 18:56 |
| kinghat | is it smart just to disable ssh access via root altogether? | 18:57 |
| tomreyn | there is a statement on sshd_config explaining what setting "UsePAM false" results in | 18:58 |
| tomreyn | what is smart surely depends on your needs and how you use it | 18:58 |
| tomreyn | it is smart to only allow those authentication methods and users to authenticate that you strictly depend on. | 18:59 |
| kinghat | ya disabling ssh root access altogether seems smart. im just not sure about my user. i have the key setup but not sure if i should kill its password access. i mean you have to share the private key around or add it to every device you need to connect from. or make their own. and managing all that. | 19:00 |
| kinghat | also i had a key setup that i used for root and my user, i know where the user pub key went, but what about the root reference? | 19:06 |
| teward | well the OTHER way is to do it by whitelist - whitelisting specific IPs to access SSH | 19:08 |
| teward | but that won't work unless you have static IPs on every side | 19:08 |
| teward | or you could set up a VPN into your box and then only allow SSH from that VPN | 19:09 |
| kinghat | ya there is that | 19:10 |
| kinghat | i want to try and find the pub key that i submitted for root to use and remove it and leave the one thats in my users `.ssh` file. | 19:11 |
| kinghat | basically, where is the roots `authorized_keys` file? | 19:15 |
| benharri | /root/.ssh/authorized_keys | 19:16 |
| kinghat | hmm i dont have that | 19:16 |
| kinghat | not sure how i was logging in via root with key. | 19:17 |
| benharri | you _can_ specify an alternate authorized_keys filename in /etc/ssh/sshd_config | 19:17 |
| benharri | but that's non-standard | 19:17 |
| kinghat | huh. odd | 19:18 |
| benharri | re: root ssh login: i usually set it to prohibit-password in the unlikely event that i muck up my normal user account or its keys | 19:18 |
| benharri | and only keep my hardware-backed gpg-agent-provided pubkey in the authorized_keys | 19:19 |
| kinghat | thats a good idea | 19:21 |
| benharri | it's definitely come in handy more than once | 19:22 |
| benharri | mucked up /etc/{passwd,shadow,group} during a migration to a new box | 19:23 |
| Ussat | so can I verify, on 16.04 I used ot use: apt-get -y autoremove --purge to remove old kernels but keep the last two, is that no longer the case ? | 20:04 |
| tomreyn | Ussat: i think that's still the case on newer releases (you're not saying which one you're asking about) if you have package "linux-image" installed. | 20:07 |
| tomreyn | "last two" is almost correct there, it also keeps the one which is currently running. | 20:07 |
| Ussat | OK so N+2 thanks | 20:11 |
| Ussat | ok, and if I dont have linux-image ? | 20:13 |
| tomreyn | i'm not certain about this but i *think* they are not removed automatically then. but this may be plain wrong or outdated, i have not tried to verify this. | 20:14 |
| Ussat | OK, thanks | 20:15 |
| Ussat | re newer systems, 18.X | 20:15 |
| Deihmos | If I wanted to remove all the packages installed with Ubuntu is that possible? | 20:20 |
| tomreyn | Deihmos: theoretically it may be, but then you'd have nothing left to work with. why would you wnat to anyways? | 20:21 |
| Deihmos | How would I do it? | 20:22 |
| sarnold | constructing the order of packages to remove so that the prerm and postrm scripts that are run don't have dependencies on already-removed packages might be difficult | 20:23 |
| tomreyn | it's a somewhat interesting experiment, but i guess it'd get somewhere between boring and daunting really quick. | 20:28 |
| tomreyn | but removing essential packages is not supported, and we haven't seen an actual use case either. | 20:28 |
| CarlenWhite | I'm screwing around with ubuntu-like and...I'm not sure what I'm looking at. | 22:36 |
| CarlenWhite | BW checkered background with a teal bordered terminal. | 22:37 |
| CarlenWhite | Holding left click in empty space gives me 'Main Menu' and middle click gives me 'Window Ops' | 22:38 |
| CarlenWhite | Oh, it's TWM | 22:40 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!