[00:06] <mason> ping sbeattie - PM?
[00:39] <mason> unping
[12:05] <ahasenack> good morning
[12:11] <ahasenack> kstenerud: try the server-next tag
[12:11] <ahasenack> kstenerud: https://bugs.launchpad.net/ubuntu/+bugs?field.tag=server-next
[12:11] <ahasenack> kstenerud: and bite-size
[12:11] <ahasenack> kstenerud: https://bugs.launchpad.net/ubuntu/+bugs?field.tag=bitesize
[12:12] <ahasenack> not all are server, though, that query needs to be refined
[12:12] <kstenerud> ok
[12:12] <ahasenack> use advanced search
[12:17] <ahasenack> kstenerud: https://bugs.launchpad.net/ubuntu/+source/fetchmail/+bug/1798786 should not be fix committed
[12:18] <ahasenack> kstenerud: I'm going to add a cosmic task, and we will leave the main task for tracking the progress in the development release of ubuntu once it opens
[12:18] <ahasenack> kstenerud: are other releases also affected?
[12:19] <ahasenack> looks like bionic is, that's where the bug was reported
[12:20] <kstenerud> I think all previous versions will be affected
[12:20] <kstenerud> it wasn't filling out a name field, but server side wasn't checking either before
[12:21] <ahasenack> can you check please and let me know? Then I can add or not more tasks if needed
[12:21] <kstenerud> ok
[12:26] <kstenerud> ahaseanck: trusty and xenial work fine
[12:33] <ahasenack> kstenerud: good, thanks for checking
[13:11] <kstenerud> ahasenack: I'm still not clear on what fields are used when in the dep-3 header. They mostly seem to be duplicates of each other
[13:12] <ahasenack> kstenerud: yeah, there can be some confusion, and on top of that you will have reviewers with different opinions
[13:12] <ahasenack> kstenerud: I can see the intention of applied-upstream, but if the patch origin is upstream already, then it's redundant in my opinion.
[13:13] <ahasenack> kstenerud: if the patch is *not* from upstream, but applied upstream, then why wouldn't the origin not be upstream already? Maybe if they were different authors
[13:14] <kstenerud> What is the Origin field for? I see options for upstream, backport, vendor, other, but no description of what any of those mean
[13:15] <ahasenack> upstream is the software author
[13:15] <ahasenack> like samba.org for samba packages, openldap.org for openldap, and so on
[13:15] <ahasenack> backport is if you had to change the patch to fit the particular ubuntu package you are patching
[13:16] <ahasenack> and vendor is if it came from redhat, debian, suse, intel, etc. Not upstream, but other distributors of the software. I rarely use that one
[13:16] <ahasenack> because eventually it gets landed upstream
[13:17] <ahasenack> note also that the presence of one field may make another one optional
[13:17] <kstenerud> So origin will be one of those 4 words, a comma, and then a url?
[13:17] <ahasenack> like author vs origin
[13:17] <ahasenack> yes, word, comma, url
[13:18] <kstenerud> What does it mean when a patch is forwarded?
[13:18] <ahasenack> if you created the fix, for example, if you forwarded it upstream or not
[13:18] <ahasenack> i.e., if you let upstream know about the fix
[13:18] <ahasenack> sometimes a fix only makes sense for ubuntu, for example, in which case Forwarded would be "not-needed" or "no"
[13:19] <kstenerud> so "forwarded" and "bug" serve the same purpose?
[13:20] <ahasenack> there are many ways to forward a patch
[13:20] <ahasenack> sometimes upstream doesn't have a bugtracker, so you forward by email
[13:20] <ahasenack> like via a mailing list
[13:22] <kstenerud> So if you needed to send via email what do you put in?
[13:22] <rbasak> If you got the patch from upstream, then I think just "Origin: upstream, ..." is sufficient.
[13:23] <rbasak> Forwarded is useful for if you wrote the patch and sent it somewhere but hasn't been upstreamed (or if a contributor wrote the patch and the same applies)
[13:23] <ahasenack> kstenerud: I had to do that once, and it wasn't clear either since it's so rare. I think I just put the words "Emailed Foo Bar <foobar>"
[13:23] <rbasak> (plus Bug, Bug-*, Last-Update, etc)
[13:24] <kstenerud> rbasak: So Forwarded has allowed values URL or no
[13:24] <kstenerud> or not-needed
[13:24] <ahasenack> Forwarded: <URL|no|not-needed, useless if you have a Bug field, optional>
[13:24] <ahasenack> it helps to have a saved-up dep3 template
[13:24] <kstenerud> yes I'm looking at the template
[13:25] <ahasenack> so it says URL in there :)
[13:25] <ahasenack> ah, you were making a statement, not a question
[13:25] <ahasenack> n/m
[13:25] <kstenerud> Yes, but if you wrote a patch that hasn't been upstreamed, there wouldn't be a URL to put in...
[13:25] <rbasak> I think you can put whatever you like in Forwarded, except that anything apart from "no" and "not-needed" means "yes" so there are only two ways of saying no.
[13:26] <ahasenack> kstenerud: the url could be to the mailing list archive showing you emailed the list with the patch
[13:26] <rbasak> "The field is really required only if the patch is vendor specific..." -- there you are :)
[13:26] <rbasak> Otherwise you'd have an Origin header.
[13:33] <kstenerud> So for an Ubuntu maintainer, does this make sense? https://pastebin.ubuntu.com/p/Wjx2y34Cst/
[13:33] <kstenerud> I want to update my document so I can remember
[13:37] <ahasenack> kstenerud: bug-<vendor> can also be Bug-Debian, Bug-Fedora, etc
[13:37] <kstenerud> Would we put Bug-Debian? Wouldn't that be for a debian maintainer?
[13:38] <ahasenack> Bug-Debian is super common
[13:38] <smoser> ahasenack: ?
[13:38] <ahasenack> the point of dep-3 headers is to record patch history
[13:38] <ahasenack> smoser: yes?
[13:38] <smoser> i m issed a ping way up
[13:38] <ahasenack> then it's gone :)
[13:39] <kstenerud> oh so that means that there's a debian bug report?
[13:39] <smoser> what was it?
[13:39] <ahasenack> smoser: maybe about the git-ubuntu build{,-source} breakage? We had to revert your branch
[13:39] <ahasenack> kstenerud: yes
[13:40] <smoser> i think that was it, but what was wrong?
[13:40] <ahasenack> kstenerud: if you use dep3changelog to construct the d/changelog message from a patch, it will also record in the d/changelog message the debian "Closes: #nnnn" string
[13:40] <ahasenack> kstenerud: sometimes debian grabs our fixes, and that string tells them that this particular ubuntu upload is also fixing a debian bug
[13:41] <ahasenack> smoser: #1799300
[13:42] <ahasenack> kstenerud: doesn't mean you have to go hunting and searching vendors' bug reports, but sometimes that is recorded in the launchpad bug already
[13:42] <kstenerud> ahasenack: dep3changelog is similar to git-ubuntu.reconstruct-changelog?
[13:42] <ahasenack> kstenerud: yes, but it also checks the syntax of the dep3 header for you, like if you missed a mandatory one
[13:43] <ahasenack> or just have an invalid syntax
[13:48] <sam_w> hi all, a preseed issue: I am loading a preseed config via https which causes certificate verification errors as the busybox installer environment seems to be missing any ca certs. I am aware of the debian-installer/allow_unauthenticated_ssl=true option, but this didn't seem to work as a boot parameter
[13:50] <rbasak> sam_w: how are you booting the installer?
[13:50] <sam_w> are you aware of any way to either d-i preseed/include an http file from a preseed file included in the boot image, or manually add ca certificates to the install environment?
[13:50] <rbasak> sam_w: I ask because the usual ways of doing that aren't secure so https brings little benefit.
[13:50] <sam_w> rbasak: usb flash drive
[13:51] <rbasak> Then you have a reasonable question :)
[13:51] <rbasak> I understand the question, but I don't know the answer, sorry.
[13:51] <rbasak> Are you sure the boot parameter syntax is correct?
[13:51] <rbasak> I was under the impression that any preseed option could be a boot parameter. If not, perhaps that one should be added to the list.
[13:53] <sam_w> rbasak: fairly sure. That was what I was wondering, if it was any or there was some mapping or explicit passthrough
[13:56] <sam_w> from grub.cfg: `linux	/install/vmlinuz noprompt auto=true priority=critical console-setup/ask_detect=false netcfg/choose_interface=auto locale=en_GB debian-installer/allow_unauthenticated_ssl=true url=<snip> quiet ---`
[14:06] <rbasak> sam_w: seems reasonable to me. The next thing to do is to dive into the code I suppose.
[14:06] <rbasak> sam_w: I'd check first that the key/value is correct, but you obviously can't do that using a regular preseed!
[14:08] <sam_w> the only other thing would be: if it was possible to have a preseed file on the iso with that option, and then include one via https
[14:09] <sam_w> but the impression I got from the docs was that preseed/include only works for the same scheme that the file it is in comes from
[14:29] <kstenerud> ahasenack: I was unable to reproduce the fetchmail bug on bionic
[14:31] <ahasenack> kstenerud: but that's where it was reported
[14:32] <ahasenack> kstenerud: and bionic and cosmic have the same exact versions
[14:32] <ahasenack>  fetchmail | 6.3.26-3build1 | bionic  | source, amd64, arm64, armhf, i386, ppc64el, s390x
[14:32] <ahasenack>  fetchmail | 6.3.26-3build1 | cosmic  | source, amd64, arm64, armhf, i386, ppc64el, s390x
[14:32] <ahasenack> you must be using the fixed package by mistake
[14:32] <kstenerud> I'll do a fresh install and see
[14:36] <kstenerud> Nope... Won't trigger on bionic, but triggers on cosmic
[14:37] <ahasenack> is it up-to-date?
[14:37] <ahasenack> apt dist-upgrade wise
[14:37] <kstenerud> yup
[14:37] <kstenerud> and both report the same version of fetchmail
[14:38] <ahasenack> what remains is the ssl version
[14:38] <kstenerud> I basically lxc launch ubuntu:cosmic or ubuntu:bionic and then https://pastebin.ubuntu.com/p/G9xHNGtQ9c/
[14:38] <ahasenack> kstenerud: oh, wait, the reporter was using 18.10, not 18.04
[14:39] <ahasenack> InstallationMedia: Ubuntu 18.04 LTS <-- he originally installed 18.04, but is now on 18.10
[14:39] <kstenerud> ok
[14:39] <ahasenack> still weird though
[14:39] <ahasenack> maybe bionic doesn't support that tls version that this triggers?
[14:39] <ahasenack> what was it, tls 1.2?
[14:39] <kstenerud> maybe. Google only does this weird stuff if you ask for TLS 1.3
[14:40] <ahasenack> can you check the ssl or gnutls library fetchmail is linked to in both cosmic and bionic? use ldd
[14:40] <kstenerud> what args do I use?
[14:40] <ahasenack> ldd <binary>
[14:44] <kstenerud> they're both the same
[14:44] <ahasenack> your test forces the tls version?
[14:47] <kstenerud> --sslproto TLS1.2+
[14:47] <kstenerud> that's as high as it goes in both versions
[14:48] <kstenerud> bionic succeeds, cosmic fails
[14:48] <kstenerud> fetchmail -d0 -vk --sslcertck --sslproto TLS1.2+ pop.gmail.com
[14:51] <smoser> i cant rbasak ping
[14:51] <smoser> so what do you want me to do. fix is this:
[14:51] <smoser>  http://paste.ubuntu.com/p/Vf2RfST58Q/
[14:52] <rbasak> That's fine if it works.
[14:52] <smoser> so just rebase my old branch?
[14:53] <rbasak> Yeah, on origin/master please. Then we can do another CI run and ahasenack can test his use cases from it too, and if all happy we can merge.
[14:53] <smoser> k
[15:04] <ahasenack> kstenerud: there is an openssl difference between bionic and cosmic
[15:04] <ahasenack> kstenerud: bionic has openssl 1.1.0, cosmic has openssl 1.1.1
[15:05] <ahasenack> ubuntu@bionic-fetchmail:~$ dpkg -S /usr/lib/x86_64-linux-gnu/libssl.so.1.1
[15:05] <ahasenack> libssl1.1:amd64: /usr/lib/x86_64-linux-gnu/libssl.so.1.1
[15:05] <ahasenack> ubuntu@bionic-fetchmail:~$ dpkg-query -W libssl1.1
[15:05] <ahasenack> libssl1.1:amd64	1.1.0g-2ubuntu4.1
[15:05] <sdeziel> I think that openssl 1.1.1 on cosmic has support for tls 1.3
[15:06] <ahasenack> it's not just that, I think some default might have changed
[15:06] <ahasenack> I can reproduce the error on cosmic with just this: openssl s_client -connect pop.gmail.com:995 -noservername
[15:07] <ahasenack> with -tls1_3 it doesn't finish the handshake
[15:07] <kstenerud> According to upstream reports it's due to Google's bizarre behavior of passing back a self-signed cert in some circumstances
[15:07] <kstenerud> such as the SNI missing in a 1.3 connection
[15:08] <kstenerud> it downgrades to 1.2+, but also sends back a completely different cert
[15:08] <ahasenack> another thing I'm thinking is that openssl 1.1.0 is setting a default sni, if none is given
[15:08] <ahasenack> there is no -noservername in openssl 1.1.0's s_client command
[15:09] <ahasenack> fetchmail's --sslproto TLS1.2+ means 1.2 *or* newer, not > 1.2
[15:09] <ahasenack> doesn't mean it's negotiating 1.3
[15:10] <ahasenack> and the output of openssl's s_client -tls1_3 suggests that 1.3 is not supported
[15:10] <kstenerud> yeah, not sure what it's actually doing under the hood. That's just the chatter from the upstream bug reports
[15:10] <ahasenack> that being said, using --sslproto TLS1.2 (which asks for 1.2 exactly) works
[15:10] <smoser> qhttps://code.launchpad.net/~smoser/usd-importer/+git/usd-importer/+merge/357826
[15:10] <ahasenack> so ok, let's leave bionic out of it
[15:10] <smoser> ahasenack, rbasak
[15:11] <rbasak> Thanks!
[15:11] <smoser> you can test ust by adding 'usd-importer/bin' to your PATH and running 'git-ubuntu build'
[15:11] <rbasak> ahasenack: once CI has passed, would you mind grabbing the built snap from CI and testing it please?
[15:11] <ahasenack> kstenerud: set the bionic task to invalid and add a comment about these tests you did, saying you couldn't reproduce it there or something lke that, even if the code is affected
[15:11] <rbasak> Or that.
[15:13] <ahasenack> yes
[15:14] <sdeziel> tcpdump would tell you if SNI is used
[15:15] <ahasenack> kstenerud: it might boil down to just the fact that openssl 1.1.1 is the one implementing tls 1.3, and 1.1.0 isn't
[15:15] <ahasenack> hence, bionic not affected
[16:02] <Kabriel> is there a way to setup my ubuntu server to be a middle man for ubuntu updates, such that other machines I have query that server and if the update is not already cached, it retrieves it, otherwise it uses the cached version.
[16:04] <xnox> Kabriel, you can setup transparent squid proxy; and install a client machines to query local net providers over avahi first....
[16:05] <xnox> Kabriel, https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=squid-deb-proxy&searchon=names
[16:05] <xnox> squid-deb-proxy & squid-deb-proxy-client
[16:15] <Kabriel> Thanks for the hint. This seems like a good tutorial: https://fabianlee.org/2018/02/08/ubuntu-a-centralized-apt-package-cache-using-squid-deb-proxy/
[16:16] <Kabriel> It lead me to apt-cacher-ng, which also looks interesting.
[16:25] <UberPope> Hiya folks! I'm on my first attempt to install Ubuntu Server on a refurb. T410
[16:26] <UberPope> The goal is to have a prototype to offer to local clients: Office server, ERP, File server, Ecommerce+WooCommerce, integrated with the ERP on the LAN.
[16:31] <xnox> Kabriel, yeah, apt-cacher-ng is the other one.
[16:32] <xnox> Kabriel, there is also a cloud-mirror proxy, as a juju charm, which is deployed typically in cloud-regions. But it's slightly more heavier to use.
[16:32] <xnox> Kabriel, that one rsyncs dists/, and caches or proxies for the pool/
[16:33] <xnox> Kabriel, or you can run a local ubuntu mirror using ubumirror scripts.... and just point all your clients to your mirror.
[16:33] <xnox> Kabriel, there are many options =)
[16:38] <Kabriel> I have a small setup -- 10 machines all running 16LTS (1 server, rest desktops). Cloud system doesn't sound right, or the mirror. I like the caching idea.
[16:38] <Kabriel> Any experince with squid vs cacher-ng
[16:38] <Kabriel> ?
[16:52] <sdeziel> Kabriel: I've been a happy user of apt-cacher-ng for many years
[16:58] <kstenerud> wow weird... sudo in cosmic always respects -p '', even if I copy the sudo from bionic (which doesn't respect -p '')
[16:58] <kstenerud> so there's some environmental issue maybe...
[16:59] <ahasenack> kstenerud: could be PAM-related, and default config related
[16:59] <ahasenack> kstenerud: the sudo manpage mentions an option about prompt overriding in /etc/sudoers
[17:00] <kstenerud> yeah, already looked in that, and sudoers.d. didn't see anything different
[17:06] <ahasenack> rbasak: dwnloading that snap from jenkins:
[17:06] <ahasenack> git-ubuntu_0+git.30720a7_amd64.snap                    2%[++                                                                                                                    ]   2.53M  63.6KB/s    eta 28m 3s
[17:06] <ahasenack> :(
[17:06] <kstenerud> wow...
[17:09] <mybalzitch> zoom zoom
[17:10] <kstenerud> hmm ok timebox up for sudo. The only ways it's supposed to override the prompt is if passprompt_override is set in sudoers (it isn't), or SUDO_PROMPT env is set (it isn't). It's not a problem with the binary because taking the bionic binary and running it from a cosmic machine works perfectly :/
[17:11] <ahasenack> +1
[17:26] <rawco> hi all
[17:26] <rawco> how’s people’s day going
[17:28] <ahasenack> it's good here
[17:28] <ahasenack> thanks
[17:29] <rawco> so, i’m trying to expand my main partition, for some reason the ubuntu installer created a 4G partition
[17:29] <rawco> and it keeps getting filled
[17:30] <rawco> https://pastebin.com/keXBG0b1
[17:30] <rawco> there’s a bunch of available space on that sdi drive
[17:30] <ahasenack> did you use lvm?
[17:30] <rawco> yes
[17:30] <ahasenack> yeah, known bug :/
[17:31] <rawco> yeah, i did have some problems when installing, had to test a couple of installer isos
[17:31] <ahasenack> https://bugs.launchpad.net/subiquity/+bug/1785321
[17:32] <rawco> yep, das it
[17:32] <rawco> so, i was wondering if i can do the expanding of the volume online
[17:32] <rawco> with growpart and resize2fs
[17:33] <rbasak> See comment 2 there in that bug
[17:33] <rbasak> lvresize has a --resizefs option
[17:34] <rbasak> Saves a call to resize2fs, though that's more useful when shrinking rather than expanding
[17:34] <rawco> rbasak: thanks, i’ll read over the bug page
[17:34] <rbasak> You can increase ext4 size online, so it should be straightforward. Note that shrinking can only be done offline, which is more of a pain for a root filesystem.
[17:35] <jelly> not using the space is a lot better bug than debian's default of "using everything, the whole VG, for last created LV and filesystem, leaving no space at all for snapshots or resizing"
[17:37] <rawco> rbasak: all done: /dev/mapper/ubuntu--vg-ubuntu--lv  108G  3.3G  100G   4% /
[17:37] <rawco> thank’s to everyone :D
[17:39] <jelly> looking at that bug report, this is in fact exactly how I'd want the "use entire disk for LVM" to work in Debian ;-)
[17:40] <lotus|NUC> rawco: can you still recall wich iso you used for install?
[17:40] <rawco> lotus|NUC: sorry, i don’t really remember what iso i used
[17:41] <rawco> i think i had to use the 18.04 iso, because 18.04.1 iso was not working with my hardware setup
[17:41] <rawco> it was a couple of months ago, sorry :(
[17:41] <rawco> i thought it was me and not the iso lol
[17:41] <rawco> so i just ignored
[17:41] <lotus|NUC> rawco: yeah might be relevant info for the channel here
[17:42] <rawco> i will lurk more here, since ya’ll are awesome
[17:43] <lotus|NUC> i have a gf already :p
[17:46] <rbasak> ahasenack: I can grab historical git-ubuntu snap binaries for you if it would help
[17:46] <ahasenack> rbasak: do you still have 439 installed? Should be trivial to reproduce the bug. kstenerud or do you have it perhaps?
[17:47] <rbasak> I'm on 440
[17:47] <jelly> ahasenack, the mind boggles, why is this a bug!  This is precisely how "use whole disk for LVM" ought to work -- PV indeed uses whole disk (apart from /boot partition)
[17:47] <rbasak> I might be able to revert.
[17:48] <ahasenack> jelly: it was unexpected, or at least not clear enough that this would happen. Some people were surprised to get "disk full errors" after installing a few more packages
[17:48] <ahasenack> at least expanding is easier than shrinking
[17:51] <jelly> it's a lot better than what d-i does.  expanding is a fully online process.  shrinking of xfs is impossible, shrinking of ext4 is offline (and unoptimized, up to 4 times slower than copying, reformatting and copying back the data if there's more than 25-50% space used)
[17:51] <rbasak> ahasenack: http://people.canonical.com/~rbasak/VAGSRAriUyDDlqsLunShJTe7503Uw4GF_439.snap.zsync and http://people.canonical.com/~rbasak/VAGSRAriUyDDlqsLunShJTe7503Uw4GF_439.snap
[17:52] <jelly> no functional change seems required, just document things and maybe put up a notification
[18:07] <rawco> what do ya’ll use for monitoring your servers
[18:07] <rawco> ELK stack?
[18:18] <ahasenack> kstenerud: remember to create a card for fetchmail, if you haven't already (I didn't find it after a quick look)
[18:23] <ahasenack> depends on how many servers, and if you have a raspberry pi3 or a 16Gb machine for monitoring :)
[18:24] <ahasenack> elk is heavy
[18:30] <rawco> i have a nice hp proliant server with sas drives and bells+whistles
[18:30] <rawco> all the gigs
[18:32] <ahasenack> grafana is pretty for the graphs
[18:32] <ahasenack> negios (or its replacement, forgot the name) is good for alerts
[18:33] <nacc_> ahasenack: icinga
[18:33] <ahasenack> that one
[18:33] <nacc_> (icinga2 i think technically)
[18:38] <sarnold> there's too many choices
[18:38] <sarnold> if there were one that sucked but it was the only one available, it'd still be the obvious choice
[18:38] <sarnold> but there's dozens :)
[19:10] <rawco> well, what do you use sarnold
[19:12] <ahasenack> I use munin on a small server, but I'm not very happy with it. I think that machine can take more. It only has 3Gb of ram and runs zfs, and that's stretching it already according to docs, but real world usage shows it has some memory free
[19:12] <ahasenack> Mem:          3.2Gi       2.1Gi       166Mi       2.0Mi       913Mi       890Mi
[19:12] <sarnold> rawco: I'm currently suffering from analysis paralysis -- where I use nothing because I can't decide what to do :(
[19:12] <rawco> sarnold: that’s my current mood lmao
[19:13] <rawco> we’re already paying for connectwise, but it’s trash for monitoring
[19:15] <teward> sarnold: Landscape.  *shot*
[19:15] <teward> (just kidding)
[19:15] <teward> sarnold: analysis paralysis is bad.  :P
[19:15] <sarnold> teward: tell me about it..
[19:22] <rawco> dehumanizing, i would say
[19:22] <rawco> i wanna surveil this goddam servers
[19:22] <rawco> 24/7
[19:24] <sdeziel> Nagios3 serves us well but we don't have a huge park (~200 machines with 2k service checks)
[19:25] <sdeziel> the webUI makes your eyes bleed so we use check-mk-multisite instead
[19:26] <sdeziel> munin is for collecting performance data (no alerting capabilities that I'm aware). For perf data and some alerting netdata is pretty nice and comes with a nice webUI
[19:26] <rawco> sdeziel: that makes sense, monitoring != performance data
[19:26] <rawco> i wonder is there’s anything out there that does everything + looks nice
[19:27] <sdeziel> rawco: well, with nagios3 we also collect perf_data for quick graphs
[19:29] <rawco> splunk ony collects logs/files and graphs them , right?
[19:29] <rawco> no actual “monitoring"
[19:36] <shubjero> rawco: zabbix, elk, grafana
[19:43] <rawco> thanks shubjero
[19:56] <waveform> actually munin does have some rudimentary alert facilities but they're not configured by default (or rather, they're configured to report via nagios by default on ubuntu - but they can be configured to report directly via e-mail)
[19:56] <waveform> here we go: https://munin.readthedocs.io/en/latest/tutorial/alert.html
[19:57] <sdeziel> good to know, thanks waveform
[19:59] <shubjero> rawco: zabbix for active monitoring of hardware and os metrics. ELK for massive log aggregation. Grafana helps fill some gaps with zabbix for us
[19:59] <shubjero> rawco: so on any server we monitor we would have a zabbix-agent and a filebeat client running