[01:10] good morning to all [01:46] leftyfb: oO ;p [01:46] you want him? [01:47] nah [01:47] got my own issues I'm dealing with [01:47] what you playin with leftyfb [01:48] main server was down for 2 hours(gotta fix monitoring). It was rebooted (don't know why) and I just finished fixing why it came back up without network(stupid -modules packages) [01:48] ouch [02:00] what server Os leftyfb [02:00] 16.04.5 [02:00] found something in the logs? [02:00] I'm not finding anything [02:00] nope [02:01] auth.log, syslog, kern.log, last [02:02] what kind of server is this? [02:02] PowerEdge R310 [02:02] Dell [02:02] i guess it'll have some more logs then [02:03] what do you mean? [02:03] the remote out of band managment stuff, can't remember dell's name for it right now [02:03] it probbaly provides some hardware monitoring log [02:03] ipmi/iDrac [02:03] right [02:04] idrac is what i meant [02:04] sel log [02:04] exactly, nothing there either? [02:04] looking ... [02:04] debug logs? [02:04] I started looking for the exact term the moment you started mentioning it :) [02:04] or the last command can help? [02:04] nope [02:04] nothing there, only "reboot" [02:04] :-/ [02:05] nothing in SEL [02:05] might be better to switch it to not reboot automatically on failure next time [02:05] ..so you can see whats on the console from idrac [02:06] you think a failure caused it to properly reboot(ACPI)? [02:07] well if trhere's nothing in the logs, it sounds like a hardware issue, right? or a kernel issue which could not get logged in time [02:07] reboot system boot 4.15.0-29-generic Sat Aug 4 18:17:43 2018 - Sat Aug 4 20:51:28 2018 (02:33) [02:07] i think there's an optio in most bioses/uefis to stop the system from getting rebooted / power cycled on failures. it's normalöly enabled. i'm suggesting you might want to disable it. [02:07] that's all I've got [02:08] leftyfb: ups could rebooted it? [02:09] if logs doesnt show anything, we should look external things? [02:09] I don't have any of the apc tools installed [02:09] hmm [02:10] leftyfb: can you trigger var log messages from the day? [02:11] morning guiverc_d [02:11] trigger? Also, /var/log/messages hasn't been a thing in years. You mean syslog/auth.log [02:12] yeah [02:13] https://pastebin.ubuntu.com/p/MzqjQ4rTMp/ [02:13] that's from syslog [02:13] lets c [02:14] reboot happened around 18:17 or so .. closer to 18:14 I think [02:15] leftyfb: what kind of service your server running/purpose? [02:15] apache2 on the host and in a couple lxc containers [02:15] should be stable that [02:15] yep [02:16] been doing this stuff for over 20 years :) [02:17] iDrac has a nice "Last Crash Screen" feature, but nothing was captured. I don't think it was a crash though [02:18] leftyfb: ssh on it? use anti bruteforce? [02:18] yep and yep [02:18] checked those attempts yet? [02:19] on the time of shutdown [02:19] root has a randomly generated 23 character password that I don't know (on purpose) only accessible via ssh keys from 2 ip address's. I have fail2ban on. ssh running on a non-standard port [02:21] check anyway :p [02:21] you never know they got in [02:21] oh this is cool as hell [02:21] iDrac saves a recording of the last 3 boot sequences [02:21] aha [02:22] well, some may like it, others may hate it [02:22] there's also https://help.ubuntu.com/lts/serverguide/kernel-crash-dump.html [02:22] but i assume you'll know [02:22] if someone "got in" to ssh via a non-standard port using my private key and somehow spoofed the 2 ip address's that's allowed, one of which is local, there's no way in hell they left any sort of tracks [02:22] the @ characters in your log make it look like a kernel crash [02:23] tomreyn: that's what I was thinking. [02:24] oh man, it shows me logging in to the local console and troubleshooting for a bit. Cool as shit [02:25] scary as shit :P [02:25] I love it [02:25] it's a dedicated interface on my LAN, not accessible to the outside [02:26] different VLANs [02:28] that's probably fine. i just don't like the idea of having the host OS effectively run in a sandbox with lots of security cameras and some other operating systems around it, where you have no idea what their limits are, how they work, and how they may be susceptible to compromise, too. [02:28] I'm partial to OOB BMC's [02:29] !info watchdog [02:29] watchdog (source: watchdog): system health checker and software/hardware watchdog handler. In component universe, is extra. Version 5.15-2 (bionic), package size 81 kB, installed size 263 kB [02:29] it was sort of my thing when I worked at Canonical :) [02:29] could be usefull too, but not sure if it can go in the past [02:30] i like to have OOBs, i just want them to be well defined, well designed, and most of all sully open source and fully documented. [02:30] I think iDrac meets most, if not all of those [02:30] kk [02:30] especially since it fully complies with ipmi [02:30] luckily we'll likely get this in the years to come. [02:31] idrac is proprietary, not open source, not documented (in terms of hardware and software), afaik. [02:31] i like lynis system scan too, but thats if you a bit more paranoid :p [02:31] well, libre software was not on my requirements list. [02:37] I really gotta setup another nagios server [02:38] another day ... gonna call it a night [02:40] nite leftyfb [02:50] Bashing-om: next time, donate to lotus computers ok? https://www.omgubuntu.co.uk/2018/08/elementary-os-hires-cassidy-james [02:50] pm for my bank account [02:50] lotuspsychje: looking. [02:50] lol [02:55] lotuspsychje: Must be a trend started ... as the GNOME Foundation aslo recently got a donation that enabled paid staffing . [02:55] yeah [02:55] next: lotus computers :p [02:56] i understand when you retire, you dont know what to do with all your money :p [02:57] lotuspsychje: I smell a fallacy somewhere :P [02:57] haha [02:59] I would be greatly pleased with a small donation of a Ryzen7 box :P [02:59] yeah sure, how about 10 with million $ [03:00] lotuspsychje: Well, yeah ..if ya gonna wish .... wish *BIG* . [03:00] to be able to help volunteers, we need a good machine right?? [03:01] * lotuspsychje hope that donate guy reads the ubuntu logs [03:02] lotuspsychje: Consider how much smoother my work flow would be with modern hardware ... now that would be support :) [03:03] yes! [03:37] wb pragmaticenigma [03:37] alo [03:37] been going slow [03:38] that'd be a nice change of pace [06:20] good morning [10:22] good afternoon to all [10:31] good morning lotuspsychje ;) [10:31] you just joined a few mins before i did ;) [10:31] hey tomreyn had a nice night :p [10:31] err after [10:31] yes, full 6 hours [10:31] too hot again? [11:14] sorry, missed your response, LP. not too hot, maybe too much daylight ;) [11:14] no sweat [11:16] not yet [11:16] haha [12:28] Hi folks [12:29] hey there BluesKaj [12:37] hey lotuspsychje [13:07] Hita BluesKaj . Great morning to you and yours! [13:08] 'Morning JimBuntu, thanks...the same to you and yours :-) [13:32] hey pragmaticenigma [13:33] hey [13:34] pragmaticenigma: support steady today [13:35] only on for a moment [13:35] sure tyt, have a coffee :p [13:35] can't ... got places to be, things to do [13:38] I do not understand how someone things a support channel is the appropriate place to talk about their misives on flash [13:39] pragmaticenigma: Hi folks. Anyone any hint why Flash Plyaer won't work on my ubuntu 18.04, neither in Firefox nor Chromium web browser ? [13:39] was his origin question [13:40] perhaps, but the commentary after wasn't OT [13:40] true [13:41] ppl think when flash works in windows, everything is ok [13:41] Also, flash is not insecure, regular updates are provided when vulnerabilities are found. Personally, I don't have it installed on my machines, only because adobe is pulling the plug 2020 (unless they're like microsoft and extend it another X years while companies finally remove it from their sites) [13:41] If I desperately need flash for something, I have google chrome installed, which includes their sandboxed implementation [13:45] flash is widely abused [13:46] but hey alot of apps got 0day exploits [13:47] it is, but to out-right dismiss it as vulnerable is spreading FUD in my mind. I have only ever been stung by flash once, and it was my own doing. I honestly believe a person has to really try to find a way to get their computer compromised these days. [13:48] i dont agree with that [13:50] windows boxes are like full of malware just by browsing the web [13:51] i always warn users for using bad protocols [13:52] vnc,ftp,flash,.. [13:52] ssh when not protected gets hammered 24/7 also [13:53] I still call FUD on that... I have several windows boxes that I maintain for family. They only run Window's Defender full time, and I run other programs once a month. These machines are more or less vanilla installs with updates installed regularly. [13:53] its not because its updated, they are not infected [13:53] i would call it reality instead of fud [13:54] lotuspsychje: you just made the claim that owning a windows machine is vulnerable [13:54] pragmaticenigma: depends from the owner [13:54] but alot of owners are infected yes [13:54] that's not a fault of the machine if the owner is doing bad things. computer do as they're told [13:55] they're not sentient... yet [13:56] the owners choice is also the Os [13:57] scan any windows pc with malwarebytes and youl get 500 hits [13:57] unless the owner maintains his Os a bit [13:58] I don't agree with that generalization [13:58] then there are rootkits, virusses,browser exploits and scanners 24/7 [13:58] the world is one big scanning warzone, thats reality mate [13:58] not a general statement [14:02] same for android..playstore is a joke full of malware [14:03] ive been warning my customers dont use pc banking on android [14:04] okay, i'm out ... this conversation is fruitless and really taking a turn for the worse [14:04] later ;) [14:05] leftyfb: server back in business? [14:05] it was last night, pretty quickly. I was just trying to find out the "why" [14:05] kk [14:37] I cringe thinking of all the packages someone_ has installed from ppa's, manually or from source. [14:37] :) [14:38] and there we go [14:42] Arch is still the best server os. It's just minimal and you know it's flaws [14:43] mwsb: It's nice to have an opinion isn't it? [14:45] It's a fact. daftykins knows [14:45] mwsb: incorrect [14:48] xD [17:14] leftyfb I solved the problem, I have downloaded all .deb files from pkgs.org and after I installed them to take place of oibaf files, I executed sudo apt -f install and that is it and everything is perfect now ." [17:14] * leftyfb sigh [17:14] some people just refuse to listen/learn and are just hell bent on making a mess [17:15] xD [20:21] BenderRodriguez: connection issues? [20:21] time to sort 'em :) [20:42] not happening [20:46] hmm? [20:51] its ok-ish, he got kicked, but only after he bragged about how he rickrolled folks, follewed by much discussion in #ubuntu, followed by more discussion in #ubunutu-ops. so he was able to deal his damge, which trolls consider winning. like he did twice yesterday, and the day besfore. [20:51] this is the guy above? or someone else [20:52] no, no. its this person now called 'rekt' in #ubuntu [20:53] previously rooik [20:55] ah [20:57] the only way to deal with such classic trolls is to kickban them immediately, leaving an explanatory (ban) message for the rest of the channel. since only then they can't do harm, can't disrupt, can't score in what they consider to be a game. [20:58] *shrug* glad to not encounter such anymore [20:58] how do you do that? got his isp on ignore? [20:59] i'm not in that channel [21:00] oh :) a hideous approach to solve this issue ;) [21:01] oh it's not from choice :) [21:01] this is why i often ask what's going on in there, can't see for myself [21:04] i see