| === Sveta_ is now known as svetlana | ||
| === svetlana is now known as sveta | ||
| snake-venom | hi found this on my server, | 05:49 |
|---|---|---|
| snake-venom | is this attacker script http://termbin.com/x3iy | 05:49 |
| snake-venom | ? | 05:49 |
| andol | That shell script makes my eyes hurt. | 06:16 |
| lordievader | Good morning | 06:59 |
| siavoshkc | Good morning, I have a https://docs.djangoproject.com/en/2.2/howto/deployment/wsgi/modwsgi/#basic-configuration for my django server. But with this config server doesn't even start | 07:13 |
| technoob | Hey guys | 07:44 |
| technoob | I need help | 07:45 |
| technoob | I already installed ubuntu server on my nuc and want to know what the end result looks like so i know it is already fully booted up | 07:45 |
| lordievader | How do you mean? | 07:48 |
| lordievader | What you see on the screen? | 07:48 |
| lotuspsychje | technoob: and what kind of NUC is it exactly? | 07:49 |
| technoob | Nuc5cpyh | 07:49 |
| lotuspsychje | technoob: did you update bios firmware to latest version? | 07:50 |
| technoob | YES | 07:50 |
| lotuspsychje | technoob: explain whats happening please as lordievader asked | 07:51 |
| technoob | I see a command prompt | 07:51 |
| technoob | But nothing else | 07:51 |
| technoob | I see just text | 07:51 |
| technoob | No username input prompt | 07:51 |
| technoob | Last line of text is "reached target cloud-init target" | 07:52 |
| technoob | lordievader | 07:58 |
| lordievader | The command prompt is all you get on a server install. A server shouldn't need more | 08:02 |
| technoob | lordievader yes but what should i all booted up server look like | 08:12 |
| technoob | I hooked up my nuc on my tv | 08:13 |
| technoob | What should i be seeing there | 08:13 |
| lordievader | On the screen you'll only see an tty login-prompt. | 08:22 |
| lordievader | If you want to know if all the services started correctly you can issue `sudo systemctl status`. | 08:22 |
| technoob | I didnt see any tty login prompt | 08:27 |
| technoob | Should i reboit? | 08:27 |
| technoob | lordievader | 08:27 |
| lordievader | Could you maybe share a screenshot? | 08:27 |
| technoob | Kinda hard im on mobile | 08:28 |
| lordievader | Photo? | 08:28 |
| technoob | Do we have a way to attach photo to this irc? | 08:34 |
| lotuspsychje | technoob: imgur.com | 08:34 |
| CarlFK | riot.im - takes some work to get set up, but then you have a nice way to irc from your phone | 08:36 |
| technoob | Ok in a minute | 08:37 |
| technoob | I rebooted my machine | 08:37 |
| technoob | Ill see if the problem still persist | 08:37 |
| technoob | Im running a celeron processor btw | 08:37 |
| technoob | So it might be slow | 08:38 |
| technoob | Also i just wanna ask | 08:40 |
| technoob | Can i run kodi in a ubuntu server? | 08:41 |
| technoob | Ubuntu server is like ubuntu desktop without gui right? | 08:41 |
| kstenerud | ubuntu server is a linux install without gui. You can technically install a gui on top of it, but if you want a gui, you'll have an easier time using the desktop flavor | 08:43 |
| technoob | I see | 08:43 |
| technoob | Can i run nextcloud server and other server apps on the same machine | 08:44 |
| kstenerud | sure | 08:44 |
| lordievader | Essentially, you can run whatever you want on the machine 😉 | 08:45 |
| kstenerud | generally people use docker to keep things nicely separated | 08:45 |
| kstenerud | It's not strictly necessary, but it keep things clean | 08:46 |
| technoob | I cant seem to launch kodi | 08:49 |
| technoob | It says that it cant find kodi.services | 08:49 |
| siavoshkc | Should www-data have access to any folder apache2 wants to access? | 08:50 |
| technoob | Systemctl start kodi. Thats the command i use | 08:50 |
| lordievader | technoob: Kodi is installed? And that is the correct service name? | 08:50 |
| technoob | I guess | 08:50 |
| technoob | I was following a instruction | 08:51 |
| technoob | In the net | 08:51 |
| lordievader | Check it then, investigate 😉 | 08:51 |
| technoob | .. | 08:52 |
| technoob | Sadface | 08:52 |
| technoob | Hmm might as well just use ubuntu desktop. Will be easier i guess | 08:53 |
| technoob | I only have like 4 gb ram though | 08:53 |
| lordievader | If you don't open too many browser tabs that is fine 😉 | 09:16 |
| kstenerud | Does anyone have experience setting up an l2tp vpn client on a command line lxd container? I tried following https://github.com/hwdsl2/setup-ipsec-vpn/blob/master/docs/clients.md#configure-linux-vpn-clients-using-the-command-line but it fails with "device not found" when looking or ppp0 | 10:34 |
| === jelly-home is now known as jelly | ||
| ahasenack | rbasak: I have to update pmdk, which, when it was created, didn't exist in debian, so its version is of the 0ubuntuN style | 11:30 |
| ahasenack | rbasak: now debian has one | 11:30 |
| ahasenack | rbasak: but git-ubuntu can't find a common ancestor to start a merge, which is expected I guess | 11:30 |
| ahasenack | rbasak: any particular way I should handle this? | 11:30 |
| rbasak | ahasenack: I guess you have a choice. Do you want to converge with Debian? | 11:30 |
| ahasenack | what does that mean? sync? | 11:31 |
| ahasenack | sync and then reapply our changes, if any remain? | 11:31 |
| rbasak | Do you want the new Ubuntu package to be based on the Debian one, plus a delta? | 11:31 |
| ahasenack | yes | 11:31 |
| ahasenack | no reason not to | 11:31 |
| rbasak | Depends on the delta I guess :) | 11:31 |
| ahasenack | unless the delta is huge, I guess | 11:31 |
| ahasenack | ok, let's say I want to | 11:31 |
| rbasak | I don't think there's any easy way of handling this. Fundamentally the "rebase" workflow doesn't work because we don't have a patchset to begin with. | 11:32 |
| ahasenack | the pmdk story has "no easy way" written all over it :) | 11:32 |
| rbasak | You could try taking the entire diff between Debian and Ubuntu currently, placing it in the working tree, and see if you can break that up into a logical delta with "git add -p". | 11:32 |
| rbasak | Probably limited to just the debian/ directory. | 11:33 |
| ahasenack | rbasak: that's ok, but what will make it have a common ancestor from now on, for future merges? | 11:37 |
| elfranne | any idea why Kerberos is ignoring my logging settings from krb5.conf ? | 11:39 |
| lotuspsychje | anyone knows if server got unattended upgrades enabled by default now? | 13:12 |
| blackflow | it doesn't | 13:13 |
| Ussat | which is a good thing | 13:17 |
| blackflow | definitely. | 13:18 |
| rbasak | I don't think it's such a bad thing nowadays. Server users should take _some_ action to ensure security updates. Taking no action is really bad. So if that action is to change the default to some other mechanism, or enable unattended-upgrades, what's the difference? | 13:40 |
| rbasak | I don't think it's such a bad thing nowadays. Server users should take _some_ action [as/if necessary] to ensure security updates. Taking no action is really bad. So if that action is to change the default to some other mechanism, or enable unattended-upgrades, what's the difference? | 13:41 |
| tomreyn | If you ask the average SMB boss what they're more worried about, production breaking due to broken patches, or software getting outdated but generally working (until things get exploited, but even exploits try not to break production), they'll surely answer the first. | 13:46 |
| tomreyn | maybe this culture will change a little to the better in the coming years when we get better laws, but it can take a good while longer. | 13:48 |
| Ussat | not sure what laws have to do with it, but having a server automated upgrade and break shit is not good | 13:52 |
| tomreyn | changing the default right now will cause those organizations' admins' taking the blame for production breaking due to bugs such as 1813873 (a regression unfixed in bionic after 1.5 weeks) or 1814555, and cause a backslash, making them even more worried about patching, if they are given time to care about that at all. | 13:53 |
| Ussat | I have set patch times, test first and then prod | 13:53 |
| tomreyn | Ussat: laws to impose fines on companies whose infrastructure is exploited, data stolen. so far the loss is mostly to users privacy, somewhat to companies' brands, but its still soemwhat cheap to them. | 13:54 |
| tomreyn | but those are politics, OT here - sorry. | 13:55 |
| Ussat | um...I am 100% against those laws. Can you gaurentee me as an admin you catch EVERYTHING ? | 13:55 |
| Ussat | all the time | 13:55 |
| Ussat | negligence sure....... | 13:55 |
| rbasak | tomreyn: that's a fair point. | 13:58 |
| rbasak | OTOH, although a large number of users deliberately don't want security updates, AIUI a large majority of those actually end up doing nothing else to get security updates. | 13:59 |
| Ussat | auto updates on desktops are one thing, servers no....thats a recipie for problems | 14:00 |
| Ussat | why not just auto update from 16 --> 18 then ? | 14:00 |
| rbasak | Ussat: my point is that auto updates are better on servers than no security updates at all. That's also a recipe for problems. | 14:00 |
| rbasak | If you want to do something special, then you can do that special thing and disable automatic updates while you're there. | 14:00 |
| Ussat | I disagree, not your job. | 14:01 |
| rbasak | If you're making some special provision anyway, then disabling automatic updates while you're there is hardly onerous. | 14:01 |
| Ussat | Again, why not auto update from 16 --> 18 then | 14:01 |
| rbasak | 16 -> 18 deliberately changes behaviour. Clearly that's unacceptable to do automatically. | 14:02 |
| blackflow | rbasak: problem with unattended upgrades is that it only does half the job, sometimes breaks stuff, and.... only does half the job but people think it does everything. | 14:02 |
| Ussat | ^^^ | 14:02 |
| rbasak | blackflow: in what way does it only do half the job? | 14:02 |
| rbasak | Is there a bug on that? | 14:02 |
| tomreyn | you need to restart services, sometimes reboot | 14:03 |
| blackflow | well for starters it doesn't reboot for new kernels. second, it doesn't restart properly when systemd or dbus is upgraded. third, not all updates are solved by systemctl restart, and not all services have post inst steps to restart. | 14:03 |
| blackflow | there's no bug. it just can't be done. reboots and all. needs human intervention. | 14:03 |
| Ussat | yup | 14:03 |
| rbasak | It does tell you human intervention is required though, with the reboot-required flag and in the motd. I hope. | 14:03 |
| rbasak | But regardless I don't think that's a justification for not turning it on by default. | 14:04 |
| tomreyn | only if a reboot is required | 14:04 |
| blackflow | who's "you"? it's unattended, remember? | 14:04 |
| rbasak | It's clearly better than the current situation. | 14:04 |
| rbasak | (in terms of landing updates) | 14:04 |
| rbasak | Perfect is the enemy of the good, etc. | 14:04 |
| blackflow | no. halfa job is worse than no job at all (so users know they have to run updates). way bettewr would be to have apticron installed so root is mailed/nagged with updates. | 14:04 |
| rbasak | That's fine to say in theory. In practice most users don't run updates at all. | 14:05 |
| blackflow | that's their problem. | 14:05 |
| Ussat | I disagree.......most SERVER users do | 14:05 |
| rbasak | The users who do and don't want automatic updates are competent enough to know and turn it off. | 14:05 |
| Ussat | Desktop is one thing, servers are a whole different ball game | 14:05 |
| Ussat | So, how woudl you schedule these updates, random times ? how would you determine they do not interfear with what is happening on the server | 14:06 |
| rbasak | I just checked. Cloud images do have unattended-upgrades enabled by default AFAICT. | 14:06 |
| blackflow | btw, when ext4 begets case insensitive filenames, will that be flipped on by default, to appease the windows migrants? | 14:07 |
| rbasak | Ussat: there is a default. If you don't like the default, you can change your configuration. Just like everything else. | 14:07 |
| rbasak | blackflow: how is that relevant? | 14:07 |
| Ussat | Yes, they default should be off | 14:07 |
| rbasak | Look, I understand your opinion. | 14:07 |
| rbasak | If you want to just keep restating it instead of actually providing some reasonable rational justfication, then what's the point of continuing this discussion? | 14:08 |
| siavoshkc | I am in state of panic | 14:08 |
| Ussat | I have provided several | 14:08 |
| blackflow | rbasak: well like I said, nagging via apticron or something, is WAY better than shoving updates by default. | 14:08 |
| siavoshkc | I accidentally deleted public_html foleder of a server | 14:08 |
| Ussat | How do you schedule the updates ? How would you determine when they happen | 14:09 |
| rbasak | AIUI, unattended-upgrades _are_ now enabled by default. | 14:09 |
| siavoshkc | Is there any way I can restore it? | 14:09 |
| blackflow | and they should _not_ be | 14:09 |
| Ussat | restore backup | 14:09 |
| blackflow | siavoshkc: yes, from backups. | 14:09 |
| rbasak | If you want to change the default, take it up with Ubuntu developers using our normal governance structure. | 14:09 |
| blackflow | yeah, windmills :) | 14:09 |
| siavoshkc | blackflow: can you please lead me on where to start? | 14:10 |
| blackflow | siavoshkc: well, you do have backups, no? | 14:11 |
| siavoshkc | blackflow: I dont know | 14:11 |
| blackflow | then you don't | 14:11 |
| tomreyn | rbasak: i enjoy this discussion, and actually i'm really on your side there, increasing the overall security by more secure defaults is a good thing. i'm just not sure whether the world is ready for it, yet, or whether it'd just backfire. if unattended security patching is really active on cloud images and this hasn't backfired, yet, this is probably a good indicator. | 14:11 |
| rbasak | Thanks :) | 14:12 |
| blackflow | siavoshkc: sorry for your loss. but hey, consider it a great learning experience: always have backups. consider also a filesystem like btrfs or zfs where you can snapshot data before any potentially breaking intervention is done ;) | 14:12 |
| tomreyn | (cloud folks are usually more forgiving and prepared for things breaking suddenly, though) | 14:12 |
| siavoshkc | blackflow: The problem is that it wasnt my system. I entered a path to delete that wasnt my bussiness. I am worrid about the owner of the site | 14:13 |
| siavoshkc | the site was up and I just deleted it | 14:13 |
| Ussat | sudo apt-get -y remove unattended-upgrades solvs it anyway | 14:14 |
| ahasenack | Ussat: fwiw, the scheduling, and many more knobs to tweak, are available in the config file in /etc/apt/apt.conf.d/ | 14:15 |
| ahasenack | I think it's 50unattended-upgrades | 14:15 |
| Ussat | ahasenack, Oh I know | 14:15 |
| siavoshkc | blackflow: I remember there was an automated snapshot mechanism | 14:15 |
| siavoshkc | can it be used to restore data? | 14:15 |
| blackflow | siavoshkc: perhaps you can use some undelete tool to try and salvage data, but that would require you to bring down the server, or pull out a disk from mirror, as every second the disk is in use reduces the chance for data recovery. | 14:15 |
| Ussat | but if you remove that, its done deal | 14:15 |
| ahasenack | sure | 14:15 |
| blackflow | siavoshkc: there's no such thing unless you're running ZFS or btrfs (and even then I don't know bout "automated" unless you've set something up | 14:15 |
| Ussat | You there are data recovery tools/services, BUT you need to ensure the data is not overwritten asap | 14:16 |
| Ussat | and its crap shoot | 14:17 |
| Ussat | and not cheap as for the services | 14:17 |
| ahasenack | yeah, and there are tools you can try. It depends if you can take the machine offline, remove the hd, and mount it elsewhere where you can poke at it | 14:19 |
| blackflow | offline and/or remove the hdd | 14:21 |
| Ussat | mount it RO....and again, needs to be done asap to avoid overwriting it | 14:22 |
| siavoshkc | If by any chance there is an automated backup. where may I find backups? | 14:26 |
| Ussat | Could be anywhere depending on how its done/software etc | 14:28 |
| Ussat | OUr backups are done nightly to a SAN then we have tape taken offsite | 14:28 |
| Ussat | so ya...could be anywhere | 14:29 |
| blackflow | btw, side question. I have an enterprisey SSD here, supposedly sata3.1 @ 6Gbps, but a dd test shows only 420 tps and ~250MB/s in iostat.... sounds.... inadequate. | 14:29 |
| lotuspsychje | blackflow: did you try the hdparm speedtest? | 14:30 |
| lotuspsychje | hdparm -tT /dev/sda something like this | 14:31 |
| blackflow | that's next, I'm clearing out the SSD before I blkdiscard it | 14:32 |
| lotuspsychje | blackflow: firmware to latest can also improve stuff | 14:32 |
| lotuspsychje | depending wich brand | 14:33 |
| blackflow | Samsung Evo 840 something something | 14:34 |
| sdeziel | siavoshkc: maybe the machine owner has backups you could ask to be restored | 14:34 |
| lotuspsychje | evo 840 needs firmware improvement patch blackflow | 14:34 |
| lotuspsychje | there's a known speed problem on it | 14:34 |
| tomreyn | Ussat: check your cron jobs for anything which looks like it coud do backups, see if you have a process running which sounds liek a backup systems' agent. | 14:39 |
| tomreyn | blackflow: evo is consumer, also | 14:40 |
| blackflow | it's already latest firmware | 14:40 |
| Ussat | tomaw, ? think you meant that for siavoshkc | 14:41 |
| blackflow | tomreyn: technically, it's MZ7GE240HMGR but any info I managed to find on it is that it's a rebranded 840 evo | 14:41 |
| tomreyn | Ussat: you're right, thanks / sorry. | 14:41 |
| tomreyn | siavoshkc: check your cron jobs for anything which looks like it coud do backups, see if you have a process running which sounds liek a backup systems' agent. | 14:41 |
| Ussat | tomaw, NP | 14:42 |
| blackflow | tomreyn: 845DC EVO | 14:42 |
| tomreyn | blackflow: probably some slight firmware changes, maybe some extra spare cells. | 14:43 |
| tomreyn | does fstrim -v report changes if you run it twice? | 14:47 |
| tomreyn | the 840ies can get really slow if you dont trim. | 14:48 |
| tomreyn | (or not successfully) | 14:48 |
| blackflow | I blkdiscarded the entire drive before testing | 14:51 |
| tomreyn | i see | 14:52 |
| cognitiaclaeves | I have a python app, test-logger.py, that I've written to write a test log message to syslog. For only that type of message, I'd like to save the log message in a file, and send the same log message out to a log aggregator (loggly) -- I'm having trouble identifying what to filter the message on based on the journalctl output. https://pastebin.com/hGip2Zz2 | 15:22 |
| cognitiaclaeves | I think this can be done by creating two filters, one to save the log entry to a file, and a duplicate target to forward the messages to loggly. | 15:23 |
| cognitiaclaeves | blackflow, is that enough detail? | 15:24 |
| blackflow | cognitiaclaeves: not quite, it's not clear what exactly you're talking about. "save the log message in a file", "what to filter"..... are you talking about rsyslog rules? | 15:25 |
| blackflow | and how are you sending to loggly? | 15:25 |
| cognitiaclaeves | Yes, rsyslog | 15:25 |
| cognitiaclaeves | Loggly has a page for that. If I can figure out how to filter the rule to a file, I think I can take it from there. | 15:25 |
| cognitiaclaeves | ( for reference: https://www.loggly.com/docs/systemd-logs/ ) | 15:26 |
| cognitiaclaeves | ( Although, now that I look at it, it looks like that will forward all logs to loggly, not just the ones I want to target. ) | 15:27 |
| blackflow | what's your rule like? | 15:27 |
| cognitiaclaeves | That's what I'm trying to figure out. | 15:28 |
| cognitiaclaeves | The documentation for rsyslog seems to refer to app_name and programme_name (I think), which I don't see in the journalctl output. | 15:28 |
| blackflow | what do you want to filter by? | 15:28 |
| blackflow | forget journalctl | 15:28 |
| blackflow | look at syslog itself and what it's logging | 15:29 |
| cognitiaclaeves | blackflow, you lost me, sorry. It's a standard ubuntu system in aws. | 15:30 |
| blackflow | journald is forwarding to rsyslog and you want to write rsyslog rules, correct? | 15:31 |
| cognitiaclaeves | Rules specific to the app, yes. | 15:31 |
| cognitiaclaeves | ( Don't know if journald is forwarding to rsyslog yet .. it may not be. ) | 15:31 |
| blackflow | right, so don't look at journalctl, but look at what is rsyslog writing. by default it has *.*;... /var/log/syslog, so take a look at that | 15:31 |
| blackflow | it is by default | 15:32 |
| cognitiaclaeves | Ah, that makes sense. | 15:32 |
| blackflow | also look at the selectors documentation https://www.rsyslog.com/doc/v8-stable/configuration/filters.html | 15:33 |
| blackflow | for program name I think you'll need an if-clause, but it's definitely doable | 15:33 |
| cognitiaclaeves | I had planned to use a property filter for that. | 15:34 |
| cognitiaclaeves | https://www.rsyslog.com/doc/v8-stable/configuration/properties.html It shows up in the property list. | 15:35 |
| cognitiaclaeves | And now that I'm looking at /var/log/syslog, I can identify programname. | 15:35 |
| blackflow | yeah | 15:36 |
| cognitiaclaeves | Ok, this might be enough to move forward with. Thanks! | 15:36 |
| blackflow | cognitiaclaeves: also keep in mind that journald by default does rate limiting. you'll have to remove it if you want _all_ messages forwarded to syslog and elsewhere | 15:36 |
| cognitiaclaeves | I just want a particular kind of message. I'll look into that. Will I also need to increase the open files setting for the OS? | 15:37 |
| blackflow | I don't see how open files are related here. you mean for rate limiting? for that, no. Only RateLimit* entries of journald.conf(5) | 15:38 |
| cognitiaclaeves | Ok, thanks. | 15:40 |
| _Trullo | ok, so I tried subnet: 255.255.255.0/24 .. didn't work, aparently it should be 192.168.1.0/24 to work properly, I have no idea how/why.. | 16:52 |
| _Trullo | pihole works perfect after like 20 reinstalls :) | 16:52 |
| teward | _Trullo: you typically won't use 255.* for anything. | 16:54 |
| teward | _Trullo: private IPv4 ranges that actually ARE private ranges are 192.168.0.0-192.168.255.255, 172.16.0.0-172.31.255.255, and 10.0.0.0-10.255.255.255 - only use Private IPv4 address space | 16:54 |
| teward | and NOT 255.* because that's a reserved range | 16:54 |
| blackflow | _Trullo: look up RFC 1918 for more info | 17:43 |
| trippeh | _Trullo: /24 is just another way of saying 255.255.255.0 as a mask | 17:59 |
| trippeh | 255.255.255.0/24 doesnt really make sense | 17:59 |
| lordcirth_ | I just had a systemd service (keepalived) restart without any apparent reason. Any tips for tracing what called the restart? | 18:02 |
| RoyK | lordcirth_: check the logs and turn on debugging if necessary | 18:13 |
| RoyK | lordcirth_: check dmesg too - perhaps it crashed - if so, it should show there with a SIGSEGV being signalled to the process | 18:14 |
| lordcirth_ | RoyK, nothing in the logs except it restarting | 18:14 |
| RoyK | lordcirth_: has this happened before? | 18:15 |
| lordcirth_ | RoyK, not that I recall | 18:15 |
| lordcirth_ | It's probably not a big deal, just weird | 18:15 |
| RoyK | I haven't seen keepalived just restart - anything in dmesg? | 18:16 |
| RoyK | run dmesg -T to get somewhat proper readable timestamps | 18:17 |
| lordcirth_ | RoyK, thanks, didn't know about -T | 18:21 |
| lordcirth_ | Yeah, nothing in dmesg for 2 days | 18:21 |
| RoyK | hm - and no cron job or something in systemd doing something funny? | 18:22 |
| === svetlana is now known as sveta | ||
| lordcirth_ | RoyK, nope | 18:38 |
| RoyK | it's just funny it restarted without anything in the logs - if it crashed and systemd restarted it, something should show up in dmesg | 18:39 |
| lordcirth_ | Yeah, I can't see any evidence of a crash | 18:39 |
| RoyK | do you have any monitoring of the system? sysstat/munin/zabbix/something? | 18:40 |
| lordcirth_ | RoyK, nagios, and keepalived emails me on all events, which is why I noticed ("Entering BACKUP state" on start) | 18:41 |
| sdeziel | lordcirth_: maybe a colleague of yours restarted it? or an config automation tool (puppet/chef/etc)? | 18:45 |
| lordcirth_ | sdeziel, yeah, I'm looking into that now | 18:46 |
| lordcirth_ | Oh, duh. I issued a dist_upgrade this morning. | 18:47 |
| RoyK | lordcirth_: oh - nagios - is that still alive? | 19:07 |
| lordcirth_ | RoyK, yes, although our install is way out of date. We are planning to upgrade to latest stable soon | 19:09 |
| lordcirth_ | RoyK, what do you use? | 19:09 |
| RoyK | lordcirth_: I used to use nagios some years back, but we're moving everything to zabbix now | 19:11 |
| RoyK | well - not everything - we'll be using NAV as well, https://nav.uninett.no/ | 19:11 |
| RoyK | because it has a few things that we can't find in other tools | 19:12 |
| lordcirth_ | RoyK, thanks, I will look at both. We have a custom inventory system with a lot of Nagios integration, etc, so we probably won't be moving | 19:12 |
| tomreyn | nagios -> zabbix, so out of the frying pan into the fire? | 19:12 |
| RoyK | I talked to the guys at uio.no - they have quite a few servers and have moved to zabbix, from nagios. | 19:13 |
| RoyK | zabbix has a steep learning curve, but once you understand the architecture, it's rather neat | 19:13 |
| lordcirth_ | RoyK, what key things does NAV have that Zabbix doesn't? | 19:14 |
| lordcirth_ | Ah, it's more of a network thing? | 19:14 |
| RoyK | stuff like switch configuration, like VLAN setup of ports, if the switch supports snmp rw | 19:14 |
| RoyK | it is | 19:14 |
| RoyK | so it's like "why not both?" | 19:14 |
| lordcirth_ | Very cool. We have a custom (argh) tool for that, which if it was open-sourced decades ago, might still be relevant. | 19:15 |
| === smoser1 is now known as smoser | ||
| RoyK | lordcirth_: take a look - with a lot of hosts, you may want to use SSDs for the storage - *lots* of things getting updated for graphite (IIRC) so it requires quite a bit of IOPS | 19:17 |
| lordcirth_ | RoyK, thanks. Our servers generally have mdraid1 SSDs for / | 19:17 |
| RoyK | should do | 19:17 |
| RoyK | althoguh I'd hate to have those things on the rootfs - I like to use separate storage or at least separate LVs for root and data | 19:19 |
| siavoshkc | tomreyn: Thank you. Unfortunately there was no backup and its a real disaster. Tomorrow our host will try to recover the wordpress files. I hope they recover something useful. I just shutdown the server to prevent moe loss. | 19:27 |
| siavoshkc | I deleted a big running wordpress site in a blinl | 19:28 |
| RoyK | backups are for cowards :D | 19:28 |
| siavoshkc | blink* | 19:28 |
| sarnold | ow | 19:28 |
| siavoshkc | RoyK: I cannot digest how there is no backup AT ALL for such a long running website | 19:28 |
| RoyK | an acquaintance | 19:28 |
| RoyK | an acquaintance of mine just found out the hard way that having a home server with 40TiB or so of storage and no backup is a bad idea - RAID6 isn't backup - bad, cheap controllers - oops | 19:29 |
| tomreyn | siavoshkc: sorry to hear this, good luck. | 19:29 |
| siavoshkc | The only good thing is that wordpress keeps many of its data in db | 19:30 |
| RoyK | siavoshkc: usually, there's chances of recovery, at least if the data resides on a RAID | 19:30 |
| RoyK | siavoshkc: the text is in the db, the images etc are on the filesystem | 19:31 |
| lordcirth_ | RoyK, yet another reason I use ZFS, not hardware raid | 19:31 |
| siavoshkc | RoyK: Yes so many things are lost | 19:32 |
| RoyK | lordcirth_: the only reason I don't have zfs on my home server, is the lack of flexibility - I've worked with rather large zfs systems for some years - they just work | 19:33 |
| lordcirth_ | RoyK, yeah, btrfs's ability to add disks and bodge storage together is very nice. LVM and such are good too. | 19:33 |
| RoyK | lordcirth_: but then - if you have a home server and want to toss in a new drive to expand things a bit, zfs won't do (for now) | 19:33 |
| RoyK | lordcirth_: I don't trust btrfs (yet) | 19:34 |
| lordcirth_ | I'm actually running / on ZFS raidz at home. 3 480GB SSDs. It's great | 19:34 |
| RoyK | lordcirth_: it's been 10 years or so with btrfs and it's still not stable | 19:34 |
| lordcirth_ | Yeah, turns out writing 5th gen filesystems is hard. Who knew? :P | 19:35 |
| RoyK | but extending VDEVs on zfs is coming - I think it's in fbsd already | 19:35 |
| lordcirth_ | Oh, cool. I'm looking forward to 0.8's encryption and faster scrubs | 19:38 |
| lordcirth_ | We have a major dataset at work that takes nearly a week to do a weekly scrub | 19:38 |
| RoyK | then don't scrub weekly :þ | 19:38 |
| RoyK | seems we're tossing out a server soon with 48x4TB disks - with a bit of luck, I could perhaps take it | 19:39 |
| sarnold | I'm looking forward to encryption and the 'special' vdev support | 19:40 |
| teward | @RoyK: send me a few of the disks lol? xD | 19:40 |
| teward | :P | 19:40 |
| teward | always in need of disk expansion on my array of disks lol | 19:40 |
| RoyK | that supermicro mobo would be nice as well as the controllers | 19:40 |
| sarnold | what's rplacing that beast? :) | 19:41 |
| RoyK | dell compellent | 19:41 |
| RoyK | well, that is, dell equallogic, old stuff - just used for CCTV cameras | 19:42 |
| RoyK | but some people at the IT dept think that ZFS and mdraid and such things are black magic and would rather sell their soul to something with a nice gui | 19:42 |
| sarnold | okay, it *would* take like five minutes to type out the zpool create command with 48 devices.. | 19:43 |
| RoyK | yeah - it's pretty hard - and dark magic - and checksums the data and all, whcih equallogic certainly doesn't (nor does compellent) | 19:44 |
| RoyK | but Dell Provides Support, meaning they tell us they need a crapload of logs and then blame us for doing something stupid | 19:45 |
| RoyK | or just say "we don't know yet" | 19:45 |
| RoyK | like when that blade centre went down during the holiday - it's powered via a UPS and one direct line - six PSUs - the UPS had some dead fans and had serious issues, but the power from the direct line was stable, but still, the blade centre just powered itself off and too all its VMs with it | 19:47 |
| sarnold | owwwwwww | 19:50 |
| lordcirth_ | six PSUs, and one failure brought it down? | 19:50 |
| lordcirth_ | Or, half of the PSUs were on the UPS? | 19:51 |
| RoyK | half on UPS half on direct | 19:51 |
| RoyK | but it should be able to run on two PSUs | 19:51 |
| lordcirth_ | Its documentation says 2/6 is enough? | 19:52 |
| RoyK | yes | 19:52 |
| RoyK | and 3/6 should at least be sufficient | 19:52 |
| RoyK | but for some reason, it just powered itself off | 19:53 |
| RoyK | Dell hasn't been able to give us an explaination of this | 19:53 |
| lordcirth_ | I bet they'll eventually say "oh, that's for if the power just cuts, dirty power is unsupported" | 19:53 |
| RoyK | something like that, yes | 19:54 |
| sarnold | 2-of-6?? wow | 19:54 |
| tomreyn | "you forgot to install the firmware update to the psu's" | 19:57 |
| teward | lol | 19:58 |
| RoyK | tomreyn: hehehe | 20:10 |
| RoyK | tomreyn: please update firmware daily and make sure you have redundancy elsewhere | 20:10 |
| tomreyn | :) i've heard a couple funny excuses already. but in the end we all just try to not become the victim of the bad conditions we have to (?) work under. | 20:16 |
| zertyuio | hello there | 21:39 |
| zertyuio | anyone used to work with mdadm everydays ? | 21:40 |
| tomreyn | zertyuio: i'm sure there are people here who worked with mdadm before, or do so daily. but that's not your actual question. please ask your actual question. | 21:42 |
| tomreyn | and hello there, too ;) | 21:42 |
| zertyuio | here is my disk of 20Go https://pastebin.ubuntu.com/p/vfnTxgdK7g/ i would like to resize it to 950Go | 21:43 |
| zertyuio | how i can do so ? | 21:44 |
| zertyuio | without losing data | 21:44 |
| tomreyn | this is very easy: | 21:44 |
| tomreyn | you do backups. | 21:44 |
| tomreyn | zertyuio: about resizing, how would you do this when there is no unallocated space? | 21:46 |
| zertyuio | easy for you mean not easy for everyone | 21:46 |
| zertyuio | as you can see on my post : https://pastebin.ubuntu.com/p/vfnTxgdK7g/ | 21:47 |
| zertyuio | there is something above 900GB of free space to use | 21:47 |
| zertyuio | i would like to do so the same way that we used to do with lvm and fdisk | 21:48 |
| lordcirth_ | zertyuio, firstly, yes, do a backup. | 21:49 |
| lordcirth_ | zertyuio, then, you can follow this: https://raid.wiki.kernel.org/index.php/Growing#Expanding_existing_partitions | 21:49 |
| zertyuio | how you can backup system volume ? | 21:50 |
| lordcirth_ | Skipping the first few instructions that are about switching the larger disks, of course | 21:50 |
| lordcirth_ | zertyuio, the root filesystem? | 21:50 |
| lordcirth_ | !backup | 21:50 |
| ubottu | There are many ways to back your system up. Here's a few: https://help.ubuntu.com/community/BackupYourSystem , https://help.ubuntu.com/community/DuplicityBackupHowto , https://wiki.ubuntu.com/HomeUserBackup , https://help.ubuntu.com/community/MondoMindi - See also !sbackup and !cloning | 21:50 |
| zertyuio | i would like to do so from rescue cd | 21:51 |
| zertyuio | are you sure about the link that you post ? | 21:53 |
| zertyuio | for backup ? | 21:53 |
| tomreyn | zertyuio: what makes you wonder about it? | 21:56 |
| zertyuio | which it ? | 21:56 |
| tomreyn | <zertyuio> are you sure about the link that you post ? | 21:57 |
| tomreyn | zertyuio: these links should indeed help creating a backup | 21:58 |
| zertyuio | yes, for example if you take example of duplicitybackup | 21:58 |
| zertyuio | it ask to install a package inside the system | 21:58 |
| zertyuio | i m not sure that it will does some good backup | 21:59 |
| zertyuio | if i follow so | 21:59 |
| zertyuio | i m expecting something that i can boot from rescue cd | 21:59 |
| zertyuio | and mount the partition | 22:00 |
| zertyuio | and backup the whole thing | 22:00 |
| tomreyn | this is cloning / imaging, bare metal recovery, but not backup. | 22:00 |
| tomreyn | you don't want to have to stop your system every time you create a backup, right? | 22:01 |
| zertyuio | yeas | 22:01 |
| zertyuio | so i install duplicity | 22:02 |
| tomreyn | so you backup only data that you will want to restore. this also enables you to do smaller, incremental, backups, of just the data you actually need. if the OS storage fails or breaks and you need to reinstall, then you just do that, restore configurations from your backup, and restore other data from your backup, and you're set. | 22:02 |
| zertyuio | in order to so | 22:04 |
| tomreyn | what you had on your mind about cloning / imaging can be done with clonezilla-live, a bootable linux distribution for this single purpose. this is not a proper backup, but something you can do now if it'd make you feel better about it. | 22:04 |
| zertyuio | wait | 22:05 |
| zertyuio | in order to so | 22:05 |
| tomreyn | ? | 22:05 |
| zertyuio | i have to seperate all data need to be backup before | 22:05 |
| zertyuio | without mixing with system partition | 22:06 |
| zertyuio | that task i forgot to so | 22:06 |
| zertyuio | i don't have separate partition for data | 22:06 |
| tomreyn | maybe you should have | 22:07 |
| tomreyn | ideally separate disks, so you don't have OS and data fail at the same time. | 22:08 |
| zertyuio | i think i found the best way | 22:08 |
| zertyuio | i will reboot my system | 22:08 |
| zertyuio | with rescue cd | 22:08 |
| zertyuio | and mount my / | 22:08 |
| zertyuio | then tar -cvf / | 22:09 |
| zertyuio | will be the best way i think | 22:09 |
| zertyuio | is it the best way ? | 22:10 |
| hal529 | Hi. I've an ubuntu 16LTS server. | 22:12 |
| hal529 | It boots/operates from a 2-disk RAID1 array. Works great; has for ages. | 22:12 |
| hal529 | I just noticed in startup logs, on exec of "/lib/systemd/systemd-modules-load", these ,essages --> "Module 'loop' is builtin" & "Failed to find module 'md'". | 22:12 |
| hal529 | In /etc/modules, there _are_ "loop" & "md" entries. | 22:12 |
| hal529 | I've never touched this on this box -- so something, maybe default install, added these. | 22:12 |
| hal529 | Do I *need* either? Safe to remove them? | 22:12 |
| tomreyn | zertyuio: this is surely not the best approach for regular backups (since it involves rebooting the systemjust for that), but maybe it is a good approach for your preparation of the RAID array changes. | 22:13 |
| zertyuio | let me do that first | 22:14 |
| tomreyn | zertyuio: note this approach may not enable you to backup databases and other data structures which require to be backued up in a special way (such as a database dump). | 22:14 |
| tomreyn | !yy.mm | hal529 | 22:15 |
| ubottu | hal529: Ubuntu version numbers are: YY.MM (YY=release year,MM=release month). Each year sees two releases, so just specifying YY is imprecise. See also https://www.ubuntu.com/about/release-cycle | 22:15 |
| tomreyn | hal529: but you made it clear enough that you have some patch level of ubuntu 16.04 LTS | 22:15 |
| zertyuio | man you are doing the backup / how database can be exclude ? | 22:15 |
| tomreyn | zertyuio: read the wikis you were pointed to, i can't explain everything. | 22:16 |
| hal529 | tomreyn: I asssume since you're asking, it makes a difference to my question? Here, I've "Description: Ubuntu 16.04.5 LTS" | 22:16 |
| zertyuio | just tell what you read about that ? | 22:16 |
| zertyuio | just tell mewhat you read about that ? | 22:17 |
| tomreyn | hal529: so loop is for loop mounts, such as you'd need for snaps, maybe ISO image mounts. md is for software RAID, managed via mdadm. | 22:19 |
| tomreyn | zertyuio: do you have databases? | 22:19 |
| hal529 | tomreyn: SUre. I know what they're _for_. I'm unclear why ubu's complaining about them here ... and what should be done about it. | 22:20 |
| hal529 | I clearly _have_ raid in place & working; mdadm's fine. So that 'md' in /etc/modules -- and the 'failed to find' message suggest that's not necessary, done somewhere or somehow else, etc. | 22:21 |
| tomreyn | hal529: i guess if you don't have a software raid it should eb safe to ignore these messages, which are probably just warnings? | 22:21 |
| hal529 | I *do* have software RIAD -- that's the point. | 22:22 |
| zertyuio | yes of course i got databases | 22:22 |
| hal529 | RAID, even ... | 22:22 |
| tomreyn | hal529: i was typing this when i hadnt spotted your message, yet | 22:22 |
| tomreyn | zertyuio: which ones then? | 22:22 |
| hal529 | np | 22:22 |
| zertyuio | mariadb | 22:22 |
| tomreyn | zertyuio: then learn how to backup mariadb and then continue with the file based backup. | 22:23 |
| tomreyn | hal529: did you reboot since, or is it that you're wondering whether that's safe now? | 22:23 |
| zertyuio | just tell me | 22:23 |
| tomreyn | zertyuio: just read it | 22:24 |
| zertyuio | why saving / can not affect database ? | 22:24 |
| hal529 | tomreyn: sorry, 'since' what? I get these messages every boot. It's been this way for ages. I'm _now_ paying attention to the messages. | 22:24 |
| zertyuio | for me it is not logical | 22:24 |
| hal529 | I didn't add these lines. They are causing warnings/errors messages on boot. Don't know if that's real problem, or just noise. | 22:25 |
| tomreyn | hal529: i see. i wouldn't worry about them if it doesn't impact your ability to boot. | 22:25 |
| tomreyn | hal529: interestingly i can't find any other mention of this message on the web: Failed to find module 'md'" | 22:28 |
| hal529 | ... which is why I'm in here ... | 22:28 |
| tomreyn | do you have a custom kernel? | 22:29 |
| hal529 | nope | 22:29 |
| tomreyn | or custom core packages such as systemd? | 22:29 |
| hal529 | nothing custom. bog-standard ubu16, with a zimbra mail server install on it. | 22:29 |
| tomreyn | is this message printed when you "systemctl restart systemd-modules-load.service" | 22:29 |
| hal529 | it _got_ to v16+ thru countless upgrades over the years -- NOT a clean install | 22:30 |
| hal529 | nope, a systemd restart of that unit FAILs -- because of these messages | 22:30 |
| hal529 | it seems ... | 22:30 |
| tomreyn | can you show the full output? | 22:30 |
| tomreyn | !paste | 22:30 |
| ubottu | For posting multi-line texts into the channel, please use https://paste.ubuntu.com | To post !screenshots use https://imgur.com/ !pastebinit to paste directly from command line | Make sure you give us the URL for your paste - see also the channel topic. | 22:30 |
| hal529 | sucks ... I was gonna paste 10K+ lines in here ;-p | 22:31 |
| tomreyn | sorry if i asked bafore: is the system fully patched? | 22:31 |
| hal529 | oops. shucks! | 22:31 |
| hal529 | yes, fully patched. will pastebin in a sec ... | 22:31 |
| tomreyn | you can install pastebinit and pipe into that or | nc termbin.com 9999 | 22:32 |
| hal529 | i'm a less-is-more kinda guy. "One moment please ..." | 22:32 |
| hal529 | tomreyn: --> https://paste.fedoraproject.org/paste/RK6-HNXYLdlCm1b525Yd3Q | 22:34 |
| hal529 | and, fwiw -> https://paste.fedoraproject.org/paste/PoN735TdWLjr0QTHgWt6Ug | 22:35 |
| tomreyn | https://termbin.com/ug6y is what i get on a 16.04 *desktop* for "sudo systemctl status systemd-modules-load.service 2>&1" | 22:38 |
| tomreyn | restart of this service gives 0 output there | 22:39 |
| hal529 | and what's in _your_ /etc/modules ? | 22:39 |
| tomreyn | $ find /etc/modules-load.d/*.conf /run/modules-load.d/*.conf /usr/lib/modules-load.d/*.conf 2>&1 | nc termbin.com 9999 | 22:41 |
| tomreyn | https://termbin.com/e6gm | 22:41 |
| tomreyn | $ cat /etc/modules | nc termbin.com 9999 | 22:42 |
| tomreyn | https://termbin.com/y41v | 22:42 |
| tomreyn | blank | 22:42 |
| hal529 | "First base"! | 22:42 |
| tomreyn | i should have a 16.04 server somewhere, looking | 22:43 |
| hal529 | I never touched that file. Ok, modify that -- within the limits of my long-term memory ... | 22:43 |
| tomreyn | mine is very weak there, but you could inspect the timestamp and compare it to the installation data | 22:44 |
| tomreyn | *datE | 22:44 |
| hal529 | On ubu, there IS a /lib/modules/4.15.0-45-generic/kernel/drivers/md/raid1.ko. | 22:44 |
| hal529 | There is *no* ".../md.ko" to be found | 22:45 |
| tomreyn | you run a HWE kernel there | 22:45 |
| hal529 | Yep, standard upgrade procedure. Seems that all the 'non-complaining' entries in /etc/modules each _do_ have a correcponding "*.ko". Neither "loop" nor "md" , otoh, *do*. | 22:47 |
| tomreyn | # ls -l /lib/modules/4.4.0-*/kernel/drivers/md/raid1.ko | 22:47 |
| tomreyn | -rw-r--r-- 1 root root 62078 Dec 5 15:15 /lib/modules/4.4.0-141-generic/kernel/drivers/md/raid1.ko | 22:47 |
| tomreyn | -rw-r--r-- 1 root root 65182 Jan 17 00:42 /lib/modules/4.4.0-142-generic/kernel/drivers/md/raid1.ko | 22:47 |
| hal529 | Which _suggests_ that they can (should?) be removed. When it comes to fubar-ing RAID arrays, especially *boot* arrays, I tend to wanna know WTF I'm doing first ... | 22:47 |
| hal529 | Are you *using* RAID? | 22:48 |
| tomreyn | no | 22:48 |
| tomreyn | but i'm on one where i do now | 22:49 |
| hal529 | Well, then I certainly wouldn't expect the module even to be automagically loaded by some install/setup/etc process | 22:49 |
| tomreyn | on the md raid1 server i have an empty /etc/modules | 22:50 |
| tomreyn | so just comments | 22:50 |
| tomreyn | and # ls -l /lib/modules/*/kernel/drivers/md/raid1.ko | 22:50 |
| tomreyn | -rw-r--r-- 1 root root 60526 Jan 19 2018 /lib/modules/4.4.0-112-generic/kernel/drivers/md/raid1.ko | 22:50 |
| tomreyn | -rw-r--r-- 1 root root 62078 Jun 14 2018 /lib/modules/4.4.0-130-generic/kernel/drivers/md/raid1.ko | 22:50 |
| hal529 | not even a 'raid1' line? | 22:50 |
| tomreyn | no | 22:50 |
| tomreyn | just lines starting with # | 22:50 |
| hal529 | clean-installed, recent ubu? or upgraded from prior versions? | 22:51 |
| tomreyn | this one should have been installed as 16.04 initially. /etc/initramfs-tools/modules is also empty except for comments | 22:51 |
| hal529 | hm. wonder if it's a carryover/artifact from prior installs ... | 22:52 |
| tomreyn | MODULES=most in /etc/initramfs-tools/initramfs.conf | 22:52 |
| tomreyn | what is yours set to there? | 22:53 |
| hal529 | I have neither such a file, nor its parent dir ... | 22:54 |
| tomreyn | you sure you're running ubuntu? | 22:54 |
| tomreyn | ;-) | 22:54 |
| hal529 | heh. i wasn just typing the same snarky question! | 22:55 |
| tomreyn | is this amd64? | 22:55 |
| tomreyn | # dpkg -S /etc/initramfs-tools/initramfs.conf | 22:55 |
| tomreyn | initramfs-tools-core: /etc/initramfs-tools/initramfs.conf | 22:55 |
| tomreyn | do you *have* an initrd? | 22:55 |
| hal529 | it's an AMD E-350. | 22:56 |
| hal529 | sure -> initrd.img-4.15.0-45-generic | 22:56 |
| tomreyn | how is it generated if not by initramfs-tools? | 22:57 |
| tomreyn | do you have initramfs-tools and initramfs-tools-core installed? | 22:57 |
| hal529 | for me, here, " lsmod | egrep "loop|raid|md"" -> https://paste.fedoraproject.org/paste/xgdfYRiojL~VE0TnDktBXA | 22:58 |
| hal529 | yes -> https://paste.fedoraproject.org/paste/1L72h8s2GexvSxq~4SSHKQ | 22:58 |
| tomreyn | so initramfs-tools-core is installed, which /etc/initramfs-tools/initramfs.conf belongs to, but you don't have the file. this suggests the package was not properly installed. | 23:01 |
| hal529 | wait . fubar <- pebkac. fat thumbs alert! | 23:01 |
| tomreyn | my lsmod looks similar btw | 23:03 |
| hal529 | https://paste.fedoraproject.org/paste/do6cqK5FCZk7CbkwQ-HS5A | 23:03 |
| hal529 | yes, MODULES = most | 23:03 |
| tomreyn | i just did "systemctl status systemd-modules-load.service" and "systemctl status systemd-modules-load.service" on this server with md raid-1 and while both generate output, no module warnings ar eprinted | 23:06 |
| tomreyn | /etc/modules is empty, and so are /etc/modules-load.d/*.conf /run/modules-load.d/*.conf /usr/lib/modules-load.d/*.conf | 23:07 |
| tomreyn | hal529: sorry i can't help more, but i'm really sleepy and should wrap it up here. | 23:08 |
| hal529 | np o/ | 23:08 |
| tomreyn | bye bye | 23:08 |
| jonfatino | Anyone here a ubiquity master? I am trying to install ubuntu from ubuntu livecd with a bash script. Not launching the installer via gui. I already have a preseed file. | 23:15 |
| === JanC_ is now known as JanC | ||
| === Kamilion|ZNC is now known as Kamilion | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!