[08:27] <Forza> Hello
[08:28] <Forza> I have an issue with 20.04lts. During reboot systemd tries to unmount the filesystem used by targetclid before stopping targetclid.
[08:28] <Forza> Then it doesn't try to unmount the filesystem again before rebooting/halting which leads to corruption
[18:21] <MIF> Ok i have something talking up aton of space
[18:21] <MIF> but I just can't find it
[18:21] <MIF> ncdu dose not give me anything that helpful
[18:23] <MIF> does anyone have any tips, or other things I can try?
[18:50] <rfm> MIF, you are running ncdu as root?  otherwise it might not see into dirs you can't read
[18:51] <MIF> it is as root
[18:51] <MIF> the only think I could find was the sys and kernal log
[18:51] <MIF> the only think I could find was the sys and kernal logs
[18:52] <MIF> but still my disk usesage is very high
[18:55] <rfm> MIF, you may have a unlinked-but-still-open file.  These can be a pain to run down.
[18:55] <MIF> how do I check?
[18:56] <rfm> MIF, 'lsof { grep '(deleted)' " will show them
[18:57] <rfm> MIF, hmm, just tried it, lsof shows more useful info than it used to when I last had this problem years ago
[18:57] <MIF> there is alot here
[18:58] <MIF> https://pastebin.com/RvUDUbSY
[18:59] <MIF> https://pastebin.com/raw/RvUDUbSY
[18:59] <MIF> ^that view is better
[19:02] <rfm> MIF, those fail2ban logs are huge (actually suspiciously so, I wonder if lsof is misreporting)
[19:05] <rfm> MIF, no idea why f2b is keeping them open.  You could try stopping f2b (which will close the files held open by pid 1063) and see if the space comes back
[19:09] <MIF> ok
[19:10] <rfm> MIF, hmm, it's just the same one file (inode 1313619) open many times, not multiple files at least
[19:10] <MIF> yep that worked
[19:10] <MIF> back to 37%
[19:10] <MIF> thanks rfm
[19:12] <rfm> MIF, this actually looks like a bug in fail2ban, might look into reporting it.  if it comes back, maybe look into setting up a periodic restart of f2b
[19:25] <MIF> ok
[19:25] <rfm> MIF, might not be a but in fail2ban, this situation would arise if somebody saw the big log and rm'ed it but didn't restart fail2ban (could even be a log rotation service if you have one)
[19:25] <MIF> that is what I did
[19:25] <MIF> I rm'd the log file
[19:25] <MIF> and did not restart fail2ban
[19:27] <rfm> MIF, ah! so all you need to do is remember to restart next time (might want to check into why f2b is writing so much to its log, perhaps you're getting really hammered by somebody trying to break in?)
[19:27] <MIF> no, I think I have it set to debug