| brobostigon | morning | 06:11 |
|---|---|---|
| zxmoypi | o/ | 07:13 |
| brobostigon | \o | 07:21 |
| Chunkyz | Evening. | 21:20 |
| Chunkyz | No.... | 21:20 |
| Chunkyz | K. | 21:20 |
| penguin42 | Yes, Evening | 21:23 |
| zmoylan-pi | nearly 2130 here... dark and stormy night. status orange wind | 21:24 |
| daftykins | :O | 21:25 |
| daftykins | i've been doing maintenance on that new client's server since 1:20pm :) although it's mostly lengthy I/O tasks so i've not really been working that long | 21:25 |
| penguin42 | zmoylan-pi: The worst wind went through about 7pm | 21:26 |
| daftykins | i discovered they were running their VM's from daisy-chained failed snapshots created by the backup program, so tracking all those delta files is what had murdered the I/O performance | 21:26 |
| penguin42 | ouch | 21:27 |
| zmoylan-pi | fun | 21:27 |
| daftykins | i came up with the cunning plan of uploading a clonezilla image, booting that in the VM and backing up only the live dataset to a NAS they have on-site :D | 21:27 |
| zmoylan-pi | west coast got status red. will have to see if that part of the island is still there tomorrow | 21:28 |
| daftykins | that saved me having to transfer each delta image at a full total of 850GB ish instead of only a 150GB backup (compressed probably) as clonezilla spat out | 21:28 |
| zmoylan-pi | you didn't bring a 1tb floppy disk?! :-P | 21:28 |
| daftykins | :D | 21:29 |
| zmoylan-pi | great fun formatting that in windows :-P | 21:29 |
| daftykins | i deleted one snapshot as a test first, took 2 hours to consolidate that - now i'm deleting the rest and it's at 78% | 21:29 |
| daftykins | once complete i'll boot with the network disconnected so the mailserver doesn't go online - and run a defrag on both virtual disks :) | 21:30 |
| zmoylan-pi | i was once handed a pc with vital data and 0 bytes free... took about 2 hours to delete 1mb of uneccessary files so it could boot :-) | 21:30 |
| daftykins | erk! | 21:30 |
| zmoylan-pi | windows is tetchy that way :-) | 21:31 |
| daftykins | did you ever that phenomenon that everyone comes out of the woodwork all at once for things they want done, just as you're already busy? | 21:31 |
| daftykins | +encounter | 21:31 |
| zmoylan-pi | simple rule. before i go onsite i have an email of what i am doing that day. i let the nice staff know i'm coming and to have anything that might need a look at to look at while i'm waiting for computer to complete tasks | 21:32 |
| daftykins | a friends biz found an edge case a bit ago where they managed to get nextcloud to fill their VPS's /, apache wouldn't permit website logins until some space was cleared | 21:32 |
| daftykins | ah in my case i mean other clients wake up and get in touch :D | 21:32 |
| zmoylan-pi | ah, very different | 21:33 |
| daftykins | woohoo we just jumped to completion | 21:34 |
| daftykins | wouldn't it be nice if this VM still booted now... | 21:35 |
| penguin42 | <drum roll> | 21:35 |
| daftykins | :D | 21:36 |
| daftykins | few things to prod first | 21:36 |
| * zmoylan-pi places bet yes... eventually... | 21:36 | |
| penguin42 | boot on a leapyear hmmmm | 21:37 |
| daftykins | what's the worst that could... | 21:37 |
| zmoylan-pi | never say that, never think that, never type that. it's asking the gods of mocking to smite some small irreplaceable part that you can't replace easily on a saturday evening | 21:38 |
| daftykins | :D | 21:42 |
| zmoylan-pi | i once spent 6 hours replacing a floppy drive. every part, every cable, every thing that could wrong went wrong. it was hilariously funny.... afterwards | 21:42 |
| daftykins | as an anecdote a few pints into an evening ;) | 21:42 |
| zmoylan-pi | here's a pc moylan before you go can you replace the dead floppy drive. no prob. 2 minute job..... famous last words... | 21:42 |
| zmoylan-pi | not helped by it been a cursed ast slimline pc that required super slim floppy drive and a 'left hand' drive cable that worked only with that make and model... | 21:43 |
| daftykins | we'll just ignore the bit where i tried to RDP into it and wondered why it wasn't working, given i'd disconnected the virtual NIC, shall we, gang? :D | 21:44 |
| * daftykins coughs | 21:44 | |
| zmoylan-pi | never happened.... been there... done that.... | 21:44 |
| zmoylan-pi | i once had my finger hovering over the enter key to delete files on hard disk that i needed to expand but didn't have the space for :-) | 21:45 |
| zmoylan-pi | just wiggle the blue wire.... https://i.imgur.com/2OjXQAJ.jpg | 22:12 |
| daftykins | heh got linked that recently too :D | 22:14 |
| daftykins | hmm it wasn't a happy bunny booted with no NIC, poked around in safe mode and making it chkdsk now | 22:14 |
| daftykins | how i wish it was on SSD... | 22:14 |
| penguin42 | daftykins: So I'm not sure which VM system you're using or how your multiple layers of storage work; but one way to flatten a complex thing is just to boot the VM and dd out the disk image | 23:44 |
| daftykins | yeah, in fact because the files themselves were huge and i knew it'd be a wasted job if i had turned up and copied them over the network, which was the original plan, i instead shutdown the VM and booted a clonezilla ISO to dump only the 'real' volumes to a network share :) | 23:53 |
| penguin42 | nod | 23:53 |
| daftykins | it's a long EOL VMware ESXi box these folks have, much like everything else on their network xD | 23:54 |
| daftykins | some numpty installed a backup product inside the VM itself and told it to back itself up to network shares, when it has been failing it kept creating a snapshot, leaving the mess as it was | 23:54 |
| daftykins | i was partly deciding between hosing them entirely and just restoring the imaged backup so that it would be sequentially written and probably sat better on the RAID volume, but in the end i just tried the snapshot deletion | 23:55 |
| daftykins | just running the OS level defrag now on the two virtual disks, should be hugely improved :D | 23:56 |
| penguin42 | nod; it's pretty amazing how old some critical stuff running is - even in big companies | 23:56 |
| daftykins | these folks haven't paid anyone for maintenance at all, it's shocking | 23:57 |
| daftykins | physical server has firmware from 2012, the HP iLO 4 has had 2 critical security updates between the version that's on here and the newest | 23:58 |
| daftykins | (that's the management NIC firmware) | 23:58 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
