| RoyK | the superblock is placed at drive start plus 4k https://raid.wiki.kernel.org/index.php/RAID_superblock_formats#The_version-1_superblock_format_on-disk_layout | 00:00 |
|---|---|---|
| fretegi | RoyK, rule out that something funky on that drive? | 00:00 |
| RoyK | I doubt it's a drive failure | 00:00 |
| fretegi | so dd like the first 5k then? | 00:00 |
| RoyK | if so, it's the weirdest I've seen so far | 00:00 |
| RoyK | 1M? | 00:00 |
| RoyK | just drop a bunch of zeros there - it doesn't matter | 00:01 |
| fretegi | RoyK, ha well ran DD then tried to zero the superblock and says "unrecognized md component device" guess superblock gone | 00:02 |
| fretegi | mdadm: /dev/sdc appears to be part of a raid array: | 00:04 |
| fretegi | level=raid0 devices=0 ctime=Wed Dec 31 17:00:00 1969 | 00:04 |
| fretegi | now thats interesting... thats from like 5 years ago.. that raid0 | 00:04 |
| fretegi | hows it still picking that up, i just ran dd bs=1M count=10k | 00:05 |
| RoyK | fretegi: ok, put a new raid on it, some crap, and run mdadm --zero-superblock. v0.9, which might be what you had back then, is at the end of the drive | 00:06 |
| RoyK | see the article I linked to above for superblock format descriptions | 00:07 |
| fretegi | yea i was reading | 00:07 |
| RoyK | fretegi: you'll need to stop the raid before removing the superblock, though, but that should be rather obvious | 00:08 |
| fretegi | yea but wont let me stop md1... | 00:09 |
| fretegi | weird | 00:09 |
| RoyK | anything in vgs/lvs/pvs? | 00:10 |
| fretegi | https://dpaste.org/1vKo | 00:10 |
| fretegi | all i did was create md1, it fired right up, never added any fs or anything to it | 00:10 |
| RoyK | check lvm anyway | 00:10 |
| RoyK | I hate it when that happens, when automatic somethings takes over | 00:11 |
| fretegi | whoa.... | 00:12 |
| RoyK | some old fun showed up? ;) | 00:12 |
| fretegi | so my VG for my sdb raid (md0) is showing active on md1... | 00:13 |
| fretegi | and md1 is sdc which has nothing | 00:13 |
| RoyK | erm… | 00:13 |
| RoyK | can you pastebin that shite? | 00:13 |
| fretegi | https://dpaste.org/wf1t | 00:14 |
| fretegi | https://dpaste.org/H1iO | 00:14 |
| RoyK | try mdadm --examine for sd[bc] | 00:15 |
| fretegi | if i vgremove servervg, does that perma delete it, or just effectively unmount it? | 00:16 |
| RoyK | is that the same name you have for the new vg? | 00:16 |
| fretegi | https://dpaste.org/UhPy | 00:17 |
| fretegi | sdb & sdc are part of different array UUID's | 00:17 |
| fretegi | no thats the thing... thats the OLD vg for md0 | 00:17 |
| fretegi | i didnt run any lvm commands on this new md1, just built the damn thing and thats it lol | 00:18 |
| fretegi | i dont show a lvol running tho | 00:19 |
| fretegi | wait yea i do, just not mounted... but avail | 00:19 |
| RoyK | well, if it doesn't contain any actual data, just remove it. But I don't know any way to undo that, so keep a backup | 00:20 |
| fretegi | how do i disassociate servervg with md1.... not sure how it was associated anyway | 00:20 |
| RoyK | iirc that "a" is "active" if you mean as in lvchange +a y | 00:20 |
| fretegi | well i can do that.. my concern is... since servervg contains the lvol that is associated with md0... if i remove it here, do i remove it there? | 00:20 |
| RoyK | pvdisplay -v should show the uuid | 00:21 |
| fretegi | RoyK, actually i got the vg stopped... had to stop lvol first whoops | 00:23 |
| fretegi | md1 stopped now too | 00:23 |
| fretegi | and superblock zero' | 00:24 |
| fretegi | zero'd | 00:24 |
| RoyK | if you don't know the correct uuid or you don't know which drive is which (linux can shuffle around disk names from time to time), write down the UUIDs in use now and whatever's linked to in /dev/disk/by-id/ to that to get the full path | 00:24 |
| fretegi | were good now | 00:24 |
| fretegi | superblock zero'd out | 00:24 |
| RoyK | if not, well, I hope you have a good backup :) | 00:24 |
| fretegi | oh i do | 00:25 |
| fretegi | as of this am | 00:25 |
| fretegi | so what form here? | 00:25 |
| RoyK | any active raids now? | 00:25 |
| fretegi | nope | 00:25 |
| fretegi | well, md0 can be restarted | 00:25 |
| RoyK | can you assemble the original? | 00:25 |
| fretegi | yup | 00:25 |
| fretegi | i can do that, just has to be manually | 00:25 |
| RoyK | doesn't matter | 00:26 |
| fretegi | md0 fired up with just one drive | 00:26 |
| RoyK | sdb? | 00:27 |
| RoyK | did lvm find it? does the fs mount (try read-only if you're nervous) | 00:27 |
| fretegi | i had to manually mount it all but yes, vg's & lvol's all properly associated with md0 and mounted to the proper directy and data appears to be intact | 00:28 |
| RoyK | good | 00:28 |
| fretegi | so what of fixing md0 ya think? | 00:29 |
| RoyK | then just create a new r0 on the other drive (sdc?) and stop the raid, then --zero-superblock that drive - it may remove the v(something old) superblock. you may have to specify superblock version - never done that with superblock removal. | 00:29 |
| fretegi | build new array like i was starting to do, rsync stuff over and ditch md0? | 00:29 |
| fretegi | did that already | 00:30 |
| fretegi | stopped md1 and zero's sdc | 00:30 |
| fretegi | yea wont let me assemble md1 now as superblock gone | 00:31 |
| fretegi | so back to a blank drive... | 00:31 |
| RoyK | try --assemble --force | 00:31 |
| RoyK | just for kicks | 00:31 |
| fretegi | md1? | 00:31 |
| RoyK | mhm | 00:31 |
| fretegi | assembly abored | 00:31 |
| fretegi | abortd | 00:31 |
| RoyK | good | 00:32 |
| fretegi | with both --run and --force | 00:32 |
| RoyK | fine | 00:32 |
| fretegi | md1 commented out of mdadm.conf too | 00:32 |
| RoyK | then, theoretically, it should work fine with just adding sdc to the array | 00:32 |
| RoyK | just don't put a bloody partition table on there afterwards | 00:33 |
| fretegi | do i need to someone let mdadm know that the prior 2nd disk in md0 is toast? | 00:33 |
| fretegi | somehow* | 00:33 |
| fretegi | still thinks its supposed to have 2, just running on 1 | 00:33 |
| RoyK | no, it won't find a superblock, so it'll treat it as a new one | 00:33 |
| fretegi | alright... adding her up here we go | 00:34 |
| RoyK | it should start resync by itself | 00:34 |
| fretegi | back to recovering | 00:34 |
| RoyK | good | 00:34 |
| fretegi | time #6 | 00:34 |
| fretegi | lol | 00:34 |
| fretegi | ok in an other 2.5 hours... ill re updated initramfs and mdadm.conf and see if good | 00:35 |
| RoyK | the other way around | 00:35 |
| fretegi | right | 00:35 |
| RoyK | but it'll probably be the same data | 00:35 |
| fretegi | yea should be | 00:35 |
| fretegi | array UUID same | 00:35 |
| RoyK | the uuid doesn't change with a disk replacement | 00:36 |
| fretegi | sure giving these drives a work out.. | 00:36 |
| RoyK | shouldn't hurt | 00:36 |
| RoyK | what sort of drives are these, btw? | 00:36 |
| fretegi | sata desktop drives lol | 00:37 |
| RoyK | model? | 00:37 |
| fretegi | good ones... but still.. not enterprise stuff... just seagate desktop stuff | 00:37 |
| RoyK | any support for SCTERC? ;) | 00:37 |
| RoyK | smartctl -l scterc /dev/sdX | 00:38 |
| fretegi | seagate 2 tb soemthing or other | 00:38 |
| fretegi | not supported | 00:38 |
| fretegi | uh oh... heres something funky.... | 00:38 |
| RoyK | bummer | 00:38 |
| RoyK | can you pastebin the full smartctl -a for those two drives? | 00:39 |
| fretegi | https://dpaste.org/enqn | 00:39 |
| fretegi | ran fdisk -l and saw that..... | 00:39 |
| fretegi | remember we DD'd it and --zero-superblock'd it as well | 00:39 |
| fretegi | did not run parted at all after the fact | 00:39 |
| RoyK | let's just hope the resync will overwrite that | 00:40 |
| fridtjof[m] | found the fix! actually makes a lot of sense now | 00:40 |
| fridtjof[m] | fixed in c9fdcf202f19fc2acdcb1ee0522ff5d61bf8c906, which is part of this patchset: https://patchwork.kernel.org/cover/10865751/ | 00:41 |
| RoyK | fretegi: https://askubuntu.com/questions/211477/how-to-remove-gpt-from-hdd | 00:41 |
| fretegi | RoyK, hate to rsyng again lol | 00:41 |
| fretegi | RoyK, ha was just reading that very article | 00:41 |
| RoyK | hehe | 00:41 |
| fretegi | RoyK, wonder if i should just rmeove this disk | 00:42 |
| fretegi | from md0, zap it and then re-enter | 00:42 |
| RoyK | just stop it, remove GPT and start over | 00:42 |
| RoyK | mdadm --fail, mdadm --remove etc | 00:42 |
| fretegi | RoyK, have to stop md0 first? | 00:43 |
| fretegi | still telling me sdc not an md device.. | 00:43 |
| RoyK | what does mdstat say now? | 00:43 |
| fretegi | still recovering | 00:43 |
| fretegi | wont let me fail or remove the drive | 00:44 |
| RoyK | mdadm /dev/md0 --fail /dev/sdc | 00:44 |
| fretegi | typo my part.. sorry | 00:44 |
| fretegi | damn.. gpt damaged... wont let me zap it | 00:49 |
| fretegi | guess can make new one and zap that | 00:49 |
| RoyK | may work | 00:49 |
| fretegi | doneski | 00:51 |
| fretegi | ok re-adding to md0... which me luck | 00:51 |
| fretegi | wish | 00:51 |
| fretegi | RoyK, well... added just fine.. back to rebuilding... no more warnings looking at fdisk | 00:52 |
| RoyK | good | 00:52 |
| fretegi | guess we wait 2.5 hours for the thing to build and cross our fingers | 00:53 |
| fretegi | ok gonna go cook some supper. appreciate you sticking around to help buddy | 00:54 |
| RoyK | well, talk to you tomorrow - it's late here | 00:54 |
| fretegi | sounds good! thanks again pal, have a great evening | 00:55 |
| fridtjof[m] | sarnold: filed a bug report! https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1895487 | 01:14 |
| ubottu | Launchpad bug 1895487 in qemu (Ubuntu) "'qemu-img convert' performance regression when target is a LVM volume" [Undecided,New] | 01:14 |
| === fretegi is now known as fretegi_away | ||
| === paride|off is now known as paride | ||
| === Wryhder is now known as Lucas_Gray | ||
| otisolsen70 | When I start up a VM with ubuntu server 12.04 LTS for some reason it does not automatically ask DHCP server for an IP Address. Instead it keeps booting up with some old IP. When I run dhclient -r enp0s3 ; dhclient enp0s3; and now it gets the right IP address from DHCP and it keeps it .... until next reboot. At which it starts up with the old again. How to fix? | 13:20 |
| tomreyn | !12.04 | 13:21 |
| ubottu | Ubuntu 12.04 LTS (Precise Pangolin) was the sixteenth release of Ubuntu. !End-of-life was April 28th 2017. See https://lists.ubuntu.com/archives/ubuntu-security-announce/2017-April/003833.html for more information | 13:21 |
| otisolsen70 | 20.04, sorry! :) | 13:21 |
| otisolsen70 | tomreyn, typo | 13:22 |
| tomreyn | oh, much better :) | 13:22 |
| === fretegi_away is now known as fretegi | ||
| tomreyn | what is it using for network management? netplan? with which renderer? | 13:23 |
| otisolsen70 | tomreyn, I have no idea. Whatever is default. Grepping for NIC name in /etc seems to indicate netplan | 13:23 |
| tomreyn | so you don't know how the network is configured, but expect dhcp to work, hmmm | 13:24 |
| otisolsen70 | tomreyn, yes, I would expect dhcp to work if nothing else is configured. | 13:25 |
| otisolsen70 | On a default install | 13:25 |
| tomreyn | so it was a fresh 20.04 installation? | 13:25 |
| otisolsen70 | Correct | 13:25 |
| tomreyn | all updates installed? | 13:26 |
| otisolsen70 | Yes | 13:26 |
| tomreyn | it must be persisting this ip address over reboots somehow, or be receiving it from the dhcp server on boot. your logs should tell which of these applies | 13:27 |
| otisolsen70 | Which logs to check? | 13:27 |
| Ussat | so, Ubuntu 18.04. MySql version 5.7.30. It seems to be hung on "/bin/bash /var/lib/dpkg/info/mysql-server-5.7.postinst configure 5.7.30-0ubuntu0.18.04.1" It LOOKS like its recreating the index. This is a 64GB database and reindexing will take a long time, I assume killing this will have very bad consiquesnces ? | 13:27 |
| tomreyn | journalctl, i'm not sure what to filter for. | 13:27 |
| Ussat | These are the mysql process running: https://pastebin.com/Q4JvUt4w | 13:28 |
| tomreyn | !info mysql-server bionic | 13:29 |
| sdeziel | Ussat: I suggest you take a look at the postinst before you decide to kill that process. I don't know myself how bad (or if at all) it would be | 13:29 |
| ubottu | mysql-server (source: mysql-5.7): MySQL database server (metapackage depending on the latest version). In component main, is optional. Version 5.7.31-0ubuntu0.18.04.1 (bionic), package size 9 kB, installed size 108 kB | 13:29 |
| otisolsen70 | I have a VM that I use for testing/dev purposes locally only. I would like it to log me in automatically in the console upon bootup. How do I do this? | 13:37 |
| RoyK | just use mariadb instead | 13:58 |
| RoyK | mysql is dead | 13:58 |
| sdeziel | otisolsen70: probably easier to setup SSH keys | 13:59 |
| otisolsen70 | sdeziel, yes, but I am asking about physical console access to the TTY | 13:59 |
| sdeziel | otisolsen70: IMHO, using the console is not a nice way to dev/test but if you really want to, that's probably doable but I don't know how, sorry | 14:00 |
| tomreyn | https://askubuntu.com/questions/819117/how-can-i-get-autologin-at-startup-working-on-ubuntu-server-16-04-1 | 14:03 |
| frickler | RoyK: if mysql is dead, why is it in main and mariadb in universe? also, would you use the latter or vendor pkgs? | 14:42 |
| frickler | jamespage: coreycb: I was wondering the same for your openstack deployments. would you support switching devstack on focal to mariadb, too? | 14:44 |
| jamespage | frickler: we're switched from percona-xtradb-cluster to mysql-8 innodb cluster as the default MySQL platform for OpenStack @ Focal | 14:45 |
| jamespage | using mysql-router for resilence | 14:45 |
| frickler | jamespage: oh, o.k., I seemed to remember something about mariadb, but then I got that misplaced somehow. so I guess we'll keep the default job on mysql | 14:47 |
| jamespage | yp | 14:47 |
| frickler | jamespage: but while I have you here, did you see my ping last week about the glibc vs. ovs bug? | 14:49 |
| frickler | https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1839592 | 14:50 |
| ubottu | Launchpad bug 1839592 in glibc (Ubuntu) "Open vSwitch (Version 2.9.2) goes into deadlocked state" [High,Confirmed] | 14:50 |
| jamespage | frickler: ah yes - I need to talk to xnox about that one on how best to proceed with a fix - backporting a whole new version is not likely to happen | 14:51 |
| frickler | jamespage: if I can help in identifying/verifying a fix, feel free to ping me. it really doesn't feel good to still run with a version from disco in prod | 14:54 |
| jamespage | I agree | 14:54 |
| === Guest49155 is now known as catalase | ||
| sarnold | fridtjof[m]: excellent work :D | 18:29 |
| coreycb | icey: I've uploaded dnspython 2.0.0+really1.16.0-2ubuntu1 | 20:05 |
| === fretegi is now known as fretegi_away | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!