RoyKthe superblock is placed at drive start plus 4k https://raid.wiki.kernel.org/index.php/RAID_superblock_formats#The_version-1_superblock_format_on-disk_layout00:00
fretegiRoyK, rule out that something funky on that drive?00:00
RoyKI doubt it's a drive failure00:00
fretegiso dd like the first 5k then?00:00
RoyKif so, it's the weirdest I've seen so far00:00
RoyKjust drop a bunch of zeros there - it doesn't matter00:01
fretegiRoyK, ha well ran DD then tried to zero the superblock and says "unrecognized md component device" guess superblock gone00:02
fretegimdadm: /dev/sdc appears to be part of a raid array:00:04
fretegi       level=raid0 devices=0 ctime=Wed Dec 31 17:00:00 196900:04
freteginow thats interesting... thats from like 5 years ago.. that raid000:04
fretegihows it still picking that up, i just ran dd bs=1M count=10k00:05
RoyKfretegi: ok, put a new raid on it, some crap, and run mdadm --zero-superblock. v0.9, which might be what you had back then, is at the end of the drive00:06
RoyKsee the article I linked to above for superblock format descriptions00:07
fretegiyea i was reading00:07
RoyKfretegi: you'll need to stop the raid before removing the superblock, though, but that should be rather obvious00:08
fretegiyea but wont let me stop md1...00:09
RoyKanything in vgs/lvs/pvs?00:10
fretegiall i did was create md1, it fired right up, never added any fs or anything to it00:10
RoyKcheck lvm anyway00:10
RoyKI hate it when that happens, when automatic somethings takes over00:11
RoyKsome old fun showed up? ;)00:12
fretegiso my VG for my sdb raid (md0) is showing active on md1...00:13
fretegiand md1 is sdc which has nothing00:13
RoyKcan you pastebin that shite?00:13
RoyKtry mdadm --examine for sd[bc]00:15
fretegiif i vgremove servervg, does that perma delete it, or just effectively unmount it?00:16
RoyKis that the same name you have for the new vg?00:16
fretegisdb & sdc are part of different array UUID's00:17
fretegino thats the thing... thats the OLD vg for md000:17
fretegii didnt run any lvm commands on this new md1, just built the damn thing and thats it lol00:18
fretegii dont show a lvol running tho00:19
fretegiwait yea i do, just not mounted... but avail00:19
RoyKwell, if it doesn't contain any actual data, just remove it. But I don't know any way to undo that, so keep a backup00:20
fretegihow do i disassociate servervg with md1.... not sure how it was associated anyway00:20
RoyKiirc that "a" is "active" if you mean as in lvchange +a y00:20
fretegiwell i can do that.. my concern is... since servervg contains the lvol that is associated with md0... if i remove it here, do i remove it there?00:20
RoyKpvdisplay -v should show the uuid00:21
fretegiRoyK, actually i got the vg stopped... had to stop lvol first whoops00:23
fretegimd1 stopped now too00:23
fretegiand superblock zero'00:24
RoyKif you don't know the correct uuid or you don't know which drive is which (linux can shuffle around disk names from time to time), write down the UUIDs in use now and whatever's linked to in /dev/disk/by-id/ to that to get the full path00:24
fretegiwere good now00:24
fretegisuperblock zero'd out00:24
RoyKif not, well, I hope you have a good backup :)00:24
fretegioh i do00:25
fretegias of this am00:25
fretegiso what form here?00:25
RoyKany active raids now?00:25
fretegiwell, md0 can be restarted00:25
RoyKcan you assemble the original?00:25
fretegii can do that, just has to be manually00:25
RoyKdoesn't matter00:26
fretegimd0 fired up with just one drive00:26
RoyKdid lvm find it? does the fs mount (try read-only if you're nervous)00:27
fretegii had to manually mount it all but yes, vg's & lvol's all properly associated with md0 and mounted to the proper directy and data appears to be intact00:28
fretegiso what of fixing md0 ya think?00:29
RoyKthen just create a new r0 on the other drive (sdc?) and stop the raid, then --zero-superblock that drive - it may remove the v(something old) superblock. you may have to specify superblock version - never done that with superblock removal.00:29
fretegibuild new array like i was starting to do, rsync stuff over and ditch md0?00:29
fretegidid that already00:30
fretegistopped md1 and zero's sdc00:30
fretegiyea wont let me assemble md1 now as superblock gone00:31
fretegiso back to a blank drive...00:31
RoyKtry --assemble --force00:31
RoyKjust for kicks00:31
fretegiassembly abored00:31
fretegiwith both --run and --force00:32
fretegimd1 commented out of mdadm.conf too00:32
RoyKthen, theoretically, it should work fine with just adding sdc to the array00:32
RoyKjust don't put a bloody partition table on there afterwards00:33
fretegido i need to someone let mdadm know that the prior 2nd disk in md0 is toast?00:33
fretegistill thinks its supposed to have 2, just running on 100:33
RoyKno, it won't find a superblock, so it'll treat it as a new one00:33
fretegialright... adding her up here we go00:34
RoyKit should start resync by itself00:34
fretegiback to recovering00:34
fretegitime #600:34
fretegiok in an other 2.5 hours... ill re updated initramfs and mdadm.conf and see if good00:35
RoyKthe other way around00:35
RoyKbut it'll probably be the same data00:35
fretegiyea should be00:35
fretegiarray UUID same00:35
RoyKthe uuid doesn't change with a disk replacement00:36
fretegisure giving these drives a work out..00:36
RoyKshouldn't hurt00:36
RoyKwhat sort of drives are these, btw?00:36
fretegisata desktop drives lol00:37
fretegigood ones... but still.. not enterprise stuff... just seagate desktop stuff00:37
RoyKany support for SCTERC? ;)00:37
RoyKsmartctl -l scterc /dev/sdX00:38
fretegiseagate 2 tb soemthing or other00:38
freteginot supported00:38
fretegiuh oh... heres something funky....00:38
RoyKcan you pastebin the full smartctl -a for those two drives?00:39
fretegiran fdisk -l and saw that.....00:39
fretegiremember we DD'd it and --zero-superblock'd it as well00:39
fretegidid not run parted at all after the fact00:39
RoyKlet's just hope the resync will overwrite that00:40
fridtjof[m]found the fix! actually makes a lot of sense now00:40
fridtjof[m]fixed in c9fdcf202f19fc2acdcb1ee0522ff5d61bf8c906, which is part of this patchset: https://patchwork.kernel.org/cover/10865751/00:41
RoyKfretegi: https://askubuntu.com/questions/211477/how-to-remove-gpt-from-hdd00:41
fretegiRoyK, hate to rsyng again lol00:41
fretegiRoyK, ha was just reading that very article00:41
fretegiRoyK, wonder if i should just rmeove this disk00:42
fretegifrom md0, zap it and then re-enter00:42
RoyKjust stop it, remove GPT and start over00:42
RoyKmdadm --fail, mdadm --remove etc00:42
fretegiRoyK, have to stop md0 first?00:43
fretegistill telling me sdc not an md device..00:43
RoyKwhat does mdstat say now?00:43
fretegistill recovering00:43
fretegiwont let me fail or remove the drive00:44
RoyKmdadm /dev/md0 --fail /dev/sdc00:44
fretegitypo my part.. sorry00:44
fretegidamn.. gpt damaged... wont let me zap it00:49
fretegiguess can make new one and zap that00:49
RoyKmay work00:49
fretegiok re-adding to md0... which me luck00:51
fretegiRoyK, well... added just fine.. back to rebuilding... no more warnings looking at fdisk00:52
fretegiguess we wait 2.5 hours for the thing to build and cross our fingers00:53
fretegiok gonna go cook some supper.  appreciate you sticking around to help buddy00:54
RoyKwell, talk to you tomorrow - it's late here00:54
fretegisounds good!  thanks again pal, have a great evening00:55
fridtjof[m]sarnold: filed a bug report! https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/189548701:14
ubottuLaunchpad bug 1895487 in qemu (Ubuntu) "'qemu-img convert' performance regression when target is a LVM volume" [Undecided,New]01:14
=== fretegi is now known as fretegi_away
=== paride|off is now known as paride
=== Wryhder is now known as Lucas_Gray
otisolsen70When I start up a VM with ubuntu server 12.04 LTS for some reason it does not automatically ask DHCP server for an IP Address. Instead it keeps booting up with some old IP. When I run dhclient -r enp0s3 ; dhclient enp0s3; and now it gets the right IP address from DHCP and it keeps it .... until next reboot. At which it starts up with the old again. How to fix?13:20
ubottuUbuntu 12.04 LTS (Precise Pangolin) was the sixteenth release of Ubuntu. !End-of-life was April 28th 2017. See https://lists.ubuntu.com/archives/ubuntu-security-announce/2017-April/003833.html for more information13:21
otisolsen7020.04, sorry! :)13:21
otisolsen70tomreyn, typo13:22
tomreynoh, much better :)13:22
=== fretegi_away is now known as fretegi
tomreynwhat is it using for network management? netplan? with which renderer?13:23
otisolsen70tomreyn, I have no idea. Whatever is default. Grepping for NIC name in /etc seems to indicate netplan13:23
tomreynso you don't know how the network is configured, but expect dhcp to work, hmmm13:24
otisolsen70tomreyn, yes, I would expect dhcp to work if nothing else is configured.13:25
otisolsen70On a default install13:25
tomreynso it was a fresh 20.04 installation?13:25
tomreynall updates installed?13:26
tomreynit must be persisting this ip address over reboots somehow, or be receiving it from the dhcp server on boot. your logs should tell which of these applies13:27
otisolsen70Which logs to check?13:27
Ussatso, Ubuntu 18.04. MySql version 5.7.30. It seems to be hung on "/bin/bash /var/lib/dpkg/info/mysql-server-5.7.postinst configure 5.7.30-0ubuntu0.18.04.1" It LOOKS like its recreating the index. This is a 64GB database and reindexing will take a long time, I assume killing this will have very bad consiquesnces ?13:27
tomreynjournalctl, i'm not sure what to filter for.13:27
UssatThese are the mysql process running: https://pastebin.com/Q4JvUt4w13:28
tomreyn!info mysql-server bionic13:29
sdezielUssat: I suggest you take a look at the postinst before you decide to kill that process. I don't know myself how bad (or if at all) it would be13:29
ubottumysql-server (source: mysql-5.7): MySQL database server (metapackage depending on the latest version). In component main, is optional. Version 5.7.31-0ubuntu0.18.04.1 (bionic), package size 9 kB, installed size 108 kB13:29
otisolsen70I have a VM that I use for testing/dev purposes locally only. I would like it to log me in automatically in the console upon bootup. How do I do this?13:37
RoyKjust use mariadb instead13:58
RoyKmysql is dead13:58
sdezielotisolsen70: probably easier to setup SSH keys13:59
otisolsen70sdeziel, yes, but I am asking about physical console access to the TTY13:59
sdezielotisolsen70: IMHO, using the console is not a nice way to dev/test but if you really want to, that's probably doable but I don't know how, sorry14:00
fricklerRoyK: if mysql is dead, why is it in main and mariadb in universe? also, would you use the latter or vendor pkgs?14:42
fricklerjamespage: coreycb: I was wondering the same for your openstack deployments. would you support switching devstack on focal to mariadb, too?14:44
jamespagefrickler: we're switched from percona-xtradb-cluster to mysql-8 innodb cluster as the default MySQL platform for OpenStack @ Focal14:45
jamespageusing mysql-router for resilence14:45
fricklerjamespage: oh, o.k., I seemed to remember something about mariadb, but then I got that misplaced somehow. so I guess we'll keep the default job on mysql14:47
fricklerjamespage: but while I have you here, did you see my ping last week about the glibc vs. ovs bug?14:49
ubottuLaunchpad bug 1839592 in glibc (Ubuntu) "Open vSwitch (Version 2.9.2) goes into deadlocked state" [High,Confirmed]14:50
jamespagefrickler: ah yes - I need to talk to xnox about that one on how best to proceed with a fix - backporting a whole new version is not likely to happen14:51
fricklerjamespage: if I can help in identifying/verifying a fix, feel free to ping me. it really doesn't feel good to still run with a version from disco in prod14:54
jamespageI agree14:54
=== Guest49155 is now known as catalase
sarnoldfridtjof[m]: excellent work :D18:29
coreycbicey: I've uploaded dnspython 2.0.0+really1.16.0-2ubuntu120:05
=== fretegi is now known as fretegi_away

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!