[01:04] cfdisk -- FATAL ERROR: Bad primary partition 1: Partition ends in the final partial cylinder -- whats wrong? [01:04] its a fresh machine, and i installed ubuntu server on one of the hard drives [01:05] pretty much exactly what it says [01:06] your partition doesn't end on a cylinder boundary [01:06] which may or may not be a problem [01:06] depending on how smart your software is [01:06] qman__: Why would that happen with a fresh install? [01:06] qman__: also, how do i find out if its a problem or not? [01:07] the installer probably used software which is smart enough to handle partitions that don't end on cylinder boundaries, while the software you're using is not [01:07] I see [01:07] cfdisk is what i was trying to use [01:07] that, or the software you're using is trying to ensure everything is aligned to 4k sectors [01:07] I'm trying to build a RAID5 with 4 disks apart from the disk i installed the OS on [01:07] cfdisk is probably just being paranoid and living in the past [01:08] I don't use cfdisk myself, so I honestly have no idea if it's a problem or not [01:08] I can ssh to the machine...was trying to use mdadm...thought would look at the disks using cfdisk [01:08] I just fdisk, n, p, 1, enter, enter, t, fd, w [01:08] just added five disks to my file server the other night that way [01:08] what exact command did you use? [01:09] did you build a raid on them? [01:09] fdisk /dev/sd?, where ? is the disk you want [01:09] and then using those letters in order in fdisk to create the partitions of the correct type [01:09] and then an mdadm command to add the disks to the array (or in your case create an array) [01:09] and then I grew my ext3 filesystem to fit after the array reshaped [01:10] on a new array you'd just make a new filesystem, or use LVM or whatever you're doing [01:10] k - question 1: how do i find out which disks are empty and what are their names...i usually use gnome-disk-utility for this, but dont know how to do this using text terminal [01:10] fdisk -l [01:11] WARNING: GPT (GUID Partition Table) detected on '/dev/sdh'! The util fdisk doesn't support GPT. Use GNU Parted. [01:11] AFAIK fdisk does not do 4k sectors by default [01:11] GPT, never done that myself [01:11] 'parted' is GNU Parted [01:11] I don't know the exact commands in parted to do what you want though [01:12] ok /dev/sde is where my os is installed -- need to make raid5 from fghi [01:12] ya, I don't think anyone is really doing partitions by 4k, but mainly by 1meg now [01:13] you need to create partitions of type "Linux RAID Autodetect", or 'fd' [01:13] then mdadm can deal with them [01:14] http://paste.pocoo.org/show/422939/ [01:14] yeah, you're using GPT, so fdisk won't work [01:14] you'll have to use a GPT aware partitioner [01:14] use parted [01:15] or if you have gui installed, gparted? [01:15] ooi, why are you using GPT? [01:15] I've no clue [01:15] I'm only using gpt anymore [01:15] I have no need for it [01:16] my data disks are all one big partition anyways [01:16] I have many raids that are >2tb [01:16] most of mine are hardware raided to >2tb [01:16] my disks are 2TB each [01:16] 4 disks -- want to put them in RAID 5 [01:16] what parted command do i need to run on each disk ? [01:17] well, you can just wipe the gpt off, and use fdisk [01:17] be simpler [01:17] I use 1TB disks right now [01:17] seems parted is better that fdisk in anycase? usage wise [01:17] next one will be a major overhaul [01:17] probably [01:17] I only suggested fdisk because that's what I know and use [01:17] I do not know parted [01:17] fdisk is a dump partitioner, it does what you say [01:18] parted is smarter [01:18] cant i just use "parted /dev/sdf" and then mkpart primary 0 2001G? [01:18] or is there a better way? [01:24] qman__: ok i do have partitions on the 4 disks i want to put in raid 5: http://paste.pocoo.org/show/422951/ [01:24] will that work? [01:24] patdk-lap: is this partitioning any good or do i need to repartition in some other way for mdadm? [01:25] well, they shouldn't have filesystems on them [01:25] mdadm doesn't care :) [01:25] I don't know what exactly that means in parted though [01:25] patdk-lap: what mdadm command do i need to create the raid 5 from 4 disks then? [01:26] mdadm --create /dev/md0 --level=5 --raid-devices=4 /dev/sd?1 /dev/sd?1 /dev/sd?1 /dev/sd?1 [01:27] you might want to tweak other options like stripe size [01:27] but that'll create one with default settings [01:28] stripe size is good, depending what what it is used for [01:28] I guess in my case I'll try this then: sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=4 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/sdi1 [01:28] my array is used mostly to store large (1GB+) files, so I increased my stripe size [01:29] -c 128 is nice [01:29] -c 1024 is good also for large streaming files [01:29] http://paste.pocoo.org/show/422955/ ? [01:29] the other thing is, you'll want to tweak your ext3/4 filesystem based on your stripe size [01:30] most of my files are a few MBs [01:30] like 10 MBs or so [01:30] more detailed info here: http://ubuntuforums.org/showthread.php?t=408461 and here: http://tldp.org/HOWTO/Software-RAID-HOWTO.html [01:30] 512K default should be fine for that, isnt it? [01:30] it depends on how it's used [01:30] when those files change [01:30] the latter is very old, but the information is still relevant, just the commands have changed some [01:30] does the while file change? or part of it? [01:31] most of my files are read only [01:31] than large is ok [01:31] how do i change the strip size to 1MB? [01:31] I would stick with 512k probably though [01:31] perfect [01:32] mdadm: /dev/sdf1 appears to contain an ext2fs file system [01:32] you don't want to go too large and cause a single drive to slow it down :) [01:32] how do i goto ext4 or some other fs? [01:32] you don't [01:32] each drive partition should not contain a filesystem [01:32] it should overrite it with the raid [01:32] they should just be partitions [01:32] the filesystem should be created on top of the RAID [01:32] perfect [01:32] that's why it's warning you [01:32] so you don't accidentally nuke a filesystem you shouldn't have [01:32] says /dev/md0 started [01:32] now i can get a FS on it? [01:33] you can either create a filesystem directly on md0, or use LVM [01:33] now watch in /proc/mdstat for progress [01:33] if you care :) [01:33] that ^^ [01:33] though it's designed to handle whatever you throw at it [01:33] I prefer to stay on the safe side, and ensure not to trust the raid until that's done [01:33] 5min speed=90111K/sec!? [01:33] it nice to make sure it gets to 1% I think, before messing with it :) [01:34] not reboot the server, not fill more space on the array than the % completed [01:34] .4% right now [01:34] it takes a long time [01:34] qman, I go both ways [01:34] done .5% [01:34] sometimes I wait till it completes 100% before anything [01:34] yep [01:34] other times I just don't [01:34] ok, so we dont add the FS onto it till its done 100%? [01:34] I did when reshaping my array [01:34] because I don't have a backup [01:34] pythonirc, it's fine now, it's just going be slow [01:34] you can create the fs [01:35] just don't start dumping files onto it [01:35] at least until it gets a bit through [01:35] sorry got booted [01:36] you can create the fs [01:36] just don't start dumping files onto it [01:36] at least until it gets a bit through [01:36] its 1% done [01:36] what command do i need to create the fs? [01:36] while it will let you put more data on the array than is allocated [01:36] what exactly it does with that data is a mystery to me [01:36] qman, it's safe :) [01:37] it just goes ahead and inits that part of the time [01:37] drive [01:37] technically you are doing that ways [01:37] cause ext throws the data randomly over the drive [01:37] mkfs.ext4 [01:37] but this also assumes it's bug free :) [01:38] though you should set the stride based on your stripe size for best performance [01:39] the numbers are much smaller because the document is old, but this section explains it well: http://tldp.org/HOWTO/Software-RAID-HOWTO-5.html#ss5.11 [01:40] also, chunk size == stripe size [01:40] qman__: mkfs.ext4 /dev/md0 ? can you please tell the exact command [01:40] or mkfs -t ext4 /dev/md0? [01:40] both are equivalent [01:40] and thats what i need? [01:41] writing inode tables [01:41] mkfs.ext4 -T largefile -E stride=512,stripe-width=3 /dev/md0 [01:41] i've 4 disks, strip-width=3 is fine? [01:42] you have 3 data disks [01:42] 4, right? [01:42] you have 3 data disks [01:42] oh one is parity? [01:42] yes [01:43] patdk-lap: thanks [01:43] now mount ? or wait? [01:44] you can mount and use it normally, but it will be slow until it is done building [01:44] 2.8% done in /proc/mdstat [01:44] perfect. [01:44] ok, so moving on to the next raid, which is even worse... [01:44] I haven't created a file server since ext4 got stable, so I didn't know the command [01:44] i hope it doesnt crash :) [01:45] you can lower your build time with the following [01:45] +1 for -binternal [01:45] echo "200000" | sudo tee /proc/sys/dev/raid/speed_limit_min [01:46] when i run parted, and do print all : Error: Invalid partition table on /dev/mapper/sil_bhagdgacbhai -- wrong signature 0. Ignore/Cancel? -- Why does this happen? [01:46] qman, not going help him much, it's already going 90MB/s the max those drives go :) [01:46] ah [01:46] didn't notice that [01:46] that happens because you're using fakeraid [01:47] with nothing on it [01:47] can i just make a fs on it? [01:47] yes [01:47] ok, next question [01:47] /dev/mapper/sil_bhagdgacbhai -- 160GB is a 4 x 40GB OCZ drives [01:48] what FS should i use on it? what raid? I was hoping that this box would show up as one drive on ubuntu, but it does not. I am hoping to use RAID 0 on this one. [01:48] fakeraid is usually terrible [01:48] what type of controller is it? [01:49] pythonirc1012: that sounds like some bullshit fakeraid controller, which you should avoid like the plague [01:49] or more importantly [01:49] it is :) [01:49] http://www.provantage.com/ocz-technology-ocz3hsd1ibs1-160g~7OCZD018.htm [01:49] does it have an actual chip [01:49] Also, don't use RAID0 for anything but ephemeral data. [01:49] yuck [01:50] pythonirc, you should have turn off raid in your bios [01:50] so you didn't have that sil_xxxxx thing, but instead a normal sd? [01:50] pythonirc1012: unless you paid $400 for your RAID card, you should use Linux md RAID. [01:50] no, it's a black box SSD with a fakeraid built into it [01:50] so he's not going to be able to turn it off [01:50] oh? [01:50] Anything less than that won't be battery-backed hardware raid, but rather some kind of fakeraid. [01:50] qman__: lame [01:51] qman__: I think i can turn it off [01:51] normally they don't have fakeraid in it, but that sandcontroller thing [01:51] patdk-lap: sandforce, it's an FTL [01:51] I'll try to switch it off tomorrow [01:51] oh, it [01:51] 's a ibis [01:51] its at work, cant do it from home [01:52] what do you guys suggest for it? switch off the fake raid 0 , and then make a raid 0 using mdadm? [01:52] ordinarily yes, but I don't think you'll be able to do that with that hardware [01:52] oh ya, they do display as a sil_xxx fakeraid, how evil [01:52] that, or buy a real RAID controller with dedicated CPU and cache [01:53] qman, won't work, it has a custom connection [01:53] the drive is actually pcie x4 [01:53] yeah [01:53] i saw it in the bios to turn off the raid on that [01:53] why are you guys saying it wont work? the controller? [01:54] pythonirc, too much unknown, I haven't used one yet [01:54] cause they aren't supported by vmware :) [01:54] patdk-lap: lets assume i've 4 40GB SSD drives that are visible to the os as /dev/sd? [01:54] just raid0 them :) [01:54] using mdadm [01:54] yep [01:55] fakeraid loads all the effort onto your main CPU anyway [01:55] you might as well use the most stable and optimized software raid, mdadm, instead of a one-off driver [01:55] just 512K default stripe? [01:55] what FS? [01:55] whatever fs you want [01:56] any that works well with SSDs? shouldnt i try to match the stripe length with whatever size the IO of SSDs are optimized for 4k? [01:57] who raid the ssd's are optimized for 4k [01:57] I can tell you for sure they aren't [01:58] 64k, 128k or 256k maybe [01:58] so 512k is probably best [01:58] patdk-lap: the IOPS they report is for 4k chunks, right? [01:58] sometimes [01:58] then? [01:58] i need iops from this device [01:58] if it's done by a proper benchmarking that is reputable, 4k yes [01:59] stripesize doesn't offect iops [01:59] man you really have no idea what your doing [01:59] you have no idea how flash works [01:59] your assuming it works the same as normal drives [01:59] and even on normal drives you wouldn't use 4k strip size [01:59] thats right strip size doesnt have any effect on IOPS [01:59] stripe size does affect iops [01:59] mdadm then should work fine, even if i want most for iops [01:59] it affects it a crapload [02:00] does the FS have any effect on IOPS? what do benchmarkers use when they look into increasing iops? [02:00] the only way to increase iops is to add more drives [02:01] to maximize iops per set raid [02:01] also: if i want to mount /dev/md0 at boot to /raid, do i just add it to /etc/fstab? or is there an easier way to do this automatically? [02:01] you want to max out the read buffer of the drive [02:01] most drives it's like 16 or 32 sectors [02:01] or used to be [02:01] but flash is different [02:01] patdk-lap: in my case, i've 4 SSDs, so that should already give good IOPS, if i read randomly all over, isnt it? [02:02] you can write to flash, 8k at a time [02:02] my app is mostly read only...no writes [02:02] but if you have to erase a sector, that sector for flash can be 16k to 512k [02:02] that large block size is what you want to optimize for in flash stripe size [02:02] cause you don't want to be writing half of that empty space at a time [02:03] and you defently want to get it aligned [02:04] nice raid now shows 5.2T free, thanks [02:04] patdk-lap: is there an easy way to add more drives to the RAID 5 that i just created using mdadm? [02:04] yep [02:04] how do i do that? [02:04] google :) [02:05] will i have trouble with ext4? [02:05] mdadm --add /dev/md1 /dev/sdj1 ? [02:05] pythonirc1012: you cannot add drives to a RAID5 or 6 array. [02:05] what happens to the ext4? [02:05] http://ubuntuforums.org/archive/index.php/t-517282.html [02:05] nothing [02:06] after the raid is fully grown [02:06] then you grow the fs [02:07] so, mdadm --add /dev/md0 /dev/sd?? [02:07] mdadm --grow /dev/md0 --raid-devices=5 [02:07] resize2fs /dev/md0 [02:07] nice [02:07] so twb is wrong then? [02:07] That is, you can add hot spares, and replacements, but not expand the array's effective size. [02:07] yep [02:07] sure you can, you have been able to for years [02:08] since like 05 [02:08] patdk-lap: hmm, maybe that's an md feature [02:08] yep we are talking mdadm :) [02:08] That rings a bell, actually [02:08] In the general case of the RAID5 algorithm, you can't [02:08] heh, try to find a raid card that doesn't let you these days :) [02:08] I expect mdadm does some rebalancing stuff [02:09] no idea [02:10] patdk-lap: thanks a lot for all the help today. [02:10] qman__: Thank you as well. [02:11] my raid is almost built 10% now :) [02:17] pythonirc1012: if you assemble the array with -binternal, it will be have a "write-intent bitmap", which is the RAID equivalent of a journal [02:18] It means that after a clean reboot or an unexpected outage, it won't have to resync the entire array (which will take hours for 2TB drives) [02:19] it's 10% done after like 30min [02:20] I don't think he should worry that much :) [02:21] Shrug [02:21] It's more for outages [02:21] that is what ups's are for :) [02:25] And when did you last replace the batteries of yours :-P [02:25] last month [02:26] cost me $300 [02:26] Nice [02:27] How many kWa you get for $300? [02:27] 3kw [02:27] but that is with an extra battery pack [02:27] so like 38min runtime full load [02:28] 3 sounds a lot lower than the numbers ISTR [02:28] heh? [02:28] For a fully populated 19" rack, we were looking at more like $1500 second-hand and $8000 new [02:28] its only done 12.5% now with 512mins remaining , and the speed is down to speed=55533K/sec [02:29] (Including the device as well as the batteries, but I imagine the batteries are 80% of the cost) [02:29] for a fully populated 19" rack, I have two 5kw's [02:29] perhaps i really need -binternal? [02:29] pythonirc, if you want, I can't imagine that wouldn't default to that though [02:29] http://paste.debian.net/121340/ are my old notes [02:30] patdk-lap: the WI bitmap won't make the initial sync faster [02:30] Sorry, bad completiong [02:30] pythonirc1012: the WI bitmap won't make the initial sync faster [02:30] pythonirc1012: it'll just mean that you don't need to do a full resync again [02:30] looks like your old notes wanted like a 2kw [02:31] is there a way to add the WI bitmap to a prebuilt/mounted raid ? [02:31] Probably I confused W with kW [02:31] (I'm a bit distracted because the financials Windows VM is on its last legs, and it's the EOFY [02:32] 3000va ups :) [02:32] If I have an mdadm array with v1.2 metadata, can I downgrade in place to 1.0 metadata? [02:33] Also, what is the latest metadata version supported by hardy (2.6.24)? [02:33] I don't think so [02:33] I can check hardy [02:33] please do, thanks [02:33] I can just blow away /boot to fix syslinux (which needs >= 1.0), but I don't want to have to blow away the main lvm one :-/ [02:34] 1, 1.0, 1.1, 1.2 [02:35] everytime i ssh to a remote machine i have to type my passphrase. how to automate this? [02:35] ssh-agent [02:35] owen1: ssh-copy-id [02:35] pythonirc, copy-id only changes it from a password to a privatekey-password :) [02:36] patdk-lap: doesnt it make it passwordless? [02:36] i just generate a key and copy it around [02:36] it makes ssh passwordless [02:36] but ssh has to ask for the privatekey password [02:36] it just neversends that password over tcp [02:37] is that bad to use? [02:37] no [02:37] but ssh-agent is what he wanted [02:37] you type your password into the agent [02:37] then when ssh needs to use your key file, it already knows the password [02:37] patdk-lap: looking into ssh-agent [02:37] patdk-lap: fantastic, thanks [02:38] I'm on zoneedit and i've a domain forwarded to a static ip-- what is TTL? Type (A,AAAA,CERT,MX,...)? [02:39] pythonirc101: BITMAP CHANGES [02:39] A write-intent bitmap can be added to, or removed from, an active array. Either internal bitmaps, or bitmaps stored in a separate file can be added. Note that if you add a bitmap stored in a file which is in a filesystem that is on the raid array being affected, the system will deadlock. The bitmap must be on a separate filesystem. [02:39] pythonirc1012, how long that entry is valid for, till a recheck is required [02:39] patdk-lap: in my case i can easily add that since i've a separate 2TB drive. [02:40] is 7200 a good number for TTL ? which is better smaller or larger? [02:40] depends [02:40] if you change it often, smaller [02:40] if you don't, larger [02:40] I just use 86400 [02:40] i dont change it at all [02:41] like past 5 years, it was the same map [02:42] mdadm --grow --bitmap=internal /dev/md0 [02:42] patdk-lap: ah, nice, I only ever passed -b to assemble and create [02:42] just that? [02:43] In a way, it's a pity mdadm will be subsumed by btrfs [02:43] yep [02:43] where is the bitmap stored? [02:43] twb, not really, depends on workload [02:43] raid itself? [02:43] yep === jkakar_ is now known as jkakar [02:43] twb, I'm sure over years, people will get used to using btrfs by default though [02:44] patdk-lap: #mdadm --grow --bitmap=internal /dev/md0 --- says --- mdadm: failed to set internal bitmap. [02:45] probably cause your raid isn't built yet [02:45] it needs to be stable first [02:45] try it tomorrow :) [02:45] 15% , will try tomorrow, thanks [02:46] patdk-lap: I'm trying to put a new server into my system so that i can take the load off an older machine [02:46] looks like it can slow down the raid 10% though, for writes [02:46] i would like to distribute the network load for the http server on these two machines [02:46] the bitmap! that slow? [02:47] ya [02:47] k, i'll skip it then :0 [02:47] before it writes anywhere on the raid, it has to update the bitmap that it is going to [02:47] then when it's done, clear that bitmap [02:47] and that bitmap is mirrored on all raid drives [02:47] patdk-lap: re 10%, I have seen people saying "no problems here" and people saying "cripples my raid" [02:48] I'm going by mdadm's numbers on their doc [02:48] https://raid.wiki.kernel.org/index.php/Write-intent_bitmap [02:48] patdk-lap: perhaps i should make the bitmap on my SSD RAID then? [02:48] how much space does it neeD? a few bits per 512K page? [02:48] * twb reads [02:48] I imagine that could kill an ssd fast [02:48] Ah, "up to 10%" [02:49] pythonirc, this only affects you when you pull the power wire out of the server [02:49] patdk-lap: if it does, it goes back to OCZ, with a thank you :) [02:49] otherwise the bitmap won't *help* [02:49] 3 years warranty must count for something [02:49] patdk-lap: I'll skip...my power is pretty stable...once a year i get a power outage. [02:50] heh, I've gone 5 years without a power outage [02:50] till this year [02:50]