mdadm (SW-RAID) and RAID6 performance. Or should I use HW-RAID?
in [Linux Hardware]
|
Prev: eSATA support in Linux / is eSATA visible as a block device?
Next: ideal operating temperature for computer equipment
From: Rahul on 5 Aug 2010 12:49 Are there any users using mdadm for RAID6? I wanted to get a performance estimate. Usually for RAID0,1,5 etc. I prefer well-tested, open-source mdadm to some sort of hardware implemented proprietary RAID card. But I've read that the checksum computation for RAID6 is computationally expensive. Hence I was wondering if it would be foolhardy to attempt a RAID6 via software-RAID. Or maybe not? It's going to be a disk-to-disk backup storage so performance is not super critical but I don't want to end up with something that crawls either. In case I am forced to go with a hardware solution, though, are there recommendations for cards that play well with Linux? I'm thinking LSI is making some good cards but if there are any caveats I'd love to hear them. The reason I'm switching over from RAID5 to RAID6 is that this time I am using eight 1-Terabyte drives in the RAID. And at this capacity I am scared by the horror stories about "unrecoverable read errors" -- Rahul
From: Aragorn on 5 Aug 2010 15:22 On Thursday 05 August 2010 18:49 in comp.os.linux.hardware, somebody identifying as Rahul wrote... > Are there any users using mdadm for RAID6? I wanted to get a > performance estimate. Usually for RAID0,1,5 etc. I prefer well-tested, > open-source mdadm to some sort of hardware implemented proprietary > RAID card. The Linux kernel supports *many* hardware RAID controllers out of the box, and most of those would fall under the "tried and tested" category. ;-) > But I've read that the checksum computation for RAID6 is > computationally expensive. Yes, it is. RAID 6 is good if you want redundancy which can cover double-disk failures, but it's the worst performer among all RAID solutions. > Hence I was wondering if it would be foolhardy to attempt a RAID6 via > software-RAID. Or maybe not? It's going to be a disk-to-disk backup > storage so performance is not super critical but I don't want to end > up with something that crawls either. I can't give you any answers on that, because it all depends on the amount and frequency of the I/O to that subsystem. However, what I *can* tell you is that a hardware RAID solution is probably going to be a lot faster, since it will be doing the checksum computations on a separate, dedicated processor, leaving your main CPU(s) available to do other stuff. Another consideration with regard to a software implementation of RAID 6 is that the speed will also be impacted by whether your kernel is preemptible or not and what interrupt timer frequency it was built with. > In case I am forced to go with a hardware solution, though, are there > recommendations for cards that play well with Linux? I'm thinking LSI > is making some good cards but if there are any caveats I'd love to > hear them. LSI, QLogic, 3Ware, Adaptec... > The reason I'm switching over from RAID5 to RAID6 is that this time I > am using eight 1-Terabyte drives in the RAID. And at this capacity I > am scared by the horror stories about "unrecoverable read errors" Why not use RAID 10? It's reliable and fast, and depending on what disk fails, you may be able to sustain a second disk failure in the same array before the first faulty disk was replaced. And with an array of eight disks, you definitely want to be having a couple of spares as well. -- *Aragorn* (registered GNU/Linux user #223157)
From: Rahul on 5 Aug 2010 17:22 Aragorn <aragorn(a)chatfactory.invalid> wrote in news:i3f31g$jum$1(a)news.eternal-september.org: > On Thursday 05 August 2010 18:49 in comp.os.linux.hardware, somebody > identifying as Rahul wrote... > The Linux kernel supports *many* hardware RAID controllers out of the > box, and most of those would fall under the "tried and tested" > category. ;-) True. But I find the user base of any HW-controller is always smaller than that of something like mdadm. > Another consideration with regard to a software implementation of RAID > 6 is that the speed will also be impacted by whether your kernel is > preemptible or not and what interrupt timer frequency it was built > with. I am running a 2.6 kernel (2.6.18-164.el51). This is supposed to have preemption. But is there a way to find out if this feature was complied into my current kernel? Same for the interrupt timer freq. How do I find it out? Based on some googling I did: cat /boot/config-`uname -r` | grep HZ # CONFIG_HZ_100 is not set # CONFIG_HZ_250 is not set CONFIG_HZ_1000=y CONFIG_HZ=1000 CONFIG_MACHZ_WDT=m [rpnabar(a)eu001 july]$ cat /boot/config-`uname -r` | grep HIGH_RES_TIMERS [no output] But not sure how to interpret this. Based on another C test code that I found online I estimate that my interrupt freq. is about 1GHz. (http://www.advenage.com/topics/linux-timer-interrupt-frequency.php) >> The reason I'm switching over from RAID5 to RAID6 is that this time I >> am using eight 1-Terabyte drives in the RAID. And at this capacity I >> am scared by the horror stories about "unrecoverable read errors" > > Why not use RAID 10? It's reliable and fast, and depending on what > disk fails, you may be able to sustain a second disk failure in the > same array before the first faulty disk was replaced. And with an > array of eight disks, you definitely want to be having a couple of > spares as well. RAID10 could work as well. But is that going to be faster on mdadm? I have heard reports that RAID10, RAID5 and RAID6 are where the HWRAID really wins over SWRAID. So not sure how to decide between RAID10 and RAID6. This is the pipeline I aim to use: [ primary server ] -> rsync -> ethernet -> [ storage server ] -> rsnapshot -> LVM -> mdadm -> SATA disk -- Rahul
From: Aragorn on 5 Aug 2010 17:52 On Thursday 05 August 2010 23:22 in comp.os.linux.hardware, somebody identifying as Rahul wrote... > Aragorn <aragorn(a)chatfactory.invalid> wrote in > news:i3f31g$jum$1(a)news.eternal-september.org: > >> On Thursday 05 August 2010 18:49 in comp.os.linux.hardware, somebody >> identifying as Rahul wrote... > >> The Linux kernel supports *many* hardware RAID controllers out of the >> box, and most of those would fall under the "tried and tested" >> category. ;-) > > True. But I find the user base of any HW-controller is always smaller > than that of something like mdadm. It depends on the market segment. For professional/ enterprise environments, the acquisition of hardware RAID equipment will be more bearable financially than for a home environment, for instance. >> Another consideration with regard to a software implementation of >> RAID 6 is that the speed will also be impacted by whether your kernel >> is preemptible or not and what interrupt timer frequency it was built >> with. > > I am running a 2.6 kernel (2.6.18-164.el51). This is supposed to have > preemption. Yes, but it may be enabled or disabled, and if enabled it could be voluntary preemption or forced preemption, and then there is the option of preempting the big kernel lock. All of these are compile-time features. > But is there a way to find out if this feature was complied into my > current kernel? Same for the interrupt timer freq. > How do I find it out? By perusing the .config for the kernel you're using, as you have done below. ;-) > Based on some googling I did: > > cat /boot/config-`uname -r` | grep HZ > # CONFIG_HZ_100 is not set > # CONFIG_HZ_250 is not set > CONFIG_HZ_1000=y > CONFIG_HZ=1000 > CONFIG_MACHZ_WDT=m > [rpnabar(a)eu001 july]$ cat /boot/config-`uname -r` | grep > [HIGH_RES_TIMERS no output] So your interrupt frequency is set to 1000 Hz, which means that if your machine has more than one CPU - with the term "CPU" referring to "a processor core", not to "a processor chip/die" - this frequency is too high for server deployment. Remember that this timer goes off for each of the CPUs in your machine. A high interrupt frequency is required for increased response towards the user, and a server aimed at high throughput does not benefit from that - on the contrary, even. The same applies to kernel preemption. Voluntary preemption is still okay, but the forced preemption (or "full preemption") used in desktop kernels is definitely not recommended for a server. > But not sure how to interpret this. > > Based on another C test code that I found online I estimate that my > interrupt freq. is about 1GHz. > (http://www.advenage.com/topics/linux-timer-interrupt-frequency.php) 1 GHz is impossible. :-) It's 1 MHz - based upon what you've pasted - but that applies for each CPU in your system. >>> The reason I'm switching over from RAID5 to RAID6 is that this time >>> I am using eight 1-Terabyte drives in the RAID. And at this capacity >>> I am scared by the horror stories about "unrecoverable read errors" >> >> Why not use RAID 10? It's reliable and fast, and depending on what >> disk fails, you may be able to sustain a second disk failure in the >> same array before the first faulty disk was replaced. And with an >> array of eight disks, you definitely want to be having a couple of >> spares as well. > > RAID10 could work as well. But is that going to be faster on mdadm? Of course it will be. There's far less overhead than with a RAID 6. > I have heard reports that RAID10, RAID5 and RAID6 are where the HWRAID > really wins over SWRAID. So not sure how to decide between RAID10 and > RAID6. > > This is the pipeline I aim to use: > > [ primary server ] -> rsync -> ethernet -> [ storage server ] -> > rsnapshot -> LVM -> mdadm -> SATA disk If you want my honest and personal advice, go with RAID 10, or perhaps a RAID 15 (i.e. a mirrored RAID 5). RAID 6 is just too slow. ;-) -- *Aragorn* (registered GNU/Linux user #223157)
From: Aragorn on 5 Aug 2010 18:17
On Thursday 05 August 2010 23:52 in comp.os.linux.hardware, somebody identifying as Aragorn wrote... > If you want my honest and personal advice, go with RAID 10, or perhaps > a RAID 15 (i.e. a mirrored RAID 5). RAID 6 is just too slow. ;-) ^^ Typo: that should read RAID 51 - think of Area 51 to help you remember. :p RAID 15 would be less efficient during an array rebuild, unless all of your disks are on the same controller, in which case it doesn't really matter. -- *Aragorn* (registered GNU/Linux user #223157) |