Configurations

From 45 Drives Technical Information Wiki
Jump to: navigation, search

There are many ways to configure storage devices depending on the need determined by each particular situation. This page will provide descriptions and how to set-up for a few common configurations. If you require information for a certain configuration not listed here, please email info@45drives.com.

When choosing what storage configuration is right for your certain application.....

See our blog post on how to decide on the right RAID level for you: http://45drives.blogspot.ca/2015/11/how-to-decide-on-best-raid_11.html

Standard RAID Levels

In terms of performance the best to worst RAID levels is as follows:

Note: The following descriptions are referencing Linux software RAID, "mdadm" and "btrfs"

RAID0 (stripe) This stripes the data across all of your disks, this allows to pull data in parallel dramatically increasing performance. The caveat is that if one disk fails, then your whole array is lost. It is not recommended that RAID0 is used by itself or without proper backups in place.

RAID1 (mirror) This mirrors data across two or more disks (typically two). There is no read penalty however there is a write penalty as every write operation needs to be done twice (once to both disks). Mirroring disks is a reliable form of redundancy; if a disk fails you have a exact copy you can rebuild your mirror from.

RAID6 (Dual Parity) This uses a combination of striping data and parity across all of the disks in the array. This has the benefit of striped read performance and redundancy; you can lose up to 2 disks in the array and still be able to rebuild lost data. The caveat is there is a severe write penalty and you will see approximately 20% reduction in performance when writing. One other important thing to keep in mind when using RAID6 is that the initial synching/rebuild times of the array will take a long time depending on the size of your array and your CPU frequency. For reference a 90 TB array will take approximately 20 hours to build.

Note: The following descriptions are referencing Linux software RAID, "ZFS"

RAIDZ1 (Single Parity) This uses a combination of striping data and parity across all of the disks in the array. This has the benefit of striped read performance and redundancy; you can lose 1 disk in the array and still be able to rebuild lost data. There is a write penalty due to the parity, but it does provide an extra level of protection that a RAID0 would be provide.

RAIDZ2 (Dual Parity) This is the ZFS equivalent to a RAID6. See RAID6 explanation above.

RAIDZ3 (Triple Parity) This uses a combination of striping data and parity across all of the disks in the array. This follows the same description as RAIDZ1 and RAIDZ2, but with triple parity you can lose up to 3 disks in the array and still be able to rebuild lost data. There is a write penalty like the others, but as you increase the number of parities, the penalty also increases.


Building common RAID Levels

To build a nested RAID you have to use an even number of disks.

Note: The following are built using the "mdadm" configuration.

RAID0+1 (Mirror of Stripes)

Raid01.png

With a mirror of stripes you can expect the improved performance of working with striped array as well as the redundancy of having them mirrored. A caveat though, when a drive fails the rebuild time will increase depending on how big your striped array is. Another important note is that since you have to have an exact copy of your array to make a mirror your effective storage capacity is half of the RAW storage. For example 44 4TB disks in a RAID01 would only have a usable capacity of 88TB (176TB Raw)

RAID1+0 (Stripe of Mirror)

RAID 10 01.svg.png

With a stripe of mirrors you get the redundancy since each disk has an exact clone as well as the improved performance since you have multiple mirrors in a stripe. Unlike RAID01, the rebuild times on a RAID10 are significantly less. Since the mirroring is per drive rather than per stripe, a failed drive drive would only need to copy over one drives worth of contents rather than the whole stripe. It is important to note that like a RAID01 your effective storage capacity is half of the RAW storage. For example 44 4TB disks in a RAID10 would only have a usable capacity of 88TB (176TB Raw)

RAID6+0 (Stripe of RAID6)

RAID 60.png

With a stripe of RAID6 you get redundancy and improved space efficiency, but will see significant performance drops when writing to the array. Due to the RAID 6 write penalty mentioned above.
If you had 44 4TB disks in a RAID60, with a width of two (i.e two RAID6s stripped) you would have a usable space of 160TB (2*(22 - 2)*4TB (176 RAW). This is a storage efficiency of 90.1%. Note that you lose two data disks for parity per RAID6 array.


Note: Below is a guide on how to build a "btrfs" RAID.


Btrfs RAID

"Btrfs" RAID supports the standard RAID levels, RAID0, RAID1, RAID5, RAID6, and RAID10. One major benefit to "btrfs" RAID is the ability to add devices to the RAID after it is created. Another benefit is the instant build time. As mentioned above, a "mdadm" RAID6 could take several hours to build, whereas a "btrfs" RAID6 builds instantly.


Note: Below is a guide on how to build a ZFS RAID.


ZFS RAID

A ZFS RAID consists of three layers, from bottom to top they are, the zpool, the VDevs, and the data sets in the file system. A VDev is one or more hard drives, usually in RAID formats intended to store data together. The combination of all the VDevs make up what is called the zpool. With ZFS, you cannot add more drives to a VDev, but you can always add more VDevs to the zpool to increase the storage capacity. A third advantage is that there is no wait while building. Similar to a "btrfs" RAID, the RAID is built instantly.




LVM Configurations

Note: You will use the mdadm or btrfs RAIDs as the physical devices in the examples below. ZFS does not apply to LVM.

  • 3 RAID + 3 LVM In this configuration, we use 3 separate RAID arrays with 15 drives each. These are then formatted with 3 seperate LVM groups so you are essentially left with 3 separate and distinct volumes for use.
  • 3 RAID + 1 LVM In this configuration, we use 3 separate RAID arrays with 15 drives each. These are then formatted with 1 LVM group so that you have one large storage volume.

Performance of various RAID levels

Shown below are the results from testing the standard Linux "mdadm" RAID vs the "btrfs" raid.

All testing was performed on a base level Q30 model here at 45 Drives.

The benchmark tool used was iozone. Tests consisted of a 1M sequential read/write of a 20GB file.
(iozone -t x -i 0 -i 1 -r 1M -s 20G ; 0<x<10)

  • 25 Drive RAID 0
Raid0-25Drive-Write.jpg


Raid0-25Drive-Read.jpg
  It can be seen that a RAID 0 has great read and write speeds reaching well over 1 GB/s. 
  One thing to consider with RAID 0 is that if one drive fails, your data is lost. 
  • 20 Drive RAID 6
Raid6-20Drive-Write.png


Raid6-20Drive-Read.png
 It can be seen that the performance of a RAID 6 is approximately three times better when using a "btrfs" RAID.
 Another positive to using the "btrfs" RAID 6 is the instantaneous build time compared to a "mdadm" RAID which takes approximately seven hours to build.
  • 20 Drive RAID 10
Raid10-20Drive-Write.png


Raid10-20Drive-Read.png
 RAID 10 is seen as one of the faster RAID configurations that provides a level of data security as well. Seen here the speeds reach approximately 1 GB/s.


Performance of ZFS RAID

Shown below are results from testing various zpool sizes along with various zfs RAID configurations.

All testing was performed on a base level Q30 model here at 45 Drives.

The benchmark tool used was iozone. Tests consisted of a 1M sequential read/write of a 20GB file.
(iozone -t x -i 0 -i 1 -r 1M -s 20G ; 0<x<10)

  • RAIDZ1

The graph shown below is for a 3 VDEV RAIDZ1 with the number of disks per VDEV changing.

ZFS-RAIDZ1-Write-3VDEV.jpg


ZFS-RAIDZ1-Read-3VDEV.jpg



The graph shown below is for a 4 VDEV RAIDZ1 with the number of disks per VDEV changing.

ZFS-RAIDZ1-Write-4VDEV.jpg


ZFS-RAIDZ1-Read-4VDEV.jpg


  • RAIDZ2

The graph shown below is for a 3 VDEV RAIDZ2 with the number of disks per VDEV changing.

ZFS-RAIDZ2-Write-3VDEV.jpg


ZFS-RAIDZ2-Read-3VDEV.jpg



The graph shown below is for a 4 VDEV RAIDZ2 with the number of disks per VDEV changing.

ZFS-RAIDZ2-Write-4VDEV.jpg


ZFS-RAIDZ2-Read-4VDEV.jpg


It can be seen that just like a "Mdadm" RAID6, there is a write penalty with RAIDZ2.


  • RAID1+0

The graphs shown below vary with the number of VDEVs. In all cases we had two disks in each VDEV.

ZFS-RAID10-Write.jpg


ZFS-RAID10-Read.jpg



  • Disclaimer - The information on this site is provided as a free support service. Protocase, and it's subsidiary company, 45 Drives, provides this support and guidance through this wiki, but can not be held responsible for any damage or loss of data that may occur through the use of any instructions found herein. Please use at your own risk.