Building a storage server, need some advice

[ENJOLRAS Fabien]

I would like to build a high density storage server which will be evolutive. Here is the hardware I have available:
-Norco RPC-4224 chassis (rackable 4U with x24 3.5 SATA disk hotswap backplane)
-Tyan Tempest i5000VF motherboard with dual 3 Ghz Xeon and 8GB ECC DDR2
-3Ware 9650SE-24M8 SATA controller (24 SATA2 channels)
I would like to begin with x5 or x6 2TB Samsung F4 "Green" hard drives, to expand to full 24 drives when my storage need will grow.
My first plan was to build a huge RAID6 array but my attention was pointed to Solaris platform for reliable Raidz3 support.
My question was, how can I grow a raidz3 array with more drives? What is the most secure and reliable method without the need to backup all the data already stored?
I've thought of creating a first raidz3 vdev with 6 drives, the adding up to 4 6-drive raidz3 vdevs. But this way I lose a lot of storage on the base 48TB?

[Ian Collins]

Post by ENJOLRAS Fabien
-Norco RPC-4224 chassis (rackable 4U with x24 3.5 SATA disk hotswap backplane)
-Tyan Tempest i5000VF motherboard with dual 3 Ghz Xeon and 8GB ECC DDR2

Max out the memory.

Post by ENJOLRAS Fabien
-3Ware 9650SE-24M8 SATA controller (24 SATA2 channels)

Is the card supported?

Post by ENJOLRAS Fabien
I would like to begin with x5 or x6 2TB Samsung F4 "Green" hard drives, to expand to full 24 drives when my storage need will grow.
My first plan was to build a huge RAID6 array but my attention was pointed to Solaris platform for reliable Raidz3 support.
My question was, how can I grow a raidz3 array with more drives? What is the most secure and reliable method without the need to backup all the data already stored?

Add more vdevs.

Post by ENJOLRAS Fabien
I've thought of creating a first raidz3 vdev with 6 drives, the adding up to 4 6-drive raidz3 vdevs. But this way I lose a lot of storage on the base 48TB?

Maybe start with an 8 drive vdev and add 2 more vdevs later.

[Orvar Korvar]

Initially, If your first vdev gets full to 90% then everything will be slow. If you then add new disks, then your first vdev is full. So you need to distribute the old data to all new disks (for instance by copying the data).

Every time you add a new vdev you need to do this rebalancing of data to new vdev.

[Ron Mexico]

My storage server at work has three RAIDZ3's, each with 12 drives plus two hot spares. I use Western Digital RE4's. In the two years I've been using this configuration, I've never had more than one drive go bad at a time.

At home I have a RAIDZ1 using 5 WD Caviar Green 1TB drives. Not a single disc error in over two years.

IMO, a six disk RAIDZ3 is beyond overkill. Make each group a RAIDZ1 with a hot spare, or a RAIDZ2 with a shelf spare if you can't spare the extra drive slots.

[Ian Collins]

Post by Ron Mexico
My storage server at work has three RAIDZ3's, each with 12 drives plus two hot spares. I use Western Digital RE4's. In the two years I've been using this configuration, I've never had more than one drive go bad at a time.
At home I have a RAIDZ1 using 5 WD Caviar Green 1TB drives. Not a single disc error in over two years.
IMO, a six disk RAIDZ3 is beyond overkill. Make each group a RAIDZ1 with a hot spare, or a RAIDZ2 with a shelf spare if you can't spare the extra drive slots.

raidz with 2TB drives opens too big a risk window for my liking. I
recently had a box take 160 hours to resilver a 500GB drive, so a
similar system could take 3 weeks to resilver a 2TB device!

[Ron Mexico]

FWIW, I resilvered a 2TB drive a few weeks ago that took less than two days. Pool is at about 75% capacity.

[Ian Collins]

[context is good!]

Post by Ron Mexico
FWIW, I resilvered a 2TB drive a few weeks ago that took less than two days. Pool is at about 75% capacity.

Resilver time is very dependent on how busy the pool is.

[Brandon High]

On Tue, Apr 26, 2011 at 1:55 AM, ENJOLRAS Fabien

Post by ENJOLRAS Fabien
-Norco RPC-4224 chassis (rackable 4U with x24 3.5 SATA disk hotswap backplane)
-Tyan Tempest i5000VF motherboard with dual 3 Ghz Xeon and 8GB ECC DDR2

You should verify that the motherboard and devices are supported. If
it doesn't use Intel NICs onboard, you may want to consider getting
some. Consider adding more memory.

Post by ENJOLRAS Fabien
My question was, how can I grow a raidz3 array with more drives? What is the most secure and reliable method without the need to backup all the data already stored?

You can't grow a raidz vdev, but you can add more vdevs to the pool.

Post by ENJOLRAS Fabien
I've thought of creating a first raidz3 vdev with 6 drives, the adding up to 4 6-drive raidz3 vdevs. But this way I lose a lot of storage on the base 48TB?

A 6-drive vdev is sub-optimal. You'd be better off going with (2^n +
p) drives, where p is the amount used for parity. For raidz3, that's
5, 7, 11, 19 and for raidz2, it's 4, 6, 10, 18. The sweet spot is the
middle of the range.

If you want to optimize for space vs. performance, three 7-drive vdevs
will give you 24TB, or two 11-drive vdevs will yield 32TB. Both
configurations leave bays available for spares.

If you want slightly higher performance, you can use more vdevs. 4 x
6-drive raidz2 vdevs will give you 32TB of space and should have
roughly twice the random iops performance of the 2x11 configuration
above.

For much higher perfomance, 12x mirrored vdevs will give you 24TB and
roughly 4 times the random iops of the 3x7 configuration. It's also
easiest to grow a zpool that's made of mirrors because you can simply
add two more drives rather than enough to make up an additional vdev.

-B