Have fun with ZFS: Setting up storage pool

Unlike filesystems that operates on a single disk, partition or logical volume (i.e. ext4 and NTFS), ZFS can span across multiple storage volumes. In ZFS, we call the storage pool we can actually store stuff zpool. Inside a zpool, there are a series of VDEV (Virtual DEVices). Data written to the zpool will be separated on all of the VDEVs, similar to RAID0¹.

If you are familiar with RAID, this may sound like a pretty bad idea, since failure of a single VDEV would bring down the whole pool. However, ZFS allows redundancy inside a VDEV to make sure they don't fail easily. There are quite a few types of VDEV²:

• single: just a single disk, no parity

• mirror: multiple volumes with the same data, VDEV won't fail as far as any single volume is good

  • similar to RAID 1

• RAIDz1, RAIDz2, RAIDz3: multiple volumes with parity data, allows 1/2/3 bad volumes before VDEV fails

  • requires 3/4/5 disks at minimum
  • uses parity data, similar to RAID 5/6

Building redundancy on VDEV level makes zpool very versatile. You can expand a storage pool via simply adding a new VDEV with any redundancy setup you wish, rather than reconstructing the whole array of volumes. Doing data parity in software also means that the Write hole issue that plagued other RAID5 systems can be mitigated.

Planning is important when designing storage system. An adequately designed system will make sure you get the most out of your hardware. Moreover, it will make life easier when bad things happen. There're three things to consider:

1. Performance
2. Space Efficiency
3. Fault Tolerance

We've already talked about space efficiency and fault tolerance, so we won't reiterate here. Performance only comes into consideration if there will be heavy upon the storage pool (i.e. you will be running databases on it). If you do need good performance, here're some rule of thumbs:

• If budget allows, use mirror.
• If RAIDz (at any level) is used, it is recommended to use \( 2^n + p \) drives to balance performance and space efficiency.

It is also very beneficial to tune ZFS to suit your workload, but this topic is beyond the scope of this article.

You should have a general plan on what the pool should look like. Now, let's create the pool.

For storage pool creation and maintenance, we use the zpool command. This command accepts subcommands, similar to git. To create a ZFS pool, we use the create subcommand:

1

zpool create -f -o ashift=12 -m <mountpoint> <pool_name> [raidz|raidz2|raidz3|mirror] <volumes>

• -f in order to mitigate Does not contain an EFI label error.

  • This may be ZFS on Linux specific. Check your manual.

• -o ashift=12 use AF (Advanced Format) 4k sector size to improve performance. If your pool is made up of SSDs, use -o ashift=13 since SSD uses 8k sector size.

  • See more detail about this topic on OpenZFS docs.
• -m <mountpoint> specify where the pool would be mounted.
• <pool_name> the name of the new pool.
• [raidz|raidz2|raidz3|mirror] <volumes> specify the type of VDEV to create and volumes to use

Note that generally, you should use volume ID rather than non-persistent volume location like sdX since volume IDs are invariant among reboots and hardware changes. You can find volume IDs by checking symlink in /dev/disk/by-id/.

1

# zpool create -f -o ashift=12 -m /mnt/data data ata-VOLUME-ID

1
2

# zpool create -f -o ashift=12 -m /mnt/data data \
      mirror ata-VOLUME-1 ata-VOLUME-2

1
2
3

# zpool create -f -o ashift=12 -m /mnt/data data \
      mirror ata-VOLUME-1 ata-VOLUME-2 \
      mirror ata-VOLUME-3 ata-VOLUME-4

1
2

# zpool create -f -o ashift=12 -m /mnt/data data \
      raidz ata-VOLUME-1 ata-VOLUME-2 ata-VOLUME-3 [...even more volumes]

Now, our pool has been created and we can check its status:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# zpool status data
  pool: data
 state: ONLINE
  scan: none requested
config:

        NAME                      STATE     READ WRITE CKSUM
        data                      ONLINE       0     0     0
          mirror-0                ONLINE       0     0     0
            ata-VOLUME-1          ONLINE       0     0     0
            ata-VOLUME-2          ONLINE       0     0     0

errors: No known data errors

Although there's not much going on here right now, this would be a very useful command later. We will check pool scrub/rebuild progress, check volume state and, if we are unlucky, check which files are affected by a data loss.

In ZFS, we can modify the pool layout after the pool has been created.

1

# zpool add [pool_name] [raidz|raidz2|raidz3|mirror] <volumes>

1

# zpool attach [pool_name] <exisitng_volume/VDEV_name> <new_volumes>

1

# zpool remove [pool_name] [devices]

If you want to use the pool on other devices, you will need to export the pool:

1

# zpool export <pool_name>

Be caution when importing the pool. By default, ZFS would use the non-persistent volume naming, which will cause issue when disk arrangement changes. Instead, you should use:

1

# zpool import -d /dev/disk/by-id <pool_name>