Using Btrfs with Multiple Devices

From btrfs Wiki
(Difference between revisions)
Jump to: navigation, search
m (Adding New Devices: deleted dash from example, probably old style leftover)
(migrate from btrfs.ipv5.de)
Line 9: Line 9:
 
=== Current Status ===
 
=== Current Status ===
  
Btrfs can add and remove devices online. Adding devices at mkfs time gives the most control over the raid levels used.
+
Btrfs can add and remove devices online. Adding devices at mkfs time gives the most control over the raid levels used.
  
Btrfs can do raid0, raid1, raid10 and it can duplicate metadata on a single spindle. When blocks are read in, checksums are verified and if there are any errors, Btrfs tries to read from an alternate copy.
+
Btrfs can do raid0, raid1, raid10 and it can duplicate metadata on a single spindle. When blocks are read in, checksums are verified and if there are any errors, Btrfs tries to read from an alternate copy.
  
 
See the [[Gotchas]] page for some current issues when using btrfs with multiple volumes of differing sizes in a RAID1 style setup.
 
See the [[Gotchas]] page for some current issues when using btrfs with multiple volumes of differing sizes in a RAID1 style setup.
Line 17: Line 17:
 
=== Creating a Multi-device FS ===
 
=== Creating a Multi-device FS ===
  
mkfs.btrfs will accept more than one device on the command line. It has options to control the raid configuration for data and metadata. Valid choices are raid0, raid1, raid10 and single. Single means that no duplication of metadata is done, which may be desired when using hardware raid.
+
mkfs.btrfs will accept more than one device on the command line. It has options to control the raid configuration for data and metadata. Valid choices are raid0, raid1, raid10 and single. Single means that no duplication of metadata is done, which may be desired when using hardware raid.
  
 
Raid10 requires at least 4 devices.
 
Raid10 requires at least 4 devices.
Line 23: Line 23:
 
  # Create a filesystem across four drives (metadata mirrored, data striped)
 
  # Create a filesystem across four drives (metadata mirrored, data striped)
 
  mkfs.btrfs /dev/sdb /dev/sdc /dev/sdd /dev/sde
 
  mkfs.btrfs /dev/sdb /dev/sdc /dev/sdd /dev/sde
 
+
 
  # Stripe the metadata without mirroring
 
  # Stripe the metadata without mirroring
 
  mkfs.btrfs -m raid0 /dev/sdb /dev/sdc
 
  mkfs.btrfs -m raid0 /dev/sdb /dev/sdc
 
+
 
  # Use raid10 for both data and metadata
 
  # Use raid10 for both data and metadata
 
  mkfs.btrfs -m raid10 -d raid10 /dev/sdb /dev/sdc /dev/sdd /dev/sde
 
  mkfs.btrfs -m raid10 -d raid10 /dev/sdb /dev/sdc /dev/sdd /dev/sde
 
+
 
  # Don't duplicate metadata on a single drive
 
  # Don't duplicate metadata on a single drive
 
  mkfs.btrfs -m single /dev/sdb
 
  mkfs.btrfs -m single /dev/sdb
Line 51: Line 51:
 
=== Finding and Listing Multi-device Filesystems ===
 
=== Finding and Listing Multi-device Filesystems ===
  
{{Note|the new command '''[[btrfs(command)|btrfs]]''' is replacing the old ones '''btrfsctl''' and '''btrfs-show'''. In the examples the btrfs commands is used. Below the old commands compared to the new one:
+
'''btrfs device scan''' is used to scan all of the block devices under /dev and probe for Btrfs volumes. This is required after loading the btrfs module if you're running with more than one device in a filesystem.
<pre>
+
OLD                        NEW
+
---------------------      --------------------------
+
btrfsctl -a                btrfs device scan
+
btrfsctl -A /dev/sdb        btrfs device scan /dev/sdb
+
btrfs-show                  btrfs filesystem show
+
</pre>
+
}}
+
 
+
'''btrfs device scan''' is used to scan all of the block devices under /dev and probe for Btrfs volumes. This is required after loading the btrfs module if you're running with more than one device in a filesystem.
+
  
 
  # Scan all devices
 
  # Scan all devices
 
  btrfs device scan
 
  btrfs device scan
 
+
 
  # Scan a single device
 
  # Scan a single device
 
  btrfs device scan /dev/sdb
 
  btrfs device scan /dev/sdb
  
'''btrfs filesystem show''' will print information about all of the Btrfs filesystems on the machine.
+
'''btrfs filesystem show''' will print information about all of the btrfs filesystems on the machine.
  
 
=== Adding New Devices ===
 
=== Adding New Devices ===
{{Note|the new command '''[[btrfs(command)|btrfs]]''' is replacing the old ones '''btrfs-vol''' and '''btrfs-show'''. In the examples the btrfs commands is used. Below the old commands compared to the new one:
 
<pre>
 
OLD                              NEW
 
--------------------------      ----------------------------------------
 
btrfs-show                      btrfs filesystem show
 
btrfs-vol -a <dev> <path>        btrfs device add <dev [<dev>...] <path>
 
btrfs-vol -b <path>              btrfs filesystem balance <path>
 
</pre>
 
}}
 
 
  
 
'''btrfs filesystem show''' gives you a list of all the btrfs filesystems on the systems and which devices they include.
 
'''btrfs filesystem show''' gives you a list of all the btrfs filesystems on the systems and which devices they include.
Line 87: Line 67:
 
'''btrfs device add''' is used to add new devices to a mounted filesystem.
 
'''btrfs device add''' is used to add new devices to a mounted filesystem.
  
'''btrfs filesystem balance''' can balance (restripe) the allocated extents across all of the existing devices.   For example:
+
'''btrfs filesystem balance''' can balance (restripe) the allocated extents across all of the existing devices. For example:
  
 
  mkfs.btrfs /dev/sdb
 
  mkfs.btrfs /dev/sdb
Line 94: Line 74:
 
  btrfs device add /dev/sdc /mnt
 
  btrfs device add /dev/sdc /mnt
  
At this point we have a filesystem with two devices, but all of the metadata and data
+
At this point we have a filesystem with two devices, but all of the metadata and data are still stored on /dev/sdb. The filesystem must be balanced to spread the files across all of the devices.
are still stored on /dev/sdb. The filesystem must be balanced to spread the files across all of the devices.
+
  
 
  btrfs filesystem balance /mnt
 
  btrfs filesystem balance /mnt
  
The balance operation will take some time. It reads in all of the FS data and metadata and rewrites it across the new device.
+
The balance operation will take some time. It reads in all of the FS data and metadata and rewrites it across the new device.
  
 
=== Removing Devices ===
 
=== Removing Devices ===
{{Note|the new command '''[[btrfs(command)|btrfs]]''' is replacing the old one '''btrfs-vol'''. In the examples the btrfs commands is used. Below the old command compared to the new one:
+
 
<pre>
+
'''btrfs device remove''' is used to remove devices online. It redistributes the any extents in use on the device being removed to the other devices in the filesystem. Example:
OLD                              NEW
+
--------------------------      ----------------------------------------
+
btrfs-vol -r <dev> <path>        btrfs device delete <dev [<dev>...] <path>
+
</pre>
+
}}
+
'''btrfs device remove''' is used to remove devices online. It redistributes the any extents in use on the device being removed to the other devices in the filesystem. Example:
+
  
 
  mkfs.btrfs /dev/sdb /dev/sdc /dev/sdd /dev/sde
 
  mkfs.btrfs /dev/sdb /dev/sdc /dev/sdd /dev/sde
Line 117: Line 90:
  
 
=== Replacing Failed Devices ===
 
=== Replacing Failed Devices ===
{{Note|the new command '''[[btrfs(command)|btrfs]]''' is replacing the old one '''btrfs-vol'''. In the examples below the btrfs commands is used. Below the old command compared to the new one:
 
<pre>
 
OLD                              NEW
 
--------------------------      ----------------------------------------
 
btrfs-vol -r missing <path>        btrfs device delete missing <path>
 
</pre>
 
}}
 
  
 
+
The example above can be used to remove a failed device if the super block can still be read. But, if a device is missing or the super block has been corrupted, the filesystem will need to be mounted in degraded mode:
The example above can be used to remove a failed device if the super block can still be read. But, if a device is missing or the super block has been corrupted, the filesystem will need to be mounted in degraded mode:
+
  
 
  mkfs.btrfs -m raid1 /dev/sdb /dev/sdc /dev/sdd /dev/sde
 
  mkfs.btrfs -m raid1 /dev/sdb /dev/sdc /dev/sdd /dev/sde
Line 141: Line 106:
 
'''btrfs device delete missing''' tells btrfs to remove the first device that is described by the filesystem metadata but not present when the FS was mounted.
 
'''btrfs device delete missing''' tells btrfs to remove the first device that is described by the filesystem metadata but not present when the FS was mounted.
  
In case of ''raidXX'' layout, you cannot go below the minimum number of the device required. So before removing a device (even the ''missing'' one) you may need to add a new one.
+
In case of ''raidXX'' layout, you cannot go below the minimum number of the device required. So before removing a device (even the ''missing'' one) you may need to add a new one. For example if you have a raid1 layout with '''two''' device, and a device fails, you must:
For example if you have a raid1 layout with '''two''' device, and a device fails, you must:
+
 
 
* mount in degraded mode
 
* mount in degraded mode
 
* add a new device
 
* add a new device
Line 149: Line 114:
 
===Mounting a multi-device filesystem from /etc/fstab===
 
===Mounting a multi-device filesystem from /etc/fstab===
  
If you don't have an initrd, or your initrd doesn't perform a <code>btrfs device scan</code>, you can still mount a multi-volume btrfs filesystem by passing ''all'' the devices in the filesystem explicitly to the mount command. A suitable /etc/fstab entry would be:
+
If you don't have an initrd, or your initrd doesn't perform a btrfs device scan, you can still mount a multi-volume btrfs filesystem by passing ''all'' the devices in the filesystem explicitly to the mount command. A suitable /etc/fstab entry would be:
 
+
<pre>
+
/dev/sdb    /mnt    btrfs    device=/dev/sdb,device=/dev/sdc,device=/dev/sdd,device=/dev/sde    0 0
/dev/sdb    /mnt    btrfs    device=/dev/sdb,device=/dev/sdc,device=/dev/sdd,device=/dev/sde    0 0
+
</pre>
+
  
[[Category:Documentation]]
+
[[Category:UserDoc]]

Revision as of 11:59, 7 May 2012

Contents

Multiple Devices

A Btrfs filesystem can be created on top of many devices, and more devices can be added after the FS has been created.

By default, metadata will be mirrored across two devices and data will be striped across all of the devices present.

If only one device is present, metadata will be duplicated on that one device.

Current Status

Btrfs can add and remove devices online. Adding devices at mkfs time gives the most control over the raid levels used.

Btrfs can do raid0, raid1, raid10 and it can duplicate metadata on a single spindle. When blocks are read in, checksums are verified and if there are any errors, Btrfs tries to read from an alternate copy.

See the Gotchas page for some current issues when using btrfs with multiple volumes of differing sizes in a RAID1 style setup.

Creating a Multi-device FS

mkfs.btrfs will accept more than one device on the command line. It has options to control the raid configuration for data and metadata. Valid choices are raid0, raid1, raid10 and single. Single means that no duplication of metadata is done, which may be desired when using hardware raid.

Raid10 requires at least 4 devices.

# Create a filesystem across four drives (metadata mirrored, data striped)
mkfs.btrfs /dev/sdb /dev/sdc /dev/sdd /dev/sde

# Stripe the metadata without mirroring
mkfs.btrfs -m raid0 /dev/sdb /dev/sdc

# Use raid10 for both data and metadata
mkfs.btrfs -m raid10 -d raid10 /dev/sdb /dev/sdc /dev/sdd /dev/sde

# Don't duplicate metadata on a single drive
mkfs.btrfs -m single /dev/sdb

Once you create a multi-device filesystem, you can use any device in the FS for the mount command:

mkfs.btrfs /dev/sdb /dev/sdc /dev/sde
mount /dev/sde /mnt

If you want to mount a multi-device filesystem using a loopback device, it's not sufficient to use mount -o loop. Instead, you'll have to set up the loopbacks manually:

# Create and mount a filesystem made of several disk images
mkfs.btrfs img0 img1 img2
losetup /dev/loop0 img0
losetup /dev/loop1 img1
losetup /dev/loop2 img2
mount /dev/loop0 /mnt/btrfs

After a reboot or reloading the btrfs module, you'll need to use btrfs device scan to discover all multi-device filesystems on the machine (see below)

Finding and Listing Multi-device Filesystems

btrfs device scan is used to scan all of the block devices under /dev and probe for Btrfs volumes. This is required after loading the btrfs module if you're running with more than one device in a filesystem.

# Scan all devices
btrfs device scan

# Scan a single device
btrfs device scan /dev/sdb

btrfs filesystem show will print information about all of the btrfs filesystems on the machine.

Adding New Devices

btrfs filesystem show gives you a list of all the btrfs filesystems on the systems and which devices they include.

btrfs device add is used to add new devices to a mounted filesystem.

btrfs filesystem balance can balance (restripe) the allocated extents across all of the existing devices. For example:

mkfs.btrfs /dev/sdb
mount /dev/sdb /mnt
# Create some files
btrfs device add /dev/sdc /mnt

At this point we have a filesystem with two devices, but all of the metadata and data are still stored on /dev/sdb. The filesystem must be balanced to spread the files across all of the devices.

btrfs filesystem balance /mnt

The balance operation will take some time. It reads in all of the FS data and metadata and rewrites it across the new device.

Removing Devices

btrfs device remove is used to remove devices online. It redistributes the any extents in use on the device being removed to the other devices in the filesystem. Example:

mkfs.btrfs /dev/sdb /dev/sdc /dev/sdd /dev/sde
mount /dev/sdb /mnt
# Put some data on the filesystem here
btrfs device delete /dev/sdc /mnt

Replacing Failed Devices

The example above can be used to remove a failed device if the super block can still be read. But, if a device is missing or the super block has been corrupted, the filesystem will need to be mounted in degraded mode:
mkfs.btrfs -m raid1 /dev/sdb /dev/sdc /dev/sdd /dev/sde
#
# sdd is destroyed or removed, use -o degraded to force the mount
# to ignore missing devices
#
mount -o degraded /dev/sdb /mnt
#
# 'missing' is a special device name
#
btrfs device delete missing /mnt

btrfs device delete missing tells btrfs to remove the first device that is described by the filesystem metadata but not present when the FS was mounted.

In case of raidXX layout, you cannot go below the minimum number of the device required. So before removing a device (even the missing one) you may need to add a new one. For example if you have a raid1 layout with two device, and a device fails, you must:

  • mount in degraded mode
  • add a new device
  • remove the missing device

Mounting a multi-device filesystem from /etc/fstab

If you don't have an initrd, or your initrd doesn't perform a btrfs device scan, you can still mount a multi-volume btrfs filesystem by passing all the devices in the filesystem explicitly to the mount command. A suitable /etc/fstab entry would be:

/dev/sdb     /mnt    btrfs    device=/dev/sdb,device=/dev/sdc,device=/dev/sdd,device=/dev/sde    0 0
Personal tools