To do this with lvm, a new special LV is first created from the faster device. This LV will hold the cache. Then, the new fast LV is attached to the main LV by way of an lvconvert command. lvconvert inserts one of the device mapper caching targets into the main LV's i/o path. The device mapper target combines the main LV and fast LV into a hybrid device that looks like the main LV, but has better performance. While the main LV is being used, portions of its data will be temporarily and transparently stored on the special fast LV.
The two kinds of caching are:
# lvcreate -n main -L Size vg /dev/slow_hhd
# lvcreate -n fast -L Size vg /dev/fast_ssd
# lvs -a
LV Attr Type Devices
fast -wi------- linear /dev/fast_ssd
main -wi------- linear /dev/slow_hhd
using dm-cache (with cachepool):
# lvconvert --type cache --cachepool fast vg/main
using dm-cache (with cachevol):
# lvconvert --type cache --cachevol fast vg/main
using dm-writecache (with cachevol):
# lvconvert --type writecache --cachevol fast vg/main
For more alteratives see:
dm-cache command shortcut
dm-cache with separate data and metadata LVs
using dm-cache (with cachepool):
# lvs -ao+devices
LV Pool Type Devices main [fast_cpool] cache main_corig(0) [fast_cpool] cache-pool fast_pool_cdata(0) [fast_cpool_cdata] linear /dev/fast_ssd [fast_cpool_cmeta] linear /dev/fast_ssd [main_corig] linear /dev/slow_hhd
using dm-cache (with cachevol):
# lvs -ao+devices
LV Pool Type Devices main [fast_cvol] cache main_corig(0) [fast_cvol] linear /dev/fast_ssd [main_corig] linear /dev/slow_hhd
using dm-writecache (with cachevol):
# lvs -ao+devices
LV Pool Type Devices main [fast_cvol] writecache main_wcorig(0) [fast_cvol] linear /dev/fast_ssd [main_wcorig] linear /dev/slow_hhd
# lvconvert --uncache vg/main
# lvs -a
LV VG Attr Type Devices main vg -wi------- linear /dev/slow_hhd
To stop caching the main LV, separate the fast LV from the main LV. This changes the type of the main LV back to what it was before the cache was attached.
# lvconvert --splitcache vg/main
# lvs -a
LV VG Attr Type Devices fast vg -wi------- linear /dev/fast_ssd main vg -wi------- linear /dev/slow_hhd
# lvcreate --type cache|writecache -n Name -L Size --cachedevice /dev/fast_ssd vg /dev/slow_hhd
The main LV is created with the specified Name and Size from the slow_hhd. A hidden fast LV is created on the fast_ssd and is then attached to the new main LV. If the fast_ssd is unused, the entire disk will be used as the cache unless the --cachesize option is used to specify a size for the fast LV. The --cachedevice option can be repeated to use multiple disks for the fast LV.
Pass this option a cachepool LV or a standard LV. When using a cache pool, lvm places cache data and cache metadata on different LVs. The two LVs together are called a cache pool. This has a bit better performance for dm-cache and permits specific placement and segment type selection for data and metadata volumes. A cache pool is represented as a special type of LV that cannot be used directly. If a standard LV is passed with this option, lvm will first convert it to a cache pool by combining it with another LV to use for metadata. This option can be used with dm-cache.
--cachevol LV
Pass this option a fast LV that should be used to hold the cache. With a cachevol, cache data and metadata are stored in different parts of the same fast LV. This option can be used with dm-writecache or dm-cache.
--cachedevice PV
This option can be used in place of --cachevol, in which case a cachevol LV will be created using the specified device. This option can be repeated to create a cachevol using multiple devices, or a tag name can be specified in which case the cachevol will be created using any of the devices with the given tag. If a named cache device is unused, the entire device will be used to create the cachevol. To create a cachevol of a specific size from the cache devices, include the --cachesize option.
If a main LV has logical block size 512 (with an existing xfs file system using that size), then it cannot use a cache pool with a 4096 logical block size. If the cache pool is attached, the main LV will likely fail to mount.
To avoid this problem, use a mkfs option to specify a 4096 block size for the file system, or attach the cache pool before running mkfs.
When a file system like xfs already exists on the main LV prior to caching, and the file system is using a block size of 512, then the writecache block size should be set to 512. (The file system will likely fail to mount if writecache block size of 4096 is used in this case.)
Check the xfs sector size while the fs is mounted:
# xfs_info /dev/vg/main
Look for sectsz=512 or sectsz=4096
The writecache block size should be chosen to match the xfs sectsz value.
It is also possible to specify a sector size of 4096 to mkfs.xfs when creating the file system. In this case the writecache block size of 4096 can be used.
The writecache block size is displayed by the command:
lvs -o writecacheblocksize VG/LV
The amount of main system memory used by dm-writecache can be a factor when selecting the writecache cachevol size and the writecache block size.
For example, --cachesettings 'high_watermark=90 writeback_jobs=4'.
To include settings when caching is started, run:
# lvconvert --type writecache --cachevol fast \ --cachesettings 'option=N' vg/main
To change settings for an existing writecache, run:
# lvchange --cachesettings 'option=N' vg/main
To clear all settings that have been applied, run:
# lvchange --cachesettings '' vg/main
To view the settings that are applied to a writecache LV, run:
# lvs -o cachesettings vg/main
Tunable settings are:
It's possible to prepare a number of different profile files in the /etc/lvm/profile directory and specify the file name of the profile when starting writecache.
Example
# cat <<EOF > /etc/lvm/profile/cache_writecache.profile allocation {
To create a cache pool of given data size and let lvm2 calculate appropriate metadata size:
# lvcreate --type cache-pool -L DataSize -n fast vg /dev/fast_ssd1
To create a cache pool from separate LV and let lvm2 calculate appropriate cache metadata size:
# lvcreate -n fast -L DataSize vg /dev/fast_ssd1
# lvconvert --type cache-pool vg/fast /dev/fast_ssd1
To create a cache pool from two separate LVs:
# lvcreate -n fast -L DataSize vg /dev/fast_ssd1
# lvcreate -n fastmeta -L MetadataSize vg /dev/fast_ssd2
# lvconvert --type cache-pool --poolmetadata fastmeta vg/fast
Then use the cache pool LV to start caching the main LV:
# lvconvert --type cache --cachepool fast vg/main
A variation of the same procedure automatically creates a cache pool when caching is started. To do this, use a standard LV as the --cachepool (this will hold cache data), and use another standard LV as the --poolmetadata (this will hold cache metadata). LVM will create a cache pool LV from the two specified LVs, and use the cache pool to start caching the main LV.
# lvcreate -n fast -L DataSize vg /dev/fast_ssd1 # lvcreate -n fastmeta -L MetadataSize vg /dev/fast_ssd2 # lvconvert --type cache --cachepool fast \ --poolmetadata fastmeta vg/main
A second cache mode is "writeback". Writeback delays writing data blocks from the cache back to the origin LV. This mode will increase performance, but the loss of a cache device can result in lost data.
With the --cachemode option, the cache mode can be set when caching is started, or changed on an LV that is already cached. The current cache mode can be displayed with the cache_mode reporting option:
lvs -o+cache_mode VG/LV
lvm.conf(5)
allocation/cache_mode
defines the default cache mode.
# lvconvert --type cache --cachemode writethrough \ --cachepool fast vg/main # lvconvert --type cache --cachemode writethrough \ --cachevol fast vg/main
Using a chunk size that is too large can result in wasteful use of the cache, in which small reads and writes cause large sections of an LV to be stored in the cache. It can also require increasing migration threshold which defaults to 2048 sectors (1 MiB). Lvm2 ensures migration threshold is at least 8 chunks in size. This may in some cases result in very high bandwidth load of transfering data between the cache LV and its cache origin LV. However, choosing a chunk size that is too small can result in more overhead trying to manage the numerous chunks that become mapped into the cache. Overhead can include both excessive CPU time searching for chunks, and excessive memory tracking chunks.
Command to display the chunk size:
lvs -o+chunksize VG/LV
lvm.conf(5) allocation/cache_pool_chunk_size
controls the default chunk size.
The default value is shown by:
lvmconfig --type default allocation/cache_pool_chunk_size
Checking migration threshold (in sectors) of running cached LV:
lvs -o+kernel_cache_settings VG/LV
User can set migration threshold via cache policy settings as "migration_threshold=<#sectors>" to set the maximum number of sectors being migrated, the default being 2048 sectors (1MiB).
Command to set migration threshold to 2MiB (4096 sectors):
lvcreate --cachepolicy 'migration_threshold=4096' VG/LV
Command to display the migration threshold:
lvs -o+kernel_cache_settings,cache_settings VG/LV
lvs -o+chunksize VG/LV
The older "mq" policy has a number of tunable parameters. The defaults are chosen to be suitable for the majority of systems, but in special circumstances, changing the settings can improve performance.
With the --cachepolicy and --cachesettings options, the cache policy and settings can be set when caching is started, or changed on an existing cached LV (both options can be used together). The current cache policy and settings can be displayed with the cache_policy and cache_settings reporting options:
lvs -o+cache_policy,cache_settings VG/LV
Change the cache policy and settings of an existing LV.
# lvchange --cachepolicy mq --cachesettings \ 'migration_threshold=2048 random_threshold=4' vg/main
lvm.conf(5)
allocation/cache_policy
defines the default cache policy.
lvm.conf(5)
allocation/cache_settings
defines the default cache settings.
Example
# cat <<EOF > /etc/lvm/profile/cache_big_chunk.profile allocation {
# lvcreate -n main -L Size vg /dev/slow # lvcreate --type raid1 -m 1 -n fast -L Size vg /dev/ssd1 /dev/ssd2 # lvconvert --type cache --cachevol fast vg/main
# lvcreate --cache --size CacheDataSize VG/LV [FastPVs]
or the longer variant
# lvcreate --type cache --size CacheDataSize \ --name NameCachePool VG/LV [FastPVs]
In this command, the specified LV already exists, and is the main LV to be cached. The command creates a new cache pool with size and given name or the name is automatically selected from a sequence lvolX_cpool, using the optionally specified fast PV(s) (typically an ssd). Then it attaches the new cache pool to the existing main LV to begin caching.
(Note: ensure that the specified main LV is a standard LV. If a cache pool LV is mistakenly specified, then the command does something different.)
(Note: the type option is interpreted differently by this command than by normal lvcreate commands in which --type specifies the type of the newly created LV. In this case, an LV with type cache-pool is being created, and the existing main LV is being converted to type cache.)
cache_check(8), cache_dump(8), cache_repair(8)