LVM ¶

By default, Raspberry Pi OS does not come with LVM. However, LVM is useful for taking snapshots in Proxmox.

Setup¶

The main setup concept is as follows:

Create an SD card with a stock Raspberry Pi OS on a host system.
Install necessary packages on SD card.
Generate new initramfs on the Pi.
Backup the OS to a tar file from the host system.
Create file system on new NVMe drive.
Mount new file system drive.
Extract tar to new drive.
Modify boot on new drive.
Boot new drive.

Equipment¶

The following equipment was used:

Raspberry Pi 5 with SD card and an NVMe hat.
Host computer with Ubuntu server installed that can read an SD card.
SD card.
500GB NVMe.

Note

My setup can only read NVMe drives from my Pi and so all of my interfacing with the NVMe is done while booted into the SD card on the Pi.

Note

I have an NVMe drive on my Pi, therefore my commnds below are with respect to /dev/nvme0n1 when working with it.

Note

When working with my SD card mounted on my host system, the drive location is /dev/mmcblk0

Note

Most of the mounting of file systems is being done in the /mnt directory of both the Pi and host system.

Host System¶

Install Raspberry Pi OS on an SD card.

SD Card¶

Insert SD card into Pi and boot from SD card.

Switch to root

sudo su

Update the Pi

(
  apt update
  apt full-upgrade
)

Install lvm2 and initramfs-tools

(
  apt install initramfs-tools
  apt install lvm2 -y
)

Update initramfs

update-initramfs -u -k all

Note

Updating initramfs may not be needed because it may update itself during the lvm2 installation.

Shutown the Pi

poweroff

Host System¶

Remove the SD card from the Pi and insert it back into the host system.

Switch to root

sudo su

Get the /dev drive location

lsblk

Output

NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0      179:0    0 119.1G  0 disk 
├─mmcblk0p1  179:1    0   512M  0 part /boot/firmware
└─mmcblk0p2  179:2    0 118.6G  0 part /
nvme0n1      259:0    0 465.8G  0 disk

Set the drive location

DRIVE=mmcblk0p

Note

The p needs to be added to the end of the drive if using an SD card or NVMe drive.

Mount the SD card partitions and archive the contents to a tar file

(
  cd /mnt
  mount /dev/${DRIVE}2 ./rpi
  mkdir -p rpi/boot/firmware
  mount /dev/${DRIVE}1 ./rpi/boot/firmware
  tar -cvzf rpi.tar.gz -C rpi ./
  umount ./rpi/boot/firmware
  umount ./rpi
)

The SD card is not archived into the /mnt/rpi.tar.gz file on the host system.

Check the tar file before using it

tar -tvf /mnt/rpi.tar.gz

SD Card¶

Insert SD card into Pi and boot from SD card.

Using parted, remove all partitions from the NVMe.

Warning

THIS DESTROYS ALL EXISTING DATA ON THE NVMe

Switch to root

sudo su

Set the drive location

DRIVE=nvme0n1p

Note

The p needs to be added to the end of the drive if using an SD card or NVMe drive.

Code

parted /dev/${DRIVE%p}

Identify the partition number: Once inside the parted interactive shell (you'll see a (parted) prompt), use the print command to list the partitions on the selected disk and find the number of the one you want to delete.

Code

(parted) print

Look at the output to identify the correct partition number (the first column).

(parted) rm 2

(parted) rm 1

(parted) quit

Check that the partitions were deleted

lsblk

Output

NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0      179:0    0 119.1G  0 disk 
├─mmcblk0p1  179:1    0   512M  0 part /boot/firmware
└─mmcblk0p2  179:2    0 118.6G  0 part /
nvme0n1      259:0    0 465.8G  0 disk

Create a 500MB FAT32 partition and the remainder of the disk as an LVM partition

(
  parted /dev/${DRIVE%p} mkpart primary fat32 2048s 512MiB && \
  parted /dev/${DRIVE%p} mkpart primary ext4 512MiB 100% && \
  parted /dev/${DRIVE%p} set 2 lvm on
)

Check that the partitions were created successfully

lsblk

Output

NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0      179:0    0 119.1G  0 disk 
├─mmcblk0p1  179:1    0   512M  0 part /boot/firmware
└─mmcblk0p2  179:2    0 118.6G  0 part /
nvme0n1      259:0    0 465.8G  0 disk 
├─nvme0n1p1  259:1    0   511M  0 part 
└─nvme0n1p2  259:2    0 465.3G  0 part

Format and label the FAT partition

mkfs.fat -F 32 -n bootfs-rpi /dev/${DRIVE}1

Setup LVM on the NVMe drive, create and format the root volume

(
  pvcreate /dev/${DRIVE}2 && \
  vgcreate pve /dev/${DRIVE}2 && \
  lvcreate -L 70G -n root pve && \
  mke2fs -t ext4 -L pve /dev/pve/root
)

Check that the volumes were created successfully

lsblk

Output

NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0      179:0    0 119.1G  0 disk 
├─mmcblk0p1  179:1    0   512M  0 part /boot/firmware
└─mmcblk0p2  179:2    0 118.6G  0 part /
nvme0n1      259:0    0 465.8G  0 disk 
├─nvme0n1p1  259:1    0   511M  0 part 
└─nvme0n1p2  259:2    0 465.3G  0 part 
  └─pve-root 254:0    0    30G  0 lvm

Transfer the archived tar file from the host system using SCP

scp user@hostip:/mnt/rpi.tar.gz /mnt

Mount the USB partitions and restore the contents

(
  cd /mnt && \
  mkdir -p ./rpi/boot/firmware && \
  mount /dev/pve/root rpi && \
  mount /dev/${DRIVE}1 ./rpi/boot/firmware && \
  tar -xvzf rpi.tar.gz -C ./rpi
)

Check that the contents transferred successfully

ls rpi

Output

bin  boot  dev  etc  home  lib  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

Note

Don't seem to need to change config.txt with initramfs.

./rpi/etc/fstab

AutomaticManual

(
  sed -i 's/^PARTUUID=[a-z0-9]*-01/\/dev\/'"${DRIVE}"'1/' /mnt/rpi/etc/fstab && \
  sed -i 's/^PARTUUID=[a-z0-9]*-02/\/dev\/pve\/root/' /mnt/rpi/etc/fstab && \
  cat /mnt/rpi/etc/fstab
)

proc            /proc           proc    defaults          0       0
/dev/nvme0n1p1  /boot/firmware  vfat    defaults          0       2
/dev/pve/root  /               ext4    defaults,noatime  0       1

./rpi/boot/firmware/cmdline.txt

AutomaticManual

(
  sed -i 's/root=PARTUUID=[a-z0-9]*-02/root=\/dev\/pve\/root/' /mnt/rpi/boot/firmware/cmdline.txt && \
  cat /mnt/rpi/boot/firmware/cmdline.txt
)

console=serial0,115200 console=tty1 root=/dev/pve/root rootfstype=ext4 fsck.repair=yes rootwait cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1

Unmount the partitions

(
  cd /mnt && \
  umount ./rpi/boot/firmware/ && \
  umount ./rpi/ && \
  rm -r rpi/
)

Reboot and hold the spacebar to get to the boot menu. Choose 6 for NVMe.

Verify that it's booting from the NVMe

lsblk

Output

NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0      179:0    0 119.1G  0 disk 
├─mmcblk0p1  179:1    0   512M  0 part 
└─mmcblk0p2  179:2    0 118.6G  0 part 
nvme0n1      259:0    0 465.8G  0 disk 
├─nvme0n1p1  259:1    0   511M  0 part /boot/firmware
└─nvme0n1p2  259:2    0 465.3G  0 part 
  ├─pve-swap 254:0    0     8G  0 lvm  
  └─pve-root 254:1    0    70G  0 lvm  /

If successful, use raspi-config to set the boot order to be NVMe drive first.

Swap ¶

This sets up a logical volume for swap instead of using the default Raspberry Pi OS file swap.

Disable dphys-swapfile file swap.

Create the LVM2 logical volume of 8GB

lvm lvcreate pve -n swap -L 8G

Format the new swap space

mkswap /dev/pve/swap

Check that the volume was created

lvdisplay

Output

  --- Logical volume ---
  LV Path                /dev/pve/swap
  LV Name                swap
  VG Name                pve
  LV UUID                Gb7n93-OBv9-YtPu-vz7K-GeXK-G5fY-W2QQ3T
  LV Write Access        read/write
  LV Creation host, time raspberrypi, 2025-03-30 05:11:22 +0100
  LV Status              available
  # open                 0
  LV Size                8.00 GiB
  Current LE             2048
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:0

/etc/fstab

Manual

 /dev/pve/swap swap swap defaults 0 0

Enable the extended logical volume

swapon -va

Test that the swap has been extended properly

cat /proc/swaps # free

Output

Filename                                Type            Size            Used            Priority
/dev/dm-0                               partition       8388592         0               -2

LVM Thin ¶

Create an LVM thin pool which allocates blocks when they are written, thereby saving disk space.

Code

(
  lvcreate -L 100G -n data pve && \
  lvconvert --type thin-pool pve/data
)

/etc/pve/storage.cfg

lvmthin: local-lvm
         thinpool data
         vgname pve
         content rootdir,images

Troubleshooting¶

Activate LVM volume group

vgchange -ay

Code

foo@pi23:/wrk $ cat sdlvm
#!/bin/false

#umount /mnt/dev/p2
#umount /mnt/dev/p1
#vgremove $(hostname)
#pvremove "$1""2"

which pvs || exit 1
[ -z "$1" ] && exit 1

E=echo
#chk and umount /dev/media/*
$E vgchange -an -y $(hostname)
$E vgremove -y $(hostname)
$E pvremove "$1""2"

$E parted -s "$1" mklabel gpt || exit 1
$E parted -s "$1" mkpart primary 4M 516M || exit 1
$E parted -s "$1" mkpart primary 516M 64G || exit 1
$E parted -s "$1" name 1 bootfs || exit 1
$E parted -s "$1" name 2 rootfs || exit 1
$E parted -s "$1" set 1 boot on || exit 1
$E parted -s "$1" set 2 lvm on || exit 1
$E pvcreate "$1""2"  || exit 1
$E vgcreate $(hostname) "$1""2" || exit 1
$E lvcreate -y -n rootfs -l 100%FREE $(hostname) || exit 1
$E mkfs.vfat "$1""1" || exit 1
$E mkfs.ext4 "/dev/"$(hostname)"/rootfs" || exit 1
$E mkdir -p /mnt/dev/p{1,2} || exit 1
$E mount "$1""1" /mnt/dev/p1 || exit 1
$E mount "/dev/"$(hostname)"/rootfs" /mnt/dev/p2 || exit 1
$E update-initramfs -u -k all
$E DRY=" " sys-rbackup /boot/firmware/ /mnt/dev/p1/ || exit 1
$E DRY=" " sys-rbackup / /mnt/dev/p2/ || exit 1
$E sed -i -e "s,root=[^[:space:]]*,root=/dev/mapper/$(hostname)-rootfs," /mnt/dev/p1/cmdline.txt || exit 1
$E sed -i -e "s,PART.*=.*-01[^[:space:]]*,PARTLABEL=bootfs," /mnt/dev/p2/etc/fstab || exit 1
$E sed -i -e "s,PART.*=.*-02[^[:space:]]*,/dev/mapper/$(hostname)-rootfs," /mnt/dev/p2/etc/fstab || exit 1

$E cat /mnt/dev/p2/etc/fstab
$E umount /mnt/dev/p2 || exit 1
$E cat /mnt/dev/p1/cmdline.txt
$E umount /mnt/dev/p1 || exit 1

LVM¶

Setup¶

Equipment¶

Host System¶

SD Card¶

Host System¶

SD Card¶

Swap¶

LVM Thin¶

Troubleshooting¶

References¶

LVM ¶

Swap ¶

LVM Thin ¶