VM disk imaging across machines with different sector sizes

2023-12-12

When transferring VM images between machines with different sector sizes (for example, one machine uses 512-byte sectors and another uses 4096-byte sectors), you may need to edit the partition table of the target device using a tool like parted. This ensures that the start and end positions align with the actual physical positions on the disk. Additionally, you may need to create a BIOS boot partition if it has gone missing.

In some cases, the machine may not boot at all. If this happens, you'll need to mount the partition using the kpartx -a command and reinstall grub using the grub-install --skip-fs-probe --boot-directory=/mnt/boot /dev/vg0/ubuntu-jammy command. You might also need to edit the /etc/fstab file to change the UUID of the root partition. There's much more to be said on this topic, but this provides a basic overview.

Correct partition table on 512-byte sector disk

fdisk -l /dev/vg0/ubuntu-jammy shows

Disk /dev/vg0/ubuntu-jammy: 20 GiB, 21474836480 bytes, 41943040 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 91928C6A-4801-4D6B-AED5-80D89AEB7BC2

Device                 Start      End  Sectors Size Type
/dev/vg0/ubuntu-jammy1  2048     4095     2048   1M BIOS boot
/dev/vg0/ubuntu-jammy2  4096 41940991 41936896  20G Linux filesystem

sfdisk -d /dev/vg0/ubuntu-jammy shows

label: gpt
label-id: 91928C6A-4801-4D6B-AED5-80D89AEB7BC2
device: /dev/vg0/ubuntu-jammy
unit: sectors
first-lba: 34
last-lba: 41943006
sector-size: 512

/dev/vg0/ubuntu-jammy1 : start=        2048, size=        2048, type=21686148-6449-6E6F-744E-656564454649, uuid=34CE0326-8129-490A-8370-78FCAE67E912
/dev/vg0/ubuntu-jammy2 : start=        4096, size=    41936896, type=0FC63DAF-8483-4772-8E79-3D69D8477DE4, uuid=372F3800-D9B1-4C0A-9EC2-739E8367A3AA

Wrong partition table on 4K sector disk

After transferring the VM iamge, e.g using dd you'll end up with a partition table which is incorrect. This is because the partition table still thinks it is on a 512-byte sector disk.

fdisk -l /dev/vg1/ubuntu-jammy shows

GPT PMBR size mismatch (41943039 != 5242879) will be corrected by write.
Disk /dev/vg1/ubuntu-jammy: 20 GiB, 21474836480 bytes, 5242880 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device                 Boot Start     End Sectors Size Id Type
/dev/vg1/ubuntu-jammy1          1 5242879 5242879  20G ee GPT

sfdisk -d /dev/vg1/ubuntu-jammy shows

GPT PMBR size mismatch (41943039 != 5242879) will be corrected by write.
label: dos
label-id: 0x00000000
device: /dev/vg1/ubuntu-jammy
unit: sectors
sector-size: 4096

/dev/vg1/ubuntu-jammy1 : start=           1, size=     5242879, type=ee

Fixing the incorrect table

On the correct table, take note of start and end sectors, and divide by 8. This is the 4K sector positions we need to use. If you're going from 4K to 512 byte, then you'll need to multiply by 8 instead. We'll assume the sector positions are multiples of 8. If it is not, there may be other ways you can deal with this, but it's not described here.

sectors 256 up to 511 for BIOS grub
sectors 512 to end of disk for main partition

Use parted to make a new partition table aligned to those values. Here are some commands which could be used, but you need to use your own values for your desk. Change units and print the partition table as you progress.

(parted) mklabel gpt
(parted) mkpart primary 256s 511s
(parted) mkpart primary 512s -1s
(parted) set 1 boot off
(parted) set 1 bios_grub on
(parted) unit s
(parted) print
Model: Linux device-mapper (linear) (dm)
Disk /dev/dm-8: 5242880s
Sector size (logical/physical): 4096B/4096B
Partition Table: gpt
Disk Flags:

Number  Start  End       Size      File system  Name     Flags
 1      256s   511s      256s                   primary  bios_grub
 2      512s   5242874s  5242363s  ext4         primary

Trying booting machine, if it doesn't boot, you'll need to re-install grub.

Read the partition table and create device maps with kpartx

kpartx /dev/vg1/ubuntu-jammy

It should return something like

vg1-ubuntu--jammy1 : 0 2048 /dev/vg1/ubuntu-jammy 2048
vg1-ubuntu--jammy2 : 0 41938904 /dev/vg1/ubuntu-jammy 4096

Mount the mapped device to /mnt

mount /dev/mapper/vg1-ubuntu--jammy2 /mnt

Run ls -l /mnt/boot and check we can see the contents of /boot on our partition.

total 249428
-rw-r--r-- 1 root root    262053 Aug 14 19:05 config-5.15.0-83-generic
-rw-r--r-- 1 root root    261844 Oct 31 07:15 config-5.15.0-89-generic
drwxr-xr-x 5 root root      4096 Dec  8 06:22 grub
lrwxrwxrwx 1 root root        28 Dec  8 05:45 initrd.img -> initrd.img-5.15.0-89-generic
-rw-r--r-- 1 root root 109539581 Dec  8 05:45 initrd.img-5.15.0-83-generic
-rw-r--r-- 1 root root 109550579 Dec  8 06:22 initrd.img-5.15.0-89-generic
lrwxrwxrwx 1 root root        28 Sep 13 02:03 initrd.img.old -> initrd.img-5.15.0-83-generic
-rw------- 1 root root   6273612 Aug 14 19:05 System.map-5.15.0-83-generic
-rw------- 1 root root   6276836 Oct 31 07:15 System.map-5.15.0-89-generic
lrwxrwxrwx 1 root root        25 Dec  8 05:45 vmlinuz -> vmlinuz-5.15.0-89-generic
-rw------- 1 root root  11615656 Aug 14 19:07 vmlinuz-5.15.0-83-generic
-rw------- 1 root root  11619464 Oct 31 07:25 vmlinuz-5.15.0-89-generic
lrwxrwxrwx 1 root root        25 Sep 13 02:03 vmlinuz.old -> vmlinuz-5.15.0-83-generic

(Re)Install grub

grub-install --skip-fs-probe --boot-directory=/mnt/boot /dev/vg1/ubuntu-jammy

Try booting again, if that doesn't work, try open a VNC port and see if that shows any more details.

Unmount partition at the end

umount /mnt

After fixing the partition table

Running fdisk -l /dev/vg1/ubuntu-jammy now shows the correct partition table.

Disk /dev/vg1/ubuntu-jammy: 20 GiB, 21474836480 bytes, 5242880 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 5C5C146F-D091-4D01-8585-C07C95C6876E

Device                 Start     End Sectors Size Type
/dev/vg1/ubuntu-jammy1   256     511     256   1M BIOS boot
/dev/vg1/ubuntu-jammy2   512 5242874 5242363  20G Linux filesystem

Running sfdisk -d /dev/vg1/ubuntu-jammy shows

label: gpt
label-id: 5C5C146F-D091-4D01-8585-C07C95C6876E
device: /dev/vg1/ubuntu-jammy
unit: sectors
first-lba: 6
last-lba: 5242874
sector-size: 4096

/dev/vg1/ubuntu-jammy1 : start=         256, size=         256, type=21686148-6449-6E6F-744E-656564454649, uuid=B96C4256-BC46-4993-8D74-4DA3206304F2, name="primary"
/dev/vg1/ubuntu-jammy2 : start=         512, size=     5242363, type=0FC63DAF-8483-4772-8E79-3D69D8477DE4, uuid=B9423558-D29B-4A23-9B25-BB8C38409E69, name="primary"

Mounting a device with a different sector size using loop device

If you need to mount a device with a 512-byte based partition table, but are on a 4K disk, you could use a loop device to mount it .

kpartx -a /dev/vg1/backup-dc2 doesn't work because the parition table sector positions don't align with where the partitions actually are on the disk.

Create loop device

losetup --find --show --sector-size 512 -r /dev/vg1/backup-dc2

It should respond with a device name like /dev/loop5.

fdisk -l /dev/loop5  # check partition table
kpartx -a /dev/loop5
mount -o ro /dev/mapper/loop5p1 /mnt

Do what you need with /mnt. Then clean up with:

umount /mnt
kpartx -d /dev/loop5
losetup -d /dev/loop5