May 6th, 2024
In my Raspberry Pi home lab I run 3 Pimox Nodes in a cluster. I don’t connect USB drive to any of them and I don’t use the microSD cards to hold virtual machines, iso disks or backup. Rather, and I use network drives from network attached storage (NAS) to hold all my virtual machines and backups. I connect each of the nodes to network drives I have on a 4B 2gb Raspberry Pi running Open Media Vault.
Why Use Shared Storage
The primary purpose is to be able to share the network storage between all the Pimox nodes. This way all the Pimox Nodes can access the same files off of a single location.
Some benefits:
- Provides one location for Virtual Machines
- I’m not limited to the storage on the MicroSD cards
- Templates and VMs can be migrated between nodes. This way you can keep a VM online when doing updates or reboots to a Node or when a Node goes offline.
- Provides one repository for iso images
- Provides one repository for backups
- Templates can be stored on one node, but be cloned to another
- If you turn on High Availability (HA) and a node goes offline, the VM is automatically migrated to another node
My Setup
I have a 2 gig Pi 4 with Open Media Vault and a 1 TB SSD drive. The OMV shares holds all my virtual machines. While sharing a single drive can cause i/o issues, I don’t run anything very read / write intensive so the shared ssd doesn’t have an issue with multiple VMs and nodes using it at one time. The key is to not boot 2+ VMs at the same time, stream from multiple services at the same time or any other set of hard drive intensive efforts at the same time. I/O could climb and cause slowdowns or the share could be knocked offline, requiring vm reboots at best, corrupted VM data at the worst.
Why Open Media Vault
I use OMV because it’s easier than building my own NFS server and managing the drives, file systems, shares, permissions, etc. at the command line. I’m lazy that way. OVM also let’s me run BTRFS vs EXT4. Another option is ZFS, but I choose not to use it. I used BTRFS instead of EXT4 because if I move to a sata controller running off of pci, I can easily setup a raid configuration. You can also use EXT4, it works great as well and I’d be using it if I didn’t have the other option.
Backups and ISO images – A Synology NAS
My other shared storage is my Synology NAS. I use it to store backups, and ISO images both directly from Pimox but also from the Proxmox Backup Server I am using. The VM backups on my NAS can then be uploaded to the cloud with the rest of my files. So I have the VM on OVM and the backups in the NAS and cloud. 3 is 2, 2 is 1 and 1 is none. I could use the NAS as a shared drive for the VMs, but then I hit up against i/o issues from the mechanical drives and other devices using the NAS.
Conclusion
Shared network storage on a NAS is a must if you are running multiple Proxmox / Pimox nodes. The configuration makes it easy to share iso, templates, backups and virtual machines.