Proxmox not booting after installing a new M.2 NVMe drive 😢

If you install a new M.2 NVMe drive inside Proxmox and also had GPU Passthrough working, after instal the M.2 NVMe, you may expereince that the NIC card isnt working. Here is a script to fix that.

This script:

  • Converts all fstab entries to UUIDs.
  • Removes all VFIO/IOMMU and GPU passthrough config.
  • Unbinds all VFIO devices, including NICs.
  • Rebuilds initramfs and updates GRUB.
  • Sets persistent naming for all NICs to avoid network conflicts after adding M.2.

After running, you can safely install the new M.2 and Proxmox should boot normally.

#!/bin/bash
echo "=== COMPLETE PASSTHROUGH REMOVAL SCRIPT ==="

# 1. Backup critical configs
echo "Backing up fstab, GRUB, and modprobe.d..."
cp /etc/fstab /etc/fstab.bak
cp /etc/default/grub /etc/default/grub.bak
mkdir -p /root/backup/modprobe.d/
cp -r /etc/modprobe.d/ /root/backup/modprobe.d.bak/

# 2. Convert fstab to UUID-based
echo "Converting fstab entries to UUIDs..."
TEMP_FSTAB=$(mktemp)
while IFS= read -r line; do
    if [[ "$line" =~ ^[[:space:]]*# ]]; then
        echo "$line"
    elif [[ "$line" =~ /dev/(nvme[^[:space:]]*|sd[a-z][0-9]*) ]]; then
        DEVICE="${BASH_REMATCH[0]}"
        if [ -b "$DEVICE" ]; then
            UUID=$(blkid -s UUID -o value "$DEVICE")
            if [ -n "$UUID" ]; then
                NEW_LINE=$(echo "$line" | sed "s|$DEVICE|UUID=$UUID|")
                echo "$NEW_LINE"
            else
                echo "$line"
            fi
        else
            echo "$line"
        fi
    else
        echo "$line"
    fi
done < /etc/fstab > "$TEMP_FSTAB"
mv "$TEMP_FSTAB" /etc/fstab

# 3. Remove VFIO/IOMMU from GRUB
echo "Cleaning GRUB of passthrough params..."
sed -i '/GRUB_CMDLINE_LINUX_DEFAULT=/ {
    s/intel_iommu=on//g
    s/amd_iommu=on//g
    s/iommu=pt//g
    s/vfio-pci.ids=[^ ]*//g
    s/rd.driver.pre=vfio-pci//g
    s/initcall_blacklist=sysfb_init//g
    s/pci=assign-busses//g
    s/pci=realloc//g
    s/  / /g
    s/^ *//g
    s/ *$//g
    s/=" *"/=""/g
}' /etc/default/grub
sed -i '/GRUB_CMDLINE_LINUX=/ {
    s/intel_iommu=on//g
    s/amd_iommu=on//g
    s/iommu=pt//g
    s/vfio-pci.ids=[^ ]*//g
}' /etc/default/grub

# 4. Remove all VFIO config files
echo "Deleting VFIO configs..."
rm -f /etc/modprobe.d/vfio*.conf
rm -f /etc/modprobe.d/pci-stub.conf
rm -f /etc/modprobe.d/blacklist-vfio.conf

# 5. Clean GPU blacklists
if [ -f /etc/modprobe.d/blacklist.conf ]; then
    sed -i '/blacklist nouveau/d; /blacklist nvidia/d; /blacklist amdgpu/d' /etc/modprobe.d/blacklist.conf
fi

# 6. Remove GPU module options
rm -f /etc/modprobe.d/nvidia.conf /etc/modprobe.d/amdgpu.conf /etc/modprobe.d/nouveau.conf

# 7. Remove passthrough services/scripts
rm -f /etc/systemd/system/vfio-bind.service
rm -f /etc/rc.local
rm -f /usr/local/bin/vfio-bind.sh
rm -f /usr/local/sbin/vfio-bind
systemctl daemon-reload 2>/dev/null

# 8. Unbind all VFIO-bound devices (GPU, NICs, other PCI)
echo "Unbinding all VFIO devices..."
for dev in /sys/bus/pci/drivers/vfio-pci/*; do
    if [ -e "$dev" ]; then
        echo "$(basename $dev)" > /sys/bus/pci/devices/$(basename $dev)/driver/unbind
    fi
done

# 9. Rebuild initramfs
echo "Rebuilding initramfs..."
update-initramfs -u -k all

# 10. Update GRUB
echo "Updating GRUB..."
update-grub

# 11. Set persistent network naming for all NICs
echo "Configuring persistent NIC naming..."
ip -o link show | awk -F': ' '/eth|enp/ {print $2, $3}' | while read IFNAME _; do
    MAC=$(cat /sys/class/net/$IFNAME/address)
    if [ -n "$MAC" ]; then
        echo "SUBSYSTEM==\"net\", ACTION==\"add\", DRIVERS==\"?*\", ATTR{address}==\"$MAC\", NAME=\"$IFNAME\"" \
        > /etc/udev/rules.d/70-persistent-net.rules
    fi
done
udevadm control --reload-rules
udevadm trigger

# 12. Verification
echo ""
echo "=== VERIFICATION ==="
grep "GRUB_CMDLINE_LINUX_DEFAULT" /etc/default/grub
ls -la /etc/modprobe.d/vfio* 2>/dev/null || echo "No VFIO config files found"
lspci -nn | grep -i "VGA\|3D controller"
echo "=== PASSTHROUGH REMOVAL COMPLETE ==="
echo ""
echo "System is ready for M.2 installation. Please shutdown and install the drive."

After Installing the M.2 run the following to verify Proxmox can see the new M.2

#List all NVMe drives
lsblk -d -o NAME,SIZE,MODEL,TYPE | grep nvme

nvme0n1  1.8T Samsung SSD 990 PRO 2TB disk
nvme1n1  3.6T Samsung SSD 990 PRO 4TB disk

# Or using PCI bus
lspci | grep -i nvme

02:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller S4LV008[Pascal]
04:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller S4LV008[Pascal]

# Check if the kernel sees the device
dmesg | grep -i nvme

nvme0n1  1.8T Samsung SSD 990 PRO 2TB disk
nvme1n1  3.6T Samsung SSD 990 PRO 4TB disk
02:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller S4LV008[Pascal]
04:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller S4LV008[Pascal]
[    4.904496] nvme 0000:02:00.0: platform quirk: setting simple suspend
[    4.904519] nvme 0000:04:00.0: platform quirk: setting simple suspend
[    4.904529] nvme nvme0: pci function 0000:02:00.0
[    4.904579] nvme nvme1: pci function 0000:04:00.0
[    4.910128] nvme nvme0: Shutdown timeout set to 10 seconds
[    4.910192] nvme nvme1: Shutdown timeout set to 10 seconds
[    4.912605] nvme nvme0: 16/0/0 default/read/poll queues
[    4.912865] nvme nvme1: 16/0/0 default/read/poll queues
[    4.914645]  nvme0n1: p1 p2 p3

Now you can add to lvm-thin storage

Notes:

  • Update NEW_DRIVE, VG_NAME, and THIN_POOL_LV to match your setup.
  • The script automatically detects filesystem type and resizes accordingly.
  • Verification steps show PV, VG, thin pool devices, and final filesystem usage.

Add new M.2 as new LVM-thin

#!/bin/bash
# Idempotent NVMe LVM-thin storage creation for Proxmox

VG_NAME="vg_nvme_auto"
LV_NAME="thinpool_nvme_auto"
STORAGE_ID="nvme-lvm-auto"

# === DETECT FREE NVMe DISK ===
FREE_DISK=$(lsblk -ndo NAME,TYPE | awk '$2=="disk" {print "/dev/"$1}' | while read d; do
    if ! pvs 2>/dev/null | grep -q "$d" && ! lsblk "$d" | grep -q part; then
        echo "$d"
        break
    fi
done)

if [ -z "$FREE_DISK" ]; then
    echo "ERROR: No free NVMe disk found."
    exit 1
fi

echo "Detected free NVMe disk: $FREE_DISK"

# === CREATE PV if not exists ===
if ! pvs | grep -q "$FREE_DISK"; then
    echo "Creating PV on $FREE_DISK..."
    pvcreate "$FREE_DISK"
else
    echo "PV already exists: $FREE_DISK"
fi

# === CREATE VG if not exists ===
if ! vgs | grep -q "$VG_NAME"; then
    echo "Creating VG $VG_NAME..."
    vgcreate "$VG_NAME" "$FREE_DISK"
else
    echo "VG already exists: $VG_NAME"
fi

# === CREATE thin pool if not exists ===
if ! lvs | grep -q "$LV_NAME"; then
    echo "Creating thin pool $LV_NAME..."
    lvcreate -l 100%FREE -T "$VG_NAME/$LV_NAME"
else
    echo "Thin pool already exists: $LV_NAME"
fi

# === ADD Proxmox storage if not exists ===
if ! pvesm status | grep -q "$STORAGE_ID"; then
    echo "Adding storage $STORAGE_ID to Proxmox..."
    pvesm add lvmthin "$STORAGE_ID" --vgname "$VG_NAME" --thinpool "$LV_NAME" --content images,rootdir
else
    echo "Proxmox storage already exists: $STORAGE_ID"
fi

# === VERIFY ===
echo ""
echo "=== Verification ==="
pvesm status | grep "$STORAGE_ID"
lvs -o lv_name,vg_name,lv_size,data_percent,metadata_percent,lv_attr | grep "$LV_NAME"
vgs | grep "$VG_NAME"
pvs | grep "$FREE_DISK"


# Next step: create the thin pool on this VG manually:

lvcreate -l 100%FREE -T vg_nvme_auto/thinpool_nvme_auto
pvesm add lvmthin nvme-lvm-auto --vgname vg_nvme_auto --thinpool thinpool_nvme_auto --content images,rootdir

# After that, verify:

lvs -o lv_name,vg_name,lv_size,data_percent,metadata_percent,lv_attr | grep thin
pvesm status | grep nvme-lvm-auto

Storage Reporter

#!/bin/bash
# Proxmox Disk Space Report Generator
REPORT_FILE="/root/proxmox_disk_report_$(date +%Y%m%d_%H%M%S).txt"

echo "=== PROXMOX COMPREHENSIVE DISK SPACE REPORT ===" > $REPORT_FILE
echo "Generated: $(date)" >> $REPORT_FILE
echo "Hostname: $(hostname)" >> $REPORT_FILE
echo "==============================================" >> $REPORT_FILE

# 1. System Filesystem Usage
echo "" >> $REPORT_FILE
echo "1. FILESYSTEM USAGE:" >> $REPORT_FILE
echo "====================" >> $REPORT_FILE
df -hT -x tmpfs -x devtmpfs >> $REPORT_FILE

# 2. Block Devices Overview
echo "" >> $REPORT_FILE
echo "2. BLOCK DEVICES DETAILS:" >> $REPORT_FILE
echo "=========================" >> $REPORT_FILE
lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT,LABEL,UUID >> $REPORT_FILE

# 3. Proxmox Storage Configuration
echo "" >> $REPORT_FILE
echo "3. PROXMOX STORAGE CONFIGURATION:" >> $REPORT_FILE
echo "================================" >> $REPORT_FILE
pvesm status >> $REPORT_FILE

# 4. Proxmox Storage Content Summary
echo "" >> $REPORT_FILE
echo "4. PROXMOX STORAGE CONTENT SUMMARY:" >> $REPORT_FILE
echo "===================================" >> $REPORT_FILE
for storage in $(pvesm status | awk '{print $1}'); do
    if [ "$storage" != "Name" ]; then
        echo "--- $storage ---" >> $REPORT_FILE
        pvesm list $storage 2>/dev/null | head -10 >> $REPORT_FILE
        echo "" >> $REPORT_FILE
    fi
done

# 5. LVM Information (if available)
echo "" >> $REPORT_FILE
echo "5. LVM STATUS:" >> $REPORT_FILE
echo "==============" >> $REPORT_FILE
if command -v pvs >/dev/null 2>&1; then
    echo "Physical Volumes:" >> $REPORT_FILE
    pvs >> $REPORT_FILE
    echo "" >> $REPORT_FILE
    echo "Volume Groups:" >> $REPORT_FILE
    vgs >> $REPORT_FILE
    echo "" >> $REPORT_FILE
    echo "Logical Volumes:" >> $REPORT_FILE
    lvs >> $REPORT_FILE
else
    echo "LVM not detected on this system" >> $REPORT_FILE
fi

# 6. LVM-Thin Specific Information
echo "" >> $REPORT_FILE
echo "6. LVM-THIN STATUS:" >> $REPORT_FILE
echo "===================" >> $REPORT_FILE
echo "Key Point: For LVM-Thin, always monitor BOTH data_percent and metadata_percent." >> $REPORT_FILE
echo "If metadata reaches 100%, the pool becomes unusable even if free space exists." >> $REPORT_FILE
if command -v lvs >/dev/null 2>&1; then
    echo "" >> $REPORT_FILE
    echo "Thin Pools:" >> $REPORT_FILE
    lvs -o lv_name,vg_name,lv_size,data_percent,metadata_percent,lv_attr | grep thin >> $REPORT_FILE
    echo "" >> $REPORT_FILE
    echo "Thin Volumes (sample):" >> $REPORT_FILE
    lvs -o name,size,data_percent,metadata_percent,pool_lv,origin,attr | grep -E "([^ ]*_thin|^LV)" | head -20 >> $REPORT_FILE
else
    echo "LVM tools not available" >> $REPORT_FILE
fi

# 7. ZFS Status (if available)
echo "" >> $REPORT_FILE
echo "7. ZFS STATUS:" >> $REPORT_FILE
echo "==============" >> $REPORT_FILE
if command -v zpool >/dev/null 2>&1; then
    echo "ZFS Pools:" >> $REPORT_FILE
    zpool list >> $REPORT_FILE
    echo "" >> $REPORT_FILE
    echo "ZFS Datasets:" >> $REPORT_FILE
    zfs list | head -15 >> $REPORT_FILE
else
    echo "ZFS not detected on this system" >> $REPORT_FILE
fi

# 8. Directory Usage Analysis
echo "" >> $REPORT_FILE
echo "8. KEY DIRECTORY USAGE:" >> $REPORT_FILE
echo "======================" >> $REPORT_FILE
echo "Proxmox directories:" >> $REPORT_FILE
du -sh /var/lib/vz 2>/dev/null >> $REPORT_FILE
du -sh /var/lib/vz/* 2>/dev/null >> $REPORT_FILE
echo "" >> $REPORT_FILE
echo "Other system directories:" >> $REPORT_FILE
du -sh /root /home /tmp /var/log 2>/dev/null >> $REPORT_FILE

# 9. Critical Usage Warnings
echo "" >> $REPORT_FILE
echo "9. CRITICAL USAGE WARNINGS:" >> $REPORT_FILE
echo "===========================" >> $REPORT_FILE

# Check filesystems over 80%
echo "Filesystems over 80% capacity:" >> $REPORT_FILE
df -h | awk 'NR>1 {gsub(/%/,""); if ($5 > 80) print $0}' >> $REPORT_FILE

# Check thin pools with high usage
if command -v lvs >/dev/null 2>&1; then
    echo "" >> $REPORT_FILE
    echo "Thin pools with high usage (over 80% in data or metadata):" >> $REPORT_FILE
    lvs -o name,data_percent,metadata_percent,attr 2>/dev/null | grep thin | \
    awk '{if ($2+0 > 80 || $3+0 > 80) print $0 "  <-- WARNING"}' >> $REPORT_FILE
    echo "" >> $REPORT_FILE
    echo "Reminder: If metadata reaches 100%, the pool becomes unusable immediately." >> $REPORT_FILE
fi

# 10. System Information
echo "" >> $REPORT_FILE
echo "10. SYSTEM INFORMATION:" >> $REPORT_FILE
echo "======================" >> $REPORT_FILE
echo "Uptime: $(uptime)" >> $REPORT_FILE
echo "Proxmox Version: $(pveversion 2>/dev/null || echo "Not available")" >> $REPORT_FILE
echo "Memory: $(free -h | awk '/Mem:/ {print $3"/"$2}')" >> $REPORT_FILE

echo "" >> $REPORT_FILE
echo "=== REPORT COMPLETE ===" >> $REPORT_FILE
echo "Report saved to: $REPORT_FILE" >> $REPORT_FILE

# Display report location
echo "Report generated: $REPORT_FILE"
echo ""
echo "To view the report:"
echo "  cat $REPORT_FILE"
echo ""
echo "To monitor live disk usage:"
echo "  watch -n 5 'df -h; echo; pvesm status'"