Proxmox¶

Proxmox is the hypervisor that I am using on most of my hardware.

I am using it over Portainer and kubernetes for ease of use and feature set.

Traefik¶

homelab/pve/traefik/conf.d/proxmox.yaml

---
http:
 #region routers 
  routers:
    proxmox01:
      entryPoints:
        - "websecure"
      rule: "Host(`pve01.l.nicholaswilde.io`)"
      middlewares:
        - default-headers@file
        - https-redirectscheme@file
      tls: {}
      service: proxmox01
    proxmox02:
      entryPoints:
        - "websecure"
      rule: "Host(`pve02.l.nicholaswilde.io`)"
      middlewares:
        - default-headers@file
        - https-redirectscheme@file
      tls: {}
      service: proxmox02
    proxmox03:
      entryPoints:
        - "websecure"
      rule: "Host(`pve03.l.nicholaswilde.io`)"
      middlewares:
        - default-headers@file
        - https-redirectscheme@file
      tls: {}
      service: proxmox03
    proxmox04:
      entryPoints:
        - "websecure"
      rule: "Host(`pve04.l.nicholaswilde.io`)"
      middlewares:
        - default-headers@file
        - https-redirectscheme@file
      tls: {}
      service: proxmox04

#endregion
#region services
  services:
    proxmox01:
      loadBalancer:
        servers:
          - url: "https://192.168.2.128:8006"
        passHostHeader: true
    proxmox02:
      loadBalancer:
        servers:
          - url: "https://192.168.2.88:8006"
        passHostHeader: true
    proxmox03:
      loadBalancer:
        servers:
          - url: "https://192.168.2.143:8006"
        passHostHeader: true
    proxmox04:
      loadBalancer:
        servers:
          - url: "https://192.168.2.67:8006"
        passHostHeader: true


#endregion

Post Installation¶

Default Port: 8006

Post Install

AMD64ARM64

bash -c "$(wget -qLO - https://github.com/community-scripts/ProxmoxVE/raw/main/misc/post-pve-install.sh)"

bash -c "$(wget -qLO - https://github.com/asylumexp/Proxmox/raw/main/misc/post-pve-install.sh)"

Add LXC IP Tag

AMD64ARM64

bash -c "$(wget -qLO - https://github.com/community-scripts/ProxmoxVE/raw/main/misc/add-lxc-iptag.sh)"

bash -c "$(wget -qLO - https://github.com/community-scripts/ProxmoxVE/raw/main/misc/add-lxc-iptag.sh)"

Update

AMD64ARM64

bash -c "$(wget -qLO - https://github.com/community-scripts/ProxmoxVE/raw/main/misc/update-lxcs.sh)"

bash -c "$(wget -qLO - https://github.com/community-scripts/ProxmoxVE/raw/main/misc/add-lxc-iptag.sh)"

Datacenter NFS Volumes¶

GUI: Datacenter -> Storage -> Add -> NFS

pve-backups

ID: pve-backups

Server: omv.l.nicholaswilde.io

Export: /export/pve-backups

pve-shared

ID: pve-shared

Server: omv.l.nicholaswilde.io

Export: /export/pve-shared

Reset Cluster Info ¶

How to reset cluster. Useful if the node IP isn't matching during join.

node

(
  systemctl stop pve-cluster corosync && \
  pmxcfs -l
)

node

(
  rm -rf /etc/corosync/*
  rm -rf /etc/pve/corosync.conf 
  killall pmxcfs
  systemctl start pve-cluster
)

The node is now separated from the cluster. You can deleted it from any remaining node of the cluster

pvecm delnode oldnode

If the command fails due to a loss of quorum in the remaining node, set the expected votes to 1 as a workaround

pvecm expected 1

And then repeat the pvecm delnode command.

Now switch back to the separated node and delete all the remaining cluster files on it. This ensures that the node can be added to another cluster again without problems.

Separated node

rm -rf /var/lib/corosync/*

As the configuration files from the other nodes are still in the cluster file system, you may want to clean those up too. After making absolutely sure that you have the correct node name, you can simply remove the entire directory recursively from /etc/pve/nodes/NODENAME.

Code

rm -rf /etc/pve/nodes/NODENAME

Warning

The node’s SSH keys will remain in the authorized_key file. This means that the nodes can still connect to each other with public key authentication. You should fix this by removing the respective keys from the /etc/pve/priv/authorized_keys file.

Static IP¶

Node¶

WIP

Container¶

WIP

VM¶

WIP

authentik ¶

Proxmox GUI

Datacenter -> Permissions -> Realms

Issuer URL: http://authentik.l.nicholaswilde.io/application/o/proxmox

Realm: authentik

Client ID: from authentik

Client Key: from authentik

Autocreate Users:

Username Claim: username

Create a Volume Group ¶

Create a partition

sgdisk -N 1 /dev/sdb

Create a [P]hysical [V]olume (PV) without confirmation and 250K metadatasize.

pvcreate --metadatasize 250k -y -ff /dev/sdb1

Create a volume group named vmdata on /dev/sdb1

vgcreate vmdata /dev/sdb1

Create a LVM-thin pool¶

lvcreate -L 80G -T -n vmstore vmdata

Resize LXC Disks ¶

Tip

This can be done from the GUI, but sometimes the LXC doesn't register the change.

From node

Stop the LXC

pct stop 108

Increase the absolute size to 20G

pct resize 108 rootfs 20G

Check the file system

e2fsck -f /dev/pve/vm-108-disk-0

Resize the file system (optional)

resize2fs /dev/pve/vm-108-disk-0

Start the LXC

pct start 108

Resize VM Disks ¶

Step 1: Increase/resize disk from GUI console¶

Step 2: Extend physical drive partition¶

check free space

fdisk -l

Extend physical drive partition

growpart /dev/sda 3

See physical drive

rootsudo

pvdisplay

sudo pvdisplay

Output

  --- Physical volume ---
  PV Name               /dev/sda3
  VG Name               ubuntu-vg
  PV Size               <30.25 GiB / not usable 16.50 KiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              7743
  Free PE               2048
  Allocated PE          5695
  PV UUID               nE8Q94-dVYI-8ZP3-VEar-vONW-Vday-L0JofP

Instruct LVM that disk size has changed

pvresize /dev/sda3

Check physical drive if has changed

pvdisplay

Output

Step 3: Extend Logical volume¶

View starting LV

lvdisplay

Resize LV

lvextend -l +100%FREE /dev/ubuntu-vg/ubuntu-lv

View changed LV

lvdisplay

Output

Step 4: Resize Filesystem¶

Resize Filesystem

resize2fs /dev/ubuntu-vg/ubuntu-lv

Confirm results

fdisk -l

Output

private key /root/.ssh/id_rsa contents do not match ¶

Run on all nodes

(
  cd /root/.ssh && \
  mv id_rsa id_rsa.bak && \
  mv id_rsa.pub id_rsa.pub.bak && \
  mv config config.bak
)

Run on all nodes

pvecm updatecerts

Pass Disk to VM ¶

List the disks by ID

ls -n /dev/disk/by-id/

Attach the disk to the VM

/sbin/qm set [VM-ID] -virtio2 /dev/disk/by-id/[DISK-ID]

Kill Backup Job ¶

vzdump -stop

ps awxf | grep vzdump

Output

2444287 ?        Ds     0:00 task UPID:server_name:00254BFF:06D36C31:63BB7524:vzdump::root@pam:

kill -9 <process id>

Email Notifications using Gmail ¶

Install dependencies

(
  apt update
  apt install -y libsasl2-modules mailutils
)

Enable 2FA for the gmail account that will be used by going to security settings.

Create app password for the account.

Go to App Passwords
Select app: Mail
Select device: Other
Type in: Proxmox or whatever you want here

Write gmail credentials to file and hash it

echo "smtp.gmail.com [email protected]:yourpassword" > /etc/postfix/sasl_passwd

Set file permissions to u=rw

chmod 600 /etc/postfix/sasl_passwd

Generate /etc/postfix/sasl_passwd.db

postmap hash:/etc/postfix/sasl_passwd

Warning

Comment out the existing line containing just relayhost= since we are using this key in our configuration we just pasted in.

Append the following to the end of the file: /etc/postfix/main.cf and comment out relayhost=

mydestination = $myhostname, localhost.$mydomain, localhost
# relayhost = 
mynetworks = 127.0.0.0/8
inet_interfaces = loopback-only
recipient_delimiter = +

compatibility = 2

relayhost = smtp.gmail.com:587
smtp_use_tls = yes
smtp_sasl_auth_enable = yes
smtp_sasl_security_options =
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_tls_CAfile = /etc/ssl/certs/Entrust_Root_Certification_Authority.pem
smtp_tls_session_cache_database = btree:/var/lib/postfix/smtp_tls_session_cache
smtp_tls_session_cache_timeout = 3600s

Example Screenshot

Screen Shot 2022-09-26 at 10 36 42 AM

Reload postfix

postfix reload

Test to make sure everything is hunky-dory

echo "sample message" | mail -s "sample subject" [email protected]

SMTP Setup ¶

Proxmox GUI

Server: smtp.gmail.com

Encryption: STARTTLS

Port: 587

Authenticate:

Username: [email protected]

Password: password

From Address: [email protected]

Recipient(s): root@pam

Addtional Recipient(s): [email protected]

test

/etc/pve/notifications.cfg

smtp: example
        mailto-user root@pam
        mailto-user admin@pve
        mailto [email protected]
        from-address [email protected]
        username pve1
        server mail.example.com
        mode starttls

The matching entry in /etc/pve/priv/notifications.cfg, containing the secret token

smtp: example
        password somepassword

Targets to notify¶

WIP

Proxmox¶

Traefik¶

Post Installation¶

Datacenter NFS Volumes¶

Static IP¶

Node¶

Container¶

VM¶

Create a LVM-thin pool¶

Step 1: Increase/resize disk from GUI console¶

Step 2: Extend physical drive partition¶

Step 3: Extend Logical volume¶

Step 4: Resize Filesystem¶

Targets to notify¶

References¶