Skip to content

Proxmox

Proxmox is the hypervisor that I am using on most of my hardware.

I am using it over Portainer and kubernetes for ease of use and feature set.

Traefik

homelab/pve/traefik/conf.d/proxmox.yaml 
---
http:
 #region routers 
  routers:
    proxmox01:
      entryPoints:
        - "websecure"
      rule: "Host(`pve01.l.nicholaswilde.io`)"
      middlewares:
        - default-headers@file
        - https-redirectscheme@file
      tls: {}
      service: proxmox01
    proxmox02:
      entryPoints:
        - "websecure"
      rule: "Host(`pve02.l.nicholaswilde.io`)"
      middlewares:
        - default-headers@file
        - https-redirectscheme@file
      tls: {}
      service: proxmox02
    proxmox03:
      entryPoints:
        - "websecure"
      rule: "Host(`pve03.l.nicholaswilde.io`)"
      middlewares:
        - default-headers@file
        - https-redirectscheme@file
      tls: {}
      service: proxmox03
    proxmox04:
      entryPoints:
        - "websecure"
      rule: "Host(`pve04.l.nicholaswilde.io`)"
      middlewares:
        - default-headers@file
        - https-redirectscheme@file
      tls: {}
      service: proxmox04

#endregion
#region services
  services:
    proxmox01:
      loadBalancer:
        servers:
          - url: "https://192.168.2.128:8006"
        passHostHeader: true
    proxmox02:
      loadBalancer:
        servers:
          - url: "https://192.168.2.88:8006"
        passHostHeader: true
    proxmox03:
      loadBalancer:
        servers:
          - url: "https://192.168.2.198:8006"
        passHostHeader: true
    proxmox04:
      loadBalancer:
        servers:
          - url: "https://192.168.2.67:8006"
        passHostHeader: true


#endregion

🛠 Post Installation

Default Port: 8006

Post Install

bash -c "$(wget -qLO - https://github.com/community-scripts/ProxmoxVE/raw/main/misc/post-pve-install.sh)"

bash -c "$(wget -qLO - https://github.com/asylumexp/Proxmox/raw/main/misc/post-pve-install.sh)"

Add LXC IP Tag

bash -c "$(wget -qLO - https://github.com/community-scripts/ProxmoxVE/raw/main/misc/add-lxc-iptag.sh)"

bash -c "$(wget -qLO - https://github.com/community-scripts/ProxmoxVE/raw/main/misc/add-lxc-iptag.sh)"

Update

bash -c "$(wget -qLO - https://github.com/community-scripts/ProxmoxVE/raw/main/misc/update-lxcs.sh)"

bash -c "$(wget -qLO - https://github.com/community-scripts/ProxmoxVE/raw/main/misc/add-lxc-iptag.sh)"

Datacenter NFS Volumes

GUI: Datacenter -> Storage -> Add -> NFS

pve-backups

ID: pve-backups

Server: omv.l.nicholaswilde.io

Export: /export/pve-backups

pve-shared

ID: pve-shared

Server: omv.l.nicholaswilde.io

Export: /export/pve-shared

Reset Cluster Info

How to reset cluster. Useful if the node IP isn't matching during join.

node

(
  systemctl stop pve-cluster corosync && \
  pmxcfs -l
)

node

(
  rm -rf /etc/corosync/*
  rm -rf /etc/pve/corosync.conf 
  killall pmxcfs
  systemctl start pve-cluster
)

The node is now separated from the cluster. You can deleted it from any remaining node of the cluster

pvecm delnode oldnode

If the command fails due to a loss of quorum in the remaining node, set the expected votes to 1 as a workaround

pvecm expected 1

And then repeat the pvecm delnode command.

Now switch back to the separated node and delete all the remaining cluster files on it. This ensures that the node can be added to another cluster again without problems.

Separated node

rm -rf /var/lib/corosync/*

As the configuration files from the other nodes are still in the cluster file system, you may want to clean those up too. After making absolutely sure that you have the correct node name, you can simply remove the entire directory recursively from /etc/pve/nodes/NODENAME.

Code

rm -rf /etc/pve/nodes/NODENAME

Warning

The node’s SSH keys will remain in the authorized_key file. This means that the nodes can still connect to each other with public key authentication. You should fix this by removing the respective keys from the /etc/pve/priv/authorized_keys file.

Static IP

⚙ Node

WIP

📦 Container

WIP

🖥 VM

WIP

authentik

Proxmox GUI

Datacenter -> Permissions -> Realms

Issuer URL: http://authentik.l.nicholaswilde.io/application/o/proxmox

Realm: authentik

Client ID: from authentik

Client Key: from authentik

Autocreate Users: ✅

Username Claim: username

Create a Volume Group

Create a partition

sgdisk -N 1 /dev/sdb

Create a [P]hysical [V]olume (PV) without confirmation and 250K metadatasize.

pvcreate --metadatasize 250k -y -ff /dev/sdb1

Create a volume group named vmdata on /dev/sdb1

vgcreate vmdata /dev/sdb1

Create a LVM-thin pool

lvcreate -L 80G -T -n vmstore vmdata

Resize VM Disks

🖼 Step 1: Increase/resize disk from GUI console

Step 2: Extend physical drive partition

check free space

fdisk -l

Extend physical drive partition

growpart /dev/sda3

See physical drive

pvdisplay
Output

Instruct LVM that disk size has changed

pvresize /dev/sda3

Check physical drive if has changed

pvdisplay
Output

🧠 Step 3: Extend Logical volume

View starting LV

lvdisplay

Resize LV

lvextend -l +100%FREE /dev/ubuntu-vg/ubuntu-lv

View changed LV

lvdisplay
Output

📂 Step 4: Resize Filesystem

Resize Filesystem

resize2fs /dev/ubuntu-vg/ubuntu-lv

Confirm results

fdisk -l
Output

🔑 private key /root/.ssh/id_rsa contents do not match

Run on all nodes

(
  cd /root/.ssh && \
  mv id_rsa id_rsa.bak && \
  mv id_rsa.pub id_rsa.pub.bak && \
  mv config config.bak
)

Run on all nodes

pvecm updatecerts

Pass Disk to VM

List the disks by ID

ls -n /dev/disk/by-id/

Attach the disk to the VM

/sbin/qm set [VM-ID] -virtio2 /dev/disk/by-id/[DISK-ID]

Kill Backup Job

vzdump -stop

ps awxf | grep vzdump
Output
2444287 ?        Ds     0:00 task UPID:server_name:00254BFF:06D36C31:63BB7524:vzdump::root@pam:

kill -9 <process id>

🔔 Email Notifications using Gmail

Install dependencies

(
  apt update
  apt install -y libsasl2-modules mailutils
)

Enable 2FA for the gmail account that will be used by going to security settings.

Create app password for the account.

  1. Go to App Passwords
  2. Select app: Mail
  3. Select device: Other
  4. Type in: Proxmox or whatever you want here

Write gmail credentials to file and hash it

echo "smtp.gmail.com [email protected]:yourpassword" > /etc/postfix/sasl_passwd

Set file permissions to u=rw

chmod 600 /etc/postfix/sasl_passwd

Generate /etc/postfix/sasl_passwd.db

postmap hash:/etc/postfix/sasl_passwd

Warning

Comment out the existing line containing just relayhost= since we are using this key in our configuration we just pasted in.

Append the following to the end of the file: /etc/postfix/main.cf and comment out relayhost=

mydestination = $myhostname, localhost.$mydomain, localhost
# relayhost = 
mynetworks = 127.0.0.0/8
inet_interfaces = loopback-only
recipient_delimiter = +

compatibility = 2

relayhost = smtp.gmail.com:587
smtp_use_tls = yes
smtp_sasl_auth_enable = yes
smtp_sasl_security_options =
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_tls_CAfile = /etc/ssl/certs/Entrust_Root_Certification_Authority.pem
smtp_tls_session_cache_database = btree:/var/lib/postfix/smtp_tls_session_cache
smtp_tls_session_cache_timeout = 3600s
Example Screenshot

Screen Shot 2022-09-26 at 10 36 42 AM

Reload postfix

postfix reload

Test to make sure everything is hunky-dory

echo "sample message" | mail -s "sample subject" [email protected]

📨 SMTP Setup

Proxmox GUI

Server: smtp.gmail.com

Encryption: STARTTLS

Port: 587

Authenticate: ✅

Username: [email protected]

Password: password

From Address: [email protected]

Recipient(s): root@pam

Addtional Recipient(s): [email protected]

test

/etc/pve/notifications.cfg 
smtp: example
        mailto-user root@pam
        mailto-user admin@pve
        mailto [email protected]
        from-address [email protected]
        username pve1
        server mail.example.com
        mode starttls
The matching entry in /etc/pve/priv/notifications.cfg, containing the secret token 
smtp: example
        password somepassword

Targets to notify

WIP

🔗 References