Proxmox VM Auto-Upgrade Script (qm + vzdump) with Network Tests and Auto-Restore

This article describes a simple (but effective) automation script for Proxmox VE that:

• creates a backup of a VM (optional),
• starts the VM if it was originally stopped,
• runs apt upgrade + reboot inside the VM,
• performs network tests before and after the update,
• restores from backup automatically if tests fail (optional),
• shuts down the VM again if it was originally stopped.

Where to download the script

You can download the script directly from my website:

soban.pl/bash/upgrade_proxmox_qm.sh

Required folders

Before you run anything, create two folders for scripts and logs:

mkdir -p /root/automate_scripts /root/logs

Install the script

Download the file into /root/automate_scripts/ and make it executable:

cd /root/automate_scripts
curl -fsSL -o upgrade_proxmox_qm.sh "https://soban.pl/bash/upgrade_proxmox_qm.sh"
chmod +x /root/automate_scripts/upgrade_proxmox_qm.sh

How it works (high level)

• Backup (vzdump snapshot + zstd) is executed first (if enabled).
• If the VM was stopped, the script starts it and waits WAIT_TIME seconds.
• Runs network tests BEFORE update:
• Executes apt update/upgrade/dist-upgrade/autoremove/clean and finally reboots the VM.
• Waits again (WAIT_TIME) and runs the same tests AFTER update.
• If tests fail and a backup exists, it performs qmrestore automatically.
• If the VM was originally stopped, it shuts down the VM at the end.

Usage

The script expects exactly two arguments:

• VMID – the Proxmox VM ID
• TIME_SEC – how long to wait after start/reboot (in seconds)

/root/automate_scripts/upgrade_proxmox_qm.sh 901 500

Tip: choose a WAIT_TIME that matches your VM boot time and apt upgrade duration. Typical values are 300 to 1000 seconds depending on the VM.

Full script (for reference)

If you prefer to keep everything in one place, below is the full script exactly as used in this setup:

#!/bin/bash

WAIT_TIME=$2  # wait time for VM start/restart (in seconds)
DO_BACKUP="y" # 'y' - create backup, 'n' - skip backup
DO_UPDATE="y" # 'y' - run system update, 'n' - skip update

echo "---------------START---------------"
date
echo "-----------------------------------"

# Argument check
if [ "$#" -ne 2 ]; then
    echo "Usage: $0 <VMID> <TIME_SEC>"
    exit 1
fi

VMID="$1"
TARGET_HOSTNAME=$(/usr/sbin/qm list | grep -w "$VMID" | awk '{print $2}')
VM_STATUS=$(/usr/sbin/qm status "$VMID" | awk '{print $2}')
HOST_MACHINE=$(hostname)
WAS_STOPPED=0

if [ "$VM_STATUS" = "stopped" ]; then
    WAS_STOPPED=1
else
    WAS_STOPPED=0
fi

# Create backup if DO_BACKUP is set to 'y'
backup_result="Backup not attempted"
if [ "$DO_BACKUP" = "y" ]; then
    echo "Creating backup for VM $VMID..."
    BACKUP_OUTPUT=$(/usr/bin/vzdump "$VMID" --storage local --mode snapshot --compress zstd)
    BACKUP_FILE=$(echo "$BACKUP_OUTPUT" | grep -oP 'vzdump-qemu-[^\s]+' | tr -d "'")

    if [ -z "$BACKUP_FILE" ]; then
        echo "Backup failed: No backup file created."
        backup_result="Backup created: NOK"
        exit 1
    else
        echo "Backup created successfully: $BACKUP_FILE"
        backup_result="Backup created: OK"
    fi
else
    echo "Backup not attempted (backup is disabled)."
fi

# Start VM if it was originally stopped
if [ "$WAS_STOPPED" -eq 1 ]; then
    echo "Starting VM $VMID for update..."
    /usr/sbin/qm start "$VMID"
    echo "Waiting $WAIT_TIME seconds for VM to start..."
    sleep "$WAIT_TIME"
fi

# Test functions
perform_ping_test() {
    source="$1"
    target="$2"
    if ssh root@"$source" "ping -c 3 -W 2 $target" >/dev/null 2>&1; then
        echo "Ping from $source to $target: OK"
        return 0
    else
        echo "Ping from $source to $target: NOK"
        return 1
    fi
}

test_curl() {
    source="$1"
    target="$2"
    if ssh root@"$source" "curl -s --head --connect-timeout 5 $target" >/dev/null 2>&1; then
        echo "Curl from $source to $target: OK"
        return 0
    else
        echo "Curl from $source to $target: NOK"
        return 1
    fi
}

perform_local_ping_test() {
    source="$1"
    target="$2"
    if ping -c 3 -W 2 "$target" >/dev/null 2>&1; then
        echo "Ping from $source to $target: OK"
        return 0
    else
        echo "Ping from $source to $target: NOK"
        return 1
    fi
}

DEFAULT_GW=$(/sbin/ip route | grep default | awk '{print $3}')

perform_tests() {
    status=0
    perform_local_ping_test "$HOST_MACHINE" "$TARGET_HOSTNAME" || status=1
    perform_ping_test "$TARGET_HOSTNAME" "8.8.8.8" || status=1
    perform_ping_test "$TARGET_HOSTNAME" "$DEFAULT_GW" || status=1
    test_curl "$TARGET_HOSTNAME" "http://google.com" || status=1
    return ${status:-0}
}

# General tests before update
echo "--- General tests BEFORE update ---"
if perform_tests; then
    echo "All network tests before update: OK"
    echo "$backup_result"

    if [ "$backup_result" == "Backup created: OK" ] || [ "$backup_result" == "Backup not attempted" ]; then
        echo "BEFORE status: OK"
    else
        echo "BEFORE status: NOK"
    fi
else
    echo "Some network tests before update: NOK"
    echo "$backup_result"
    echo "BEFORE status: NOK"
    [ "$WAS_STOPPED" -eq 1 ] && /usr/sbin/qm shutdown "$VMID"
    exit 1
fi
echo "-----------------------------------"

# System update and reboot
update_result="Upgrade system: NOK"
if [ "$DO_UPDATE" = "y" ]; then
    echo "--- Updating VM $VMID ---"
    if ssh root@"$TARGET_HOSTNAME" "/usr/bin/apt update && \
                          /usr/bin/apt upgrade -y && \
                          /usr/bin/apt dist-upgrade -y && \
                          /usr/bin/apt autoremove -y && \
                          /usr/bin/apt autoclean && \
                          /usr/bin/apt clean && \
                          /usr/bin/apt --purge autoremove && \
                          /sbin/reboot"; then
        echo "---END Updating VM $VMID ---"
        echo "Upgrade system: OK"
        update_result="Upgrade system: OK"
    else
        echo "Upgrade system: NOK"
        update_result="Upgrade system: NOK"
        [ "$WAS_STOPPED" -eq 1 ] && /usr/sbin/qm shutdown "$VMID"
        echo "Update failed. Exiting."
        exit 1
    fi
fi

# Wait for VM reboot if update was performed
if [ "$DO_UPDATE" = "y" ]; then
    echo "Waiting $WAIT_TIME seconds for VM to reboot..."
    sleep "$WAIT_TIME"
fi

# Tests after reboot if update was performed
if [ "$DO_UPDATE" = "y" ]; then
    echo "--- Network tests AFTER update ---"
    if perform_tests; then
        echo "All network tests after update: OK"
        echo "$update_result"
        echo "AFTER status: OK"
    else
        echo "Some network tests after update: NOK"
        echo "$update_result"
        echo "Attempting to restore from backup..."

        if [ "$DO_BACKUP" = "y" ]; then
            /usr/sbin/qm stop "$VMID"
            /usr/sbin/qmrestore "/var/lib/vz/dump/$BACKUP_FILE" "$VMID" --force

            if [ "$WAS_STOPPED" -eq 1 ]; then
                /usr/sbin/qm shutdown "$VMID"
            fi

            echo "Restore attempt finished. Please check VM status."
            echo "AFTER status: NOK"
        else
            echo "No backup available for restoration. The VM might not function properly after failed update."
            echo "AFTER status: NOK"
        fi
    fi
    echo "-----------------------------------"
fi

# Shut down VM if it was originally stopped
if [ "$WAS_STOPPED" -eq 1 ]; then
    echo "Shutting down VM $VMID (originally stopped)."
    /usr/sbin/qm shutdown "$VMID"
fi

echo "---------------END---------------"
date
echo "-----------------------------------"

Cron jobs (weekly schedule + per-VM logs)

Below is an example crontab that runs upgrades once a week (different VM each weekday) and writes separate logs to /root/logs/.

Edit root crontab:

crontab -e

Paste rules like these (comments in English):

15 0 * * 1 /root/automate_scripts/upgrade_proxmox_qm.sh 901 500 > /root/logs/kali.log 2>&1                 # Monday: upgrade Kali Linux (VMID 901)
15 0 * * 2 /root/automate_scripts/upgrade_proxmox_qm.sh 903 500 > /root/logs/proxmox-test-01.log 2>&1      # Tuesday: upgrade proxmox-test-01 (VMID 903)
15 0 * * 3 /root/automate_scripts/upgrade_proxmox_qm.sh 904 500 > /root/logs/ubuntu-lte.log 2>&1           # Wednesday: upgrade ubuntu-lte (VMID 904)
15 0 * * 4 /root/automate_scripts/upgrade_proxmox_qm.sh 905 1000 > /root/logs/backup-machine.log 2>&1      # Thursday: upgrade backup-machine (VMID 905)

Quick checks / troubleshooting

• Make sure root can SSH into the VM by hostname (the script uses ssh root@<vm-hostname>).
• Make sure DNS (or /etc/hosts) resolves the VM hostname correctly from the Proxmox host.
• If apt needs user interaction, ensure your VMs are set up for non-interactive upgrades (or pin/hold packages that prompt).

That’s it. Drop the script into /root/automate_scripts/, send logs into /root/logs/, and your weekly VM maintenance becomes mostly fire-and-forget.

Artykuł Proxmox VM Auto-Upgrade Script (qm + vzdump) with Network Tests and Auto-Restore pochodzi z serwisu soban.

Upgrading Apache Cassandra to a newer version is a significant task that database administrators undertake to ensure their systems benefit from new features, enhanced security measures, and improved performance. This guide provides a detailed walkthrough for upgrading Apache Cassandra from version 3.1.15 and higher to the latest 4.1.x version, specifically on Ubuntu 20.04.5 LTS, with an emphasis on pre-upgrade cleaning operations to manage disk space effectively.

Pre-upgrade Preparation

Backup Configuration Directory:

Before initiating the upgrade, it’s crucial to back up the Cassandra configuration directory. This precaution allows for a swift restoration of the configuration should any issues arise during the upgrade process. Utilize the following command to create a backup, incorporating the current date into the folder name for easy identification:

# cp -r /etc/cassandra/ /root/cassandra-conf-bkp-$(date +%Y%m%d)

Pre-Cleanup Operations

Preparation is key to a smooth upgrade. Begin with maintenance commands to guarantee data integrity and optimize space usage, especially important for systems with limited disk space.

Scrub Data:

Execute nodetool scrub to clean and reorganize data on disk. Given that this operation may be time-consuming, particularly for databases with large amounts of data or limited disk space, it’s a critical step for a healthy upgrade process.

Clear Snapshots:

To further manage disk space, use nodetool clearsnapshot to remove existing snapshots, freeing up space for the upgrade process. To delete all snapshots on the node, simply use this method if you’re running out of space:

# nodetool clearsnapshot --all

Cleanup Data:

Perform a nodetool cleanup to purge unnecessary data. In scenarios where disk space is a premium, it’s advisable to execute a scrub operation without generating a snapshot to conserve space:

# nodetool scrub --no-snapshot

Draining and Stopping Cassandra

Drain the Node:

Prior to halting the Cassandra service, ensure all data in memory is flushed to disk with nodetool drain.

# nodetool drain

Stop the Cassandra Service:

Cease the running Cassandra services to proceed with the upgrade safely:

# systemctl stop cassandra.service

Upgrading Cassandra

Update Source List:

Edit the repository sources to point to the new version of Cassandra by adjusting the cassandra.sources.list file:

# echo "deb https://debian.cassandra.apache.org 41x main" > /etc/apt/sources.list.d/cassandra.sources.list

Upgrade Packages:

With the repository sources updated, refresh the package list and upgrade the packages. When executing the apt upgrade command, you can keep pressing Enter as the default option is ‘N’ (No):

# apt update && apt upgrade

Modify Configuration:

Adjust the Cassandra configuration for version 4.1.x by commenting out or deleting deprecated options:

# for var in thrift_prepared_statements_cache_size_mb start_rpc rpc_port rpc_server_type thrift_framed_transport_size_in_mb request_scheduler; do sed -i "/$var:/s/^/#/" /etc/cassandra/cassandra.yaml; done

Update JAMM Library:

Ensure the Java Agent Memory Manager (JAMM) library is updated to enhance performance:

# sed -i 's|jamm-0.3.0.jar|jamm-0.3.2.jar|g' /etc/cassandra/cassandra-env.sh

Backup and update the JVM options file:

It’s a good practice to back up configuration files before making changes. This step renames the existing jvm-server.options file to jvm-server.options.orig as a backup. Then, it copies the jvm.options file to jvm-server.options to apply the standard JVM options for Cassandra servers.

# cd /etc/cassandra/
# mv jvm-server.options jvm-server.options.orig && cp -p jvm.options jvm-server.options

Optimization and Verification

Optimize Memory Usage:

Post-upgrade, it’s beneficial to evaluate and optimize memory usage and swap space to ensure efficient Cassandra operation:

# swapoff -a && swapon -a

Restart the Cassandra Service:

Apply the new version by restarting the Cassandra service:

# systemctl start cassandra.service

Verify Upgrade:

Confirm the success of the upgrade by inspecting the cluster’s topology and state, ensuring all nodes are functional:

# nodetool describecluster
# nodetool status

By adhering to this comprehensive guide, database administrators can effectively upgrade Apache Cassandra to version 4.1.x, capitalizing on the latest advancements and optimizations the platform has to offer, while ensuring data integrity and system performance through careful pre-upgrade preparations.

Optimization and Verification

After successfully upgrading Apache Cassandra to version 4.1.x and ensuring the cluster is fully operational, it’s crucial to conduct post-upgrade maintenance to optimize the performance and security of your database system. This section outlines essential steps and considerations to maintain a healthy and efficient Cassandra environment.

Monitor Performance and Logs

In the immediate aftermath of the upgrade, closely monitor the system’s performance, including CPU, memory usage, and disk I/O, to identify any unexpected behavior or bottlenecks. Additionally, review the Cassandra system logs for warnings or errors that may indicate potential issues requiring attention.

Tune and Optimize

Based on the performance monitoring insights, you may need to adjust Cassandra’s configuration settings for optimal performance. Consider tuning parameters related to JVM options, compaction, and read/write performance, keeping in mind the specific workload and data patterns of your application.

Run nodetool upgradesstables

To ensure that all SSTables are updated to the latest format, execute nodetool upgradesstables on each node in the cluster. This operation will rewrite SSTables that are not already in the current format, which is essential for taking full advantage of the improvements and features in Cassandra 4.1.x (Check the space, and if required, delete all snapshots as shown above.):

# time nodetool upgradesstables

This process can be resource-intensive and should be scheduled during off-peak hours to minimize impact on live traffic.

Implement Security Enhancements

Cassandra 4.1.x includes several security enhancements. Review the latest security features and best practices, such as enabling client-to-node encryption, node-to-node encryption, and advanced authentication mechanisms, to enhance the security posture of your Cassandra cluster.

Review and Update Backup Strategies

With the new version in place, reassess your backup strategies to ensure they are still effective and meet your recovery objectives. Verify that your backup and restore procedures are compatible with Cassandra 4.1.x and consider leveraging new tools or features that may have been introduced in this release for more efficient data management.

Artykuł Upgrading Apache Cassandra from Version 3.1.15 and Higher to 4.1.x on Ubuntu 20.04.5 LTS: A Comprehensive Guide pochodzi z serwisu soban.

# yum update

error: rpmdb: BDB0113 Thread/process 16934/140571185240320 failed: BDB1507 Thread died in Berkeley DB library

error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery

error: cannot open Packages index using db5 -  (-30973)

error: cannot open Packages database in /var/lib/rpm

CRITICAL:yum.main:

 

Error: rpmdb open failed

The fixing of this problem, you can try rebuild db:

# rpm --rebuilddb

And then should work:

# yum update

That’s all!

Artykuł Error on Oracle Linux 7 rpmdb: BDB0113 pochodzi z serwisu soban.

# cat /root/update.sh
#!/bin/bash
apt-get update
apt-get upgrade -q -y
apt-get autoremove -y -q
rm -rf /var/cache/apt/archives/*.deb

You can download the script from:

$ cd /root/
$ wget https://soban.pl/bash/update.sh

The script cleans unnecessary deb files after the update.
Keep in mind the permissions and capabilities of the script:

# chmod +x /root/update.sh

In crontab, I set the day after automatic backup in proxmoxe:

10 4 * * 3 /root/update.sh > /dev/null 2>&1

Of course, the script can be added in the production environment, but it should not be added to the crontab.

Artykuł Automatic update of the Debian Linux test environment pochodzi z serwisu soban.

# cat /root/check_update.sh
#!/bin/bash
email=soban@soban.pl
hostnam=`hostname`
apt-get update
# sometimes change number of lines (for example proxmox 4.4 - 4; proxmox 5 - 5)
if [ `apt-get --just-print upgrade | wc -l` != 5 ]
then
        apt-get --just-print upgrade |  mail -s "Update checker $hostnam" $email
fi

You can download the script from:

$ cd /root/
$ wget https://soban.pl/bash/check_update.sh

You should set the correct e-mail address in the script, for now it is soban@soban.pl
Notification it’s looks something like that:

Should the script be executed:

# chmod +x /root/check_update.sh

Also added to the crontab, every day to extend a new update:

00 06 * * * /root/check_update.sh > /dev/null 2>&1

Remember don’t to upgrade your system on Friday! Good luck!

Artykuł E-mail notification about new updates Proxmox pochodzi z serwisu soban.

And now, we going to update OS, but before we do that – we will make copy of:

cp /etc/apt/sources.list /root/sources.list_buster

There is sorces setup for buster:

# cat /etc/apt/sources.list
deb http://ftp.debian.org/debian buster main contrib
deb http://ftp.debian.org/debian buster-updates main contrib
deb http://security.debian.org buster/updates main contrib
deb http://nginx.org/packages/debian/ buster nginx
deb-src http://nginx.org/packages/debian/ buster nginx

Before we upgrade the debian system version to 11, we need to do a full upgrade:

# apt update && apt upgrade -q -y && apt dist-upgrade -q -y && apt autoremove -y -q

Then we use sed to replace “buster” on “bullseye”:

# sed -i 's/buster/bullseye/g' /etc/apt/sources.list

There here’s the effect:

# cat /etc/apt/sources.list

To be sure, we will do a comparison of the backed up file:

And then we can do update:

# apt update && apt upgrade -q -y && apt dist-upgrade -q -y && apt autoremove -y -q

So that it is not too easy, to start the task in the form of:

Specifically about this error:

Err:5 http://security.debian.org bullseye/updates Release
  404  Not Found [IP: 199.232.150.132 80]
Reading package lists... Done
E: The repository 'http://security.debian.org bullseye/updates Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

To fix it, use any editor on the file (vi in ​​my case):

# vi /etc/apt/sources.list

We remove line 4 – “deb http://security.debian.org buster/updates main contrib” as you can see, we go out and save :

Of course, any other editor like nano is also good for this case.
The contents of the /etc/apt/sources.list file now looks like this:

# cat /etc/apt/sources.list
deb http://ftp.debian.org/debian bullseye main contrib
deb http://ftp.debian.org/debian bullseye-updates main contrib
deb http://nginx.org/packages/debian/ bullseye nginx
deb-src http://nginx.org/packages/debian/ bullseye nginx

We can try to update the system again:

# apt update && apt upgrade -q -y && apt dist-upgrade -q -y && apt autoremove -y -q

Now you need get some the system upgrade process has started. You can go for a coffee, or not 😉
There will be questions.
And more information, albout apt listchanges:

Just press ‘q’ and enter.

In this case we press enter.

Services to restart:

Enter agien.

This is question, about ssh deamon configuration:

In my case, I press enter because I don’t want to make changes to its configuration.

If all went well, we can reboot the system:

# reboot

Now check the system:

Congratulations, we are on the new version of the system!
At this point, we can verify all services, for example whether the website is working properly. If it is OK, upgrade the production environment.

In my cases, I have problem PHP new version.
PHP have no persmision on nginx to user – when I try enter to page:

# tail -f /var/log/nginx/error-*.log 
unix:/var/run/php/php-fpm.sock failed (13: Permission denied)

So we need to fix that in this way:

# service php7.4-fpm status
* php7.4-fpm.service - The PHP 7.4 FastCGI Process Manager
     Loaded: loaded (/lib/systemd/system/php7.4-fpm.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2021-10-01 14:59:14 CEST; 3min 28s ago
       Docs: man:php-fpm7.4(8)
    Process: 289 ExecStartPost=/usr/lib/php/php-fpm-socket-helper install /run/php/php-fpm.sock /etc/php/7.4/fpm/pool.d/www.conf 74 (code=exited, status=0/S>
   Main PID: 116 (php-fpm7.4)
     Status: "Processes active: 0, idle: 2, Requests: 0, slow: 0, Traffic: 0req/sec"
        CPU: 364ms
     CGroup: /system.slice/php7.4-fpm.service
             |-116 php-fpm: master process (/etc/php/7.4/fpm/php-fpm.conf)
             |-287 php-fpm: pool www
             `-288 php-fpm: pool www

Oct 01 14:59:11 soban-pl systemd[1]: Starting The PHP 7.4 FastCGI Process Manager...
Oct 01 14:59:14 soban-pl systemd[1]: Started The PHP 7.4 FastCGI Process Manager.

There is something wrong with the permissions:

# grep www-data /etc/php/7.4/fpm/pool.d/www.conf
user = www-data
group = www-data
listen.owner = www-data
listen.group = www-data

Let’s make some backup:

# cp /etc/php/7.4/fpm/pool.d/www.conf /etc/php/7.4/fpm/pool.d/www.conf.backup.$(date +%F)

I will replace www-data to nginx:

# sed -i 's/www-data/nginx/g' /etc/php/7.4/fpm/pool.d/www.conf

Now it’s looks better:

# grep nginx /etc/php/7.4/fpm/pool.d/www.conf
user = nginx
group = nginx
listen.owner = nginx
listen.group = nginx

And restart services:

# service php7.4-fpm restart

Also checking status:

Another problem is that, mariadb was removed, so I install it agien:

# apt-get install mariadb-server mariadb-client -y

and that resolved all my problems.

To be 100% sure, I reloaded the entire machine.

Artykuł Upgrade Debian GNU/Linux 10 (buster) to 11 (bullseye) pochodzi z serwisu soban.