Upgrading Apache Cassandra from Version 3.1.15 and Higher to 4.1.x on Ubuntu 20.04.5 LTS: A Comprehensive Guide

Upgrading Apache Cassandra to a newer version is a significant task that database administrators undertake to ensure their systems benefit from new features, enhanced security measures, and improved performance. This guide provides a detailed walkthrough for upgrading Apache Cassandra from version 3.1.15 and higher to the latest 4.1.x version, specifically on Ubuntu 20.04.5 LTS, with an emphasis on pre-upgrade cleaning operations to manage disk space effectively.

Pre-upgrade Preparation

Backup Configuration Directory:

Before initiating the upgrade, it’s crucial to back up the Cassandra configuration directory. This precaution allows for a swift restoration of the configuration should any issues arise during the upgrade process. Utilize the following command to create a backup, incorporating the current date into the folder name for easy identification:

Pre-Cleanup Operations

Preparation is key to a smooth upgrade. Begin with maintenance commands to guarantee data integrity and optimize space usage, especially important for systems with limited disk space.

Scrub Data:

Execute nodetool scrub to clean and reorganize data on disk. Given that this operation may be time-consuming, particularly for databases with large amounts of data or limited disk space, it’s a critical step for a healthy upgrade process.

Clear Snapshots:

To further manage disk space, use nodetool clearsnapshot to remove existing snapshots, freeing up space for the upgrade process. To delete all snapshots on the node, simply use this method if you’re running out of space:

Cleanup Data:

Perform a nodetool cleanup to purge unnecessary data. In scenarios where disk space is a premium, it’s advisable to execute a scrub operation without generating a snapshot to conserve space:

Draining and Stopping Cassandra

Drain the Node:

Prior to halting the Cassandra service, ensure all data in memory is flushed to disk with nodetool drain.

Stop the Cassandra Service:

Cease the running Cassandra services to proceed with the upgrade safely:

Upgrading Cassandra

Update Source List:

Edit the repository sources to point to the new version of Cassandra by adjusting the cassandra.sources.list file:

Upgrade Packages:

With the repository sources updated, refresh the package list and upgrade the packages. When executing the apt upgrade command, you can keep pressing Enter as the default option is ‘N’ (No):

Modify Configuration:

Adjust the Cassandra configuration for version 4.1.x by commenting out or deleting deprecated options:

Update JAMM Library:

Ensure the Java Agent Memory Manager (JAMM) library is updated to enhance performance:

Backup and update the JVM options file:

It’s a good practice to back up configuration files before making changes. This step renames the existing jvm-server.options file to jvm-server.options.orig as a backup. Then, it copies the jvm.options file to jvm-server.options to apply the standard JVM options for Cassandra servers.

Optimization and Verification

Optimize Memory Usage:

Post-upgrade, it’s beneficial to evaluate and optimize memory usage and swap space to ensure efficient Cassandra operation:

Restart the Cassandra Service:

Apply the new version by restarting the Cassandra service:

Verify Upgrade:

Confirm the success of the upgrade by inspecting the cluster’s topology and state, ensuring all nodes are functional:

By adhering to this comprehensive guide, database administrators can effectively upgrade Apache Cassandra to version 4.1.x, capitalizing on the latest advancements and optimizations the platform has to offer, while ensuring data integrity and system performance through careful pre-upgrade preparations.

Optimization and Verification

After successfully upgrading Apache Cassandra to version 4.1.x and ensuring the cluster is fully operational, it’s crucial to conduct post-upgrade maintenance to optimize the performance and security of your database system. This section outlines essential steps and considerations to maintain a healthy and efficient Cassandra environment.

Monitor Performance and Logs

In the immediate aftermath of the upgrade, closely monitor the system’s performance, including CPU, memory usage, and disk I/O, to identify any unexpected behavior or bottlenecks. Additionally, review the Cassandra system logs for warnings or errors that may indicate potential issues requiring attention.

Tune and Optimize

Based on the performance monitoring insights, you may need to adjust Cassandra’s configuration settings for optimal performance. Consider tuning parameters related to JVM options, compaction, and read/write performance, keeping in mind the specific workload and data patterns of your application.

Run nodetool upgradesstables

To ensure that all SSTables are updated to the latest format, execute nodetool upgradesstables on each node in the cluster. This operation will rewrite SSTables that are not already in the current format, which is essential for taking full advantage of the improvements and features in Cassandra 4.1.x (Check the space, and if required, delete all snapshots as shown above.):

This process can be resource-intensive and should be scheduled during off-peak hours to minimize impact on live traffic.

Implement Security Enhancements

Cassandra 4.1.x includes several security enhancements. Review the latest security features and best practices, such as enabling client-to-node encryption, node-to-node encryption, and advanced authentication mechanisms, to enhance the security posture of your Cassandra cluster.

Review and Update Backup Strategies

With the new version in place, reassess your backup strategies to ensure they are still effective and meet your recovery objectives. Verify that your backup and restore procedures are compatible with Cassandra 4.1.x and consider leveraging new tools or features that may have been introduced in this release for more efficient data management.

Error on Oracle Linux 7 rpmdb: BDB0113

Sometimes on Oracle Linux there is problem, when you try use “yum” and get error like this:

The fixing of this problem, you can try rebuild db:

And then should work:

That’s all!

Automatic update of the Debian Linux test environment

A convenient way to maintain the test environment is automatic updating.
However, remember to set the backup, e.g. the day before – I always set it like that in proxmoxe.
The script that updates the Debian system looks like this:

You can download the script from:

The script cleans unnecessary deb files after the update.
Keep in mind the permissions and capabilities of the script:

In crontab, I set the day after automatic backup in proxmoxe:

Of course, the script can be added in the production environment, but it should not be added to the crontab.

E-mail notification about new updates Proxmox

This script send you e-mail notification about new updates:

You can download the script from:

You should set the correct e-mail address in the script, for now it is soban@soban.pl
Notification it’s looks something like that:

Should the script be executed:

Also added to the crontab, every day to extend a new update:

Remember don’t to upgrade your system on Friday! Good luck!

Upgrade Debian GNU/Linux 10 (buster) to 11 (bullseye)

Before we start upgrading the Debian system version, please make a snapshot or a possible backup of the system. Such a change entails significant changes that can damage the system. Of course, before you upgrade the system in production, it is best to upgrade to the testing environment first. The upgrade process affects the entire system. Services may not be available at this time. If the system is doing the hosting for your website then it may not be available!

Always bear in mind that the production environment is different from test environment, so I recommend that you do it carefully. A good practice is to keep a time interval between the upgrade of test and production environment, in my case it is a week. Remember not to make changes to production on Friday!

This is how we check the version of the system:

And now, we going to update OS, but before we do that – we will make copy of:

There is sorces setup for buster:

Before we upgrade the debian system version to 11, we need to do a full upgrade:

Then we use sed to replace “buster” on “bullseye”:

There here’s the effect:

To be sure, we will do a comparison of the backed up file:

And then we can do update:

So that it is not too easy, to start the task in the form of:

Specifically about this error:

To fix it, use any editor on the file (vi in ​​my case):

We remove line 4 – “deb http://security.debian.org buster/updates main contrib” as you can see, we go out and save :

Of course, any other editor like nano is also good for this case.
The contents of the /etc/apt/sources.list file now looks like this:

We can try to update the system again:

Now you need get some the system upgrade process has started. You can go for a coffee, or not 😉
There will be questions.
And more information, albout apt listchanges:

Just press ‘q’ and enter.

In this case we press enter.

Services to restart:

Enter agien.

This is question, about ssh deamon configuration:

In my case, I press enter because I don’t want to make changes to its configuration.

If all went well, we can reboot the system:

Now check the system:

Congratulations, we are on the new version of the system!
At this point, we can verify all services, for example whether the website is working properly. If it is OK, upgrade the production environment.

In my cases, I have problem PHP new version.
PHP have no persmision on nginx to user – when I try enter to page:

So we need to fix that in this way:

There is something wrong with the permissions:

Let’s make some backup:

I will replace www-data to nginx:

Now it’s looks better:

And restart services:

Also checking status:

Another problem is that, mariadb was removed, so I install it agien:

and that resolved all my problems.

To be 100% sure, I reloaded the entire machine.