You can upgrade an existing AWS MSK cluster running Apache Kafka 2.4.1 to version 3.5.1. Here’s what you need to know:

• Upgrade path: AWS MSK supports in-place upgrades for Apache Kafka versions. This means you can upgrade directly from 2.4.1 to 3.5.1 without needing to create a new cluster.
• Client compatibility: Ensure your Kafka client applications are compatible with Apache Kafka 3.5.1. MSK only updates the server-side software during an upgrade. You’ll need to update your clients separately to leverage the new features in 3.5.1.
• Upgrade process: You can upgrade your MSK cluster using the AWS Management Console or the AWS CLI. The AWS documentation provides a guide for upgrading the Apache Kafka version using the console https://docs.aws.amazon.com/msk/latest/developerguide/version-support.html.

Additional notes:

• While AWS no longer offers creating new MSK clusters with Apache Kafka 2.4.1, they do provide a bug-fix version 2.4.1.1 which is compatible with existing 2.4.1 client applications. Upgrading to 2.4.1.1 before upgrading to 3.5.1 is recommended but not mandatory.
• Upgrading to a new major version (like 2.4.1 to 3.5.1) might introduce breaking changes. It’s recommended to thoroughly review the Apache Kafka release notes for 3.5.1 to identify any potential compatibility issues with your applications https://archive.apache.org/dist/kafka/3.5.1/RELEASE_NOTES.html.

Upgrading AWS MSK (Amazon Managed Streaming for Kafka) from Apache Kafka version 2.4.1 to 3.5.1 is a significant version jump, involving many intermediate versions and potentially significant changes in Kafka’s functionality and performance characteristics. Here’s a general approach on how to manage this upgrade:

1. Check Compatibility and Features:
   • Review the release notes for each major and minor Kafka version between 2.4.1 and 3.5.1. This will help you understand new features, deprecated features, and any breaking changes that could impact your applications.
   • Verify that your client applications are compatible with Kafka 3.5.1. Client compatibility can be a major concern when upgrading Kafka, as newer brokers typically support older clients, but the reverse is not always true.
2. Plan the Upgrade Path:
   • Given the number of version jumps, it might be safer and more manageable to upgrade through one or more intermediate versions rather than directly to 3.5.1. Each step should be planned based on the least amount of breaking changes and the greatest stability.
   • Check AWS MSK documentation or support to see what upgrade paths are officially supported. AWS MSK sometimes limits the versions to which direct upgrades are possible.
3. Test the Upgrade:
   • Before applying changes to your production environment, replicate your environment and perform the upgrade on the test cluster. This allows you to identify potential issues without affecting your production data.
   • Test all aspects of your Kafka deployment in this test environment, including data ingestion, stream processing, and any integrations with other services.
4. Perform the Upgrade:
   • Once testing is complete and you’re confident in the upgrade path, plan the upgrade of your production environment. This might involve some downtime, depending on your configuration and the specifics of how AWS MSK handles the upgrade process.
   • Follow the best practices for backup and disaster recovery. Ensure that you have reliable backups before starting the upgrade.
5. Monitor After Upgrade:
   • After upgrading, monitor the Kafka cluster closely. Check for any performance regressions or unexpected behavior.
   • Pay attention to the AWS MSK metrics and logs to ensure that the new version is stable and performs as expected.
6. Utilize AWS Support:
   • If you encounter any issues or if you need specific guidance, consider contacting AWS support. They can provide guidance tailored to your specific situation and configuration.