Infrastructure Update Changelog

❗️ This update transitions MySQL from v5.7 to v8 and incurs full downtime of up to 30 minutes.

How to update

This update uses a new update process that leverages pre-generated CloudFormation changesets that you only need to execute.

Your Gitpod Account Manager will provide you with links to the changeset for the role and the infrastructure. To apply the upgrade, you will need to visit these individually and execute the pre-generated changeset. Ensure to do this in the following order: first the role changeset, then the infrastructure changeset. Follow the process laid out on Updating the Gitpod Dedicated Infrastructure. If you need access to the full, updated CloudFormation template, please contact your Gitpod Account Manager.

Changelog

• Major MySQL upgrade to v8
• Switches CA to rds-ca-rsa2048-g1 to avoid expiry in Aug 2024
• Policy to enable lambda updates
• Policies to support cleaner stack deletion

Expected CloudFormation Change Set

The change set being generated as part of this CF change is expected to include the following changes:

❗️ This update impacts running workspaces and should not be done during working hours. You can expect a downtime of 4 minutes after the CloudFormation Change Set is applied as new nodes are spun up.

How to update

• We have improved the template distribution process starting with this release. We now distribute templates as S3 URLs, which are readable only from the cell AWS account. This paves the way for a much more simplified upgrade process moving forward, so stay tuned!

• Your Gitpod Account Manager will provide you with two updated CloudFormation templates in the form of S3 URLs(one for the infrastructure template role and one for Gitpod itself) that both need to be applied as change sets.

• Follow the process laid out on Updating the Gitpod Dedicated Infrastructure

Changelog

• Support for in-place AMI updates for all clusters
• Improved cell lambda image update workflow, triggered by new Gitpod Dedicated releases. This equips a faster rollout of cell Lambda code changes. This does not change the permission scope of the lambdas.
• Automated changeset creation process, thereby simplifying future infrastructure upgrades. After this upgrade you will be able to preview all the changes automatically in the stack’s Changesets section
• Dynamic autoscaler configurations enabling fine tuning of cluster scaling capacity to optimally accommodate varying workloads. You can talk to your Gitpod Account manager to get this configured
• Export of historical logs, thereby enhancing the ease of debugging issues that customers encounter
• Various bug fixes

Expected CloudFormation Change Set

The change set being generated as part of this CF change is expected to include the following changes:

Changes to Gitpod Role CF template

Changes to Gitpod Instance CF template

❗️ This update impacts running workspaces and should not be done during working hours. You can expect a downtime of 5 minutes after the CloudFormation Change Set is applied as new nodes are spun up.

ℹ️ Creating the change set can take longer than usual. Further, once the change stack is applied, the clean up step will take longer than usual - up to 40 minutes (see below for reasoning). The Gitpod instance can be used as normal during this time. Future updates will take less time again.

How to update

• Your Gitpod Account Manager will provide you with two CloudFormation templates (one for the infrastructure template role and one for Gitpod itself) that both need to be applied as change sets.

• Follow the process laid out on Updating the Gitpod Dedicated Infrastructure

Changelog

• Support for custom CA certificates (important: An application release is necessary to fully roll out this feature. You can ask your Gitpod Account Manager whether your instance has received the required release)
• Disabled scaling the instance to 0 nodes during working hours (6:00 to 22:00 local time to the instance) to speed up the workspace starts in the morning. Scale to 0 is still enabled on weekends.
• Improvements of log groups associated with Lambda functions to reduce cost and align function names with AWS conventions. This requires all lambdas to be recreated, leading to the longer than usual clean up time mentioned above.
• Enforce use of IMDSv2 AWS metadata endpoint for EC2 instances
• Various bug fixes

Expected CloudFormation Change Set

The change set being generated as part of this CF change is expected to include the following changes:

Changes to the stack for the role used to execute the Gitpod CF template:

Changes to Gitpod CF template

How to update

• Follow the process laid out in Updating the Gitpod Dedicated Infrastructure

• This update does not impact running workspaces and can be done during working hours.

Changelog

• Update to the application controller (Lambda) to improve the ordering of its operations
• Turn off debug mode for the telemetry controller as it was logging too much
• Turn off AZ rebalancing which was impacting the stability of some nodes and thus workspaces
• Set workspace DNS resolvers to be local VPC resolver IP instead of public DNS lookup. This resolves networking issues in environments where public DNS lookups are blocked. This is the first of a two part roll out process, the second part is an application change.

Expected CloudFormation Change Set

The change set being generated as part of this CF change is expected to include the following 14 changes: