Security Log

Background

On November 1st, 2022 the OpenSSL Project patched two buffer overflow vulnerabilities (CVE-2022-3786; CVE-2022-3602). Under certain circumstances, an exploit could have resulted into an application crash (denial of service) or potential remote code execution.

Remediation

We have updated all our container images to include the latest OpenSSL version today #14333.

Update (21.10.2022)

We’ve discovered an additional instance of where a secret could end up in the logs - specifically the installer logs. This has been fixed as part of the 2022.09.2 hotfix, specifically via PR #13934.

Background

We have identified three issues in total that can lead to environment variables being written into logs, outlined below:

The first issue is regarding the ws-manager pod logging environment variables. If these environment variables include sensitive information, these would be included in the pod’s logs. This has since been fixed.

The second issue regards our common-go component, it includes redaction logic to sanitize log content. This is used by ws-manager and image-builder-mk3 pods, which redacts log content based on these keywords. This has been enhanced [1] [2] and is included in this release.

Finally we’ve identified where secrets can be logged into Kubernetes audit logs. More specifically, a user’s environment variables - and their SCM access token - were attached to the workspace pods in clear text. On platforms which log the entire Pod spec to the Kubernetes audit logs (e.g. GKE) this would lead to environment variable values - and the SCM access token - being logged. If such environment variables contain secrets these secrets would be logged to Kubernetes audit logs.

To fix this we now ship environment variables and the SCM token as a short-lived Kubernetes secret (the secret is deleted once the workspace pod is running). More details can be found here.

Remediation

For SaaS we will be taking measures to remove sensitive data from logs where applicable.

If you are running a self-hosted installation of Gitpod we recommend updating to version 2022.09.2 and to search for and remove any logs coming from ws-manager, installer and image-builder-mk3 that contain sensitive information.

We deployed a security update to Gitpod. This prevents an unsecured Google API endpoint from being available which can lead to information disclosure between workspaces.

If you are running a self-hosted installation of Gitpod, this is unlikely to affect you - we recommend updating to version 2022.01 nevertheless.

Background

We have been notified about this issue on February 20th which has been reviewed, validated and remediated the same day. Furthermore we conducted a root-cause analysis to identify and remediate the underlying issues of this exploit.

Lessons we learned

• We are now running active checks that will terminate the session when a workspace is connecting to the metadata API #8334
• The issue will be subject to penetration testing procedures
• We are adding automated tests which ensure the aforementioned checks remain in effect

Kudos to Alan and the Team

We would like to thank Alan Cao for notifying us about this issue including a comprehensive write-up that allowed an easy revalidation. Also, the Gitpod teams understood and remediated the issue swiftly - the timely response is key.