How To (Force) Delete a GCP Cloud Composer Environment
GCP Cloud Composer is a wonderful product aimed at delivering a managed, easy-to-use Apache Airflow deployment. It comes as a single resource (or so we think), runs of GKE — Google Kubernetes Cluster — and allows data engineers to run their pipelines without knowing much about infrastructure (or so we think).
We use Cloud Composer a lot. We see graphs like this, a lot:
Sometimes, when redeploying or deleting a Composer environment from terraform or UI you will experience errors that cannot be recovered.
Cloud Composer operations — be it creation, modification or deletion — can take anywhere between 10 and 40 min. If you see an operation going beyond 54 min (yep, that precise number) — it is likely it will fail. Pretty frustrating, isn’t it? Waiting for an hour only to see an error you can’t recover from.
In this case, here is how you can delete the environment manually, using Google Cloud Console and the CLI tool: gcloud
DRAGONS! These steps cannot be recovered from. Make sure you know what you are doing.
In the Google Cloud Console:
- Navigate to Delete the GKE cluster, that corresponds to the environment
- Delete the Google Storage bucket used by the environment (Composer → Look at your environment→ DAGs folder)
Click on the “DAGs”, note the name of the bucket, then go to Storage Browser and delete the bucket:
Pro Tip: To find any section or any resource in GCP, click on Search and type in a key word, like “Composer” or “your-project-name”
Delete the remaining resources using CLI
- Delete the GKE deployments using CLI
gcloud deployment-manager deployments list --project $PROJECT_NAMEgcloud deployment-manager deployments delete $DEPLOYMENT_NAME --project $PROJECT_NAME# If it fails with RESOURCE_NOT_FOUND, do this:
gcloud deployment-manager deployments delete $DEPLOYMENT_NAME --project $PROJECT_NAME --delete-policy=ABANDON
2. Finally, proceed with deleting the Composer environment:
gcloud composer environments delete $COMPOSER_ENV_NAME --location $GCP_REGION