Professional Cloud DevOps EngineerDemo

SRE postmortems should result in blameless, concrete preventative action items that improve systems and processes. Requiring successful pre-release execution of test cases that detect the relevant error type directly reduces the likelihood that the same software-change failure reaches users. Google SRE guidance recommends preventative fixes and emphasizes that changing automated systems and processes is more reliable than trying to change human behavior.

Google Cloud Managed Service for Prometheus managed collection is Google’s recommended approach for Kubernetes environments. It collects Prometheus metrics from GKE workloads while the managed Kubernetes operator handles collector deployment, scrape configuration, scaling, and related operational maintenance; the collected data supports Cloud Monitoring querying and alerting.

GKE SYSTEM logging collects system logs required for cluster operations, while WORKLOAD logging collects logs generated by non-system application containers. Configuring the development cluster with --logging=SYSTEM prevents verbose application workload logs from being ingested into Cloud Logging while preserving GKE operational/system logs. About GKE logs

Binary Authorization can require a trusted attestation before an image is deployed to production. An automated load-test workload should create the attestation only after the test succeeds, using an asymmetric signing key held in Cloud KMS. Workload Identity Federation for GKE is the recommended way for a GKE workload to obtain the IAM authorization needed to use Google Cloud APIs, avoiding less-secure, long-lived service account JSON keys stored as Kubernetes Secrets.

For HTTP traffic, GKE Ingress provisions an external Application Load Balancer. With container-native load balancing, the load balancer targets Pod endpoints directly through network endpoint groups, avoiding the extra load-balancing hop through a node and its NodePort. External GKE Ingress/Application Load Balancers do not support Standard Tier routing.

Anthos Config Management provides GitOps-based configuration management for Kubernetes clusters: it uses the Git repository as the source of truth and reconciles repository changes into the enrolled clusters. Constraint templates and parameterized Policy Controller constraints can therefore be versioned in GitHub and automatically applied consistently across GKE clusters.

Artifact Registry is a fully managed Google Cloud artifact repository service that supports container images and Helm 3 charts stored as OCI artifacts in Docker repositories. It integrates with GKE and Cloud Run deployments, uses IAM for access control, and can be protected by VPC Service Controls.

Compute Engine machine type rightsizing recommendations use historical CPU and memory utilization metrics to identify a suitable VM machine type. The Recommender API exposes these recommendations programmatically, providing the direct Google-recommended method for selecting and applying a rightsizing recommendation.

Binary Authorization breakglass is an emergency, Pod-specific policy override enabled with the image-policy.k8s.io/break-glass: "true" label. It permits the otherwise denied image for that deployment while automatically recording a breakglass event in Cloud Audit Logs, preserving an auditable exception without weakening cluster-wide policy.

Google Cloud does not allow deletion of an instance template while a managed instance group references it. Setting create_before_destroy = true in the instance template resource’s lifecycle block makes Terraform create the replacement template before destroying the existing one, allowing the managed instance group to switch references first and avoiding the failed deletion order.

Cloud Build triggers can start CI/CD workflows in response to source-code changes. A sound pipeline runs unit tests on each push, builds and stores the container image only after those tests pass, deploys that tested image to a test environment for integration and acceptance tests, and promotes it to production only when those tests succeed. Smoke tests after production deployment verify that the deployed application is operational.

Cloud Run instance-based billing, formerly called CPU always allocated, provides CPU even outside request processing. This supports background and asynchronous work performed by monitoring agents such as OpenTelemetry, preventing trace-export work from being paused after a response is returned.

The Private Logs Viewer role (roles/logging.privateLogViewer) grants read-only access to Data Access audit logs, whereas Logs Viewer does not. Granting that role to a group containing the security team follows Google Cloud IAM guidance to manage shared access through groups instead of individual user grants.

A canary deployment that correlates with a sharp rise in user-facing errors and latency should be rolled back immediately to stop exposing users to the suspected faulty release. Rollback is the fastest mitigation; monitoring, root-cause investigation, and postmortem documentation follow once user impact is contained.

Folders organize projects into logical groups such as environments and serve as policy-inheritance points. A production folder and a development folder let environment-specific IAM and organization policies apply consistently to every child project, while separating the projects into environment groups for cost organization and reporting.

The recommended approach is to keep secrets out of source code, container images, and build pipelines, and instead deliver them securely to each instance only when it is created. Using an encrypted configuration management system to inject the secret at instance creation time ensures the secret remains encrypted at rest and in transit until it is needed, limits its exposure to only the specific running instance, and avoids storing plaintext credentials in images, logs, or version control — thereby minimizing the blast radius if a breach occurs. This aligns with Google Cloud's guidance on secret management, which emphasizes retrieving secrets at runtime/deploy time through secure, access-controlled channels rather than embedding them in artifacts or exposing them broadly.

Google Cloud recommends storing sensitive dependency values, including API keys, in Secret Manager. Cloud Run can expose a Secret Manager secret to the container as a secret-backed environment variable, with access controlled through the service identity's Secret Manager permissions.

Cloud Storage provides a secure upload destination controlled with IAM permissions. A Cloud Storage object-finalize event can initiate processing when a new batch is uploaded. Compute Engine managed instance groups can autoscale by adding VMs as demand increases and removing them as demand falls; an instance template can supply the preinstalled processing software. This matches an irregular, CPU-bound batch workload while avoiding continuously running processing capacity.

Cloud Build steps execute sequentially by default. Setting the later independent step to waitFor: ['-'] makes it start at build start alongside the first step. A subsequent step without waitFor waits for all preceding steps, or it can explicitly list both step IDs, so baz starts only after foo and bar succeed. Configuring the order of build steps

HTTP response latency is typically recorded as a distribution because it preserves the population of request durations in histogram buckets, allowing the latency experience across users to be analyzed. A GAUGE metric with a DISTRIBUTION value type is valid for this purpose, and a Heatmap displays a distribution-valued time series over time and latency buckets.

The Stackdriver/Cloud Logging agent on Compute Engine is a Fluentd-based agent. You can edit its local configuration to insert the built-in filter_record_transformer filter plugin, which can remove or transform specific fields (using its remove_keys parameter) from log records as they flow through the agent, before they are ever sent to Cloud Logging. This is the fastest remediation because it only requires updating the Fluentd agent configuration on the instance and restarting the agent — it does not require an application code change, and it avoids the extra latency and engineering effort of an intermediate storage-and-processing pipeline.

Latency SLO thresholds should reflect the level of performance at which users are satisfied, with modest headroom above current observed percentile latency to accommodate ordinary variation. A 190 ms p90 target and a 330 ms p95 target retain a meaningful performance commitment without setting the objective exactly at the current baseline. Google Cloud guidance defines SLOs around user happiness and recommends 28 days as a starting measurement period.

Rising oldest-unacknowledged-message age together with a large increase in undelivered messages indicates that Pub/Sub subscribers are not keeping up with the message volume. Adding Pod replicas creates more concurrent subscriber instances and raises processing throughput, reducing the backlog and inventory-update delay. The Pods are not near their CPU or memory limits, so increasing those limits is not the relevant response.

Effective postmortems are blameless learning records: they analyze the causes of an incident, identify how its impact or outcome could be improved, and produce actions that prevent recurrence. Collaborative authorship by incident participants improves completeness, while broad internal sharing spreads the lessons learned and helps prevent similar incidents elsewhere. Google SRE guidance recommends blameless analysis, corrective actions, collaboration, and sharing postmortems as widely as possible.