Associate Data PractitionerDemo

AutoML Natural Language provides managed custom sentiment-model training and deployment from labeled text, avoiding the operational and machine-learning engineering overhead of building and hosting a TensorFlow or Spark MLlib model. Using the raw review text retains sentiment-bearing language, emojis, and slang for the custom text model. AutoML Text is now deprecated, with Google directing new custom text classification and sentiment workloads to Vertex AI Gemini prompting and tuning.

Cloud Storage Object Lifecycle Management applies rules based on conditions such as object age. Its SetStorageClass action can move older objects to lower-cost classes, and its Delete action can remove objects after the required retention period, aligning cost optimization with defined tiering and retention policies.

Dataflow Job Builder is a visual Google Cloud console interface for creating and running Dataflow pipelines without code. It supports Pub/Sub input, transformation steps, and BigQuery output, including streaming from Pub/Sub to BigQuery with transformations.

BigQuery table snapshots retain a point-in-time copy independently of the base table and can be restored to a writable table. They provide retention beyond BigQuery time travel’s maximum seven-day recovery window, while time travel can recover more recent changes made after the latest weekly snapshot when the deletion is within that window.

Dataflow provides managed real-time stream processing from Pub/Sub and can transform the records before writing them. BigQuery is a SQL data warehouse that Looker can query for analysis. Google Cloud’s Dataflow architecture explicitly describes Pub/Sub ingestion, Dataflow transformation, BigQuery storage, and Looker BI analysis.

Dataflow is designed for scalable batch and streaming data pipelines: it can read data from Cloud Storage, apply transformations, and write the transformed results to BigQuery. Google Cloud provides Cloud Storage-to-BigQuery Dataflow templates that include transformation support, making it appropriate for a growing 100 GB JSON workload.

BigQuery row-level access policies enforce filters on table rows for specified users or groups, such as a filter that permits a representative to see only rows for their assigned region. Policy tags and data masking control column access or presentation, while dataset-level IAM permissions do not filter individual table rows. Use row-level security

Cloud Composer is a fully managed Apache Airflow service designed for data-driven workflows. Its DAGs define task ordering and dependencies, support scheduled execution, and use Google Cloud operators to interact with Cloud Storage, Apache Spark, and BigQuery.

BigQuery materialized views precompute query results, reducing the work required for repeated dashboard joins and aggregations. They are automatically refreshed as base-table data changes and can use base-table changes to produce up-to-date results, making them appropriate for low-latency dashboards over streaming data.

Dataform is a managed BigQuery transformation service that uses SQLX and JavaScript, supports version-controlled repositories, enables reusable modular code, and provides data-quality assertions. These assertions test output and source data as part of the transformation workflow.

Cloud Logging provides the pipeline error messages needed to diagnose intermittent failures, and Cloud Monitoring exposes operational metrics such as CPU utilization and memory usage that can help identify resource pressure or other conditions associated with those failures.

A BigQuery ML LOGISTIC_REG model performs classification using a label and input features. Naming churned as label identifies the target without INPUT_LABEL_COLS, while SELECT * EXCEPT(churned) retains the demographic and purchase-history columns as features and prevents the target from being used as a feature. Create an ML model in BigQuery ML by using SQL

Cloud Storage can publish an OBJECT_FINALIZE notification to Pub/Sub whenever a new object is successfully created. A Dataflow streaming pipeline can consume these events, read the referenced CSV files, perform cleaning, deduplication, and BigQuery-based enrichment, and write the transformed data to BigQuery with low latency and without operating Composer workflows or Dataproc clusters.

BigQuery authorized views share the output of a defined query with specified users while withholding access to the underlying source dataset. An authorized view can expose only the required aggregated performance data and continues to reflect source-data updates, enforcing least-privilege access.

Looker custom fields can be created ad hoc from the field picker without Develop permission (subject to the create_custom_fields permission), and they generate SQL that runs against the database. Table calculations instead operate on data after it has been retrieved.

BigQuery can clean and transform data in place with GoogleSQL expressions and data-manipulation features. SQL functions can standardize formats and handle missing values, while query logic can identify or remove duplicates, making direct BigQuery transformation the fastest appropriate approach for data already stored there.

Cloud Run functions can be triggered by Cloud Storage object events through Eventarc, including when a new object is finalized. A function can initiate a BigQuery load job for each uploaded CSV, providing event-driven ingestion without operating scheduled orchestration or a cluster.

BigQuery Data Transfer Service has a Google Ads connector that loads Google Ads data into BigQuery, supports daily scheduled transfers, and allows backfill runs to retrieve historical data outside the normal refresh window. It provides the required managed, low-code automation without operating Cloud Composer, Dataflow, or a separate storage pipeline.

BigQuery uses IAM roles to control access to its resources. Granting authorized principals the least-permissive predefined or custom roles at the appropriate resource scope provides the permissions required to query the data without granting unnecessary access.

Cloud Storage Nearline is a low-cost class intended for data read or modified on average once per month or less, including data accessed monthly for analysis. Cloud Storage classes provide low-latency online access, so Nearline meets the rapid-retrieval requirement while costing less at rest than Standard storage.

Cloud SQL multi-region backups provide disaster-recovery capability for a regional outage while retaining the primary database in its local region, avoiding synchronous cross-region write latency. Cloud SQL high availability uses synchronous replication only between zones within the configured region, so it does not protect against loss of that entire region. Cross-region replicas replicate asynchronously and can lose recent committed transactions during a regional outage.

Looker supports centralized, scalable governance for enterprise BI. Google recommends using groups to manage users’ access controls, then applying roles and associated data, feature, and content permissions to those groups. This avoids administering access individually or duplicating reports and dashboards for each team.

The Google-provided Pub/Sub-to-BigQuery Dataflow template is a streaming pipeline that reads Pub/Sub messages and writes them to BigQuery. It can apply a user-defined function to transform each incoming message, providing a managed, low-development solution for high-volume near-real-time processing.

When existing pipeline tooling has a built-in BigQuery write connector, redirecting the current pipelines to BigQuery by reconfiguring their data mappings minimizes reimplementation effort and accelerates migration. Google Cloud recommends this approach for legacy ETL pipelines, provided the connector uses scalable BigQuery ingestion mechanisms such as batch loads or streaming inserts.