QuestionQ114

VerifiedMaintaining and automating data workloads

You migrated a Hadoop job from an on-premises cluster to Dataproc and GCS. Your Spark job is a complex analytical workload with many shuffle operations, and the initial data consists of Parquet files averaging 200–400 MB each. After migrating to Dataproc, you observe some performance degradation and want to optimize it. Your organization is highly cost-sensitive, so you want to keep using Dataproc on preemptible VMs, with only 2 non-preemptible workers for this workload.

What should you do?

Explanation

Shuffle data is written to worker-local disks. SSD-backed disks provide substantially better IOPS for shuffle-heavy Spark workloads, while increasing the preemptible workers’ boot-disk size provides additional shuffle capacity and can increase persistent-disk throughput. Keeping the source data in Cloud Storage avoids relying on transient HDFS storage on a preemptible-worker cluster.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!