Certified Associate Developer for Apache SparkDemo

Spark accumulators let worker tasks contribute values that are automatically propagated to the driver, where the accumulator value can be read. A custom accumulator can define merge logic that retains the maximum processing time, including maxima keyed by worker node. Broadcast variables distribute a read-only value to workers and cannot collect worker-side updates.

broadcast() marks each supplied DataFrame as suitable for a broadcast join, and DataFrame.join() returns a joined DataFrame that can be joined again. Thus, broadcasting df2 in the first inner join and df3 in the chained inner join is valid. The predicate df2.id == df3.id correctly links the second join to the df2 column retained from the first join.

A broadcast hint marks a DataFrame as small enough for a broadcast join, making that relation the broadcast/build side. Broadcasting the much larger employeesDF replicates it across executors and is inefficient; the smaller storesDF should be broadcast instead. Apache Spark documents that a BROADCAST hint makes the hinted relation the broadcast side, regardless of spark.sql.autoBroadcastJoinThreshold.

A broadcast join distributes the smaller build-side DataFrame to executors, so the large transaction DataFrame can be joined without being shuffled across the cluster. Broadcasting the 1 GB lookup DataFrame therefore avoids shuffling the 128 GB DataFrame. Spark documents that a broadcast hint makes the hinted relation the broadcast join build side.

A skewed partition is processed by a single task, whose memory is limited by the executor running it; memory from other executors is not combined for that task. Scenario #6 provides only 12.5 GB per executor, the smallest per-executor memory allocation, so it has the least capacity for an unusually large partition and is the most susceptible to an out-of-memory error. Spark defines executor memory as the memory available to each executor process.

A grouped count is a supported stateful aggregation in Spark Structured Streaming. The engine maintains aggregation state by Id across incoming micro-batches and can emit the counts using an appropriate streaming output mode.

sort('region_id') sorts the DataFrame in ascending order by default, and take(3) retrieves the first three rows. Selecting region before region_id yields key-value pairs in the required region → region_id order, for IDs 0, 1, and 2.

Spark SQL's approx_count_distinct returns an estimated distinct count (using HyperLogLog++), not an exact one. Its optional relative-standard-deviation parameter controls the approximation tolerance; supplying it does not make the result exact. Use an exact distinct-count aggregate, such as count_distinct, when an exact value is required.

DataFrame.write provides a DataFrameWriter, whose json(path) method saves the DataFrame in JSON format at the specified path.

In PySpark, DataFrame.dropDuplicates() without a subset argument removes duplicate rows by considering all columns. It therefore retains one instance of each fully identical transaction record while preserving records that differ in any field.

PySpark DataFrame.union(other) returns a new DataFrame containing rows from both DataFrames and resolves corresponding columns by position rather than by name. Apache Spark: DataFrame.union

Spark Scala’s DataFrameNaFunctions.fill returns a new DataFrame and provides an overload that takes a numeric replacement value together with Seq[String] column names. Supplying 30000 and Seq("sqft") therefore replaces null numeric values only in sqft.

PySpark DataFrame.union returns a new DataFrame containing rows from both inputs and resolves corresponding columns by position rather than by name. PySpark DataFrame.union documentation

Multiplying a numeric value by 3.14159 produces a floating-point result. A PySpark UDF’s declared returnType specifies its Spark SQL output type, so this result should use T.DoubleType() and a Python float return annotation. The function must also multiply its defined parameter, t, rather than the undefined name answer.

approx_count_distinct("user_id") estimates the number of unique user IDs, and avg("transaction_amount") calculates the mean transaction amount. Passing both expressions to df.agg(...) computes the two global aggregates together in one aggregation operation, avoiding separate distinct-count and average workflows. Apache Spark documents approx_count_distinct as an aggregate that estimates distinct cardinality and DataFrame.agg as aggregation across the whole DataFrame.

Spark task slots represent executor capacity available for concurrent task execution, typically with a 1:1 relationship between task slots and available CPU cores. They enable parallel processing within a Spark application.

PySpark SQL DataFrame.join accepts a string as the name of a shared join column, but the how parameter defaults to inner. An outer join therefore requires an explicit join type such as "outer", e.g. storesDF.join(employeesDF, "storeId", "outer").

PySpark DataFrame.sort returns a DataFrame sorted by the specified column, and its ascending parameter defaults to True. Sorting the string column division this way produces alphabetical order.

withColumn returns a new DataFrame with the named column added or replaced. col("storeId") supplies a Column expression, and Column.cast(StringType()) converts that expression to Spark SQL's string data type.

PySpark DataFrame.drop(*cols) accepts multiple column names and returns a new DataFrame without those specified columns. Dropping first_name, last_name, email, and birthdate removes the PII fields, leaving the non-PII columns.

SparkSession createDataset creates a Dataset from a Scala sequence, and Dataset.toDF() converts that typed Dataset into a DataFrame. Creating a Dataset from List(years) and invoking toDF therefore produces a one-column DataFrame whose value is the integer list. IntegerType is a Spark SQL data-type descriptor, not the DataFrame conversion method required by this expression.

DataFrame.coalesce(1) reduces the number of partitions without a shuffle by default, using a narrow dependency. It returns a new DataFrame.

spark.udf.register associates a Python function with a SQL function name, which can then be invoked in selectExpr. The cube operation returns Python integer values, so IntegerType() is the matching declared Spark SQL return type. PySpark documents that a registered Python function’s produced value must match its specified return type.

The recursiveFileLookup option directs Spark to load files recursively under the specified base directory, so it includes every year/month/day subtree. This option is intended for the nested layout shown. Executable PySpark must call .parquet(...); the .paquet(...) spelling in the selected choice is a typographical error.