Certified Associate Developer for Apache SparkPreview

Which of the following operations can be used to return a DataFrame with no duplicate rows? Please select the most complete answer.

Which of the following Spark properties is used to configure the maximum size of an automatically broadcasted DataFrame when performing a join?

The code block shown below contains an error. The code block intended to return a DataFrame containing a column dayOfYear, an integer representation of the day of the year from column openDate from DataFrame storesDF. Identify the error.

Note that column openDate is of type integer and represents a date in the UNIX epoch format – the number of seconds since midnight on January 1st, 1970.

A sample of storesDF is displayed below:

Code block:

storesDF.withColumn("dayOfYear", dayofyear(col("openDate")))

Which of the following statements about slots is incorrect?

Which of the following storage levels should be used to store as much data as possible in memory on two cluster nodes while storing any data that does not fit in memory on disk to be read in when needed?

In what order should the below lines of code be run in order to write DataFrame storesDF to file path filePath as parquet and partition by values in column division?
Lines of code:

1. .write() \
2. .partitionBy("division") \
3. .parquet(filePath)
4. .storesDF \
5. .repartition("division")
6. .write \
7. .path(filePath, "parquet")

Which of the following operations can be used to return the number of rows in a DataFrame?

Which of the following operations returns a GroupedData object?

Which of the following code blocks fails to return a new DataFrame that is the result of an inner join between DataFrame storesDF and DataFrame employeesDF on column storeId and column employeeId?

Which of the following cluster configurations is most likely to experience delays due to garbage collection of a large Dataframe?

Note: each configuration has roughly the same compute power using 100GB of RAM and 200 cores.

The code block shown below should cache DataFrame storesDF only in Spark's memory. Choose the response that correctly fil ls in the numbered blanks within the code block to complete this task.

Code block:

1.2(3).count()

Which of the following code blocks returns a DataFrame containing only the rows from DataFrame storesDF where the value in column sqft is less than or equal to 25,000 OR the value in column customerSatisfaction is greater than or equal to 30?

The code block shown below contains an error. The code block is intended to return a new DataFrame from DataFrame storesDF where column storeId is of the type string. Identify the error.

Code block:

storesDF.withColumn(“storeId”, cast(col(“storeId”), StringType()))

The code block shown below should return a new DataFrame where column division from DataFrame storesDF has been renamed to column state and column managerName from DataFrame storesDF has been renamed to column managerFullName. Choose the response that correctly fills in the numbered blanks within the code block to complete this task.

Code block:

storesDF. 1(2, 2).4(5, 6)

The code block shown below should read a CSV at the file path filePath into a DataFrame with the specified schema schema. Choose the response that correctly fills in the numbered blanks within the code block to complete this task.

Code block:

1.2.3(4).format("csv").5(6)

Which of the following code blocks returns a new DataFrame where column division is the first two characters of column division in DataFrame storesDF?

The code block shown below should return a collection of summary statistics for column sqft in DataFrame storesDF. Choose the response that correctly fills in the numbered blanks within the code block to complete this task.

Code block:

storesDF.1(2)

The code block shown below should return a new DataFrame where rows in DataFrame storesDF with missing values in every column have been dropped. Choose the response that correctly fills in the numbered blanks within the code block to complete this task.

Code block:

storesDF.1.2__(3__ = 4)

Which of the following code blocks returns a new DataFrame with column storeReview where the pattern "End" has been removed from the end of column storeReview in DataFrame storesDF?

A sample DataFrame storesDF is below:

The code block shown below contains an error. The code block is intended to create a single-column DataFrame from Python list years which is made up of integers. Identify the error.

Code block:

spark.createDataFrame(years, IntegerType)

Which of the following operations will fail to trigger evaluation?

Which of the following code blocks returns a new DataFrame where column sqft from DataFrame storesDF has had its missing values replaced with the value 30,000?

A sample of DataFrame storesDF is below:

Which of the following statements about the Spark DataFrame is true?

Which of the following code blocks returns the number of rows in DataFrame storesDF for each unique value in column division?

Which of the following code blocks will always return a new 4-partition DataFrame from the 8-partition DataFrame storesDF without inducing a shuffle?