Associate Data Practitioner

You need to design a data pipeline that processes large volumes of raw server log data stored in Cloud Storage. The data must be cleaned, transformed, and aggregated before it is loaded into BigQuery for analysis. The transformation requires complex data manipulation using Spark scripts developed by your team. You need to implement a solution that uses your team's existing skill set, processes data at scale, and minimizes cost. What should you do?