They need to be USC/GC
We are seeking a skilled Scala Data Engineer to join our team, focusing on designing and optimizing data pipelines to process large-scale transactional data. The role requires enterprise-level experience with Spark and Scala, handling high-volume data loads, and optimizing memory usage for efficient processing. You will play a key role in managing and improving our data pipelines, ensuring they can handle spikes in sales and transactional data flow.
Key Responsibilities:
- Develop and maintain data pipelines using Scala and Apache Spark.
- Write and optimize complex SQL queries for large datasets.
- Ensure efficient memory management and performance tuning of Spark jobs.
- Use Airflow for workflow orchestration.
- Process data on GCP platforms, including BigQuery.
- Collaborate with teams to handle data processing during high-traffic events, such as sales spikes.
Skills:
- Must have high-level knowledge of Spark Apache and SQL.
- Complex SQL queries.
- Huge amount of data loads.
- Memory experience.
- Enterprise experience.
- Data pipeline work.
- Spark memory experience.
- PySpark to write pseudo code.
- Airflow is required.
- GCP and BigQuery.
Qualifications:
- Proficiency in Scala and Apache Spark for data engineering tasks.
- Strong experience with complex SQL queries and large-scale data processing.
- Hands-on experience with workflow orchestration tools like Airflow.
- Knowledge of GCP services, especially BigQuery.
- Understanding of memory management and optimization for Spark workloads.
- Experience with PySpark for pseudocode or prototyping is a plus.
- Proven background working on enterprise-level data pipelines.
Job Type: Contract
Pay: $108,029.73 - $130,100.31 per year
Schedule:
Experience:
- Scala: 1 year (Required)
- PySpark: 1 year (Required)
- Airflow: 1 year (Required)
Ability to Commute:
- Sunnyvale, CA 94085 (Required)
Work Location: In person
