Data Engineer Job Description
Focus is good comms , Python Spark. Knowledge of AtScale a Big Big Plus.
Job Responsibility
- Design and implement scalable data pipelines using PySpark/Python.
- Develop and optimize SQL queries for data extraction, transformation, and loading (ETL) processes.
- Utilize Databricks on AWS to build and manage data workflows.
- Able to read Legacy code and understand it
- Excellent know how on preparing script for data validation
- Monitor and troubleshoot data pipelines to ensure smooth operation.
- Document data processes and workflows for future reference and maintenance.
- Work closely with existing team to understand the requirement and deliver accordingly
Technical Skills :
- Strong proficiency in PySpark and Python programming.
- Extensive experience with SQL and database management.
- Hands-on experience with Databricks on AWS.
- Familiarity with AWS services such as S3, Lambda, Glue, and Redshift.
- Knowledge of data warehousing concepts and ETL processes.
- Excellent problem-solving skills and attention to detail.
- Strong communication and collaboration skills.
Job Types: Contract, Temporary
Pay: $60.00 per hour
Expected hours: 40 per week
Schedule:
- 8 hour shift
- Monday to Friday
Work Location: In person
