DCI Job Requirement for:
Data Pipeline Engineer
Location: New York, NY (Hybrid Remote)
Job Description:
Serve as a Data Pipeline engineer
Ensure customers work with updated and accurate data
Be the first to respond to data health failures on key pipelines
Read code to identify fixes and make changes to improve data pipeline performance
Work alongside pipeline engineers from other customer accounts
Maintain and build schedules so that pipelines run
Set up and maintain health checks on different pipelines
Respond to, triage, and debug pipelines when there is a problem
- Including limited out of hours support for critical issues
Read and write code changes and/or modify monitoring set-up where necessary
Know and understand how to navigate the pipelines and documentation
Follow SOPs to contact other teams and data providers when data is incorrect
Communicate outages with the end users of a pipeline
Contribute to and monitor tooling improvements (where feasible)
Qualifications:
BS in Computer Science / Engineering (or related career field)
5+ years of data engineering
Comfortable reading and writing code in SQL, Python, Pyspark and Java
Basic understanding of parallel data processing and the basics of tuning Spark jobs
Practical experience with performing root cause analysis
Experience documenting lessons learned from production incidents
Ability to work within an agile team
*Must be available to respond if there is a critical outage during their assigned “on-call” weeks. After an on-call weekend the engineer will receive 2 days off*
Active DoD Top-Secret clearance
Job Type: Full-time
Salary: $150,000-$200,000
Job Type: Full-time
Pay: $150,000.00 - $200,000.00 per year
Benefits:
- 401(k)
- 401(k) matching
- Dental insurance
- Employee assistance program
- Flexible schedule
- Flexible spending account
- Health insurance
- Health savings account
- Life insurance
- Paid time off
- Parental leave
- Professional development assistance
- Relocation assistance
- Retirement plan
- Tuition reimbursement
- Vision insurance
Schedule:
Experience:
- Data engineering: 5 years (Preferred)
- python programmin: 5 years (Preferred)
- data pipeline: 3 years (Preferred)
- agile team: 3 years (Preferred)
Security clearance:
Work Location: In person