DCI Job Requirement for:
Data Pipeline Engineer
Location: NYC, NY
Job Description:
Serves as a Data Pipeline Engineer
Ensure that customers work with updated and accurate data
Be the first to respond to data health failures on key pipelines
Diagnose, resolve, and prevent issues encountered in the field
Collaborate with customer-facing teams to increase reliability of data pipelines
Reduce operational overhead of production workflows and leverage data
Take part in a 24/7 on-call rotation responsible for:
- Coordinating team's response to mission-critical incidents
- Ensuring efficient resolution with minimal customer impact
Read code to identify fixes and increase stability of data pipelines
Work alongside pipeline engineers from other customer accounts
Maintain and build schedules so that pipelines run effectively
Set up and maintaining health checks on different pipelines
Respond to, triage, and debug pipelines when there is a problem
Read and write code changes and/or modify monitoring set-up
Know and understanding how to navigate the pipelines and documentation
Follow SOPs to contact other teams/providers if data is incorrect
Communicate outages with the end users of a pipeline
Contribute to and monitor tooling improvements (where feasible)
Qualifications:
BS in Computer Science / Engineering (or related career field)
3+ years of experience with the following:
- Pipeline Engineering
- Data Engineering
- Parallel data processing and basics of optimizing Spark jobs is a plus.
Comfortable reading and writing code in:
- Python, SQL, Pyspark and Java
Basic understanding of Spark and familiar with tuning Spark jobs
Practical experience with performing root cause analysis and documentation
Experience working in an agile environment
Required: Active DoD Secret Clearance
Job Type: Full-time
Salary: $140,000-$180,000
Job Type: Full-time
Pay: $140,000.00 - $180,000.00 per year
Benefits:
- 401(k)
- Dental insurance
- Health insurance
Compensation Package:
- 1099 contract
- Bonus opportunities
- Performance bonus
- Retention bonus
- Signing bonus
Schedule:
Experience:
- Python coding/programming: 5 years (Preferred)
- data engineering: 5 years (Preferred)
- SQL programming: 5 years (Preferred)
Security clearance:
Work Location: In person
