Data Engineer - Databricks

Affine Inc • Wilmington, DE, US • 1w ago

Duties:

Design and build data pipelines using Spark-SQL and PySpark in Azure Databricks
Design and build ETL pipelines using ADF
Build and maintain a Lakehouse architecture in ADLS / Databricks.
Perform data preparation tasks including data cleaning, normalization, deduplication, type conversion etc.
Work with DevOps team to deploy solutions in production environments.
Control data processes and take corrective action when errors are identified. Corrective action may include executing a work around process and then identifying the cause and solution for data errors.
Participate as a full member of the global Analytics team, providing solutions for and insights into data related items.
Collaborate with your Data Science and Business Intelligence colleagues across the world to share key learnings, leverage ideas and solutions and to propagate best practices.
You will lead projects that include other team members and participate in projects led by other team members.
Apply change management tools including training, communication and documentation to manage upgrades, changes and data migrations.

Must Have Skills:

Azure Databricks
Azure Data Factory
PySpark
Spark - SQL
ADLS/Azure Data Lake Storage

Job Type: Full-time

Benefits:

Health insurance
Paid time off
Referral program

Compensation Package:

Bonus opportunities
Performance bonus
Yearly pay

Schedule:

8 hour shift
Day shift
Monday to Friday

Experience:

Azure Data Lake: 3 years (Required)
Azure Databricks: 1 year (Required)

Ability to Commute:

Wilmington, DE 19805 (Required)

Ability to Relocate:

Wilmington, DE 19805: Relocate with an employer provided relocation package (Required)

Work Location: In person

Related Jobs

Lead Data Engineer

Capital One • Wilmington, DE, US • 2d ago

Data Science

2d ago

Senior Data Engineer - BankTech

Capital One • Wilmington, DE, US • 3w ago

Data Science

3w ago

Regional Solar Site Mgr & ES III- ROI DG- NY/NJ/PA

NextEra • Full-time • Marcus Hook, Pennsylvania • 2m ago

Data Science

2m ago

Operations Manager

Confidential Company • Full-time • West Chester, PA, US • $80k - $150k / year • 3w ago

Product Management

3w ago