The Data Engineer will design, develop, and maintain data pipelines, focus on SQL and data warehousing, and utilize tools like Azure Databricks and Python for data integration and performance optimization.
KF - Data Engineer B. - Job7353Summary:
We are seeking an experienced and skilled Data Engineer to join our team. As a Data Engineer, your primary responsibility will be to design, develop, and maintain data pipelines and solutions using modern technologies. You should have expertise in SQL with a focus on data warehousing, as well as experience with Azure Databricks, PySpark, Azure Data Factory, and Azure Data Lake. Strong knowledge of data engineering fundamentals and working with Parquet/Delta tables in Azure Data Lake. Proficiency in Python programming and the ability to create data pipelines is necessary.
Responsibilities:- Designing and developing data pipelines to extract, transform, and load data from various sources into the data warehouse leveraging Python & notebooks.
- Writing complex SQL queries for data extraction and manipulation from the data warehouse.
- Building and maintaining ETL processes using Azure Databricks with PySpark.
- Implementing data integration workflows using Azure Data Factory.
- Collaborating with cross functional teams including developers, data analysts, and business stakeholders to understand requirements and deliver high quality solutions.
- Optimizing performance of the data pipelines and ensuring scalability and reliability of the systems.
- Monitoring Data quality and troubleshooting issues in collaboration with the operations team.
- Maintaining documentation of the design and implementation of the data pipelines.
- Ability to collaborate on best practices in code creation while maintaining communication with the team for new business logic transformations.
- Expertise in SQL, ideally with experience in working with data warehousing concepts.
- Strong firsthand experience with Azure Databricks and Spark.
- Proficiency in designing and implementing data integration workflow using Azure Data Factory.
- Demonstrates proficiency in Python programming and the ability to develop scalable data engineering pipelines in Python.
- Solid understanding of data engineering fundamentals including data modeling, data transformation, change data capture and performance optimization techniques.
- Experience working with Azure Data Lake for storing large data sets, maintaining Parquet/Delta tables, and performing efficient querying.
- Experience with version control systems and familiarity with CI/CD practices.
- Strong interpersonal skills, ability to clearly communicate, and voice concerns in a group setting.
- Initiative-taking, self-reliant approach with a willingness to learn business logic and work with critical faults. Candidates should be able to independently understand business requirements without relying on subject matter experts for ongoing explanations.
- Ability to collaborate effectively in planning and refinement sessions.
Top Skills
Azure Data Factory
Azure Data Lake
Azure Databricks
Pyspark
Python
SQL
Similar Jobs
Digital Media • Design
The Brand Operations Manager will oversee the Brand Division's operations, coordinating workflows, managing timelines and budgets, and driving collaboration across departments to ensure efficiency and timely asset delivery.
Top Skills:
AsanaDam PlatformsShotgrid
Professional Services • Consulting • Financial Services
Lead client transactions and relationships, manage teams, develop financing strategies, and drive new business opportunities in capital markets across Africa.
Top Skills:
Financial ModelingInvestor Presentations
Professional Services • Consulting • Financial Services
Leading multiple client transactions, structuring funding vehicles, coaching junior team members, and engaging with investors to grow African markets.
Top Skills:
Commercial Deal DocumentationFinancial Modelling
What you need to know about the Belfast Tech Scene
If asked to name the birthplace of the RMS Titanic, you might not say Belfast. Similarly, if asked to name Europe's leading destination for foreign direct investment in new software development, Belfast might not come to mind. Yet, both are true. The city has emerged as a tech powerhouse, recently ranked among the best in the U.K. for tech careers — especially for software developers. It also leads the U.K. with the highest percentage of software development jobs advertised.
