Definitive Healthcare Logo

Definitive Healthcare

Senior Big Data Engineer

Posted Yesterday
Remote
Hybrid
Hiring Remotely in United States
Senior level
Remote
Hybrid
Hiring Remotely in United States
Senior level
The Senior Big Data Engineer will design and develop scalable data pipelines, manage data integration, performance tuning, and collaborate with teams to enhance data engineering practices.
The summary above was generated by AI

At Definitive Healthcare, our passion is to transform data, analytics and expertise into healthcare commercial intelligence. We help clients uncover the right markets, opportunities and people, so they can shape tomorrow's healthcare industry. Our SaaS platform creates new paths to commercial success in the healthcare market, so companies can identify where to go next.
Our employees are kind, collaborative, energetic, approachable and driven. On top of that, we value the unique perspectives, backgrounds and voices of our employees. Why? Because their diverse experiences drive new ideas and help us build a better community.
For over 10 years, we've built a collaborative culture driven by employees who share a passion for improving the healthcare ecosystem, enjoy giving back to the local community and value diversity and inclusion.
One of the hallmarks of our culture is our commitment to community service. Through the DefinitiveCares program, employees can work with their choice of more than 40 charitable organizations, supporting causes from hunger and homelessness to healthcare, LGBTQ+ issues, racial justice, women's initiatives and more. 2021 marked the sixth year that we had 100% employee participation in DefinitiveCares.
We also provide a range of opportunities for employees to connect with each other. Employees can join any of our employee run affinity groups supporting causes such as women's empowerment, LGBTQ+, Black, indigenous and people of color (BIPOC), disabilities and working parents and potential for many more. Affinity groups often enable greater education companywide through training, events and speaker series.
We're also a great place to work. For five years in a row, we've been recognized by the Boston Business Journal and the Boston Globe as a best place to work in Massachusetts. In 2022, Energage recognized us for Culture Excellence in Compensation & Benefits, Innovation, Great Leadership, Purpose & Value and Work-Life Flexibility!
Think you'd be a good addition to our team? Explore our available positions here. We'd love the chance to get to know you.
Responsibilities:

  • Design and Develop Data Pipelines:
    • Build and maintain scalable data pipelines using Python, Spark, and Databricks.
    • Implement data workflows and ETL processes using Apache Airflow.
  • Data Integration and Management:
    • Integrate data from various sources (AWS, GCP, on-premises) into a unified data warehouse.
    • Handle variety of data formats such as csv, text, xml, parquet, delta etc.,
    • Ensure data quality and integrity through effective data cleansing and curation practices.
    • Manage and optimize data storage solutions, ensuring high availability and performance.
    • Automate observability of data and workloads
  • Metadata Management and Governance:
    • Implement and manage Unity Catalog for metadata management.
    • Ensure data governance policies are followed, including data security, privacy, and compliance.
    • Develop and maintain data documentation and data dictionaries.
    • Automate data observability across pipelines
  • Performance Tuning and Troubleshooting:
    • Optimize Spark jobs for performance and efficiency.
    • Investigate and resolve performance bottlenecks in Spark applications.
    • Utilize JVM tuning techniques to improve application performance.
  • Data Maturity Lifecycle:
    • Implement and manage the Medallion architecture for data maturity lifecycle.
    • Ensure data is appropriately processed and categorized at different stages (bronze, silver, gold) to maximize its usability and value.
  • Collaboration and Continuous Improvement:
    • Work closely with data scientists, analysts, and other stakeholders to understand data needs and deliver solutions.
    • Implement CI/CD pipelines to automate deployment and testing of data infrastructure.
    • Stay up to date with the latest industry trends and technologies to continuously improve data engineering practices.


Required Skills and Qualifications:

  • Technical Skills:
    • Hands-on Python or Scala programming.
    • Strong experience with Apache Spark and Databricks.
    • Hands-on experience with Apache Airflow or similar workflow orchestration tools.
    • Data modeling and processing fundamentals with large-scale volume of data
    • Knowledge of data cleansing and curation techniques.
    • Familiarity with Unity Catalog or other metadata management tools.
    • Understanding of data governance principles and best practices.
    • Experience with cloud platforms (AWS and GCP).
    • Strong understanding of normalization and denormalization.
    • Proficiency in CI/CD tools and practices (e.g., Jenkins, GitLab CI, etc.).
    • Experience with JVM tuning and Spark job performance investigation.
    • Experience with Medallion architecture for data maturity lifecycle.
    • Familiarity with containerization
  • Soft Skills:
    • Excellent problem-solving and analytical skills.
    • Strong communication and collaboration skills.
    • Ability to work independently and as part of a team.
    • Detail-oriented with a focus on delivering high-quality work.


Preferred Qualifications:

  • Certification in cloud platforms (AWS Certified Data Analytics, Google Cloud Professional Data Engineer, etc.).
  • Familiarity with SQL and NoSQL databases.
  • Experience in a similar role within a fast-paced, data-driven environment.


Why we love Definitive, and why you will too!

  • Industry leading products
  • Work hard, and have fun doing it
  • Incredibly fast growth means limitless opportunity
  • Flexible and dynamic culture
  • Work alongside some of the most talented and dedicated teammates
  • Definitive Cares, our community service group, gives all of us a chance to give back
  • Competitive benefits package including great healthcare benefits and a 401(k) match


What our Employees are saying about us on Glassdoor:
"Great Work atmosphere, great work life balance, excellent company to work for, amazing top notch product, incredible customer service, lots of tools to help you succeed."
-Business Development Manager
"Great team. Amazing growth. Employees are treated very well."
-Research Analyst
"I have waited 36 years to work at a dream job for a dream company and I am so happy to have finally got there."
-Profile Analyst
If you don't fit all of these qualifications, but believe you're still a great fit, feel free to apply and tell us why in your cover letter.
If you are a California, Colorado, New York City or Washington resident and this role is a remote role, you can receive additional information about the compensation and benefits for this role, which we will provide upon request.
Definitive Hiring Philosophy
Definitive Healthcare is an equal opportunity employer that celebrates diversity and is committed to creating an inclusive workplace with equal opportunity for all applicants and teammates. Our goal is to recruit the most talented people from a diverse candidate pool regardless of race, color, religion, age, gender, gender identity, sexual orientation or any other status. If you're interested in working in a fast growing, exciting working environment - we encourage you to apply!
Privacy
Your privacy is important to us. Please review our Candidate Privacy Notice which tells you how we use and process your personal information
Please note : All communications regarding the hiring process at Definitive Healthcare will come directly from one of our corporate recruiters or coordinators with an @definitivehc.com email address. We will never request any money transfer or purchase of equipment with a promise of reimbursement. If you receive any suspicious communications, please reach out to [email protected] to confirm your status in the application process.

Top Skills

Apache Airflow
AWS
Ci/Cd
Databricks
GCP
Gitlab Ci
Jenkins
Python
Spark
Unity Catalog

Similar Jobs at Definitive Healthcare

Yesterday
Remote
Hybrid
Framingham, MA, USA
Mid level
Mid level
Healthtech • Software
The Data Engineer will build and maintain scalable data pipelines, integrate data sources, manage metadata, optimize performance, and collaborate with stakeholders.
Top Skills: Apache AirflowAWSCi/CdDatabricksPythonSparkSQLSsis
6 Hours Ago
Remote
Hybrid
Framingham, MA, USA
Senior level
Senior level
Healthtech • Software
The Associate Principal will bridge client needs with analytics, creating specifications, managing communication, and developing data-driven strategies for Pharmas using medical claims data.
Top Skills: MS OfficePower BIPythonRScalaSQLTableau
6 Hours Ago
Remote
Hybrid
Framingham, MA, USA
Mid level
Mid level
Healthtech • Software
As an Account Executive, you will manage and grow client accounts, engage with decision makers, and drive growth opportunities while exceeding quotas.
Top Skills: SaaS

What you need to know about the Belfast Tech Scene

If asked to name the birthplace of the RMS Titanic, you might not say Belfast. Similarly, if asked to name Europe's leading destination for foreign direct investment in new software development, Belfast might not come to mind. Yet, both are true. The city has emerged as a tech powerhouse, recently ranked among the best in the U.K. for tech careers — especially for software developers. It also leads the U.K. with the highest percentage of software development jobs advertised.

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account