AWS Databricks Engineer

Overview Join a team where innovation meets mission. Our AI, cloud, cyber, and modernization solutions save agencies thousands of hours, safeguard national security, and strengthen health and humanitarian missions worldwide. With 1,700+ team members, 1,500+ AI/data experts, and 100+ prime contracts, we deliver at scale and with purpose. We’ve been recognized as a Top Workplace by the Washington Post for six straight years and named to the Inc. 5000 Fastest Growing Private Companies 13 of the past 14 years. Credence is a welcoming home for those looking to grow and contribute to positive change. We encourage all employees to expand beyond their boundaries and dive into important, world-changing federal challenges. Position Summary Credence has an immediate need for an AWS Databricks Engineer to join our growing AI and Automation practice. You will serve as a key technical contributor responsible for designing, building, and optimizing scalable cloud-based data solutions using AWS and Databricks. You’ll develop robust data pipelines, implement modern data lakehouse architectures, and collaborate across engineering, data science, and stakeholder teams to deliver secure, high-performance data capabilities that advance federal missions. Responsibilities include, but are not limited to, the duties listed below: - Databricks Engineering Design, build, and optimize scalable data pipelines and processing workflows using Databricks, Apache Spark, PySpark, and Delta Lake. - Data Lakehouse Architecture Design and implement modern lakehouse solutions using Databricks and AWS services. Apply medallion architecture (Bronze/Silver/Gold layers) to improve data quality, governance, lineage, performance, and accessibility. - AWS Data Integration Build and maintain ETL/ELT pipelines integrating structured, semi-structured, and unstructured data across AWS services such as S3, Glue, RDS, Redshift, Lambda, and related cloud data platforms. - Data Transformation & Optimization Develop efficient data transformation workflows using SQL, Python, and PySpark. Optimize Spark workloads, queries, clusters, and storage configurations for performance, reliability, and cost efficiency. - Data Governance & Quality Implement data quality, cataloging, lineage, governance, and access-control practices across Databricks and AWS environments, including the use of Unity Catalog where applicable. - Cloud Automation & DevOps Automate data platform deployments and workflows using Infrastructure as Code (IaC), CI/CD pipelines, Git-based development practices, and AWS-native deployment tools. - Collaborative Engineering Work closely with data engineers, software engineers, data scientists, cloud engineers, and mission stakeholders to translate business and technical requirements into production-ready data solutions. - Technical Rigor & Documentation Develop clean, maintainable, and well-documented code while following engineering, security, and federal development standards. Support code reviews, testing, troubleshooting, and technical design documentation. Requirements - U.S. citizenship with the ability to obtain and maintain a Secret clearance is required. - Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, Engineering, or a related technical field. - Hands-on experience in data engineering, cloud engineering, or related technical roles. - Hands-on experience building and maintaining production data solutions using Databricks, Apache Spark, and PySpark. - Strong experience with AWS data services such as S3, Glue, RDS, Redshift, Lambda, IAM, and CloudWatch. - Strong understanding of ETL/ELT processes, data lake and lakehouse architectures, data modeling, and large-scale data processing. - Hands-on experience with Delta Lake, SQL, Python, and Spark-based data processing. - Experience developing, scheduling, monitoring, and troubleshooting production data pipelines and workflows. - Experience with Git and CI/CD platforms such as GitLab, GitHub Actions, Jenkins, or AWS CodePipeline. - Understanding of cloud security, identity and access management, encryption, and secure data-handling practices. - Strong communication skills and a client-oriented mindset with the ability to collaborate across technical and non-technical teams. Preferred - Experience implementing Databricks Lakehouse Architecture and medallion architecture using Bronze, Silver, and Gold data layers. - Experience with Databricks Unity Catalog, Delta Live Tables, Databricks Workflows, or similar data orchestration and governance capabilities. - Strong Python proficiency and experience with data engineering libraries and frameworks such as PySpark, Pandas, and Pydantic. - Experience with Infrastructure as Code tools such as Terraform, OpenTofu, AWS CDK, or CloudFormation. - AWS and/or Databricks certifications, such as AWS Certified Data Engineer - Associate, AWS Certified Solutions Architect, or Databricks Certified Data Engineer credentials. - Knowledge of federal cybersecurity, RMF, FedRAMP, or other government regulatory and compliance frameworks. - Experience delivering cloud-native data solutions within federal government or other highly regulated environments. Salary Range: $155,000 - $185,000 annually. Actual compensation will be determined based on the selected candidate's experience, education, certifications, skills, and overall qualifications. Benefits - Health Care Plan (Medical, Dental & Vision) - Retirement Plan (401k, IRA) - Life Insurance (Basic, Voluntary & AD&D) - Paid Time Off (Vacation, Sick & Public Holidays) - Family Leave (Maternity, Paternity) - Short Term & Long Term Disability - Training & Development - Wellness Resources