Job Overview
AI Harness Cloud Automation Engineer - Fulltime, Fort Worth, TX
Responsibilities
- Develops and manages automated pipelines (using tools like Azure DevOps, GitHub Actions, Databricks Jobs and Workflows, etc.) for seamless application code deployments and infrastructure provisioning, ensuring repeatability, compliance, and industry best practices across all DTAP environments, including integration with Azure Kubernetes Service (AKS), containerisation, and event‑driven architectures.
- Implements automation strategies and best practices for environment segregation, access control, and secure deployment to Development, Test, Acceptance and Production stages, including Databricks workspace configuration, access management, and Azure service registration.
- Continuously monitors, traces, and assesses the health and performance of deployed infrastructure using Azure Monitor, Application Insights, Databricks monitoring tools and other observability solutions, proactively identifying and resolving issues with attention to PostgreSQL/SQL databases and supporting event/topic subscriptions.
- Supports and automates deployment pipelines for AI, data engineering and Databricks code, ensuring scalable and reliable delivery of advanced analytics, machine learning workloads and integration with cloud-based AI/ML models.
- Integrates automated compliance and security checks into deployment pipelines, including static/dynamic code analysis, vulnerability scanning, secrets management, Databricks security validations and adherence to regulatory and organisational requirements.
- Works collaboratively with developers, data engineers and product teams to reinforce agile methodologies, ensure code quality and streamline release cycles, including coordination around Databricks workflow integration and collaborative development practices.
- Documents architecture, pipelines, operational procedures and incident response protocols—including Databricks workflows and operational best practices—enabling transparency, knowledge transfer and continuous improvement across teams.
- Applies a working knowledge of agile principles, continuous delivery and modern DevOps processes (including Infrastructure as Code tools such as Terraform, Ansible and Azure Resource Manager), adapting practices to enhance team productivity and project outcomes, with emphasis on Databricks integration and automation.
- Leverages AI tools, coding assistants, prompt engineering and Databricks capabilities to optimise productivity, accelerate development, automate repetitive tasks and support cloud-based data engineering and AI/ML workloads.
- Remains updated on the latest Azure, Databricks, automation, security and AI technologies, contributing to continuous process improvement and evolving best practices in cloud engineering and DevOps.
Qualifications
- 4+ years of infrastructure experience in Azure Containers, Databricks, ADO, event hubs/topics, Kafka and Logic Apps.
- Bachelor’s degree in Computer Science, Engineering, Information Technology or a related technical field.
- Experience using AI for building pipelines or managing infrastructure.
- Demonstrated proficiency with Azure Kubernetes Service (AKS), containers, Kubernetes orchestration, relational databases (PostgreSQL/SQL), Azure service registration and event-driven architecture.
- Advanced skills in pipeline automation, Infrastructure as Code (IaC) using tools like Terraform, Ansible, Azure Resource Manager and CI/CD implementation with Azure DevOps, GitHub Actions or similar platforms.
- Hands‑on experience designing and enforcing automation and environment segregation for DTAP (Development, Test, Acceptance, Production) environments, adhering to industry best practices for compliance and security.
- Familiarity with monitoring and observability solutions, such as Azure Monitor, Application Insights, Databricks monitoring tools and other cloud‑native observability frameworks.
- Working knowledge of integrating security and compliance validations—such as vulnerability scanning, static/dynamic code analysis and secrets management—into automated deployment workflows.
- Solid understanding of agile methodologies, DevOps principles, continuous improvement and collaborative development practices in a cloud environment.
- Excellent organisational and prioritisation skills, with the ability to balance multiple issues and projects in a fast-paced environment.
- Excellent verbal, written and interpersonal communication skills.
- Must be self‑motivated with strong initiative, high level of accountability and attention to detail.
- Must be a self‑starter with the ability to work independently and as part of a team.
- Ability to always maintain a high degree of ethical standards and complete confidentiality.
Preferred Skills
- Expertise in Databricks engineering, including advanced pipeline orchestration, comprehensive observability practices and proficiency with MLFlow and related toolsets.
- Experience supporting AI/ML models, AI gateways and AI governance in cloud environments.
- Ability to use AI-powered tools and prompt engineering to automate tasks and enhance team efficiency.
- Relevant certifications (e.g., Azure Certified Solutions Architect, Kubernetes, Databricks, security credentials).
#J-18808-Ljbffr