Azure Databricks DevOps Administrator New York City, New York 

Azure Databricks DevOps Administrator New York City, New York 
> 
> Description: 
> 
> Job Location: New York City, New York - Hybrid (3 days onsite) 
> 
> Job Title: Azure Databricks DevOps Administrator 
> 
> Duration: Long term Contract / CTH 
> 
> Target Start Date: ASAP 
> 
> Experience 
> 
> As an Azure Databricks DevOps Administrator, you will be responsible
> for managing and maintaining the Databricks platform, ensuring optimal
> performance and security across workspaces, clusters, and jobs. This
> role is crucial in overseeing user administration, managing Azure
> storage solutions, and implementing CI/CD pipelines using Azure
> DevOps. By leveraging a deep understanding of Databricks architecture
> and proficiency in scripting languages, the Azure DevOps administrator
> will automate tasks and enhance the efficiency of data operations.
> Strong communication skills and a commitment to high-quality support
> will enable effective collaboration with cross-functional teams,
> directly contributing to the department's mission of delivering robust
> data solutions and insights. 
> 
> Job Responsibilities 
> 
>      * Hands-on experience in Azure Cloud services, Networking Concepts,
> Security - Cloud and on-premises system, Deployment using Azure
> DevOps, Azure Cloud Monitoring and Cost controls and Terraform -
> hands-on experience
>      * CI/CD Pipeline Management: Design, implement, and manage
> Continuous Integration/Continuous Deployment (CI/CD) pipelines using
> Azure DevOps and GitHub. Ensure the pipelines are efficient, reliable,
> and scalable.
>      * Infrastructure as Code (IaC): Automate the provisioning and
> management of infrastructure, with a focus on Azure Databricks and
> Azure Data Factory in a private network environment, using tools like
> Terraform and ARM templates.
>      * Environment Management: Create and manage development, testing,
> and production environments, ensuring consistency, security, and
> alignment with organizational requirements.
>      * Security: Implement security best practices throughout the CI/CD
> pipeline, including secrets management, secure code scanning, and
> compliance with security standards.
>      * Monitoring & Logging: Set up and maintain monitoring and logging
> for applications and infrastructure using Azure Monitor, Log
> Analytics, and related tools to ensure system reliability and
> performance.
>      * Automation: Identify opportunities for automation to streamline
> processes, reduce manual errors, and improve operational efficiency.
>      * Policy Enforcement: Establish and enforce policies such as branch
> policies, pull request reviews, and pipeline approvals to maintain
> code quality and compliance with organizational standards.
>      * Manage and maintain Azure Databricks Platform, workspaces,
> clusters, and jobs
>      * Oversee user administration including access controls and
> permissions
>      * Handle library installations, runtime management, and policy
> enforcement
>      * Implement/analyze cost control measures
>      * Administer Unity Catalog for data governance and security
>      * Collaborate with data engineers, data scientists, and analysts to
> optimize and streamline data workflows, and analytical pipelines on
> the Databricks platform
>      * Manage Azure storage solutions, including Blob Storage and Data
> Lake Storage
>      * Administer Azure Key Vault for secure storage of secrets and keys
>      * Configure and manage Azure Data Factory for data integration and
> ETL processes
>      * Implement and manage VNETs, firewalls, Azure policies and security
> best practices
>      * Set up budgets and alerts to monitor and control Azure costs and
> spend
>      * Configure alerts for proactive issue detection and resolution
>      * Databricks Lakehouse Monitoring
> 
> Minimum Qualifications 
> 
>      * Minimum of Bachelor's degree in Computer science, Information
> systems, or Engineering
>      * Experience: 7+ years of professional experience in DevOps, with a
> strong focus on Azure. Knowledge of Data Platforms like Databricks,
> Data Factory (preferred)
>      * Proficiency in scripting languages such as Python/PySpark,
> PowerShell, or Bash
>      * Experience in automating administrative tasks and workflows
>      * Knowledge of security best practices and compliance requirements
>      * Experience with ETL processes, data pipelines, and big data
> technologies
>      * Experience with backup and restore procedures for Databricks and
> Azure services
>      * Ability to troubleshoot and resolve issues in a timely and
> efficient manner
>      * Strong verbal and written communication skills
>      * Ability to document processes, procedures, and configurations
> clearly
>      * Commitment to providing high-quality support to internal/external
> users and stakeholders
>      * Ability to understand and address the needs of internal and
> external customers
>      * Team player with the ability to work collaboratively with
> cross-functional teams
>      * Flexibility to adapt to changing requirements and priorities
>      * Continuous Learning: Eagerness to learn new technologies and
> continuously improve skills to stay current in a rapidly evolving
> field.