Create and manage Azure Databricks workspaces, and Unity Catalog setup allocating resources and configuring settings based on specific needs for all environments (Dev/Test/Prod).
Design and implement access control mechanisms, user permissions, and security policies to safeguard sensitive data.
Configure and launch Databricks clusters, optimizing performance and cost through effective instance type selection, auto-scaling, and termination policies.
Monitor cluster health and performance, troubleshoot issues proactively, and adjust configurations as necessary.
Install and configure libraries, dependencies, and environment variables essential for data engineering and analytics tasks.
Implement robust security practices, including network security settings, integration with Azure Active Directory, and data encryption.
Stay informed about security best practices and updates, applying them proactively to maintain a secure environment.
Monitor resource utilization and costs associated with Databricks usage, identifying opportunities for optimization and cost reduction.
Analyze cluster performance metrics and implement tuning strategies to ensure optimal query execution times and resource utilization.
Required Qualifications
Bachelor's degree in Computer Science, Information Technology, or related field.
Must have Unix and Windows background.
Must have strong experience with all Azure cloud components including Storage Accounts, and Azure Databricks in an admin capacity.
knowledge of Azure Active Directory, Snowflake, and Azure Cloud components is a PLUS.
Experience in the administration and performance tuning of Databricks is required for success in this position.