Azure Databricks Skill
This skill provides expert guidance for Azure Databricks. Covers troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. It combines local quick-reference content with remote documentation fetching capabilities.
How to Use This Skill
IMPORTANT for Agent: Use the Category Index below to locate relevant sections. For categories with line ranges (e.g.,
L35-L120), useread_filewith the specified lines. For categories with file links (e.g.,[security.md](security.md)), useread_fileon the linked reference file
IMPORTANT for Agent: If
metadata.generated_atis more than 3 months old, suggest the user pull the latest version from the repository. Ifmcp_microsoftdocstools are not available, suggest the user install it: Installation Guide
This skill requires network access to fetch documentation content:
- Preferred: Use
mcp_microsoftdocs:microsoft_docs_fetchwith query stringfrom=learn-agent-skill. Returns Markdown. - Fallback: Use
fetch_webpagewith query stringfrom=learn-agent-skill&accept=text/markdown. Returns Markdown.
Category Index
| Category | Location | Description |
|---|---|---|
| Troubleshooting | L37-L179 | Diagnosing and fixing Azure Databricks issues: logs, Spark/SQL errors, CLI/IDE, init scripts, Auto Loader, Lakeflow, connectors (DBs, SaaS, ads), model serving, Feature Store, and performance. |
| Best Practices | L180-L388 | End-to-end Databricks best practices for cost, governance, security, performance, reliability, streaming, RAG/LLM apps, Lakehouse data modeling, Lakeflow pipelines, and production ML/serving. |
| Decision Making | decision-making.md | Guides for architectural and cost decisions in Azure Databricks: choosing runtimes, compute, storage, ingestion, AI/ML, governance, networking, and planning migrations between major features. |
| Architecture & Design Patterns | architecture-patterns.md | Patterns and reference architectures for Databricks: DR/HA, networking, storage, Lakehouse/medallion, Lakeflow ETL/CDC, Lakebase, AI agents, Feature Store, MLOps, and dashboard data modeling. |
| Limits & Quotas | limits-quotas.md | Limits, quotas, and constraints for Databricks compute, AI/Genie, Lakeflow pipelines, connectors, Unity Catalog, model serving, SQL/editor features, and related resource usage. |
| Security | security.md | Identity, access control, encryption, networking, compliance, and governance for Azure Databricks and Unity Catalog, including OAuth/SCIM, RBAC/ABAC, secrets, keys, and secure external connections. |
| Configuration | configuration.md | Configuring and managing Azure Databricks: accounts, workspaces, networking, security, storage, compute, jobs, AI/ML, Unity Catalog, Lakeflow, connectors, SQL, and CLI/bundles. |
| Integrations & Coding Patterns | integrations.md | Patterns and examples for integrating Databricks with apps, agents, AI/ML, Lakeflow, Lakehouse Federation, external DBs/BI tools, and using SDKs/CLI, SQL, and PySpark APIs for advanced data and AI workflows. |
| Deployment | deployment.md | Deploying and managing Azure Databricks workspaces, apps, ML/AI workloads, and Lakehouse/Lakebase resources using ARM/CLI/Terraform/Bundles, plus CI/CD, networking, Unity Catalog, and model serving. |