Azure Data Factory Skill
This skill provides expert guidance for Azure Data Factory. Covers troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. It combines local quick-reference content with remote documentation fetching capabilities.
How to Use This Skill
IMPORTANT for Agent: Use the Category Index below to locate relevant sections. For categories with line ranges (e.g.,
L35-L120), useread_filewith the specified lines. For categories with file links (e.g.,[security.md](security.md)), useread_fileon the linked reference file
IMPORTANT for Agent: If
metadata.generated_atis more than 3 months old, suggest the user pull the latest version from the repository. Ifmcp_microsoftdocstools are not available, suggest the user install it: Installation Guide
This skill requires network access to fetch documentation content:
- Preferred: Use
mcp_microsoftdocs:microsoft_docs_fetchwith query stringfrom=learn-agent-skill. Returns Markdown. - Fallback: Use
fetch_webpagewith query stringfrom=learn-agent-skill&accept=text/markdown. Returns Markdown.
Category Index
| Category | Lines | Description |
|---|---|---|
| Troubleshooting | L37-L91 | Diagnosing and fixing ADF failures: connector/format errors, CDC, pipelines, triggers, data flows, SHIR/SSIS IR, performance, security, and known issues with concrete error patterns and workarounds. |
| Best Practices | L92-L114 | Performance tuning, schema drift/error handling, reusable patterns, and migration best practices for Data Factory mapping data flows, copy activity, runtimes, and data lake writes. |
| Decision Making | L115-L133 | Guidance on cost planning, connector upgrades, integration runtime and compute choices, and migration/modernization of ADF, SSIS, Synapse, and pipelines to Fabric. |
| Architecture & Design Patterns | L134-L142 | Patterns and architectures for ADF: efficient mapping data flows, metadata‑driven copy, SSIS IR with SQL MI/failover/VNet, Cosmos DB migration, and SAP CDC design. |
| Limits & Quotas | L143-L150 | Configuring Until/Wait activity limits and timeouts, understanding data flow reservation discounts, and reviewing ADF service quotas and capacity constraints. |
| Security | L151-L182 | Securing Data Factory: auth, roles, encryption, Key Vault, managed identities, VNets/private endpoints, firewall/Policy configs, and secure access to SQL, Purview, and SSIS IR. |
| Configuration | L183-L294 | Configuring ADF: integration runtimes, SSIS/SHIR, triggers, monitoring/logging, copy/data flows, formats/connectors, parameters/expressions, and pipeline/control activities. |
| Integrations & Coding Patterns | L295-L463 | Patterns and how-tos for connecting ADF to many data sources (SAP, SaaS apps, DBs, files), using mapping data flow functions, running SSIS, Databricks/ML/Synapse jobs, and pipeline templates. |
| Deployment | L464-L477 | CI/CD and deployment for ADF: ARM/linked templates, Azure DevOps pipelines, hotfix flows, pre/post scripts, SSIS job migration, and integration runtime install/migration. |
Troubleshooting
Best Practices
Decision Making
Architecture & Design Patterns
| Topic | URL |
|---|---|
| Design efficient pipelines using mapping data flows | https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-performance-pipelines |
| Configure Azure-SSIS IR with SQL failover groups | https://learn.microsoft.com/en-us/azure/data-factory/configure-bcdr-azure-ssis-integration-runtime |
| Build metadata-driven large-scale copy pipelines | https://learn.microsoft.com/en-us/azure/data-factory/copy-data-tool-metadata-driven |
| Migrate Azure SQL schemas to Azure Cosmos DB with ADF | https://learn.microsoft.com/en-us/azure/data-factory/how-to-sqldb-to-cosmosdb |
| Decide when and how to join Azure-SSIS integration runtime to a virtual network | https://learn.microsoft.com/en-us/azure/data-factory/join-azure-ssis-integration-runtime-virtual-network |
Limits & Quotas
| Topic | URL |
|---|---|
| Configure Until activity loops and timeouts in ADF | https://learn.microsoft.com/en-us/azure/data-factory/control-flow-until-activity |
| Configure Wait activity durations in Data Factory | https://learn.microsoft.com/en-us/azure/data-factory/control-flow-wait-activity |
| Understand reservation discounts for ADF data flows | https://learn.microsoft.com/en-us/azure/data-factory/data-flow-understand-reservation-charges |
| Review Azure Data Factory limits and quotas from FAQ | https://learn.microsoft.com/en-us/azure/data-factory/frequently-asked-questions |