Skip to content
azure-databricks logo

Azure Databricks Skill

azure-databricks

Expert knowledge for Azure Databricks development including troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. Use when working with Unity Catalog, Lakeflow, Lakebase, Delta Sh...

MicrosoftDocs/Agent-Skills0installs772stars

SKILL.md

Full skill instructions

Azure Databricks Skill

This skill provides expert guidance for Azure Databricks. Covers troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. It combines local quick-reference content with remote documentation fetching capabilities.

How to Use This Skill

IMPORTANT for Agent: Use the Category Index below to locate relevant sections. For categories with line ranges (e.g., L35-L120), use read_file with the specified lines. For categories with file links (e.g., [security.md](security.md)), use read_file on the linked reference file

IMPORTANT for Agent: If metadata.generated_at is more than 3 months old, suggest the user pull the latest version from the repository. If mcp_microsoftdocs tools are not available, suggest the user install it: Installation Guide

This skill requires network access to fetch documentation content:

  • Preferred: Use mcp_microsoftdocs:microsoft_docs_fetch with query string from=learn-agent-skill. Returns Markdown.
  • Fallback: Use fetch_webpage with query string from=learn-agent-skill&accept=text/​markdown. Returns Markdown.

Category Index

CategoryLocationDescription
TroubleshootingL37-L148Diagnosing and fixing Databricks errors and failures across compute, Spark/​SQL, connectors/​Lakeflow, ML/​AI, and tooling (CLI, VS Code, Git), including specific error codes and performance issues.
Best PracticesL149-L307End-to-end Databricks best practices for performance, cost, reliability, governance, streaming, ML/​GenAI, BI, Lakeflow, and operations (compute, storage, security, monitoring, and CI/​CD).
Decision MakingL308-L399Guides for architectural and migration decisions: choosing compute, runtimes, Unity Catalog, Lakebase, ML/​AI options, and planning upgrades, costs, and data/​app/​ETL migrations on Azure Databricks.
Architecture & Design PatternsL400-L439Architectural blueprints and design patterns for Databricks: DR/​HA, networking, storage, governance, medallion/​RAG/​AI agents, streaming, Lakebase, and model/​MLOps deployment.
Limits & Quotaslimits-quotas.mdLimits, quotas, and constraints for Databricks compute, AI/​BI, model serving, connectors, Lakeflow, Lakebase, SQL types, governance (Unity Catalog, ABAC), and Free Edition usage.
Securitysecurity.mdIdentity, access control, encryption, networking, compliance, and governance for Azure Databricks and Unity Catalog, including tokens, SCIM, OAuth, ABAC, Lakeflow, Lakebase, and Delta Sharing.
Configurationconfiguration.mdConfiguring and governing Azure Databricks: accounts, workspaces, networking, compute, Unity Catalog, pipelines, ML/​AI/​GenAI, connectors, SQL/​session settings, and cost/​usage monitoring.
Integrations & Coding Patternsintegrations.mdPatterns and how-tos for integrating Databricks with external systems, tools, and agents: connectors, federation, streaming, AI/​ML/​LLM APIs, Lakeflow, Lakebase, MLflow, and SDK/​CLI automation.
Deploymentdeployment.mdDeploying and operating Azure Databricks: workspace setup, CI/​CD, apps and AI agents, model serving, Lakeflow ingestion, Unity Catalog migration, and serverless/​compute configuration.

Troubleshooting

TopicURL
Troubleshoot Azure Databricks compute startup issueshttps://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/
Resolve Databricks classic compute termination error codeshttps://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/cluster-error-codes
Debug Spark applications using Databricks Spark UIhttps://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/debugging-spark-ui
Troubleshoot Apache Kafka streaming on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/connect/streaming/kafka/faq
Troubleshoot common Delta Sharing access errorshttps://learn.microsoft.com/en-us/azure/databricks/delta-sharing/troubleshooting
Troubleshoot common Databricks CLI issueshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/troubleshooting
Diagnose and fix Databricks Connect Python issueshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/python/troubleshooting
Diagnose and fix Databricks Connect Scala issueshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/scala/troubleshooting
Troubleshoot common Databricks Terraform provider errorshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/terraform/troubleshoot
Resolve common issues with Databricks VS Code extensionhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/vscode-ext/faqs
Troubleshoot Databricks VS Code extension errorshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/vscode-ext/troubleshooting
Resolve ARITHMETIC_OVERFLOW errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/arithmetic-overflow-error-class
Handle CAST_INVALID_INPUT errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/cast-invalid-input-error-class
Diagnose DC_GA4_RAW_DATA_ERROR in GA4 connectorhttps://learn.microsoft.com/en-us/azure/databricks/error-messages/dc-ga4-raw-data-error-error-class
Understand DC_SFDC_API_ERROR in Databricks connectorshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/dc-sfdc-api-error-error-class
Diagnose DC_SQLSERVER_ERROR in SQL Server connectorhttps://learn.microsoft.com/en-us/azure/databricks/error-messages/dc-sqlserver-error-error-class
Understand DELTA_ICEBERG_COMPAT_V1_VIOLATION errorshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/delta-iceberg-compat-v1-violation-error-class
Resolve DIVIDE_BY_ZERO error in Azure Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/error-messages/divide-by-zero-error-class
Interpret and handle Azure Databricks error conditionshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/error-classes
Fix EWKB_PARSE_ERROR geometry parsing issueshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/ewkb-parse-error-error-class
Fix EWKT_PARSE_ERROR geometry parsing issueshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/ewkt-parse-error-error-class
Resolve GEOJSON_PARSE_ERROR in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/geojson-parse-error-error-class
Address GROUP_BY_AGGREGATE errors in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/error-messages/group-by-aggregate-error-class
Handle H3_INVALID_CELL_ID errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-invalid-cell-id-error-class
Interpret and resolve H3_INVALID_GRID_DISTANCE_VALUE in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-invalid-grid-distance-value-error-class
Handle H3_INVALID_RESOLUTION_VALUE errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-invalid-resolution-value-error-class
Resolve H3_NOT_ENABLED errors and tier requirementshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-not-enabled-error-class
Fix INSUFFICIENT_TABLE_PROPERTY errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/insufficient-table-property-error-class
Troubleshoot INVALID_ARRAY_INDEX errors in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/error-messages/invalid-array-index-error-class
Troubleshoot INVALID_ARRAY_INDEX_IN_ELEMENT_AT in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/invalid-array-index-in-element-at-error-class
Resolve MISSING_AGGREGATION errors in Databricks querieshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/missing-aggregation-error-class
Diagnose ROW_COLUMN_ACCESS errors for filters and maskshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/row-column-access-error-class
Interpret Azure Databricks SQLSTATE error codeshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/sqlstates
Fix TABLE_OR_VIEW_NOT_FOUND errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/table-or-view-not-found-error-class
Resolve UNRESOLVED_ROUTINE function resolution errorshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/unresolved-routine-error-class
Understand UNSUPPORTED_TABLE_OPERATION errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/unsupported-table-operation-error-class
Understand UNSUPPORTED_VIEW_OPERATION errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/unsupported-view-operation-error-class
Troubleshoot WKB_PARSE_ERROR for geometry parsinghttps://learn.microsoft.com/en-us/azure/databricks/error-messages/wkb-parse-error-error-class
Troubleshoot WKT_PARSE_ERROR for geometry parsinghttps://learn.microsoft.com/en-us/azure/databricks/error-messages/wkt-parse-error-error-class
Troubleshoot MLflow 2 Agent Evaluation issueshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-evaluation/troubleshooting
Debug custom AI code agents on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/debug-agent
Troubleshoot common Genie Space issueshttps://learn.microsoft.com/en-us/azure/databricks/genie/troubleshooting
Resolve common Confluence connector ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/confluence-faq
Troubleshoot authentication and rate limit errors for Confluencehttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/confluence-troubleshoot
Troubleshoot Dynamics 365 Lakeflow ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/d365-troubleshoot
Resolve common issues with Databricks managed connectorshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/faq
Troubleshoot Google Ads connector ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/google-ads-troubleshoot
Troubleshoot Google Analytics raw data ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/google-analytics-troubleshoot
Resolve common Databricks Google Drive connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/google-drive-faq
Troubleshoot Databricks Google Drive ingestion failureshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/google-drive-troubleshoot
Troubleshoot Databricks HubSpot connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/hubspot-troubleshoot
Resolve common Azure Databricks Jira connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/jira-faq
Troubleshoot Jira Lakeflow ingestion errorshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/jira-troubleshoot
Troubleshoot Meta Ads ingestion connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/meta-ads-troubleshoot
Diagnose and fix MySQL Lakeflow Connect ingestionhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/mysql-troubleshoot
Troubleshoot common Outlook connector ingestion errorshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/outlook-troubleshoot
Pendo connector FAQs for Databricks ingestionhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/pendo-faq
Troubleshoot Databricks Pendo connector errors and failureshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/pendo-troubleshoot
Troubleshoot PostgreSQL Lakeflow Connect ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/postgresql-troubleshoot
Troubleshoot query-based connector cursor and errorshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/query-based-troubleshoot
Troubleshoot Databricks RabbitMQ ingestion errorshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/rabbitmq-troubleshoot
Troubleshoot Databricks Salesforce ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/salesforce-troubleshoot
Diagnose and fix Databricks ServiceNow connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/servicenow-troubleshoot
Troubleshoot Microsoft SharePoint connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sharepoint-troubleshoot
Troubleshoot Databricks Slack logs connector errorshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/slack-access-integration-logs-troubleshoot
Troubleshoot Databricks Smartsheet connector errorshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/smartsheet-troubleshoot
Answer common SQL Server Lakeflow Connect connector questionshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sql-server-faq
Resolve SQL Server Lakeflow Connect ingestion problemshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sql-server-troubleshoot
Troubleshoot TikTok Ads connector in Lakeflowhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/tiktok-ads-troubleshoot
Fix UNITY_CATALOG_INITIALIZATION_FAILED in Databricks pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/uc-initialization-troubleshoot
Troubleshoot Workday HCM connector in Lakeflowhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/workday-hcm-troubleshoot
Diagnose and fix Databricks Workday connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/workday-reports-troubleshoot
Diagnose and fix Zendesk Support connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/zendesk-support-troubleshoot
Troubleshoot common Zoom Logs connector errorshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/zoom-logs-troubleshoot
Diagnose Zerobus Ingest API errors and handlinghttps://learn.microsoft.com/en-us/azure/databricks/ingestion/zerobus-errors
Inspect logs for Databricks init script executionhttps://learn.microsoft.com/en-us/azure/databricks/init-scripts/logs
Test and validate Databricks ODBC driver connectionshttps://learn.microsoft.com/en-us/azure/databricks/integrations/odbc/testing
Configure and troubleshoot Lakeflow Jobs with many taskshttps://learn.microsoft.com/en-us/azure/databricks/jobs/large-jobs
Troubleshoot and repair Azure Databricks Lakeflow job failureshttps://learn.microsoft.com/en-us/azure/databricks/jobs/repair-job-failures
Manage and debug Foundation Model Fine-tuning runshttps://learn.microsoft.com/en-us/azure/databricks/large-language-models/foundation-model-training/view-manage-runs
Monitor and troubleshoot Lakeflow Spark pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/observability
Use pipeline query history for debugging and tuninghttps://learn.microsoft.com/en-us/azure/databricks/ldp/query-history
Recover Databricks pipelines from checkpoint failureshttps://learn.microsoft.com/en-us/azure/databricks/ldp/recover-streaming
User guides, migration, and troubleshooting for AI Runtimehttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/ai-runtime/guides
Troubleshoot Databricks Feature Store issueshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/troubleshooting-and-limitations
Troubleshoot Databricks Model Serving endpoint issueshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/model-serving-debug
Diagnose Databricks model serving issues with Genie Codehttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/model-serving-genie-code
Debug Python code in Databricks notebookshttps://learn.microsoft.com/en-us/azure/databricks/notebooks/debugger
Troubleshoot failing Spark jobs and executors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/failing-spark-jobs
Use Databricks Spark jobs timeline for debugginghttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/jobs-timeline
Diagnose long-running Spark stages in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/long-spark-stage
Investigate high I/​O Spark stages in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/long-spark-stage-io
Debug slow low-I/​O Spark stages in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/slow-spark-stage-low-io
Identify expensive reads in Spark DAG on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-dag-expensive-read
Diagnose gaps between Spark jobs in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-job-gaps
Diagnose and fix Spark memory issues on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-memory-issues
Troubleshoot Azure Databricks Partner Connect issueshttps://learn.microsoft.com/en-us/azure/databricks/partner-connect/troubleshoot
Retrieve exceptions from terminated StreamingQueryhttps://learn.microsoft.com/en-us/azure/databricks/pyspark/reference/classes/streamingquery/exception
Debug streaming queries with explain planshttps://learn.microsoft.com/en-us/azure/databricks/pyspark/reference/classes/streamingquery/explain
Troubleshoot Databricks Git folder sync errorshttps://learn.microsoft.com/en-us/azure/databricks/repos/errors-troubleshooting
Fetch cursor rows and handle SQLSTATE in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/control-flow/fetch-stmt
Use GET DIAGNOSTICS for SQL error handling in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/control-flow/get-diagnostics-stmt
Open cursors and handle errors with OPEN in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/control-flow/open-stmt
Validate UTF-8 strings and handle INVALID_UTF8_STRINGhttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/functions/validate_utf8
Interpret Databricks SQL query performance insightshttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/performance-insights
Use Databricks SQL query history to debug performancehttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-history
Analyze Databricks SQL query profiles to find bottleneckshttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-profile
Inspect Structured Streaming state data for monitoring and debugginghttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/read-state

Best Practices

TopicURL
Use default Databricks policy families to enforce compute best practiceshttps://learn.microsoft.com/en-us/azure/databricks/admin/clusters/policy-families
Apply identity best practices in Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/best-practices
Configure default deletion vectors for Databricks Delta tableshttps://learn.microsoft.com/en-us/azure/databricks/admin/workspace-settings/deletion-vectors
Apply best practices to Azure Databricks serverless workspaceshttps://learn.microsoft.com/en-us/azure/databricks/admin/workspace/serverless-workspaces-best-practices
Improve Databricks AI Search performance and latencyhttps://learn.microsoft.com/en-us/azure/databricks/ai-search/best-practices
Optimize Databricks AI Search cost managementhttps://learn.microsoft.com/en-us/azure/databricks/ai-search/cost-management
Optimize Databricks AI Search retrieval qualityhttps://learn.microsoft.com/en-us/azure/databricks/ai-search/retrieval-quality
Identify and clean up unused AI Search endpointshttps://learn.microsoft.com/en-us/azure/databricks/ai-search/unused-endpoints
Migrate Databricks library installs from init scriptshttps://learn.microsoft.com/en-us/azure/databricks/archive/compute/libraries-init-scripts
Apply compute policy best practices in Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/archive/compute/policies-best-practices
Use DBIO for transactional writes to cloud storage in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/archive/legacy/dbio-commit
Optimize skewed joins in Databricks using skew hintshttps://learn.microsoft.com/en-us/azure/databricks/archive/legacy/skew-join
Migrate from Databricks Deep Learning Pipelineshttps://learn.microsoft.com/en-us/azure/databricks/archive/spark-3.x-migration/deep-learning-pipelines
Apply Azure Databricks administration best practiceshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/administration
Optimize BI performance with Databricks SQL warehouseshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/bi-serving
Prepare and model data for high-performance BI on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/bi-serving-data-prep
Configure Databricks SQL warehouses for optimal BI servinghttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/bi-serving-sql-serving
Apply Azure Databricks compute creation best practiceshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/compute
Implement Azure Databricks production job scheduling best practiceshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/jobs
Best practices for Power BI dashboards on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/power-bi
Apply Databricks compute configuration best practiceshttps://learn.microsoft.com/en-us/azure/databricks/compute/cluster-config-best-practices
Use flexible node types for reliable Databricks computehttps://learn.microsoft.com/en-us/azure/databricks/compute/flexible-node-types
Apply best practices for Databricks poolshttps://learn.microsoft.com/en-us/azure/databricks/compute/pool-best-practices
Optimize Azure Databricks serverless compute usagehttps://learn.microsoft.com/en-us/azure/databricks/compute/serverless/best-practices
Tune Databricks SQL warehouses for BI workloadshttps://learn.microsoft.com/en-us/azure/databricks/compute/sql-warehouse/bi-workload-settings
Use system table queries to monitor SQL warehouseshttps://learn.microsoft.com/en-us/azure/databricks/compute/sql-warehouse/monitor/queries
Control large interactive queries with Query Watchdoghttps://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/query-watchdog
Apply observability best practices for Databricks jobs and pipelineshttps://learn.microsoft.com/en-us/azure/databricks/data-engineering/observability-best-practices
Apply best practices for Unity Catalog ABAC policieshttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/abac/best-practices
Optimize performance of Unity Catalog ABAC policieshttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/abac/performance
Understand ABAC policy evaluation behavior in Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/abac/policy-evaluation
Apply Unity Catalog best practices for data governancehttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/best-practices
Work with legacy Hive metastore database objectshttps://learn.microsoft.com/en-us/azure/databricks/database-objects/hive-metastore
Follow DBFS root storage recommendations in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/dbfs/dbfs-root
Migrate from DBFS mounts to Unity Catalog external locationshttps://learn.microsoft.com/en-us/azure/databricks/dbfs/mounts
Apply DBFS and Unity Catalog usage best practiceshttps://learn.microsoft.com/en-us/azure/databricks/dbfs/unity-catalog
Optimize Delta Sharing to reduce cross-region egress costshttps://learn.microsoft.com/en-us/azure/databricks/delta-sharing/manage-egress
Apply Delta Lake best practices on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/delta/best-practices
Optimize Databricks tables with liquid clusteringhttps://learn.microsoft.com/en-us/azure/databricks/delta/clustering
Improve query performance with Databricks data skippinghttps://learn.microsoft.com/en-us/azure/databricks/delta/data-skipping
Drop or replace Delta and Unity Catalog tables safelyhttps://learn.microsoft.com/en-us/azure/databricks/delta/drop-table
Optimize Delta table file layout on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/delta/optimize
Handle Delta Lake limitations on Amazon S3https://learn.microsoft.com/en-us/azure/databricks/delta/s3-limitations
Choose selective overwrite options in Delta Lakehttps://learn.microsoft.com/en-us/azure/databricks/delta/selective-overwrite
Control Delta table data file size on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/delta/tune-file-size
Vacuum Delta tables and manage retention safelyhttps://learn.microsoft.com/en-us/azure/databricks/delta/vacuum
Optimize VARIANT data performance with shredding on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/delta/variant-shredding
Apply MLOps Stack best practices with bundleshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/mlops-stacks
Apply Databricks-specific CI/​CD workflows and patternshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/best-practices
Apply security and performance best practices for Databricks appshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-apps/best-practices
Test Databricks Connect for Python code with pytesthttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/python/testing
Handle async queries and interruptions in Databricks Connecthttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/queries
Choose between Databricks volumes and workspace fileshttps://learn.microsoft.com/en-us/azure/databricks/files/files-recommendations
Design effective evaluation sets for Databricks agentshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-evaluation/evaluation-set
Measure RAG performance with Databricks metricshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/evaluate-assess-performance
Evaluate and monitor RAG apps on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/fundamentals-evaluation-monitoring-rag
Optimize Databricks RAG application qualityhttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/quality-overview
Improve Databricks RAG chain qualityhttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/quality-rag-chain
Apply product-specific prompting tips for Genie Codehttps://learn.microsoft.com/en-us/azure/databricks/genie-code/tips
Curate high-quality Genie Spaces for accuracyhttps://learn.microsoft.com/en-us/azure/databricks/genie/best-practices
Configure Databricks Auto Loader for production workloadshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/production
Apply common COPY INTO data loading patternshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/copy-into/examples
Apply common patterns for Lakeflow ingestionhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/common-patterns
Analyze Lakeflow Connect costs with system.billing.usagehttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/monitor-costs
Maintain Lakeflow managed ingestion pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/pipeline-maintenance
Maintain and operate PostgreSQL ingestion pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/postgresql-maintenance
RabbitMQ connector behavioral FAQs and guidancehttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/rabbitmq-faq
Enable incremental ingestion for Salesforce formula fieldshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/salesforce-formula-fields
SharePoint connector FAQs and behavioral guidancehttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sharepoint-faq
Use Databricks init scripts for cluster customizationhttps://learn.microsoft.com/en-us/azure/databricks/init-scripts/
Reference external files safely in Databricks init scriptshttps://learn.microsoft.com/en-us/azure/databricks/init-scripts/referencing-files
Configure and optimize compute for Lakeflow Jobshttps://learn.microsoft.com/en-us/azure/databricks/jobs/compute
Build metadata-driven For each jobs with control tableshttps://learn.microsoft.com/en-us/azure/databricks/jobs/how-to/foreach-sql-lookup-tutorial
Apply best practices for configuring classic Lakeflow Jobshttps://learn.microsoft.com/en-us/azure/databricks/jobs/run-classic-jobs
Apply Databricks lakehouse cost optimization practiceshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/cost-optimization/best-practices
Apply data and AI governance best practices on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/data-governance/best-practices
Design observability and monitoring strategy for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/observability
Implement interoperability and usability best practices on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/interoperability-and-usability/best-practices
Apply operational excellence practices in Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/operational-excellence/best-practices
Optimize performance efficiency in Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/performance-efficiency/best-practices
Apply reliability best practices on Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/reliability/best-practices
Optimize Lakeflow pipelines with enhanced autoscalinghttps://learn.microsoft.com/en-us/azure/databricks/ldp/auto-scaling
Best practices for Lakeflow Spark Declarative Pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/best-practices
Use advanced AUTO CDC features and monitor processing metricshttps://learn.microsoft.com/en-us/azure/databricks/ldp/cdc-advanced
Develop and test Lakeflow Spark Declarative Pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/develop
Manage Python dependencies in Databricks pipelines safelyhttps://learn.microsoft.com/en-us/azure/databricks/ldp/developer/external-dependencies
Implement Databricks expectations at scale in pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/expectation-patterns
Apply data quality expectations in Databricks pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/expectations
Reduce high initialization times in Lakeflow pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/fix-high-init
Backfill historical data with Lakeflow pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/flows-backfill
Run full refresh operations for Databricks streaming tables safelyhttps://learn.microsoft.com/en-us/azure/databricks/ldp/full-refresh-st
Optimize Databricks stateful streaming with watermarkshttps://learn.microsoft.com/en-us/azure/databricks/ldp/stateful-processing
Restart the Python process to refresh Databricks librarieshttps://learn.microsoft.com/en-us/azure/databricks/libraries/restart-python-process
Apply data loading best practices on Databricks AI Runtimehttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/ai-runtime/dataloading
Apply Hyperopt best practices on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/automl-hyperparam-tuning/hyperopt-best-practices
Use covariates to improve Databricks AutoML forecastinghttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/automl/automl-covariate-forecast
Implement point-in-time correct feature joins for time series MLhttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/time-series
Benchmark Databricks LLM provisioned throughput endpointshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/foundation-model-apis/prov-throughput-run-benchmark
Apply Databricks batch model inference patternshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-inference/
Validate Databricks models before serving deploymenthttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/model-serving-pre-deployment-validation
Monitor Databricks Model Serving quality and healthhttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/monitor-diagnose-endpoints
Optimize Databricks Model Serving endpoints for productionhttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/production-optimization
Plan and execute load testing for Databricks serving endpointshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/what-is-load-test
Tune and scale Ray clusters on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/ray/scale-ray
Apply deep learning best practices on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-model/dl-best-practices
Adapt Apache Spark workloads for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/migration/spark
Evaluate and monitor Databricks AI agents with MLflowhttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/
Align Azure Databricks LLM judges with human evaluatorshttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/align-judges
Evaluate and compare MLflow prompt versions effectivelyhttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/prompt-version-mgmt/prompt-registry/evaluate-prompts
Use manual MLflow tracing for production GenAI appshttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/tracing/app-instrumentation/manual-tracing/
Log and analyze GenAI user feedback with MLflowhttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/tracing/collect-user-feedback/
Analyze GenAI traces for errors and performancehttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/tracing/observe-with-traces/analyze-traces
Run Databricks notebooks safely and efficientlyhttps://learn.microsoft.com/en-us/azure/databricks/notebooks/run-notebook
Test and schedule Databricks notebook codehttps://learn.microsoft.com/en-us/azure/databricks/notebooks/test-notebooks
Apply performance optimization recommendations on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/
Use adaptive query execution on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/aqe
Leverage cost-based optimizer in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/optimizations/cbo
Improve read performance with Databricks disk cachehttps://learn.microsoft.com/en-us/azure/databricks/optimizations/disk-cache
Improve Delta query performance with dynamic file pruning on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/dynamic-file-pruning
Use incremental refresh for Databricks materialized viewshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/incremental-refresh
Accelerate data access with predictive I/​Ohttps://learn.microsoft.com/en-us/azure/databricks/optimizations/predictive-io
Optimize Azure Databricks range join performancehttps://learn.microsoft.com/en-us/azure/databricks/optimizations/range-join
Diagnose Databricks Spark cost and performance in UIhttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/
Debug skew and spill in Databricks Spark stageshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/long-spark-stage-page
Handle Databricks spot instance losses effectivelyhttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/losing-spot-instances
Resolve long Spark stages with a single taskhttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/one-spark-task
Optimize many small Spark jobs on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/small-spark-jobs
Mitigate overloaded Spark driver on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-driver-overloaded
Detect unnecessary data rewriting in Databricks Spark writeshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-rewriting-data
Best practices for setting up Databricks Partner Connecthttps://learn.microsoft.com/en-us/azure/databricks/partner-connect/best-practice
Handle to_utc_timestamp semantics in Spark Databrickshttps://learn.microsoft.com/en-us/azure/databricks/pyspark/reference/functions/to_utc_timestamp
Configure networking for Databricks Lakehouse Federation data sourceshttps://learn.microsoft.com/en-us/azure/databricks/query-federation/networking
Optimize performance of Lakehouse Federation querieshttps://learn.microsoft.com/en-us/azure/databricks/query-federation/performance-recommendations
Transform complex and nested data types in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/semi-structured/complex-types
Use higher-order functions on arrays in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/semi-structured/higher-order-functions
Differences between VARIANT and JSON strings in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/semi-structured/variant-json-diff
Work with OBJECT type and VARIANT schemas in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/data-types/object-type
Use VARIANT type and Iceberg compatibility in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/data-types/variant-type
Collect table statistics with ANALYZE TABLE for optimizationhttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/sql-ref-syntax-aux-analyze-compute-statistics
Benchmark Databricks SQL warehouses with the TPC-DS datasethttps://learn.microsoft.com/en-us/azure/databricks/sql/tpcds-eval
Author effective SQL patterns for Databricks alertshttps://learn.microsoft.com/en-us/azure/databricks/sql/user/alerts/query-patterns
Optimize Databricks SQL queries with RELY constraintshttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-optimization-constraints
Use Structured Streaming checkpoints safely on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/checkpoints
Run multiple Structured Streaming queries per clusterhttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/multiple-streams
Optimize Databricks Structured Streaming for productionhttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/production
Optimize and monitor Databricks real-time streaming performancehttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/real-time/performance
Manage stateful Structured Streaming in Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/stateful-streaming
Optimize stateless Structured Streaming queries on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/stateless-streaming
Apply watermarks for stateful streaming on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/watermarks
Analyze Databricks table size and optimize storage costshttps://learn.microsoft.com/en-us/azure/databricks/tables/size
Design data models optimized for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/transform/data-modeling
Optimize join performance for Azure Databricks workloadshttps://learn.microsoft.com/en-us/azure/databricks/transform/optimize-joins
Clean and validate data with Databricks batch and streaminghttps://learn.microsoft.com/en-us/azure/databricks/transform/validate
Optimize Unity Catalog batch Python UDF performancehttps://learn.microsoft.com/en-us/azure/databricks/udf/python-batch-udf
Download internet data into Azure Databricks volumeshttps://learn.microsoft.com/en-us/azure/databricks/volumes/download-internet-files

Decision Making

TopicURL
Manage and change Azure Databricks subscription tierhttps://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/account
Plan migration from Standard to Premium Databricks workspaceshttps://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/standard-tier
Migrate Databricks identity from SCIM to automatic managementhttps://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/automatic-identity-management/migrate-to-aim
Decide when and how to use serverless Databricks workspaceshttps://learn.microsoft.com/en-us/azure/databricks/admin/workspace/serverless-workspaces
Manage Unity AI Gateway budgets and spendhttps://learn.microsoft.com/en-us/azure/databricks/ai-gateway/budgets-beta
Track Databricks AI Search costs with usage policieshttps://learn.microsoft.com/en-us/azure/databricks/ai-search/budget-policies
Decide and migrate from dbx to Databricks bundleshttps://learn.microsoft.com/en-us/azure/databricks/archive/dev-tools/dbx/dbx-migrate
Plan and migrate from legacy Databricks online tableshttps://learn.microsoft.com/en-us/azure/databricks/archive/machine-learning/feature-store/online-tables
Migrate optimized LLM endpoints to provisioned throughputhttps://learn.microsoft.com/en-us/azure/databricks/archive/machine-learning/migrate-provisioned-throughput
Decide when to use Databricks Light runtimehttps://learn.microsoft.com/en-us/azure/databricks/archive/runtime/light
Plan migration of Databricks workloads to Spark 3.xhttps://learn.microsoft.com/en-us/azure/databricks/archive/spark-3.x-migration/
Choose connection patterns for metric views in BI toolshttps://learn.microsoft.com/en-us/azure/databricks/business-semantics/metric-views/bi-tools
Choose and manage the Unity Catalog default cataloghttps://learn.microsoft.com/en-us/azure/databricks/catalogs/default
Choose appropriate Azure Databricks compute typeshttps://learn.microsoft.com/en-us/azure/databricks/compute/choose-compute
Decide when and how to use GPU Databricks computehttps://learn.microsoft.com/en-us/azure/databricks/compute/gpu
Decide when and how to use Azure Databricks poolshttps://learn.microsoft.com/en-us/azure/databricks/compute/pool-index
Plan migration from classic to serverless Databricks computehttps://learn.microsoft.com/en-us/azure/databricks/compute/serverless/migration
Choose Databricks serverless options for streaminghttps://learn.microsoft.com/en-us/azure/databricks/compute/serverless/streaming
Choose and manage Azure Databricks SQL warehouse sizing and scalinghttps://learn.microsoft.com/en-us/azure/databricks/compute/sql-warehouse/warehouse-behavior
Choose between Databricks SQL warehouse typeshttps://learn.microsoft.com/en-us/azure/databricks/compute/sql-warehouse/warehouse-types
Choose between ABAC and table-level filters in Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/abac/abac-vs-rls-cm
Choose between managed and external Unity Catalog assetshttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/managed-versus-external
Plan and execute upgrade of Databricks workspaces to Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/upgrade/
Prepare and migrate to Unity Catalog–only Databricks workspaceshttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/upgrade/uc-only-migration
Choose Delta Lake protocol versions and feature setshttps://learn.microsoft.com/en-us/azure/databricks/delta/feature-compatibility
Choose local development tools for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/
Migrate from legacy to new Databricks CLIhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/migrate
Plan migration from legacy Databricks Connect runtimeshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect-legacy
Migrate from older to new Databricks Connect for Pythonhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/python/migrate
Migrate Scala projects to Databricks Connect 13.3+https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/scala/migrate
Choose and use Databricks SDKs for automationhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/sdks
Decide between CDKTF and Databricks Terraform providerhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/terraform/cdktf
Decide when to migrate agents to Databricks Appshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/migrate-agent-to-apps
Set Genie budgets and cost controls in Unity AI Gatewayhttps://learn.microsoft.com/en-us/azure/databricks/genie/budgets
Choose between Databricks Free Edition and free trialhttps://learn.microsoft.com/en-us/azure/databricks/getting-started/free-trial-vs-free-edition
Choose incremental ingestion options from cloud object storagehttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/
Select Auto Loader file detection mode for your workloadhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/file-detection-modes
Plan migration of existing data to Delta Lake on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/data-migration/
Plan and configure MySQL ingestion with Lakeflow Connecthttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/mysql
Understand Slack logs connector requirements and supporthttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/slack-access-integration-logs-faq
Understand Zoom Logs connector requirements and capabilitieshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/zoom-logs-faq
Choose and start with Databricks ODBC and JDBC drivershttps://learn.microsoft.com/en-us/azure/databricks/integrations/jdbc-odbc-bi
Migrate from Simba Spark ODBC to Databricks ODBChttps://learn.microsoft.com/en-us/azure/databricks/integrations/odbc/migration
Plan and manage production workloads with Lakeflow Jobshttps://learn.microsoft.com/en-us/azure/databricks/jobs/
Migrate from Spark Submit task to supported Databricks job typeshttps://learn.microsoft.com/en-us/azure/databricks/jobs/spark-submit
Plan production Azure Databricks lakehouse deploymentshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/
Design Azure Databricks workspace strategyhttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/workspace-strategy
Choose a programming language for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/languages/overview
Plan migration from deprecated Foundation Model Fine-tuninghttps://learn.microsoft.com/en-us/azure/databricks/large-language-models/foundation-model-training/
Assess environment version compatibility for Lakeflow pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/developer/environment-version-compatibility
Choose triggered vs continuous mode for pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/pipeline-mode
Migrate Databricks online tables to Lakebasehttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/migrate-from-online-tables
Upgrade workspace feature tables to Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/uc/upgrade-feature-table-to-uc
Choose Databricks-hosted foundation models via APIshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/foundation-model-apis/supported-models
Migrate Databricks models to Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/manage-model-lifecycle/migrate-to-uc
Upgrade ML workflows to Unity Catalog modelshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/manage-model-lifecycle/upgrade-workflows
Migrate from legacy MLflow Model Serving to Databricks Model Servinghttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/migrate-model-serving
Choose between Spark and Ray on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/ray/spark-ray-overview
Plan for Databricks generative AI model lifecyclehttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/retired-models-policy
Decide when to use distributed training on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-model/distributed-training/
Choose and train deep-learning recommenders on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-recommender-models
Plan migration of data applications to Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/migration/
Scope and plan ETL pipeline migration to Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/migration/etl
Select a migration path from Parquet to Delta Lakehttps://learn.microsoft.com/en-us/azure/databricks/migration/parquet-to-delta-lake
Plan migration from data warehouse to Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/migration/warehouse-to-lakehouse
Migrate from Agent Evaluation to MLflow 3 on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/agent-eval-migration
Quick reference for migrating to MLflow 3https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/agent-eval-migration-reference
Choose between open source and managed MLflow on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/overview/oss-managed-diff
Choose Lakebase backup and restore methodshttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/backup-methods
Choose between Databricks Apps and external Lakebase appshttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/build-applications
Plan and manage Lakebase upgrade to Autoscalinghttps://learn.microsoft.com/en-us/azure/databricks/oltp/upgrade-to-autoscaling
Choose pandas options on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/pandas/
Choose between Azure Databricks and Fabric integrationshttps://learn.microsoft.com/en-us/azure/databricks/partners/bi/fabric
Migrate legacy Databricks query federation to Lakehouse Federationhttps://learn.microsoft.com/en-us/azure/databricks/query-federation/migrate
Plan and execute Databricks Runtime 11.x migrationhttps://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/11.x-migration
Migrate workloads to Databricks Runtime 12.x safelyhttps://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/12.x-migration
Plan and execute Databricks Runtime 13.x migrationhttps://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/13.x-migration
Migrate workloads to Databricks Runtime 14.x safelyhttps://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/14.x-migration
Assess Databricks Runtime support lifecycle and upgradeshttps://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/databricks-runtime-ver
Choose Azure Databricks serverless SKUs and DBU rateshttps://learn.microsoft.com/en-us/azure/databricks/resources/pricing
Plan and manage networking costs for Databricks serverlesshttps://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/cost-management
Choose and use Databricks workspace export optionshttps://learn.microsoft.com/en-us/azure/databricks/security/privacy/export-workspace-data
Decide between Spark Connect and Spark Classic on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/spark/connect-vs-classic
Choose between SparkR and sparklyr on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sparkr/sparkr-vs-sparklyr
Use SYNC to upgrade Hive tables to Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/sql-ref-syntax-aux-sync
Choose and size SQL warehouses for alertshttps://learn.microsoft.com/en-us/azure/databricks/sql/user/alerts/compute
Choose Structured Streaming output modes on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/output-mode
Choose and implement Databricks transaction modeshttps://learn.microsoft.com/en-us/azure/databricks/transactions/transaction-modes

Architecture & Design Patterns

TopicURL
Plan disaster recovery architecture for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/admin/disaster-recovery
Apply Databricks agent system design patternshttps://learn.microsoft.com/en-us/azure/databricks/agents/agent-system-design-patterns
Design materialization strategy for Databricks metric viewshttps://learn.microsoft.com/en-us/azure/databricks/business-semantics/metric-views/materialization
Implement fan-in and fan-out in Lakeflow pipelineshttps://learn.microsoft.com/en-us/azure/databricks/data-engineering/fan-in-fan-out
Choose patterns for external access to Unity Catalog datahttps://learn.microsoft.com/en-us/azure/databricks/external-access/
Use Compatibility Mode for external Delta and Iceberg readshttps://learn.microsoft.com/en-us/azure/databricks/external-access/compatibility-mode
Build an IDP pipeline with Databricks AI Functionshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-bricks/idp-pipeline-tutorial
Design intelligent document processing pipelines on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-bricks/intelligent-document-processing
Design measurement infrastructure for RAG quality on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/evaluate-enable-measurement
Design and tune Databricks RAG inference chainshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/fundamentals-inference-chain-rag
Design cost optimization architecture for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/cost-optimization/
Apply data and AI governance architecture on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/data-governance/
Design Delta Lake and medallion architecture on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/delta-lake
Plan high availability and disaster recovery for Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/ha-dr
Design Azure Databricks network and connectivity architecturehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/network
Design storage architecture for Azure Databricks and Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/storage
Design interoperability and usability architecture for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/interoperability-and-usability/
Design operational excellence architecture for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/operational-excellence/
Design performance efficiency architecture for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/performance-efficiency/
Apply Databricks lakehouse reference architectureshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/reference
Design reliability architecture for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/reliability/
Apply medallion lakehouse architecture on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse/medallion
Choose Databricks model deployment patternshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/mlops/deployment-patterns
Design MLOps workflows on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/mlops/mlops-workflow
Configure high availability for Lakebase instanceshttps://learn.microsoft.com/en-us/azure/databricks/oltp/instances/create/high-availability
Use Lakebase for durable AI agent state and memoryhttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/state-management
Set up Databricks Serverless Private Git with Private Linkhttps://learn.microsoft.com/en-us/azure/databricks/repos/serverless-private-git
Apply data exfiltration protection reference architectureshttps://learn.microsoft.com/en-us/azure/databricks/security/network/data-exfiltration-protection/architecture
Choose Azure Databricks network reference architectureshttps://learn.microsoft.com/en-us/azure/databricks/security/network/deployment-architecture/
Deploy hardened connectivity architecture for Databrickshttps://learn.microsoft.com/en-us/azure/databricks/security/network/deployment-architecture/hardened-connectivity
Design isolated environment architecture for Databrickshttps://learn.microsoft.com/en-us/azure/databricks/security/network/deployment-architecture/isolated-environment
Implement managed security network architecture for Databrickshttps://learn.microsoft.com/en-us/azure/databricks/security/network/deployment-architecture/managed-security
Choose patterns for modeling semi-structured data on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/semi-structured/
Choose async checkpointing for Databricks stateful querieshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/async-checkpointing
Use async progress tracking in Databricks streaminghttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/async-progress-checking
Decide when and how to partition Delta tableshttps://learn.microsoft.com/en-us/azure/databricks/tables/partitions