The SDK is a thin, generated wrapper over the Databricks REST API. You make one client — w = WorkspaceClient() — and every service hangs off it as a property: w.clusters, w.jobs, w.catalogs. Each call maps one-to-one to a REST endpoint, takes and returns typed dataclasses, and authenticates itself from the environment. Learn the shape once and all ~110 services feel the same.
install · auth · clientcomputejobs & pipelinesSQL & dataUnity Catalogworkspace · ML · accountgotcha · pattern★most common
Verified 2026-08-27 against the Databricks SDK for Python documentation (Getting Started, Authentication, Long-running operations, Pagination, Workspace & Account client references) and the PyPI release history. · databricks-sdk 0.133.0 (19 Aug 2026), Python ≥3.10 · Beta: production-supported, but pin the version — type names occasionally change between minor releases. Bundled in Databricks Runtime 13.1+; %pip install --upgrade for the newest.
Mental model — one client, a property per service, each call a typed REST round-trip
↔ swipe to see the whole diagram
One client, then everything — auth, list, create-and-wait, run SQL
# pip install databricks-sdk — auth comes from env / .databrickscfg, no secrets in codefrom databricks.sdk import WorkspaceClient
w = WorkspaceClient() # ★ the one line you always start with# list — .list() returns a lazy Iterator[T], pagination handled for youfor c in w.clusters.list():
print(c.cluster_id, c.cluster_name, c.state)
# create and block until the cluster is RUNNING (a long-running operation)
info = w.clusters.create_and_wait(
cluster_name="my-cluster",
spark_version=w.clusters.select_spark_version(long_term_support=True),
node_type_id=w.clusters.select_node_type(local_disk=True),
autotermination_minutes=20, num_workers=2)
# run SQL on a warehouse and read rows straight back
r = w.statement_execution.execute_statement(
warehouse_id="abc123", statement="SELECT * FROM main.default.people LIMIT 10")
print(r.result.data_array)
# typed errors, not string matchingfrom databricks.sdk.errors import ResourceDoesNotExist
try:
w.clusters.get(cluster_id="nope")
except ResourceDoesNotExist as e:
print("gone:", e)
Part ISetup, authentication & the clientinstall · auth flow · WorkspaceClient · AccountClient · config
01Install & import★
databricks-sdk 0.133 · Python ≥3.10
pip install databricks-sdkThe whole SDK, one package. No cloud-specific variants — it detects AWS / Azure / GCP itself.★
from databricks.sdk import WorkspaceClient
w = WorkspaceClient()w is the conventional name (short for workspace). Press TAB after w. for autocompletion of every service.★
from databricks.sdk import AccountClient
a = AccountClient()Account-level operations (users, workspaces, billing). Conventionally a.
%pip install --upgrade databricks-sdk
dbutils.library.restartPython()In a notebook the SDK is pre-bundled (Runtime 13.1+); upgrade then restart Python to pick up the new version.notebook
pip install "databricks-sdk[notebook,openai]"Extras: notebook for in-notebook helpers, openai for the OpenAI-compatible serving client.
import databricks.sdk; databricks.sdk.version.__version__Beta package — pin the exact version; minor releases occasionally rename types for consistency.
02Authentication that just works★
unified auth, resolved from the environment
w = WorkspaceClient() # zero argsIn a notebook it reads the execution context. Elsewhere it reads env vars then .databrickscfg. This is the happy path.★
export DATABRICKS_HOST=https://... DATABRICKS_TOKEN=dapi...The two env vars that cover most CI and scripts. host + a personal access token (PAT).
# ~/.databrickscfg
[DEFAULT]
host = https://myorg.cloud.databricks.com
token = dapi...Profile-based config, shared with the CLI and Terraform. Pick one with WorkspaceClient(profile="prod").
w = WorkspaceClient(host="https://...", token="dapi...")Explicit args work but avoid hard-coding secrets — they leak into version control.gotcha
w = WorkspaceClient(auth_type="external-browser")Opens a browser for interactive SSO — handy on a dev laptop. auth_type also forces a specific method.
# order tried Databricks native → Azure → GCPPAT first, then OAuth/WIF, then cloud-native. Methods whose config is absent are skipped automatically.
03Config & tuning the client
timeouts, retries, debugging
from databricks.sdk.core import Config
w = WorkspaceClient(config=Config(retry_timeout_seconds=300))Every knob lives on Config. Pass a prebuilt one, or set fields as client kwargs directly.
WorkspaceClient(http_timeout_seconds=60, retry_timeout_seconds=300)Per-request timeout vs total retry budget (default 5 min). Raise the latter for slow long-running ops.
WorkspaceClient(profile="prod", config_file="~/.dbcfg")Point at a non-default profile or a non-default credentials file.
WorkspaceClient(rate_limit=10) # req/sec capThrottle yourself below the API's limit when running large fan-out jobs.
w.config.host · w.config.auth_typeInspect what actually got resolved — the fastest way to debug "which host / which credential am I using".
from databricks.sdk import useragent
useragent.with_product("my-tool", "1.2.0")Tag requests for attribution; partners use with_partner(). Shows up in the User-Agent header.
04Workspace vs Account★
two clients, two scopes
w = WorkspaceClient() # inside one workspaceClusters, jobs, notebooks, catalogs, SQL — anything scoped to a single workspace.★
a.workspaces.list() · a.users.list() · a.metastores.list()The account client exposes its own service set — workspace management, not cluster management.
# notebook auto-auth does NOT work for AccountClientImplicit context auth is workspace-only. The account client always needs explicit host + account_id.gotcha
w = WorkspaceClient(profile="unified")
a = AccountClient(profile="unified")On a unified host, one profile drives both — workspace_id routes workspace calls, account_id routes account calls.0.11x
a.get_workspace_client(ws) # account → workspaceSpin up a workspace client for a specific workspace from the account client.
05The universal call shape★
learn once, applies to all ~110 services
w.<service>.<verb>(...)Every service is a property; every verb is a method. w.jobs.create, w.catalogs.list, w.secrets.put_secret.★
.list() → Iterator[T] · .get(id) → T · .create(...) → TCRUD verbs are consistent across services. .delete(), .update(), .edit() round it out.
obj.as_dict() · SomeType.from_dict(d)Every dataclass round-trips to/from plain dicts — handy for JSON, logging, or diffing.
from databricks.sdk.service.compute import State
if c.state == State.RUNNING: ...Enums, not magic strings. Import them from databricks.sdk.service.<area>.
import databricks.sdk.service.jobs as j
j.Task(...) j.RunSubmitTaskSettings(...)Request bodies are dataclasses in the same module. Aliasing the module (as j) keeps call sites short.
w.api_client.do("GET", "/api/2.0/...") # escape hatchFor a brand-new endpoint the typed method hasn't caught up to, call the raw REST client directly.
06Errors, logging & retries
typed exceptions, standard logging
from databricks.sdk.errors import ResourceDoesNotExist, PermissionDenied, ResourceAlreadyExistsCatch by type, not by parsing messages. All defined in databricks.sdk.errors.★
try:
w.jobs.get(job_id=1)
except ResourceDoesNotExist:
...Consistent across services even though the raw API error shapes differ.
from databricks.sdk.errors import DatabricksErrorThe base class — catch it to handle any platform error, then inspect .error_code.
logging.getLogger("databricks.sdk").setLevel(logging.DEBUG)Logs every request/response as > and <. Tokens are auto-redacted to **REDACTED**.
WorkspaceClient(debug_truncate_bytes=1024)Debug logs truncate long JSON by default (96 bytes) — raise it to see full payloads.
# 429 / 503 + connection resets auto-retried with backoffYou rarely write retry loops yourself; the client handles transient failures within retry_timeout_seconds.
Part IICompute & orchestrationclusters · pools · libraries · jobs · pipelines · repos
07Clusters★
w.clusters · create, start, terminate
for c in w.clusters.list():
print(c.cluster_id, c.cluster_name, c.state)Lazy iterator over all clusters. Each item is a typed ClusterDetails.★
info = w.clusters.create_and_wait(cluster_name="c1", spark_version="15.4.x-scala2.12", node_type_id="m5d.large", num_workers=2, autotermination_minutes=20)_and_wait blocks until RUNNING. Always set autotermination_minutes to avoid idle spend.★
w.clusters.select_spark_version(long_term_support=True)Resolve a valid runtime string instead of hard-coding; select_node_type(local_disk=True) does the same for instances.
w.clusters.ensure_cluster_is_running(cid)Idempotent "make sure it's up" — what dbutils calls internally.
w.clusters.events(cluster_id=cid) · w.clusters.get(cluster_id=cid)Event history for debugging autoscaling/termination; get for a live snapshot.
08Pools, policies & init scripts
w.instance_pools · w.cluster_policies
w.instance_pools.create(instance_pool_name="p1", node_type_id="m5d.large", min_idle_instances=2)Warm pools of ready VMs cut cluster start and autoscale times.★
w.instance_pools.list() · w.instance_pools.delete(instance_pool_id=pid)Point a cluster at one with instance_pool_id= instead of node_type_id=.
w.cluster_policies.create(name="pol", definition="{...}")Policies constrain what clusters users can create — instance types, autoscaling bounds, tags.
w.policy_families.list() # prebuilt templatesDatabricks-maintained starting points you clone rather than write from scratch.
w.global_init_scripts.create(name="gis", script=b64, enabled=True)Run a script on every cluster launch workspace-wide — script must be base64-encoded.
w.instance_profiles.add(instance_profile_arn=arn) # AWSRegister an IAM instance profile so clusters can assume a cloud role.
09Libraries on clusters
w.libraries · install & check status
from databricks.sdk.service.compute import Library, PythonPyPiLibraryLibrary specs are dataclasses — PyPI, Maven, wheel, jar, or a requirements file.
w.libraries.install(cluster_id=cid, libraries=[Library(pypi=PythonPyPiLibrary(package="pandas==2.2.0"))])Install is async — the call returns before the library is ready.★
w.libraries.cluster_status(cluster_id=cid)Poll here for INSTALLED vs PENDING vs FAILED before you run code that needs the library.gotcha
Library(whl="dbfs:/libs/my.whl") · Library(jar="dbfs:/libs/x.jar")Point at artifacts on DBFS, a Volume, or cloud storage.
w.libraries.uninstall(cluster_id=cid, libraries=[...])Uninstall takes effect on the next cluster restart, not immediately.
10Jobs — create & run★
w.jobs · the workflow orchestrator
import databricks.sdk.service.jobs as jAlias the module — you'll reference j.Task, j.NotebookTask, j.JobCluster a lot.★
created = w.jobs.create(name="etl", tasks=[j.Task(task_key="t1", notebook_task=j.NotebookTask(notebook_path="/Repos/etl"), new_cluster=cluster_spec)])A job is a name plus a list of Tasks; each task has a type (notebook, python, SQL, dbt) and compute.★
w.jobs.run_now_and_wait(job_id=created.job_id)Trigger an existing job and block for the run to finish. Returns the terminal Run.
waiter = w.jobs.submit(run_name="one-off", tasks=[...])
run = waiter.result(timeout=timedelta(minutes=15))submit runs a one-time job with no saved definition — great for ad-hoc work.
depends_on=[j.TaskDependency(task_key="t1")]Wire task DAGs by key; SDK sends the graph, Databricks schedules it.
w.jobs.update(job_id=id, new_settings=...) · w.jobs.delete(job_id=id)reset replaces the whole definition; update patches part of it.
11Jobs — runs & monitoring
w.jobs.list_runs · poll, inspect, repair
for run in w.jobs.list_runs(job_id=id, expand_tasks=False):
print(run.run_id, run.state.result_state)History as a lazy iterator. expand_tasks=True to see per-task detail.★
run = w.jobs.get_run(run_id=rid)
print(run.run_page_url)run_page_url deep-links to the UI — drop it into alerts and logs.
w.jobs.wait_get_run_job_terminated_or_skipped(run_id=rid, timeout=timedelta(minutes=30), callback=on_update)Poll from a separate thread/service; the callback fires on each state change.
w.jobs.get_run_output(run_id=task_run_id)Pull a task's return value or error — note it takes the task run id, not the parent.gotcha
w.jobs.repair_run(run_id=rid, rerun_tasks=["t2"])Re-run only the failed tasks instead of the whole job — cheaper recovery.
w.jobs.cancel_run(run_id=rid) · w.jobs.cancel_all_runs(job_id=id)Stop a stuck run, or clear the queue for a job.
12Pipelines (Lakeflow)
w.pipelines · declarative ETL
import databricks.sdk.service.pipelines as pThe declarative pipelines API (formerly Delta Live Tables). Create, start, and inspect update runs.
w.pipelines.create(name="dlt", libraries=[p.PipelineLibrary(notebook=p.NotebookLibrary(path="/Repos/dlt"))], target="main.etl")Pipeline = a set of source notebooks/files plus a target schema.★
w.pipelines.start_update(pipeline_id=pid, full_refresh=False)Kick an update; full_refresh=True reprocesses all data from scratch.
w.pipelines.list_pipeline_events(pipeline_id=pid)Structured event log — data quality expectations, flow progress, errors.
w.pipelines.get(pipeline_id=pid).latest_updatesCheck the most recent update's state without scanning the whole event stream.
w.pipelines.stop(pipeline_id=pid)Halt a running update gracefully.
13Repos & Git
w.repos · w.git_credentials
w.git_credentials.create(git_provider="gitHub", git_username="me", personal_access_token="ghp_...")Register a Git PAT once so Databricks can act on your behalf.
repo = w.repos.create(url="https://github.com/org/repo", provider="gitHub", path="/Repos/me/repo")Clone a repo into the workspace. path must live under /Repos.★
w.repos.update(repo_id=repo.id, branch="main")Checkout a branch or tag — this is how CI pulls the latest code before a job run.
for r in w.repos.list():
print(r.path, r.branch, r.head_commit_id)Audit which repos and commits are deployed across the workspace.
w.repos.delete(repo_id=repo.id)Remove the checkout; the remote is untouched.
14Run commands on a cluster
w.command_execution · REPL over REST
import databricks.sdk.service.compute as ccExecute Python/Scala/SQL/R on a running cluster — a programmatic REPL.
ctx = w.command_execution.create_and_wait(cluster_id=cid, language=cc.Language.PYTHON)An execution context is a persistent session on the cluster; reuse it across commands.
res = w.command_execution.execute_and_wait(cluster_id=cid, context_id=ctx.id, language=cc.Language.PYTHON, command="print(1+1)")Blocks for the result; read res.results.data.★
w.command_execution.destroy(cluster_id=cid, context_id=ctx.id)Tear the context down when done to free cluster resources.
# prefer jobs/statement_execution for productionCommand execution is great for interactive tooling; scheduled work belongs in Jobs or SQL statements.
15Long-running operations★
the _and_wait / Wait / result() pattern
w.clusters.create_and_wait(...) # blocksAny op with a wait has a <verb>_and_wait twin that returns only once the target state is reached.★
waiter = w.clusters.create(...) # returns a Wait
info = waiter.result(timeout=timedelta(minutes=10))The non-wait form returns a Wait; call .result() when you actually need to block.
waiter.result(callback=lambda x: print(x.state))The callback fires between polls — log progress without busy-waiting.
from datetime import timedelta
.result(timeout=timedelta(minutes=20))Defaults are sensible (~20 min) but override for slow clusters or big jobs.gotcha
# covers clusters · jobs · pipelines · warehouses · command execThe same waiter shape recurs across every long-running service.
from databricks.sdk.errors import OperationFailedA wait that ends in a failed terminal state raises — wrap it if you need to branch on failure.
Part IIISQL, data & Unity Catalogwarehouses · statements · catalogs · tables · volumes · grants
16Run SQL statements★
w.statement_execution · SQL over REST
r = w.statement_execution.execute_statement(warehouse_id=wid, statement="SELECT * FROM main.default.t LIMIT 100")Runs synchronously (up to a timeout) and returns inline results. The everyday way to query.★
r.result.data_array · r.manifest.schema.columnsRows as a list of lists; column names/types in the manifest. No Spark session needed.
execute_statement(..., wait_timeout="30s", on_wait_timeout="CONTINUE")If it exceeds the wait, keep running server-side and poll by statement_id.
w.statement_execution.get_statement(statement_id=sid)
w.statement_execution.get_statement_result_chunk_n(sid, chunk_index=1)Fetch status and page through large result sets chunk by chunk.gotcha
parameters=[StatementParameterListItem(name="id", value="42")]Parameterise with named markers — safer than string-formatting SQL.
disposition="EXTERNAL_LINKS"# for big resultsReturn presigned links to result files instead of inline rows when the payload is large.
17SQL warehouses
w.warehouses · the SQL compute
for wh in w.warehouses.list():
print(wh.id, wh.name, wh.state)A warehouse is the compute that runs your SQL statements — separate from clusters.★
w.warehouses.create_and_wait(name="wh", cluster_size="Small", auto_stop_mins=10, enable_serverless_compute=True)Serverless starts in seconds; set auto_stop_mins so it parks when idle.
w.warehouses.start_and_wait(id=wid)
w.warehouses.stop_and_wait(id=wid)Start before a batch, stop after — or rely on auto-stop.
w.warehouses.get_workspace_warehouse_config()Workspace-wide defaults: security policy, data access config, SQL configs.
w.warehouses.edit(id=wid, cluster_size="Medium")Resize or reconfigure in place; takes effect on next start.
18Queries, alerts & dashboards
w.queries · w.alerts · w.lakeview
w.queries.create(query=Query(display_name="q", query_text="SELECT ...", warehouse_id=wid))Saved queries (the v2 API — queries_legacy is the old one).v2
w.alerts.create(alert=Alert(...)) # alerts_v2 for the new APIFire when a query result crosses a threshold; wire to notification destinations.
w.lakeview.list() · w.lakeview.get(dashboard_id=did)Manage Lakeview (AI/BI) dashboards; lakeview_embedded for token-based embedding.
w.genie.create_message_and_wait(space_id=sid, conversation_id=cid, content="top 5 customers?")Drive Genie — the no-code natural-language BI experience — programmatically.new
w.query_history.list()Audit what SQL ran, on which warehouse, how long it took.
19Unity Catalog namespace★
catalog → schema → table, three levels
w.catalogs.create(name="main") · w.catalogs.list()The top level of UC's three-part namespace catalog.schema.table.★
w.schemas.create(name="sales", catalog_name="main")
w.schemas.list(catalog_name="main")Schemas (a.k.a. databases) live inside a catalog. list is scoped by parent.★
for t in w.tables.list(catalog_name="main", schema_name="sales"):
print(t.full_name, t.table_type)Metadata only — managed/external, columns, storage location. Not the row data.
w.tables.get(full_name="main.sales.orders")Most UC calls key off the fully-qualified catalog.schema.name.
w.metastores.list() · w.metastores.current()The metastore is the top container; current() shows the one attached to this workspace.
w.functions.list(...) · w.model_versions.list(...)UDFs and UC-registered model versions share the same namespace and governance.
20Volumes & files★
w.files · w.volumes · governed storage
w.volumes.create(catalog_name="main", schema_name="ops", name="landing", volume_type=VolumeType.MANAGED)A Volume is UC-governed file storage — the modern replacement for DBFS paths.★
w.files.upload("/Volumes/main/ops/landing/f.csv", contents, overwrite=True)Read/write files by URI. Works on Volumes; the Files API is a plain HTTP file interface.★
resp = w.files.download("/Volumes/main/ops/landing/f.csv")
data = resp.contents.read()download streams the bytes back; .contents is a file-like object.
w.files.list_directory_contents("/Volumes/main/ops/landing")List, plus create_directory / delete for folder management.
w.dbfs.open(path, write=True, overwrite=True) # legacy DBFSDBFS still works via w.dbfs, but new work should target Volumes for governance.legacy
21Grants & governance
w.grants · secure by default
import databricks.sdk.service.catalog as ucUC is secure by default — nothing is readable until granted.
w.grants.get(securable_type=uc.SecurableType.TABLE, full_name="main.sales.orders")Read the current privilege list on any securable (catalog, schema, table, volume).★
w.grants.update(securable_type=..., full_name=..., changes=[uc.PermissionsChange(principal="eng@co", add=[uc.Privilege.SELECT])])Add/remove privileges by principal. Enums for privilege names, not strings.
w.storage_credentials.list() · w.external_locations.list()Credentials authorise cloud storage; external locations bind a path to a credential.
w.connections.create(...) # Lakehouse FederationRegister external data sources (Postgres, Snowflake, ...) to query without ingesting.
w.workspace_bindings.get(name="main")Restrict which workspaces can see a catalog — OPEN vs ISOLATED.
22Data-plane vs control-plane
where a call actually goes
# control plane: metadata & managementMost calls — list, create, grants — hit the control plane at your workspace host.
# data plane: bytes & inferenceFile upload/download and serving-endpoint queries route to the data plane, sometimes a different host.gotcha
w.serving_endpoints_data_plane.query(...)Explicit data-plane clients exist where latency and locality matter.
# the SDK picks the right endpoint for youYou rarely think about it — but it explains why some calls need extra network egress rules.
w.temporary_table_credentials.generate_temporary_table_credentials(...)Short-lived, downscoped creds for reading table/volume data directly from cloud storage.
Part IVWorkspace, ML, account & the patterns that bitenotebooks · secrets · identity · serving · dbutils · gotchas
23Workspace objects & notebooks
w.workspace · import, export, list
for obj in w.workspace.list("/Users/me"):
print(obj.path, obj.object_type)Walk the workspace tree — notebooks, folders, files, dashboards.★
import databricks.sdk.service.workspace as ws
w.workspace.import_(path="/Users/me/nb", format=ws.ImportFormat.SOURCE, language=ws.Language.PYTHON, content=b64, overwrite=True)Upload a notebook — content is base64. Note the trailing underscore: import_ (import is a keyword).gotcha
exported = w.workspace.export_(path="/Users/me/nb", format=ws.ExportFormat.SOURCE)Round-trips out; exported.content is base64 you decode.
w.workspace.mkdirs("/Shared/pipeline") · w.workspace.delete(path, recursive=True)Folder management; recursive=True to remove a non-empty tree.
w.workspace.get_status("/Users/me/nb")Exists-check plus object metadata without downloading content.
24Secrets
w.secrets · scopes & keys
w.secrets.create_scope(scope="prod")A scope groups secrets. Back it by Databricks or (on Azure) Key Vault.★
w.secrets.put_secret(scope="prod", key="db_pw", string_value="...")Write a secret. In notebooks it prints as [REDACTED] when referenced.
for s in w.secrets.list_secrets(scope="prod"):
print(s.key) # metadata onlyYou can't read a secret value back through the API — only inside a notebook via dbutils.secrets.get.gotcha
w.secrets.put_acl(scope="prod", principal="eng", permission=AclPermission.READ)Grant a group read access to a scope.
w.secrets.delete_secret(scope="prod", key="db_pw")Remove a single secret or the whole scope with delete_scope.
25Identity & permissions
w.users · w.groups · w.current_user
me = w.current_user.me()
print(me.user_name, me.active)Who am I? The quickest smoke-test that auth works at all.★
for u in w.users.list():
print(u.user_name, u.id)SCIM-based user directory. groups and service_principals mirror the same shape.
w.groups.create(display_name="eng", members=[ComplexValue(value=uid)])Group membership by id. Assign access to the group, not individuals.
w.permissions.set(request_object_type="jobs", request_object_id=jid, access_control_list=[...])The generic permissions API governs jobs, clusters, notebooks, warehouses — one call shape for all.
w.permissions.get(request_object_type="clusters", request_object_id=cid)Read the current ACL before you change it — update merges, set replaces.gotcha
w.tokens.create(comment="ci", lifetime_seconds=86400)Mint a PAT programmatically; token_management lets admins see/revoke everyone's.
26Model serving & ML
w.serving_endpoints · w.vector_search_*
w.serving_endpoints.create_and_wait(name="ep", config=EndpointCoreConfigInput(served_entities=[...]))Stand up a model-serving endpoint and wait until it's ready.★
w.serving_endpoints.query(name="ep", dataframe_records=[{...}])Send inference requests; for LLMs pass messages= / inputs=.
client = w.serving_endpoints.get_open_ai_client()An OpenAI-compatible client for foundation-model endpoints — drop-in for existing OpenAI code.new
w.vector_search_indexes.create_index(...) · .query_index(...)Managed vector search — build an index, run ANN queries for RAG.
w.experiments.create_experiment(name="/Users/me/exp")
w.registered_models.list(...)MLflow tracking + the UC model registry, both under the workspace client.
w.serving_endpoints.update_config_and_wait(...)Roll out a new served model version with traffic splitting; waits for the update to land.
27Account-level admin
a.* · workspaces, identity, billing
a = AccountClient(host="https://accounts.cloud.databricks.com", account_id="...")Account operations always need explicit host + account_id (no notebook auto-auth).gotcha
for ws in a.workspaces.list():
print(ws.workspace_name, ws.workspace_status)Provision, list, and manage the workspaces in your account.★
a.users.list() · a.groups.list() · a.service_principals.list()Account-level identity; sync to workspaces via a.workspace_assignment.
a.metastores.list() · a.storage.list()Account-scoped UC metastores and their storage roots.
a.budgets.list() · a.billable_usage.download(...)Cost governance — budgets and raw billable-usage CSVs.
a.custom_app_integration.create(name="app", redirect_urls=[...])Register a custom OAuth app for building SSO webapps on Databricks.
28dbutils from the SDK
w.dbutils · fs & secrets anywhere
dbutils = w.dbutils
files = dbutils.fs.ls("/")A client-side dbutils — fs and secrets work locally, no notebook required.★
from databricks.sdk.runtime import dbutilsAlternative import; needs auth already present in env vars.
dbutils.fs.cp("/a", "/b") · dbutils.fs.mkdirs("/c")Most fs ops are implemented natively in Python inside the SDK.
dbutils.secrets.get(scope="prod", key="db_pw")The only way to actually read a secret value — the REST API won't return it.
WorkspaceClient(cluster_id="....") # for non-native opsSome dbutils ops still need a cluster; set cluster_id and the SDK ensures it's running.gotcha
29Pagination & dataclass patterns★
idioms that keep code clean
for x in w.jobs.list(): ... # never think about pages.list() is a generator — it fetches pages lazily as you iterate. Don't collect unless you must.★
all_jobs = list(w.jobs.list()) # materialise, carefullyFine for hundreds; for tens of thousands, filter server-side or stream instead.gotcha
spec = ClusterSpec(...); w.jobs.create(..., new_cluster=spec)Build request bodies as dataclasses and reuse them — type-checked, autocompleted, diffable.
settings.as_dict() # store/inspect JobSettings.from_dict(d)Persist a definition as JSON, then reconstruct it later — the basis of "jobs as code".
from databricks.sdk.service.jobs import RunResultState
if run.state.result_state == RunResultState.SUCCESS: ...Compare against enums; states differ between life-cycle and result — check the right field.
# filter args beat client-side filteringMany lists accept filter= / name args — push the predicate to the server to fetch less.
30The traps worth knowing
what surprises people first
# SDK ≠ PySparkThe SDK manages the platform. To run DataFrame/Spark code use pyspark or Databricks Connect — a different package.★
import_ export_ # trailing underscoresMethods colliding with Python keywords get a trailing _. Same for a few dataclass fields.
w.secrets.get_secret(...) # returns metadata, not the valueReading secret values is notebook-only via dbutils.secrets.get — by design.gotcha
# delete vs permanent_delete on clustersdelete = terminate (config kept). To truly remove, use permanent_delete.
# pin the version databricks-sdk==0.133.0It's Beta — type names occasionally change between minor releases. Read the CHANGELOG on upgrade.beta
AccountClient() # won't auto-auth in a notebookNotebook implicit auth is workspace-only. The account client needs explicit config.
1 · Anatomy of a callclient · service · method · typed body
↔ swipe to see the whole diagram
2 · Which client, which scopew for a workspace, a for the account
↔ swipe to see the whole diagram
3 · The authentication laddertried in order, first match wins
↔ swipe to see the whole diagram
4 · SDK vs the neighbourswhat each Databricks Python package is for
↔ swipe to see the whole diagram
Worth memorising
w = WorkspaceClient()
The one line you always start with. Zero args works in notebooks, CI, and locally — auth resolves from the environment. a = AccountClient() for account-level work.
w.<service>.<verb>(...)
Every one of ~110 services is a property; every call maps 1:1 to a REST endpoint and takes/returns typed dataclasses. Learn the shape once.
.list() is a lazy Iterator[T]
Just for x in w.jobs.list() — pagination is invisible. Only list(...) it into memory when the result set is small.
create_and_wait / .result()
Long-running ops (clusters, jobs, pipelines, warehouses) have an _and_wait twin that blocks until the target state, or return a Wait you .result(timeout=...).
dataclasses & enums, not dicts
Request bodies are typed objects from databricks.sdk.service.<area>; states are enums. obj.as_dict() / T.from_dict(d) round-trip to JSON.
catch typed errors
from databricks.sdk.errors import ResourceDoesNotExist, PermissionDenied — handle by exception type, never by parsing messages. Base class is DatabricksError.
the SDK is not PySpark
It automates the platform — clusters, jobs, catalogs, permissions. To run Spark/DataFrame code use pyspark or Databricks Connect, a separate package.
catalog.schema.table
Unity Catalog is a three-level namespace. Most UC calls key off the fully-qualified full_name; list is scoped by its parent (catalog_name=, schema_name=).
secrets: write yes, read no
You can put and list secrets via the API but never read a value back — that's notebook-only through dbutils.secrets.get, by design.
always set autotermination
On clusters and warehouses, autotermination_minutes / auto_stop_mins is the difference between a tidy bill and an idle one. delete terminates; permanent_delete removes.
import_ , export_
Methods (and a few fields) that collide with Python keywords carry a trailing underscore. w.workspace.import_(...), not import(...).
account client needs explicit auth
Notebook auto-auth is workspace-only. AccountClient always needs host (the accounts URL) and account_id.
pin the version
It's Beta and production-supported, but minor releases occasionally rename types. Pin databricks-sdk==x.y.z and skim the CHANGELOG before upgrading.
escape hatch: w.api_client.do()
For an endpoint the typed methods haven't caught up to yet, call the raw REST client directly with a method and path.