Skip to content

Commit 6937d27

Browse files
nkvuongalexott
andauthored
[Doc] Add deprecation notice to databricks_dbfs_file and databricks_mount (#4876)
## Changes - Add deprecation notice to `databricks_dbfs_file` and `databricks_mount` - Resolves #4865 ## Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [x] relevant change in `docs/` folder - [x] has entry in `NEXT_CHANGELOG.md` file --------- Co-authored-by: Alex Ott <[email protected]>
1 parent 42c4698 commit 6937d27

File tree

11 files changed

+32
-61
lines changed

11 files changed

+32
-61
lines changed

NEXT_CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919
* Refreshed `databricks_job` documentation ([#4861](https://github.com/databricks/terraform-provider-databricks/pull/4861)).
2020
* Document `environment` block in `databricks_pipeline` ([#4878](https://github.com/databricks/terraform-provider-databricks/pull/4878)).
2121
* Updated documentation for `databricks_disable_legacy_dbfs_setting` resource ([#4870](https://github.com/databricks/terraform-provider-databricks/pull/4870)).
22+
* Add deprecation notice to `databricks_dbfs_file` and `databricks_mount` ([#4876](https://github.com/databricks/terraform-provider-databricks/pull/4876))
2223

2324
### Exporter
2425

docs/guides/workspace-management.md

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -201,10 +201,8 @@ resource "databricks_permissions" "pool" {
201201

202202
Depending on your preferences and needs, you can
203203

204-
* Manage JAR, Wheel & Egg libraries through [databricks_dbfs_file](../resources/dbfs_file.md).
205-
* List entries on DBFS with [databricks_dbfs_file_paths](../data-sources/dbfs_file_paths.md) data source.
206-
* Get contents of small files with [databricks_dbfs_file](../data-sources/dbfs_file.md) data source.
207-
* Mount your storage using [databricks_mount](../resources/mount.md) resource.
204+
* Create UC volumes using [databricks_volume](../resources/volume.md) resource.
205+
* Manage JAR & Wheel libraries through [databricks_workspace_file](../resources/workspace_file.md) or [databricks_file](../resources/file.md)
208206

209207
## Part 4: Advanced configuration
210208

docs/resources/cluster.md

Lines changed: 6 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -156,27 +156,27 @@ To install libraries, one must specify each library in a separate configuration
156156

157157
-> Please consider using [databricks_library](library.md) resource for a more flexible setup.
158158

159-
Installing JAR artifacts on a cluster. Location can be anything, that is DBFS or mounted object store (s3, adls, ...)
159+
Installing JAR artifacts on a cluster. Location can be a workspace file, Unity Catalog volume or cloud object storage location (s3, ADLS, ...)
160160

161161
```hcl
162162
library {
163-
jar = "dbfs:/FileStore/app-0.0.1.jar"
163+
jar = "/Volumes/catalog/schema/volume/app-0.0.1.jar"
164164
}
165165
```
166166

167-
Installing Python EGG artifacts. Location can be anything, that is DBFS or mounted object store (s3, adls, ...)
167+
Installing Python EGG artifacts (Deprecated)
168168

169169
```hcl
170170
library {
171171
egg = "dbfs:/FileStore/foo.egg"
172172
}
173173
```
174174

175-
Installing Python Wheel artifacts. Location can be anything, that is DBFS or mounted object store (s3, adls, ...)
175+
Installing Python Wheel artifacts. Location can be a workspace file, Unity Catalog volume or cloud object storage location (s3, ADLS, ...)
176176

177177
```hcl
178178
library {
179-
whl = "dbfs:/FileStore/baz.whl"
179+
whl = "/Volumes/catalog/schema/volume/baz.whl"
180180
}
181181
```
182182

@@ -223,16 +223,6 @@ library {
223223

224224
### cluster_log_conf
225225

226-
Example of pushing all cluster logs to DBFS:
227-
228-
```hcl
229-
cluster_log_conf {
230-
dbfs {
231-
destination = "dbfs:/cluster-logs"
232-
}
233-
}
234-
```
235-
236226
Example of pushing all cluster logs to S3:
237227

238228
```hcl
@@ -268,7 +258,7 @@ There are a few more advanced attributes for S3 log delivery:
268258

269259
To run a particular init script on all clusters within the same workspace, both automated/job and interactive/all-purpose cluster types, please consider the [databricks_global_init_script](global_init_script.md) resource.
270260

271-
It is possible to specify up to 10 different cluster-scoped init scripts per cluster. Init scripts support DBFS, cloud storage locations, and workspace files.
261+
It is possible to specify up to 10 different cluster-scoped init scripts per cluster. Init scripts support volumes, cloud storage locations, and workspace files.
272262

273263
Example of using a Databricks workspace file as init script:
274264

@@ -290,16 +280,6 @@ init_scripts {
290280
}
291281
```
292282

293-
Example of taking init script from DBFS (deprecated):
294-
295-
```hcl
296-
init_scripts {
297-
dbfs {
298-
destination = "dbfs:/init-scripts/install-elk.sh"
299-
}
300-
}
301-
```
302-
303283
Example of taking init script from S3:
304284

305285
```hcl
@@ -573,7 +553,6 @@ The following resources are often used in the same context:
573553
* [databricks_instance_profile](instance_profile.md) to manage AWS EC2 instance profiles that users can launch [databricks_cluster](cluster.md) and access data, like [databricks_mount](mount.md).
574554
* [databricks_job](job.md) to manage [Databricks Jobs](https://docs.databricks.com/jobs.html) to run non-interactive code in a [databricks_cluster](cluster.md).
575555
* [databricks_library](library.md) to install a [library](https://docs.databricks.com/libraries/index.html) on [databricks_cluster](cluster.md).
576-
* [databricks_mount](mount.md) to [mount your cloud storage](https://docs.databricks.com/data/databricks-file-system.html#mount-object-storage-to-dbfs) on `dbfs:/mnt/name`.
577556
* [databricks_node_type](../data-sources/node_type.md) data to get the smallest node type for [databricks_cluster](cluster.md) that fits search criteria, like amount of RAM or number of cores.
578557
* [databricks_pipeline](pipeline.md) to deploy [Lakeflow Declarative Pipelines](https://docs.databricks.com/aws/en/dlt).
579558
* [databricks_spark_version](../data-sources/spark_version.md) data to get [Databricks Runtime (DBR)](https://docs.databricks.com/runtime/dbr.html) version that could be used for `spark_version` parameter in [databricks_cluster](cluster.md) and other resources.

docs/resources/dbfs_file.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@ subcategory: "Storage"
33
---
44
# databricks_dbfs_file Resource
55

6+
-> Please switch to [databricks_file](file.md) or [databricks_workspace_file](workspace_file.md) to manage files. Databricks recommends against storing any production data or sensitive information in the DBFS root.
7+
68
This is a resource that lets you manage relatively small files on [Databricks File System (DBFS)](https://docs.databricks.com/data/databricks-file-system.html). The best use cases are libraries for [databricks_cluster](cluster.md) or [databricks_job](job.md). You can also use [databricks_dbfs_file](../data-sources/dbfs_file.md) and [databricks_dbfs_file_paths](../data-sources/dbfs_file_paths.md) data sources.
79

810
-> This resource can only be used with a workspace-level provider!

docs/resources/global_init_script.md

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,5 @@ The following resources are often used in the same context:
7777
* [End to end workspace management](../guides/workspace-management.md) guide.
7878
* [databricks_cluster](cluster.md) to create [Databricks Clusters](https://docs.databricks.com/clusters/index.html).
7979
* [databricks_cluster_policy](cluster_policy.md) to create a [databricks_cluster](cluster.md) policy, which limits the ability to create clusters based on a set of rules.
80-
* [databricks_dbfs_file](dbfs_file.md) to manage relatively small files on [Databricks File System (DBFS)](https://docs.databricks.com/data/databricks-file-system.html).
80+
* [databricks_workspace_file](workspace_file.md) to manage small files in Databricks workspace
8181
* [databricks_ip_access_list](ip_access_list.md) to allow access from [predefined IP ranges](https://docs.databricks.com/security/network/ip-access-list.html).
82-
* [databricks_mount](mount.md) to [mount your cloud storage](https://docs.databricks.com/data/databricks-file-system.html#mount-object-storage-to-dbfs) on `dbfs:/mnt/name`.

docs/resources/job.md

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -258,7 +258,7 @@ The `power_bi_task` triggers a Power BI semantic model update.
258258

259259
#### spark_python_task Configuration Block
260260

261-
* `python_file` - (Required) The URI of the Python file to be executed. [databricks_dbfs_file](dbfs_file.md#path), cloud file URIs (e.g. `s3:/`, `abfss:/`, `gs:/`), workspace paths and remote repository are supported. For Python files stored in the Databricks workspace, the path must be absolute and begin with `/`. For files stored in a remote repository, the path must be relative. This field is required.
261+
* `python_file` - (Required) The URI of the Python file to be executed. Cloud file URIs (e.g. `s3:/`, `abfss:/`, `gs:/`), workspace paths and remote repository are supported. For Python files stored in the Databricks workspace, the path must be absolute and begin with `/`. For files stored in a remote repository, the path must be relative. This field is required.
262262
* `source` - (Optional) Location type of the Python file. When set to `WORKSPACE` or not specified, the file will be retrieved from the local Databricks workspace or cloud location (if the python_file has a URI format). When set to `GIT`, the Python file will be retrieved from a Git repository defined in `git_source`.
263263
* `WORKSPACE`: The Python file is located in a Databricks workspace or at a cloud filesystem URI.
264264
* `GIT`: The Python file is located in a remote Git repository.
@@ -632,12 +632,8 @@ The following resources are often used in the same context:
632632
* [databricks_cluster](cluster.md) to create [Databricks Clusters](https://docs.databricks.com/clusters/index.html).
633633
* [databricks_cluster_policy](cluster_policy.md) to create a [databricks_cluster](cluster.md) policy, which limits the ability to create clusters based on a set of rules.
634634
* [databricks_current_user](../data-sources/current_user.md) data to retrieve information about [databricks_user](user.md) or [databricks_service_principal](service_principal.md), that is calling Databricks REST API.
635-
* [databricks_dbfs_file](../data-sources/dbfs_file.md) data to get file content from [Databricks File System (DBFS)](https://docs.databricks.com/data/databricks-file-system.html).
636-
* [databricks_dbfs_file_paths](../data-sources/dbfs_file_paths.md) data to get list of file names from get file content from [Databricks File System (DBFS)](https://docs.databricks.com/data/databricks-file-system.html).
637-
* [databricks_dbfs_file](dbfs_file.md) to manage relatively small files on [Databricks File System (DBFS)](https://docs.databricks.com/data/databricks-file-system.html).
638635
* [databricks_global_init_script](global_init_script.md) to manage [global init scripts](https://docs.databricks.com/clusters/init-scripts.html#global-init-scripts), which are run on all [databricks_cluster](cluster.md#init_scripts) and [databricks_job](job.md#new_cluster).
639636
* [databricks_instance_pool](instance_pool.md) to manage [instance pools](https://docs.databricks.com/clusters/instance-pools/index.html) to reduce [cluster](cluster.md) start and auto-scaling times by maintaining a set of idle, ready-to-use instances.
640-
* [databricks_instance_profile](instance_profile.md) to manage AWS EC2 instance profiles that users can launch [databricks_cluster](cluster.md) and access data, like [databricks_mount](mount.md).
641637
* [databricks_jobs](../data-sources/jobs.md) data to get all jobs and their names from a workspace.
642638
* [databricks_library](library.md) to install a [library](https://docs.databricks.com/libraries/index.html) on [databricks_cluster](cluster.md).
643639
* [databricks_node_type](../data-sources/node_type.md) data to get the smallest node type for [databricks_cluster](cluster.md) that fits search criteria, like amount of RAM or number of cores.

docs/resources/library.md

Lines changed: 8 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ Installs a [library](https://docs.databricks.com/libraries/index.html) on [datab
1010
-> `databricks_library` resource would always start the associated cluster if it's not running, so make sure to have auto-termination configured. It's not possible to atomically change the version of the same library without cluster restart. Libraries are fully removed from the cluster only after restart.
1111

1212
## Plugin Framework Migration
13+
1314
The library resource has been migrated from sdkv2 to plugin framework。 If you encounter any problem with this resource and suspect it is due to the migration, you can fallback to sdkv2 by setting the environment variable in the following way `export USE_SDK_V2_RESOURCES="databricks_library"`.
1415

1516
## Installing library on all clusters
@@ -32,14 +33,14 @@ resource "databricks_library" "cli" {
3233
## Java/Scala JAR
3334

3435
```hcl
35-
resource "databricks_dbfs_file" "app" {
36+
resource "databricks_file" "app" {
3637
source = "${path.module}/app-0.0.1.jar"
37-
path = "/FileStore/app-0.0.1.jar"
38+
path = "/Volumes/catalog/schema/volume/app-0.0.1.jar"
3839
}
3940
4041
resource "databricks_library" "app" {
4142
cluster_id = databricks_cluster.this.id
42-
jar = databricks_dbfs_file.app.dbfs_path
43+
jar = databricks_file.app.path
4344
}
4445
```
4546

@@ -61,14 +62,14 @@ resource "databricks_library" "deequ" {
6162
## Python Wheel
6263

6364
```hcl
64-
resource "databricks_dbfs_file" "app" {
65+
resource "databricks_file" "app" {
6566
source = "${path.module}/baz.whl"
66-
path = "/FileStore/baz.whl"
67+
path = "/Volumes/catalog/schema/volume/baz.whl"
6768
}
6869
6970
resource "databricks_library" "app" {
7071
cluster_id = databricks_cluster.this.id
71-
whl = databricks_dbfs_file.app.dbfs_path
72+
whl = databricks_file.app.path
7273
}
7374
```
7475

@@ -92,16 +93,14 @@ resource "databricks_library" "fbprophet" {
9293

9394
Installing Python libraries listed in the `requirements.txt` file. Only Workspace paths and Unity Catalog Volumes paths are supported. Requires a cluster with DBR 15.0+.
9495

95-
9696
```hcl
9797
resource "databricks_library" "libraries" {
9898
cluster_id = databricks_cluster.this.id
9999
requirements = "/Workspace/path/to/requirements.txt"
100100
}
101101
```
102102

103-
104-
## Python EGG
103+
## Python EGG (Deprecated)
105104

106105
```hcl
107106
resource "databricks_dbfs_file" "app" {
@@ -128,7 +127,6 @@ resource "databricks_library" "rkeops" {
128127
}
129128
```
130129

131-
132130
## Import
133131

134132
!> Importing this resource is not currently supported.
@@ -141,11 +139,7 @@ The following resources are often used in the same context:
141139
* [databricks_clusters](../data-sources/clusters.md) data to retrieve a list of [databricks_cluster](cluster.md) ids.
142140
* [databricks_cluster](cluster.md) to create [Databricks Clusters](https://docs.databricks.com/clusters/index.html).
143141
* [databricks_cluster_policy](cluster_policy.md) to create a [databricks_cluster](cluster.md) policy, which limits the ability to create clusters based on a set of rules.
144-
* [databricks_dbfs_file](../data-sources/dbfs_file.md) data to get file content from [Databricks File System (DBFS)](https://docs.databricks.com/data/databricks-file-system.html).
145-
* [databricks_dbfs_file_paths](../data-sources/dbfs_file_paths.md) data to get list of file names from get file content from [Databricks File System (DBFS)](https://docs.databricks.com/data/databricks-file-system.html).
146-
* [databricks_dbfs_file](dbfs_file.md) to manage relatively small files on [Databricks File System (DBFS)](https://docs.databricks.com/data/databricks-file-system.html).
147142
* [databricks_global_init_script](global_init_script.md) to manage [global init scripts](https://docs.databricks.com/clusters/init-scripts.html#global-init-scripts), which are run on all [databricks_cluster](cluster.md#init_scripts) and [databricks_job](job.md#new_cluster).
148143
* [databricks_job](job.md) to manage [Databricks Jobs](https://docs.databricks.com/jobs.html) to run non-interactive code in a [databricks_cluster](cluster.md).
149-
* [databricks_mount](mount.md) to [mount your cloud storage](https://docs.databricks.com/data/databricks-file-system.html#mount-object-storage-to-dbfs) on `dbfs:/mnt/name`.
150144
* [databricks_pipeline](pipeline.md) to deploy [Lakeflow Declarative Pipelines](https://docs.databricks.com/aws/en/dlt).
151145
* [databricks_repo](repo.md) to manage [Databricks Repos](https://docs.databricks.com/repos.html).

docs/resources/mlflow_experiment.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ data "databricks_current_user" "me" {}
1414
1515
resource "databricks_mlflow_experiment" "this" {
1616
name = "${data.databricks_current_user.me.home}/Sample"
17-
artifact_location = "dbfs:/tmp/my-experiment"
17+
artifact_location = "s3://bucket/my-experiment"
1818
1919
tags {
2020
key = "key1"
@@ -23,7 +23,7 @@ resource "databricks_mlflow_experiment" "this" {
2323
tags {
2424
key = "key2"
2525
value = "value2"
26-
}
26+
}
2727
}
2828
```
2929

@@ -32,7 +32,7 @@ resource "databricks_mlflow_experiment" "this" {
3232
The following arguments are supported:
3333

3434
* `name` - (Required) Name of MLflow experiment. It must be an absolute path within the Databricks workspace, e.g. `/Users/<some-username>/my-experiment`. For more information about changes to experiment naming conventions, see [mlflow docs](https://docs.databricks.com/applications/mlflow/experiments.html#experiment-migration).
35-
* `artifact_location` - Path to dbfs:/ or s3:// artifact location of the MLflow experiment.
35+
* `artifact_location` - Path to artifact location of the MLflow experiment.
3636
* `tags` - Tags for the MLflow experiment.
3737

3838
## Attribute Reference

docs/resources/mount.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,14 +3,16 @@ subcategory: "Storage"
33
---
44
# databricks_mount Resource
55

6-
This resource will [mount your cloud storage](https://docs.databricks.com/data/databricks-file-system.html#mount-object-storage-to-dbfs) on `dbfs:/mnt/name`. Right now it supports mounting AWS S3, Azure (Blob Storage, ADLS Gen1 & Gen2), Google Cloud Storage. It is important to understand that this will start up the [cluster](cluster.md) if the cluster is terminated. The read and refresh terraform command will require a cluster and may take some time to validate the mount.
6+
-> Please switch to [databricks_volume](volume.md). DBFS mounts are deprecated.
77

88
-> This resource can only be used with a workspace-level provider!
99

1010
-> When `cluster_id` is not specified, it will create the smallest possible cluster in the default availability zone with name equal to or starting with `terraform-mount` for the shortest possible amount of time. To avoid mount failure due to potentially quota or capacity issues with the default cluster, we recommend specifying a cluster to use for mounting.
1111

1212
-> CRUD operations on a databricks mount require a running cluster. Due to limitations of terraform and the databricks mounts APIs, if the cluster the mount was most recently created / updated using no longer exists AND the mount is destroyed as a part of a terraform apply, we mark it as deleted without cleaning it up from the workspace.
1313

14+
This resource will [mount your cloud storage](https://docs.databricks.com/data/databricks-file-system.html#mount-object-storage-to-dbfs) on `dbfs:/mnt/name`. Right now it supports mounting AWS S3, Azure (Blob Storage, ADLS Gen1 & Gen2), Google Cloud Storage. It is important to understand that this will start up the [cluster](cluster.md) if the cluster is terminated. The read and refresh terraform command will require a cluster and may take some time to validate the mount.
15+
1416
This resource provides two ways of mounting a storage account:
1517

1618
1. Use a storage-specific configuration block - this could be used for the most cases, as it will fill most of the necessary details. Currently we support following configuration blocks:

docs/resources/permissions.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -525,7 +525,7 @@ data "databricks_current_user" "me" {}
525525
526526
resource "databricks_mlflow_experiment" "this" {
527527
name = "${data.databricks_current_user.me.home}/Sample"
528-
artifact_location = "dbfs:/tmp/my-experiment"
528+
artifact_location = "s3://bucket/my-experiment"
529529
description = "My MLflow experiment description"
530530
}
531531
@@ -863,7 +863,7 @@ resource "databricks_permissions" "query_usage" {
863863

864864
## SQL Alert (AlertV2) usage
865865

866-
[Alert V2](https://docs.databricks.com/sql/user/security/access-control/alert-acl.html) which is the new version of SQL Alert have 4 possible permission levels: `CAN_READ`, `CAN_RUN`, `CAN_EDIT`, and `CAN_MANAGE`.
866+
[Alert V2](https://docs.databricks.com/sql/user/security/access-control/alert-acl.html) which is the new version of SQL Alert have 4 possible permission levels: `CAN_READ`, `CAN_RUN`, `CAN_EDIT`, and `CAN_MANAGE`.
867867

868868
```hcl
869869
resource "databricks_group" "auto" {
@@ -889,7 +889,7 @@ resource "databricks_permissions" "app_usage" {
889889
}
890890
```
891891

892-
## SQL Alert (legacy) usage
892+
## SQL Alert (legacy) usage
893893

894894
[SQL alerts](https://docs.databricks.com/sql/user/security/access-control/alert-acl.html) have three possible permissions: `CAN_VIEW`, `CAN_RUN` and `CAN_MANAGE`:
895895

@@ -1081,4 +1081,4 @@ resource "databricks_permissions" "model_usage" {
10811081
Import command:
10821082
```bash
10831083
terraform import databricks_permissions.model_usage /registered-models/<registered_model_id>
1084-
```
1084+
```

0 commit comments

Comments
 (0)