Databricks submit jar
jar file to azure databricks cluster from local.
The idea is to push the fat jar to databrick’s dbfs using
Configure databricks CLI connection
Generate and copy PAT token in databricks workspace/User Settings. Then go:
databricks configure --token
Workspace url: https://adb-3772194496043630.10.azuredatabricks.net/
jar file to databricks workspace
First, pack to create the JAR
cd [ProjectRoot] sbt assembly
Then push the jar to Databricks. (Before, run
rm to delete jar from databricks if it exists already).
dbfs rm dbfs:/lib/jars/Arch.Pocs.Databricks.Application.jar dbfs cp ./target/scala-2.12/Arch.Pocs.Databricks.Application.jar dbfs:/lib/jars/Arch.Pocs.Databricks.Application.jar