Repository navigation
Stop appending UseNativeQuery for the Databricks JDBC 3.4.2 driver - #343
developer-rpai wants to merge 3 commits into
Conversation
Document the driver version, stop appending UseNativeQuery, and default Spark ODBC connections to the Databricks ODBC driver while still allowing the legacy driver name. Co-authored-by: Rakesh Pai <developer-rpai@users.noreply.github.com>
|
Some thoughts on this:
|
|
Thanks for the thoughts. On the driver version: agreed, syncing with WebAPI makes sense, I will check the Databricks JDBC version WebAPI currently ships and align this PR to it. On propagating user identity instead of requiring a service principal: I agree that is the better direction long term. It is a bigger change than the driver swap itself, and with the Simba driver going away on Oct 31 I would rather land the swap first and track identity propagation as a follow-up, so this PR does not block on it. Happy to be corrected if you see it differently. |
| 3,sql server,sqlServerV9.2.0.zip,https://ohdsi.github.io/DatabaseConnectorJars/ | ||
| 4,oracle,oracleV19.8.zip,https://ohdsi.github.io/DatabaseConnectorJars/ | ||
| 5,spark,DatabricksJDBC-3.4.2.zip,https://databricks-bi-artifacts.s3.us-east-2.amazonaws.com/simbaspark-drivers/jdbc/3.4.2/ | ||
| 5,spark,databricks-jdbc-2.6.40.jar,https://repo1.maven.org/maven2/com/databricks/databricks-jdbc/2.6.40/ |
There was a problem hiding this comment.
As far as I can tell we already are on 3.4.2, which (despite the URL) is not a Simba model. Did I get that wrong? If so, why is this code changing away from 3.4.2?
There was a problem hiding this comment.
Good question — you're right that the jar currently listed is the 3.4.2 build; the reason it caught my eye is that it's served from the simbaspark-drivers S3 path, so I read it as part of the Simba distribution that goes away. The move to 2.6.40 in this PR wasn't meant as a downgrade: konstjar suggested syncing the version with WebAPI, and WebAPI's Databricks profile pins databricks-jdbc 2.6.40 from Maven, so I aligned to that. If you'd rather stay on 3.4.2, I'm happy to keep 3.4.2 and just re-point/re-host it instead — which do you prefer?
There was a problem hiding this comment.
Quick correction to my earlier reply, I had the versions backwards. 3.4.2 is the open source Databricks driver and 2.6.40 is the older Simba based one, so moving to 2.6.40 was a step back. I just pushed a commit that keeps the download on the existing 3.4.2 build and drops the ODBC changes from this PR, so the ODBC path is the same as on main. What's left is removing the UseNativeQuery append (a Simba only property) and the docs, which now list the driver as V3.4.2.
| connectionString | ||
| } | ||
|
|
||
| resolveSparkOdbcDriver <- function(extraSettings) { |
There was a problem hiding this comment.
I don't think we should actively support ODBC drivers. We either use JDBC, or at some point may transition to some native R driver package.
There was a problem hiding this comment.
That makes sense — I'll drop the ODBC support from this PR and keep it JDBC-only. I added the ODBC resolution while wiring up the Spark connection path, but I agree we shouldn't actively support it if the direction is JDBC now and possibly a native R driver later. I'll push a commit removing resolveSparkOdbcDriver and its call sites.
Revert the downloadJdbcDrivers() spark entry to the existing DatabricksJDBC 3.4.2 build and document it as V3.4.2. Remove the ODBC driver name default and override added in this PR, so the existing ODBC path is unchanged.
developer-rpai
changed the title
Closes #331.
The Spark connection path still pointed at the Simba JDBC driver. This switches it to the Databricks JDBC driver (3.4.2):
R/Connect.RandR/Drivers.R: driver references and download details updated from Simba to Databricks.DESCRIPTION: JDBC driver version bumped accordingly.NEWS.md: entry for the driver switch.tests/testthat/test-sparkDriver.Rcovering the Spark driver selection.One commit, no other connection backends touched.