Apache Calcite Cosmos Adapter

Query Azure Cosmos DB with SQL, through Apache Calcite, from .NET.

Containers become relational tables. As much of each query as Cosmos can evaluate is translated to Cosmos SQL and executed by the service; whatever it cannot — relational joins, set operations, functions it has no counterpart for — Calcite evaluates in-process over the rows that come back. Calcite runs in-process via IKVM: no JVM, no JDBC, no second process.

Read the user manual →

Install

dotnet add package Apache.Calcite.Cosmos.Adapter
dotnet add package Apache.Calcite.Data

A first query

using Apache.Calcite.Data;

const string model = """
{
  "version": "1.0",
  "defaultSchema": "COSMOS",
  "schemas": [{
    "name": "COSMOS",
    "type": "custom",
    "factory": "Apache.Calcite.Cosmos.Adapter.CosmosSchemaFactory, Apache.Calcite.Cosmos.Adapter",
    "operand": {
      "endpoint": "https://myaccount.documents.azure.com:443/",
      "database": "inventory",
      "containers": [ "products" ]
    }
  }]
}
""";

_ = new Apache.Calcite.Cosmos.Adapter.CosmosSchemaFactory();   // load the assembly the model names

await using var connection = new CalciteConnection(new CalciteConnectionStringBuilder
{
    Model = "inline:" + model,
    CaseSensitive = true,
    DefaultNullCollation = "LOW",
}.ConnectionString);

await connection.OpenAsync();

await using var command = connection.CreateCommand();
command.CommandText = """
    SELECT c."id", JSON_VALUE(c."DOC", '$.name') AS "name"
    FROM "products" AS c
    WHERE c."$.category" = 'bikes'
    ORDER BY c."_ts" DESC FETCH NEXT 10 ROWS ONLY
    """;

await using var reader = await command.ExecuteReaderAsync();
while (await reader.ReadAsync())
    Console.WriteLine($"{reader.GetString(0)} {reader.GetString(1)}");

With no key, the adapter signs in with Microsoft Entra ID as whoever the process runs as. The whole query is answered by Cosmos, routed to one partition, as one statement.

Three things to get right from the start, each explained in the manual:

• Name the factory assembly-qualified, and make sure its assembly is loaded — Installation.
• Set DefaultNullCollation = "LOW", or most sorts run in process — Sorting and paging.
• Use ExecuteReaderAsync and ReadAsync — the Cosmos SDK has no synchronous reads — Connections.

What it does

• Pushdown of filters, projections, sorts, row limits, aggregation and array traversal, partially where a whole operator cannot go — Filtering, Sorting, Aggregation, Arrays.
• Partition routing and point reads, recovered from the predicate.
• Joins that fetch only the documents another source's keys can match, and joins of a container to itself answered with one read — Joins.
• Writes — INSERT, UPDATE and DELETE as item operations — Writing data.
• Declared facts: a JSON Schema and UNIQUE constraints that let UUIDs, timestamps and numbers stored as text be compared, sorted and routed at the service — JSON Schema, Dates and times, Uniqueness.
• Full text, vector and geodesic search — Full text and vector, Geography.
• Request-unit telemetry through System.Diagnostics — Monitoring.

It never infers a document's shape by sampling, and declines to push down anything Cosmos would answer differently from SQL — the query still runs, in process.

Documentation

• User manual — installation, configuration, querying, writing, performance, troubleshooting and reference
• DESIGN.md — why it works the way it does, and what the service was measured to do
• TODO.md — what is left
• Apache Calcite for .NET — the ADO.NET provider

Building

dotnet build Apache.Calcite.Cosmos.slnx

Part of the test suite runs against Cosmos DB, starting an emulator in Docker where none is running, or against a real account named by COSMOS_TEST_ENDPOINT and COSMOS_TEST_KEY — Appendix E.

License

Apache License 2.0.