New features for 2.4.0 in sqllin-dsl

CHANGELOG.md

@@ -7,12 +7,20 @@
 ### All
 
 * Update `Kotlin`'s version to `2.4.20`
-* Update `AGP`'s version to `9.4.0`
+* Update `AGP`'s version to `9.4.1`
 * Migrate the Android instrumented tests to `Robolectric`, they now run on the JVM as host unit tests against `API 26` and `API 37`, and no longer need an emulator
 * Move the `sqllin-driver` tests back into the `sqllin-driver` module's `commonTest`, and remove the `sqllin-driver-test` module
 
 ### sqllin-dsl
 
+* New DSL API: `DatabaseScope#INSERT_OR_IGNORE` for SQL syntax `INSERT OR IGNORE`
+* New DSL API: projection, which reads `SELECT` results into a narrower `@Serializable` type naming the columns to select, given as the type argument of the clause function: `X<R>()`, `WHERE<R>(...)`, `ORDER_BY<R>(...)`, `LIMIT<R>(...)` and `GROUP_BY<R>(...)`, after `SELECT` or `SELECT_DISTINCT`. A type that doesn't fit the table is rejected with an `IllegalArgumentException` when the statement is built. To support it, the public `DatabaseScope#select` functions now take a result type separate from the table's; this is source-compatible, but on Kotlin/Native a library compiled against an earlier version may have to be recompiled
+* New DSL API: result columns, which select expressions such as aggregate functions into properties of a result type with `AS`, as in `table SELECT listOf(count(X) AS AuthorStats::books, sum(pages) AS AuthorStats::totalPages) GROUP_BY author`, or `table SELECT (count(X) AS BookCount::books)` for a single one. The result type's other properties are read from their columns, as in a projection. They work after `SELECT` and `SELECT_DISTINCT`, followed by `WHERE`, `GROUP_BY`, `ORDER_BY` and `LIMIT`. A property must have the type of its expression's values, which is checked at compile time, and be nullable when its expression can be `NULL`, which is checked when the statement is built. An aggregate query without `GROUP BY` returns a row even when no rows match, in which every column and every aggregate function except `count` is `NULL`; as `GROUP_BY` can still follow when the statement is built, this is checked when the scope ends, before any of its statements runs
+* New DSL API: `INSERT`, `INSERT_OR_IGNORE` and `INSERT_OR_REPLACE` taking a `SelectStatement` of the table's row type, for SQL syntax `INSERT INTO ... SELECT`, as in `ArchiveTable INSERT (PersonTable SELECT WHERE<Archive>(PersonTable.age GT 60))`. Every column is inserted, the primary key copied as it is selected. The `SELECT` becomes part of the `INSERT`, so it no longer runs on its own
+* New API: `Table#withName`, which returns a table with the same structure under another name. With `INSERT INTO ... SELECT`, it rebuilds a table in a migration, for a change `ALTER TABLE` can't make, such as adding a constraint, or dropping a column on SQLite older than 3.35, which on Android means below API 34. The guide to modifying the database now describes the procedure, and documents `INSERT_OR_IGNORE` and `INSERT_OR_REPLACE`
+* **Breaking change**: `ClauseElement`, `ClauseNumber` and `ClauseString` now take a type parameter, the type of their values, such as `ClauseNumber<Int>` for an `Int` column and `ClauseNumber<Long>` for `count(X)`, which is what lets `AS` check the type of a property. Code that only uses the DSL is unaffected; code that names these types has to add a type argument, such as `ClauseElement<*>` where any element is accepted. The public constructors of `ClauseNumber`, `ClauseString`, `ClauseBoolean`, `ClauseBlob` and `ClauseEnum`, which the generated table objects call, now take whether the column is nullable instead of whether the element is a function. The generated code is regenerated by the build, but on Kotlin/Native a library compiled against an earlier version has to be recompiled
+* **Breaking change**: The SQL functions now return elements of the type of the values SQLite returns for them: `count`, `length`, `instr` and `random` a `ClauseNumber<Long>`, `avg` and `round` a `ClauseNumber<Double>`, the string functions and `group_concat` a `ClauseString<String>`, and `max`, `min` and `abs` an element of the same kind and type as their argument. So `max` and `min` of a String column are now a `ClauseString`, compared with strings in `HAVING`, where they used to be a `ClauseNumber`. `sum` is overloaded by the type of its column: of a column of integers or Booleans it is a `ClauseNumber<Long>`, and of a `Float` or `Double` column a `ClauseNumber<Double>`. A `sum` of a String, BLOB, enum or `ULong` column no longer compiles; the last because SQLite stores a `ULong` above `Long.MAX_VALUE` as a negative number, which made the sum wrong
+* Fix documentation: the SQL functions guide listed a `sign` function, which isn't available, and its `HAVING (count(X) > 2)` example didn't compile; it is `HAVING (count(X) GT 2)`. It no longer says that functions can only be used in conditions
 * **Breaking change**: The parameter of annotation `@PrimaryKey` renamed from `isAutoincrement` to `autoIncrement`, aligning it with the name already used in the documentation and with the naming of the other annotations. Call sites using the named argument `@PrimaryKey(isAutoincrement = true)` must be updated to `@PrimaryKey(autoIncrement = true)`; positional usage such as `@PrimaryKey(true)` is unaffected
 * **Breaking change**: The nullability of a `@PrimaryKey` property now decides who supplies its value. A `Long?` key is assigned by the database, as before. A non-null `Long` key is now allowed: it remains an `INTEGER PRIMARY KEY`, a rowid alias, but is supplied by the caller and written by every `INSERT`. A key of any other type must be non-null and is declared `NOT NULL`. Previously every `@PrimaryKey` was forced to be nullable, against the annotation's own documentation, and because SQLite does not let `PRIMARY KEY` imply `NOT NULL` on such a column, a `String` key could hold `NULL` in any number of rows. `autoIncrement = true` now requires a `Long?` key, and a `ULong?` key, which used to be stored as `NULL` because it was left out of `INSERT` without being a rowid alias, is now rejected. To migrate, drop the `?` from any non-`Long` `@PrimaryKey`; a single-column `@CompositePrimaryKey` that only existed to hold a caller-supplied `Long` key can become `@PrimaryKey val id: Long`
 * **Breaking change**: `@CompositePrimaryKey` now requires at least two properties. A single-column primary key is declared with `@PrimaryKey`, which for a `Long` key maps to `INTEGER`, a rowid alias, where a single-column `@CompositePrimaryKey` mapped it to `BIGINT`. To migrate, replace a lone `@CompositePrimaryKey` with `@PrimaryKey`
@@ -22,13 +30,16 @@
 * Fix documentation: the `CREATE_INDEX` and `CREATE_UNIQUE_INDEX` examples referenced a `KClass.table` extension that does not exist in the library, and referred to columns by property reference instead of through the generated table object
 * Fix: the SQL string functions added in 2.2.0, `substr`, `trim`, `ltrim`, `rtrim`, `replace`, `instr` and `printf`, now carry the same DSL marker as the other SQL functions, so IntelliJ IDEA highlights their calls the same way
 * Fix documentation: the installation guide now declares the task dependencies the generated code needs, as SQLlin's own builds always did. Without them Gradle fails the build, in particular when another KSP processor runs in the same module. It also states that each generated object is named after its class with a `Table` suffix, not after the table
+* Fix: the string arguments of `replace`, `instr`, `printf` and `group_concat` were put into the SQL between single quotes without escaping, so a `'` in one broke the statement, and a crafted one could change what the statement does, such as making a condition true for every row. A `'` is now escaped as `''`, the only escape SQLite has in a string literal, so any string stays a literal
+* Fix: a comparison between two elements, such as `length(name) GT pages` or `HAVING (max(pages) GT min(pages))`, qualified both with their table's name even when one was a function, producing invalid SQL such as `book.length(name)`. A function is now written as it is
 
 ### sqllin-driver
 
 * Update `sqlite-jdbc`'s version to `3.53.4.0`
 
 ### sqllin-processor
 
+* Update `KSP`'s version to `2.3.12`
 * Fix: the visibility of the class annotated with `@DBRow` is now propagated to the generated table object. An `internal` `@DBRow` class used to produce a `public` object, which failed to compile with `EXPOSED_SUPER_CLASS`, `EXPOSED_FUNCTION_RETURN_TYPE` and `EXPOSED_RECEIVER_TYPE`. A `@DBRow` class that is neither `public` nor `internal` is now reported as an error
 * Fix: every `SetClause` property generated for a column declared after a nullable `Long` `@PrimaryKey` was typed nullable regardless of the column's own declaration, so `UPDATE ... SET { column = null }` compiled against `NOT NULL` columns and failed only at runtime. Each property now takes the nullability its own column declares
 * Fix: the columns of a `@CompositePrimaryKey` are now declared `NOT NULL`. SQLite, unlike standard SQL, does not let a table-level `PRIMARY KEY` imply it on a rowid table, so such a key used to accept `NULL`, and any number of rows sharing the same key once a `NULL` was part of it. SQLlin itself could not write those `NULL`s, but anything else writing to the database could, and SQLlin then read them back as `0` or an empty string. This only changes the schema of tables created from now on; an existing table keeps its schema, as SQLite cannot add `NOT NULL` to an existing column

ROADMAP.md

@@ -1,21 +1,32 @@
 # SQLlin Roadmap
 
+## High Priority
+
+* Support observable queries that return a `Flow` re-emitting whenever the tables they read change, e.g. to invalidate a Paging source
+* Support UNION of SELECTs whose result type isn't the table's row type, such as projections, result columns and joins. `UNION` is typed by the table's row type but reads the rows into its first SELECT's result type, so using the results of such a union fails with a `ClassCastException`
+
 ## Medium Priority
 
 * Support WASM platform DSL
 * Support CREATE VIRTUAL TABLE DSL
 * Support CREATE VIEW DSL
 * Support CREATE TRIGGER DSL
 * Support JOIN sub-query DSL
-* Support more functions
+* Support more functions, such as `coalesce` and `ifnull`, as well as `CAST` and literal values in expressions, e.g. to convert data while copying it to a rebuilt table with `INSERT INTO ... SELECT`
+* Support type converters: store a property of any type through a serializer that encodes it to a type SQLite supports, with type-safe WHERE and SET on its column, e.g. to store instances of kotlinx.datetime
 
 ## Low Priority
 
-* Support store instances of kotlinx.datetime
 * Support CHECK keyword
+* Support using a query's results within the same transaction, so that a read-modify-write is one transaction
+* Support upsert, `INSERT ... ON CONFLICT (target) DO UPDATE` and `DO NOTHING`, which updates the conflicting row in place where `INSERT OR REPLACE` deletes and re-inserts it
 
 ## Supported
 
+* Support INSERT INTO ... SELECT (2.4.0 ✅)
+* Support SQL functions in SELECT results (2.4.0 ✅)
+* Support SELECT projection (2.4.0 ✅)
+* Support INSERT OR IGNORE (2.4.0 ✅)
 * Support INSERT OR REPLACE (2.3.0 ✅)
 * Support FOREIGN KEY DSL (2.2.0 ✅)
 * Support CREATE INDEX DSL (2.2.0 ✅)

gradle/libs.versions.toml

@@ -1,11 +1,11 @@
 [versions]
 
 kotlin = "2.4.20"
-agp = "9.4.0"
-ksp = "2.3.11"
+agp = "9.4.1"
+ksp = "2.3.12"
 serialization = "1.11.0"
 coroutines = "1.11.0"
-androidx-annotation = "1.10.0"
+androidx-annotation = "1.11.0"
 androidx-test = "1.7.0"
 robolectric = "4.17"
 junit = "4.13.2"

sqllin-dsl-test/src/androidHostTest/kotlin/com/ctrip/sqllin/dsl/test/AndroidTest.kt

@@ -72,6 +72,24 @@ class AndroidTest {
     @Test
     fun testInsertOrReplace() = commonTest.testInsertOrReplace()
 
+    @Test
+    fun testInsertOrIgnore() = commonTest.testInsertOrIgnore()
+
+    @Test
+    fun testProjection() = commonTest.testProjection()
+
+    @Test
+    fun testResultColumns() = commonTest.testResultColumns()
+
+    @Test
+    fun testResultColumnChecks() = commonTest.testResultColumnChecks()
+
+    @Test
+    fun testInsertSelect() = commonTest.testInsertSelect()
+
+    @Test
+    fun testTableRebuild() = commonTest.testTableRebuild()
+
     @Test
     fun testCreateInDatabaseScope() = commonTest.testCreateInDatabaseScope()
 
@@ -120,6 +138,12 @@ class AndroidTest {
     @Test
     fun testStringAggregateFunctions() = commonTest.testStringAggregateFunctions()
 
+    @Test
+    fun testFunctionStringArguments() = commonTest.testFunctionStringArguments()
+
+    @Test
+    fun testFunctionComparisons() = commonTest.testFunctionComparisons()
+
     @Test
     fun testIndexOperations() = commonTest.testIndexOperations()