New features for 2.4.0 in sqllin-dsl - #127
Merged
Merged
Conversation
INSERT_OR_REPLACE resolves a PRIMARY KEY or UNIQUE conflict by deleting the existing row and inserting the new one, which replaces the existing row's other columns. When the existing row should win instead, as when a paged list fetches an item it already holds and must keep that item's position, there was no way to say so. INSERT_OR_IGNORE writes INSERT OR IGNORE INTO: each entity that conflicts with an existing row is skipped and that row is left exactly as it is, while the other entities are inserted. Like INSERT_OR_REPLACE it always writes the primary key column, as a conflict on a key left out of the statement could never be seen. A null key that the database assigns still can't conflict. As SQLite documents, and as checked here, OR IGNORE also skips a row that would violate NOT NULL, which can't happen for a non-null property, while a FOREIGN KEY violation still fails the statement. The KDoc says so. The test covers a conflict on the primary key, on another UNIQUE column and on a composite key, and it fails if the primary key isn't written. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A SELECT always read rows into the table's own row type, so reading a few
columns meant fetching and decoding all of them. The column list was already
built from the deserializer's descriptor, and JOIN already took a result type of
its own, but a single-table SELECT tied its result type to the table's.
The result type is now given to the clause function, as with JOIN:
PersonTable SELECT X<NameAndAge>()
PersonTable SELECT WHERE<NameAndAge>(cond) ORDER_BY PersonTable.age LIMIT 10
for X, WHERE, ORDER BY, LIMIT and GROUP BY, after both SELECT and
SELECT_DISTINCT, and the chained clauses keep it. Only the columns named by the
type's properties are selected. Without a type argument a SELECT reads the
table's row type exactly as before, and every existing call resolves as it did.
A projection type has to fit the table: each property must be a column, of that
column's type, and nullable if the column is, as a NULL read into a non-null
property would quietly become 0 or "". A mismatch throws an
IllegalArgumentException while the statement is built.
Two other shapes were ruled out by compiling them. An overload of SELECT(X) that
was generic only in its return type made every existing `SELECT X` ambiguous, so
the no-clause form is a function, X<R>(), declared next to the object X. And the
projection can't be inferred from the type the statement is assigned to, as the
existing overload is the more specific one, so the type argument is required.
The public DatabaseScope.select functions now take a result type separate from
the table's. That is source-compatible and keeps their JVM signatures, but on
Kotlin/Native a library compiled against an earlier version may need to be
recompiled.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A SELECT could only read columns: the column list came from the result type's
property names, and the SQL functions returned elements usable only in
conditions, with no Kotlin type and no way to name a result. So `count(*)` or a
per-group `sum` could not be selected at all.
An expression is now selected into a property of the result type with AS, and
the type's other properties are read from their columns, as in a projection:
table SELECT listOf(count(X) AS AuthorStats::books, sum(pages) AS AuthorStats::totalPages) GROUP_BY author
table SELECT (count(X) AS BookCount::books) WHERE (price LT 20.0)
It follows the existing convention of a single argument or a Kotlin collection,
as INSERT and GROUP_BY do, rather than adding a function that would look like a
SQL keyword without being one. It works after SELECT and SELECT_DISTINCT, and
is followed by WHERE, GROUP BY, ORDER BY and LIMIT, through a new
ResultColumnSelectStatement.
To check the type of a property at compile time, ClauseElement, ClauseNumber
and ClauseString take a type parameter, the type of their values. The generated
accessors give each column its property's type, and each function the type of
the values SQLite returns for it: count, length, instr and random a Long, avg
and round a Double, the string functions and group_concat a String, max, min
and abs the type of their argument. sum is overloaded by column type, Long for
integers and Booleans, Double for reals; it no longer takes a String, BLOB,
enum or ULong column. AS takes a KProperty1<R, P?> of the element's type P, so
count(X) goes into a Long property, not an Int or a String one. Mixing result
types in one listOf doesn't compile either.
Nullability can't be checked through a property reference, as KProperty1 is
covariant, so it is checked when the statement is built: an element knows
whether it can be NULL in a row, or in a group for an aggregate function. One
case depends on what follows: without GROUP BY an aggregate query returns one
row even when no rows match, in which every column and every aggregate except
count is NULL. As GROUP BY can still be appended then, the statements carry
that error until GROUP BY clears it, and the scope reports it when it ends,
before any of its statements runs, transactions included. A property given
two expressions, renamed with @SerialName, or given another table's column is
rejected too.
max and min now return an element of their argument's kind, so they can be a
Boolean, BLOB or enum element as well; these now respect isFunction like the
numeric and string ones, so that a condition on such a function isn't prefixed
with the table name.
Documentation: a result columns section in the advanced query guide, and the
SQL functions guide no longer says functions are for conditions only. It also
listed a sign function, which is disabled, and had an example using `>`
instead of GT, which didn't compile.
ROADMAP: N5 is supported. This also commits the earlier roadmap decisions:
observable queries (N1) as high priority, type converters merged with the
kotlinx.datetime item (N8) as medium priority, and using a query's results
within the same transaction (N6) as low priority.
Tests: jvmTest (49) and testAndroidHostTest on API 26 and 37 (98) pass. The
native test sources compile for macosArm64, linuxX64, mingwX64 and
watchosArm32 but were not run, as this machine is an Intel Mac. The
compile-time rejections were checked with a temporary file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The full upsert, INSERT ... ON CONFLICT (target) DO UPDATE or DO NOTHING, updates the conflicting row in place, where INSERT OR REPLACE deletes it and inserts a new one, firing delete triggers and cascades. It is deferred: it needs SQLite 3.24, which the Android framework only has from API 30 on, while SQLlin supports API 24; the DSL can't attach a clause to INSERT yet; and the common cases already have a way, INSERT_OR_IGNORE for de-duplication, and an UPDATE followed by INSERT_OR_IGNORE in a transaction for an update in place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jvmTest and testAndroidHostTest of sqllin-dsl-test and sqllin-driver pass, and the macosArm64 tests and the sample compile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SQLite's ALTER TABLE can only rename a table, and add, rename or drop a column; dropping one needs SQLite 3.35, which Android only has from API 34 on. Any other change, such as adding a constraint or changing the primary key, rebuilds the table: create the new structure under a temporary name, copy the rows with INSERT INTO ... SELECT, drop the old table and rename the new one. The DSL could do all of it but the copy, and could only create a table under the name its @DBRow class fixes at compile time. INSERT, INSERT_OR_IGNORE and INSERT_OR_REPLACE now also take a SELECT of the table's row type: ArchiveTable INSERT (PersonTable SELECT WHERE<Archive>(PersonTable.age GT 60)) The column list is the row type's properties, in the order the SELECT selects them, so every column gets a value and the primary key is copied as it is selected. Requiring the table's own row type makes that complete at compile time. The SELECT becomes part of the INSERT: it is removed from the statements of its scope, so it no longer runs on its own, and its deferred GROUP BY check from N5 runs when it is taken. Table.withName returns a table with the same structure under another name, its CREATE TABLE statement renamed. It is a plain function, not a SQL keyword, so it is lowercase and carries no DSL marker. Its KDoc and the guide explain the rebuild, and why the new table is renamed rather than the old one: with SQLite's default settings, renaming a table also renames the references to it in other tables' foreign keys, which would then point at the dropped table. That was checked with sqlite3: with legacy_alter_table off, renaming the old table first rewrote the child's REFERENCES, while the documented order left it in place. The guide to modifying the database gains a section on rebuilding a table, and its Insert section covers INSERT ... SELECT. It also documents INSERT_OR_IGNORE and INSERT_OR_REPLACE, which it didn't mention. Tests: testInsertSelect covers copying with a WHERE parameter, a SELECT taken by an INSERT having no results of its own, filling a table from a grouped aggregate and rejecting the ungrouped one, and the three INSERTs on a primary key conflict. testTableRebuild runs a real migration from version 1 to 2: it renames a column, adds a UNIQUE constraint and drops a column, then checks the rows and keys, the constraint, and that another table's foreign key still points at the rebuilt table. jvmTest (51) and testAndroidHostTest on API 26 and 37 (102) pass, so the rebuild works on API 26, where DROP COLUMN doesn't. The native test sources compile for macosArm64, linuxX64, mingwX64 and watchosArm32 but were not run, as this machine is an Intel Mac. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
INSERT INTO ... SELECT copies rows into a rebuilt table, but converting them on the way needs what the DSL doesn't have yet: CAST to change a value's type, coalesce and ifnull to replace a NULL, such as when making a column NOT NULL, and literal values. They join the medium priority item for more functions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Table.UNION returns a statement typed by the table's row type, but decodes the rows with its first SELECT's deserializer. A union of projections, result columns or joins therefore compiles, and its results fail with a ClassCastException when used; checked with a union of two projections. This predates 2.4.0, as joins already had their own result type, but projections and result columns make it easier to reach. Supporting such unions changes the public UNION functions, which are inline, so it is deferred to the roadmap rather than fixed in 2.4.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
replace, instr, printf and group_concat put their string arguments into the
SQL between single quotes as they were. A ' in one broke the statement with a
syntax error, and a crafted one could rewrite it: checked with sqlite-jdbc,
instr(name, "zzz') + 1 + ('") GT 0 became
WHERE instr(name,'zzz') + 1 + ('')>?
which matched every book, though no name contains that string. Any of these
arguments that comes from user input was an injection point.
The arguments are now written as SQL string literals with each ' doubled,
the only escape SQLite has in one, so whatever the string holds stays inside
the literal. Binding them as parameters was considered, but a function
element is SQL text without parameters, and making elements carry them
through WHERE, HAVING, ORDER BY, GROUP BY and result columns, in order, is a
much larger change that escaping makes unnecessary.
testFunctionStringArguments covers a ' in each of the four functions, and the
crafted argument above now matches no rows; the test fails with the old code.
jvmTest (52) and testAndroidHostTest on API 26 and 37 (104) pass, and the
macosArm64 tests compile.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A comparison between two elements, such as length(name) GT pages, wrote
both as table.valueName. For a column that is right, but a function's value
name is the call itself, so it produced
WHERE book.length(name)<book.pages
which SQLite rejects with a syntax error; checked with sqlite-jdbc. A
comparison with a plain value already left a function unqualified, but the
comparisons between two elements, in ClauseNumber, ClauseString, ClauseBlob
and ClauseEnum, didn't check.
ClauseElement.appendSQL writes an element the way the value comparisons
always did, a column qualified by its table and a function as it is, and the
four element comparisons use it on both sides. Only comparisons involving a
function change, and those never worked.
testFunctionComparisons covers a function compared with a column, a column
with a function, two aggregates in HAVING, string functions, and max and min
of an enum column; the test fails with the old code. jvmTest (53) and
testAndroidHostTest on API 26 and 37 (106) pass, and the macosArm64 tests
compile.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New features for 2.4.0 in
sqllin-dsl, one commit per issue, plus two fixes found along the way, a dependency update and roadmap changes. Each commit message explains its change in detail, andCHANGELOG.mdhas an entry for each.New features
INSERT_OR_IGNORE, for SQLINSERT OR IGNORE. (d1c2d42)SELECTreads rows into a narrower@Serializabletype given to the clause function, as intable SELECT X<BookTitle>()orWHERE<BookTitle>(...), and selects only the columns it names. A type that doesn't fit the table is rejected when the statement is built. (bda5d27)AS, as intable SELECT listOf(count(X) AS AuthorStats::books, sum(pages) AS AuthorStats::totalPages) GROUP_BY author. A property's type is checked at compile time and its nullability when the statement is built; for an aggregate query withoutGROUP BY, which returns a row ofNULLs when no rows match, when the scope ends, before any statement runs. (0ba59f8)INSERT INTO ... SELECTforINSERT,INSERT_OR_IGNOREandINSERT_OR_REPLACE, andTable#withName, which together rebuild a table in a migration, for a changeALTER TABLEcan't make, such as adding a constraint, or dropping a column below Android API 34. (365123b)Breaking changes
These may stop existing code from compiling. Migration is described in
CHANGELOG.md.ClauseElement,ClauseNumberandClauseStringtake a type parameter, the type of their values. Code that only uses the DSL is unaffected; code that names these types needs a type argument. The public constructors of theClause*classes, which the generated code calls, now take a column's nullability instead of whether the element is a function. (0ba59f8)maxandminof a String column are now aClauseString, andsumis overloaded by column type, so asumof a String, BLOB, enum orULongcolumn no longer compiles. (0ba59f8)DatabaseScope#selectfunctions and theClause*classes changed their signatures. (bda5d27, 0ba59f8)Fixes
These predate 2.4.0; they were found by reading the code during this work, then confirmed with sqlite-jdbc.
replace,instr,printfandgroup_concatwere not escaped. A'in one broke the statement, and a crafted one rewrote it:instr(name, "zzz') + 1 + ('") GT 0matched every row. Any such argument taken from user input was an injection point. A'is now doubled, the only escape SQLite has in a string literal. (907351d)book.length(name)<book.pages. A function is now written as it is. (2220147)Other changes
sign; the guide to modifying the database describesINSERT INTO ... SELECT, rebuilding a table,INSERT_OR_IGNOREandINSERT_OR_REPLACE.UNIONof other result types as high (c6b6cd7), see below.Testing
jvmTest(53 tests) andtestAndroidHostTeston Robolectric API 26 and 37 (106 tests) pass locally. The tests added for the two fixes fail with the old code. The table rebuild test passes on API 26, whose SQLite lacksDROP COLUMN.macosArm64,linuxX64,mingwX64andwatchosArm32. CI on this PR is their first run.count(X)into anIntproperty or mixing result types in onelistOf, were verified by compiling temporary files.Known issue, not addressed here
UNIONreturns a statement typed by the table's row type, but decodes with its first member's result type, so a union of joins, projections or result columns fails with aClassCastExceptionwhen its results are used. This predates 2.4.0; projections and result columns make it easier to reach. Fixing it changes the public, inlineUNIONfunctions, so it is on the roadmap as high priority instead. (c6b6cd7)🤖 Generated with Claude Code