Add SQLite3MultipleCiphers 2.3.3 support (SQLite 3.53.0) #1
Open
gainskills wants to merge 98 commits into
Open
Add SQLite3MultipleCiphers 2.3.3 support (SQLite 3.53.0) #1gainskills wants to merge 98 commits into
gainskills wants to merge 98 commits into
Conversation
use same workaround as authelia/authelia#6404 before mattn#1177 fixes the build otherwise
action/upload-artifact@v1 has been deprecated for a while. It seems like GitHub Actions will now cancel workflows if it is still using v1 of the action. This upgrades to the latest v4 of the action. See https://github.blog/news-insights/product-news/get-started-with-v4-of-github-actions-artifacts/ for details.
The commit removes the use of runtime.SetFinalizer to finalize
SQLiteRows since only serves to close the associated SQLiteStmt which
already has a registered finalizer.
It also fixes a race and potential panic in SQLiteRows.Close around the
SQLiteRows.s field (*SQLiteStmt) which is accessed without a mutex being
held, but modified with it held (null'd out). Further the mutex we are
holding is that of the SQLiteStmt so a subsequent call to Close will
cause a panic sine it'll attempt to dereference a nil field. The fix
here is to add a mutex for closing to SQLiteRows.
Since we now also set the s field to nil when closing this commit
removes the "closed" field (since checking if s is nil is the same) and
also changes the type of "nc" (number of columns) to an int32 so that we
can pack the nc and cls fields, and add the close mutex without making
the struct any bigger.
```
goos: darwin
goarch: arm64
pkg: github.com/charlievieth/go-sqlite3
cpu: Apple M4 Pro
│ x1.txt │ x4.txt │
│ sec/op │ sec/op vs base │
Suite/BenchmarkExec/Params-14 719.2n ± 2% 716.9n ± 1% ~ (p=0.897 n=10)
Suite/BenchmarkExec/NoParams-14 506.5n ± 3% 500.1n ± 0% -1.25% (p=0.002 n=10)
Suite/BenchmarkExecContext/Params-14 1.584µ ± 0% 1.567µ ± 1% -1.07% (p=0.007 n=10)
Suite/BenchmarkExecContext/NoParams-14 1.524µ ± 1% 1.524µ ± 1% ~ (p=0.539 n=10)
Suite/BenchmarkExecStep-14 443.9µ ± 3% 441.4µ ± 0% -0.55% (p=0.011 n=10)
Suite/BenchmarkExecContextStep-14 447.8µ ± 1% 442.9µ ± 0% -1.10% (p=0.000 n=10)
Suite/BenchmarkExecTx-14 1.643µ ± 1% 1.640µ ± 0% ~ (p=0.642 n=10)
Suite/BenchmarkQuery-14 1.968µ ± 3% 1.821µ ± 1% -7.52% (p=0.000 n=10)
Suite/BenchmarkQuerySimple-14 1.207µ ± 2% 1.040µ ± 1% -13.84% (p=0.000 n=10)
Suite/BenchmarkQueryContext/Background-14 2.400µ ± 1% 2.320µ ± 0% -3.31% (p=0.000 n=10)
Suite/BenchmarkQueryContext/WithCancel-14 8.847µ ± 5% 8.512µ ± 4% -3.79% (p=0.007 n=10)
Suite/BenchmarkParams-14 2.131µ ± 2% 1.967µ ± 1% -7.70% (p=0.000 n=10)
Suite/BenchmarkStmt-14 1.444µ ± 1% 1.359µ ± 1% -5.89% (p=0.000 n=10)
Suite/BenchmarkRows-14 61.57µ ± 1% 60.24µ ± 1% -2.16% (p=0.000 n=10)
Suite/BenchmarkStmtRows-14 60.15µ ± 1% 59.08µ ± 1% -1.78% (p=0.000 n=10)
Suite/BenchmarkQueryParallel-14 960.9n ± 1% 420.8n ± 2% -56.21% (p=0.000 n=10)
geomean 4.795µ 4.430µ -7.62%
```
A valid sqlite header must always be included (like in the other files) but sqlite3-binding.h explicitly guards against the system library case.
Because of sqlite's flexible typing, even though the column was declared as jsonb, the values are stored as the TEXT-typed json because they're converted to strings by Value/Scan. If the table is strict with a BLOB column, the example fails because of the type mismatch. This can be fixed by using the `jsonb()` function to convert incoming string-typed json and the `json()` function to convert outgoing binary-typed jsonb. The example is expanded to show both of these approaches. Note that both approaches use the same string-typed marshalling functions because the conversion to jsonb occurs within sqlite3, not within the Go code. SQLite docs state that the binary format is internal and applications shouldn't try to generate it: https://sqlite.org/json1.html#jsonb
GCP (at least Google Compute Engine) is a VM, and does not have restrictions relating to gcc. Also it appears that this whole section refers to general compiling, rather than compiling on GCP.
* Add ability to set an int64 file control * Update documentation * Remove duplicate err check in test * Update sqlite3.go Co-authored-by: rittneje <rittneje@gmail.com> --------- Co-authored-by: rittneje <rittneje@gmail.com>
reduce CGO call overhead for exec and bind paths
stmtCacheSize is immutable after connection open, so checking it before the lock avoids mutex overhead when cache is not enabled.
When stmtCacheSize <= 0, stmtCacheCount >= stmtCacheSize is always true, so the explicit check is unnecessary.
Finalize all cached statements even if one fails. Leaving a finalized statement in the cache map would be a use-after-finalize bug per SQLite documentation.
prepareWithCache now delegates to prepare and sets cacheKey afterward, removing the useCache boolean parameter.
This avoids an unnecessary reset when the cache is full, guarantees a statement cannot enter the cache without being reset/cleared, and fixes a leak where sqlite3_finalize was not called when reset failed.
Clarify that each connection in the sql.DB pool maintains its own independent statement cache.
[codex] add opt-in statement cache
Updated security policy to reflect supported versions and reporting guidelines.
based on SQLite 3.51.2
Owner
|
Hey @gainskills, thanks for opening this PR. I actually cannot merge it because my |
Author
Hi @jgiannuzzi , Please let me know once the new branch is ready, and I’ll update the PR accordingly. Alternatively, you can create a new branch with the changes that made this easier. Thanks, |
sqlite3_prepare_v2 returns SQLITE_OK with a NULL statement handle when the input contains no SQL. exec() already handled this; query() forwarded the NULL handle to bind(), which crashed in sqlite3_clear_bindings(NULL). Make query() skip NULL statements like exec() does, and make SQLiteRows safe against a nil underlying statement so the empty-rows return value does not crash. Closes mattn#1390
…-panic Fix panic when querying input with no SQL (only comments/whitespace)
evict least-recently-used stmt when cache is full
Upgrade SQLite to version 3053000
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
aes128cbc,aes256cbc,chacha20,sqlcipher, andrc4._cipher,_key,_kdf_iter,_hmac_check,_hmac_use,_legacy,_legacy_page_size,_mc_legacy_wal,_plaintext_header_size. Cipher PRAGMAs run beforeany schema-touching SQL;
_keyruns last in the cipher block so key derivation triggers after configuration.sqlite3mc-2.2.7), upgraded to MC 2.3.3 (adds thread-safety fix insqlite3mc_cipher_name, symbol-conflict fix when statically linking withlibsodium, and secure clearing of cipher data structures on free).
mattn/masterso the cipher integration coexists with upstream's CGO/perf rewrite, opt-in statement cache (_stmt_cache_size), Go 1.21 floor, and recent bug fixes.Test plan
go build ./...succeedsgo test -run Cipher -v ./...passes (end-to-end cipher integration test insqlite3_cipher_test.go)go test ./...passes (full suite, no regressions on the merged tree)_stmt_cache_size_keylast in the cipher block)sqlite3ext.hentries (SQLite 3.52.0+ function pointers) compile cleanly with the existing build flags🤖 Generated with Claude Code