[daun]

Issue

The Eloquent driver sorts text byte-by-byte. Flat files sort text alphabetically for the site's language. So the same content comes back in a different order depending on the driver. Capitals before lowercase, umlauts at the end and 10 before 2 etc. It's most noticeable when sorting assets by filename or entries by title.

Flat files	Eloquent driver
2.jpg	10.jpg
10.jpg	2.jpg
apfel.jpg	Zebra.jpg
Äpfel.jpg	apfel.jpg
Zebra.jpg	Äpfel.jpg

Solution

A new opt-in config: when enabled, sorting by a text field gives the same order as flat files. The driver mimics Statamic's Comparator. Number and date fields still sort as numbers and dates. Open to better names for the config.

• case-insensitive
• natural number order (2 before 10, IMG_3 before IMG_20)
• using the current site's locale

// config/statamic/eloquent-driver.php

'sqlite_collation' => true,

Off by default

It does make sorting slower. SQLite doesn't have alphabetical sorting built in, so the comparison is done by PHP. When not using an index (e.g. sorting entries by title), it is around twice as slow, so I'd suggest making this opt-in to allow performance-minded folks to keep their Eloquent speed-up :)

Benchmarks

Sorting…	1,000 items	10,000 items	100,000 items
Asset browser by filename, first page	0.02 → 0.4ms	0.02 → 3.5ms	0.02 → 33ms
Entries by title, first page	0.4 → 1.1ms	3.9 → 7.7ms	43 → 76ms
All entries by title	0.6 → 2.9ms	6.5 → 40ms	84 → 523ms

Drive-by fixes

• Sorting assets by a custom field without filtering by container threw a TypeError (AssetQueryBuilder::getJsonCasts() returned an array instead of a collection)
• orderByDesc() and reorder() now apply the same number/date casting as orderBy()

Limitations

• Requires PHP's intl extension
• Without intl, sorting is only made case-insensitive

[daun]

Come to think of it, the name of the new config key is pretty nondescript. Very open to suggestions here :)

[ryanmitchell]

Thanks for this, it's a really nice PR. I've pushed a commit (3e39e2f) with two changes:

1. The fallback without intl now matches Comparator
   Without intl, Comparator::strings() uses strnatcmp() on lowercased values, which is case-insensitive and natural. The previous NOCASE fallback only ignored ASCII case, so 10 still sorted before 2. SqliteCollation now registers a STATAMIC_NATURAL collation that does the same as Comparator in that setup. The comparison callback now comes from a new SqliteCollation::comparator(?string $locale) method, where a null locale means the fallback. That keeps it testable in CI.

2. Mapped entry columns that don't hold text are no longer collated
   With map_data_to_columns, mapped columns aren't in the model's $casts, so integer and datetime columns were getting collate added. The order was still right, because SQLite ignores collation for non-text values, but it stopped those columns using their indexes. EntryQueryBuilder::uncollatedColumns() now reads the column types from the schema and only collates mapped columns SQLite treats as text (type containing CHAR, CLOB or TEXT). It's cached in Blink per table.

I added tests for both.

A few other thoughts, none of them blocking:

• SECONDARY strength vs Comparator: Comparator lowercases and then compares at tertiary strength. SECONDARY also ignores other tertiary differences, like full-width vs half-width characters, so it's not exactly the same. That's fine for real content, but the "Match how Comparator ignores case" comment slightly overstates the reality.
• Performance note in the README: most of the asset browser slowdown comes from the index on name, filename and path columns no longer being used for the sort. Saying that plainly in the README would help people judge the trade-off before turning it on.
• Registering when the query is built: the collation is registered is called, but its name is baked into the SQL. If the connectionreconnects between building and running the query (DB::reconnect(), long-running workers), the query fails with no such collation sequence. That's unlikely with SQLite, so I'm happy to leave it, but registering when the the connection is established would be sturdier. A small side effect: toSql() now opens a connection.
• Remaining differences from flat files: Comparator sorts nulls last when ascending, and SQLite sorts them first. MySQL and Postgres aren't covered either. Postgres ICU collations (und-u-kn-true) could give the same order natively, which could be nice follow-up. Neither needs to happen in this PR. Just noting that "Match Stache sort order" is SQLite-only for now.
• The getJsonCasts() fix: good catch, that TypeError was a real bug.
• Possible fix for the index slowdown: SQLite can index with a custom collation, e.g. CREATE INDEX … ON assets (basename COLLATE STATAMIC_UNICODE_EN_US), and an ORDER BY with the same collation will use it. That would recover most of the asset browser slowdown. The catch is that anything writing to the table then needsthe collation registered (the sqlite3 CLI, GUIs, other apps), otherwise inserts fail, and the index is tied to one locale. So I wouldn't ship it as a migration, but it could be a documented option for people with large asset containers. The alternative that works on every database would be a precomputed sort-key column (Collator::getSortKey()) on hot fields like basename. Both are probably a follow-up if the slowdown turns out to matter in practice.