[ehennestad]

Motivation

Problem — A user who reads NWB files with nwbRead waits for MATLAB classes to be generated on every call, even when the file uses the same schema version as the previous one. Reading the same 2.11.0 file three times in a row takes 6.8 s, 4.7 s and 4.3 s; only the first read needs new classes. nwbRead regenerates all classes from the specification embedded in the file each time, and the classes it writes are identical to the ones it wrote on the previous call. Most of the time spent reading a small file is this regeneration.

Solution — Generated classes are kept per namespace version and copied back when the specification, the generator and the dependencies are unchanged. The second and third reads of the same file now take about 1 s each. generateCore uses the same cache.

Additional fix: classes of the previously generated version remain

Problem — When a user generates one schema version and then another, classes that only the first version defines stay in +types. After generateCore("2.11.0") followed by generateCore("2.7.0"), types.core.EventsTable, a type that 2.7.0 does not have, is still on the path. Generation writes the classes of the new version over the old folder but never removes the files the new version does not write. Nothing reports it, so the user can create objects of a type that does not exist in the active schema version.

Solution — The folder of a namespace is emptied before its classes are generated, so only the classes of the new version remain.

Why in this PR — The cache stores a copy of the folder right after generation. Without emptying it first, the cache entry for 2.7.0 would permanently contain EventsTable from 2.11.0, and every later switch to 2.7.0 would restore it.

What changed

• nwbRead and generateCore reuse classes generated earlier for the same namespace version, when the specification, the generator code and the dependencies match.
• A file written by MatNWB and the schema bundled with MatNWB give the same match, because the comparison is made on the parsed specification, not on the YAML or JSON text.
• Changing the specification, editing the generator code, or changing a dependency (for example hdmf-common under core) makes the classes generate again.
• Several variants of one schema version are kept side by side, so switching between them does not regenerate: for example files written by PyNWB, whose embedded 2.11.0 specification differs from the one bundled with MatNWB, and files written by MatNWB. At most three variants are kept per namespace version; the one used least recently is removed.
• The cache lives in generated-types-cache in the MatNWB folder (gitignored). The GeneratedTypesCacheFolder preference in the matnwb group moves it elsewhere.
• Adding MatNWB with addpath(genpath(...)), as the intro tutorial shows, does not put the cached classes on the path.
• Switching to another schema version removes the classes that only the previous version defines.
• generateExtension and nwbInstallExtension are unchanged and still generate every time.

Implementation notes

• New namespace matnwb.internal.typecache. generateNamespaces takes parsed namespaces (from spec.generate), sorts them by dependency, and for each one copies the cache entry into the save folder on a hit, or generates into the save folder and replaces the entry on a miss.
• A cache entry is <cache folder>/resources/<namespace>/<version>/<key>/, where <key> is the first 12 characters of the cache key. It holds +types/+<namespace>/, namespace.mat (the same content as namespaces/<name>.mat), record.json with the full key, and last-used.txt. The record is written last, so an entry without it counts as incomplete.
• Copying from an entry or writing it stores the time of use in last-used.txt, as text, because file modification times can be too coarse to order entries used in quick succession. Writing a fourth entry for a namespace version removes the least recently used one; an incomplete entry counts as least recently used.
• The resources level keeps the entries out of genpath, which skips folders with that name; each entry holds a +types folder that would otherwise shadow the active classes.
• The cache key is a SHA-256 over the parsed specification (name, version, dependencies and schema; not the namespace declaration, which writers format differently), the code in +file, +spec and +schemes, and the keys of the dependencies.
• A specification embedded by PyNWB can differ from the schema bundled with MatNWB for the same version. The two have different keys and are kept as separate entries.
• With a cache hit, about 0.4 s of a read is spent parsing the specification and about 0.3 s on hashing and copying.

Examples

Reading the same file repeatedly

The snippet writes an empty 2.11.0 file and reads it three times in a fresh MATLAB session. Each line is the time for one nwbRead call.

nwb = NwbFile(identifier="demo", session_description="demo", ...
    session_start_time=datetime(2024, 1, 1, TimeZone="UTC"));
nwbExport(nwb, "demo.nwb");

for iRead = 1:3
    tic
    nwb = nwbRead("demo.nwb");
    fprintf("read %d (schema %s): %.1f s\n", iRead, nwb.nwb_version, toc)
end

Before — Every read regenerates the classes, so each one takes several seconds.

read 1 (schema 2.11.0): 6.8 s
read 2 (schema 2.11.0): 4.7 s
read 3 (schema 2.11.0): 4.3 s

After — The first read generates the classes and stores them. The next two copy them from the cache.

read 1 (schema 2.11.0): 7.5 s
read 2 (schema 2.11.0): 1.1 s
read 3 (schema 2.11.0): 1.0 s

Switching schema version

The snippet generates 2.11.0, then 2.7.0, and checks whether the class file for EventsTable, which only 2.11.0 defines, is still there.

generateCore("2.11.0");
generateCore("2.7.0");
fprintf("types.core version: %s\n", types.core.Version())
fprintf("EventsTable.m present: %d\n", isfile(fullfile(misc.getMatnwbDir(), "+types", "+core", "EventsTable.m")))

Before — The active version is 2.7.0, but the 2.11.0 class is still present.

types.core version: 2.7.0
EventsTable.m present: 1

After — Only the classes of 2.7.0 remain.

types.core version: 2.7.0
EventsTable.m present: 0

Switching between files written by PyNWB and by MatNWB

The snippet reads a 2.11.0 file written by PyNWB 4.2.0 and a 2.11.0 file written by MatNWB, twice each, after generateCore("2.11.0"). The two files embed different variants of the 2.11.0 specification.

files = ["pynwb.nwb", "matnwb.nwb", "pynwb.nwb", "matnwb.nwb"];
labels = ["pynwb file", "matnwb file", "pynwb file", "matnwb file"];
for i = 1:numel(files)
    tic; nwbRead(files(i)); fprintf("read %d (%s): %.1f s\n", i, labels(i), toc);
end

After — The PyNWB variant is generated once. Each later switch copies the matching variant from the cache. (Before this change every read regenerated, as in the first example.)

read 1 (pynwb file): 6.4 s
read 2 (matnwb file): 1.0 s
read 3 (pynwb file): 0.9 s
read 4 (matnwb file): 0.8 s

How to test

Run the "Reading the same file repeatedly" snippet above in a fresh MATLAB session: the second and third reads should take about 1 s. Then run the new unit tests, which use a small test schema and a temporary cache folder:

results = runtests("tests.unit.TypeCacheTest");
disp(table(results))

Checklist

• Have you ensured the PR description clearly describes the problem and solutions?
• Have you checked to ensure that there aren't other open or previously closed Pull Requests for the same change?
• If this PR fixes an issue, is the first line of the PR description fix #XX where XX is the issue number?

🤖 Generated with Claude Code