Using a new WebAssembly target in place of a JS reimplementation from-scratch?

[chances]

See chances/dbpf-d#4.

Justification

• Using WebAssembly we could more easily target other runtimes like Deno, Bun, and their ilk.
• This effort could also increase execution performance, given the properties of a systems-level programming language.

[chances]

I'd also like to discuss extracting your specification into it's own GitHub organization that we share.

[anonhostpi]

I like the idea, but I have a few concerns. Primarily, the most pressing concern is multithreading. I'm currently working on a multithreading implementation that leverages Web Workers (and Worker Threads in Node.js).

AFAIK the only way to achieve multithreading in WASM is through Web Workers. I believe this means that there would still have to be some level of interop between wasm and js to make multithreading usable.

Other concerns I have are:

• bundling: it needs to support a bundler, so that it can work in the browser. Currently, I use WebPack
• limited experience: I have limited experience with wasm, but I pick up pretty fast
• KaiTai support: currently, the project uses KaiTai structs for parsing data not associated with a common mime type (like GEOMs).
  • I am open to switching to a different cross-language serialization library, if another library would be better for this.
• (inevitably) write support: to make the tool fully rounded, I would like to eventually support write ops as well. This is another consideration for KaiTai. I believe KaiTai, (now) supports write ops in JS/TS

[anonhostpi]

For multithreading, I'm currently using a client-server relationship between the main (client) and Worker Threads (servers) to match the Worker.postMessage design.

As for shared data, I'm primarily targeting Blob URLs, since Blobs are paged. For Node.js, I'm just using File URLs and fs ops for reading file data.

• Whenever fs.openAsBlob gets full support, I will inevitably switch to it, so I can get paging support in Node.

[chances]

Regarding your parallelism concerns:

D has a fairly robust infrastructure of primitives in the standard library to support both SMP Concurrency and full-blown Parallelism. However, I am not sure how the LLVM backend handles parallelism in compiled WASM modules. (Note to self: Research the parallelism story and its support (or lack thereof) in LDC.)

Re: Shared Data

Note to self: Regarding parallelism concerns, SharedArrayBuffer has limited support across JS runtimes.

[chances]

Or, perhaps, we just focus on the inverse, i.e. create a WebAssembly target in this library by compiling a shared library via AssemblyScript?

---

@anonhostpi How far along is SimCity 4 support in this library?

[anonhostpi]

Or, perhaps, we just focus on the inverse, i.e. create a WebAssembly target in this library by compiling a shared library via AssemblyScript?

I was thinking that a good starting point would be to implement wasm in isolation on each thread interoped with the main thread via JS. It would be a hybrid implementation of wasm and JS/TS. Where wasm is responsible for parsing Blobs and JS/TS is responsible for interop.

SharedArrays will likely not be necessary, as the file content is best handled/shared via Blob/File URLs, and the content of each parsed entry can be serialized and returned back to the main thread via postMessage (or similar)

@anonhostpi How far along is SimCity 4 support in this library?

So, I'm still working on writing handlers for the contents of each DBPF entry, but the library currently supports parsing the header and index table for any DBPF version up to v2.1 (v1.x included)

Support for DBPF version 3.x is in the works, but it requires that I dig into Spore (and possibly Darkspore)

[chances]

@anonhostpi How far along is SimCity 4 support in this library?

So, I'm still working on writing handlers for the contents of each DBPF entry, but the library currently supports parsing the header and index table for any DBPF version up to v2.1 (v1.x included)

Support for DBPF version 3.x is in the works, but it requires that I dig into Spore (and possibly Darkspore)

This is good to hear! My D library barely functions, if at all, to parse v1.x archives.

Since I'd rather not reinvent the wheel, I'm going to experiment in my fork by wrapping this library in an Extism plugin.