WebAssembly (emscripten) support

[iTrooz]

Hi, I'm not committing to anything, but I'd like to add support for WebAssembly/emscripten to this library. (WerWolv/ImHex#1299)

(The implementation would use the browser's methods to let the user pick a file/folder in their filesystem)

Is that something you'd be interested in ?

[btzy]

AFAIK this is a sort of hard/impossible thing to do properly.

TL;DR Browsers are designed to run websites in a sandbox so interacting with the host filesystem is not straightforward. I don't know of a good design to make what you are requesting work properly, and there are many complications.

---

Opening/saving files the traditional way

The traditional way to load a file from the user's machine is to add an <input type="file"> element and have the user click on it. File filters and a switch to select multiple files are supported. With appropriate JavaScript and CSS, it is possible to hide the actual element and click on it programmatically, which is what most websites do. It produces a File object that you can directly read from in JavaScript, but I'm not sure if it will have a filepath that agrees with the native filepath. Furthermore, even if it has a filepath that agrees, there is no way I know of to obtain the File object back from the filepath.

There has also been no way to show a save dialog in a browser. The best you can do is to save something to the downloads folder automatically. I wrote this Brainfuck compiler a long time ago but the "Open" and "Download" buttons on the top bar demonstrate how any browser-based implementation would behave.

There is also no way to show a folder picker.

---

The experimental File System API

This new API contains ways to show a file picker (both single and multiple files) or directory picker for input, and a file picker for output (save dialog). This is the most promising, however, there are two reasons (at least that I can tell) that make it problematic for NFDe:

1. Similar to the traditional way, all the pickers return a wrapper object (FileSystemHandle) instead of an underlying path, and there seems to be no way to reconstruct the handle from a path. (Presumably there are security concerns regarding a website knowing the actual path of the file.) This does not play nice with NFDe as it the return values are all filepaths (i.e. strings).
2. This new API is async. You call Window.showOpenFilePicker(), which returns immediately, and you get a callback later when it is done. This again doesn't fit into NFDe.

Also, this is experimental and currently only available in Chrome, Edge, and Opera (not sure if behind a feature flag).

---

Emscripten's file system

I haven't actually used Emscripten before, but from what I understand, they use an in-memory file system that has no access to the real files on the host filesystem. All the files in that file system must be downloaded (from the web) with the website. I suppose it's possible to write new files to that filesystem, so we can read a file from the host filesystem either using the traditional or experimental way, and then create the corresponding file in Emscripten's filesystem and write the data there. This should be mostly transparent for reading files, except if somebody else is concurrently modifying the file (Emscripten's in-memory file system will not get the modifications). However, writing files will pose a problem because the files will be written to the in-memory file system, so the changes will not be persisted by default. If we manually copy the file out after it is done writing, at this point we would have to ask the user for the real location to store the file (or just download it to the downloads directory if using the traditional way), which would surprise the user.

The proper way to resolve this (not sure if doable in Emscripten at this point, but I doubt so) would be to install some hook on Emscripten's filesystem API to invoke a custom handler instead of just reading/writing from the in-memory storage, and then in that custom handler we call the experimental File System API to open/save to the host filesystem. I don't know how difficult this would be, but it seems like the only proper way to do what the user would expect when they click on the "Save" button in your application. (Of course this would still not solve the two concerns with the experimental File System API above.)

---

So yeah I don't expect this to work anytime soon. Integrating with the experimental File System API sounds like something that the Emscripten project should be interested in implementing, so I think we might just need to wait for them to get around to doing this.

I don't think any of the other cross platform file dialog libraries out there have a proper Emscripten implementation, but do let me know if you come across one - I'd be interested to see how they implement it.

[iTrooz]

Hey ! Thanks for your thorough answer ! This helped me a lot to bring ImHex to the web (spoiler, in the end I did the same thing as you with https://btzy.github.io/jelly/bf.html, you can see https://github.com/WerWolv/ImHex/blob/ed8c0794bb4be617f969d455b37370c0a0cb5ae3/lib/libimhex/source/helpers/fs.cpp#L99 for implementation details)

So, a few things to note that might be interesting to you:

There is also no way to show a folder picker.

There is https://caniuse.com/?search=webkitdirectory which seems to do just that, and have relatively large adoption

Similar to the traditional way, all the pickers return a wrapper object (FileSystemHandle) instead of an underlying path, and there seems to be no way to reconstruct the handle from a path. (Presumably there are security concerns regarding a website knowing the actual path of the file.) This does not play nice with NFDe as it the return values are all filepaths (i.e. strings).

You can indeed solve this with the emscripten file system, that's how we do it in ImHex (see the impl details)

This new API is async. You call Window.showOpenFilePicker(), which returns immediately, and you get a callback later when it is done. This again doesn't fit into NFDe.

I don't know much about it, but there seem to be an ASYNCIFY option in emscripten that lets you call async APIs from C++. It will pause the whole program until the async call returns: https://web.dev/asyncify/ https://emscripten.org/docs/porting/asyncify.html

The proper way to resolve this (not sure if doable in Emscripten at this point, but I doubt so) would be to install some hook on Emscripten's filesystem API to invoke a custom handler instead of just reading/writing from the in-memory storage, and then in that custom handler we call the experimental File System API to open/save to the host filesystem.

I think you could solve this problem with https://emscripten.org/docs/api_reference/Filesystem-API.html#FS.trackingDelegate%5Bcallback%20name%5D to detect when the file is closed, but you would still need the experimental API, yeah

All in all, I don't feel like implementing this anymore, as this solution is complicated, half-baked and not even compatible with every browser, but I learned a lot of things by discussing here, so thank you !

[btzy]

Thanks for the information! What you've described seems to more or less work (though it would take quite a bit of effort and I'm not sure if it's really worth it).

There might be an issue if the user opens two files with the same name but different paths though, e.g. /path/to/A/file.bin and /path/to/B/file.bin. It seems that both will appear as /savedFiles/file.bin in the emscripten filesystem, which would break things if the user is allowed to have two files open at once. There's FileSystemHandle.isSameEntry() though, so we could probably compare the new file with all existing files with the same name, and if none are the same then we put the new file in a newly generated subdirectory.

[jimmywarting]

i would still like to be able to use this on the server side (nodejs, deno, bun)

[btzy]

I agree it will be nice to have, and it seems reasonable that a server-side implementation would be different from a web one, due to permission limitations on the web. I am new to server-side JavaScript/WebAssembly and so am not quite confident I'll be able to do everything the proper way, but I'm happy to look at a PR if you'd like to give this a try!

[Froncu]

Hi everyone! Any progress on this? I would love to see this implemented :)

[btzy]

No progress from me, but still happy to accept a PR.