Make Blob Building Incremental#2

Merged

itzmeanjan merged 49 commits intomainfrom

make-blob-building-incremental

Jul 25, 2025

Owner

itzmeanjan commented Jul 23, 2025

break command, which takes arbitrary size file and breaks it down to proof-carrying chunks, can now work with any sized file, without loading the whole file content in memory. Meaning, if you want to erasure-code a 100GB file, on a machine with 16GB RAM, it should be possible.

Demo 👇

Also this PR introduces tokio as async runtime for break command.

itzmeanjan added 26 commits

July 16, 2025 13:21


          Incrementally updatable blob-builder struct

6d96f7c

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Add API comments for BlobBuilder

f03db61

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Update how chunkset-id is computed

62e92a8

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Update tests to work with new BlobBuilder API

e4d8738

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Update benchmarks to work with new BlobBuilder API

b7fd629

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Update doc-test in lib.rs

7598b34

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Add new dependency for figuring out number of logical CPUs

65dae42

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Update decds CLI tool to use new BlobBuilder incremental API

30af22c

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Trap failure of any bash commands in test script

b3977f1

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Blob-builder update method can now exploit rayon-based data-paralle…

0712cd9

…lism

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Bump rlnc to latest version

f7bbe24

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Put independent components of Blob Builder (CLI command break) into…

e08db87

… different threads, communicating over MPSC sync channels

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Address issue in building file-path when finalizing erasure-coded chunks

7a4299e

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Make inter-thread communication channel async

cd779a9

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Update test-runner shell script to compile with platform-specific opt…

1290b68

…imization

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Change buffer sizes - it demonstrates better performance

738eb01

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Update decds CLI test script to work with 1GB random file

da70e18

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Add tokio - async runtime engine as a dependency of the project

0098b9b

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Rewrite logic for breaking arbitrary-size blob data into proof-carryi…

ac63ae3

…ng chunks, using `tokio` async runtime

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Refactor code-block into its own function

412a933

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Add new Makefile recipe for ease of building executable

9eaccb3

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Update custom Cargo profile optimized with aggressive optimization …

50cf066

…settings

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Show progress of chunkify process with updating text

6173e89

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Add progress text in break command output

e26faae

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Each tokio task computes blob-level Merkle inclusion proof for itself

3eb6260

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Simplify script which tests decds CLI

e1a2c70

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>

itzmeanjan self-assigned this

itzmeanjan added 3 commits

July 23, 2025 17:11


          Remove unused optimization flag declaration from BASH script

27064f2

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Detailed log for kind of error, that was encountered during execution…

9c6be1c

… of `break` command

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Use console crate to lookup terminal size

c1955b4

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>

itzmeanjan added 20 commits

July 23, 2025 19:33


          Check for errors when adding chunk to repairing chunkset

6deb550

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Make bash script for generic so that it runs on both Linux and Mac

ae3972d

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Update github actions script to run renamed BASH script

e3c0807

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Do not use -p option with time command

563eb07

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Update Blob repairing tests to not ignore return value of add_chunk

7eaf413

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Use JoinSet to spawn a lot of worker tasks

34c7e7a

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          For communicating in between blob-reader and blob-builder tasks use M…

…utex wrapped vector and oneshot channel instead of mpsc channel

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Human readable byte presentation function now displays upto two place…

13ac327

…s after decimal

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          For communication in between blob-builder and partial-chunk writer ta…

d906e0e

…sks, use Mutex wrapped vector and oneshot channel instead of mpsc channel

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Update github actions CI script to check that decds works with MSRV

c6af16d

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Mention make build recipe to avoid system-wide installation of decds

4f31178

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Update code-coverage report and add instrutions to uninstall decds

e563d26

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Implement Error and Debug traits for custom error types

132b9ec

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Update example in lib.rs

fcfead5

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Mention script on test decds CLI

1a443f5

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          decds CLI version command now shows git commit from which it was bu…

cfa813d

…ilt and build date

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Update bash script to show which version of decds CLI it is using

4fd5260

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Bump version to 0.2.0 - ready for release

294ebb4

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Bump dependency rlnc to latest version

83a3aa4

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>


          Do not use features from rust-lang/rust#53667, stay compatible to MSR…

59494c5

…V 1.85.0

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>

itzmeanjan merged commit 5cdde34 into main

12 checks passed

itzmeanjan deleted the make-blob-building-incremental branch

July 25, 2025 13:43

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet