Skip to content

librocksdb_sys: Implement a writable file interface#851

Merged
ti-chi-bot[bot] merged 4 commits into
tikv:masterfrom
Leavrth:implement_mem_env_sst_reader
Jun 2, 2026
Merged

librocksdb_sys: Implement a writable file interface#851
ti-chi-bot[bot] merged 4 commits into
tikv:masterfrom
Leavrth:implement_mem_env_sst_reader

Conversation

@Leavrth

@Leavrth Leavrth commented Jun 1, 2026

Copy link
Copy Markdown

Implement a writable file interface to enable SST data streaming via memory-copy level writes instead of KV-level writes.

let env = Arc::new(Env::new_mem());
let file = env
    .new_writable_file(path, EnvOptions::new())
    .unwrap();
let sst_file_content_stream = self.download_file_from_external_storage(path);
io::copy(&mut sst_file_content_stream, &mut file).unwrap();

The implementation is modeled after sequential_file

Summary by CodeRabbit

  • New Features

    • Added writable-file support to the RocksDB environment: create, write/append, flush, close, and safe cleanup; integrates with Rust's standard I/O (Write/flush) and re-exports the writable-file type.
  • Tests

    • Added tests exercising writable-file lifecycle: create, write, flush, idempotent close, read-back verification, and deletion.

Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
@ti-chi-bot ti-chi-bot Bot added the dco-signoff: yes Indicates the PR's author has signed the dco. label Jun 1, 2026
@coderabbitai

coderabbitai Bot commented Jun 1, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 127aa28a-b235-4c72-adcd-52fa4f762ba9

📥 Commits

Reviewing files that changed from the base of the PR and between 897faa8 and 5411724.

📒 Files selected for processing (1)
  • librocksdb_sys/crocksdb/crocksdb/c.h
🚧 Files skipped from review as they are similar to previous changes (1)
  • librocksdb_sys/crocksdb/crocksdb/c.h

📝 Walkthrough

Walkthrough

Adds end-to-end WritableFile support: C API wrappers, librocksdb_sys FFI bindings, a safe Rust WritableFile type implementing io::Write with close/Drop, crate re-export, and integration tests that create/write/flush/close/read/delete files.

Changes

WritableFile Support Stack

Layer / File(s) Summary
C API writable file contract and implementation
librocksdb_sys/crocksdb/crocksdb/c.h, librocksdb_sys/crocksdb/c.cc
New C functions create writable files via Env, append and flush data, close handles, and destroy wrappers. Error propagation uses SaveError and Status translation.
FFI bindings to Rust
librocksdb_sys/src/lib.rs
Adds DBWritableFile opaque type and FFI function declarations (crocksdb_writable_file_create, crocksdb_writable_file_append, crocksdb_writable_file_flush, crocksdb_writable_file_close, crocksdb_writable_file_destroy) bridging the C API to Rust.
Safe Rust WritableFile wrapper and public API
src/rocksdb.rs, src/lib.rs
Env::new_writable_file factory creates wrapped files; WritableFile struct implements io::Write for appending, flush(), explicit close(), RAII Drop cleanup, and FFI error-to-io::Error translation. Type re-exported at crate root.
WritableFile integration tests
src/rocksdb.rs
test_env_operations updated to create writable files, write/flush/close (including idempotent close), read back contents via sequential file API, verify contents, and assert deletion behavior.

Sequence Diagram

sequenceDiagram
  participant Caller
  participant Env
  participant RustWrapper as WritableFile (Rust)
  participant FFI as librocksdb_sys FFI
  participant C_API as RocksDB C API

  Caller->>Env: new_writable_file(path, opts)
  Env->>FFI: crocksdb_writable_file_create(path, opts, err)
  FFI->>C_API: Env::NewWritableFile -> rocksdb::WritableFile*
  C_API-->>FFI: WritableFile* / Status
  FFI-->>RustWrapper: DBWritableFile pointer or err
  Caller->>RustWrapper: write(data)
  RustWrapper->>FFI: crocksdb_writable_file_append(data, len, err)
  FFI->>C_API: WritableFile::Append -> Status
  C_API-->>FFI: Status (OK/error)
  FFI-->>RustWrapper: ok / err
  Caller->>RustWrapper: flush()
  RustWrapper->>FFI: crocksdb_writable_file_flush(err)
  RustWrapper->>FFI: crocksdb_writable_file_close(err)
  RustWrapper->>FFI: crocksdb_writable_file_destroy()
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 I nibble bytes from C to Rust tonight,
A WritableFile that feels just right,
With append and flush and tidy close,
My tiny hops write rows on rows,
Hooray — the rabbit's files take flight!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 5.56% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely summarizes the main change: implementing a writable file interface for the librocksdb_sys bindings.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@ti-chi-bot ti-chi-bot Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jun 1, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
librocksdb_sys/crocksdb/crocksdb/c.h (1)

1797-1809: ⚡ Quick win

Consider consistent naming between typedef and function prefix.

The typedef is crocksdb_writablefile_t (no underscore between "writable" and "file") but the functions use crocksdb_writable_file_* (with underscore). This differs from the existing pattern where typedef and function names match—for example, crocksdb_sequential_file_t uses crocksdb_sequential_file_* functions, and crocksdb_readoptions_t uses crocksdb_readoptions_* functions.

For consistency, either:

  • Change the typedef to crocksdb_writable_file_t, or
  • Change function names to crocksdb_writablefile_*

This mismatch could confuse API users and lead to typos when calling these functions.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@librocksdb_sys/crocksdb/crocksdb/c.h` around lines 1797 - 1809, The typedef
name crocksdb_writablefile_t is inconsistent with the function family
crocksdb_writable_file_*; rename the typedef to crocksdb_writable_file_t (and
any struct/tag declarations and all references) so it matches the functions
crocksdb_writable_file_create, crocksdb_writable_file_append,
crocksdb_writable_file_flush, crocksdb_writable_file_close, and
crocksdb_writable_file_destroy; update all usages and forward declarations in
the codebase and headers to the new typedef name to maintain the established
naming pattern.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/rocksdb.rs`:
- Around line 2923-2983: The WritableFile Drop currently calls
crocksdb_writable_file_destroy directly and can leak an un-closed file; make
close() idempotent and have Drop attempt a best-effort close before destroy. Add
a closed: bool (or closed: AtomicBool if multithreaded) field to WritableFile,
initialize it in WritableFile::new, update close(&mut self) to return early if
closed is true and otherwise call crocksdb_writable_file_close and set closed =
true (preserve the existing Result<String> behavior), and change Drop::drop to
call close() in an unsafe, ignored-error way and then call
crocksdb_writable_file_destroy only once (set inner = ptr::null_mut() after
destroy to avoid double-destroy).

---

Nitpick comments:
In `@librocksdb_sys/crocksdb/crocksdb/c.h`:
- Around line 1797-1809: The typedef name crocksdb_writablefile_t is
inconsistent with the function family crocksdb_writable_file_*; rename the
typedef to crocksdb_writable_file_t (and any struct/tag declarations and all
references) so it matches the functions crocksdb_writable_file_create,
crocksdb_writable_file_append, crocksdb_writable_file_flush,
crocksdb_writable_file_close, and crocksdb_writable_file_destroy; update all
usages and forward declarations in the codebase and headers to the new typedef
name to maintain the established naming pattern.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 9566ac32-74df-4993-8e01-857df1b6c77f

📥 Commits

Reviewing files that changed from the base of the PR and between c9aefbe and 9164fc4.

📒 Files selected for processing (5)
  • librocksdb_sys/crocksdb/c.cc
  • librocksdb_sys/crocksdb/crocksdb/c.h
  • librocksdb_sys/src/lib.rs
  • src/lib.rs
  • src/rocksdb.rs

Comment thread src/rocksdb.rs
Leavrth added 3 commits June 1, 2026 19:50
Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
@ti-chi-bot ti-chi-bot Bot added the lgtm label Jun 2, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jun 2, 2026

Copy link
Copy Markdown

[LGTM Timeline notifier]

Timeline:

  • 2026-06-02 02:20:46.331095712 +0000 UTC m=+235347.401413102: ☑️ agreed by YuJuncen.

@ti-chi-bot ti-chi-bot Bot added the approved label Jun 2, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jun 2, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: glorv, YuJuncen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@Leavrth

Leavrth commented Jun 2, 2026

Copy link
Copy Markdown
Author

/ok-to-test

@ti-chi-bot ti-chi-bot Bot added the ok-to-test Indicates a PR is ready to be tested. label Jun 2, 2026
@Leavrth

Leavrth commented Jun 2, 2026

Copy link
Copy Markdown
Author

/retest

@ti-chi-bot ti-chi-bot Bot merged commit 2f9ebd7 into tikv:master Jun 2, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved dco-signoff: yes Indicates the PR's author has signed the dco. lgtm ok-to-test Indicates a PR is ready to be tested. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants