Unexpected benchmark result

[zthhhhh]

Hi vectorchord team,

I recently used the ann-benchmarks tool to conduct a preliminary evaluation of pgvector and VectorChord. The build time of the ivf-rabitq index is significantly faster than that of the hnsw index, even though I set sufficiently large shared_buffers and maintenance_work_mem to eliminate the potential impact of disk I/O.

However, the qps isn't meet expectations. Below are the test results:

I use docker image tensorchord/vchord-postgres:pg18-v1.0 to build postgres, and ll tests were run on the dbpedia-openai-1000k-angular dataset and same hardware. Here are the parameter settings:

1. pgvector-hnsw

name: pgvector_hnsw
run_groups:
test1:
arg_groups: [{M: [16], efConstruction: 64}]
args: {}
query_args: [[10, 20, 30, 40, 60, 80, 120, 160, 200,250, 300,350, 400, 500, 600, 700, 800]]

2. pgvector_ivfflat

name: pgvector_ivfflat
run_groups:
test2:
arg_groups: [{lists: [2000]}]
args: {}
query_args: [[1, 10, 20, 50 ,100, 200, 300, 400, 600, 800]]

3. vchord

name: vchordrq
run_groups:
test3:
arg_groups: [{lists: [2000]}]
args: {}
query_args: [[1, 10, 20, 50 ,100, 200, 300, 400, 600, 800]]

I’d like to understand what might be causing the performance to fall short of pgvector’s, and I sincerely apologize If I missed relevant details while reading the VectorChord documentation.

Thank you for your attention.

[VoVAllen]

We tested this dataset a long time ago, and the results were similar to what you posted. This dataset is very strange—on hnsw, even small ef_search (120) can achieve excellent results (99%). As for IVF indexing, scanning 5% of the vectors (nprobe=100, nlist=2000) still fails to achieve similar recall. But this basically only happens with this dataset; you should see different results if you try another dataset.

[zthhhhh]

Thanks for your attension, I will with other open-source datasets. Also, are there any test results for VectorChord on other datasets? I want to evaluate whether my test results meet expectations.

[mertalev]

DBPedia is a knowledge graph, so maybe graph indices are more suited to structuring it. It'd be interesting to see how vchordg performs on this dataset since it's also graph-based.