Added embedding benchmark along with new config file and plot updates

[radoslavralev]

Note

Introduces reranking and a unified, provider-agnostic embedding stack, plus end-to-end benchmarking utilities.

• Adds optional cross-encoder reranking (--cross_encoder_model, --rerank_k) across local and Redis workflows; retrieval upgraded to top-k with CE selection
• Refactors embeddings to provider-agnostic interfaces: new EmbeddingModel, SimilarityMatcher, and embedding_providers (HuggingFace, OpenAI, Gemini); updates RedisVectorIndex to use providers and correct vector handling
• Normalizes embeddings and fixes dtype in Redis/data paths; adjusts threshold sweep to use dynamic max
• New benchmarking pipeline (run_benchmark.py, run_benchmark.sh) with bootstrap runs, model/CE sweeps, and organized outputs
• Adds analysis/plot scripts: precision@1 table/plots, multi-model precision-vs-CHR with AUC bars, precision-over-threshold; expands run_chr_analysis.sh
• Updates dependencies (google-genai, openai, peft, radon, pylint config) and removes large dataset CSVs from repo

^{Written by Cursor Bugbot for commit fbfe4e7. This will update automatically on new commits. Configure here.}

[Copilot]

Pull request overview

This PR adds optional cross-encoder reranking functionality to improve retrieval quality, implements top-k candidate retrieval in embedding matching, normalizes embeddings before Redis operations, enhances the Redis index handling with dimension probing, and improves visualization with dual-subplot charts. The changes span the core matching logic, benchmarking infrastructure, and result visualization.

Key Changes

• Adds cross-encoder reranking with configurable top-k retrieval for both Redis-based and standard neural embedding matching
• Implements top-k retrieval logic in blockwise embedding matching with support for variable k values
• Normalizes embeddings to float32 and unit length before Redis operations, with dimension probing to handle models with incorrect config dimensions

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 16 comments.

Show a summary per file

File	Description
src/customer_analysis/query_engine.py	Adds embedding dimension probing, enables trust_remote_code, and always overwrites Redis index on initialization
src/customer_analysis/embedding_interface.py	Implements top-k retrieval in blockwise matching methods with support for variable k; updates dimension inference to probe models first
src/customer_analysis/data_processing.py	Integrates cross-encoder reranking in both run_matching and run_matching_redis; adds embedding normalization before Redis operations
scripts/plot_multiple_precision_vs_cache_hit_ratio.py	Creates dual-subplot visualization with precision-CHR curves and AUC bar chart; adds numpy version compatibility for trapezoid/trapz
run_benchmark.sh	Provides example shell script for running benchmarks with cross-encoder models
run_benchmark.py	Extends benchmark loop to iterate over cross-encoder models and include them in output paths
evaluation.py	Adds command-line arguments for cross-encoder model and rerank_k parameter

Comments suppressed due to low confidence (2)

src/customer_analysis/embedding_interface.py:414

• The docstring for calculate_best_matches_with_cache_large_dataset does not document the new k parameter or how it affects the return value shapes. The function returns arrays with shape (num_sentences,) when k=1 but (num_sentences, k) when k>1. This should be documented to avoid confusion.

        """Large-dataset variant: find best cache match for each sentence using memmaps.

        Writes two memmaps (rows for sentences, cols for cache), normalised, and
        performs blockwise dot-products. If `sentence_offset` is provided and the
        cache corresponds to the same corpus, the self-similarity diagonal is masked.
        """

src/customer_analysis/embedding_interface.py:490

• The docstrings for calculate_best_matches_with_cache and calculate_best_matches_from_embeddings_with_cache do not document the new k parameter or how it affects return value shapes. When k=1, arrays have shape (N,), but when k>1, they have shape (N, k). This should be documented.

        """
        Calculate the best similarity match for each sentence against all other
        sentences using a neural embedding model.
        """

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The top-k retrieval logic in blockwise matching lacks test coverage. Since the repository has comprehensive tests and this is a significant new feature that changes the shape of return values and introduces complex merging logic, it should have tests covering: 1) k > 1 with various cache sizes, 2) edge cases where block size < k, 3) self-similarity masking with k > 1, and 4) correct sorting and merging across blocks.

[cursor]

Empty UV index URL breaks package installation

High Severity

The [[tool.uv.index]] URL is set to an empty string "" instead of being removed entirely. An empty URL is not a valid package index URL and will cause uv to fail when attempting to resolve or install packages from this index. The section should either be removed completely or contain a valid URL.

 

[cursor]

Cross-encoder reranking creates invalid pairs when cache is small

Medium Severity

When using cross-encoder reranking with k > 1 and the cache has fewer entries than k, the code iterates over all k indices in best_indices[i], but some of these indices are initialized to 0 with scores of -inf (from init_results). This causes cache_list[idx] to repeatedly access cache_list[0] for invalid entries, creating spurious query-cache pairs. The cross-encoder then scores these invalid pairs, which could lead to incorrect match selection.

 

[cursor]

Missing try/finally for memmap cleanup causes temp file leak

Medium Severity

In _calculate_best_matches_large_dataset and calculate_best_matches_with_cache_large_dataset, temporary memmap files are created via _write_embeddings_memmap, but the cleanup code that removes these files is not wrapped in a try/finally block. If an exception is raised during _compute_blockwise_best_matches or _compute_blockwise_best_matches_two_sets, the cleanup code is never executed and temporary files accumulate on disk. Over time with repeated failures, this could fill up disk space.

Additional Locations (1)

• src/customer_analysis/similarity_matcher.py#L451-L475