For rebuttal, add GLiNER2 support for CV evaluation and define GLINER_SCHEMA#101
For rebuttal, add GLiNER2 support for CV evaluation and define GLINER_SCHEMA#101
Conversation
for more information, see https://pre-commit.ci
There was a problem hiding this comment.
Pull request overview
Adds optional GLiNER2-based extraction support to the CV evaluation pipeline by introducing a GLiNER schema definition, a new evaluator implementation, and a CLI flag to select it.
Changes:
- Define
GLINER_SCHEMAfor CV field extraction. - Add
GLiNEREvaluatorand route evaluator selection via--use_gliner2. - Expose
--use_gliner2in the argument parser for CV evaluation.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
src/gimbench/cv/schema.py |
Adds GLINER_SCHEMA used by GLiNER2 JSON extraction. |
src/gimbench/cv/evaluators.py |
Introduces GLiNEREvaluator and updates evaluator selection logic. |
src/gimbench/arguments.py |
Adds --use_gliner2 CLI flag for CV evaluation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| "phone_number::str::Phone number", | ||
| "email::str::Email address", | ||
| "highest_level_degree::[Bachelor|Master|PhD]::str::Highest educational degree", | ||
| "university::str::University name", |
There was a problem hiding this comment.
GLINER_SCHEMA entries appear to use a consistent field::type::description format, but highest_level_degree has an extra segment (highest_level_degree::[Bachelor|Master|PhD]::str::...). This inconsistency is likely to break whatever parsing GLiNER2.extract_json does for schema strings. Align this entry’s format with the others (and with the expected GLiNER2 schema syntax) so it uses the same number of ::-separated parts.
| raise ImportError( | ||
| "The 'gliner2' package is required but not installed. " | ||
| "Please install it manually using `pip install gliner2` or `uv add gliner2` " | ||
| "to evaluate using this model." |
There was a problem hiding this comment.
The comment says the CV content will be truncated due to a GLiNER2 length limit, but the code passes cv_content directly with no truncation. Either implement truncation based on the model’s documented limit or update/remove the comment to reflect actual behavior.
| if isinstance(extracted_item, dict): | ||
| for field in CV_FIELDS: | ||
| val = extracted_item.get(field, "") | ||
| extraction[field] = str(val) if val is not None else "" | ||
| return extraction | ||
|
|
||
|
|
There was a problem hiding this comment.
conduct_eval silently prioritizes --use_outlines over --use_gliner2 if both flags are set. This makes CLI behavior ambiguous and can lead to unexpected evaluator selection. Consider validating these flags as mutually exclusive (e.g., in validate_and_standardize or here) and raising a clear error when both are provided.
| def _add_cv_eval_args(parser): | ||
| parser.add_argument("--use_outlines", action="store_true", help="Whether to use outlines in CV evaluation") | ||
| parser.add_argument("--use_gliner2", action="store_true", help="Whether to use GLiNER2 in CV evaluation") | ||
| parser.add_argument( |
There was a problem hiding this comment.
--use_outlines and --use_gliner2 can both be set, but evaluator selection currently depends on precedence elsewhere. Consider making these options mutually exclusive at the argument-parsing/validation layer so users get an immediate, clear error if they pass both.
No description provided.