Interop idea: EvalRun/EvalResult ↔ EvalPort (open interchange format for eval results)

[adhabnr-ux]

Context

I'm the maintainer of EvalPort — an Apache-2.0 open interchange spec (TestCase / Grader / ResultSet) for portable LLM evaluation data, with a Python/TS SDK (evalport-sdk) and 32 real framework adapters merged so far (DeepEval, Ragas, MLflow, LangSmith, Braintrust, Phoenix, Weave, and others — full list in the README). I'm not affiliated with FlowyML — just went through flowyml/evals/ because your GenAI eval stack (scorer breakdown, baseline comparison, regression alerts) is one of the more complete ones I've seen in an ML pipeline framework, and it maps onto EvalPort surprisingly cleanly. Filing this as an idea, not a PR — happy to build it if there's interest.

Where it maps

Looking at flowyml/evals/dataset.py, core.py, and run.py:

• EvalDataset.create_genai(name, examples=[{"inputs", "outputs", "expected", "context"}, ...]) — each example already carries a pre-generated outputs, not just a prompt. That's the same shape DeepEval's own adapter (adapters/deepeval-openeval-adapter) handles: grading outputs that were produced elsewhere, rather than EvalPort's own run command generating them. inputs/expected/context line up directly with EvalPort's TestCase.input / expected_output / context; outputs lines up with ResultSet.results[].actual_output.
• Scorer.score() / score_batch() → ScorerFeedback(name, value, rationale, passed, metadata) (in evals/core.py) maps almost field-for-field onto EvalPort's grader_results[] entries (grader_id, score, passed, reason, metadata). Scorers without a matching EvalPort grader type (exact_match/regex/llm_judge/etc.) would just serialize as type: "custom".
• EvalResult.summary (per-scorer aggregate) and EvalResult.scores (per-scorer, per-example feedback) → ResultSet.summary.by_grader and ResultSet.results[].grader_results.
• EvalRun.compare_with() / EvalResult.regressions_from() — this is the part I'd actually flag as valuable both ways: once a FlowyML EvalRun is serialized as an EvalPort ResultSet, it can be diffed against a ResultSet produced by any other EvalPort adapter (Ragas, DeepEval, MLflow, whatever produced the baseline), not just another FlowyML run. Regression detection stops being FlowyML-only.

Concrete sketch

Something like this, using the real field names above:

def to_openeval(dataset: "EvalDataset", result: "EvalResult") -> dict:
    """FlowyML EvalDataset + EvalResult -> EvalPort ResultSet."""
    examples = dataset.data if dataset.data_format == "genai" else []
    results = []
    for i, ex in enumerate(examples):
        grader_results = []
        for scorer_name, feedbacks in result.scores.items():
            fb = feedbacks[i] if i < len(feedbacks) else feedbacks[0]
            grader_results.append({
                "grader_id": scorer_name,
                "type": "custom",
                "score": fb.value if isinstance(fb.value, (int, float)) else None,
                "passed": bool(fb.passed) if fb.passed is not None else True,
                "reason": fb.rationale,
                "metadata": fb.metadata,
            })
        results.append({
            "test_case_id": ex.get("id", f"tc_{i}"),
            "actual_output": ex.get("outputs"),
            "grader_results": grader_results,
            "passed": all(g["passed"] for g in grader_results),
        })

    return {
        "version": "1.0.0",
        "suite_id": dataset.name,
        "suite_version": dataset.version or "0",
        "run_id": result.eval_id,
        "started_at": result.created_at,
        "completed_at": result.created_at,
        "results": results,
        "summary": {
            "total": len(results),
            "passed": sum(r["passed"] for r in results),
            "failed": sum(not r["passed"] for r in results),
            "pass_rate": result.pass_rate,
            "by_grader": {k: {"avg_score": v} for k, v in result.summary.items()},
        },
        "metadata": {"experiment": result.experiment, **result.metadata},
    }

from_openeval() would go the other way — turn a ResultSet (e.g. one exported by the Ragas or DeepEval adapters) into an EvalDataset + EvalResult, so FlowyML's EvalRun.compare_with() / regression alerting could run against eval results that never touched FlowyML.

Comparable adapters already merged

For reference, the shape and rigor I'd aim for — real to_openeval()/from_openeval(), tests against EvalPort's actual validator, a README:

• adapters/deepeval-openeval-adapter — closest analog, since it's also grading pre-generated outputs rather than driving generation.
• adapters/ragas-openeval-adapter — relevant since your README already lists Ragas as one of FlowyML's built-in adapter integrations, so a flowyml-openeval-adapter would sit right alongside it in spirit.

The actual ask

Is this something the FlowyML maintainers would want either as:

1. A flowyml-openeval-adapter package living in EvalPort's adapters/ (same pattern as the DeepEval/Ragas ones), maintained by me with FlowyML as the upstream reference, or
2. A small to_openeval()/from_openeval() example living in flowyml/evals/ itself, if you'd rather it be first-party?

No pressure either way — genuinely just flagging that the shapes line up unusually well and I'd rather ask than build something in the dark. Happy to open a draft PR against whichever repo makes more sense once there's a steer.

— Sahi, independent contributor (not affiliated with this project)

[piotrlaczkowski]

Hello @adhabnr-ux !!! Thank you for this initiative !

I think this might actually better fit with my second project: AgentoMatic -> dedicated to GenAI and optimization, so do not hesitate to take a look and propose improvement.

For not I did not hear, nor see any extension proposition from the community, so I suppose there is no particular interest ... ?
Hard to tell ... but do not hesitate to contribute to this one as well, maybe as a plugin / module -> this might be quite an idea !