Docs   |   Discord

Thabit

Open source platform to evaluate multiple LLM models with custom datasets.

Demo

Installation

pip3 install thabit

Test

pytest tests

Build

pip3 install -e .

Docs

Visit https://docs.thabit.ai

TODO:

• Validate the input dataset.

• UI for adding/editing config.

• Visulaise Output (using UI).

• Run eval per dataset (add folders for dataset and for evals). This is to simplify visualising results later using the UI.

  root
  ├── datasets
  │ └── a
  └── evals
    └── a