Aller au contenu
login
arrow_backRetour aux issues
adhabnr-ux/evalport #16

Adapter wanted: Vellum (to_openeval() / from_openeval())

ecoDébutant help wanted good first issue

descriptionDescription

Vellum (`pip install vellum-ai`, currently 1.14.7, actively released) is about as literal a naming match to EvalPort as any framework in this ecosystem gets — its SDK's own vocabulary is "Test Suite" / "Test Case" / "Test Suite Run." Verified directly against the real installed package, not the docs: ```python import vellum.types as t print(t.TestSuiteTestCase.model_fields.keys()) # ['id', 'external_id', 'label', 'input_values', 'evaluation_values'] print(t.TestSuiteRunExecution.model_fields.keys()) # ['id', 'test_case_id', 'outputs', 'metric_results'] print(t.TestSuiteRunExecutionMetricResult.model_fields.keys()) # ['metric_id', 'outputs', 'metric_label', 'metric_definition'] ``` `TestSuiteTestCase` → EvalPort `TestCase` almost directly (`id`/`external_id` → `id`, `input_values` → `input`, `evaluation_values` → `expected_output`/context). `TestSuiteRunExecution` → EvalPort `Result` (`test_case_id` links back, `outputs` is the actual output, `metric_results` is the list of per-metric grades). `TestSuiteRunExecutionMetricResult` → EvalPort `GraderResult` (`metric_id`/`metric_label` → `grader_id`, `outputs` carries the metric's actual value — note it's a typed union, `TestSuiteRunMetricNumberOutput`/`StringOutput`/`JsonOutput`/`ErrorOutput`/`ArrayOutput`, so only the numeric variant should become a real `score`; the rest need the same honest `score: null` + raw-value-in-metadata treatment other adapters in this repo use for non-numeric grader results). One real wrinkle worth flagging up front, since it affects the input-mapping design: `input_values`/`evaluation_values` on `TestSuiteTestCase` are lists of `NamedTestCase*VariableValue` — a *typed, named* variable system (`NamedTestCaseStringVariableValue`, `NamedTestCaseChatHistoryVariableValue`, `NamedTestCaseJsonVariableValue`, `NamedTestCaseArrayVariableValue`, and others), not a flat dict. Worth reading `vellum/types/named_test_case_*_variable_value.py` directly before deciding how to flatten it into EvalPort's `input: string | string[]`. **What to build:** a standalone adapter package at `adapters/vellum-openeval-adapter/`, following the exact shape of [`adapters/autogen-openeval-adapter`](adapters/autogen-openeval-adapter/) (the reference implementation — read its README and source first). At minimum: `to_openeval()`/`from_openeval()` for `TestSuiteTestCase` ↔ an EvalPort suite, and a results-side function converting `TestSuiteRunExecution`/`TestSuiteRunExecutionMetricResult` into an EvalPort `ResultSet`, handling the typed-output-union honestly as described above. `pyproject.toml` depending on `evalport-sdk` with a real, verified-minimum pinned `vellum` extra per [`CONTRIBUTING.md`'s packaging convention](.github/CONTRIBUTING.md#adding-a-new-converter). `tests/` with a round-trip test validated against the real `openeval.validate.validate_suite()`/`validate_result_set()`. **Definition of done:** builds and installs cleanly, tests pass locally against the real installed `vellum-ai` package, the round-trip validates against the real EvalPort spec, README with install + a two-line usage example — same bar as every other adapter in `adapters/`. Comment here if you want to talk through the named-variable-value flattening before writing code. Full contributor guide: [`.github/CONTRIBUTING.md`](.github/CONTRIBUTING.md).
codeOuvre sur GitHub