Aller au contenu
login
arrow_backRetour aux issues
adhabnr-ux/evalport #15

Adapter wanted: Humanloop (to_openeval() / from_openeval())

ecoDébutant help wanted good first issue

descriptionDescription

Humanloop (`pip install humanloop`, currently 0.8.38, actively released) is a genuinely good fit for a `good first issue` — its SDK has dedicated `datasets`, `evaluators`, and `evaluations` submodules with a data model that maps onto EvalPort almost directly, verified by importing the real package and reading its types, not guessed from docs: ```python from humanloop.types.datapoint_response import DatapointResponse print(DatapointResponse.model_fields.keys()) # dict_keys(['inputs', 'messages', 'target', 'id']) ``` A `DatapointResponse` (one row of a Humanloop `Dataset`) is `inputs` (a flat key-value dict) or `messages` (chat history), an optional `target`, and an `id` — that's essentially EvalPort's `TestCase.input`/`expected_output`/`id` already. On the results side, `humanloop.types.EvaluationResponse` (an evaluation run) references `EvaluationRunResponse`/`EvaluatorAggregate`, and each per-datapoint judgment comes back as an `EvaluatorLogResponse` with an `EvaluatorLogResponseJudgment` — the per-test-case, per-evaluator score that maps to EvalPort's `GraderResult`. **What to build:** a standalone adapter package at `adapters/humanloop-openeval-adapter/`, following the exact shape of [`adapters/autogen-openeval-adapter`](adapters/autogen-openeval-adapter/) (the reference implementation — read its README and source first). At minimum: - `to_openeval(datapoints, ...)` — converts a list of `DatapointResponse`-shaped dicts (or the real Humanloop objects) into an EvalPort suite, handling both the `inputs` (dict) and `messages` (chat history) input shapes honestly rather than picking one and dropping the other. - `from_openeval(suite)` — the reverse, producing Humanloop-datapoint-ready dicts. - A results-side function converting a real evaluation run's `EvaluatorLogResponse`/`EvaluatorLogResponseJudgment` objects into an EvalPort `ResultSet` — note `EvaluatorReturnTypeEnum` means a judgment can be boolean, numeric, select/categorical, or text, so (like several existing adapters) a non-numeric judgment should honestly become `score: null` with the raw judgment preserved in `metadata`, not a fabricated number. - `pyproject.toml` depending on `evalport-sdk`, with a real, verified-minimum pinned extra for `humanloop` per [`CONTRIBUTING.md`'s packaging convention](.github/CONTRIBUTING.md#adding-a-new-converter) (install fresh, run the tests, pin what you actually verified). - `tests/` with a round-trip test validated against the real `openeval.validate.validate_suite()`/`validate_result_set()` — not a mock. **Definition of done:** builds and installs cleanly, tests pass locally against the real installed `humanloop` package, the round-trip validates against the real EvalPort spec, and the README explains install + a two-line usage example — same bar as every other adapter in `adapters/`. Comment here if you want to talk through the `inputs`-vs-`messages` mapping or the judgment-type handling before writing code — happy to help either way. Full contributor guide: [`.github/CONTRIBUTING.md`](.github/CONTRIBUTING.md).
codeOuvre sur GitHub