arrow_backRetour aux issues
adhabnr-ux/evalport
#15
Débutant
Ouvrirarrow_forward
Débutant
Ouvrirarrow_forward
Débutant
Ouvrirarrow_forward
Adapter wanted: Humanloop (to_openeval() / from_openeval())
ecoDébutant
help wanted
good first issue
descriptionDescription
Humanloop (`pip install humanloop`, currently 0.8.38, actively released) is a genuinely good fit for a `good first issue` — its SDK has dedicated `datasets`, `evaluators`, and `evaluations` submodules with a data model that maps onto EvalPort almost directly, verified by importing the real package and reading its types, not guessed from docs:
```python
from humanloop.types.datapoint_response import DatapointResponse
print(DatapointResponse.model_fields.keys())
# dict_keys(['inputs', 'messages', 'target', 'id'])
```
A `DatapointResponse` (one row of a Humanloop `Dataset`) is `inputs` (a flat key-value dict) or `messages` (chat history), an optional `target`, and an `id` — that's essentially EvalPort's `TestCase.input`/`expected_output`/`id` already. On the results side, `humanloop.types.EvaluationResponse` (an evaluation run) references `EvaluationRunResponse`/`EvaluatorAggregate`, and each per-datapoint judgment comes back as an `EvaluatorLogResponse` with an `EvaluatorLogResponseJudgment` — the per-test-case, per-evaluator score that maps to EvalPort's `GraderResult`.
**What to build:** a standalone adapter package at `adapters/humanloop-openeval-adapter/`, following the exact shape of [`adapters/autogen-openeval-adapter`](adapters/autogen-openeval-adapter/) (the reference implementation — read its README and source first). At minimum:
- `to_openeval(datapoints, ...)` — converts a list of `DatapointResponse`-shaped dicts (or the real Humanloop objects) into an EvalPort suite, handling both the `inputs` (dict) and `messages` (chat history) input shapes honestly rather than picking one and dropping the other.
- `from_openeval(suite)` — the reverse, producing Humanloop-datapoint-ready dicts.
- A results-side function converting a real evaluation run's `EvaluatorLogResponse`/`EvaluatorLogResponseJudgment` objects into an EvalPort `ResultSet` — note `EvaluatorReturnTypeEnum` means a judgment can be boolean, numeric, select/categorical, or text, so (like several existing adapters) a non-numeric judgment should honestly become `score: null` with the raw judgment preserved in `metadata`, not a fabricated number.
- `pyproject.toml` depending on `evalport-sdk`, with a real, verified-minimum pinned extra for `humanloop` per [`CONTRIBUTING.md`'s packaging convention](.github/CONTRIBUTING.md#adding-a-new-converter) (install fresh, run the tests, pin what you actually verified).
- `tests/` with a round-trip test validated against the real `openeval.validate.validate_suite()`/`validate_result_set()` — not a mock.
**Definition of done:** builds and installs cleanly, tests pass locally against the real installed `humanloop` package, the round-trip validates against the real EvalPort spec, and the README explains install + a two-line usage example — same bar as every other adapter in `adapters/`.
Comment here if you want to talk through the `inputs`-vs-`messages` mapping or the judgment-type handling before writing code — happy to help either way. Full contributor guide: [`.github/CONTRIBUTING.md`](.github/CONTRIBUTING.md).
Issues similaires
calkit/calkit
star53
Poids du dépôt moyen
VS Code extension should be robust to YAML parser errors
Seeing this error: ``` Failed to read calkit.yaml: YAMLParseError: A block sequence may not be used as an implicit map…
Python
bug
good first issue
fu351/Doberman-Core
star211
Poids du dépôt léger
dash: a manual Refresh control
The dashboard polls: `refreshStats()` (`src/doberman/dash/app.py:408`) every 5 s and `refreshPending()` (`:546`) every …
Python
enhancement
good first issue
fu351/Doberman-Core
star211
Poids du dépôt léger
dash: "Copy details" button on each pending-approval card
Each pending-approval card in the dashboard (`renderPending`, `src/doberman/dash/app.py:448-544`) shows the risk badge,…
Python
enhancement
good first issue