The problem

Ragas evaluates LLM applications, but it needs an LLM to act as the judge. Teams on Oracle Cloud Infrastructure had no direct path: the only route to OCI Generative AI models was through LangChain or LlamaIndex wrappers, which meant extra dependencies and an extra layer to debug just to run an evaluation.

Separately, while working in the codebase I found a validator that looked correct but never ran: MultiTurnSample.validate_user_input() checked a generator expression, which is always truthy, so its type check could never fail.

Approach

PR #2321 — native OCI Generative AI support

• —An OCIGenAIWrapper that extends BaseRagasLLM and talks to the OCI Python SDK directly, with no LangChain or LlamaIndex in the path.
• —An oci_genai_factory() helper so setup is one call, with support for the OCI CLI config, environment variables, or a manual config dict.
• —Full async/await support, error handling for authentication, missing-model and permission failures, and usage tracking through Ragas analytics events.
• —The OCI SDK ships as an optional dependency (ragas[oci]) and imports fail gracefully when it is absent, so existing installs are untouched. No breaking changes.
• —An integration guide, a runnable example script, and an entry in the integrations index so the feature is discoverable, not just merged.

PR #2426 — fixing a validator that never validated

The fix was one line, but the more useful part was proving it: two new tests that fail on the old logic, one rejecting invalid message types and one accepting valid HumanMessage, AIMessage and ToolMessage inputs.

Tech deep-dive

The OCI wrapper follows the same pattern as the other Ragas LLMs, so it drops into evaluate() with no special casing:

from ragas import evaluate
from ragas.llms import oci_genai_factory

llm = oci_genai_factory(
    model_id="cohere.command",
    compartment_id="ocid1.compartment.oc1..example",
)

result = evaluate(dataset, llm=llm)

The validator bug is a classic Python trap. A bare generator expression is an object, and objects are truthy, so the guard below can never trigger. Wrapping it in all() makes it evaluate every message:

# before: always truthy, never validates
if not (isinstance(m, (HumanMessage, AIMessage, ToolMessage)) for m in messages):
    raise ValueError(...)

# after
if not all(isinstance(m, (HumanMessage, AIMessage, ToolMessage)) for m in messages):
    raise ValueError(...)

Outcomes & learnings

Both PRs were reviewed and merged into the main Ragas repository. OCI Generative AI is now a first-class LLM option for anyone running Ragas evaluations on Oracle Cloud, with tests, docs and an example behind it.

The work sits close to my day job building enterprise AI at Oracle: evaluation is what makes LLM systems trustworthy enough to ship, and contributing to the tooling that does it is the same instinct as my retail AI evaluation patent.