Evaluate RAG with Online Inference#

In this tutorial, we demonstrate how to evaluate a Retrieval-Augmented Generation (RAG) pipeline using an online inference approach. You will learn how to use a deployed LLM service to process evaluation queries, retrieve supporting context, and generate responses.

Here is the architecture diagram for the RAG evaluation pipeline with online inference:

https://raw.githubusercontent.com/ray-project/ray/refs/heads/master/doc/source/ray-overview/examples/e2e-rag/images/online_inference_rag_evaluation.png

Anyscale-Specific Configuration

Note: This tutorial is optimized for the Anyscale platform. When running on open source Ray, additional configuration is required. For example, you’ll need to manually:

Configure your Ray Cluster: Set up your multi-node environment (including head and worker nodes) and manage resource allocation (e.g., autoscaling, GPU/CPU assignments) without the Anyscale automation. See the Ray Cluster Setup documentation for details: https://docs.ray.io/en/latest/cluster/getting-started.html.
Manage Dependencies: Install and manage dependencies on each node since you won’t have Anyscale’s Docker-based dependency management. Refer to the Ray Installation Guide for instructions on installing and updating Ray in your environment: https://docs.ray.io/en/latest/ray-core/handling-dependencies.html.
Set Up Storage: Configure your own distributed or shared storage system (instead of relying on Anyscale’s integrated cluster storage). Check out the Ray Cluster Configuration guide for suggestions on setting up shared storage solutions: https://docs.ray.io/en/latest/train/user-guides/persistent-storage.html.

Prerequisites#

Before you move on to the next steps, please make sure you have all the required prerequisites in place.

Pre-requisite #1: You must have finished the data ingestion in Chroma DB with CHROMA_PATH = "/mnt/cluster_storage/vector_store" and CHROMA_COLLECTION_NAME = "anyscale_jobs_docs_embeddings". For setup details, please refer to Notebook #2.

Pre-requisite #2: You must have deployed the LLM service with `Qwen/Qwen2.5-32B-Instruct` model. For setup details, please refer to Notebook #3.

Initlize the RAG components#

First, initializing the necessary components:

Embedder: Converts your questions into a embedding the system can search with.
ChromaQuerier: Searches our document chunks for matches using the vector DB Chroma.
LLMClient: Sends questions to the language model and gets answers back.

from rag_utils import  Embedder, LLMClient, ChromaQuerier, render_rag_prompt

EMBEDDER_MODEL_NAME = "intfloat/multilingual-e5-large-instruct"
CHROMA_PATH = "/mnt/cluster_storage/vector_store"
CHROMA_COLLECTION_NAME = "anyscale_jobs_docs_embeddings"


# Initialize client
model_id='Qwen/Qwen2.5-32B-Instruct' ## model id need to be same as your deployment 
base_url = "http://localhost:8000/" ## replace with your own service base url
api_key = "fake-key" ## replace with your own api key


# Initialize the components for rag.
querier = ChromaQuerier(CHROMA_PATH, CHROMA_COLLECTION_NAME, score_threshold=0.8)
embedder = Embedder(EMBEDDER_MODEL_NAME)
llm_client = LLMClient(base_url=base_url, api_key=api_key, model_id=model_id)

Load the Evaluation Data#

The evaluation data is stored in a CSV file (evaluation_data/rag-eval-questions.csv) that contains 64 user queries grouped by category.

These queries cover a range of topics—from technical questions about Anyscale and its relationship with Ray, to casual, ethically sensitive, and non-English requests. This diverse dataset helps assess the system’s performance on a wide variety of inputs.

Feel free to add more categories or questions as needed.

import pandas as pd


# Load questions from CSV file
csv_file = "evaluation_data/rag-eval-questions.csv"  # Ensure this file exists in the correct directory
df = pd.read_csv(csv_file)

print("first 5 rows:\n\n", df.head(5))

Evaluate the RAG Pipeline Using Online Inference#

This section shows how to use online inference with the deployed LLM service to evaluate the RAG system. Although this method is straightforward, it might be slow for large datasets. Online inference is best suited for smaller datasets during initial evaluations.

def eval_rag(df, output_csv="eval_results.csv", num_requests=None):
    """
    Process each row in the DataFrame, obtain answers using the LLM client, and save the results to a CSV file.

    Parameters:
        df (pd.DataFrame): DataFrame containing 'category' and 'user_request' columns.
        output_csv (str): The file path to save the CSV results.
        num_requests (int, optional): Number of requests to evaluate. If None, all requests will be evaluated.
    """
    responses = []
    
    # If num_requests is specified, limit the DataFrame to that number of rows.
    if num_requests is not None:
        df = df.head(num_requests)
    
    for idx, row in df.iterrows():
        category = row['category']
        user_request = row['user_request']
        
        # Print the evaluation statement for the user request.
        print(f"Evaluating user request #{idx}: {user_request}")
        
        chat_history = ""
        company = "Anyscale"
        
        # Query for context
        user_request_embedding = embedder.embed_single(user_request)
        context = querier.query(user_request_embedding, n_results=10)
        
        # Create prompt using render_rag_prompt.
        prompt = render_rag_prompt(company, user_request, context, chat_history)
        
        # Get the answer from the chat model client.
        answer = llm_client.get_response(prompt, temperature=0)
        
        responses.append({
            "Category": category,
            "User Request": user_request,
            "Context": context,
            "Answer": answer
        })
    
    # Convert responses to DataFrame and save as CSV
    output_df = pd.DataFrame(responses)
    output_df.to_csv(output_csv, index=False)
    print(f"CSV file '{output_csv}' has been created with questions and answers.")

import os
# CI caps the eval via RAG_EVAL_N; users get the full 63-row eval by default.
eval_rag(df, output_csv="eval_results_online_inference_qwen32b.csv", num_requests=int(os.environ.get("RAG_EVAL_N", "0")) or None)

Evaluate the Results and Improve RAG Quality#

After running the evaluation, open the resulting CSV file (eval_results_online_inference.csv) to review:

The user request.
The retrieved context from the vector store.
The generated answer from the LLM service.

You can manually review the evaluation results, marking responses as good or bad, and refine the prompt iteratively to improve performance.

Save the high-quality responses as a golden dataset for future reference. Once you have a substantial golden dataset, you can leverage more advanced LLMs—potentially with reasoning capabilities—to act as an LLM judge, comparing new RAG results against the golden dataset.

Scalability Considerations: Why Online Inference May Not Be Ideal#

While online inference is simple to implement, it has limitations for large-scale evaluations:

Production Stability: High-volume requests can overload the production LLM API, potentially affecting service stability.
Overhead: Deploying a dedicated evaluation service adds complexity.
Cost: Continuously running production services for evaluation can lead to unnecessary costs if not properly managed.

In the next tutorial, we will demonstrate how to use Ray Data LLM to perform batch inference, which is more scalable and efficient for processing large datasets.