Custom vLLM models#

This page provides an example of serving a custom vLLM model with Ray Serve LLM.

The example model is a custom reward model composed of Qwen3-0.6B with its vocabulary LM head replaced by a scalar reward head. The model reuses vLLM’s Qwen3 backbone, scores the last token, and returns the reward through vLLM’s /classify endpoint.

How does Ray Serve LLM support custom vLLM models?#

Ray Serve LLM serves any architecture that vLLM supports. When your model isn’t a built-in vLLM architecture, you add it with a vLLM plugin: a custom package that registers the architecture in vLLM’s model registry without patching vLLM. Ray Serve LLM then serves it through the same OpenAI-compatible API as any other model.

Prerequisites#

Install Ray with the LLM extra:

pip install "ray[llm]"

Write the plugin#

Compose the plugin as an installable Python package. The following is an example layout:

qwen3-reward-plugin/          # project directory -- run `pip install .` here
├── setup.py                  # packaging metadata and the vLLM entry point
└── qwen3_reward_plugin/      # the importable package
    ├── __init__.py           # register(): adds the architecture to vLLM's model registry
    ├── qwen3_rm.py           # Qwen3CustomRewardModel: the model class
    └── serve_hook.py         # RewardModelServer: registers the plugin in the Ray Serve LLM replica

Model class#

Subclass vLLM’s Qwen3ForCausalLM to reuse its backbone, then attach the reward head and a last-token classification pooler. See the full model class here.

Register the architecture#

vLLM calls register() in every process it starts, the driver, the engine core, and each rank worker, through vLLM’s load_general_plugins().

"""vLLM plugin entry point: register the custom architecture."""
def register() -> None:
    from vllm import ModelRegistry

    if "Qwen3CustomRewardModel" not in ModelRegistry.get_supported_archs():
        ModelRegistry.register_model(
            "Qwen3CustomRewardModel",
            "qwen3_reward_plugin.qwen3_rm:Qwen3CustomRewardModel",
        )

Package it#

Declare the vllm.general_plugins entry point in setup.py. vLLM discovers the package and runs register() when the package installs.

from setuptools import setup

setup(
    name="qwen3-reward-plugin",
    version="0.1.0",
    packages=["qwen3_reward_plugin"],
    # vLLM discovers and runs this entry point in every process (the engine core
    # and each rank worker) via load_general_plugins().
    entry_points={
        "vllm.general_plugins": [
            "qwen3_reward = qwen3_reward_plugin:register",
        ],
    },
)

Install the plugin into the image your Ray cluster runs, alongside vLLM, so every node and every vLLM engine and worker process can discover it:

pip install .  # from the plugin project directory

Deploy the model#

Configure the model as a pooling deployment, enable direct streaming, then launch it with the Python API or a YAML config.

Configure the model#

Set runner="pooling" because a reward model encodes rather than generates, and use engine_kwargs.hf_overrides to select the custom architecture and configure a single regression score (num_labels=1, problem_type="regression" for an identity activation).

The plugin entry point registers the architecture in the vLLM engine and worker processes. Ray Serve LLM also resolves and validates the architecture while it builds the engine configuration, so register it there too: point server_cls at an LLMServer subclass whose import calls register().

"""LLMServer subclass for ``LLMConfig.server_cls``: importing it registers the plugin."""

from ray.llm._internal.serve.core.server.llm_server import LLMServer

from qwen3_reward_plugin import register

register()

class RewardModelServer(LLMServer):
    pass

Enable direct streaming#

Serve with direct streaming so vLLM’s native /classify route is exposed. Export both environment variables before starting Serve:

export RAY_SERVE_ENABLE_HA_PROXY=1
export RAY_SERVE_LLM_ENABLE_DIRECT_STREAMING=1

Deploy#

Deploy with the Python API or an equivalent YAML config:

Python

from ray import serve
from ray.serve.llm import LLMConfig, build_openai_app

# Serve Qwen3-0.6B with a scalar reward head from the `qwen3_reward_plugin`
# vLLM plugin (pip install it into your cluster image).
llm_config = LLMConfig(
    model_loading_config=dict(
        model_id="qwen3-reward",
        model_source="Qwen/Qwen3-0.6B",
    ),
    # vLLM's entry point registers the architecture in the engine and worker
    # processes. Ray Serve LLM also resolves it while building the engine config.
    server_cls="qwen3_reward_plugin.serve_hook:RewardModelServer",
    engine_kwargs=dict(
        runner="pooling",
        hf_overrides=dict(
            architectures=["Qwen3CustomRewardModel"],
            num_labels=1,
            problem_type="regression",
        ),
        max_model_len=4096,
    ),
)

# Requires direct streaming so vLLM's native /classify route is exposed.
app = build_openai_app({"llm_configs": [llm_config]})
serve.run(app, blocking=True)

Save this as app.py and run it to deploy.

YAML

# config.yaml
applications:
- args:
    llm_configs:
        - model_loading_config:
            model_id: qwen3-reward
            model_source: Qwen/Qwen3-0.6B
          # Register the plugin architecture in the Ray Serve LLM processes.
          server_cls: qwen3_reward_plugin.serve_hook:RewardModelServer
          engine_kwargs:
            runner: pooling
            hf_overrides:
                architectures:
                    - Qwen3CustomRewardModel
                num_labels: 1
                problem_type: regression
            max_model_len: 4096
  import_path: ray.serve.llm:build_openai_app
  name: custom_vllm_app
  route_prefix: "/"

Deploy it with serve run custom_vllm_config.yaml.

Query the model#

Run the following command to verify your model is serving. This query sends text to the /classify endpoint and reads the scalar reward from data[0].probs[0]. Until you provide trained reward-head weights (see the next section), this value is arbitrary.