vLLM
Load vLLM module
You’ll first need to load the vLLM module
module load vllmOffline Batched Inference
Create a vLLM python script. E.g.
from vllm import LLM, SamplingParams
prompts = [
"Hello, my name is",
"The president of the United States is",
"The capital of France is",
"The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
llm = LLM(model="Qwen/Qwen2.5-1.5B-Instruct")
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
prompt = output.prompt
generated_text = output.outputs[0].text
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")Run your python script
python my-vllm-script.pyOnline Serving
To protect your vLLM service, it’s best to set an API key
export VLLM_API_KEY="random_string_goes_here"Start the vLLM service and save the process ID (so you can terminate it later). The command below will start the service and send output to a file called vllm.log. Note that by default, vLLM starts on port 8000. You may need to specify a different port if that one is already in use.
vllm serve Qwen/Qwen2.5-1.5B-Instruct --port 8000 --enable-auto-tool-choice --tool-call-parser hermes vllm.log 2>&1 &
PID=$!Wait for the service to start (note that this may take a couple of minutes). Once it’s running, you can now send a query to the vLLM service. Note that you need to substitute API_KEY_HERE with your actual API key from above. E.g.
curl http://localhost:8000/v1/completions \
-H "Authorization: Bearer API_KEY_HERE"
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-1.5B-Instruct",
"prompt": "San Francisco is a",
"max_tokens": 7,
"temperature": 0
}'Kill the vLLM service when finished
kill $PID