Benchmarking SGLang (gpt-oss-120b) on NVIDIA Spark (GB10): Toward AI Performance on the First AI Personal Computer

By : Jason Chan, Jackie Siu, John Cheung, 12 Nov 2025

Abstract

This report benchmarks the SGLang (gpt-oss-120b) model on NVIDIA Spark (GB10)—an AI-capable personal computer designed for on-device large language model (LLM) inference. Using a PowerShell-based load-testing framework, we evaluate throughput, latency, and concurrency scaling from 1 to 20 simultaneous users. Results show that SGLang scales efficiently up to ~12 concurrent users, with a sustained aggregate wall-clock throughput of ~175 tokens/sec at full load. The findings demonstrate that modern AI PCs can deliver multi-user inference performance, reinforcing their role in hybrid local–cloud AI architectures.

1. Introduction

The emergence of AI Personal Computers (AI PCs), exemplified by the NVIDIA Spark (GB10), enables desktop-level LLM inference that previously required datacenter GPUs. Running inference locally improves privacy, latency, and cost predictability, especially in research and institutional environments. This report examines the performance of the SGLang runtime using CUDA Graph batching on the Spark (GB10). We benchmark real concurrency scaling from 1 to 20 user sessions, focusing on throughput (tokens/sec) and latency (p95) to understand GPU saturation behavior and efficiency.

2. Experimental Setup

2.1 Hardware and Environment

• System: NVIDIA Spark (GB10)
• Model: openai/gpt-oss-120b
• Framework: SGLang (CUDA Graph enabled)
• Container: lmsysorg/sglang:spark
• Concurrent load: 1–20 users (10 requests per user)
• Output length: ~500 words per response

Server Launch Command:

sudo docker run --gpus all \
  --shm-size 32g \
  -p 30000:30000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v ~/tiktoken_encodings:/tiktoken_encodings \
  --env "HF_TOKEN=<REDACTED>" \
  --ipc=host \
  lmsysorg/sglang:spark \
  python3 -m sglang.launch_server \
    --model-path openai/gpt-oss-120b \
    --host 0.0.0.0 --port 30000 \
    --reasoning-parser gpt-oss --tool-call-parser gpt-oss

2.2 Load Testing Methodology

A PowerShell-based test harness issued 10 requests per user at each concurrency level (1–20). Each request prompted the model to write ~500 words. The harness recorded: p95 latency (ms) per request, average tokens/sec per request, and combined throughput (wall-clock) for all requests. Timeouts scaled from 180s to 262s with concurrency to avoid premature terminations.

Client Script (excerpt)

$Endpoint = "http://<REDACTED>:30000/v1/chat/completions"
$RequestsPerUser = 10
$MaxConcurrency = 20
$Body = '{
  "model": "default",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Write approximately 500 words about tokenization and throughput in large language models."}
  ],
  "temperature": 0.7,
  "max_tokens": 2048
}'

3. Results

3.1 Performance Summary

3.2 Visualization

Figure 1 — Combined Throughput (Wall-Clock) vs. Concurrent Users

Figure 2 — p95 Latency vs. Concurrent Users (seconds)#

Figure 3 — Per-Request Avg Tokens/s vs. Concurrent Users

4. Discussion

The Spark GB10 scales linearly up to approximately 12 concurrent users, achieving a maximum wall-clock throughput of about 175 tokens/sec with minimal deviation across repeated tests. This reflects excellent scheduling efficiency in SGLang’s CUDA Graph runtime and effective batching across multiple sequences. The close correlation between per-user and wall-clock throughput values demonstrates low queuing overhead and optimized kernel utilization.

Latency increased proportionally to concurrency, which aligns with expected GPU saturation behavior. Beyond 12 users, throughput gain diminished due to full decode pipeline utilization, confirming that the Spark GB10 GPU reached its optimal operating point. Nevertheless, the system maintained full stability without timeouts or dropped requests even at 20 concurrent clients, illustrating both the reliability and efficiency of the serving stack.

These findings are significant because they showcase the ability of a single-device AI PC to deliver multi-user inference. This suggests that local AI workstations can serve as cost-effective, privacy-preserving compute nodes in hybrid deployment topologies.

5. Conclusion and Future Work

SGLang (gpt-oss-120b) running on NVIDIA Spark (GB10) demonstrated robust multi-user inference scaling. Single-user throughput averaged about 52 tokens/sec, while full-load (20 users) performance reached 175 tokens/sec combined throughput. The absence of any errors or request timeouts throughout all tests underscores the maturity of SGLang’s inference stack. These results validate the Spark GB10 as a capable AI PC platform for both research and production workloads.

Future evaluations will include multi-node scaling using ConnectX-7, and comparative benchmarking of alternative frameworks such as vLLM and TensorRT-LLM to further explore edge-level deployment optimization.

Similar to other unified memory platforms such as the Mac Studio, the NVIDIA Spark demonstrates both the advantages and limitations of this architecture. As AI PCs gain popularity, what software innovations or adaptations will be needed to fully leverage this new trend?

Appendix

A. Client Script (excerpt) — see Section 2.2 for the PowerShell harness.

B. Server Log Excerpts:

[2025-11-05 09:53:53] Decode batch. #running-req: 19, gen throughput (token/s): 174.16
[2025-11-05 10:00:09] Decode batch. #running-req: 20, gen throughput (token/s): 178.62
[2025-11-05 10:06:15] Decode batch. #running-req: 20, gen throughput (token/s): 185.46

References

1. L. Zheng et al., “SGLang: Fast Structured Generation with Large Language Models,” arXiv:2403.12031, 2024.

2. NVIDIA Corporation, “NVIDIA Spark gb10 Technical Overview,” NVIDIA Whitepaper, 2025.

#Latency is a critical aspect of LLM performance because it directly affects user-perceived responsiveness. While throughput indicates system capacity, latency reveals how efficiently requests are scheduled and executed under load. Monitoring the 95th percentile (p95) latency helps identify bottlenecks and ensures interactive stability in multi-user environments.