LATEST
AI Tools

SGLang Consortium: NVIDIA, AMD, Intel and Google Unite Behind Open AI Inference

Updated November 16, 2026
13 min read
Editor's Note: This article was human-reviewed for accuracy
Row of server towers with blue and red lighting in a data center

Quick Answer & Key Takeaways

  • • LMSYS Org launched the SGLang Consortium on November 12, 2026, with NVIDIA, AMD, Intel, Google, Meta, Cloudflare, Modal and RadixArk as founding members.
  • • The group will shape the roadmap of SGLang, the open-source inference engine behind some of the largest AI deployments, while its maintainers keep final say on technical decisions.
  • • SGLang v1.0 arrived the same week, with day-zero support for 217 models and adoption across more than 330 organizations.

On November 12, 2026, eight companies that normally compete for the same AI infrastructure budgets agreed to share a roadmap. LMSYS Org, the nonprofit behind the open-source inference engine SGLang, launched the SGLang Consortium with NVIDIA, AMD, Intel, Google, Meta, Cloudflare, Modal and RadixArk as founding members. The announcement came in the opening keynote of the first SGLang Summit in San Francisco.

The timing is not an accident. Inference, the work of actually running a model after it is trained, is expected to account for roughly two-thirds of all AI compute in 2026, up from about half in 2025, according to Deloitte figures cited by LMSYS. The engine that serves a model decides how fast that compute runs and what it costs on each company's hardware. That is the layer these eight companies just agreed to build together, in the open, instead of each building its own.

What SGLang actually is

SGLang is an open-source inference framework for large language models, vision-language models and diffusion models. In plain terms, it is the software that sits between a model and its users: it takes incoming requests, schedules them across GPUs, manages memory, and returns generated text, images or video. It is the same layer DeepSeek open-sourced its own production engine for, DeepInfer, and SGLang's GitHub repository describes it as optimized for agentic workloads, large-scale serving and reinforcement-learning rollouts, the three workloads that now dominate production AI.

Its core idea is RadixAttention. When many requests share the same opening text, a system prompt, a conversation history, a set of tool definitions, SGLang stores the computed work for that shared prefix once in a radix tree and reuses it for every request that matches. Later requests that hit the same prefix skip the recomputation. That matters most for chatbots, agents and retrieval pipelines, where the same long prompt is sent again and again.

The project is Apache 2.0 licensed, free to use commercially, and hosted by LMSYS Org. When we checked on November 15, 2026, the repository showed 38,905 stars and 10,512 forks, with 22,555 commits since it was created in January 2024. LMSYS says SGLang runs on hundreds of thousands of GPUs and powers production inference at Meta, Microsoft, Google, Amazon, Thinking Machines Lab, Cloudflare, Reflection and Modal. The project's own PyPI page puts the figure at more than 400,000 GPUs worldwide. Both numbers come from the project itself, not from an independent audit.

Hardware support is unusually wide for an inference engine. The README lists NVIDIA GPUs from A100 through B200 and GB300, AMD Instinct MI300X through MI355X, Google TPUs, Intel Arc GPUs and Xeon CPUs, Apple Silicon, Huawei Ascend NPUs and Moore Threads GPUs. AWS Trainium support arrived this week through a separate plugin, covered below. The TPU side of that story connects to Google's own eighth-generation chips, which we covered in our TPU 8tx and 8i Max piece, and to the broader question of escaping NVIDIA's CUDA lock-in, which we compared in TorchTPU vs TorchNeuron.

What the consortium is, and what it is not

The consortium is a governance body, not a product launch. No new feature shipped with the announcement itself. Each founding member names a technical representative and joins two roadmap meetings a year, contributing engineering expertise, deployment feedback, testing resources or joint development. Working groups may collaborate between meetings, with public updates on roadmap direction and progress.

Five focus areas were named for the group's initial work: support for frontier models, reasoning workloads, multimodal inference, distributed serving, and hardware optimization. LMSYS was explicit about one point: its maintainers keep responsibility for technical decisions and code acceptance, and the project stays open to the entire community. Membership buys a seat at the roadmap table, not control of the project.

The member list is the story. NVIDIA, AMD and Intel build competing accelerators. Google and Meta run some of the largest inference fleets in the world. Cloudflare and Modal serve models to customers. RadixArk, founded by Ying Sheng and Banghua Zhu, employs several of SGLang's original creators and core maintainers. Ying Sheng, co-author of the original SGLang paper and RadixArk's CEO, said the teams running SGLang at the largest scale "see what inference needs next before anyone else does." NVIDIA's Ian Buck said better software helps developers get more performance from NVIDIA compute and that lower inference costs make AI more accessible. AMD's Ramine Roane said SGLang gives developers a fast, open inference engine that runs across hardware, including AMD Instinct GPUs.

The summit itself ran November 12 to 13 at the Fort Mason Center for Arts and Culture in San Francisco, with more than 100 speakers across reinforcement learning, multimodal serving and hardware sessions.

What shipped the same week: SGLang v1.0

The consortium was not the only announcement at the summit. SGLang also released version 1.0, a milestone the project announced with adoption across more than 330 organizations and day-zero support for 217 models, meaning those models were supported the day they were released. The v1.0 announcement lists Rust-rewritten hot paths, a zero-overhead scheduler, full-graph coverage, speculative decoding, unified memory, prefill-decode disaggregation and a hierarchical cache. Detailed release notes were still pending at announcement time, with SGLang saying details were coming soon, so treat the feature list as the announcement's own summary rather than a finished changelog.

A reinforcement-learning framework, Miles v0.2, shipped alongside v1.0. One figure from the summit deserves a flag rather than a headline: slides shown at the summit put agent-written code at about 95 percent of merged pull requests, up from none a year earlier. That is a claim from the project's own slides, reported by summit coverage, not an independently verified statistic.

The day-zero support number is worth pausing on. A model that is supported on release day can be served in production immediately, without waiting for the engine to catch up. SGLang has built its reputation on exactly that, with day-one support for DeepSeek V3 and R1 in January 2025 and for OpenAI's gpt-oss in August 2025, according to the project's own release history.

SGLang-Neuron: the engine now runs on AWS Trainium

The most concrete software release of the week was SGLang-Neuron v0.1.0, a plugin co-developed by the SGLang team at RadixArk and AWS Annapurna Labs that brings the SGLang serving stack to AWS Trainium as a native PyTorch plugin. AWS lists it as public preview as of November 12, 2026, after a closed beta in October. It is open source under Apache 2.0 and reached through an AWS Deep Learning Container, with the same SGLang interface developers already use.

The release post's tables list 17 models in all: 12 language models and 5 diffusion models, by FourWeekMBA's count of the tables. The largest deployment named in the release is Kimi-K3, which runs across three Trainium3 nodes, one for prefill and two for decode. GLM-5.2 serves a 1M-token context. The one throughput figure in the release is for the hierarchical cache, which keeps evicted cache pages in host memory: on GPT-OSS-120B, the team reports 72 to 75 percent more output throughput. That figure covers one model and one cache setting, measured by the developers, not independently.

Two limits matter. First, public preview is not general availability: AWS and the SGLang team say features, APIs and supported models can change. Second, neither AWS nor the SGLang team compares Trainium serving speed or cost with NVIDIA GPUs in this release, and neither gives prices for running these models on Trainium.

Intel and RadixArk announced a separate collaboration the same day, to optimize SGLang for Intel CPUs, GPUs and accelerators, including support for Intel Advanced Matrix Extensions (AMX), with day-zero readiness for Intel hardware as the stated goal. No measured performance results or release dates came with that announcement.

SGLang vs vLLM: the comparison that decides most deployments

SGLang's main rival is vLLM, the inference engine that began in UC Berkeley's Sky Computing Lab and introduced PagedAttention, the technique of treating the KV cache like operating-system virtual memory, allocating it in fixed-size pages on demand instead of reserving one large block per request. Both engines are Apache 2.0, both expose an OpenAI-compatible API, and both do continuous batching. The differences are in what they cache, what hardware they cover, and who governs them.

SGLangvLLM
LicenseApache 2.0Apache 2.0
GitHub stars (checked November 15, 2026)38,90596,971
Forks10,51224,862
Project startedJanuary 2024February 2023
Core caching ideaRadixAttention: radix tree over shared KV-cache prefixes, automatic reusePagedAttention: KV cache in fixed-size pages, automatic prefix caching as an option
Model supportDay-zero support for 217 models at v1.0; strong on DeepSeek, Kimi and other large mixture-of-experts models200+ model architectures on Hugging Face, including encoder-decoder models
HardwareNVIDIA, AMD, Google TPU, Intel, Apple Silicon, Huawei Ascend, Moore Threads; AWS Trainium via SGLang-Neuron public previewNVIDIA, AMD, Intel GPUs and CPUs, Google TPU, Intel Gaudi, Huawei Ascend, Apple Silicon and more
GovernanceLMSYS Org plus the new SGLang Consortium; maintainers keep technical decisionsCommunity project from UC Berkeley's Sky Computing Lab, with contributors from many institutions and companies
API accessYes, OpenAI-compatible API serverYes, OpenAI-compatible API server plus Anthropic Messages API and gRPC

Which one is faster depends on the traffic, and an independent comparison updated November 2, 2026 makes the point bluntly: there is no single, current, independent head-to-head between the two engines. The comparisons that exist are either first-party or third-party one-offs on one model and one machine. The same comparison's practitioner reports show the winner flipping by model family, with one reporter finding SGLang about 20 percent faster on a text model and about 30 percent slower on the vision variant of the same model, on the same stack.

The usable rule from that comparison is about workload shape, not brand. If more than half of your requests share a long prefix, a fixed system prompt, a conversation history, an agent loop, SGLang's RadixAttention reuses that prefix automatically and is the engine to test first. If every prompt is unique, or you are memory-bound before you are compute-bound, vLLM's PagedAttention is the lever that matters. For DeepSeek, Kimi and other large mixture-of-experts models, SGLang's day-zero support and tuned kernels give it an edge. For encoder-decoder models, vLLM is the listed option.

One row in that November 2 comparison is already out of date, and it shows why this week matters. It listed AWS Trainium as hardware vLLM covers and SGLang does not. SGLang-Neuron's public preview, announced November 12, closes that gap, though as a preview rather than a finished release. Check the current support matrix before deciding on hardware grounds.

What it costs

SGLang itself costs nothing. The Apache 2.0 license covers commercial use, there is no paid tier of the engine, and no hosted SGLang service with a per-token price exists to quote. The bill is the compute it runs on: the GPUs, TPUs or Trainium chips, the machines around them, and the engineers who operate the deployment. Enterprise adoption help, technical consulting and sponsorship go through LMSYS Org, which lists a contact address rather than a published price.

That is the honest way to read the consortium, too. Nothing in the announcement changes what anyone pays for inference this week. What changes is who sets the priorities for the software that determines how much inference costs on each chip. If the consortium's joint engineering work lands in the open codebase, the savings show up as lower serving costs for everyone running SGLang, on any member's hardware. If it stays a press release, nothing changes. The test is merged code and public cross-vendor benchmarks, not the founding-member list.

What the announcement does not mean

Four mistakes are easy to make with this story. First, the consortium shipped no code. It is a governance announcement, and its value will be decided by what its members merge into the project over the next year. Second, eight famous members do not prove the engine is faster. No independent benchmark accompanied the launch, and the performance figures attached to SGLang v1.0 and SGLang-Neuron are the projects' own measurements on their own workloads. Third, the big adoption numbers are project-reported: more than 330 organizations, day-zero support for 217 models, more than 400,000 GPUs. They are plausible given the project's production users, but no third party has audited them. Fourth, the 95 percent agent-written pull request figure is a slide from the summit, not a verified engineering metric. Treat it as a signal about how the project builds software, not as a fact about code quality.

One more caution for anyone choosing an engine this week: the v1.0 detailed release notes were still pending when the milestone was announced, and SGLang-Neuron is a public preview whose features and supported models can change. If you deploy either, pin the version and read the notes when they land.

How to try it

A simple path is the Docker image, which includes SGLang and its dependencies: docker pull lmsysorg/sglang:latest. Alternatively, install it in a Python environment with uv pip install --prerelease=allow sglang. The project's quickstart walks through launching a model and sending a first request, and its cookbook gives ready-to-run launch commands by model and hardware. The documentation lives at docs.sglang.io.

For Trainium, the path is the AWS Deep Learning Container for SGLang-Neuron, with the plugin selected at server start. For Intel hardware, the optimization work was announced without a release date, so check the project's platform guides before planning a deployment on it.

Is SGLang free to use?

Yes. SGLang is Apache 2.0 licensed, which covers commercial use with no license fee. You pay for the hardware it runs on and the people who operate it, not for the software. The consortium changes nothing about the license.

Does the consortium change who owns SGLang?

No. SGLang stays hosted by LMSYS Org, and its maintainers keep responsibility for technical decisions and code acceptance. Founding members name technical representatives and join two roadmap meetings a year to shape priorities. The project stays open to the whole community, not just to members.

SGLang or vLLM: which should I pick?

It depends on your traffic. Shared prefixes, multi-turn chat and agent loops favor SGLang, because RadixAttention reuses shared prefixes automatically. Unique one-shot prompts and memory-bound batches favor vLLM, because PagedAttention removes KV-cache waste. There is no current independent head-to-head, so measure both on your own workload before committing.

Can I run SGLang on AWS Trainium now?

Yes, in public preview. SGLang-Neuron v0.1.0, released November 12, 2026, brings the SGLang serving stack to Trainium2 and Trainium3 through a native PyTorch plugin, with 17 models listed in the release and the same SGLang interface. Features, APIs and supported models can change while it is in preview.

What is RadixAttention?

RadixAttention is SGLang's way of managing the KV cache, the memory that stores what a model has already computed for a prompt. It organizes cached prefixes in a radix tree, so any prefix two requests share is computed once and reused automatically, with no configuration. It pays off most in multi-turn chat, agent loops and retrieval pipelines with long shared prompts.

Eight rivals agreeing to share one inference roadmap is unusual, and it is a bet that the serving layer is now infrastructure, like a Linux distribution or a web server, rather than a competitive advantage to keep private. The consortium's first year will show whether that bet holds. Watch for merged code, public cross-vendor benchmarks, and the v1.0 release notes. If those arrive, the announcement will have been the moment open inference got a shared roadmap. If they do not, it was a photograph of eight logos. Either way, the engine itself is free, and you can test it on your own workload today.

Sources

13 min read
20 views
edit_note Written by

Ratul Ahamed

Founder & Editor-in-Chief

Ratul Ahamed is the founder and editor of AI Core Updates, an independent site covering AI tools, models and pricing. He is a web developer based in Bangladesh and also runs All In One Tools, a free collection of browser-based utilities. Contact: [email protected]

Read all stories by Ratul Ahamed arrow_forward
verified_user Human-Reviewed Editorial

forum Comments (0)

Leave a Reply

chat_bubble_outline

No comments yet. Be the first to join the conversation!

Weekly AI Digest

Get our weekly AI tools digest

One email a week analyzing new AI tools, model updates, and practical workflows. Tested and summarized with zero fluff. Unsubscribe any time.

bolt
Hands-on AI Tool Reviews
Pricing math, limits, and alternatives.
verified_user
Strict Anti-Spam Policy
1-click unsubscribe at any time.
mark_email_unread

Subscribe to Digest

Sent weekly • No spam

person_outline
mail_outline

No spam ever. You can unsubscribe with a single click at any time.

cookie

Cookie Settings

We use cookies to enhance your browsing experience, serve personalized content, and analyze our traffic. By clicking "Accept All", you consent to our use of cookies. Learn more