Collectives with Zero-SMs

October 02, 2026

These are my personal notes on how NCCL’s zero-SM collective/RMA operation paths work.

What the heck is Zero-SM?

Typically, everything done inside of a GPU is done using SMs: they have the logic necessary to load and store data to and from general memory. Modern GPUs however have new mechanisms to perform data transfer, specifically the Copy Engine (CE) and Tensor Memory Accelerator (TMA). This is in addition to the default “SM” path.

There are various tradeoffs between these 3 transfer mechanisms I won’t get into here (see ParallelKittens for a systematic review). For the purposes of this post, the one we care about is the copy engine/CE, which can enable the transfer of data with no SMs whatsoever, hence “zero-SMs”. This can be insanely valuable: the entire point of kernel performance engineering is to keep you expensive SMs running 24/7, so offloading all communication and peer data movement to the CE can give you a very large performance bump, if you structure your collective in the right way. If you overlap zero-SM operations with compute work, in theory there is no contention for resources!

NCCL’s Zero-SM Capabilities (as of 2026)

Let’s break things down at a top-level view, because things can get confusing quickly. NCCL has three main kinds of collectives: legacy, symmetric, and “zero-SM”:

There’s a lot to be said about all 3, but zooming into the zero-SM path, we can see that the collective you run first splits by whether all peers are in the same NVLD, vs. when some are on different nodes (and thus require movement via the NIC). This is dubbed “flat” and “hierarchical” respectively; in general, we call inter + intra-node zero-SM “hierarchical” because there is an implicit hierarchy of interconnects that it is crossing. For simplicity we can divide this into simply “intra” and “inter” node interconnects, but in reality modern GPUs are exposing more tiers that we’ll have to account for in the future (see “Blackwell stitch bandwidth” and “die localization”). In the zero-SM path, it’s important to note how all intranode transfer is done via the Copy Engine, and all internode transfer is done via a “Proxy Thread” - i.e. a thread that polls for enqueued work on the host CPU: note the complete lack of SMs needed!

I call this “less-flexible high throughput” because it’s designed for large message sizes just like legacy, but the difference is you don’t have to use any SMs to accomplish this data transfer. This has the tradeoff of not being able to use multicast for example: so you can overlap things better but you lose SM-centric capabilities. There are caveats with “use”, but in general this is my mental model for how these collectives are designed.

NCCL’s host RMA APIs (ncclPutSignal, ncclSignal, ncclWaitSignal) are also on the zero-SM path: they are extremely similar to collectives in that they use CE for intranode data transfer but a network proxy thread for internode data transfer. You can find all the logic in these files:

Both of these are pretty straightforward: they are fundamentally cudaMemcpyAsync calls on a stream. It doesn’t get more complicated than that: the confusion lies in the use of various intermediate functions that wrap some of the lower level complexity. The TLDR is that we need to set up the signalling infrastructure, then create a (src, dst, bytesize) work job, and then submit that to the cudaMemcpyAsync call in order to initiate the CE transfer. We poll as needed on the signals to mark completion!

Here’s a general workflow of how intranode CE transfers work for collectives, as defined in ce_coll.cc:

For collectives, our signalling infrastructure uses “Ready” and “Complete” arrays to do all the necessary signalling for every peer rank. Because NCCL devs define the entire dataflow of the zero-SM collective, the signals are a lot more simple: we just have a ready and complete counter for each peer.

Here’s a general workflow of how intranode CE transfers work for RMA operations, as defined in rma_ce.cc:

RMA is more complex: we don’t know if the RMA operation is launched standalone or part of a graph. If it is standalone, we can use a counter and thus one buffer for the signal and one buffer for the shadow (the shadow buffer simply tracks the last counter value whose operation is completed). If it is launched as part of a graph, we need a separate data structure for Signal and Ack, because the CE cannot easily increment counter values (at least as of now).

Both collectives and RMA operations use a network proxy path for internode data transfers (see rma_proxy_launch.cc):

The code for this is in the rma/rma_proxy* files: it looks complex, but fundamentally it’s just a single thread (dubbed the “proxy”) who runs a while-loop that continuously polls a work queue/”desc” queue. It then takes jobs, and submits them to an “in-progress” queue, before polling for completion. Upon completion of the network RMA operation (handled separately by the ncclNet plugin since there are too many NIC brands out there to count), it updates the done/op sequence number as needed.

Now we have the three fundamental primitives, covering all CE-based intranode data transfer, and proxy thread-based internode NIC data transfer. Using this, it’s relatively straightforward to compose the underlying functions and build whatever 0-SM collectives or RMA operations you want!