Cutting Payload aka Packet Trimming—the CRISPR-Cas9 of Network Flow Control?

(Edited to add more pointers to implementations at the end— SL.)

One of my tasks in the (backbone) network team at Switch involves identifying and following "trends" outside our bubble that might somehow end up affecting our business. A current trend with possible impact is "AI"—maybe you have heard of it—which today means mostly machine learning using (deep) artificial neural networks, and in particular large language models (LLMs).

There are already noticeable changes in how datacenters are built to accommodate LLMs—check out what OCP has been doing in the past few years. These changes include (datacenter) networks, which must now support the needs of large GPU clusters, in particular for LLM training. To use those high-performance—and high-cost—GPUs effectively, one wants to be able to copy large amounts of data between GPUs anywhere in the cluster, which today might contain 32768 GPUs or so, with some striving up to a scale of millions. These transfers typically use remote memory-to-memory transfer without CPU involvement (RDMA). There are significant efforts to make this work over some sort of standardized Ethernet-based network. This could be done with ROCE (RDMA over Converged Ethernet) (v2), where the "Converged" hints at some modifications beyond traditional Ethernet.

Some techniques were already considered for addition to Ethernet for datacenter networks before the LLM hype wave, for example PFC (priority flow control) to support "lossless" operation of the network for specific subsets of traffic, or novel ECMP (equal-cost multipath) approaches that don't necessarily preserve per-flow ordering (using "spraying" instead).

Recently I was excited when I heard about another technique to be added: packet "trimming". It reminded me of a great (video) presentation by Mark Handley (UCL) from SIGCOMM 2017. I'll add the link later. For now I'll refer to this work as "the NDP paper".

I remember being impressed when I saw the video, and I also found that it was well received. Just having a paper accepted at SIGCOMM is one of the highest achievements for (Internet-affine) network researchers, and this paper was awarded best paper that year. I also remember thinking something along the lines of... "wow, that was an awesome presentation and must have been the talk of the conference... but what are the chances of this being implemented in production networks within my lifetime?". Focused on "classical" Internet backbones and enterprise networks, I thought that this was just a bit too revolutionary/"disruptive" to have much of a chance in our environment, which has shown strong resistance to seemingly much simpler changes (AQM, ECN, making addresses wider...). But 7–8 years later and thanks to the investment craze kicked off by ChatGPT, maybe the time is ripe!

One thing I was wondering about was whether the NDP paper was the first to introduce "trimming". Upon re-watching the video after several years (and understanding a bit more than at previous attempts :-) I caught Mark referring to previous work from Tsinghua as the inspiration for the trimming. A bit of searching turned up this NSDI 2014 presentation of the paper "Catch the Whole Lot in an Action: Rapid Precise Packet Loss Notification in Data Centers" by Peng Cheng (then a Ph.D. student at Tsinghua University, now at Microsoft Research Asia) et al. It includes a nice introduction where Peng Cheng talks about cooking and his use of the scissors as a multipurpose tool, suggesting that his "cut payload" approach could be similarly broadly useful. (A bit like the CRISPR "gene scissors" in bioengineering today, no?) Maybe Peng Cheng et al.'s work could be considered for a "test of time" award.

So where are we now, in 2025? Looks like concepts from the NSDI 2014 and SIGCOMM 2017 papers (trimming, spraying...) will be added to the Ultra Ethernet standard. It is not clear to me how the trimming capability will be used for congestion management. Maybe not at all in the initial standard?

Thinking a bit further: Assuming that trimming becomes a standard function in data center switches, could we use it in other network contexts? For example, to reduce "bufferbloat" for broadband users, or generally in the wide-area Internet... If people see potential for this, then I guess some work (including standardization work, presumably in the IETF) would be needed to ensure that trimmed packets could safely be sent over the Internet, and on Internet-deployable transport protocols (based on something like NDP?) that can make use of them.

My intuition says that there is some potential there, and if done well, this novel idea could become a useful addition to the Internet.

But one assumption that justifies trimming is that "serialization delay" (i.e. the time it takes to put the bits of a packet on the wire) is significant relative to "propagation time" (the time it takes the packet to travel through the network). This can be true for networks that are relatively small in geographical size—for example within a data center—or for networks that have relatively "slow" (low-bitrate) links; or both, of course! (There is also a dependence on packet size, but for now let's assume that the maximum packet size is similar for all networks, between ~1500 and ~9000 bytes. This is the case in practice today, although it would be interesting to study in how far trimming could make large-MTU networks more viable.)

So assuming that trimming makes sense in a DC network where end-to-end round trips take on the order of a microsecond (100m distance) and link speeds are on the order of 100Gb/s, it should make just as much sense on a network with one millisecond RTT and links of 100Mb/s, or one with 100ms RTT and links of 1Mb/s. OK, such delay/rate combinations aren't frequently seen in classical "research & education networking" anymore these days. But some interesting link technologies used at the network "edge" may sometimes exhibit such low rates—Wi-Fi when the station is far away from the base, or low-energy short/medium-distance wireless.

My gut feeling is that packet trimming may become commonplace in data center networks, provided that the AI "summer" continues for a few years. It may disappear again because the complexities (integrating it with higher-level protocols) may outweigh the gains. But it's encouraging that such "outside-the-box" ideas can gain traction in the market—when the stars align just right.

Oh yeah, here is the promised link to Mark Handley's presentation: Re-architecting datacenter networks and stacks for low latency and high performance, SIGCOMM 2017 proceedings. See the "supplementary material" section for the video.

Known Packet Trimming Implementations

Cumulus Linux ≥5.14 supports Packet Trimming on NVIDIA Spectrum chipsets (Release Notes).

Arista EOS ≥4.35 supports Packet Trimming on (Tomahawk 5-based?) 800G switches (Release Notes).

HPE Juniper supports Packet Trimming (called DCN, Drop Congestion Notification) on Tomahawk 5-based switches (Configuration Guide, LinkedIn post).

Broadcom's MRC (Multi-Path Reliable Connections) on Tomahawk 5/6 and Thor Ultra 800GE NICs uses source-controlled "spraying" and packet trimming (blog post by N. Vaidya, C. Raicu, and E. Spada), May 2026.

Enabling BBR on Ubuntu Xenial

BBR (Bottleneck Bandwidth and RTT) is a novel congestion control scheme for TCP. It was developed by several Google engineers and has been in use on Google.com and Youtube. See the PERT KB for more background.
Here's how you can enable it on a machine running Ubuntu 16.04 LTS "Xenial Xerus".
Note: This is mostly useful for systems that send fairly large TCP flows. And the configuration is incompatible with other packet schedulers such as fq_codel, so consider the tradeoff.

Upgrade Linux Kernel to ≥4.9

The BBR implementation was merged into the master branch of the Linux kernel before the 4.9 release. Xenial uses the 4.4 kernel by default, but you can install a newer one (4.10 as of this writing) from a preview repository of the "hardware enablement" kernel provided by Canonical:

sudo apt install -y --install-recommends \
  linux-generic-hwe-16.04-edge

Enable fq Packet Scheduler

BBR fundamentally requires a packet scheduler that can "pace" outgoing packets per flow.  Enable this by adding an "up" clause (shown in boldface below) to each interface in its corresponding configuration file, e.g. /etc/network/interfaces.d/eth0.cfg:

auto eth0
iface eth0 inet dhcp
  up tc qdisc replace dev $IFACE root fq pacing
iface eth0 inet6 auto

Make BBR the Default Congestion Control Algorithm for TCP

echo net.ipv4.tcp_congestion_control=bbr | \
  sudo tee /etc/sysctl.d/90-bbr

Activate by Rebooting

A reboot should now enable all necessary components: the recent kernel, a pacing-capable packet scheduler, and BBR as the default.

You can check whether BBR is actually in effect by using the ss -i command while having at least one TCP connection open—which is already guaranteed to be the case if you are logged in over SSH.

$ ss --tcp -i
State       Recv-Q Send-Q                                               Local Address:Port                                                                Peer Address:Port
ESTAB       0      1848                         2001:620:5ca1:2f0:f816:3eff:fe9c:71af:ssh                                                           2001:620:0:69::107:57795
  bbr wscale:5,7 rto:208 rtt:6.078/0.237 ato:48 mss:1182 cwnd:12 bytes_acked:156685 bytes_received:10033 segs_out:851 segs_in:977 send 18.7Mbps lastrcv:4 lastack:4 pacing_rate 87.2Mbps unacked:4 retrans:0/13 rcv_rtt:12 rcv_space:28560
ESTAB       0      0                            2001:620:5ca1:2f0:f816:3eff:fe9c:71af:ssh                                                           2001:620:0:69::107:61315
  bbr wscale:5,7 rto:220 rtt:18.674/25.752 ato:40 mss:1182 cwnd:11 bytes_acked:8025 bytes_received:5125 segs_out:86 segs_in:122 send 5.6Mbps lastsnd:470176 lastrcv:480332 lastack:470068 pacing_rate 60.5Mbps rcv_rtt:12 rcv_space:28560

The "-i" part of the output mentions "bbr" and includes some BBR-specific metrics as well as the pacing rate applied to each outbound TCP flow.

Riding the SDN Hype Wave

In case you haven't noticed, Software-Defined Networking has become the guiding meme for most innovation in networks over the past few years.  It's a great meme because it sounds cool and slightly mysterious.  The notion was certainly inspired by Software-Defined Radio, which had a relatively well-defined meaning, and has since spread to other areas such as Software-Defined Storage and, coming soon, the Software-Defined Economy.
As a networking veteran who has fought in the OSI and ATM wars, I like making fun of these fads like the next person—buzzword bingo anyone? But actually, I consider the SDN hype a really good thing.  Why? Because, to quote Cisco CTO Padmasree Warrior, "networking is cool again", and that's not just good for her company but a breath of fresh air for the industry as a whole.
What I like in particular is that SDN (because nobody knows exactly what it means and where its limits are...) legitimates ideas that would have quickly been rejected before ("it has been known for years that this doesn't scale/that this-and-that needs to be done in hardware/...").  Of course this also means that many tired ideas will get another chance by being rebranded as SDN, but personally I think that this does less damage.

SDN Beyond OpenFlow

The public perception of SDN has been pretty much driven by OpenFlow's vision of separating forwarding plane (the "hardware" function) and control plane, and using external software to drive networks, usually using a "logically centralized" control approach.

The Open Networking Forum attempts to codify this close association of SDN and OpenFlow by publishing their own SDN definition.  OpenFlow has huge merits as a concrete proposal that can be (and is) implemented and used in real systems.  Therefore it deserves a lot of credit for making people take the "SDN" vision seriously.  But I think the SDN meme is too beautiful to be left confined to OpenFlow-based and "logically centralized" approaches.  I much prefer JR Rivers's (Cumulus Networks) suggestion for what SDN should be: "How can I write software to do things that used to be super hard and do them super easy?" That's certainly more inclusive!

x86 as a Viable High-Speed Packet Processing Platform

Something that I definitely consider an SDN approach is to revisit generic computing hardware (mostly defined as "x86" these days) and see what you can do in terms of interesting packet processing on such platforms.  It turns out that these boxes have come a long way over the past few years! In particular, recent Intel server CPUs (Sandy Bridge/Ivy Bridge) have massively increased memory bandwidth compared to previous generations, CPU cores to spare.  On the interface front, most/all of today's 10 Gigabit Ethernet adapters have many helpful performance features such as multiple receive/transmit queues, segmentation offload, hardware virtualization support and so on.  So is it now possible to do line-rate 10Gb/s packet processing on this platform?

The dirty secret is that even the leading companies in ASIC-based backbone routers are already using regular multi-core CPUs for high-performance middleboxes such as firewalls (as opposed to previous generations that had to use expensive-to-design and program network processors, FPGAs and/or ASICs).

Intel has its DPDK (Data Plane Development Kit) to support high-performance applications using their network adapters and processors, and there are several existence proofs now that you can do interesting packet processing on multiple 10Gb/s interfaces using one core or less per interface—and you can get many of those cores in fairly inexpensive boxes.

Snabb Switch

One of my favorite projects in this space is Luke Gorrie's Snabb Switch.  If CPU-based forwarding approaches are at the fringe of SDN, Snabb Switch is at the fringe of CPU-based forwarding approaches... hm, maybe I just like being different.

Snabb Switch is based on the Lua scripting language and on the excellent LuaJIT implementation.  It runs entirely in user space, which means that it can avoid all user/kernel interface issues that make high-performance difficult, but also means that it has to implement its own device drivers in user space! Fortunately Lua is a much friendlier platform for developing those, and one of Luke's not-so-secret missions for Snabb is that "writing device drivers should be fun again".

The Snabb Switch project has gained of traction over the year or so since its inception.  A large operator is investigating its use in an ambitious backbone/NFV project; high-performance integration with the QEMU/KVM hypervisor has been upstreamed; and the integration into OpenStack Networking is making good progress, with some hope of significant parts being integrated for the "Juno" release.  And my colleague and long-time backbone engineering teammate Alex Gall is developing a standalone L2VPN (Ethernet over IPv6) appliance based on Snabb Switch, with the ultimate goal of removing our only business requirement for MPLS in the backbone.  Now that should convince even the curmudgeons who fought in the X.25 wars, no?

The final proof that Snabb Switch's world domination is inevitable is that it was featured in the pilot episode of Ivan Pepelnjaks new Software Gone Wild podcast.

(In fact that is the very reason for this post, because yours truly also had a (small) appearance in that episode, and I had to give up the address of my blog... and now I'm afraid that some of the venerable readers of Ivan's blog will follow the link and find that nothing has been posted here lately, even less so related to networking.  Welcome anyway!)

Come on in, the water's fine!

So turn your bullshit detectors way down, embrace the hype, and enjoy the ride! There are plenty of good ideas waiting to be implemented once we free ourselves from the rule of those ASIC-wielding vertically integrated giga-companies...

Hello World with ØMQ (ZeroMQ), Part I

Why I am interested in ØMQ

Several months ago, I stumbled across an interview with Pieter Hintjens about ØMQ (ZeroMQ) in episode 195 of FLOSS Weekly by Randal L. Schwartz. From what I got from the interview, ØMQ is a pretty powerful "message queue" system that is somehow implemented in a light-weight way as a linkable library. There are also many language bindings, including all fashionable and many exotic languages (but, sadly, not Common Lisp).
I had heard about message queue systems for a long time, but have never really used any, and they always seemed a little scary. The currently popular message queue system seems to be RabbitMQ, and despite the cute name, I hear that it is somewhat big. At the same time, I'm sure that message queues serve a useful purpose, and may be a great basis for distributed systems with fewer reinvented wheels and, thus, better behavior (including performance), so they probably deserve a closer look. And ØMQ seems to be successful in several respects, and at the same time "lightweight" enough for me to understand something. This makes it an attractive system to investigate.

First steps

I have finally found time to start reading the ØMQ Guide during a train ride from Geneva to Zurich.
The introduction ("Fixing the world") at first looks a little pompous, but is in fact full of very good thoughts, both original and convincing, about big problems in programming large distributed software systems. Apparently ØMQ aspires at solving an important part of these problems. Judging from the introduction, the people who wrote this seem very smart. And from the interview, I know that the designers have had a lot of practical experience building real systems, and they knew the deficiencies (but also the achievements) of other messaging systems before they started rolling their own.

Walking through the "Hello World" example

So now I'm reading through the first, "Hello World", example in the guide, trying to understand as much as possible about ØMQ's concepts. The example starts with the server side (which responds "World"). I cut & paste the complete code to a file server.c, which compiles easily enough on my Ubuntu 12.04 system with libzmq-dev installed:

    : leinen@momp2[zmq]; gcc -c server.c
    : leinen@momp2[zmq]; gcc -o server server.o -lzmq
    : leinen@momp2[zmq]; 

When trying to understand an ØMQ API call, I first guess a little what the names and arguments could mean, then I look at the respective man page to validate my guesses and to learn the parts that I was unable to guess.

    void *context = zmq_init (1);

Why is the result type void *, rather than something like ZqmContext *? I'll explain in a mpment why I would strongly prefer the latter.
And wouldn't it be nice if the size of the thread pool (the 1 in the call) could be left undefined? The ØMQ system could either optimize it dynamically, or it could be controlled by standardized external configuration. But maybe this isn't really practical anyway.

    //  Socket to talk to clients
    void *responder = zmq_socket (context, ZMQ_REP);

The socket call is simple enough, taking just one argument beyond the necessarily required context - the socket type (here: ZMQ_REP), "which determines the semantics of communication over the socket." So what does ZMQ_REP mean, and what other types are available? Aha, "REP" is for "reply", and the complementary type is ZMQ_REQ, for "request". So these must be for the standard request/response pattern in classical client/server protocols.
Other types include ZMQ_PUB/ZMQ_SUB for pub/sub protocols, ZMQ_PUSH/ZMQ_PULL for processing pipelines, and a few others related to, if I understand correctly, load balancing, request routing etc.
That's a great choice of patterns to support, because they cover a huge subspace of socket applications in real applications.

    zmq_bind (responder, "tcp://*:5555");

In passing, I notice that this call doesn't take a "context" argument. This probably means that it gets the context from the socket (here: "responder"). Convenient. But, coming back to my previous complaint, responder is also of type void *. So what happens when someone is only slightly confused and passes context to zmq_bind? The compiler certainly has no way of catching this. In fact, when I deliberately introduce this error, I find that even the library doesn't catch this at runtime! I just get a server program that mysteriously sits there and doesn't listen on any TCP port. That is really not nice. Maybe real programmers don't make these kinds of mistakes, but I have my doubts. As an old Lisper, I'm certainly not religious about static type checking. But type checking at some point would really be beneficial.
OK, back to the zmq_bind call. We have a responding socket, so we need to bind it to a "listening" port. The API uses URL-like strings to specify endpoint addresses (tcp://*:5555). This is fine, although I'm slightly worried whether the API developers try to adhere to any standards here (is there a standard for "tcp" URLs?), or whether they just make things up.
The URL-like string approach is certainly superior to what people using the C socket API have to put up with: Use the right sockaddr structures, use getaddrinfo() (and NOT use gethostbyname() anymore! :-), do address resolution error handling by hand, etc.

ØMQ could support IPv6, but does it?

One can easily imagine that the library just Does The Right Thing (DTRT) concerning multiple network-protocol support, e.g. that the above call results in a socket that accepts connections over both IPv4 and IPv6 if those are supported.

Not just yet, it seems.

Unfortunately this doesn't seem to be the case however: When I run the compiled server.c binary under system-call tracing, I get

    : leinen@momp2[zmq]; strace -e bind ./server
    bind(16, {sa_family=AF_INET, sin_port=htons(5555), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
      C-c C-c: leinen@momp2[zmq]; 

So this is obviously an IPv4-only socket. Still, if I look inside libzmq.a (in GNU Emacs in case you are curious, though nm | grep would also work), I notice that it references getaddrinfo, but not gethostbyname. So there is at least some chance that someone thought of IPv6. Maybe one has to set special options, or maybe the people who built the Ubuntu (or Debian) package forgot to activate IPv6, or whatever?
Looking at the man page of zmq_tcp, ("ØMQ unicast transport using TCP"), it only talks about "IPv4 addresses". The way I interpret this is that they only support IPv4 right now, but at least they don't ignore the existence of IPv6, otherwise they would have just said "IP addresses" and still meant IPv4 only. So there is at least a weak hope that IPv6 could once be supported.
When I briefly had connectivity (because the train stopped at a station with WiFi), I googled for [zmq ipv6] and found a few entries that suggest that there has been some work on this. Maybe the most promising result I got from Google was this:
[zeromq-dev] IPv6 support - Grokbase
http://grokbase.com/t/zeromq/zeromq-dev/118fjjmpmc/ipv6-support
15 Aug 2011 – (5 replies) Hi all, Steven McCoy's IPv6 patches were merged into the master. The change should be completely backward compatible.

So this fairly new. I also noticed that the libzmq in my brand-new Ubuntu 12.04 is only version 2.11.11, and some other Google hits suggest that IPv6 support is planned for ØMQ 3.0. So when I get home I'll check whether I can find ØMQ 3 or better, and use that in preference to the Ubuntu package.

What about multi-transport support?

So we learned that multiple network-layer support (e.g. IPv4 and IPv6) seems possible, and even planned. What about support for multiple different transports? This is certainly not possible using low-level sockets, and it seems difficult, but not impossible, to provide this transparently using a single ZMQ-level socket. For example, a server endpoint could listen requests on both TCP and SCTP connections.
I have a hunch that ØMQ doesn't support this, because the standard interface has only a single scalar socket-type argument. In some sense this is a pity, because having a single socket that supports multiple protocols would make it easier to write programs that live in a multi-protocol world, just like support for multiple network protocols in a single socket makes it easier to write programs that live in a world with multiple such protocols, like IPv4 and IPv6 these days.
Conceptually, ØMQ looks powerful enough to support this without too much pain. It might also be possible to build "Happy Eyeballs" functionality into the library, so that in such multi-protocol contexts, the library could make sure that a reasonably well-performing substrate is always used, so that developers using the library don't have to worry about possible problems when they add multi-protocol support.

Swiss trains go too fast, or Switzerland is too small

So I only got through a whopping two lines of Hello World code for now. But I hope my sidelines and confused thoughts didn't turn you off, and you still agree with me that ØMQ is something worth watching. I sure hope I'll find the time to continue exploring it.