Introduction

TCP transport protocol, for a long time, has proven to be phenomenally successful. This is designed in the late 1970, when there were only about 100 hosts and networklink had speeds of 10Kbps. Since then, we've seen the incredibly growth of the internet to billions of host and link speeds of 100Gbps or more.

Looking at these number, and knowning that TCP still serves as the primary transport protocol for almost every application, gives me the urge to take a deeper look into its mechanisms, and understand if it's acutally efficient as said. It's an extraodinary achivement in the engineering field, to have designed such mechanism that could survive radical changes in underlying technology.

However, I still believe that nowadays, some groundbreaking changes must have caused some impact on TCP's design, who know 😛, and this is just a chance for me to confirm validity of this though.

Requirement

Before jumping into any specific problems with TCP, it's important to be on the same page while talking about challenges that must be addressed by any transport protocol in the massive data world, to me, these requirements are important in today world:

• Reliable delivery: data must be reliably deliverd from one host to another, in spite of transient failure in the network.
• Low latency: With mordern networking hardware improvement, processing time is already cut to a few microseconds, and transport protocol surely must not add significanly to this latency, so applications could experience latencies close to the hardware limit, otherwise, transport will unnecessarily becomes bottleneck.
• High throughput: With today high requirement and massive data requirement, it's easy to understand this requirement. To serve a massive requirement of data, the protocol need to be able to deliver massive data with high throughput.
• Congestion control: Have the mechanism to able to balance between the load and the speed of the network, so no downgrade in performance should be visible.

TCP and its limitation

Stream orientation:

The main data model for TCP is a stream of bytes. Meanwhile, datacenter application usually exchange discrete messages to implement RPCs.

Ambiguous message delivery

When messages are serialized in a TCP stream, TCP has no knowledge of message boundaries, this can lead to the following possible situation when an application reads from the TCP stream:

• it could receive the complete message (if the read buffer is larger than the message length, but there is no subsequent message immediately follows)
• it could receive less than a full message (when the read buffer is smaller then the message length)
• it could receive parts of multiple messages (when the read buffer is much larger than the message length, and multiple messages have already written to the TCP stream)

There is a known workaround for this one at the application level: the TCP based application could marks message boundaries when they serialize message (for example, prefixing each message with its length), and the server application uses this information to reassemble message on receipt. This though introduces extra complexity and overheads (in both processing time and data transmitting, or compute cost for maintaining state of partially-received messages)

Insufficient software message load balancing

Another issue with this is that, with modern application, streaming model is disastrous for software load balancing. To fully utilize the power of multi core hardware, applications usually uses a collection of threads to serve requests arriving across a collection of streams. Ideally, all of threads would wait for incoming messages on any streams, or, in other words, messages distributed evenly across threads.

However, because of previous known limitation, there is no guarantee that a read operation on specific thread at a time could returns an entire message. It's possible that parts of a single message are received by different threads. While it could theoretically possible that threads coordinate and reassemble the entire message in one of the thread until that thread is happy to release the stream. But this is too hard to be a pragmatic solution.

To work around this limitation, usually, system rely on inferior forms of load balancing, in which each stream is owned by a single thread.

• The first solution, is divide a collection of streams statically among the threads, where each thread handles all the requests arriving on its assigned stream. (This is known approach used by memcached). This approach is prone to hot spots, tho, where one thread receives a unequal share of incoming requests
• The second solution, is dedicates one thread to read all incoming messages from all streams, and dispatch them to other threads for processing. This allows much better load balancing across worker thread, but the dispatcher thread now become the single bottleneck. Further more, each message need to travel through 2 separate threads before being actually processed, which mean considerable latency and software overhead should be noticeable.

In short, the fundamental problem with streaming is that the unit in which data is received (bytes) does not correspond to the unit of work (message)

Tail latency with head-of-line (HOL) blocking

TCP strictly enforces in-order delivery. Message in a single stream must be received in the exact order they were sent. This means a short message can be delayed behind a long message in the same stream, there is a possibility that the server might sit idle waiting for the rest of long message to arrive, while the sort message is unnecessarily hold up in the queue, and this usually affect the tail latency noticeably.

Lack of round trip guarantee

While offers excellent reliability thought packet ordering guarantee, this seems isn't exactly what async/sync RPC application need. Application wants round trip guarantees. A client application wants a promise that its requests will be delivered and processed, and it will receive a response, and if any of these fails, the client would want an error notification.

TCP stream, tho, only guarantees best-effort delivery of data in one direction. In other words, client will receive no notification if the server doesn't send a response. As a result, client need to maintain their own understanding about the request state by implementing end-to-end timeout mechanism (for example), even though TCP already has timers of its own. These mechanism surely introduce additional overheads.

Connection orientation

TCP requires long-live connection state for each peer that the host communicates with. To give background concept, connection represents the state of the specific TCP socket, which include source and target IP, ports, and its delivering state (current packet sequence number, etc). In other words, TCP is stateful, and TCP requires extra effort to maintain this state.

Unavoidable space/time overhead

For massive scale system, this is undesirable in a sense of high overheads in space or time. To be more specific, each connection could requires about 2000 bytes of state for each TCP TCP socket in Linux kernel, excluding package buffers, plus additional state at application level. Under massively scaled system, where it need to maintain hundreds of thousands of connection, this is prohibitively expensive.

Tho, some known applications have a way to reduce this overheads, that is make application threads communicate through a collection of proxy threads that manage connection to all upstream (service meshes). This allows a single connection for each upstream to be shared across all the application threads on that host, tho it introduces overhead by forcing traffic though proxies (I see this somewhere before 😛).

Another alternative is using connectionless protocol like UDP. UDP is acceptable in system that can tolerate its unreliability, this tho sacrifices congestion control.

Extra resources for setup phase (aka TCP handshake)

Without understanding about other peer, the source and target host don't know how to transmitted data (as said, this is stateful). In TCP, this phase has a nontrivial cost, since it requires an additional round-trip between hosts.

This is fine with traditional system, where connections have been long-lived, so the setup cost can be amortized across a large number of requests. However, in modern server less environment, where applications have very short lifetimes, this upfront latency penalty is much harder to absorb.

Bandwidth sharing ("fair" scheduling)

In TCP, when a host's link is overloaded, TCP kicks into a congestion control, see [[TCP Congestion Control]]. This could make TCP attempts to share the available bandwidth equally among the active connections, this is also known as "fair" scheduling.

Poorly underload

Fair scheduling performs poorly under heavy RPC load. When receiving several large messages simultanously, bandwidth sharing causes all of them to transfer slowly (instead of trying to finish each one ASAP). There are run-to-completion approaches such as SRPT provide better overall response time because they dedicate all the available resource to a single, shortest task at a time, ensuring that it finishes quickly, tho, it's incredibly difficult to implement such with TCP due to the lack of message boundaries.

This fair sharing from TCP also discriminates heavily against short messages. Their transmission time, which should naturally be considerably much more smaller, noticeably increases when being forced to share bandwidth equally with massive data transfers.

Sender-driven congestion control

TCP drives congestion control from senders, which voluntarily slow their packet transmission rate when detect congestion.

Sender are blind

Senders have no first-hand knowledge of congestion, which can happen either in the core fabric or at edge link, so they can only rely on internal congestion signals.

Client only could guesses there is a traffic jam by trusting these clues: packet drops due to buffer occupancy at router, increase in RTT or some specific notification from smart switch (ECN)

Congestion control in TCP weird in a sense of these limitation:

• Congestion can only be detected when there is noticeable downgrade in performance (timeout or more packet lost). at this point, the performance is already hurt somehow.
• TCP doesn't take advantage of priority queues in modern network switches, thus all packets are treated equally and queues generated by long message will cause delays for short message

Moreover, it takes about 1 RTT for a sender to find out about traffic congestion, meaning sender make decisions based on out-of-date information. As messages get shorter and network get faster, more and more message will be completed in less than 1 RTT, which make these kind of approach less and less reliable.

In-order packet delivery

In modern setup, there is a technique called packet spraying, where each package is independently routed through the switching fabric to balance loads on links.

Impossible network balancing

However, due to TCP packet order guaranteee, this cannot be used effectively with TCP, since it could change the order in which packet arrive at their destination. Instead, TCP networks must use flow-consistent routing, where all packets from a given connection need to follow the same path through the network fabric. Flow-consistent routing ensures the in-order packet transimission, but it virtually guarantees that there will be overloaded link along the way, even with low load network (due to its nature of unequally balancing). And also, due to this deterministic packet flow, all connection hash to the same overloaded link also experience downgrade despite its transmission rate.

This is not only happens in network fabric, it also causes hot spots in software. To give an example, Linux performs load balancing in software by distributing the handling of incoming packets across multiple cores, this is a way to sustain high packet rates. Each incomming packet is processed by the kernel on different cores before reaching the application. In order to ensure this in-order packet delivery, all of packets from a connection must go through the same sequence of cores. And like the example with link, this could also create uneven core loading when high connection hash to the same core.