kolo-transport opens QUIC version 1 (RFC 9000) connections over UDP, secured by TLS 1.3 (RFC 9001), with the loss recovery and Reno congestion control of RFC 9002 and an optional BBR controller after Cardwell et al. It gives each peer bidirectional and unidirectional streams, with hard limits on unread data, streams and connections. The connection is encrypted, but the peer is not authenticated: kolo-secure adds identity on top, using kolo-id keys.
Install
go get github.com/ciranda-dev/kolo-transportBuilds without cgo.
Quick start
A listener and a dialer in one program on 127.0.0.1. The dialer opens a stream, and the listener echoes it back.
package main
import (
"context"
"fmt"
"io"
"log"
"time"
kolotransport "github.com/ciranda-dev/kolo-transport"
)
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
l, err := kolotransport.Listen(ctx, "127.0.0.1:0", nil)
if err != nil {
log.Fatal(err)
}
defer l.Close()
go func() {
c, err := l.Accept(ctx)
if err != nil {
return
}
s, err := c.AcceptStream(ctx)
if err != nil {
return
}
io.Copy(s, s)
s.Close()
}()
c, err := kolotransport.Dial(ctx, l.Addr().String(), nil)
if err != nil {
log.Fatal(err)
}
defer c.Close()
s, err := c.OpenStream(ctx)
if err != nil {
log.Fatal(err)
}
if _, err := s.Write([]byte("hello kolo")); err != nil {
log.Fatal(err)
}
s.Close()
reply, err := io.ReadAll(s)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", reply)
}It prints hello kolo. s.Close() ends only the send side, so the dialer can
still read the reply.
API reference
The API reference is generated from the Go source and is in English.
Package kolotransport carries multiplexed streams over one QUIC (RFC 9000) connection between two Kolo peers.
It does not authenticate the peer. Each Listener presents an ephemeral self-signed certificate and the dialer accepts any such certificate, so TLS 1.3 gives confidentiality and integrity against passive observers but no identity, and an active man in the middle is not detected. Identity must come from a higher layer, for example kolo-id signatures over ExportKey.
Experimental: the API may change before v1.0.0.
Constants
const ALPN = "kolo/0"ALPN is the TLS application protocol both ends must offer.
const DefaultMaxConnReceiveWindow = 512 << 10DefaultMaxConnReceiveWindow is the default MaxConnReceiveWindow: 512 KiB.
const DefaultMaxGrownConnReceiveWindow = 8 << 20DefaultMaxGrownConnReceiveWindow is the default MaxGrownConnReceiveWindow: 8 MiB.
const DefaultMaxGrownMemory = 64 << 20DefaultMaxGrownMemory is the default MaxGrownMemory: 64 MiB.
const DefaultMaxGrownStreamReceiveWindow = 4 << 20DefaultMaxGrownStreamReceiveWindow is the default MaxGrownStreamReceiveWindow: 4 MiB.
const DefaultMaxMemory = 1 << 30DefaultMaxMemory is the default MaxMemory: 1 GiB of receive window.
const DefaultMaxStreamReceiveWindow = 256 << 10DefaultMaxStreamReceiveWindow is the default MaxStreamReceiveWindow: 256 KiB.
const MaxCode uint64 = 1<<62 - 1MaxCode is the largest error code QUIC can carry.
const MaxPacketSize = 1200MaxPacketSize is the largest non-QUIC datagram WritePacket sends and ReadPacket returns.
Variables
var (
// ErrInvalidConfig means a Config field is out of range.
ErrInvalidConfig = errors.New("kolotransport: invalid config")
// ErrInvalidArgument means a bad code, close message, ExportKey label, PacketConn, listen or dial address, packet or destination (see the README).
ErrInvalidArgument = errors.New("kolotransport: invalid argument")
// ErrClosed means this side closed the listener, connection or stream.
ErrClosed = errors.New("kolotransport: closed")
// ErrPeerClosed means the peer closed the connection or no longer has state for it.
ErrPeerClosed = errors.New("kolotransport: peer closed the connection")
// ErrRefused means the peer refused the connection, for example because its listener is full.
ErrRefused = errors.New("kolotransport: connection refused")
// ErrHandshake means the TLS or QUIC handshake failed.
ErrHandshake = errors.New("kolotransport: handshake failed")
// ErrTimeout means the handshake or idle timeout expired.
ErrTimeout = errors.New("kolotransport: timeout")
// ErrTransport means a QUIC transport error or another failure of the connection.
ErrTransport = errors.New("kolotransport: transport error")
// ErrStreamReset means the peer reset the stream or asked this side to stop sending.
ErrStreamReset = errors.New("kolotransport: stream reset by peer")
// ErrMaxDialed means Listener.Dial already has MaxDialed connections open.
ErrMaxDialed = errors.New("kolotransport: too many dialed connections")
// ErrStreamCanceled means this side canceled the stream.
ErrStreamCanceled = errors.New("kolotransport: stream canceled")
)Types
type Config
type Config struct {
// MaxConns caps the connections of a Listener, handshaking or open. Listener only. Default 100, from 1 to 10000.
MaxConns int
// MaxConnsPerIP caps them per source address, the one a connection started from. Listener only. Default 4 (or MaxConns if lower), at most MaxConns.
MaxConnsPerIP int
// MaxDialed caps the connections Listener.Dial has open, which MaxConns and MaxConnsPerIP do not count. Listener only. Default 32, from 1 to 10000.
MaxDialed int
// MaxIncomingStreams caps the concurrent streams the peer may open, bidirectional and unidirectional each. Default 16, from 1 to 1000.
MaxIncomingStreams int
// HandshakeTimeout bounds the whole handshake and the life of a Retry token. Default 10s, from 1s to 60s.
HandshakeTimeout time.Duration
// IdleTimeout closes a connection with no network activity. Default 30s, from 1s to 10m.
IdleTimeout time.Duration
// KeepAlivePeriod sends a PING after this long without receiving, so connections stay open, also those never accepted.
// Default 5s (or IdleTimeout/2 if lower), from 500ms to below IdleTimeout.
KeepAlivePeriod time.Duration
// MaxMemory must cover (MaxConns + MaxDialed) times MaxConnReceiveWindow (unread data, not heap), in bytes; the default leaves headroom and does not bound heap (see the README). Listener only. Default 1 GiB, from 64 MiB to 64 GiB.
MaxMemory int64
// AcceptFunc filters attempts by the source IP a connection starts from, never port or identity; false or a panic refuses.
// Listener only. Nil allows all.
AcceptFunc func(remote netip.AddrPort) bool
// MaxStreamReceiveWindow caps the unread data buffered per stream, in bytes, on Listen and Dial. Advanced. Default 256 KiB (or MaxConnReceiveWindow if lower), from 64 KiB to 16 MiB.
MaxStreamReceiveWindow int64
// MaxConnReceiveWindow caps the unread data buffered per connection, in bytes; a node that dials N peers can hold N times it. Advanced. Default 512 KiB, from 64 KiB to 64 MiB, at least MaxStreamReceiveWindow.
MaxConnReceiveWindow int64
// WindowGrowth lets Conn.AllowWindowGrowth raise the receive windows of a connection the application trusts, up to the Grown values below; every other connection keeps its initial windows. Default false: no window ever grows.
WindowGrowth bool
// MaxGrownStreamReceiveWindow is the largest stream window a trusted connection can reach, in bytes. WindowGrowth only. Default 4 MiB (or MaxStreamReceiveWindow if higher), from MaxStreamReceiveWindow to 16 MiB.
MaxGrownStreamReceiveWindow int64
// MaxGrownConnReceiveWindow is the largest connection window a trusted connection can reach, in bytes. WindowGrowth only. Default 8 MiB (or MaxConnReceiveWindow if higher), from MaxConnReceiveWindow and MaxGrownStreamReceiveWindow to 64 MiB.
MaxGrownConnReceiveWindow int64
// MaxGrownMemory is the connection window growth all trusted connections of a Listener share, in bytes (a dialed Conn has its own); growth stops when it is used up and returns when a connection ends. WindowGrowth only. Default 64 MiB, from 1 MiB to 64 GiB.
MaxGrownMemory int64
// Network selects the UDP network of the socket Listen opens. Listener only. Default "udp4", the only accepted value: IPv6 is refused.
Network string
// CongestionControl picks the controller that paces what this end sends: "reno" (the default), "cubic" or "bbr". "bbr" keeps a link full where random loss makes the others crawl, but takes the bandwidth of other flows and holds a standing queue (see the guide).
CongestionControl string
}Config sets the limits of a Listener or a dialed Conn. A zero field takes its default. Config is not comparable.
type Conn
type Conn struct {
// contains filtered or unexported fields
}Conn is a connection to one peer, carrying multiplexed streams.
func Dial
func Dial(ctx context.Context, addr string, cfg *Config) (*Conn, error)Dial connects to the listener at an IPv4 addr from a new udp4 socket (on loopback for a loopback addr), released when the connection ends.
func (Conn) AcceptStream
func (c *Conn) AcceptStream(ctx context.Context) (*Stream, error)AcceptStream waits for the next bidirectional stream opened by the peer.
func (Conn) AcceptUniStream
func (c *Conn) AcceptUniStream(ctx context.Context) (*ReceiveStream, error)AcceptUniStream waits for the next unidirectional stream opened by the peer.
func (Conn) AllowWindowGrowth
func (c *Conn) AllowWindowGrowth() boolAllowWindowGrowth marks the connection as trusted, so its receive windows can grow up to the MaxGrown values while MaxGrownMemory lasts. Call it once the peer is one the application authorized, such as after a handshake that proved its identity; a connection never marked keeps its initial windows. It reports whether growth is on for the connection: false when Config.WindowGrowth is off or the connection has ended. It is safe to call more than once.
func (Conn) Close
func (c *Conn) Close() errorClose closes the connection with code 0.
func (Conn) CloseWithError
func (c *Conn) CloseWithError(code uint64, msg string) errorCloseWithError closes the connection, sending code and msg to the peer. An invalid code or msg (see ErrInvalidArgument) still closes the connection, with code 0 and no message.
func (Conn) Context
func (c *Conn) Context() context.ContextContext is canceled when the connection ends; its context.Cause is the error the connection's calls return.
func (Conn) ExportKey
func (c *Conn) ExportKey(label string) ([32]byte, error)ExportKey derives 32 bytes from the TLS session (RFC 9846 exporter); both ends get the same bytes only without a man in the middle. Use a unique non-empty label such as "kolo-<lib>/<purpose>/v0", and sign label, role and key together, never the key alone.
func (Conn) LocalAddr
func (c *Conn) LocalAddr() net.AddrLocalAddr returns the local UDP address.
func (Conn) OpenStream
func (c *Conn) OpenStream(ctx context.Context) (*Stream, error)OpenStream opens a bidirectional stream, waiting while the peer's stream limit is reached. The peer sees the stream only once data or a Close is sent on it.
func (Conn) OpenUniStream
func (c *Conn) OpenUniStream(ctx context.Context) (*SendStream, error)OpenUniStream opens a unidirectional stream, waiting while the peer's stream limit is reached.
func (Conn) RemoteAddr
func (c *Conn) RemoteAddr() net.AddrRemoteAddr returns the peer's UDP address.
func (Conn) Stats
func (c *Conn) Stats() ConnStatsStats returns a snapshot of the connection's statistics. It is cheap, does not wait on a network read and is safe from any goroutine. The fields are read one by one, not atomically, so a moving value can disagree with its neighbours. After the connection has ended it returns the zero ConnStats.
type ConnStats
type ConnStats struct {
// SmoothedRTT is the exponentially weighted moving average of the round-trip time samples (RFC 9002, section 5.3).
// It is 0 until the current path has a sample, which a new connection has once its handshake completes.
SmoothedRTT time.Duration
// MinRTT is the lowest round-trip time seen on the current path, which is close to the propagation delay.
MinRTT time.Duration
// LatestRTT is the last round-trip time sample on the current path.
LatestRTT time.Duration
// RTTVariation is the mean deviation of the samples from SmoothedRTT (the RTTVAR of RFC 9002), a measure of jitter.
RTTVariation time.Duration
// BytesSent counts every byte sent, including packets later declared lost and retransmitted data.
BytesSent uint64
// BytesReceived counts every byte received, including duplicates and packets that could not be processed.
BytesReceived uint64
// PacketsSent counts every packet sent, including packets later declared lost.
PacketsSent uint64
// PacketsReceived counts every packet received, including duplicates and packets that could not be processed.
PacketsReceived uint64
// PacketsLost counts the packets of this end the peer did not acknowledge in time, and each congestion signal of ECN
// (a peer that marks packets can raise it without any loss, as the built-in sender counts it too). A packet that arrives
// late after it was declared lost still counts, so PacketsLost over PacketsSent is an upper bound of the loss rate.
PacketsLost uint64
// BytesLost is the size in bytes of the packets counted in PacketsLost.
BytesLost uint64
// CongestionWindow is how many bytes this end may have in flight, as set by the congestion controller.
// A transfer is limited by the congestion window when BytesInFlight stays near it.
CongestionWindow uint64
// BytesInFlight is how many bytes of packets this end sent that the peer has not yet acknowledged and this end has not declared lost.
BytesInFlight uint64
// CongestionController is the name of the controller sending on this connection: "reno", "cubic" or "bbr".
CongestionController string
// BandwidthEstimate is the bottleneck bandwidth the controller measured, in bytes per second: the highest delivery
// rate of the last ten round trips. Only "bbr" measures it; it is 0 for "reno" and "cubic" and before the first sample.
BandwidthEstimate uint64
// MaxDatagramSize is the largest UDP payload in bytes this end sends on the current path. It starts at the
// initial size and grows as path MTU discovery succeeds.
MaxDatagramSize uint64
// PathChanges counts how often the connection moved to a new network path, for example when the peer's NAT gave it a new port.
// The RTT fields, the congestion window and BandwidthEstimate start over on a new path.
PathChanges uint64
}ConnStats is a snapshot of what a connection measured about its network path and its own traffic. Byte counts are QUIC packet bytes: they include QUIC headers and retransmissions, not UDP or IP headers.
type Error
type Error struct {
Err error
Code uint64
}Error is a connection or stream error with its numeric code; Unwrap returns one of the sentinels.
func (Error) Error
func (e *Error) Error() stringError returns the sentinel text and the code.
func (Error) Unwrap
func (e *Error) Unwrap() errorUnwrap returns the sentinel.
type Listener
type Listener struct {
// contains filtered or unexported fields
}Listener accepts connections on one UDP socket and can dial from it.
func Listen
func Listen(ctx context.Context, addr string, cfg *Config) (*Listener, error)Listen opens a UDP socket on addr, such as "127.0.0.1:0", and accepts connections on it.
func ListenConn
func ListenConn(ctx context.Context, pc net.PacketConn, cfg *Config) (*Listener, error)ListenConn is Listen on pc, which the Listener owns and closes, also when ListenConn fails (see the README). quic-go may set the socket's buffer sizes and DF bit.
func (Listener) Accept
func (l *Listener) Accept(ctx context.Context) (*Conn, error)Accept waits for the next connection whose handshake has completed. Up to 32 that are never accepted stay open until Close.
func (Listener) Addr
func (l *Listener) Addr() net.AddrAddr returns the local address of the listener.
func (Listener) Close
func (l *Listener) Close() errorClose refuses new connections, closes every connection of the listener with code 0, then the socket. It is safe to call more than once.
func (Listener) Dial
func (l *Listener) Dial(ctx context.Context, addr netip.AddrPort) (*Conn, error)Dial connects to addr from the listener's socket, with the listener's Config; MaxDialed bounds these connections.
func (Listener) ReadPacket
func (l *Listener) ReadPacket(ctx context.Context, b []byte) (int, netip.AddrPort, error)ReadPacket waits for the next non-QUIC datagram of at most MaxPacketSize bytes; the sender may be spoofed (see the README). A b shorter than the datagram gets its first len(b) bytes and the rest is dropped without an error; the count is the bytes copied.
func (Listener) WritePacket
func (l *Listener) WritePacket(b []byte, to netip.AddrPort) errorWritePacket sends b as one non-QUIC datagram from the listener's socket, blocking as long as its WriteTo does; b[0]&0xC0 must be 0.
type ReceiveStream
type ReceiveStream struct {
// contains filtered or unexported fields
}ReceiveStream is the receiving end of a unidirectional stream.
func (ReceiveStream) CancelRead
func (s *ReceiveStream) CancelRead(code uint64) errorCancelRead asks the peer to stop sending, with code (0 and ErrInvalidArgument if above MaxCode).
func (ReceiveStream) Read
func (s *ReceiveStream) Read(p []byte) (int, error)Read reads data from the peer; it returns io.EOF after the peer closed the stream.
func (ReceiveStream) SetReadDeadline
func (s *ReceiveStream) SetReadDeadline(t time.Time) errorSetReadDeadline sets the read deadline; an expired deadline returns os.ErrDeadlineExceeded.
type SendStream
type SendStream struct {
// contains filtered or unexported fields
}SendStream is the sending end of a unidirectional stream.
func (SendStream) CancelWrite
func (s *SendStream) CancelWrite(code uint64) errorCancelWrite abandons the stream with code (0 and ErrInvalidArgument if above MaxCode).
func (SendStream) Close
func (s *SendStream) Close() errorClose ends the stream.
func (SendStream) SetWriteDeadline
func (s *SendStream) SetWriteDeadline(t time.Time) errorSetWriteDeadline sets the write deadline; an expired deadline returns os.ErrDeadlineExceeded.
func (SendStream) Write
func (s *SendStream) Write(p []byte) (int, error)Write writes data to the peer, blocking while flow control is exhausted.
type Stream
type Stream struct {
// contains filtered or unexported fields
}Stream is a bidirectional stream. Read and Write may be used from different goroutines.
func (Stream) CancelRead
func (s *Stream) CancelRead(code uint64) errorCancelRead asks the peer to stop sending, with code (0 and ErrInvalidArgument if above MaxCode).
func (Stream) CancelWrite
func (s *Stream) CancelWrite(code uint64) errorCancelWrite abandons the send side with code (0 and ErrInvalidArgument if above MaxCode).
func (Stream) Close
func (s *Stream) Close() errorClose ends the send side; the receive side stays open.
func (Stream) Read
func (s *Stream) Read(p []byte) (int, error)Read reads data from the peer; it returns io.EOF after the peer closed its send side.
func (Stream) SetDeadline
func (s *Stream) SetDeadline(t time.Time) errorSetDeadline sets the read and write deadlines; an expired deadline returns os.ErrDeadlineExceeded.
func (Stream) SetReadDeadline
func (s *Stream) SetReadDeadline(t time.Time) errorSetReadDeadline sets the read deadline.
func (Stream) SetWriteDeadline
func (s *Stream) SetWriteDeadline(t time.Time) errorSetWriteDeadline sets the write deadline.
func (Stream) Write
func (s *Stream) Write(p []byte) (int, error)Write writes data to the peer, blocking while flow control is exhausted.
Good to know
Limits
v0.1.x is IPv4 only. Sockets are udp4. An IPv6 address, the v4-mapped form
::ffff:a.b.c.d included, is refused with ErrInvalidArgument, and a
Network other than "udp4" with ErrInvalidConfig. A host name with no
IPv4 address fails to resolve.
Config fields (nil or zero means the default; out of range is
ErrInvalidConfig):
MaxConns: 100 connections per listener, handshaking or open, from 1 to 10000. Past it, new attempts are refused and the dialer getsErrRefused.MaxConnsPerIP: 4 per source address (orMaxConnsif lower), from 1 toMaxConns. Past it, attempts from that address are refused.MaxDialed: 32 connections open fromListener.Dial, from 1 to 10000. Past it,Listener.DialreturnsErrMaxDialed. These do not count againstMaxConnsorMaxConnsPerIP.MaxIncomingStreams: 16 concurrent streams the peer may open per connection, bidirectional and unidirectional each, from 1 to 1000. Past it, the peer'sOpenStreamwaits.HandshakeTimeout: 10s, from 1s to 60s. It is also the life of a Retry token, so a peer whose round trip is longer than about half of it cannot connect.IdleTimeout: 30s, from 1s to 10m. Each side's idle timeout is min(ownIdleTimeout, max(peer'sIdleTimeout, 5s)), so below 5s the two sides can time out at different moments.KeepAlivePeriod: 5s (orIdleTimeout/2 if lower), from 500ms to belowIdleTimeout. Keep-alive is always on.MaxStreamReceiveWindow: 256 KiB of unread data per stream, from 64 KiB to 16 MiB.MaxConnReceiveWindow: 512 KiB of unread data per connection, from 64 KiB to 64 MiB, at leastMaxStreamReceiveWindow.MaxMemory: 1 GiB, from 64 MiB to 64 GiB.Listenchecks (MaxConns+MaxDialed) ×MaxConnReceiveWindow≤MaxMemoryand returnsErrInvalidConfigotherwise. The defaults need 66 MiB. It checks unread data, not heap.CongestionControl:"reno"(the default),"cubic"or"bbr". Any other value isErrInvalidConfig. See "Congestion control" below.WindowGrowth: off by default. With it on,Conn.AllowWindowGrowthlets a connection's receive windows grow up toMaxGrownStreamReceiveWindow(4 MiB, fromMaxStreamReceiveWindowto 16 MiB) andMaxGrownConnReceiveWindow(8 MiB, fromMaxConnReceiveWindowto 64 MiB), withinMaxGrownMemory(64 MiB, from 1 MiB to 64 GiB, shared by the connections of aListener). The threeMaxGrownoptions needWindowGrowth.
Fixed values:
- Accept queue: 32 handshaked connections that wait for
Accept. Past it, new connections are refused. CallAccepteven on a listener used only to dial. - Non-QUIC datagrams:
MaxPacketSizeis 1200 bytes.WritePacketrefuses a larger, empty, or QUIC-looking one (b[0]&0xC0 != 0) withErrInvalidArgument, andReadPacketdrops a larger one. The queue holds 32 datagrams (about 45 KiB) and drops the rest, so read promptly. CloseWithError: the code is at mostMaxCode(2^62-1) and the message at most 256 bytes. Out of range, the connection still closes, with code 0, and the call returnsErrInvalidArgument.CancelReadandCancelWritetreat a code aboveMaxCodethe same way.ExportKeyreturns 32 bytes and needs a non-empty label.- NEW_TOKEN tokens expire at any age, even within one clock tick or after the
wall clock steps back (
MaxTokenAgeis the smallest duration), so every new attempt goes through Retry. Listener.Closecloses aPacketConnthat ignores read deadlines after 1s.- With
ListenConn, the conn must carry datagrams of at least 1280 bytes; anything over 1452 bytes is truncated.
Heap is not bounded by MaxMemory. Measured in the README, with 1-byte
frames each unread byte costs about 176 heap bytes, and each byte read on a
stream still open keeps about 96. The worst case per connection is:
2 × MaxIncomingStreams × MaxStreamReceiveWindow × 96
+ MaxConnReceiveWindow × 176At the defaults that is about 856 MiB per connection. To bound the heap,
lower MaxConns, MaxIncomingStreams or the windows, set GOMEMLIMIT, read
streams promptly and close them in both directions when done.
Faster transfers for trusted peers
A stream moves at most window / round trip: with the default 256 KiB stream window, about 21 Mbit/s at 100 ms. The windows are small because a hostile peer can make an application hold up to 176 heap bytes per unread byte, so they start small for every connection and grow only for one the application has marked as trusted, within a budget that all of them share.
cfg := &kolotransport.Config{WindowGrowth: true}
l, err := kolotransport.Listen(ctx, "127.0.0.1:0", cfg)
// ...
conn, err := l.Accept(ctx)
// The application learns who the peer is, for example from a handshake that
// proved its key, and decides it is one it wants to serve fast:
if peerIsAuthorized(conn) {
conn.AllowWindowGrowth()
}- kolo-transport never learns who a peer is, so it cannot decide. The
application calls
Conn.AllowWindowGrowthonce its own layer has authenticated the peer and its policy admits it. It returns false whenWindowGrowthis off, which is the default and changes nothing, or when the connection has ended. - Growth is the auto-tuning of the receive window that the QUIC stack has: a window doubles while the
application reads it faster than a few round trips, up to
MaxGrownStreamReceiveWindow(4 MiB, up to 16 MiB) andMaxGrownConnReceiveWindow(8 MiB, up to 64 MiB). It never shrinks. MaxGrownMemory(64 MiB, from 1 MiB to 64 GiB) is the growth that all marked connections of oneListenercan hold together,Listener.Dialconnections included. A step of growth is refused when the budget is not enough, and the budget returns when the connection ends.Listenalso checks thatMaxMemorycovers the initial windows plus that growth.- When you turn growth on for a public socket, keep
MaxConnReceiveWindowequal toMaxStreamReceiveWindow: the QUIC stack does not let the application refuse the growth of a stream window, only of the connection's, so with equal values a connection nobody marked stays at its start, as without growth.
Measured with 100 Mbit/s each way and one stream read as fast as it arrives, a marked connection read at the link rate, 12.24 MB/s, at 20 ms, 50 ms and 100 ms of round trip, against 10.10, 4.43 and 2.01 MB/s with the fixed windows.
Congestion control
Config.CongestionControl picks the controller that paces what this end
sends: "reno", the default, "cubic" or "bbr". Reno halves
its window at every lost packet, so on a path that also loses packets at
random, as a mobile link can, one stream over 20 Mbit/s with 1 percent loss and
120 ms of round trip reads at 1.4 to 1.8 Mbit/s. BBR estimates the rate of the
link instead and does not slow down for isolated losses.
cfg := &kolotransport.Config{CongestionControl: "bbr"}
conn, err := kolotransport.Dial(ctx, addr, cfg)- Each end paces its own sending, so set it on the end that sends the data, or
on both for a transfer that goes both ways. It applies to every connection the
Configmakes (Dial,Listener.Accept,Listener.Dial), and a connection keeps its controller when a NAT gives the dialer a new port. Any other value isErrInvalidConfig. - Measured over a simulated link of 20 Mbit/s each way, 60 ms of delay each way
and 1 percent random loss each way, with 8 MiB windows: one stream reads at
1.83 Mbit/s with Reno and 18.86 Mbit/s with BBR.
"cubic"does not help on that path, since it backs off at every loss like Reno. The paths measured have independent random loss only, not the bursts, reordering or policers of a real network. - BBR is only as fast as the receive window: with the default windows a stream
moves at most window / round trip, so use it together with larger windows or
WindowGrowth. - Keep calls and live media on
"reno". BBR keeps a standing queue in the bottleneck (about 100 ms, 150 ms with a bufferbloated router in the measurements) and takes bandwidth from Reno flows that share it: it is for a transfer the user asked for and wants fast, not for background traffic. - A BBR connection keeps an 88-byte sample per packet number between the
oldest packet it waits for and the newest it sent. The span is capped at
40000 packet numbers, so a peer that never acknowledges cannot make it grow:
the oldest samples are dropped, which costs only bandwidth samples. The cap
bounds it at 5.5 MiB per BBR connection.
"reno"and"cubic"have no such structure.
Connection statistics
Conn.Stats() returns a ConnStats snapshot of what one connection measured.
It is cheap (a few atomic reads), never blocks on the network, is safe from
any goroutine, and returns the zero value once the connection has ended, so
read it before you close the connection if you want the final numbers. Each
field is read on its own, so two fields can disagree for one sample.
s := conn.Stats()
fmt.Printf("rtt %v (min %v) window %d in flight %d lost %d of %d packets via %s\n",
s.SmoothedRTT, s.MinRTT, s.CongestionWindow, s.BytesInFlight, s.PacketsLost, s.PacketsSent, s.CongestionController)SmoothedRTT,MinRTT,LatestRTTandRTTVariation: the round trip of the current path and its jitter.BytesSent,BytesReceived,PacketsSentandPacketsReceived: QUIC packets over the life of the connection, retransmissions and duplicates included.PacketsLostandBytesLost: packets this end sent that were not acknowledged in time. An ECN congestion signal counts as one lost packet.CongestionWindowandBytesInFlight: what the controller allows in flight and what is in flight now.CongestionController:"reno","cubic"or"bbr".BandwidthEstimate: the bottleneck bandwidth BBR measured, in bytes per second, 0 for the other two.MaxDatagramSize: the largest UDP payload this end sends on the current path, at most 1452 bytes.PathChanges: how often the connection moved to a new path; the round-trip fields, the window and the estimate start over on a new path.
BytesInFlight near CongestionWindow means the controller limits the
transfer. A SmoothedRTT far above MinRTT means a queue on the path.
BytesInFlight far below CongestionWindow while the transfer is slow means
the network is not the limit: the application does not write fast enough, or
the peer's receive windows are full.
Security
AcceptFuncfilters by source IP, not identity. Filter by IP, never by port: a Retry token proves only the IP, and can be replayed from that IP forHandshakeTimeout. A peer can move its connection to another address after the handshake. IfAcceptFuncreturns false or panics, the attempt is refused. It runs on the goroutine that handles every new attempt, so keep it fast and non-blocking.- The sender of
ReadPacketmay be spoofed. Do not dial it, and do not answer with more bytes than it sent, until it has proved it receives at that address. Non-QUIC datagrams are not filtered byAcceptFuncand carry no protection. Dial,Listener.DialandWritePacketrefuse port 0, IPv6, unspecified, multicast,0.0.0.0/8and240.0.0.0/4destinations. Loopback, private, link-local and subnet broadcast addresses are allowed, so filter any address a peer supplies before you dial or write to it.- The sockets
ListenandDialcreate haveSO_BROADCASTcleared.ListenConnleaves it as is. - Errors never include text from the peer, only a numeric code.
Known limits
- Experimental. The API may change before v1.0.0.
- QUIC datagrams are not exposed in v0.1.x.
- kolo-transport carries its own copy of its QUIC implementation, a pinned
release plus one patch: the hook for a congestion controller and the
counters
Conn.Statsreads. A script and a test check that the copy is the pinned release plus that patch and nothing else. A fix to the QUIC implementation reaches kolo-transport only when the copy is updated. - The package
Dialhas no cap on the number of connections, so a node that dials N peers can hold N ×MaxConnReceiveWindowof unread data. - Not bounded: the number of distinct addresses (peers on more than
MaxConns/MaxConnsPerIPaddresses can still fillMaxConns), the CPU and bandwidth spent per packet, and the OS UDP receive buffer. - On
ListenConnwith a conn that is not a*net.UDPConn: no ECN, no GSO, no batched reads, and path MTU discovery is off, so packets stay at 1280 bytes.
Errors
| Error | Message |
|---|---|
ErrClosed | kolotransport: closed |
ErrHandshake | kolotransport: handshake failed |
ErrInvalidArgument | kolotransport: invalid argument |
ErrInvalidConfig | kolotransport: invalid config |
ErrMaxDialed | kolotransport: too many dialed connections |
ErrPeerClosed | kolotransport: peer closed the connection |
ErrRefused | kolotransport: connection refused |
ErrStreamCanceled | kolotransport: stream canceled |
ErrStreamReset | kolotransport: stream reset by peer |
ErrTimeout | kolotransport: timeout |
ErrTransport | kolotransport: transport error |
Test errors with errors.Is against the sentinels. Errors with a code are an
*Error; use errors.As to read Code, and its Unwrap returns the
sentinel. context.Cause of Conn.Context gives the same errors.
References
- RFC 9000, QUIC: A UDP-Based Multiplexed and Secure Transport (Proposed Standard, May 2021).
- Takes: QUIC version 1, Retry on every new connection (8.1.2), 1200-byte first packets (14), stream and error-code semantics.
- Not implemented: 0-RTT, session resumption, QUIC version 2 (RFC 9369), unreliable datagrams (RFC 9221).
- RFC 9001, Using TLS to Secure QUIC (Proposed Standard, May 2021).
- Takes: the TLS 1.3 handshake inside QUIC, ALPN
kolo/0. - Not implemented: session tickets, early data.
- Takes: the TLS 1.3 handshake inside QUIC, ALPN
- RFC 9002, QUIC Loss Detection and Congestion Control (Proposed Standard, May 2021).
- Takes: RTT estimation (5.3), loss detection, and the Reno controller (7) that is the default.
- Not implemented: nothing removed.
- RFC 6582, The NewReno Modification to TCP's Fast Recovery Algorithm (Proposed Standard, April 2012).
- Takes: one window reduction per loss episode.
- RFC 9438, CUBIC for Fast and Long-Distance Networks (Proposed Standard, August 2023).
- Takes: the opt-in
"cubic"controller. - Not implemented: conformance to every section was not checked.
- Takes: the opt-in
- Cardwell, Cheng, Gunn, Hassas Yeganeh and Jacobson, BBR: Congestion-Based Congestion Control, ACM Queue 14(5), 2016.
- Takes: the opt-in
"bbr"controller: bottleneck bandwidth and round-trip estimates, pacing. - Not implemented: BBRv3 of draft-ietf-ccwg-bbr; loss is not a signal.
- Takes: the opt-in
- RFC 8899, Packetization Layer Path MTU Discovery for Datagram Transports (Proposed Standard, September 2020).
- Takes: path MTU discovery above 1200 bytes.
- RFC 9846, The Transport Layer Security (TLS) Protocol Version 1.3, section 7.5 (Proposed Standard, July 2026; obsoletes RFC 8446).
- Takes: the exporter behind
Conn.ExportKey.
- Takes: the exporter behind