kolo docsbeta

kolo-transport

One QUIC connection between two Kolo peers, with multiplexed streams and hard limits.

On this page

kolo-transport opens QUIC version 1 (RFC 9000) connections over UDP, secured by TLS 1.3 (RFC 9001), with the loss recovery and Reno congestion control of RFC 9002 and an optional BBR controller after Cardwell et al. It gives each peer bidirectional and unidirectional streams, with hard limits on unread data, streams and connections. The connection is encrypted, but the peer is not authenticated: kolo-secure adds identity on top, using kolo-id keys.

Install

go get github.com/ciranda-dev/kolo-transport

Builds without cgo.

Quick start

A listener and a dialer in one program on 127.0.0.1. The dialer opens a stream, and the listener echoes it back.

main.go
package main

import (
	"context"
	"fmt"
	"io"
	"log"
	"time"

	kolotransport "github.com/ciranda-dev/kolo-transport"
)

func main() {
	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
	defer cancel()

	l, err := kolotransport.Listen(ctx, "127.0.0.1:0", nil)
	if err != nil {
		log.Fatal(err)
	}
	defer l.Close()
	go func() {
		c, err := l.Accept(ctx)
		if err != nil {
			return
		}
		s, err := c.AcceptStream(ctx)
		if err != nil {
			return
		}
		io.Copy(s, s)
		s.Close()
	}()

	c, err := kolotransport.Dial(ctx, l.Addr().String(), nil)
	if err != nil {
		log.Fatal(err)
	}
	defer c.Close()
	s, err := c.OpenStream(ctx)
	if err != nil {
		log.Fatal(err)
	}
	if _, err := s.Write([]byte("hello kolo")); err != nil {
		log.Fatal(err)
	}
	s.Close()
	reply, err := io.ReadAll(s)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("%s\n", reply)
}

It prints hello kolo. s.Close() ends only the send side, so the dialer can still read the reply.

API reference

The API reference is generated from the Go source and is in English.

Package kolotransport · github.com/ciranda-dev/kolo-transport · v0.1.0

Package kolotransport carries multiplexed streams over one QUIC (RFC 9000) connection between two Kolo peers.

It does not authenticate the peer. Each Listener presents an ephemeral self-signed certificate and the dialer accepts any such certificate, so TLS 1.3 gives confidentiality and integrity against passive observers but no identity, and an active man in the middle is not detected. Identity must come from a higher layer, for example kolo-id signatures over ExportKey.

Experimental: the API may change before v1.0.0.

Constants

const ALPN = "kolo/0"

ALPN is the TLS application protocol both ends must offer.

const DefaultMaxConnReceiveWindow = 512 << 10

DefaultMaxConnReceiveWindow is the default MaxConnReceiveWindow: 512 KiB.

const DefaultMaxGrownConnReceiveWindow = 8 << 20

DefaultMaxGrownConnReceiveWindow is the default MaxGrownConnReceiveWindow: 8 MiB.

const DefaultMaxGrownMemory = 64 << 20

DefaultMaxGrownMemory is the default MaxGrownMemory: 64 MiB.

const DefaultMaxGrownStreamReceiveWindow = 4 << 20

DefaultMaxGrownStreamReceiveWindow is the default MaxGrownStreamReceiveWindow: 4 MiB.

const DefaultMaxMemory = 1 << 30

DefaultMaxMemory is the default MaxMemory: 1 GiB of receive window.

const DefaultMaxStreamReceiveWindow = 256 << 10

DefaultMaxStreamReceiveWindow is the default MaxStreamReceiveWindow: 256 KiB.

const MaxCode uint64 = 1<<62 - 1

MaxCode is the largest error code QUIC can carry.

const MaxPacketSize = 1200

MaxPacketSize is the largest non-QUIC datagram WritePacket sends and ReadPacket returns.

Variables

var (
	// ErrInvalidConfig means a Config field is out of range.
	ErrInvalidConfig = errors.New("kolotransport: invalid config")
	// ErrInvalidArgument means a bad code, close message, ExportKey label, PacketConn, listen or dial address, packet or destination (see the README).
	ErrInvalidArgument = errors.New("kolotransport: invalid argument")
	// ErrClosed means this side closed the listener, connection or stream.
	ErrClosed = errors.New("kolotransport: closed")
	// ErrPeerClosed means the peer closed the connection or no longer has state for it.
	ErrPeerClosed = errors.New("kolotransport: peer closed the connection")
	// ErrRefused means the peer refused the connection, for example because its listener is full.
	ErrRefused = errors.New("kolotransport: connection refused")
	// ErrHandshake means the TLS or QUIC handshake failed.
	ErrHandshake = errors.New("kolotransport: handshake failed")
	// ErrTimeout means the handshake or idle timeout expired.
	ErrTimeout = errors.New("kolotransport: timeout")
	// ErrTransport means a QUIC transport error or another failure of the connection.
	ErrTransport = errors.New("kolotransport: transport error")
	// ErrStreamReset means the peer reset the stream or asked this side to stop sending.
	ErrStreamReset = errors.New("kolotransport: stream reset by peer")
	// ErrMaxDialed means Listener.Dial already has MaxDialed connections open.
	ErrMaxDialed = errors.New("kolotransport: too many dialed connections")
	// ErrStreamCanceled means this side canceled the stream.
	ErrStreamCanceled = errors.New("kolotransport: stream canceled")
)

Types

type Config

type Config struct {
	// MaxConns caps the connections of a Listener, handshaking or open. Listener only. Default 100, from 1 to 10000.
	MaxConns int
	// MaxConnsPerIP caps them per source address, the one a connection started from. Listener only. Default 4 (or MaxConns if lower), at most MaxConns.
	MaxConnsPerIP int
	// MaxDialed caps the connections Listener.Dial has open, which MaxConns and MaxConnsPerIP do not count. Listener only. Default 32, from 1 to 10000.
	MaxDialed int
	// MaxIncomingStreams caps the concurrent streams the peer may open, bidirectional and unidirectional each. Default 16, from 1 to 1000.
	MaxIncomingStreams int
	// HandshakeTimeout bounds the whole handshake and the life of a Retry token. Default 10s, from 1s to 60s.
	HandshakeTimeout time.Duration
	// IdleTimeout closes a connection with no network activity. Default 30s, from 1s to 10m.
	IdleTimeout time.Duration
	// KeepAlivePeriod sends a PING after this long without receiving, so connections stay open, also those never accepted.
	// Default 5s (or IdleTimeout/2 if lower), from 500ms to below IdleTimeout.
	KeepAlivePeriod time.Duration
	// MaxMemory must cover (MaxConns + MaxDialed) times MaxConnReceiveWindow (unread data, not heap), in bytes; the default leaves headroom and does not bound heap (see the README). Listener only. Default 1 GiB, from 64 MiB to 64 GiB.
	MaxMemory int64
	// AcceptFunc filters attempts by the source IP a connection starts from, never port or identity; false or a panic refuses.
	// Listener only. Nil allows all.
	AcceptFunc func(remote netip.AddrPort) bool
	// MaxStreamReceiveWindow caps the unread data buffered per stream, in bytes, on Listen and Dial. Advanced. Default 256 KiB (or MaxConnReceiveWindow if lower), from 64 KiB to 16 MiB.
	MaxStreamReceiveWindow int64
	// MaxConnReceiveWindow caps the unread data buffered per connection, in bytes; a node that dials N peers can hold N times it. Advanced. Default 512 KiB, from 64 KiB to 64 MiB, at least MaxStreamReceiveWindow.
	MaxConnReceiveWindow int64
	// WindowGrowth lets Conn.AllowWindowGrowth raise the receive windows of a connection the application trusts, up to the Grown values below; every other connection keeps its initial windows. Default false: no window ever grows.
	WindowGrowth bool
	// MaxGrownStreamReceiveWindow is the largest stream window a trusted connection can reach, in bytes. WindowGrowth only. Default 4 MiB (or MaxStreamReceiveWindow if higher), from MaxStreamReceiveWindow to 16 MiB.
	MaxGrownStreamReceiveWindow int64
	// MaxGrownConnReceiveWindow is the largest connection window a trusted connection can reach, in bytes. WindowGrowth only. Default 8 MiB (or MaxConnReceiveWindow if higher), from MaxConnReceiveWindow and MaxGrownStreamReceiveWindow to 64 MiB.
	MaxGrownConnReceiveWindow int64
	// MaxGrownMemory is the connection window growth all trusted connections of a Listener share, in bytes (a dialed Conn has its own); growth stops when it is used up and returns when a connection ends. WindowGrowth only. Default 64 MiB, from 1 MiB to 64 GiB.
	MaxGrownMemory int64
	// Network selects the UDP network of the socket Listen opens. Listener only. Default "udp4", the only accepted value: IPv6 is refused.
	Network string
	// CongestionControl picks the controller that paces what this end sends: "reno" (the default), "cubic" or "bbr". "bbr" keeps a link full where random loss makes the others crawl, but takes the bandwidth of other flows and holds a standing queue (see the guide).
	CongestionControl string
}

Config sets the limits of a Listener or a dialed Conn. A zero field takes its default. Config is not comparable.

type Conn

type Conn struct {
	// contains filtered or unexported fields
}

Conn is a connection to one peer, carrying multiplexed streams.

func Dial
func Dial(ctx context.Context, addr string, cfg *Config) (*Conn, error)

Dial connects to the listener at an IPv4 addr from a new udp4 socket (on loopback for a loopback addr), released when the connection ends.

func (Conn) AcceptStream
func (c *Conn) AcceptStream(ctx context.Context) (*Stream, error)

AcceptStream waits for the next bidirectional stream opened by the peer.

func (Conn) AcceptUniStream
func (c *Conn) AcceptUniStream(ctx context.Context) (*ReceiveStream, error)

AcceptUniStream waits for the next unidirectional stream opened by the peer.

func (Conn) AllowWindowGrowth
func (c *Conn) AllowWindowGrowth() bool

AllowWindowGrowth marks the connection as trusted, so its receive windows can grow up to the MaxGrown values while MaxGrownMemory lasts. Call it once the peer is one the application authorized, such as after a handshake that proved its identity; a connection never marked keeps its initial windows. It reports whether growth is on for the connection: false when Config.WindowGrowth is off or the connection has ended. It is safe to call more than once.

func (Conn) Close
func (c *Conn) Close() error

Close closes the connection with code 0.

func (Conn) CloseWithError
func (c *Conn) CloseWithError(code uint64, msg string) error

CloseWithError closes the connection, sending code and msg to the peer. An invalid code or msg (see ErrInvalidArgument) still closes the connection, with code 0 and no message.

func (Conn) Context
func (c *Conn) Context() context.Context

Context is canceled when the connection ends; its context.Cause is the error the connection's calls return.

func (Conn) ExportKey
func (c *Conn) ExportKey(label string) ([32]byte, error)

ExportKey derives 32 bytes from the TLS session (RFC 9846 exporter); both ends get the same bytes only without a man in the middle. Use a unique non-empty label such as "kolo-<lib>/<purpose>/v0", and sign label, role and key together, never the key alone.

func (Conn) LocalAddr
func (c *Conn) LocalAddr() net.Addr

LocalAddr returns the local UDP address.

func (Conn) OpenStream
func (c *Conn) OpenStream(ctx context.Context) (*Stream, error)

OpenStream opens a bidirectional stream, waiting while the peer's stream limit is reached. The peer sees the stream only once data or a Close is sent on it.

func (Conn) OpenUniStream
func (c *Conn) OpenUniStream(ctx context.Context) (*SendStream, error)

OpenUniStream opens a unidirectional stream, waiting while the peer's stream limit is reached.

func (Conn) RemoteAddr
func (c *Conn) RemoteAddr() net.Addr

RemoteAddr returns the peer's UDP address.

func (Conn) Stats
func (c *Conn) Stats() ConnStats

Stats returns a snapshot of the connection's statistics. It is cheap, does not wait on a network read and is safe from any goroutine. The fields are read one by one, not atomically, so a moving value can disagree with its neighbours. After the connection has ended it returns the zero ConnStats.

type ConnStats

type ConnStats struct {
	// SmoothedRTT is the exponentially weighted moving average of the round-trip time samples (RFC 9002, section 5.3).
	// It is 0 until the current path has a sample, which a new connection has once its handshake completes.
	SmoothedRTT time.Duration
	// MinRTT is the lowest round-trip time seen on the current path, which is close to the propagation delay.
	MinRTT time.Duration
	// LatestRTT is the last round-trip time sample on the current path.
	LatestRTT time.Duration
	// RTTVariation is the mean deviation of the samples from SmoothedRTT (the RTTVAR of RFC 9002), a measure of jitter.
	RTTVariation time.Duration

	// BytesSent counts every byte sent, including packets later declared lost and retransmitted data.
	BytesSent uint64
	// BytesReceived counts every byte received, including duplicates and packets that could not be processed.
	BytesReceived uint64
	// PacketsSent counts every packet sent, including packets later declared lost.
	PacketsSent uint64
	// PacketsReceived counts every packet received, including duplicates and packets that could not be processed.
	PacketsReceived uint64
	// PacketsLost counts the packets of this end the peer did not acknowledge in time, and each congestion signal of ECN
	// (a peer that marks packets can raise it without any loss, as the built-in sender counts it too). A packet that arrives
	// late after it was declared lost still counts, so PacketsLost over PacketsSent is an upper bound of the loss rate.
	PacketsLost uint64
	// BytesLost is the size in bytes of the packets counted in PacketsLost.
	BytesLost uint64

	// CongestionWindow is how many bytes this end may have in flight, as set by the congestion controller.
	// A transfer is limited by the congestion window when BytesInFlight stays near it.
	CongestionWindow uint64
	// BytesInFlight is how many bytes of packets this end sent that the peer has not yet acknowledged and this end has not declared lost.
	BytesInFlight uint64
	// CongestionController is the name of the controller sending on this connection: "reno", "cubic" or "bbr".
	CongestionController string
	// BandwidthEstimate is the bottleneck bandwidth the controller measured, in bytes per second: the highest delivery
	// rate of the last ten round trips. Only "bbr" measures it; it is 0 for "reno" and "cubic" and before the first sample.
	BandwidthEstimate uint64
	// MaxDatagramSize is the largest UDP payload in bytes this end sends on the current path. It starts at the
	// initial size and grows as path MTU discovery succeeds.
	MaxDatagramSize uint64
	// PathChanges counts how often the connection moved to a new network path, for example when the peer's NAT gave it a new port.
	// The RTT fields, the congestion window and BandwidthEstimate start over on a new path.
	PathChanges uint64
}

ConnStats is a snapshot of what a connection measured about its network path and its own traffic. Byte counts are QUIC packet bytes: they include QUIC headers and retransmissions, not UDP or IP headers.

type Error

type Error struct {
	Err  error
	Code uint64
}

Error is a connection or stream error with its numeric code; Unwrap returns one of the sentinels.

func (Error) Error
func (e *Error) Error() string

Error returns the sentinel text and the code.

func (Error) Unwrap
func (e *Error) Unwrap() error

Unwrap returns the sentinel.

type Listener

type Listener struct {
	// contains filtered or unexported fields
}

Listener accepts connections on one UDP socket and can dial from it.

func Listen
func Listen(ctx context.Context, addr string, cfg *Config) (*Listener, error)

Listen opens a UDP socket on addr, such as "127.0.0.1:0", and accepts connections on it.

func ListenConn
func ListenConn(ctx context.Context, pc net.PacketConn, cfg *Config) (*Listener, error)

ListenConn is Listen on pc, which the Listener owns and closes, also when ListenConn fails (see the README). quic-go may set the socket's buffer sizes and DF bit.

func (Listener) Accept
func (l *Listener) Accept(ctx context.Context) (*Conn, error)

Accept waits for the next connection whose handshake has completed. Up to 32 that are never accepted stay open until Close.

func (Listener) Addr
func (l *Listener) Addr() net.Addr

Addr returns the local address of the listener.

func (Listener) Close
func (l *Listener) Close() error

Close refuses new connections, closes every connection of the listener with code 0, then the socket. It is safe to call more than once.

func (Listener) Dial
func (l *Listener) Dial(ctx context.Context, addr netip.AddrPort) (*Conn, error)

Dial connects to addr from the listener's socket, with the listener's Config; MaxDialed bounds these connections.

func (Listener) ReadPacket
func (l *Listener) ReadPacket(ctx context.Context, b []byte) (int, netip.AddrPort, error)

ReadPacket waits for the next non-QUIC datagram of at most MaxPacketSize bytes; the sender may be spoofed (see the README). A b shorter than the datagram gets its first len(b) bytes and the rest is dropped without an error; the count is the bytes copied.

func (Listener) WritePacket
func (l *Listener) WritePacket(b []byte, to netip.AddrPort) error

WritePacket sends b as one non-QUIC datagram from the listener's socket, blocking as long as its WriteTo does; b[0]&0xC0 must be 0.

type ReceiveStream

type ReceiveStream struct {
	// contains filtered or unexported fields
}

ReceiveStream is the receiving end of a unidirectional stream.

func (ReceiveStream) CancelRead
func (s *ReceiveStream) CancelRead(code uint64) error

CancelRead asks the peer to stop sending, with code (0 and ErrInvalidArgument if above MaxCode).

func (ReceiveStream) Read
func (s *ReceiveStream) Read(p []byte) (int, error)

Read reads data from the peer; it returns io.EOF after the peer closed the stream.

func (ReceiveStream) SetReadDeadline
func (s *ReceiveStream) SetReadDeadline(t time.Time) error

SetReadDeadline sets the read deadline; an expired deadline returns os.ErrDeadlineExceeded.

type SendStream

type SendStream struct {
	// contains filtered or unexported fields
}

SendStream is the sending end of a unidirectional stream.

func (SendStream) CancelWrite
func (s *SendStream) CancelWrite(code uint64) error

CancelWrite abandons the stream with code (0 and ErrInvalidArgument if above MaxCode).

func (SendStream) Close
func (s *SendStream) Close() error

Close ends the stream.

func (SendStream) SetWriteDeadline
func (s *SendStream) SetWriteDeadline(t time.Time) error

SetWriteDeadline sets the write deadline; an expired deadline returns os.ErrDeadlineExceeded.

func (SendStream) Write
func (s *SendStream) Write(p []byte) (int, error)

Write writes data to the peer, blocking while flow control is exhausted.

type Stream

type Stream struct {
	// contains filtered or unexported fields
}

Stream is a bidirectional stream. Read and Write may be used from different goroutines.

func (Stream) CancelRead
func (s *Stream) CancelRead(code uint64) error

CancelRead asks the peer to stop sending, with code (0 and ErrInvalidArgument if above MaxCode).

func (Stream) CancelWrite
func (s *Stream) CancelWrite(code uint64) error

CancelWrite abandons the send side with code (0 and ErrInvalidArgument if above MaxCode).

func (Stream) Close
func (s *Stream) Close() error

Close ends the send side; the receive side stays open.

func (Stream) Read
func (s *Stream) Read(p []byte) (int, error)

Read reads data from the peer; it returns io.EOF after the peer closed its send side.

func (Stream) SetDeadline
func (s *Stream) SetDeadline(t time.Time) error

SetDeadline sets the read and write deadlines; an expired deadline returns os.ErrDeadlineExceeded.

func (Stream) SetReadDeadline
func (s *Stream) SetReadDeadline(t time.Time) error

SetReadDeadline sets the read deadline.

func (Stream) SetWriteDeadline
func (s *Stream) SetWriteDeadline(t time.Time) error

SetWriteDeadline sets the write deadline.

func (Stream) Write
func (s *Stream) Write(p []byte) (int, error)

Write writes data to the peer, blocking while flow control is exhausted.

Good to know

Limits

v0.1.x is IPv4 only. Sockets are udp4. An IPv6 address, the v4-mapped form ::ffff:a.b.c.d included, is refused with ErrInvalidArgument, and a Network other than "udp4" with ErrInvalidConfig. A host name with no IPv4 address fails to resolve.

Config fields (nil or zero means the default; out of range is ErrInvalidConfig):

  • MaxConns: 100 connections per listener, handshaking or open, from 1 to 10000. Past it, new attempts are refused and the dialer gets ErrRefused.
  • MaxConnsPerIP: 4 per source address (or MaxConns if lower), from 1 to MaxConns. Past it, attempts from that address are refused.
  • MaxDialed: 32 connections open from Listener.Dial, from 1 to 10000. Past it, Listener.Dial returns ErrMaxDialed. These do not count against MaxConns or MaxConnsPerIP.
  • MaxIncomingStreams: 16 concurrent streams the peer may open per connection, bidirectional and unidirectional each, from 1 to 1000. Past it, the peer's OpenStream waits.
  • HandshakeTimeout: 10s, from 1s to 60s. It is also the life of a Retry token, so a peer whose round trip is longer than about half of it cannot connect.
  • IdleTimeout: 30s, from 1s to 10m. Each side's idle timeout is min(own IdleTimeout, max(peer's IdleTimeout, 5s)), so below 5s the two sides can time out at different moments.
  • KeepAlivePeriod: 5s (or IdleTimeout/2 if lower), from 500ms to below IdleTimeout. Keep-alive is always on.
  • MaxStreamReceiveWindow: 256 KiB of unread data per stream, from 64 KiB to 16 MiB.
  • MaxConnReceiveWindow: 512 KiB of unread data per connection, from 64 KiB to 64 MiB, at least MaxStreamReceiveWindow.
  • MaxMemory: 1 GiB, from 64 MiB to 64 GiB. Listen checks (MaxConns + MaxDialed) × MaxConnReceiveWindow ≤ MaxMemory and returns ErrInvalidConfig otherwise. The defaults need 66 MiB. It checks unread data, not heap.
  • CongestionControl: "reno" (the default), "cubic" or "bbr". Any other value is ErrInvalidConfig. See "Congestion control" below.
  • WindowGrowth: off by default. With it on, Conn.AllowWindowGrowth lets a connection's receive windows grow up to MaxGrownStreamReceiveWindow (4 MiB, from MaxStreamReceiveWindow to 16 MiB) and MaxGrownConnReceiveWindow (8 MiB, from MaxConnReceiveWindow to 64 MiB), within MaxGrownMemory (64 MiB, from 1 MiB to 64 GiB, shared by the connections of a Listener). The three MaxGrown options need WindowGrowth.

Fixed values:

  • Accept queue: 32 handshaked connections that wait for Accept. Past it, new connections are refused. Call Accept even on a listener used only to dial.
  • Non-QUIC datagrams: MaxPacketSize is 1200 bytes. WritePacket refuses a larger, empty, or QUIC-looking one (b[0]&0xC0 != 0) with ErrInvalidArgument, and ReadPacket drops a larger one. The queue holds 32 datagrams (about 45 KiB) and drops the rest, so read promptly.
  • CloseWithError: the code is at most MaxCode (2^62-1) and the message at most 256 bytes. Out of range, the connection still closes, with code 0, and the call returns ErrInvalidArgument. CancelRead and CancelWrite treat a code above MaxCode the same way.
  • ExportKey returns 32 bytes and needs a non-empty label.
  • NEW_TOKEN tokens expire at any age, even within one clock tick or after the wall clock steps back (MaxTokenAge is the smallest duration), so every new attempt goes through Retry.
  • Listener.Close closes a PacketConn that ignores read deadlines after 1s.
  • With ListenConn, the conn must carry datagrams of at least 1280 bytes; anything over 1452 bytes is truncated.

Heap is not bounded by MaxMemory. Measured in the README, with 1-byte frames each unread byte costs about 176 heap bytes, and each byte read on a stream still open keeps about 96. The worst case per connection is:

Text
2 × MaxIncomingStreams × MaxStreamReceiveWindow × 96
  + MaxConnReceiveWindow × 176

At the defaults that is about 856 MiB per connection. To bound the heap, lower MaxConns, MaxIncomingStreams or the windows, set GOMEMLIMIT, read streams promptly and close them in both directions when done.

Faster transfers for trusted peers

A stream moves at most window / round trip: with the default 256 KiB stream window, about 21 Mbit/s at 100 ms. The windows are small because a hostile peer can make an application hold up to 176 heap bytes per unread byte, so they start small for every connection and grow only for one the application has marked as trusted, within a budget that all of them share.

Go
cfg := &kolotransport.Config{WindowGrowth: true}
l, err := kolotransport.Listen(ctx, "127.0.0.1:0", cfg)
// ...
conn, err := l.Accept(ctx)
// The application learns who the peer is, for example from a handshake that
// proved its key, and decides it is one it wants to serve fast:
if peerIsAuthorized(conn) {
	conn.AllowWindowGrowth()
}
  • kolo-transport never learns who a peer is, so it cannot decide. The application calls Conn.AllowWindowGrowth once its own layer has authenticated the peer and its policy admits it. It returns false when WindowGrowth is off, which is the default and changes nothing, or when the connection has ended.
  • Growth is the auto-tuning of the receive window that the QUIC stack has: a window doubles while the application reads it faster than a few round trips, up to MaxGrownStreamReceiveWindow (4 MiB, up to 16 MiB) and MaxGrownConnReceiveWindow (8 MiB, up to 64 MiB). It never shrinks.
  • MaxGrownMemory (64 MiB, from 1 MiB to 64 GiB) is the growth that all marked connections of one Listener can hold together, Listener.Dial connections included. A step of growth is refused when the budget is not enough, and the budget returns when the connection ends. Listen also checks that MaxMemory covers the initial windows plus that growth.
  • When you turn growth on for a public socket, keep MaxConnReceiveWindow equal to MaxStreamReceiveWindow: the QUIC stack does not let the application refuse the growth of a stream window, only of the connection's, so with equal values a connection nobody marked stays at its start, as without growth.

Measured with 100 Mbit/s each way and one stream read as fast as it arrives, a marked connection read at the link rate, 12.24 MB/s, at 20 ms, 50 ms and 100 ms of round trip, against 10.10, 4.43 and 2.01 MB/s with the fixed windows.

Congestion control

Config.CongestionControl picks the controller that paces what this end sends: "reno", the default, "cubic" or "bbr". Reno halves its window at every lost packet, so on a path that also loses packets at random, as a mobile link can, one stream over 20 Mbit/s with 1 percent loss and 120 ms of round trip reads at 1.4 to 1.8 Mbit/s. BBR estimates the rate of the link instead and does not slow down for isolated losses.

Go
cfg := &kolotransport.Config{CongestionControl: "bbr"}
conn, err := kolotransport.Dial(ctx, addr, cfg)
  • Each end paces its own sending, so set it on the end that sends the data, or on both for a transfer that goes both ways. It applies to every connection the Config makes (Dial, Listener.Accept, Listener.Dial), and a connection keeps its controller when a NAT gives the dialer a new port. Any other value is ErrInvalidConfig.
  • Measured over a simulated link of 20 Mbit/s each way, 60 ms of delay each way and 1 percent random loss each way, with 8 MiB windows: one stream reads at 1.83 Mbit/s with Reno and 18.86 Mbit/s with BBR. "cubic" does not help on that path, since it backs off at every loss like Reno. The paths measured have independent random loss only, not the bursts, reordering or policers of a real network.
  • BBR is only as fast as the receive window: with the default windows a stream moves at most window / round trip, so use it together with larger windows or WindowGrowth.
  • Keep calls and live media on "reno". BBR keeps a standing queue in the bottleneck (about 100 ms, 150 ms with a bufferbloated router in the measurements) and takes bandwidth from Reno flows that share it: it is for a transfer the user asked for and wants fast, not for background traffic.
  • A BBR connection keeps an 88-byte sample per packet number between the oldest packet it waits for and the newest it sent. The span is capped at 40000 packet numbers, so a peer that never acknowledges cannot make it grow: the oldest samples are dropped, which costs only bandwidth samples. The cap bounds it at 5.5 MiB per BBR connection. "reno" and "cubic" have no such structure.

Connection statistics

Conn.Stats() returns a ConnStats snapshot of what one connection measured. It is cheap (a few atomic reads), never blocks on the network, is safe from any goroutine, and returns the zero value once the connection has ended, so read it before you close the connection if you want the final numbers. Each field is read on its own, so two fields can disagree for one sample.

Go
s := conn.Stats()
fmt.Printf("rtt %v (min %v) window %d in flight %d lost %d of %d packets via %s\n",
	s.SmoothedRTT, s.MinRTT, s.CongestionWindow, s.BytesInFlight, s.PacketsLost, s.PacketsSent, s.CongestionController)
  • SmoothedRTT, MinRTT, LatestRTT and RTTVariation: the round trip of the current path and its jitter.
  • BytesSent, BytesReceived, PacketsSent and PacketsReceived: QUIC packets over the life of the connection, retransmissions and duplicates included.
  • PacketsLost and BytesLost: packets this end sent that were not acknowledged in time. An ECN congestion signal counts as one lost packet.
  • CongestionWindow and BytesInFlight: what the controller allows in flight and what is in flight now.
  • CongestionController: "reno", "cubic" or "bbr". BandwidthEstimate: the bottleneck bandwidth BBR measured, in bytes per second, 0 for the other two.
  • MaxDatagramSize: the largest UDP payload this end sends on the current path, at most 1452 bytes. PathChanges: how often the connection moved to a new path; the round-trip fields, the window and the estimate start over on a new path.

BytesInFlight near CongestionWindow means the controller limits the transfer. A SmoothedRTT far above MinRTT means a queue on the path. BytesInFlight far below CongestionWindow while the transfer is slow means the network is not the limit: the application does not write fast enough, or the peer's receive windows are full.

Security

  • AcceptFunc filters by source IP, not identity. Filter by IP, never by port: a Retry token proves only the IP, and can be replayed from that IP for HandshakeTimeout. A peer can move its connection to another address after the handshake. If AcceptFunc returns false or panics, the attempt is refused. It runs on the goroutine that handles every new attempt, so keep it fast and non-blocking.
  • The sender of ReadPacket may be spoofed. Do not dial it, and do not answer with more bytes than it sent, until it has proved it receives at that address. Non-QUIC datagrams are not filtered by AcceptFunc and carry no protection.
  • Dial, Listener.Dial and WritePacket refuse port 0, IPv6, unspecified, multicast, 0.0.0.0/8 and 240.0.0.0/4 destinations. Loopback, private, link-local and subnet broadcast addresses are allowed, so filter any address a peer supplies before you dial or write to it.
  • The sockets Listen and Dial create have SO_BROADCAST cleared. ListenConn leaves it as is.
  • Errors never include text from the peer, only a numeric code.

Known limits

  • Experimental. The API may change before v1.0.0.
  • QUIC datagrams are not exposed in v0.1.x.
  • kolo-transport carries its own copy of its QUIC implementation, a pinned release plus one patch: the hook for a congestion controller and the counters Conn.Stats reads. A script and a test check that the copy is the pinned release plus that patch and nothing else. A fix to the QUIC implementation reaches kolo-transport only when the copy is updated.
  • The package Dial has no cap on the number of connections, so a node that dials N peers can hold N × MaxConnReceiveWindow of unread data.
  • Not bounded: the number of distinct addresses (peers on more than MaxConns / MaxConnsPerIP addresses can still fill MaxConns), the CPU and bandwidth spent per packet, and the OS UDP receive buffer.
  • On ListenConn with a conn that is not a *net.UDPConn: no ECN, no GSO, no batched reads, and path MTU discovery is off, so packets stay at 1280 bytes.

Errors

ErrorMessage
ErrClosedkolotransport: closed
ErrHandshakekolotransport: handshake failed
ErrInvalidArgumentkolotransport: invalid argument
ErrInvalidConfigkolotransport: invalid config
ErrMaxDialedkolotransport: too many dialed connections
ErrPeerClosedkolotransport: peer closed the connection
ErrRefusedkolotransport: connection refused
ErrStreamCanceledkolotransport: stream canceled
ErrStreamResetkolotransport: stream reset by peer
ErrTimeoutkolotransport: timeout
ErrTransportkolotransport: transport error

Test errors with errors.Is against the sentinels. Errors with a code are an *Error; use errors.As to read Code, and its Unwrap returns the sentinel. context.Cause of Conn.Context gives the same errors.

References