kolo docsbeta

kolo-relay

An optional, self-hosted relay that joins two Kolo peers that cannot connect directly.

On this page

kolo-relay lets two peers connect when neither can reach the other, for example both behind symmetric NAT or CGNAT, or behind a firewall that only lets traffic out. A peer reserves a slot on a relay with a reachable address; another peer asks the relay for that peer's key, and the relay joins the two and copies bytes between them, with allocations, lifetimes and quotas modeled on TURN (RFC 8656). The two peers run an ordinary kolo-secure connection end to end, so the relay forwards ciphertext it cannot read. Like the relay policy of WebRTC, the relay is off unless the application turns it on: it never finds relays, never tries a direct path first and never falls back to a relay by itself. See Relay.

All steps

Direct, or through a relay

The relay is never a silent fallback: your app chooses it, by its address and key.

  1. When a direct path exists, the peers connect directly and no relay takes part.
  2. The target dials the relay with kolo-secure and sends Reserve. The relay keeps the reservation for up to an hour.
  3. The initiator dials the relay and sends Connect with the target's key.
  4. The relay tells the target with Incoming and the initiator's key. The target accepts or refuses, and the relay answers OK.
  5. The relay splices the two streams. Inside them the peers run their own kolo-secure connection, end to end, and the target checks it is the announced initiator.
  6. The relay cannot read the packets. It learns both keys, both addresses, when the session starts and ends, and the sizes and timing of the packets. Each session has limits: 1 hour, 1 GiB each way, 60 seconds idle.

Install

go get github.com/ciranda-dev/kolo-relay

Builds without cgo.

Quick start

This program starts a relay on loopback, reserves a slot for one peer and dials it from another peer through the relay.

main.go
package main

import (
	"context"
	"fmt"
	"io"
	"log"
	"time"

	koloid "github.com/ciranda-dev/kolo-id"
	kolorelay "github.com/ciranda-dev/kolo-relay"
)

func main() {
	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
	defer cancel()

	relayID, err := koloid.Generate()
	if err != nil {
		log.Fatal(err)
	}
	targetID, err := koloid.Generate()
	if err != nil {
		log.Fatal(err)
	}
	dialerID, err := koloid.Generate()
	if err != nil {
		log.Fatal(err)
	}

	relay, err := kolorelay.Listen(ctx, "127.0.0.1:0", relayID, nil)
	if err != nil {
		log.Fatal(err)
	}
	defer relay.Close()
	addr, key := relay.Addr().String(), relayID.PublicKey()

	res, err := kolorelay.Reserve(ctx, addr, key, targetID, nil)
	if err != nil {
		log.Fatal(err)
	}
	defer res.Close()
	go func() {
		c, err := res.Accept(ctx)
		if err != nil {
			return
		}
		s, err := c.AcceptStream(ctx)
		if err != nil {
			return
		}
		io.Copy(s, s)
		s.Close()
	}()

	c, err := kolorelay.Dial(ctx, addr, key, dialerID, targetID.PublicKey(), nil)
	if err != nil {
		log.Fatal(err)
	}
	defer c.Close()
	fmt.Printf("connected to %x through relay %x\n", c.RemotePublicKey()[:4], c.RelayPublicKey()[:4])
	s, err := c.OpenStream(ctx)
	if err != nil {
		log.Fatal(err)
	}
	if _, err := s.Write([]byte("hello kolo")); err != nil {
		log.Fatal(err)
	}
	s.Close()
	reply, err := io.ReadAll(s)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("%s\n", reply)
}

It prints the first bytes of the target's key and the relay's key, then hello kolo echoed back by the target. In a real deployment the relay runs on a host with a public address and loads its identity with koloid.Load; peers get its address and key from their own configuration, and a target tells others which relay it reserved on through the application.

API reference

The API reference is generated from the Go source and is in English.

Package kolorelay · github.com/ciranda-dev/kolo-relay · v0.1.0

Package kolorelay lets two Kolo peers that cannot reach each other connect through a relay they chose.

The peers run a kolo-secure connection end to end; its packets travel inside streams of each peer's own connection to the relay, so the relay forwards ciphertext of a session it is not part of. Using a relay is always an explicit choice of the application.

Experimental: the API may change before v1.0.0.

Variables

var (
	// ErrInvalidConfig means a Config or ServerConfig field is out of range, or a kolo-transport Network in it is not "udp4".
	ErrInvalidConfig = errors.New("kolorelay: invalid config")
	// ErrInvalidArgument means a nil or zero identity, an IPv6 address (v4-mapped included),
	// or a target key that is not 32 bytes or is the local or the relay's key.
	ErrInvalidArgument = errors.New("kolorelay: invalid argument")
	// ErrClosed means the Server or Reservation is closed.
	ErrClosed = errors.New("kolorelay: closed")
	// ErrProtocol means a relay message was malformed, truncated, too large or unexpected.
	ErrProtocol = wire.ErrProtocol
	// ErrUnsupportedVersion means the other side speaks another kolo-relay version.
	ErrUnsupportedVersion = wire.ErrUnsupportedVersion
	// ErrUnavailable means the target cannot be reached through the relay: no reservation,
	// MaxSessionsPerTarget sessions, or the relay or the target refused.
	ErrUnavailable = errors.New("kolorelay: target unavailable through the relay")
	// ErrNotAllowed means the relay does not let this key reserve.
	ErrNotAllowed = errors.New("kolorelay: not allowed by the relay")
	// ErrLimit means MaxReservations, MaxSessions or MaxSessionsPerPeer was reached; trying later may work.
	ErrLimit = limits.ErrLimit
	// ErrReservationEnded means the reservation ended: its lifetime passed, the key reserved again,
	// the relay closed, or the connection was lost.
	ErrReservationEnded = errors.New("kolorelay: reservation ended")
)

Types

type Config

type Config struct {
	// Relay configures the connection to the relay. Nil means kolo-secure's defaults with a 5s keep-alive; a non-nil value is used as given.
	Relay *kolosecure.Config
	// Peer configures the end-to-end connection. Nil means the defaults; kolorelay sets the listener-only transport fields.
	Peer *kolosecure.Config
	// ControlTimeout bounds each exchange with the relay and the wait for a relayed peer's handshake.
	// Default 20s, 5s above the relay's default ConnectTimeout; from 1s to 60s.
	ControlTimeout time.Duration
	// MaxPending caps the incoming sessions not yet returned by Accept. Default 16, from 1 to 256.
	MaxPending int
	// SendQueue is how many packets a session queues before it drops the oldest. Default 512, from 8 to 1024.
	SendQueue int
	// AllowFunc says whether an initiator may reach this reservation. Nil allows every key; a panic refuses.
	AllowFunc func(initiator ed25519.PublicKey) bool
}

Config sets the limits of a peer that reserves on a relay or dials through one. A zero field takes its default.

type Conn

type Conn struct {
	// contains filtered or unexported fields
}

Conn is a connection to a peer through a relay; the peer proved RemotePublicKey end to end, and the relay only sees ciphertext.

func Dial
func Dial(ctx context.Context, relayAddr string, relayKey ed25519.PublicKey, id *koloid.Identity, target ed25519.PublicKey, cfg *Config) (*Conn, error)

Dial connects to target through the relay at relayAddr, whose key is relayKey, and returns once target has proved its key end to end. The relay must hold a reservation for target. Nothing is tried directly: the connection always goes through the relay.

func (Conn) AcceptStream
func (c *Conn) AcceptStream(ctx context.Context) (*kolotransport.Stream, error)

AcceptStream waits for the next bidirectional stream opened by the peer.

func (Conn) AcceptUniStream
func (c *Conn) AcceptUniStream(ctx context.Context) (*kolotransport.ReceiveStream, error)

AcceptUniStream waits for the next unidirectional stream opened by the peer.

func (Conn) Close
func (c *Conn) Close() error

Close closes the connection with code 0 and releases its resources.

func (Conn) CloseWithError
func (c *Conn) CloseWithError(code uint64, msg string) error

CloseWithError closes the connection, sending code and msg to the peer, as kolosecure.Conn.CloseWithError.

func (Conn) Context
func (c *Conn) Context() context.Context

Context is canceled when the connection ends, also when the relay ends the session; its context.Cause is a kolo-transport error.

func (Conn) OpenStream
func (c *Conn) OpenStream(ctx context.Context) (*kolotransport.Stream, error)

OpenStream opens a bidirectional stream, as kolosecure.Conn.OpenStream.

func (Conn) OpenUniStream
func (c *Conn) OpenUniStream(ctx context.Context) (*kolotransport.SendStream, error)

OpenUniStream opens a unidirectional stream, as kolosecure.Conn.OpenUniStream.

func (Conn) RelayAddr
func (c *Conn) RelayAddr() net.Addr

RelayAddr returns the UDP address of the relay this connection goes through.

func (Conn) RelayPublicKey
func (c *Conn) RelayPublicKey() ed25519.PublicKey

RelayPublicKey returns a copy of the key of the relay this connection goes through.

func (Conn) RemotePublicKey
func (c *Conn) RemotePublicKey() ed25519.PublicKey

RemotePublicKey returns a copy of the peer's kolo-id public key, proved end to end.

func (Conn) Stats
func (c *Conn) Stats() kolotransport.ConnStats

Stats returns the transport statistics of the end-to-end connection, which crosses the relay: its round trip time is the whole path. It is the zero value after the connection has ended.

type Reservation

type Reservation struct {
	// contains filtered or unexported fields
}

Reservation makes this peer reachable through one relay and receives the connections other peers make to it there.

func Reserve
func Reserve(ctx context.Context, relayAddr string, relayKey ed25519.PublicKey, id *koloid.Identity, cfg *Config) (*Reservation, error)

Reserve asks the relay at relayAddr, whose key is relayKey, to make id reachable there until the relay ends the reservation or Close.

func (Reservation) Accept
func (r *Reservation) Accept(ctx context.Context) (*Conn, error)

Accept waits for the next peer that connected through the relay and proved its key end to end. After the relay ends the reservation it returns an error wrapping ErrReservationEnded; connections already returned keep working.

func (Reservation) Close
func (r *Reservation) Close() error

Close ends the reservation and closes its connection to the relay, with every connection Accept returned. It is safe to call more than once.

type Server

type Server struct {
	// contains filtered or unexported fields
}

Server is a relay: peers reserve on it, and it joins other peers to them, copying bytes it cannot read.

func Listen
func Listen(ctx context.Context, addr string, id *koloid.Identity, cfg *ServerConfig) (*Server, error)

Listen starts a relay on addr, such as "127.0.0.1:0", authenticated as id. Clients must know id's public key.

func (Server) Addr
func (s *Server) Addr() net.Addr

Addr returns the relay's local UDP address.

func (Server) Close
func (s *Server) Close() error

Close ends every session and reservation, closes every connection and waits for them. It is safe to call more than once.

type ServerConfig

type ServerConfig struct {
	// Secure configures the relay's listener. Nil means the relay defaults in the README; a non-nil value is used as given.
	Secure *kolosecure.Config
	// MaxReservations caps the reserved keys. Default 64, from 1 to 100000.
	MaxReservations int
	// MaxSessions caps the sessions being set up or open. Default 16, from 1 to 10000.
	MaxSessions int
	// MaxSessionsPerPeer caps the sessions one key has started. Default 4 or MaxSessions if lower, at most MaxSessions.
	MaxSessionsPerPeer int
	// MaxSessionsPerTarget caps the sessions to one reserved key; beyond it the relay answers Unavailable.
	// Default 8 or MaxSessions if lower, at most MaxSessions.
	MaxSessionsPerTarget int
	// ReservationLifetime ends a reservation; the peer must reserve again. Default 1h, from 1s to 24h.
	ReservationLifetime time.Duration
	// MaxSessionDuration ends a session. Default 1h, from 1s to 24h.
	MaxSessionDuration time.Duration
	// MaxSessionBytes ends a session once this many bytes went one way. Default 1 GiB, from 64 KiB to 1 TiB.
	MaxSessionBytes int64
	// SessionIdleTimeout ends a session with no byte either way. Default 60s, from 1s to 10m.
	// Keep-alive is always on, so it ends a session between live peers only if it is not above their KeepAlivePeriod (5s by default).
	SessionIdleTimeout time.Duration
	// ConnectTimeout bounds how long the target may take to accept a session. Default 15s, from 1s to 60s.
	// Default clients wait 20s for the answer, so above about 15s they need a longer ControlTimeout.
	ConnectTimeout time.Duration
	// ControlTimeout bounds how long a client may take to send its first message. Default 15s, from 1s to 60s.
	ControlTimeout time.Duration
	// RateBytes is the read rate per client connection, in bytes per second. Default 1 MiB, from 16 KiB to 1 GiB.
	RateBytes int64
	// RateBurst is the most a client connection may send at once above RateBytes. Default 256 KiB, from 16 KiB to 64 MiB.
	RateBurst int64
	// FlushInterval is the shortest time between two writes of less than a full buffer in one direction of a session.
	// Default 1ms, from 100µs to 100ms.
	FlushInterval time.Duration
	// AllowReserve says whether a key may reserve. Nil allows every key, which Listen reports with a warning; a panic refuses.
	AllowReserve func(key ed25519.PublicKey) bool
	// AllowConnect says whether from may reach to. Nil allows every pair; a panic refuses.
	// It must be fast: Server.Close waits for calls in progress.
	AllowConnect func(from, to ed25519.PublicKey) bool
	// Logger receives the warning Listen logs when AllowReserve is nil. Nil means slog.Default().
	Logger *slog.Logger
}

ServerConfig sets the limits of a relay. A zero field takes its default.

Good to know

Conn.Stats returns the kolo-transport statistics of the end-to-end connection that crosses the relay: round-trip time of the whole path, losses, the congestion window and the controller. It is the zero value once the connection has ended.

Limits

The defaults suit a trial on a host with about 8 GiB and leave the relay open to every key. Before you expose a relay, set AllowReserve and size MaxConns, the receive windows and MaxSessions for your host; the repository's limits page has the recipe for a 300 MiB host.

  • IPv4 only. An IPv6 address given to Listen, Reserve or Dial, v4-mapped included, is ErrInvalidArgument before anything is sent. A kolo-transport Network other than "udp4" in Config or ServerConfig is ErrInvalidConfig.
  • ServerConfig (a zero field takes the default; out of range is ErrInvalidConfig):
    • MaxReservations: 64, from 1 to 100000. Past it, Reserve gets ErrLimit.
    • MaxSessions (being set up or open): 16, from 1 to 10000. Past it, ErrLimit.
    • MaxSessionsPerPeer (started by one key): 4, or MaxSessions if lower. Past it, ErrLimit.
    • MaxSessionsPerTarget (towards one reserved key): 8, or MaxSessions if lower. Past it, ErrUnavailable.
    • ReservationLifetime: 1h, from 1s to 24h. Then the reservation ends and the peer must reserve again; sessions it already carries go on.
    • MaxSessionDuration: 1h, from 1s to 24h.
    • MaxSessionBytes (each direction): 1 GiB, from 64 KiB to 1 TiB. Exactly that many bytes pass, then the whole session ends.
    • SessionIdleTimeout (no byte either way): 60s, from 1s to 10m. Keep-alive is always on (5s by default), so a session between two live peers does not reach a timeout above their KeepAlivePeriod.
    • ConnectTimeout (target accepts a session): 15s, from 1s to 60s. Above about 15s, clients need a ControlTimeout longer than their default 20s.
    • ControlTimeout (client's first message): 15s, from 1s to 60s.
    • RateBytes (read from each client connection, per second): 1 MiB, from 16 KiB to 1 GiB.
    • RateBurst: 256 KiB, from 16 KiB to 64 MiB.
    • FlushInterval: 1ms, from 100µs to 100ms.
    • Logger: nil means slog.Default(). It receives the one warning of an open relay.
  • With Secure nil, the relay listens with MaxConns 128, MaxConnsPerIP 4, MaxDialed 1, MaxIncomingStreams 2, MaxStreamReceiveWindow 64 KiB and MaxConnReceiveWindow 256 KiB. A non-nil Secure is used as given, without these values.
  • Every session needs its own client connection, so one source address holds at most MaxConnsPerIP sessions.
  • Config (for Reserve and Dial; a zero field takes the default; out of range is ErrInvalidConfig):
    • Relay: nil means kolo-secure defaults with a KeepAlivePeriod of 5s.
    • ControlTimeout (each exchange with the relay): 20s, from 1s to 60s. A relay that does not answer in time gives an error wrapping os.ErrDeadlineExceeded.
    • MaxPending (sessions not yet returned by Accept): 16, from 1 to 256.
    • SendQueue (packets queued per session): 512, from 8 to 1024. When full, the oldest packet is dropped, as on UDP.
  • A zero HandshakeTimeout in Peer is 10s. A target waits at most ControlTimeout plus both handshake timeouts (40s at the defaults) for an announced peer, then frees the slot.
  • Fixed sizes: a control message is at most 64 bytes, a packet frame at most 1452 bytes, and the relay's copy buffer is 64 KiB per direction. Inner packets stay at 1280 bytes, since path MTU discovery is off on the relayed path.
  • Reservation.Close and Conn.Close can take about 2s: the last packet gets at most 1s to go out, then the other side gets at most 1s to end its stream. When the relay ends a session, each side's Conn closes within about 1s.
  • Server.Close gives each initiator up to 1s to close its connection, then ends every reservation.
  • Throughput at the defaults: the 64 KiB stream window limits a session to 64 KiB per round trip in each direction, RateBytes limits each client connection to 1 MiB/s, and the relay's egress is at most 2 × MaxSessions × RateBytes, 32 MiB/s. Only loopback tests exist.
  • Memory: the relay's worst-case live heap at the defaults is about 7.3 GiB (7506.8 MiB). Set GOMEMLIMIT a little above the bound of your configuration, for example GOMEMLIMIT=8500MiB at the defaults. On a small host, lower MaxSessions and the listener limits through Secure, and recompute with the formula in the README. A connection whose end chose CongestionControl: "bbr" adds up to 5.5 MiB of BBR samples, a term the formula omits; the relay's own listener uses Reno unless its Secure says otherwise.

Security

  • The relay is not a TLS endpoint of the peers' connection. It cannot read or change the inner packets, and it cannot pose as either peer. It can always drop, delay or end them.
  • The relay learns both keys of every session, both clients' addresses, which keys hold reservations, when sessions start and end, and the sizes and timing of the inner packets.
  • The relay never dials anyone and only joins two of its own authenticated clients, towards a key that reserved, so it is not an open proxy.
  • Only the holder of a key can reserve it. A newer reservation of the same key replaces the older one.
  • Dial returns only once target has proved its key end to end. A target that receives a peer whose proved key is not the one the relay announced closes that connection with code 1 and never returns it.
  • A client key or a target key that is not 32 bytes ends the connection with code 1.
  • A panic in AllowReserve, AllowConnect or AllowFunc refuses. AllowConnect must be fast: Server.Close waits for calls in progress.
  • ErrUnavailable is one answer for several causes on purpose. A key can still learn its own AllowConnect decision and whether a target is reserved, but nothing about other keys.
  • No error carries anything the peer or the relay sent, except numeric codes.

Known limits

  • Accept does not reserve again after the relay ends a reservation. Call Close after Accept fails, then Reserve again.
  • Conn has no RemoteAddr: the only address on the path is the relay's (RelayAddr).
  • Each packet is encrypted twice and goes through two flow controls (to the relay and end to end), and both limit throughput. Throughput on real networks is not measured.
  • The relay is experimental. The API may change before v1.0.0.

Errors

ErrorMessage
ErrClosedkolorelay: closed
ErrInvalidArgumentkolorelay: invalid argument
ErrInvalidConfigkolorelay: invalid config
ErrLimitkolorelay: relay limit reached
ErrNotAllowedkolorelay: not allowed by the relay
ErrProtocolkolorelay: invalid message
ErrReservationEndedkolorelay: reservation ended
ErrUnavailablekolorelay: target unavailable through the relay
ErrUnsupportedVersionkolorelay: unsupported protocol version

Test errors with errors.Is. The error from Accept after the relay ends a reservation wraps both ErrReservationEnded and the kolo-transport error with the relay's code (2 lifetime over, 3 replaced, 4 relay closing). kolo-secure and kolo-transport errors pass through, for example kolosecure.ErrUnexpectedPeer when the relay does not hold relayKey.

References