kolo docsbeta

kolo-codec

Canonical binary encoding for Kolo, where every value has exactly one valid encoding.

On this page

Peers sign and hash the bytes they exchange, so a value must always encode to the same bytes. kolo-codec gives every value exactly one valid encoding: unsigned integers are minimal unsigned LEB128 varints as defined by the DWARF Version 5 standard, signed integers are mapped to unsigned ones by alternating sign (0, -1, 1, -2 become 0, 1, 2, 3), and strings are UTF-8 (RFC 3629). It rejects anything else with an error, and reads untrusted input without panicking. kolo-log and the libraries above it use it for everything they sign, hash, send or store. It has no schema and no JSON form.

Install

go get github.com/ciranda-dev/kolo-codec

No dependencies outside the standard library.

Quick start

Write a number and a string, read them back, and see a second encoding of a value rejected.

main.go
package main

import (
	"errors"
	"fmt"
	"log"

	kolocodec "github.com/ciranda-dev/kolo-codec"
)

func main() {
	var e kolocodec.Encoder
	e.PutUvarint(300)
	e.PutString("hello")
	data := e.Bytes()
	fmt.Printf("encoded: %x\n", data)

	d := kolocodec.NewDecoder(data)
	seq, err := d.Uvarint()
	if err != nil {
		log.Fatal(err)
	}
	body, err := d.String(1024)
	if err != nil {
		log.Fatal(err)
	}
	if err := d.Done(); err != nil {
		log.Fatal(err)
	}
	fmt.Println(seq, body)

	// 80 00 is a second, longer encoding of 0.
	_, err = kolocodec.NewDecoder([]byte{0x80, 0x00}).Uvarint()
	fmt.Println("non-canonical:", errors.Is(err, kolocodec.ErrNonCanonical))
}

It prints encoded: ac020568656c6c6f, then 300 hello and non-canonical: true. Read fields in the order they were written, and call Done at the end so trailing bytes are an error.

API reference

The API reference is generated from the Go source and is in English.

Package kolocodec · github.com/ciranda-dev/kolo-codec · v0.1.0

Package kolocodec encodes values into canonical bytes for Kolo and decodes them without panicking on hostile input. Canonical is byte-level: strings must be valid UTF-8 but are not normalized (no NFC).

This package is experimental and its API may change before v1.0.0.

Variables

var (
	// ErrShortBuffer is returned when the input ends before a value does.
	ErrShortBuffer = errors.New("kolocodec: short buffer")
	// ErrNonCanonical is returned when the input is not the one valid encoding of its value.
	ErrNonCanonical = errors.New("kolocodec: non-canonical encoding")
	// ErrTooLarge is returned when a length or value is over the caller's max or a uvarint overflows 64 bits.
	ErrTooLarge = errors.New("kolocodec: value too large")
	// ErrInvalidUTF8 is returned by [Decoder.String] and [Encoder.PutValidString] for invalid UTF-8.
	ErrInvalidUTF8 = errors.New("kolocodec: invalid UTF-8")
	// ErrTrailingBytes is returned by [Decoder.Done] when input is left.
	ErrTrailingBytes = errors.New("kolocodec: trailing bytes")
)

Types

type Decoder

type Decoder struct {
	// contains filtered or unexported fields
}

Decoder reads canonical encodings from a byte slice; a failed read consumes nothing.

func NewDecoder
func NewDecoder(b []byte) *Decoder

NewDecoder returns a Decoder that reads from b.

func (Decoder) Bool
func (d *Decoder) Bool() (bool, error)

Bool reads one byte and returns ErrNonCanonical if it is not 0 or 1.

func (Decoder) Bytes
func (d *Decoder) Bytes(max int) ([]byte, error)

Bytes reads a length-prefixed byte slice of at most max bytes and returns a copy.

func (Decoder) Done
func (d *Decoder) Done() error

Done returns ErrTrailingBytes if any input is left.

func (Decoder) Fixed
func (d *Decoder) Fixed(n int) ([]byte, error)

Fixed reads exactly n bytes with no length and returns a copy.

func (Decoder) Len
func (d *Decoder) Len() int

Len returns the number of bytes left to read.

func (Decoder) String
func (d *Decoder) String(max int) (string, error)

String reads a length-prefixed UTF-8 string of at most max bytes.

func (Decoder) Uint32
func (d *Decoder) Uint32() (uint32, error)

Uint32 reads 4 bytes, little-endian.

func (Decoder) Uint64
func (d *Decoder) Uint64() (uint64, error)

Uint64 reads 8 bytes, little-endian.

func (Decoder) Uvarint
func (d *Decoder) Uvarint() (uint64, error)

Uvarint reads a minimal unsigned LEB128 varint, rejecting non-minimal encodings.

func (Decoder) UvarintMax
func (d *Decoder) UvarintMax(max uint64) (uint64, error)

UvarintMax reads a uvarint like Decoder.Uvarint and returns ErrTooLarge if it is over max.

func (Decoder) Varint
func (d *Decoder) Varint() (int64, error)

Varint reads a zigzag-encoded minimal varint written by Encoder.PutVarint.

type Encoder

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder appends canonical encodings to a buffer; the zero value is ready to use.

func (Encoder) Bytes
func (e *Encoder) Bytes() []byte

Bytes returns a copy of the encoded output.

func (Encoder) Len
func (e *Encoder) Len() int

Len returns the number of bytes written, without copying them.

func (Encoder) PutBool
func (e *Encoder) PutBool(v bool)

PutBool writes v as one byte, 0 or 1.

func (Encoder) PutBytes
func (e *Encoder) PutBytes(b []byte)

PutBytes writes the length of b as a uvarint, then b.

func (Encoder) PutFixed
func (e *Encoder) PutFixed(b []byte)

PutFixed writes b with no length, for fixed-size values such as keys and hashes.

func (Encoder) PutString
func (e *Encoder) PutString(s string)

PutString writes s like Encoder.PutBytes; use Encoder.PutValidString to reject invalid UTF-8.

func (Encoder) PutUint32
func (e *Encoder) PutUint32(v uint32)

PutUint32 writes v as 4 bytes, little-endian.

func (Encoder) PutUint64
func (e *Encoder) PutUint64(v uint64)

PutUint64 writes v as 8 bytes, little-endian.

func (Encoder) PutUvarint
func (e *Encoder) PutUvarint(v uint64)

PutUvarint writes v as a minimal unsigned LEB128 varint.

func (Encoder) PutValidString
func (e *Encoder) PutValidString(s string) error

PutValidString writes s like Encoder.PutString, or returns ErrInvalidUTF8 and writes nothing.

func (Encoder) PutVarint
func (e *Encoder) PutVarint(v int64)

PutVarint writes v as a zigzag-encoded minimal varint.

func (Encoder) Reset
func (e *Encoder) Reset()

Reset discards the output and keeps the buffer for reuse; earlier results of Encoder.Bytes are unaffected.

Good to know

ValueEncoding
unsigned integerminimal unsigned LEB128 varint
signed integerzigzag, then the unsigned varint
uint32, uint64fixed width, little-endian
boolone byte, 0 or 1
bytes, stringvarint length, then the data
fixed-size value (key, hash)the raw bytes, no length

Limits

  • A varint is at most 10 bytes and 64 bits. Longer input returns ErrTooLarge, and a varint with redundant bytes returns ErrNonCanonical.
  • Bytes(max) and String(max) return ErrTooLarge for a length over max or a negative max, and ErrShortBuffer for a length past the end of the input. Both are checked before anything is allocated.
  • UvarintMax(max) returns ErrTooLarge for a value over max.
  • Fixed(n) returns ErrTooLarge for a negative n.
  • A bool byte other than 0 or 1 returns ErrNonCanonical.

Security

  • Decoding untrusted input returns an error, never a panic.
  • A failed read consumes nothing.
  • Bytes, Fixed and Encoder.Bytes return copies, so changing the input or output later does not change what you hold.
  • Encoder.Len is the size of the output so far, without a copy, and Encoder.Reset empties the encoder and keeps its buffer, so a hot path can reuse it. Results of Encoder.Bytes taken earlier are not affected.
  • Error messages never include values taken from the input.

Known limits

  • Experimental. The API may change before v1.0.0.
  • No Unicode normalization (no NFC).
  • PutString writes invalid UTF-8 as given, and String then rejects it. Use PutValidString to refuse it when writing, and Bytes for arbitrary data.

Errors

ErrorMessage
ErrInvalidUTF8kolocodec: invalid UTF-8
ErrNonCanonicalkolocodec: non-canonical encoding
ErrShortBufferkolocodec: short buffer
ErrTooLargekolocodec: value too large
ErrTrailingByteskolocodec: trailing bytes

Test errors with errors.Is. ErrTooLarge can come wrapped with a short reason, such as a uvarint that overflows 64 bits; the other errors are returned as they are.

References

  • DWARF Version 5, the Debugging Information Format standard.
    • Takes: the unsigned LEB128 encoding, used only in its minimal form.
    • Not implemented: signed LEB128; signed integers are mapped to unsigned ones by alternating sign instead.
  • RFC 3629, UTF-8, a transformation format of ISO 10646 (Internet Standard, November 2003).
    • Takes: the definition of valid UTF-8 that every string is checked against.
    • Not implemented: nothing left out.