Peers sign and hash the bytes they exchange, so a value must always encode to the same bytes. kolo-codec gives every value exactly one valid encoding: unsigned integers are minimal unsigned LEB128 varints as defined by the DWARF Version 5 standard, signed integers are mapped to unsigned ones by alternating sign (0, -1, 1, -2 become 0, 1, 2, 3), and strings are UTF-8 (RFC 3629). It rejects anything else with an error, and reads untrusted input without panicking. kolo-log and the libraries above it use it for everything they sign, hash, send or store. It has no schema and no JSON form.
Install
go get github.com/ciranda-dev/kolo-codecNo dependencies outside the standard library.
Quick start
Write a number and a string, read them back, and see a second encoding of a value rejected.
package main
import (
"errors"
"fmt"
"log"
kolocodec "github.com/ciranda-dev/kolo-codec"
)
func main() {
var e kolocodec.Encoder
e.PutUvarint(300)
e.PutString("hello")
data := e.Bytes()
fmt.Printf("encoded: %x\n", data)
d := kolocodec.NewDecoder(data)
seq, err := d.Uvarint()
if err != nil {
log.Fatal(err)
}
body, err := d.String(1024)
if err != nil {
log.Fatal(err)
}
if err := d.Done(); err != nil {
log.Fatal(err)
}
fmt.Println(seq, body)
// 80 00 is a second, longer encoding of 0.
_, err = kolocodec.NewDecoder([]byte{0x80, 0x00}).Uvarint()
fmt.Println("non-canonical:", errors.Is(err, kolocodec.ErrNonCanonical))
}It prints encoded: ac020568656c6c6f, then 300 hello and
non-canonical: true. Read fields in the order they were written, and call
Done at the end so trailing bytes are an error.
API reference
The API reference is generated from the Go source and is in English.
Package kolocodec encodes values into canonical bytes for Kolo and decodes them without panicking on hostile input. Canonical is byte-level: strings must be valid UTF-8 but are not normalized (no NFC).
This package is experimental and its API may change before v1.0.0.
Variables
var (
// ErrShortBuffer is returned when the input ends before a value does.
ErrShortBuffer = errors.New("kolocodec: short buffer")
// ErrNonCanonical is returned when the input is not the one valid encoding of its value.
ErrNonCanonical = errors.New("kolocodec: non-canonical encoding")
// ErrTooLarge is returned when a length or value is over the caller's max or a uvarint overflows 64 bits.
ErrTooLarge = errors.New("kolocodec: value too large")
// ErrInvalidUTF8 is returned by [Decoder.String] and [Encoder.PutValidString] for invalid UTF-8.
ErrInvalidUTF8 = errors.New("kolocodec: invalid UTF-8")
// ErrTrailingBytes is returned by [Decoder.Done] when input is left.
ErrTrailingBytes = errors.New("kolocodec: trailing bytes")
)Types
type Decoder
type Decoder struct {
// contains filtered or unexported fields
}Decoder reads canonical encodings from a byte slice; a failed read consumes nothing.
func NewDecoder
func NewDecoder(b []byte) *DecoderNewDecoder returns a Decoder that reads from b.
func (Decoder) Bool
func (d *Decoder) Bool() (bool, error)Bool reads one byte and returns ErrNonCanonical if it is not 0 or 1.
func (Decoder) Bytes
func (d *Decoder) Bytes(max int) ([]byte, error)Bytes reads a length-prefixed byte slice of at most max bytes and returns a copy.
func (Decoder) Done
func (d *Decoder) Done() errorDone returns ErrTrailingBytes if any input is left.
func (Decoder) Fixed
func (d *Decoder) Fixed(n int) ([]byte, error)Fixed reads exactly n bytes with no length and returns a copy.
func (Decoder) Len
func (d *Decoder) Len() intLen returns the number of bytes left to read.
func (Decoder) String
func (d *Decoder) String(max int) (string, error)String reads a length-prefixed UTF-8 string of at most max bytes.
func (Decoder) Uint32
func (d *Decoder) Uint32() (uint32, error)Uint32 reads 4 bytes, little-endian.
func (Decoder) Uint64
func (d *Decoder) Uint64() (uint64, error)Uint64 reads 8 bytes, little-endian.
func (Decoder) Uvarint
func (d *Decoder) Uvarint() (uint64, error)Uvarint reads a minimal unsigned LEB128 varint, rejecting non-minimal encodings.
func (Decoder) UvarintMax
func (d *Decoder) UvarintMax(max uint64) (uint64, error)UvarintMax reads a uvarint like Decoder.Uvarint and returns ErrTooLarge if it is over max.
func (Decoder) Varint
func (d *Decoder) Varint() (int64, error)Varint reads a zigzag-encoded minimal varint written by Encoder.PutVarint.
type Encoder
type Encoder struct {
// contains filtered or unexported fields
}Encoder appends canonical encodings to a buffer; the zero value is ready to use.
func (Encoder) Bytes
func (e *Encoder) Bytes() []byteBytes returns a copy of the encoded output.
func (Encoder) Len
func (e *Encoder) Len() intLen returns the number of bytes written, without copying them.
func (Encoder) PutBool
func (e *Encoder) PutBool(v bool)PutBool writes v as one byte, 0 or 1.
func (Encoder) PutBytes
func (e *Encoder) PutBytes(b []byte)PutBytes writes the length of b as a uvarint, then b.
func (Encoder) PutFixed
func (e *Encoder) PutFixed(b []byte)PutFixed writes b with no length, for fixed-size values such as keys and hashes.
func (Encoder) PutString
func (e *Encoder) PutString(s string)PutString writes s like Encoder.PutBytes; use Encoder.PutValidString to reject invalid UTF-8.
func (Encoder) PutUint32
func (e *Encoder) PutUint32(v uint32)PutUint32 writes v as 4 bytes, little-endian.
func (Encoder) PutUint64
func (e *Encoder) PutUint64(v uint64)PutUint64 writes v as 8 bytes, little-endian.
func (Encoder) PutUvarint
func (e *Encoder) PutUvarint(v uint64)PutUvarint writes v as a minimal unsigned LEB128 varint.
func (Encoder) PutValidString
func (e *Encoder) PutValidString(s string) errorPutValidString writes s like Encoder.PutString, or returns ErrInvalidUTF8 and writes nothing.
func (Encoder) PutVarint
func (e *Encoder) PutVarint(v int64)PutVarint writes v as a zigzag-encoded minimal varint.
func (Encoder) Reset
func (e *Encoder) Reset()Reset discards the output and keeps the buffer for reuse; earlier results of Encoder.Bytes are unaffected.
Good to know
| Value | Encoding |
|---|---|
| unsigned integer | minimal unsigned LEB128 varint |
| signed integer | zigzag, then the unsigned varint |
uint32, uint64 | fixed width, little-endian |
| bool | one byte, 0 or 1 |
| bytes, string | varint length, then the data |
| fixed-size value (key, hash) | the raw bytes, no length |
Limits
- A varint is at most 10 bytes and 64 bits. Longer input returns
ErrTooLarge, and a varint with redundant bytes returnsErrNonCanonical. Bytes(max)andString(max)returnErrTooLargefor a length overmaxor a negativemax, andErrShortBufferfor a length past the end of the input. Both are checked before anything is allocated.UvarintMax(max)returnsErrTooLargefor a value overmax.Fixed(n)returnsErrTooLargefor a negativen.- A bool byte other than 0 or 1 returns
ErrNonCanonical.
Security
- Decoding untrusted input returns an error, never a panic.
- A failed read consumes nothing.
Bytes,FixedandEncoder.Bytesreturn copies, so changing the input or output later does not change what you hold.Encoder.Lenis the size of the output so far, without a copy, andEncoder.Resetempties the encoder and keeps its buffer, so a hot path can reuse it. Results ofEncoder.Bytestaken earlier are not affected.- Error messages never include values taken from the input.
Known limits
- Experimental. The API may change before v1.0.0.
- No Unicode normalization (no NFC).
PutStringwrites invalid UTF-8 as given, andStringthen rejects it. UsePutValidStringto refuse it when writing, andBytesfor arbitrary data.
Errors
| Error | Message |
|---|---|
ErrInvalidUTF8 | kolocodec: invalid UTF-8 |
ErrNonCanonical | kolocodec: non-canonical encoding |
ErrShortBuffer | kolocodec: short buffer |
ErrTooLarge | kolocodec: value too large |
ErrTrailingBytes | kolocodec: trailing bytes |
Test errors with errors.Is. ErrTooLarge can come wrapped with a short
reason, such as a uvarint that overflows 64 bits; the other errors are
returned as they are.
References
- DWARF Version 5, the Debugging Information Format standard.
- Takes: the unsigned LEB128 encoding, used only in its minimal form.
- Not implemented: signed LEB128; signed integers are mapped to unsigned ones by alternating sign instead.
- RFC 3629, UTF-8, a transformation format of ISO 10646 (Internet Standard, November 2003).
- Takes: the definition of valid UTF-8 that every string is checked against.
- Not implemented: nothing left out.