kolo docsbeta

kolo-video

Camera capture, I420 frames, VP8 and VP9 with libvpx, and optional H.264 from the operating system's own codec. No cgo, no C compiler and nothing to install.

On this page

kolo-video lists and opens cameras, delivers I420 frames (with the BT.601 matrix), converts other pixel formats to I420, draws frames into RGBA images, and encodes and decodes VP8 (RFC 6386) and VP9 for real-time use. H.264 is optional and comes from the operating system's own codec. It does not send anything over the network and does not sync audio with video. The Go packages are kolovideo, vpx and h264.

All steps

From a Frame to packets and back

VP8 and VP9 run everywhere. H.264 uses the codec of the operating system, so it is optional.

  1. A Frame is one I420 picture: 8-bit planes, BT.601, limited range. It comes from a camera, from a converter such as FromNV12 or from your own code.
  2. vpx encodes it as VP8 or VP9 with libvpx 1.16.0. The native engine is a prebuilt library loaded without cgo; the portable engine is the same code translated to Go. The package picks one and tells you which.
  3. h264 encodes it as H.264 with the codec the operating system ships: VideoToolbox on macOS, Media Foundation on Windows. It is optional: Capabilities says whether it is available, and on Linux it is not.
  4. Each encoder returns one packet for a frame: a VP8 or VP9 frame, or an Annex B access unit with its SPS and PPS before every keyframe. A sender can send VP8 always and H.264 when it can. Your application sends the packets.
  5. The receiver decodes the stream it picked. It takes H.264 only if its own system can decode it, and VP8 otherwise. Each decoder checks a packet before it decodes it.
  6. The decoder returns a Frame, or an error, and then waits for a keyframe.

Install

go get github.com/ciranda-dev/kolo-video

To pin this release, run go get github.com/ciranda-dev/kolo-video@v0.1.0. It needs Go 1.26 or newer. There is no cgo, no C compiler, no libvpx, no pkg-config and no environment variable to set, so it builds with CGO_ENABLED=0. The VP8 and VP9 codec is libvpx 1.16.0 in two forms that carry the same code: a prebuilt library embedded in the module and loaded without cgo (full speed), and the same code translated to Go (works everywhere, real time up to 640x480). The package picks one at run time and tells you which. The first build of the vpx package compiles generated Go and peaks at about 570 MB of memory on a Mac; later builds use the build cache.

Quick start

This program lists the cameras (it never opens one), then encodes 30 generated frames with VP8 and decodes them again. Run it with go run . <dir>, where <dir> is where the native engine keeps its library; an application normally leaves CacheDir empty, which uses the user cache directory.

main.go
package main

import (
	"errors"
	"fmt"
	"log"
	"os"
	"time"

	kolovideo "github.com/ciranda-dev/kolo-video"
	"github.com/ciranda-dev/kolo-video/vpx"
)

func main() {
	if len(os.Args) != 2 {
		log.Fatal("usage: quickstart <cache directory>")
	}
	cams, err := kolovideo.Devices()
	if err != nil && !errors.Is(err, kolovideo.ErrNoDevice) && !errors.Is(err, kolovideo.ErrUnsupported) {
		log.Fatal(err)
	}
	for _, c := range cams {
		fmt.Printf("camera %q default=%v\n", c.Name, c.Default)
	}

	rt, err := vpx.Load(vpx.LoadConfig{CacheDir: os.Args[1]})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("libvpx %s, %v engine\n", rt.Version, rt.Engine)

	enc, err := vpx.NewEncoder(vpx.Config{Codec: vpx.VP8, Width: 320, Height: 240})
	if err != nil {
		log.Fatal(err)
	}
	defer enc.Close()
	dec, err := vpx.NewDecoder(vpx.DecoderConfig{Codec: vpx.VP8})
	if err != nil {
		log.Fatal(err)
	}
	defer dec.Close()

	var bytes, frames int
	for i := range 30 {
		f, err := kolovideo.NewFrame(320, 240)
		if err != nil {
			log.Fatal(err)
		}
		for j := range f.Y {
			f.Y[j] = byte(16 + (j/320+j%320+4*i)%220)
		}
		f.Timestamp = time.Duration(i) * time.Second / 30

		pkts, err := enc.Encode(f, false)
		if err != nil {
			log.Fatal(err)
		}
		for _, p := range pkts {
			out, err := dec.Decode(p)
			if err != nil {
				log.Fatal(err)
			}
			if out != nil {
				bytes += len(p.Data)
				frames++
			}
		}
	}
	fmt.Printf("%d frames encoded and decoded, %d bytes\n", frames, bytes)
}

The engine line depends on the platform (native on macOS and Windows, portable on Linux by default). The last line does not:

Text
30 frames encoded and decoded, 13596 bytes

H.264 has the same shape, with one extra step: ask the system first. This program encodes one frame with H.264, reads the codec string from the keyframe and decodes it again. Where the system has no H.264 codec it prints why, and you publish VP8 only.

h264.go
package main

import (
	"errors"
	"fmt"
	"log"

	kolovideo "github.com/ciranda-dev/kolo-video"
	"github.com/ciranda-dev/kolo-video/h264"
)

func main() {
	enc, err := h264.NewEncoder(h264.Config{Width: 640, Height: 360})
	if errors.Is(err, h264.ErrUnavailable) {
		fmt.Println("H.264 is not available:", err) // publish VP8 only
		return
	}
	if err != nil {
		log.Fatal(err)
	}
	defer enc.Close()
	dec, err := h264.NewDecoder(h264.DecoderConfig{})
	if err != nil {
		log.Fatal(err)
	}
	defer dec.Close()

	frame, err := kolovideo.NewFrame(640, 360)
	if err != nil {
		log.Fatal(err)
	}
	pkts, err := enc.Encode(frame, false) // the first frame is a keyframe
	if err != nil {
		log.Fatal(err)
	}
	sps, err := h264.ParseKeyframe(pkts[0].Data)
	if err != nil {
		log.Fatal(err)
	}
	out, err := dec.Decode(pkts[0])
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("%s, hardware %v, %d bytes, decoded %dx%d\n", sps.Codec(), enc.Hardware(), len(pkts[0].Data), out.Width, out.Height)
}

On a Mac with VideoToolbox it printed:

Text
avc3.42c01e, hardware true, 739 bytes, decoded 640x360

In a Linux container, where there is no engine, it printed:

Text
H.264 is not available: kolovideo/h264: no H.264 engine on linux/arm64 in this release; V4L2 stateful H.264 devices: none

To capture, open a camera and read frames. OpenCapture never asks for permission: on macOS call RequestAccess first. The repository has examples/snapshot, which saves camera frames as PNG files (-list lists the cameras), examples/vp8, which encodes and decodes generated frames and prints the engine, the size and the quality, and examples/h264, which does the same with H.264 and falls back to VP8 where the system has no codec.

API reference

The API reference is generated from the Go source and is in English.

Package kolovideo · github.com/ciranda-dev/kolo-video · v0.1.0

Package kolovideo captures camera video and converts, scales and renders raw I420 frames for Kolo. Subpackage vpx encodes and decodes VP8 and VP9; subpackage h264 encodes and decodes H.264 with the operating system's codec where there is one. Nothing needs to be installed and no package uses cgo.

Experimental: the API may change before v1.0.0.

Constants

const (
	// DefaultWidth, DefaultHeight and DefaultFrameRate are the common
	// browser capture default, 640x480 at 30 fps. vpx.DefaultFrameRate and
	// h264.DefaultFrameRate are the encoders' rate for frames without a
	// Duration, the same value.
	DefaultWidth     = 640
	DefaultHeight    = 480
	DefaultFrameRate = 30
	// MinFrameRate and MaxFrameRate bound a wanted frame rate: one frame per
	// hour, so that a frame duration in 90 kHz ticks fits the 32-bit duration of
	// libvpx, and 1000 frames per second, the finest rate a 90 kHz clock gives
	// 90 ticks to.
	MinFrameRate = 1.0 / 3600
	MaxFrameRate = 1000
	// DefaultBuffer frames are held for a slow reader: one late read is
	// absorbed without adding latency.
	DefaultBuffer = 2
	// DefaultStallTimeout is the wait for a frame before the device counts
	// as lost, the timeout of the V4L2 reference capture loop.
	DefaultStallTimeout = 2 * time.Second
	// DefaultMaxPixels bounds the capture size: 8,912,896 pixels, the VP9
	// level 5 picture size (4096x2176), which holds 4K UHD. vpx.DefaultMaxPixels
	// bounds decoded VP8 and VP9 frames with the same value; h264.DefaultMaxPixels
	// bounds the coded size of H.264 pictures and is a different value.
	DefaultMaxPixels = 8912896
)

Capture defaults.

Variables

var (
	// ErrInvalidConfig means a configuration field is out of range.
	ErrInvalidConfig = errors.New("kolovideo: invalid config")
	// ErrInvalidFrame means a frame or raw image has inconsistent sizes.
	ErrInvalidFrame = errors.New("kolovideo: invalid frame")
	// ErrNoDevice means no camera matches the request.
	ErrNoDevice = errors.New("kolovideo: no such device")
	// ErrUnsupportedFormat means the device or stream offers no format this
	// package can convert.
	ErrUnsupportedFormat = errors.New("kolovideo: unsupported format")
	// ErrPermission means the system has not granted camera access.
	ErrPermission = errors.New("kolovideo: camera access not granted")
	// ErrDeviceLost means the camera was disconnected, failed, or sent no
	// frame within the stall timeout.
	ErrDeviceLost = errors.New("kolovideo: device lost")
	// ErrClosed means the capture, encoder or decoder was closed.
	ErrClosed = errors.New("kolovideo: closed")
	// ErrUnsupported means this platform has no backend for the feature:
	// camera capture on a system other than macOS, Linux, and Windows on
	// amd64 or arm64, or a system whose camera framework is missing.
	ErrUnsupported = errors.New("kolovideo: not supported on this platform")
)

Functions

func RequestAccess

func RequestAccess(ctx context.Context) error

RequestAccess asks the user for camera access where the system requires it (macOS) and returns ErrPermission if it is refused. It may show a system prompt. On Linux and Windows it returns nil at once; where capture is not supported it returns ErrUnsupported.

Types

type Capture

type Capture struct {
	// contains filtered or unexported fields
}

Capture is an open camera. Its methods are safe for concurrent use.

func OpenCapture
func OpenCapture(cfg CaptureConfig) (*Capture, error)

OpenCapture opens a camera and starts capturing.

func (Capture) Close
func (c *Capture) Close() error

Close stops the camera. Blocked and later Read calls return ErrClosed. A zero Capture, which holds no camera, can be closed and read too. Close is idempotent; only the first call can return an error.

func (Capture) Device
func (c *Capture) Device() Device

Device returns the camera in use.

func (Capture) Mode
func (c *Capture) Mode() Mode

Mode returns the mode in use.

func (Capture) Read
func (c *Capture) Read(ctx context.Context) (*Frame, error)

Read returns the oldest buffered frame, waiting for one if needed. After the camera is lost, buffered frames are returned first, then an error wrapping ErrDeviceLost or ErrPermission. After Close it returns ErrClosed.

func (Capture) Stats
func (c *Capture) Stats() Stats

Stats returns the frame counters.

type CaptureConfig

type CaptureConfig struct {
	// DeviceID is a Device.ID; empty means the default camera.
	DeviceID string
	// Width and Height are the wanted size, DefaultWidth by DefaultHeight
	// if both are zero. The nearest supported mode is used.
	Width, Height int
	// FrameRate is the wanted rate, DefaultFrameRate if zero. It is capped
	// at the chosen mode's highest rate.
	FrameRate float64
	// Buffer is the number of frames held while the reader is slow; when
	// it is full the oldest frame is dropped. Default DefaultBuffer.
	Buffer int
	// StallTimeout is how long Read waits for a frame before it returns
	// ErrDeviceLost. Default DefaultStallTimeout; negative disables it.
	StallTimeout time.Duration
	// MaxPixels is the largest mode accepted, so a faulty camera cannot
	// make every frame a huge allocation. Default DefaultMaxPixels.
	MaxPixels int
}

CaptureConfig selects a camera and mode. Zero fields take defaults.

type Device

type Device struct {
	// ID identifies the camera to OpenCapture. It is stable while the
	// camera stays connected.
	ID   string
	Name string
	// Default marks the system's preferred camera.
	Default bool
	// Modes lists supported modes when the platform reports them without
	// starting the camera; it is empty on Windows. A Linux node held exclusively
	// by another program is listed with no modes.
	Modes []Mode
}

Device is a camera.

func Devices
func Devices() ([]Device, error)

Devices lists the cameras connected now.

type Frame

type Frame struct {
	Width, Height int
	Y, U, V       []byte
	// YStride and UVStride are the bytes between the starts of two rows.
	YStride, UVStride int
	// Timestamp is the capture time on the system monotonic clock.
	Timestamp time.Duration
	// Duration is the nominal display time of the frame.
	Duration time.Duration
}

Frame is a raw I420 picture: planar YUV 4:2:0, 8 bits per sample, BT.601 limited range. The U and V planes are (Width+1)/2 by (Height+1)/2.

func FromBGRA
func FromBGRA(width, height int, src []byte, stride int) (*Frame, error)

FromBGRA converts 32-bit B, G, R, A pixels to a new I420 frame. Alpha is ignored.

func FromNV12
func FromNV12(width, height int, y []byte, yStride int, uv []byte, uvStride int) (*Frame, error)

FromNV12 converts an NV12 picture (a Y plane and an interleaved UV plane) to a new I420 frame.

func FromRGBA
func FromRGBA(width, height int, src []byte, stride int) (*Frame, error)

FromRGBA converts 32-bit R, G, B, A pixels, such as image.RGBA.Pix, to a new I420 frame. Alpha is ignored.

func FromUYVY
func FromUYVY(width, height int, src []byte, stride int) (*Frame, error)

FromUYVY converts a packed 4:2:2 picture in U Y0 V Y1 order to a new I420 frame. The width must be even.

func FromYUY2
func FromYUY2(width, height int, src []byte, stride int) (*Frame, error)

FromYUY2 converts a packed 4:2:2 picture in Y0 U Y1 V order (also called YUYV) to a new I420 frame. The width must be even.

func NewFrame
func NewFrame(width, height int) (*Frame, error)

NewFrame allocates a black frame with tightly packed planes.

func (Frame) RGBA
func (f *Frame) RGBA() (*image.RGBA, error)

RGBA converts the frame to a new image of the same size.

func (Frame) Validate
func (f *Frame) Validate() error

Validate reports whether the sizes, strides and plane lengths agree.

func (Frame) YCbCr
func (f *Frame) YCbCr() *image.YCbCr

YCbCr returns an image that shares the frame's planes, or nil if the frame is invalid. The image/color package treats the samples as full range, so use RGBA or Renderer for exact colors.

type Mode

type Mode struct {
	Width, Height int
	// FrameRate is the highest rate in Device.Modes, 0 if unknown. In
	// Capture.Mode it is the rate the camera was set to, or the requested
	// rate if the camera does not report one.
	FrameRate float64
	// Format is the camera's native format; captured frames are always
	// I420.
	Format PixelFormat
}

Mode is one capture size and frame rate a camera supports.

type PixelFormat

type PixelFormat int

PixelFormat is a camera's native pixel layout.

const (
	FormatUnknown PixelFormat = iota
	FormatI420
	FormatNV12
	FormatYUY2
	FormatUYVY
	FormatBGRA
	FormatMJPEG
)

Pixel formats a camera may deliver.

func (PixelFormat) String
func (p PixelFormat) String() string

String returns the format name.

type Renderer

type Renderer struct {
	Scale ScaleMode
	// contains filtered or unexported fields
}

Renderer draws frames into RGBA images of any size. It keeps a scratch image between calls, so reuse one Renderer per video stream. A Renderer is not safe for concurrent use.

func (Renderer) Render
func (r *Renderer) Render(dst *image.RGBA, f *Frame) error

Render draws f into dst.Bounds(), keeping the frame's aspect ratio. Scaling uses approximate bilinear interpolation.

type ScaleMode

type ScaleMode int

ScaleMode says how Renderer maps a frame onto a destination of another shape.

const (
	// Fit shows the whole frame and fills the bars left over with black.
	Fit ScaleMode = iota
	// Fill covers the whole destination and crops the frame's overflow.
	Fill
)

type Stats

type Stats struct {
	// Frames is the number of frames received from the camera.
	Frames uint64
	// Dropped is the number of frames discarded because Buffer was full.
	Dropped uint64
	// Invalid is the number of camera frames that could not be converted.
	Invalid uint64
}

Stats counts frames since OpenCapture.

Package vpx

github.com/ciranda-dev/kolo-video/vpx

Package vpx encodes and decodes VP8 and VP9 video with libvpx 1.16.0, set up for real-time use. It needs no installed library, compiler or cgo.

Two engines run the same libvpx. The native engine is a prebuilt shared library embedded in the package and loaded without cgo; it is the default on macOS and Windows, and on Linux only with the build tag kolovideo_native, so that binaries stay static and start on any distribution. The portable engine is the same code translated to Go; it works everywhere, runs single-threaded and encodes in real time up to 640x480. When the native engine cannot load, the portable engine is used and LoadError says why. Streams from the two engines are byte-identical only when the encoder runs one thread.

Experimental: the API may change before v1.0.0.

Constants

const (
	// DefaultFrameRate is the rate assumed for frames without a Duration:
	// kolovideo.DefaultFrameRate.
	DefaultFrameRate = kolovideo.DefaultFrameRate
	// DefaultBitsPerPixel sets the default bitrate: 0.07 bits per pixel per
	// frame, about 4.4 Mbit/s for 1080p at 30 fps.
	DefaultBitsPerPixel = 0.07
	// DefaultKeyframeInterval starts a new group of pictures every 2 s.
	DefaultKeyframeInterval = 2 * time.Second
)

Encoder defaults.

const DefaultMaxPacketBytes = 16 << 20

DefaultMaxPacketBytes bounds a packet: 16 MiB, the default MaxBytes of the kolo-media playout. A VP8 or VP9 frame at DefaultMaxPixels is far smaller.

const DefaultMaxPixels = kolovideo.DefaultMaxPixels

DefaultMaxPixels bounds decoded frames: 8,912,896 pixels, the VP9 level 5 picture size (4096x2176), as kolovideo.DefaultMaxPixels for capture.

Variables

var (
	// ErrCorrupt means a packet is truncated or not a valid frame.
	ErrCorrupt = errors.New("kolovideo/vpx: corrupt packet")
	// ErrTooLarge means a packet is larger than MaxPacketBytes or declares a
	// frame larger than MaxPixels.
	ErrTooLarge = errors.New("kolovideo/vpx: packet or frame too large")
	// ErrNeedKeyframe means the decoder drops packets until a keyframe,
	// at the start and after a decoding error.
	ErrNeedKeyframe = errors.New("kolovideo/vpx: waiting for a keyframe")
	// ErrCodec means libvpx failed for a reason other than bad input.
	ErrCodec = errors.New("kolovideo/vpx: codec error")
)
var (
	// ErrNativeUnsupported means this platform or build has no native library:
	// Linux without the kolovideo_native tag, or another platform.
	ErrNativeUnsupported = native.ErrUnsupported
	// ErrNativeIntegrity means the embedded library does not match its checksum.
	ErrNativeIntegrity = native.ErrIntegrity
	// ErrNativeABI means the library does not offer the interface this package
	// expects.
	ErrNativeABI = native.ErrABI
	// ErrNativeSignature means the code signature of a library file on macOS
	// is not valid.
	ErrNativeSignature = native.ErrSignature
	// ErrNativeCache means the cache directory or file cannot be used: another
	// user owns it or a directory above it can be changed by others. That
	// includes a group-writable directory such as a home with mode 0775 and a
	// private group: use mode 0755 or set LoadConfig.CacheDir.
	ErrNativeCache = native.ErrCache
)

The reasons the native engine can fail to load. LoadError, Runtime.NativeErr and the error of Load with EngineNative match them with errors.Is; the error text names the platform, each way that was tried and the operating system's message.

var ErrUnavailable = errors.New("kolovideo/vpx: libvpx unavailable")

ErrUnavailable means the requested libvpx engine cannot run.

Functions

func LoadError
func LoadError() error

LoadError returns why the native engine is not in use, nil if it is. On Linux builds without the kolovideo_native tag it says so.

func Notices
func Notices() string

Notices returns the licence and patent grant texts of the code in this package: libvpx, the C runtime parts of the portable engine and the native libraries, and the translator's helpers. An application that ships binaries must include them.

func WriteLibrary
func WriteLibrary(dir string) (string, error)

WriteLibrary writes the embedded native library into dir and returns its path, for an application that signs the library and ships it, then names it in LoadConfig.Library (on macOS also as Contents/Frameworks/libkolovpx.dylib, which is loaded without configuration). It errors when no library is embedded.

Types

type Codec
type Codec int

Codec is a video codec.

const (
	VP8 Codec = iota + 1
	VP9
)

Supported codecs.

func (Codec) String
func (c Codec) String() string

String returns the codec name.

type Config
type Config struct {
	Codec Codec
	// Width and Height of every frame: 1 to 16383 for VP8, 1 to 65536 for
	// VP9.
	Width, Height int
	// FrameRate is the nominal rate, DefaultFrameRate if zero; otherwise
	// kolovideo.MinFrameRate to kolovideo.MaxFrameRate.
	FrameRate float64
	// Bitrate is the target in bits per second, DefaultBitsPerPixel times
	// pixels times FrameRate if zero.
	Bitrate int
	// KeyframeInterval is the time between keyframes, measured on frame
	// timestamps; DefaultKeyframeInterval if zero, negative for keyframes
	// only on request.
	KeyframeInterval time.Duration
}

Config sets up an Encoder. Zero fields take defaults.

type Decoder
type Decoder struct {
	// contains filtered or unexported fields
}

Decoder decodes packets into I420 frames. Its methods are safe for concurrent use.

func NewDecoder
func NewDecoder(cfg DecoderConfig) (*Decoder, error)

NewDecoder creates a decoder. Call Close to free it.

func (Decoder) Close
func (d *Decoder) Close() error

Close frees the decoder. It is idempotent.

func (Decoder) Decode
func (d *Decoder) Decode(p Packet) (*kolovideo.Frame, error)

Decode decodes one packet. It returns a nil frame and nil error when the packet holds no frame to show. Frames may change size at keyframes. After any error, packets are refused with ErrNeedKeyframe until the next keyframe.

type DecoderConfig
type DecoderConfig struct {
	Codec Codec
	// MaxPixels is the largest frame accepted, checked in each frame
	// header before decoding; DefaultMaxPixels if zero.
	MaxPixels int
	// MaxPacketBytes bounds a packet, checked before it is copied;
	// DefaultMaxPacketBytes if zero, at most 256 MiB.
	MaxPacketBytes int
}

DecoderConfig sets up a Decoder.

type Encoder
type Encoder struct {
	// contains filtered or unexported fields
}

Encoder encodes I420 frames. Its methods are safe for concurrent use.

func NewEncoder
func NewEncoder(cfg Config) (*Encoder, error)

NewEncoder creates an encoder. Call Close to free it.

func (Encoder) Close
func (e *Encoder) Close() error

Close frees the encoder. It is idempotent.

func (Encoder) Config
func (e *Encoder) Config() Config

Config returns the configuration with defaults filled in.

func (Encoder) Encode
func (e *Encoder) Encode(f *kolovideo.Frame, forceKeyframe bool) ([]Packet, error)

Encode compresses one frame of the configured size. It returns no packet when rate control drops the frame. A keyframe is made when forceKeyframe is set, for the first frame, and every KeyframeInterval.

func (Encoder) SetBitrate
func (e *Encoder) SetBitrate(bps int) error

SetBitrate changes the target bitrate in bits per second, effective from the next frame.

type EngineKind
type EngineKind int

EngineKind names a libvpx engine.

const (
	EngineAuto EngineKind = iota
	EngineNative
	EnginePortable
)

Engines. The native engine is the prebuilt library, the portable engine is the same libvpx translated to Go.

func Engine
func Engine() EngineKind

Engine returns the engine in use, loading the default one if needed. It returns EngineAuto if no engine can run.

func (EngineKind) String
func (k EngineKind) String() string

String returns the engine name.

type Limits
type Limits struct {
	Engine EngineKind
	Encode RealTime
	Decode RealTime
}

Limits are the largest modes the engine in use handled in real time.

func Capabilities
func Capabilities() Limits

Capabilities returns reference limits, not a run-time probe: the largest encode and decode modes measured as real time for the engine in use, so a caller can cap resolution. Real time means a mean time per frame below the 33.3 ms period for a camera-like stream, on an Apple M5: the native engine reaches 1920x1080 at 30 fps for encode and decode, VP8 and VP9; the portable engine encodes up to 640x480 at 30 fps and decodes up to 1920x1080. A slower CPU reaches less. It returns the zero Limits if no engine can run.

type LoadConfig
type LoadConfig struct {
	// Engine is EngineAuto if zero: native, with the portable engine as the
	// fallback when the native library cannot load.
	Engine EngineKind
	// CacheDir is the root of the native library cache, os.UserCacheDir()/kolo-video
	// if empty.
	CacheDir string
	// Library is the path of a libkolovpx the application ships and signs
	// itself, loaded instead of the embedded one. It is not checked against a
	// hash. An altered library that still carries a valid signature can be
	// killed by the operating system when it is loaded, which cannot be caught.
	Library string
}

LoadConfig selects and sets up the engine. Library and CacheDir apply to the native engine only.

type Packet
type Packet struct {
	Data     []byte
	Keyframe bool
	// Timestamp and Duration come from the frame and pass through the
	// decoder unchanged.
	Timestamp time.Duration
	Duration  time.Duration
}

Packet is one encoded frame.

type RealTime
type RealTime struct {
	Width, Height int
	FrameRate     float64
}

RealTime is a picture size and frame rate.

type Runtime
type Runtime struct {
	Engine  EngineKind
	Version string // libvpx version, such as "v1.16.0"
	// NativeErr is why the native engine is not in use, nil if it is. Check it
	// after Load with EngineAuto: ErrNativeIntegrity and ErrNativeSignature are
	// not reported any other way.
	NativeErr error
}

Runtime describes the engine in use.

func Load
func Load(cfg LoadConfig) (Runtime, error)

Load selects the libvpx engine. It is optional: NewEncoder, NewDecoder, Engine, LoadError and Capabilities load the default configuration when nothing is loaded (on macOS and Windows that extracts and loads the native library), so call Load first to choose another one. Once an engine is active, Load with another configuration returns kolovideo.ErrInvalidConfig and the first engine stays. With Engine set to EngineNative, a native load failure returns ErrUnavailable wrapping the cause and nothing is loaded.

With EngineAuto a failed native load is not an error: the portable engine runs and Load returns nil. Read Runtime.NativeErr (or LoadError) to learn why. ErrNativeIntegrity and ErrNativeSignature there mean that the embedded bytes or a bundle library did not verify, which is corruption or tampering, not the plain absence of a native library (ErrNativeUnsupported); log them.

Package h264

github.com/ciranda-dev/kolo-video/h264

Package h264 encodes and decodes H.264 video with the codec of the operating system: VideoToolbox on macOS, the Media Foundation H.264 transforms on Windows desktop editions. It contains no H.264 encoder or decoder and needs no installed library, compiler or cgo. Where the system has no usable codec, Capabilities and the constructors report why with an *UnavailableError.

Packets are Annex B access units with 4-byte start codes, one picture each, holding only slices, SPS and PPS; every keyframe starts with its SPS and PPS. Encoders produce Constrained Baseline, Main or High within level 5.1, without B-frames, one packet per frame. Decoders check every packet in Go before the system sees it and refuse streams outside those profiles and levels.

Experimental: the API may change before v1.0.0.

Constants

const (
	// DefaultFrameRate is the rate assumed for frames without a Duration:
	// kolovideo.DefaultFrameRate.
	DefaultFrameRate = kolovideo.DefaultFrameRate
	// DefaultBitsPerPixel sets the default bitrate: about 4.4 Mbit/s for
	// 1080p at 30 fps.
	DefaultBitsPerPixel = 0.07
	// DefaultKeyframeInterval starts a new group of pictures every 2 s.
	DefaultKeyframeInterval = 2 * time.Second
	// DefaultMaxPixels bounds the coded size of decoded pictures: 36864
	// macroblocks, the level 5.1 MaxFS.
	DefaultMaxPixels = 9437184
	// DefaultMaxPacketBytes bounds a packet, as kolo-media's playout MaxBytes.
	DefaultMaxPacketBytes = 16 << 20
	// MaxBitrate is the level 5.1 MaxBR in bits per second, in the Baseline
	// and Main unit.
	MaxBitrate = 240000000
)

Defaults and bounds. The level limits are those of level 5.1 (ITU-T H.264 Table A-1), the highest level every engine documents.

Variables

var (
	// ErrUnavailable means this system has no H.264 codec kolo-video can use;
	// the error returned is an *UnavailableError that wraps it.
	ErrUnavailable = errors.New("kolovideo/h264: no H.264 codec on this system")
	// ErrCorrupt means a packet is not valid H.264 or the system decoder
	// reported bad data.
	ErrCorrupt = errors.New("kolovideo/h264: corrupt packet")
	// ErrTooLarge means a packet, its NAL unit count or its picture exceeds a
	// bound.
	ErrTooLarge = errors.New("kolovideo/h264: packet or picture too large")
	// ErrNeedKeyframe means the decoder drops packets until an IDR with its
	// SPS and PPS.
	ErrNeedKeyframe = errors.New("kolovideo/h264: waiting for a keyframe")
	// ErrUnsupportedStream means the stream uses a profile, level or tool this
	// package refuses.
	ErrUnsupportedStream = errors.New("kolovideo/h264: unsupported stream")
	// ErrProfile means the system encoder cannot produce the requested
	// profile.
	ErrProfile = errors.New("kolovideo/h264: profile not available from the system encoder")
	// ErrCodec means the system codec failed for a reason other than bad
	// input.
	ErrCodec = errors.New("kolovideo/h264: system codec error")
)

Types

type Config
type Config struct {
	// Width and Height are even, 48 to 4096, at most 36864 macroblocks, and
	// macroblocks times FrameRate at most 983040 (level 5.1).
	Width, Height int
	// FrameRate is kolovideo.MinFrameRate to MaxFrameRate, DefaultFrameRate
	// if zero.
	FrameRate float64
	// Bitrate is the target in bits per second, 1 to MaxBitrate;
	// DefaultBitsPerPixel times pixels times FrameRate if zero.
	Bitrate int
	// KeyframeInterval is the time between keyframes on frame timestamps,
	// DefaultKeyframeInterval if zero, negative for keyframes on request only.
	KeyframeInterval time.Duration
	// Profile is ConstrainedBaseline if zero.
	Profile Profile
}

Config sets up an Encoder. Zero fields take defaults.

type Decoder
type Decoder struct {
	// contains filtered or unexported fields
}

Decoder decodes H.264 packets with the system codec. Its methods are safe for concurrent use. Every packet is checked in Go before the system sees it, and pictures come out in decoding order.

func NewDecoder
func NewDecoder(cfg DecoderConfig) (*Decoder, error)

NewDecoder creates a decoder. Call Close to free it.

func (Decoder) Close
func (d *Decoder) Close() error

Close frees the decoder. It is idempotent. It waits for a Decode in progress.

func (Decoder) Decode
func (d *Decoder) Decode(p Packet) (*kolovideo.Frame, error)

Decode returns the picture in p, or nil and nil when p holds no picture. After any error, packets are refused with ErrNeedKeyframe until an IDR with its SPS and PPS.

func (Decoder) Hardware
func (d *Decoder) Hardware() bool

Hardware reports whether the system decodes this stream in hardware.

func (Decoder) SPS
func (d *Decoder) SPS() (s SPS, ok bool)

SPS returns the active sequence parameter set, ok false before the first keyframe.

type DecoderConfig
type DecoderConfig struct {
	// MaxPixels bounds the coded picture size, macroblocks times 256 before
	// cropping, which is what the system decoder allocates: 1920x1080 is
	// coded as 1920x1088. DefaultMaxPixels if zero, at most DefaultMaxPixels.
	MaxPixels int
	// MaxPacketBytes bounds a packet, DefaultMaxPacketBytes if zero, at most
	// 256 MiB so that bit offsets fit in 32 bits.
	MaxPacketBytes int
}

DecoderConfig sets up a Decoder. Zero fields take defaults.

type Encoder
type Encoder struct {
	// contains filtered or unexported fields
}

Encoder encodes I420 frames with the system codec. Its methods are safe for concurrent use.

func NewEncoder
func NewEncoder(cfg Config) (*Encoder, error)

NewEncoder creates an encoder. On Windows it also checks the profile; call Close to free it.

func (Encoder) Close
func (e *Encoder) Close() error

Close frees the encoder. It is idempotent.

func (Encoder) Config
func (e *Encoder) Config() Config

Config returns the configuration with defaults filled in.

func (Encoder) Encode
func (e *Encoder) Encode(f *kolovideo.Frame, forceKeyframe bool) ([]Packet, error)

Encode returns the access unit for f, or none when the system encoder dropped the frame. A keyframe is made for the first frame, every KeyframeInterval and when forceKeyframe is set.

func (Encoder) Hardware
func (e *Encoder) Hardware() bool

Hardware reports whether the system encodes this stream in hardware.

func (Encoder) SetBitrate
func (e *Encoder) SetBitrate(bps int) error

SetBitrate changes the target bitrate from the next frame.

type Limits
type Limits struct{ Encode, Decode Support }

Limits is the encode and decode support of this system.

func Capabilities
func Capabilities() Limits

Capabilities probes the system codec once per process and returns the result.

type Packet
type Packet struct {
	Data []byte
	// Keyframe is set by the encoder; Decode ignores it and reads the NAL types.
	Keyframe bool
	// Timestamp and Duration pass through unchanged.
	Timestamp time.Duration
	Duration  time.Duration
}

Packet is one access unit: Annex B, 4-byte start codes, NAL types 1, 5, 7 and 8 only, one picture. A keyframe starts with an SPS and a PPS, then an IDR picture.

type Profile
type Profile int

Profile is an H.264 profile.

const (
	ConstrainedBaseline Profile = iota + 1 // profile_idc 66 with constraint_set1
	Main                                   // profile_idc 77
	High                                   // profile_idc 100
)

Profiles.

func (Profile) String
func (p Profile) String() string

String returns the profile name.

type SPS
type SPS struct {
	ID                             uint32
	ProfileIDC, Constraints, Level uint8
	// Width and Height are the picture size after cropping.
	Width, Height int
	MaxRefFrames  uint32
	FullRange     bool
	// Matrix is matrix_coefficients, 2 (unspecified) without a colour
	// description.
	Matrix uint8
}

SPS holds the sequence parameter set fields this package checks.

func ParseKeyframe
func ParseKeyframe(packet []byte) (SPS, error)

ParseKeyframe returns the first SPS of an Annex B keyframe packet, checked as Decoder checks it with the default bounds. It never panics.

func (SPS) Codec
func (s SPS) Codec() string

Codec returns the codec string, such as "avc3.42c01f" (RFC 6381 3.3).

func (SPS) Profile
func (s SPS) Profile() (p Profile, ok bool)

Profile classifies the SPS by RFC 6184 Table 5; ok is false for other profiles.

type Support
type Support struct {
	Available bool
	// Hardware reports that the system has a hardware codec for this direction.
	Hardware bool
	// Engine is "VideoToolbox" or "Media Foundation".
	Engine string
	// Profiles lists what the engine produces (encode) or accepts (decode). On
	// Windows the encode list is probed at run time; on macOS it comes from
	// the API and each encoder confirms its profile at the first keyframe.
	Profiles []Profile
	// MaxWidth and MaxHeight are the limits the system vendor documents, 0
	// when it documents none. No system documents or reports a frame-rate
	// limit, so none is reported.
	MaxWidth, MaxHeight int
	// Err says why Available is false.
	Err error
}

Support describes one direction of the system codec.

type UnavailableError
type UnavailableError struct {
	GOOS, GOARCH string
	// Engine is "VideoToolbox", "Media Foundation", or "" when none is built in.
	Engine string
	// Reason is the system's message, or why no engine exists for this platform.
	Reason string
	// Code is the OSStatus or HRESULT, 0 if none.
	Code int64
	// Devices lists, on Linux, the V4L2 stateful H.264 devices found and not used.
	Devices []string
}

UnavailableError says why H.264 is absent.

func (UnavailableError) Error
func (e *UnavailableError) Error() string

Error returns the reason with the platform and code.

func (UnavailableError) Unwrap
func (e *UnavailableError) Unwrap() error

Unwrap returns ErrUnavailable.

Good to know

How a frame travels

  1. A camera, a converter or your code makes a Frame: I420, BT.601, limited range.
  2. vpx encodes it as VP8 or VP9, and h264 as H.264 when the system has a codec.
  3. Each encoder returns one packet for a frame. Your application sends it.
  4. The receiver decodes the packet with the matching decoder, which checks it first.
  5. The decoder returns a Frame, or an error and a wait for a keyframe.

Frames and capture

  • A Frame holds 8-bit Y, U and V planes, plus Timestamp and Duration. NewFrame(w, h) allocates a black frame. Every function that takes a frame calls Validate and returns ErrInvalidFrame instead of panicking.
  • FromNV12, FromYUY2, FromUYVY, FromBGRA and FromRGBA convert a raw picture with any stride to a new frame. Frame.RGBA() converts back, and Renderer{Scale: Fit} draws into an *image.RGBA with letterboxing (Fill crops).
  • Devices() lists cameras without opening one. OpenCapture picks the supported mode nearest to the request and Capture.Read returns frames. When the reader is slow at most Buffer frames are kept and the oldest is dropped. A disconnect, a capture failure, or no frame for StallTimeout ends the capture with ErrDeviceLost after the buffered frames are read.
  • The backends are AVFoundation on macOS (through purego), Media Foundation on Windows (through COM) and V4L2 on Linux (pure Go, including Motion-JPEG). On macOS OpenCapture returns ErrPermission until the user grants access, and never asks: RequestAccess asks. Frames are always converted to BT.601 limited range I420.
  • Elsewhere, Devices, RequestAccess and OpenCapture return ErrUnsupported: the platform has no capture backend. Frames, conversion, rendering and the codecs still work.

VP8 and VP9

  • vpx.NewEncoder and vpx.NewDecoder work in real time: one pass, constant bitrate, no lag, error resilience and frame dropping. The first frame and one frame per KeyframeInterval (2 s by default) are keyframes, Encode(f, true) forces one, and SetBitrate changes the target from the next frame. Encode returns no packet when rate control drops a frame.
  • The decoder checks every frame header against MaxPixels before libvpx allocates anything (ErrTooLarge), decodes from its own copy of the packet, and refuses packets with ErrNeedKeyframe from the start and after any error until a keyframe arrives.
  • vpx.DecoderConfig.MaxPacketBytes refuses a packet above it with ErrTooLarge, before the packet is copied: 16 MiB by default, at most 256 MiB (vpx.DefaultMaxPacketBytes, the same bound as in h264). The decoder keeps at most 1 MiB of packet buffer between calls.
  • Two engines run the same libvpx 1.16.0, and vpx.Load makes the choice once per process. The native engine is a prebuilt library per target, loaded without cgo; the portable engine is the same code translated to Go. vpx.Capabilities() lists the largest real-time modes of the engine in use: native 1920x1080 at 30 fps for both directions, portable 640x480 at 30 fps for encode and 1920x1080 at 30 fps for decode.
  • On Linux the default binary is static and uses the portable engine; build with -tags kolovideo_native on glibc 2.28 or later for the native one. A signed macOS app needs one more step for the native engine, and Windows Smart App Control may refuse the embedded DLL. In both cases the portable engine runs and vpx.LoadError() says why.

H.264

  • h264.Capabilities() says whether the system can encode and decode H.264, whether the codec is hardware, which engine it is (VideoToolbox or Media Foundation) and which profiles it offers. Where it is absent, the error is an *UnavailableError that matches h264.ErrUnavailable and names the platform and the reason.
  • kolo-video contains no H.264 code. VideoToolbox on macOS and the Media Foundation H.264 transforms on Windows desktop editions do the coding; nothing is installed, bundled or downloaded. On Linux and on every other platform there is no engine, and h264 does not import vpx or any native library.
  • Profiles are Constrained Baseline (the default), Main and High, up to level 5.1, progressive 4:2:0 at 8 bits, with no B-frames. Sizes are even and from 48 to 4096 on each side. The encoder returns one packet for a frame, inside the same Encode call.
  • A packet is one Annex B access unit with 4-byte start codes. Every keyframe starts with its SPS and PPS, so a receiver can start at any keyframe. h264.ParseKeyframe(packet) and SPS.Codec() give the codec string, such as avc3.42c01f. The string comes from the first keyframe and is advisory.
  • The recommended pair for a sender is VP8 always and H.264 when Capabilities says it can encode: two renditions of one source. A receiver that cannot decode H.264 takes VP8. Nothing falls back by itself: your application checks Capabilities and chooses. kolo-media v0.1.0 does not carry H.264: its catalog accepts VP8 and VP9 for video.
  • The decoder checks every packet in Go before the system sees it, so a stream outside the supported profiles, levels and tools is ErrUnsupportedStream, a damaged packet ErrCorrupt, an oversized one ErrTooLarge, and after any error it returns ErrNeedKeyframe until a keyframe arrives.
  • SetBitrate changes the target from the next frame. The macOS hardware encoder followed a change from 1.9 to 0.5 Mbit/s to within 1 percent; the macOS software encoder was not converged (988 kbit/s for a 500 kbit/s target).

Platforms

PlatformVP8 and VP9H.264 (OS codec)Camera backend
macOS arm64native, portable as fallbackVideoToolbox, hardware on the Mac that ran itAVFoundation through purego
macOS amd64sameVideoToolboxsame
Windows arm64native, portable as fallbackMedia Foundation, softwareMedia Foundation through COM
Windows amd64samesamesame
Linux amd64, arm64portable; native with -tags kolovideo_nativenone: ErrUnavailableV4L2 in pure Go
Otherportablenone: ErrUnavailablenone: ErrUnsupported

What ran for H.264:

  • macOS arm64, an Apple M5: round trips at 640x480, 1280x720 and 1920x1080 in Constrained Baseline, Main and High decoded every frame with luma PSNR of at least 42.69 dB, with the hardware encoder and decoder. The software path (hardware disabled) and damaged input were tested too, and the binary ran ad-hoc signed with the hardened runtime. Under Rosetta, amd64 passed the same tests. No Intel Mac was used.
  • Windows 11 ARM64 in a VM: the same round trips with luma PSNR of at least 42.86 dB, one picture for every packet from the first, software transforms only. The VM's x64 emulation passed the functional tests. Native Windows amd64 was not run.
  • The macOS and Windows encoders each produced streams that the other system's decoder decoded.
  • Linux, in Debian and Alpine containers: the absence message, a static binary and the parsers. No V4L2 stateful codec device exists on any test machine, so detection never ran against a real driver.
  • Not validated: Linux encode and decode (not provided), VA-API, Windows hardware transforms, N editions and Windows Server, a Developer ID signed and notarized app, a real camera feeding the H.264 encoder, and browsers and phones.

A camera stream was run on one Mac's built-in camera only. Linux and Windows capture never ran on a real camera.

Limits

A value outside a range returns ErrInvalidConfig or ErrInvalidFrame. The full tables are in Limits.

  • Frames: 1 to 65536 on each side. CaptureConfig.MaxPixels and DecoderConfig.MaxPixels bound pictures, 8,912,896 pixels for VP8 and VP9 and 9,437,184 for H.264.
  • VP8 frames are 1 to 16383 on each side, and the largest VP9 frame is 65536.
  • H.264 encoder: even sizes from 48 to 4096, at most 36864 macroblocks, and macroblocks times frame rate at most 983040. That is 1920x1080 up to 120 fps and 3840x2160 up to 30 fps. Bitrate 1 to 240,000,000 bit/s.
  • H.264 decoder: profiles 66, 77 and 100, levels 1 to 5.1, at most 16 MiB per packet by default, and one picture per packet.
  • The portable engine uses at most 1 GiB of memory per instance.

Security

  • Every frame header is checked in Go against MaxPixels before libvpx allocates anything, and the decoder works on its own copy of the packet.
  • For H.264, only NAL unit types 1, 5, 7 and 8 reach the system. A parameter set is accepted only in a keyframe packet, and the level, the picture size and the reference frames are checked against Table A-1 of the H.264 specification.
  • On Windows, damaged slices that pass the checks are concealed by the system and returned as pictures, so Decode does not report them, and a crash inside the decoder ends the application. A hang inside the system codec blocks Decode and Close.
  • The embedded native libraries are checked against their SHA-256 before every load, and the cache directory and every directory above it are checked for ownership and permissions. A signed macOS app can refuse the library, and the portable engine runs.
  • The library makes no network connection, starts no process and uses no cgo.

Status

  • Done: frames and conversions, camera capture on macOS, Windows and Linux, VP8 and VP9 on two engines, and H.264 through VideoToolbox and Media Foundation, with fuzz targets and tests.
  • Tested: H.264 ran on one Mac and in one Windows VM, as the platforms list says. A camera stream ran on one Mac only.
  • Not done: H.264 on Linux, and kolo-media does not carry H.264 yet.
  • Experimental. The API may change before v1.0.0.

Errors

ErrorMessage
ErrClosedkolovideo: closed
ErrDeviceLostkolovideo: device lost
ErrInvalidConfigkolovideo: invalid config
ErrInvalidFramekolovideo: invalid frame
ErrNoDevicekolovideo: no such device
ErrPermissionkolovideo: camera access not granted
ErrUnsupportedkolovideo: not supported on this platform
ErrUnsupportedFormatkolovideo: unsupported format
vpx.ErrCodeckolovideo/vpx: codec error
vpx.ErrCorruptkolovideo/vpx: corrupt packet
vpx.ErrNativeABInative libvpx ABI mismatch
vpx.ErrNativeCachenative libvpx cache is unusable
vpx.ErrNativeIntegrityembedded native libvpx does not match its checksum
vpx.ErrNativeSignaturenative libvpx code signature is not valid
vpx.ErrNativeUnsupportedno native libvpx
vpx.ErrNeedKeyframekolovideo/vpx: waiting for a keyframe
vpx.ErrTooLargekolovideo/vpx: packet or frame too large
vpx.ErrUnavailablekolovideo/vpx: libvpx unavailable
h264.ErrCodeckolovideo/h264: system codec error
h264.ErrCorruptkolovideo/h264: corrupt packet
h264.ErrNeedKeyframekolovideo/h264: waiting for a keyframe
h264.ErrProfilekolovideo/h264: profile not available from the system encoder
h264.ErrTooLargekolovideo/h264: packet or picture too large
h264.ErrUnavailablekolovideo/h264: no H.264 codec on this system
h264.ErrUnsupportedStreamkolovideo/h264: unsupported stream

Every error starts with kolovideo:, or kolovideo/vpx: or kolovideo/h264: for the codec packages, and wraps a sentinel, so errors.Is works. ErrUnsupported means this platform has no capture backend, and the error names GOOS and GOARCH. h264.ErrUnavailable comes as an *UnavailableError that says why the system has no codec.

References