kolo-video lists and opens cameras, delivers I420 frames (with the BT.601 matrix), converts other pixel formats to I420, draws frames into RGBA images, and encodes and decodes VP8 (RFC 6386) and VP9 for real-time use. H.264 is optional and comes from the operating system's own codec. It does not send anything over the network and does not sync audio with video. The Go packages are kolovideo, vpx and h264.
From a Frame to packets and back
VP8 and VP9 run everywhere. H.264 uses the codec of the operating system, so it is optional.
- A Frame is one I420 picture: 8-bit planes, BT.601, limited range. It comes from a camera, from a converter such as FromNV12 or from your own code.
- vpx encodes it as VP8 or VP9 with libvpx 1.16.0. The native engine is a prebuilt library loaded without cgo; the portable engine is the same code translated to Go. The package picks one and tells you which.
- h264 encodes it as H.264 with the codec the operating system ships: VideoToolbox on macOS, Media Foundation on Windows. It is optional: Capabilities says whether it is available, and on Linux it is not.
- Each encoder returns one packet for a frame: a VP8 or VP9 frame, or an Annex B access unit with its SPS and PPS before every keyframe. A sender can send VP8 always and H.264 when it can. Your application sends the packets.
- The receiver decodes the stream it picked. It takes H.264 only if its own system can decode it, and VP8 otherwise. Each decoder checks a packet before it decodes it.
- The decoder returns a Frame, or an error, and then waits for a keyframe.
Install
go get github.com/ciranda-dev/kolo-videoTo pin this release, run go get github.com/ciranda-dev/kolo-video@v0.1.0. It needs Go 1.26 or newer. There is no cgo, no C compiler, no libvpx, no pkg-config and no environment variable to set, so it builds with CGO_ENABLED=0. The VP8 and VP9 codec is libvpx 1.16.0 in two forms that carry the same code: a prebuilt library embedded in the module and loaded without cgo (full speed), and the same code translated to Go (works everywhere, real time up to 640x480). The package picks one at run time and tells you which. The first build of the vpx package compiles generated Go and peaks at about 570 MB of memory on a Mac; later builds use the build cache.
Quick start
This program lists the cameras (it never opens one), then encodes 30 generated frames with VP8 and decodes them again. Run it with go run . <dir>, where <dir> is where the native engine keeps its library; an application normally leaves CacheDir empty, which uses the user cache directory.
package main
import (
"errors"
"fmt"
"log"
"os"
"time"
kolovideo "github.com/ciranda-dev/kolo-video"
"github.com/ciranda-dev/kolo-video/vpx"
)
func main() {
if len(os.Args) != 2 {
log.Fatal("usage: quickstart <cache directory>")
}
cams, err := kolovideo.Devices()
if err != nil && !errors.Is(err, kolovideo.ErrNoDevice) && !errors.Is(err, kolovideo.ErrUnsupported) {
log.Fatal(err)
}
for _, c := range cams {
fmt.Printf("camera %q default=%v\n", c.Name, c.Default)
}
rt, err := vpx.Load(vpx.LoadConfig{CacheDir: os.Args[1]})
if err != nil {
log.Fatal(err)
}
fmt.Printf("libvpx %s, %v engine\n", rt.Version, rt.Engine)
enc, err := vpx.NewEncoder(vpx.Config{Codec: vpx.VP8, Width: 320, Height: 240})
if err != nil {
log.Fatal(err)
}
defer enc.Close()
dec, err := vpx.NewDecoder(vpx.DecoderConfig{Codec: vpx.VP8})
if err != nil {
log.Fatal(err)
}
defer dec.Close()
var bytes, frames int
for i := range 30 {
f, err := kolovideo.NewFrame(320, 240)
if err != nil {
log.Fatal(err)
}
for j := range f.Y {
f.Y[j] = byte(16 + (j/320+j%320+4*i)%220)
}
f.Timestamp = time.Duration(i) * time.Second / 30
pkts, err := enc.Encode(f, false)
if err != nil {
log.Fatal(err)
}
for _, p := range pkts {
out, err := dec.Decode(p)
if err != nil {
log.Fatal(err)
}
if out != nil {
bytes += len(p.Data)
frames++
}
}
}
fmt.Printf("%d frames encoded and decoded, %d bytes\n", frames, bytes)
}The engine line depends on the platform (native on macOS and Windows, portable on Linux by default). The last line does not:
30 frames encoded and decoded, 13596 bytesH.264 has the same shape, with one extra step: ask the system first. This program encodes one frame with H.264, reads the codec string from the keyframe and decodes it again. Where the system has no H.264 codec it prints why, and you publish VP8 only.
package main
import (
"errors"
"fmt"
"log"
kolovideo "github.com/ciranda-dev/kolo-video"
"github.com/ciranda-dev/kolo-video/h264"
)
func main() {
enc, err := h264.NewEncoder(h264.Config{Width: 640, Height: 360})
if errors.Is(err, h264.ErrUnavailable) {
fmt.Println("H.264 is not available:", err) // publish VP8 only
return
}
if err != nil {
log.Fatal(err)
}
defer enc.Close()
dec, err := h264.NewDecoder(h264.DecoderConfig{})
if err != nil {
log.Fatal(err)
}
defer dec.Close()
frame, err := kolovideo.NewFrame(640, 360)
if err != nil {
log.Fatal(err)
}
pkts, err := enc.Encode(frame, false) // the first frame is a keyframe
if err != nil {
log.Fatal(err)
}
sps, err := h264.ParseKeyframe(pkts[0].Data)
if err != nil {
log.Fatal(err)
}
out, err := dec.Decode(pkts[0])
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s, hardware %v, %d bytes, decoded %dx%d\n", sps.Codec(), enc.Hardware(), len(pkts[0].Data), out.Width, out.Height)
}On a Mac with VideoToolbox it printed:
avc3.42c01e, hardware true, 739 bytes, decoded 640x360In a Linux container, where there is no engine, it printed:
H.264 is not available: kolovideo/h264: no H.264 engine on linux/arm64 in this release; V4L2 stateful H.264 devices: noneTo capture, open a camera and read frames. OpenCapture never asks for permission: on macOS call RequestAccess first. The repository has examples/snapshot, which saves camera frames as PNG files (-list lists the cameras), examples/vp8, which encodes and decodes generated frames and prints the engine, the size and the quality, and examples/h264, which does the same with H.264 and falls back to VP8 where the system has no codec.
API reference
The API reference is generated from the Go source and is in English.
Package kolovideo captures camera video and converts, scales and renders raw I420 frames for Kolo. Subpackage vpx encodes and decodes VP8 and VP9; subpackage h264 encodes and decodes H.264 with the operating system's codec where there is one. Nothing needs to be installed and no package uses cgo.
Experimental: the API may change before v1.0.0.
Constants
const (
// DefaultWidth, DefaultHeight and DefaultFrameRate are the common
// browser capture default, 640x480 at 30 fps. vpx.DefaultFrameRate and
// h264.DefaultFrameRate are the encoders' rate for frames without a
// Duration, the same value.
DefaultWidth = 640
DefaultHeight = 480
DefaultFrameRate = 30
// MinFrameRate and MaxFrameRate bound a wanted frame rate: one frame per
// hour, so that a frame duration in 90 kHz ticks fits the 32-bit duration of
// libvpx, and 1000 frames per second, the finest rate a 90 kHz clock gives
// 90 ticks to.
MinFrameRate = 1.0 / 3600
MaxFrameRate = 1000
// DefaultBuffer frames are held for a slow reader: one late read is
// absorbed without adding latency.
DefaultBuffer = 2
// DefaultStallTimeout is the wait for a frame before the device counts
// as lost, the timeout of the V4L2 reference capture loop.
DefaultStallTimeout = 2 * time.Second
// DefaultMaxPixels bounds the capture size: 8,912,896 pixels, the VP9
// level 5 picture size (4096x2176), which holds 4K UHD. vpx.DefaultMaxPixels
// bounds decoded VP8 and VP9 frames with the same value; h264.DefaultMaxPixels
// bounds the coded size of H.264 pictures and is a different value.
DefaultMaxPixels = 8912896
)Capture defaults.
Variables
var (
// ErrInvalidConfig means a configuration field is out of range.
ErrInvalidConfig = errors.New("kolovideo: invalid config")
// ErrInvalidFrame means a frame or raw image has inconsistent sizes.
ErrInvalidFrame = errors.New("kolovideo: invalid frame")
// ErrNoDevice means no camera matches the request.
ErrNoDevice = errors.New("kolovideo: no such device")
// ErrUnsupportedFormat means the device or stream offers no format this
// package can convert.
ErrUnsupportedFormat = errors.New("kolovideo: unsupported format")
// ErrPermission means the system has not granted camera access.
ErrPermission = errors.New("kolovideo: camera access not granted")
// ErrDeviceLost means the camera was disconnected, failed, or sent no
// frame within the stall timeout.
ErrDeviceLost = errors.New("kolovideo: device lost")
// ErrClosed means the capture, encoder or decoder was closed.
ErrClosed = errors.New("kolovideo: closed")
// ErrUnsupported means this platform has no backend for the feature:
// camera capture on a system other than macOS, Linux, and Windows on
// amd64 or arm64, or a system whose camera framework is missing.
ErrUnsupported = errors.New("kolovideo: not supported on this platform")
)Functions
func RequestAccess
func RequestAccess(ctx context.Context) errorRequestAccess asks the user for camera access where the system requires it (macOS) and returns ErrPermission if it is refused. It may show a system prompt. On Linux and Windows it returns nil at once; where capture is not supported it returns ErrUnsupported.
Types
type Capture
type Capture struct {
// contains filtered or unexported fields
}Capture is an open camera. Its methods are safe for concurrent use.
func OpenCapture
func OpenCapture(cfg CaptureConfig) (*Capture, error)OpenCapture opens a camera and starts capturing.
func (Capture) Close
func (c *Capture) Close() errorClose stops the camera. Blocked and later Read calls return ErrClosed. A zero Capture, which holds no camera, can be closed and read too. Close is idempotent; only the first call can return an error.
func (Capture) Device
func (c *Capture) Device() DeviceDevice returns the camera in use.
func (Capture) Mode
func (c *Capture) Mode() ModeMode returns the mode in use.
func (Capture) Read
func (c *Capture) Read(ctx context.Context) (*Frame, error)Read returns the oldest buffered frame, waiting for one if needed. After the camera is lost, buffered frames are returned first, then an error wrapping ErrDeviceLost or ErrPermission. After Close it returns ErrClosed.
func (Capture) Stats
func (c *Capture) Stats() StatsStats returns the frame counters.
type CaptureConfig
type CaptureConfig struct {
// DeviceID is a Device.ID; empty means the default camera.
DeviceID string
// Width and Height are the wanted size, DefaultWidth by DefaultHeight
// if both are zero. The nearest supported mode is used.
Width, Height int
// FrameRate is the wanted rate, DefaultFrameRate if zero. It is capped
// at the chosen mode's highest rate.
FrameRate float64
// Buffer is the number of frames held while the reader is slow; when
// it is full the oldest frame is dropped. Default DefaultBuffer.
Buffer int
// StallTimeout is how long Read waits for a frame before it returns
// ErrDeviceLost. Default DefaultStallTimeout; negative disables it.
StallTimeout time.Duration
// MaxPixels is the largest mode accepted, so a faulty camera cannot
// make every frame a huge allocation. Default DefaultMaxPixels.
MaxPixels int
}CaptureConfig selects a camera and mode. Zero fields take defaults.
type Device
type Device struct {
// ID identifies the camera to OpenCapture. It is stable while the
// camera stays connected.
ID string
Name string
// Default marks the system's preferred camera.
Default bool
// Modes lists supported modes when the platform reports them without
// starting the camera; it is empty on Windows. A Linux node held exclusively
// by another program is listed with no modes.
Modes []Mode
}Device is a camera.
func Devices
func Devices() ([]Device, error)Devices lists the cameras connected now.
type Frame
type Frame struct {
Width, Height int
Y, U, V []byte
// YStride and UVStride are the bytes between the starts of two rows.
YStride, UVStride int
// Timestamp is the capture time on the system monotonic clock.
Timestamp time.Duration
// Duration is the nominal display time of the frame.
Duration time.Duration
}Frame is a raw I420 picture: planar YUV 4:2:0, 8 bits per sample, BT.601 limited range. The U and V planes are (Width+1)/2 by (Height+1)/2.
func FromBGRA
func FromBGRA(width, height int, src []byte, stride int) (*Frame, error)FromBGRA converts 32-bit B, G, R, A pixels to a new I420 frame. Alpha is ignored.
func FromNV12
func FromNV12(width, height int, y []byte, yStride int, uv []byte, uvStride int) (*Frame, error)FromNV12 converts an NV12 picture (a Y plane and an interleaved UV plane) to a new I420 frame.
func FromRGBA
func FromRGBA(width, height int, src []byte, stride int) (*Frame, error)FromRGBA converts 32-bit R, G, B, A pixels, such as image.RGBA.Pix, to a new I420 frame. Alpha is ignored.
func FromUYVY
func FromUYVY(width, height int, src []byte, stride int) (*Frame, error)FromUYVY converts a packed 4:2:2 picture in U Y0 V Y1 order to a new I420 frame. The width must be even.
func FromYUY2
func FromYUY2(width, height int, src []byte, stride int) (*Frame, error)FromYUY2 converts a packed 4:2:2 picture in Y0 U Y1 V order (also called YUYV) to a new I420 frame. The width must be even.
func NewFrame
func NewFrame(width, height int) (*Frame, error)NewFrame allocates a black frame with tightly packed planes.
func (Frame) RGBA
func (f *Frame) RGBA() (*image.RGBA, error)RGBA converts the frame to a new image of the same size.
func (Frame) Validate
func (f *Frame) Validate() errorValidate reports whether the sizes, strides and plane lengths agree.
func (Frame) YCbCr
func (f *Frame) YCbCr() *image.YCbCrYCbCr returns an image that shares the frame's planes, or nil if the frame is invalid. The image/color package treats the samples as full range, so use RGBA or Renderer for exact colors.
type Mode
type Mode struct {
Width, Height int
// FrameRate is the highest rate in Device.Modes, 0 if unknown. In
// Capture.Mode it is the rate the camera was set to, or the requested
// rate if the camera does not report one.
FrameRate float64
// Format is the camera's native format; captured frames are always
// I420.
Format PixelFormat
}Mode is one capture size and frame rate a camera supports.
type PixelFormat
type PixelFormat intPixelFormat is a camera's native pixel layout.
const (
FormatUnknown PixelFormat = iota
FormatI420
FormatNV12
FormatYUY2
FormatUYVY
FormatBGRA
FormatMJPEG
)Pixel formats a camera may deliver.
func (PixelFormat) String
func (p PixelFormat) String() stringString returns the format name.
type Renderer
type Renderer struct {
Scale ScaleMode
// contains filtered or unexported fields
}Renderer draws frames into RGBA images of any size. It keeps a scratch image between calls, so reuse one Renderer per video stream. A Renderer is not safe for concurrent use.
func (Renderer) Render
func (r *Renderer) Render(dst *image.RGBA, f *Frame) errorRender draws f into dst.Bounds(), keeping the frame's aspect ratio. Scaling uses approximate bilinear interpolation.
type ScaleMode
type ScaleMode intScaleMode says how Renderer maps a frame onto a destination of another shape.
const (
// Fit shows the whole frame and fills the bars left over with black.
Fit ScaleMode = iota
// Fill covers the whole destination and crops the frame's overflow.
Fill
)type Stats
type Stats struct {
// Frames is the number of frames received from the camera.
Frames uint64
// Dropped is the number of frames discarded because Buffer was full.
Dropped uint64
// Invalid is the number of camera frames that could not be converted.
Invalid uint64
}Stats counts frames since OpenCapture.
Package vpx
Package vpx encodes and decodes VP8 and VP9 video with libvpx 1.16.0, set up for real-time use. It needs no installed library, compiler or cgo.
Two engines run the same libvpx. The native engine is a prebuilt shared library embedded in the package and loaded without cgo; it is the default on macOS and Windows, and on Linux only with the build tag kolovideo_native, so that binaries stay static and start on any distribution. The portable engine is the same code translated to Go; it works everywhere, runs single-threaded and encodes in real time up to 640x480. When the native engine cannot load, the portable engine is used and LoadError says why. Streams from the two engines are byte-identical only when the encoder runs one thread.
Experimental: the API may change before v1.0.0.
Constants
const (
// DefaultFrameRate is the rate assumed for frames without a Duration:
// kolovideo.DefaultFrameRate.
DefaultFrameRate = kolovideo.DefaultFrameRate
// DefaultBitsPerPixel sets the default bitrate: 0.07 bits per pixel per
// frame, about 4.4 Mbit/s for 1080p at 30 fps.
DefaultBitsPerPixel = 0.07
// DefaultKeyframeInterval starts a new group of pictures every 2 s.
DefaultKeyframeInterval = 2 * time.Second
)Encoder defaults.
const DefaultMaxPacketBytes = 16 << 20DefaultMaxPacketBytes bounds a packet: 16 MiB, the default MaxBytes of the kolo-media playout. A VP8 or VP9 frame at DefaultMaxPixels is far smaller.
const DefaultMaxPixels = kolovideo.DefaultMaxPixelsDefaultMaxPixels bounds decoded frames: 8,912,896 pixels, the VP9 level 5 picture size (4096x2176), as kolovideo.DefaultMaxPixels for capture.
Variables
var (
// ErrCorrupt means a packet is truncated or not a valid frame.
ErrCorrupt = errors.New("kolovideo/vpx: corrupt packet")
// ErrTooLarge means a packet is larger than MaxPacketBytes or declares a
// frame larger than MaxPixels.
ErrTooLarge = errors.New("kolovideo/vpx: packet or frame too large")
// ErrNeedKeyframe means the decoder drops packets until a keyframe,
// at the start and after a decoding error.
ErrNeedKeyframe = errors.New("kolovideo/vpx: waiting for a keyframe")
// ErrCodec means libvpx failed for a reason other than bad input.
ErrCodec = errors.New("kolovideo/vpx: codec error")
)var (
// ErrNativeUnsupported means this platform or build has no native library:
// Linux without the kolovideo_native tag, or another platform.
ErrNativeUnsupported = native.ErrUnsupported
// ErrNativeIntegrity means the embedded library does not match its checksum.
ErrNativeIntegrity = native.ErrIntegrity
// ErrNativeABI means the library does not offer the interface this package
// expects.
ErrNativeABI = native.ErrABI
// ErrNativeSignature means the code signature of a library file on macOS
// is not valid.
ErrNativeSignature = native.ErrSignature
// ErrNativeCache means the cache directory or file cannot be used: another
// user owns it or a directory above it can be changed by others. That
// includes a group-writable directory such as a home with mode 0775 and a
// private group: use mode 0755 or set LoadConfig.CacheDir.
ErrNativeCache = native.ErrCache
)The reasons the native engine can fail to load. LoadError, Runtime.NativeErr and the error of Load with EngineNative match them with errors.Is; the error text names the platform, each way that was tried and the operating system's message.
Functions
func LoadError
func LoadError() errorLoadError returns why the native engine is not in use, nil if it is. On Linux builds without the kolovideo_native tag it says so.
func Notices
func Notices() stringNotices returns the licence and patent grant texts of the code in this package: libvpx, the C runtime parts of the portable engine and the native libraries, and the translator's helpers. An application that ships binaries must include them.
func WriteLibrary
func WriteLibrary(dir string) (string, error)WriteLibrary writes the embedded native library into dir and returns its path, for an application that signs the library and ships it, then names it in LoadConfig.Library (on macOS also as Contents/Frameworks/libkolovpx.dylib, which is loaded without configuration). It errors when no library is embedded.
Types
type Codec
type Codec intCodec is a video codec.
const (
VP8 Codec = iota + 1
VP9
)Supported codecs.
func (Codec) String
func (c Codec) String() stringString returns the codec name.
type Config
type Config struct {
Codec Codec
// Width and Height of every frame: 1 to 16383 for VP8, 1 to 65536 for
// VP9.
Width, Height int
// FrameRate is the nominal rate, DefaultFrameRate if zero; otherwise
// kolovideo.MinFrameRate to kolovideo.MaxFrameRate.
FrameRate float64
// Bitrate is the target in bits per second, DefaultBitsPerPixel times
// pixels times FrameRate if zero.
Bitrate int
// KeyframeInterval is the time between keyframes, measured on frame
// timestamps; DefaultKeyframeInterval if zero, negative for keyframes
// only on request.
KeyframeInterval time.Duration
}Config sets up an Encoder. Zero fields take defaults.
type Decoder
type Decoder struct {
// contains filtered or unexported fields
}Decoder decodes packets into I420 frames. Its methods are safe for concurrent use.
func NewDecoder
func NewDecoder(cfg DecoderConfig) (*Decoder, error)NewDecoder creates a decoder. Call Close to free it.
func (Decoder) Close
func (d *Decoder) Close() errorClose frees the decoder. It is idempotent.
func (Decoder) Decode
func (d *Decoder) Decode(p Packet) (*kolovideo.Frame, error)Decode decodes one packet. It returns a nil frame and nil error when the packet holds no frame to show. Frames may change size at keyframes. After any error, packets are refused with ErrNeedKeyframe until the next keyframe.
type DecoderConfig
type DecoderConfig struct {
Codec Codec
// MaxPixels is the largest frame accepted, checked in each frame
// header before decoding; DefaultMaxPixels if zero.
MaxPixels int
// MaxPacketBytes bounds a packet, checked before it is copied;
// DefaultMaxPacketBytes if zero, at most 256 MiB.
MaxPacketBytes int
}DecoderConfig sets up a Decoder.
type Encoder
type Encoder struct {
// contains filtered or unexported fields
}Encoder encodes I420 frames. Its methods are safe for concurrent use.
func NewEncoder
func NewEncoder(cfg Config) (*Encoder, error)NewEncoder creates an encoder. Call Close to free it.
func (Encoder) Close
func (e *Encoder) Close() errorClose frees the encoder. It is idempotent.
func (Encoder) Config
func (e *Encoder) Config() ConfigConfig returns the configuration with defaults filled in.
func (Encoder) Encode
func (e *Encoder) Encode(f *kolovideo.Frame, forceKeyframe bool) ([]Packet, error)Encode compresses one frame of the configured size. It returns no packet when rate control drops the frame. A keyframe is made when forceKeyframe is set, for the first frame, and every KeyframeInterval.
func (Encoder) SetBitrate
func (e *Encoder) SetBitrate(bps int) errorSetBitrate changes the target bitrate in bits per second, effective from the next frame.
type EngineKind
type EngineKind intEngineKind names a libvpx engine.
const (
EngineAuto EngineKind = iota
EngineNative
EnginePortable
)Engines. The native engine is the prebuilt library, the portable engine is the same libvpx translated to Go.
func Engine
func Engine() EngineKindEngine returns the engine in use, loading the default one if needed. It returns EngineAuto if no engine can run.
func (EngineKind) String
func (k EngineKind) String() stringString returns the engine name.
type Limits
type Limits struct {
Engine EngineKind
Encode RealTime
Decode RealTime
}Limits are the largest modes the engine in use handled in real time.
func Capabilities
func Capabilities() LimitsCapabilities returns reference limits, not a run-time probe: the largest encode and decode modes measured as real time for the engine in use, so a caller can cap resolution. Real time means a mean time per frame below the 33.3 ms period for a camera-like stream, on an Apple M5: the native engine reaches 1920x1080 at 30 fps for encode and decode, VP8 and VP9; the portable engine encodes up to 640x480 at 30 fps and decodes up to 1920x1080. A slower CPU reaches less. It returns the zero Limits if no engine can run.
type LoadConfig
type LoadConfig struct {
// Engine is EngineAuto if zero: native, with the portable engine as the
// fallback when the native library cannot load.
Engine EngineKind
// CacheDir is the root of the native library cache, os.UserCacheDir()/kolo-video
// if empty.
CacheDir string
// Library is the path of a libkolovpx the application ships and signs
// itself, loaded instead of the embedded one. It is not checked against a
// hash. An altered library that still carries a valid signature can be
// killed by the operating system when it is loaded, which cannot be caught.
Library string
}LoadConfig selects and sets up the engine. Library and CacheDir apply to the native engine only.
type Packet
type Packet struct {
Data []byte
Keyframe bool
// Timestamp and Duration come from the frame and pass through the
// decoder unchanged.
Timestamp time.Duration
Duration time.Duration
}Packet is one encoded frame.
type RealTime
type RealTime struct {
Width, Height int
FrameRate float64
}RealTime is a picture size and frame rate.
type Runtime
type Runtime struct {
Engine EngineKind
Version string // libvpx version, such as "v1.16.0"
// NativeErr is why the native engine is not in use, nil if it is. Check it
// after Load with EngineAuto: ErrNativeIntegrity and ErrNativeSignature are
// not reported any other way.
NativeErr error
}Runtime describes the engine in use.
func Load
func Load(cfg LoadConfig) (Runtime, error)Load selects the libvpx engine. It is optional: NewEncoder, NewDecoder, Engine, LoadError and Capabilities load the default configuration when nothing is loaded (on macOS and Windows that extracts and loads the native library), so call Load first to choose another one. Once an engine is active, Load with another configuration returns kolovideo.ErrInvalidConfig and the first engine stays. With Engine set to EngineNative, a native load failure returns ErrUnavailable wrapping the cause and nothing is loaded.
With EngineAuto a failed native load is not an error: the portable engine runs and Load returns nil. Read Runtime.NativeErr (or LoadError) to learn why. ErrNativeIntegrity and ErrNativeSignature there mean that the embedded bytes or a bundle library did not verify, which is corruption or tampering, not the plain absence of a native library (ErrNativeUnsupported); log them.
Package h264
Package h264 encodes and decodes H.264 video with the codec of the operating system: VideoToolbox on macOS, the Media Foundation H.264 transforms on Windows desktop editions. It contains no H.264 encoder or decoder and needs no installed library, compiler or cgo. Where the system has no usable codec, Capabilities and the constructors report why with an *UnavailableError.
Packets are Annex B access units with 4-byte start codes, one picture each, holding only slices, SPS and PPS; every keyframe starts with its SPS and PPS. Encoders produce Constrained Baseline, Main or High within level 5.1, without B-frames, one packet per frame. Decoders check every packet in Go before the system sees it and refuse streams outside those profiles and levels.
Experimental: the API may change before v1.0.0.
Constants
const (
// DefaultFrameRate is the rate assumed for frames without a Duration:
// kolovideo.DefaultFrameRate.
DefaultFrameRate = kolovideo.DefaultFrameRate
// DefaultBitsPerPixel sets the default bitrate: about 4.4 Mbit/s for
// 1080p at 30 fps.
DefaultBitsPerPixel = 0.07
// DefaultKeyframeInterval starts a new group of pictures every 2 s.
DefaultKeyframeInterval = 2 * time.Second
// DefaultMaxPixels bounds the coded size of decoded pictures: 36864
// macroblocks, the level 5.1 MaxFS.
DefaultMaxPixels = 9437184
// DefaultMaxPacketBytes bounds a packet, as kolo-media's playout MaxBytes.
DefaultMaxPacketBytes = 16 << 20
// MaxBitrate is the level 5.1 MaxBR in bits per second, in the Baseline
// and Main unit.
MaxBitrate = 240000000
)Defaults and bounds. The level limits are those of level 5.1 (ITU-T H.264 Table A-1), the highest level every engine documents.
Variables
Types
type Config
type Config struct {
// Width and Height are even, 48 to 4096, at most 36864 macroblocks, and
// macroblocks times FrameRate at most 983040 (level 5.1).
Width, Height int
// FrameRate is kolovideo.MinFrameRate to MaxFrameRate, DefaultFrameRate
// if zero.
FrameRate float64
// Bitrate is the target in bits per second, 1 to MaxBitrate;
// DefaultBitsPerPixel times pixels times FrameRate if zero.
Bitrate int
// KeyframeInterval is the time between keyframes on frame timestamps,
// DefaultKeyframeInterval if zero, negative for keyframes on request only.
KeyframeInterval time.Duration
// Profile is ConstrainedBaseline if zero.
Profile Profile
}Config sets up an Encoder. Zero fields take defaults.
type Decoder
type Decoder struct {
// contains filtered or unexported fields
}Decoder decodes H.264 packets with the system codec. Its methods are safe for concurrent use. Every packet is checked in Go before the system sees it, and pictures come out in decoding order.
func NewDecoder
func NewDecoder(cfg DecoderConfig) (*Decoder, error)NewDecoder creates a decoder. Call Close to free it.
func (Decoder) Close
func (d *Decoder) Close() errorClose frees the decoder. It is idempotent. It waits for a Decode in progress.
func (Decoder) Decode
func (d *Decoder) Decode(p Packet) (*kolovideo.Frame, error)Decode returns the picture in p, or nil and nil when p holds no picture. After any error, packets are refused with ErrNeedKeyframe until an IDR with its SPS and PPS.
func (Decoder) Hardware
func (d *Decoder) Hardware() boolHardware reports whether the system decodes this stream in hardware.
func (Decoder) SPS
func (d *Decoder) SPS() (s SPS, ok bool)SPS returns the active sequence parameter set, ok false before the first keyframe.
type DecoderConfig
type DecoderConfig struct {
// MaxPixels bounds the coded picture size, macroblocks times 256 before
// cropping, which is what the system decoder allocates: 1920x1080 is
// coded as 1920x1088. DefaultMaxPixels if zero, at most DefaultMaxPixels.
MaxPixels int
// MaxPacketBytes bounds a packet, DefaultMaxPacketBytes if zero, at most
// 256 MiB so that bit offsets fit in 32 bits.
MaxPacketBytes int
}DecoderConfig sets up a Decoder. Zero fields take defaults.
type Encoder
type Encoder struct {
// contains filtered or unexported fields
}Encoder encodes I420 frames with the system codec. Its methods are safe for concurrent use.
func NewEncoder
func NewEncoder(cfg Config) (*Encoder, error)NewEncoder creates an encoder. On Windows it also checks the profile; call Close to free it.
func (Encoder) Close
func (e *Encoder) Close() errorClose frees the encoder. It is idempotent.
func (Encoder) Config
func (e *Encoder) Config() ConfigConfig returns the configuration with defaults filled in.
func (Encoder) Encode
func (e *Encoder) Encode(f *kolovideo.Frame, forceKeyframe bool) ([]Packet, error)Encode returns the access unit for f, or none when the system encoder dropped the frame. A keyframe is made for the first frame, every KeyframeInterval and when forceKeyframe is set.
func (Encoder) Hardware
func (e *Encoder) Hardware() boolHardware reports whether the system encodes this stream in hardware.
func (Encoder) SetBitrate
func (e *Encoder) SetBitrate(bps int) errorSetBitrate changes the target bitrate from the next frame.
type Limits
type Limits struct{ Encode, Decode Support }Limits is the encode and decode support of this system.
func Capabilities
func Capabilities() LimitsCapabilities probes the system codec once per process and returns the result.
type Packet
type Packet struct {
Data []byte
// Keyframe is set by the encoder; Decode ignores it and reads the NAL types.
Keyframe bool
// Timestamp and Duration pass through unchanged.
Timestamp time.Duration
Duration time.Duration
}Packet is one access unit: Annex B, 4-byte start codes, NAL types 1, 5, 7 and 8 only, one picture. A keyframe starts with an SPS and a PPS, then an IDR picture.
type Profile
type Profile intProfile is an H.264 profile.
const (
ConstrainedBaseline Profile = iota + 1 // profile_idc 66 with constraint_set1
Main // profile_idc 77
High // profile_idc 100
)Profiles.
func (Profile) String
func (p Profile) String() stringString returns the profile name.
type SPS
type SPS struct {
ID uint32
ProfileIDC, Constraints, Level uint8
// Width and Height are the picture size after cropping.
Width, Height int
MaxRefFrames uint32
FullRange bool
// Matrix is matrix_coefficients, 2 (unspecified) without a colour
// description.
Matrix uint8
}SPS holds the sequence parameter set fields this package checks.
func ParseKeyframe
func ParseKeyframe(packet []byte) (SPS, error)ParseKeyframe returns the first SPS of an Annex B keyframe packet, checked as Decoder checks it with the default bounds. It never panics.
func (SPS) Codec
func (s SPS) Codec() stringCodec returns the codec string, such as "avc3.42c01f" (RFC 6381 3.3).
func (SPS) Profile
func (s SPS) Profile() (p Profile, ok bool)Profile classifies the SPS by RFC 6184 Table 5; ok is false for other profiles.
type Support
type Support struct {
Available bool
// Hardware reports that the system has a hardware codec for this direction.
Hardware bool
// Engine is "VideoToolbox" or "Media Foundation".
Engine string
// Profiles lists what the engine produces (encode) or accepts (decode). On
// Windows the encode list is probed at run time; on macOS it comes from
// the API and each encoder confirms its profile at the first keyframe.
Profiles []Profile
// MaxWidth and MaxHeight are the limits the system vendor documents, 0
// when it documents none. No system documents or reports a frame-rate
// limit, so none is reported.
MaxWidth, MaxHeight int
// Err says why Available is false.
Err error
}Support describes one direction of the system codec.
Good to know
How a frame travels
- A camera, a converter or your code makes a
Frame: I420, BT.601, limited range. vpxencodes it as VP8 or VP9, andh264as H.264 when the system has a codec.- Each encoder returns one packet for a frame. Your application sends it.
- The receiver decodes the packet with the matching decoder, which checks it first.
- The decoder returns a
Frame, or an error and a wait for a keyframe.
Frames and capture
- A
Frameholds 8-bitY,UandVplanes, plusTimestampandDuration.NewFrame(w, h)allocates a black frame. Every function that takes a frame callsValidateand returnsErrInvalidFrameinstead of panicking. FromNV12,FromYUY2,FromUYVY,FromBGRAandFromRGBAconvert a raw picture with any stride to a new frame.Frame.RGBA()converts back, andRenderer{Scale: Fit}draws into an*image.RGBAwith letterboxing (Fillcrops).Devices()lists cameras without opening one.OpenCapturepicks the supported mode nearest to the request andCapture.Readreturns frames. When the reader is slow at mostBufferframes are kept and the oldest is dropped. A disconnect, a capture failure, or no frame forStallTimeoutends the capture withErrDeviceLostafter the buffered frames are read.- The backends are AVFoundation on macOS (through purego), Media Foundation on Windows (through COM) and V4L2 on Linux (pure Go, including Motion-JPEG). On macOS
OpenCapturereturnsErrPermissionuntil the user grants access, and never asks:RequestAccessasks. Frames are always converted to BT.601 limited range I420. - Elsewhere,
Devices,RequestAccessandOpenCapturereturnErrUnsupported: the platform has no capture backend. Frames, conversion, rendering and the codecs still work.
VP8 and VP9
vpx.NewEncoderandvpx.NewDecoderwork in real time: one pass, constant bitrate, no lag, error resilience and frame dropping. The first frame and one frame perKeyframeInterval(2 s by default) are keyframes,Encode(f, true)forces one, andSetBitratechanges the target from the next frame.Encodereturns no packet when rate control drops a frame.- The decoder checks every frame header against
MaxPixelsbefore libvpx allocates anything (ErrTooLarge), decodes from its own copy of the packet, and refuses packets withErrNeedKeyframefrom the start and after any error until a keyframe arrives. vpx.DecoderConfig.MaxPacketBytesrefuses a packet above it withErrTooLarge, before the packet is copied: 16 MiB by default, at most 256 MiB (vpx.DefaultMaxPacketBytes, the same bound as inh264). The decoder keeps at most 1 MiB of packet buffer between calls.- Two engines run the same libvpx 1.16.0, and
vpx.Loadmakes the choice once per process. The native engine is a prebuilt library per target, loaded without cgo; the portable engine is the same code translated to Go.vpx.Capabilities()lists the largest real-time modes of the engine in use: native 1920x1080 at 30 fps for both directions, portable 640x480 at 30 fps for encode and 1920x1080 at 30 fps for decode. - On Linux the default binary is static and uses the portable engine; build with
-tags kolovideo_nativeon glibc 2.28 or later for the native one. A signed macOS app needs one more step for the native engine, and Windows Smart App Control may refuse the embedded DLL. In both cases the portable engine runs andvpx.LoadError()says why.
H.264
h264.Capabilities()says whether the system can encode and decode H.264, whether the codec is hardware, which engine it is (VideoToolboxorMedia Foundation) and which profiles it offers. Where it is absent, the error is an*UnavailableErrorthat matchesh264.ErrUnavailableand names the platform and the reason.- kolo-video contains no H.264 code. VideoToolbox on macOS and the Media Foundation H.264 transforms on Windows desktop editions do the coding; nothing is installed, bundled or downloaded. On Linux and on every other platform there is no engine, and
h264does not importvpxor any native library. - Profiles are Constrained Baseline (the default), Main and High, up to level 5.1, progressive 4:2:0 at 8 bits, with no B-frames. Sizes are even and from 48 to 4096 on each side. The encoder returns one packet for a frame, inside the same
Encodecall. - A packet is one Annex B access unit with 4-byte start codes. Every keyframe starts with its SPS and PPS, so a receiver can start at any keyframe.
h264.ParseKeyframe(packet)andSPS.Codec()give the codec string, such asavc3.42c01f. The string comes from the first keyframe and is advisory. - The recommended pair for a sender is VP8 always and H.264 when
Capabilitiessays it can encode: two renditions of one source. A receiver that cannot decode H.264 takes VP8. Nothing falls back by itself: your application checksCapabilitiesand chooses. kolo-media v0.1.0 does not carry H.264: its catalog accepts VP8 and VP9 for video. - The decoder checks every packet in Go before the system sees it, so a stream outside the supported profiles, levels and tools is
ErrUnsupportedStream, a damaged packetErrCorrupt, an oversized oneErrTooLarge, and after any error it returnsErrNeedKeyframeuntil a keyframe arrives. SetBitratechanges the target from the next frame. The macOS hardware encoder followed a change from 1.9 to 0.5 Mbit/s to within 1 percent; the macOS software encoder was not converged (988 kbit/s for a 500 kbit/s target).
Platforms
| Platform | VP8 and VP9 | H.264 (OS codec) | Camera backend |
|---|---|---|---|
| macOS arm64 | native, portable as fallback | VideoToolbox, hardware on the Mac that ran it | AVFoundation through purego |
| macOS amd64 | same | VideoToolbox | same |
| Windows arm64 | native, portable as fallback | Media Foundation, software | Media Foundation through COM |
| Windows amd64 | same | same | same |
| Linux amd64, arm64 | portable; native with -tags kolovideo_native | none: ErrUnavailable | V4L2 in pure Go |
| Other | portable | none: ErrUnavailable | none: ErrUnsupported |
What ran for H.264:
- macOS arm64, an Apple M5: round trips at 640x480, 1280x720 and 1920x1080 in Constrained Baseline, Main and High decoded every frame with luma PSNR of at least 42.69 dB, with the hardware encoder and decoder. The software path (hardware disabled) and damaged input were tested too, and the binary ran ad-hoc signed with the hardened runtime. Under Rosetta, amd64 passed the same tests. No Intel Mac was used.
- Windows 11 ARM64 in a VM: the same round trips with luma PSNR of at least 42.86 dB, one picture for every packet from the first, software transforms only. The VM's x64 emulation passed the functional tests. Native Windows amd64 was not run.
- The macOS and Windows encoders each produced streams that the other system's decoder decoded.
- Linux, in Debian and Alpine containers: the absence message, a static binary and the parsers. No V4L2 stateful codec device exists on any test machine, so detection never ran against a real driver.
- Not validated: Linux encode and decode (not provided), VA-API, Windows hardware transforms, N editions and Windows Server, a Developer ID signed and notarized app, a real camera feeding the H.264 encoder, and browsers and phones.
A camera stream was run on one Mac's built-in camera only. Linux and Windows capture never ran on a real camera.
Limits
A value outside a range returns ErrInvalidConfig or ErrInvalidFrame. The full tables are in Limits.
- Frames: 1 to 65536 on each side.
CaptureConfig.MaxPixelsandDecoderConfig.MaxPixelsbound pictures, 8,912,896 pixels for VP8 and VP9 and 9,437,184 for H.264. - VP8 frames are 1 to 16383 on each side, and the largest VP9 frame is 65536.
- H.264 encoder: even sizes from 48 to 4096, at most 36864 macroblocks, and macroblocks times frame rate at most 983040. That is 1920x1080 up to 120 fps and 3840x2160 up to 30 fps. Bitrate 1 to 240,000,000 bit/s.
- H.264 decoder: profiles 66, 77 and 100, levels 1 to 5.1, at most 16 MiB per packet by default, and one picture per packet.
- The portable engine uses at most 1 GiB of memory per instance.
Security
- Every frame header is checked in Go against
MaxPixelsbefore libvpx allocates anything, and the decoder works on its own copy of the packet. - For H.264, only NAL unit types 1, 5, 7 and 8 reach the system. A parameter set is accepted only in a keyframe packet, and the level, the picture size and the reference frames are checked against Table A-1 of the H.264 specification.
- On Windows, damaged slices that pass the checks are concealed by the system and returned as pictures, so
Decodedoes not report them, and a crash inside the decoder ends the application. A hang inside the system codec blocksDecodeandClose. - The embedded native libraries are checked against their SHA-256 before every load, and the cache directory and every directory above it are checked for ownership and permissions. A signed macOS app can refuse the library, and the portable engine runs.
- The library makes no network connection, starts no process and uses no cgo.
Status
- Done: frames and conversions, camera capture on macOS, Windows and Linux, VP8 and VP9 on two engines, and H.264 through VideoToolbox and Media Foundation, with fuzz targets and tests.
- Tested: H.264 ran on one Mac and in one Windows VM, as the platforms list says. A camera stream ran on one Mac only.
- Not done: H.264 on Linux, and kolo-media does not carry H.264 yet.
- Experimental. The API may change before v1.0.0.
Errors
| Error | Message |
|---|---|
ErrClosed | kolovideo: closed |
ErrDeviceLost | kolovideo: device lost |
ErrInvalidConfig | kolovideo: invalid config |
ErrInvalidFrame | kolovideo: invalid frame |
ErrNoDevice | kolovideo: no such device |
ErrPermission | kolovideo: camera access not granted |
ErrUnsupported | kolovideo: not supported on this platform |
ErrUnsupportedFormat | kolovideo: unsupported format |
vpx.ErrCodec | kolovideo/vpx: codec error |
vpx.ErrCorrupt | kolovideo/vpx: corrupt packet |
vpx.ErrNativeABI | native libvpx ABI mismatch |
vpx.ErrNativeCache | native libvpx cache is unusable |
vpx.ErrNativeIntegrity | embedded native libvpx does not match its checksum |
vpx.ErrNativeSignature | native libvpx code signature is not valid |
vpx.ErrNativeUnsupported | no native libvpx |
vpx.ErrNeedKeyframe | kolovideo/vpx: waiting for a keyframe |
vpx.ErrTooLarge | kolovideo/vpx: packet or frame too large |
vpx.ErrUnavailable | kolovideo/vpx: libvpx unavailable |
h264.ErrCodec | kolovideo/h264: system codec error |
h264.ErrCorrupt | kolovideo/h264: corrupt packet |
h264.ErrNeedKeyframe | kolovideo/h264: waiting for a keyframe |
h264.ErrProfile | kolovideo/h264: profile not available from the system encoder |
h264.ErrTooLarge | kolovideo/h264: packet or picture too large |
h264.ErrUnavailable | kolovideo/h264: no H.264 codec on this system |
h264.ErrUnsupportedStream | kolovideo/h264: unsupported stream |
Every error starts with kolovideo:, or kolovideo/vpx: or kolovideo/h264: for the codec packages, and wraps a sentinel, so errors.Is works. ErrUnsupported means this platform has no capture backend, and the error names GOOS and GOARCH. h264.ErrUnavailable comes as an *UnavailableError that says why the system has no codec.
References
- RFC 6386, VP8 Data Format and Decoding Guide (Informational, November 2011).
- Takes: the frame tag and the key frame header (section 9.1).
- Not implemented: nothing left out.
- RFC 7741, RTP Payload Format for VP8 Video (Proposed Standard, March 2016).
- Takes: the 90 kHz timebase (sections 4.1 and 6.1).
- Not implemented: the RTP packetization.
- ITU-T H.264, Advanced video coding for generic audiovisual services (06/2026, in force).
- Takes: the Annex B byte stream, the level limits of Table A-1 and the SPS fields.
- Not implemented: B slices, interlace, profiles other than 66, 77 and 100.
- RFC 6381, The 'Codecs' and 'Profiles' Parameters for "Bucket" Media Types (Proposed Standard, August 2011).
- Takes: the
avc3.codec string (section 3.3). - Not implemented: nothing left out.
- Takes: the
- RFC 6184, RTP Payload Format for H.264 Video (Proposed Standard, May 2011).
- Takes: the profile classification (Table 5).
- Not implemented: the RTP packetization.
- ITU-T T.81, Information technology - Digital compression and coding of continuous-tone still images - Requirements and guidelines (09/92, in force).
- Takes: the default Huffman tables for MJPEG cameras (Annex K.3).
- Not implemented: nothing left out.
- ITU-R BT.601-7, Studio encoding parameters of digital television for standard 4:3 and wide screen 16:9 aspect ratios (03/2011, in force).
- Takes: the I420 matrix, limited range.
- Not implemented: nothing left out.