kolo docsbeta

kolo-files

An encrypted tree of files and folders with its whole history. Check out any earlier version, compare two, and keep only the files a device needs.

On this page

kolo-files keeps folders, names and versions on top of kolo-blobs, with names in Unicode Normalization Form C (UAX #15). A Volume is a tree of files and folders kept in two kolo-store logs of one owner: a metadata log with one op per change (create a folder, write a file, move, delete) and a content log of kolo-blobs objects. The tree is rebuilt by replaying the metadata log, and every op is a version: Snapshot checks out any earlier tree and Diff compares two. Each file has its own key, kept inside its create op, so a kolo-pin pinning peer that holds both logs sees neither names, keys nor content. Volume and Snapshot implement io/fs, so fs.WalkDir, fs.ReadFile and fs.Glob work on them; to write one to disk, follow Writing a volume to disk, not os.CopyFS. A reader keeps a replica of the metadata log and reads content from a content replica or, entry by entry, from another peer; marks choose which files a device keeps. The Go package is named kolofiles.

All steps

Ops, nodes and paths

No tree is kept on disk. It is rebuilt by replaying the metadata log from its first entry.

  1. Every change is one op in the metadata log. Entry 0 is VOLUME, which names the content log. REMOVE and PURGE, for deletes, are not drawn.
  2. An op names a node, not a path. Every file and folder gets a random 16-byte node ID when it is created. The root is the all-zero ID.
  3. A move is one op that sets a node's parent and name. The node ID stays, so a moved folder takes its subtree along and the history follows the file.
  4. Replaying the log builds the tree. A path follows io/fs: slash-separated, names in NFC with no control characters, 765 bytes a name and 2048 names deep at most by default.

Install

go get github.com/ciranda-dev/kolo-files

Builds with CGO_ENABLED=0 for a pure Go binary. Share files between two terminals shares a folder and each new version of it with another peer, step by step.

Quick start

This program keeps a small tree in a volume on disk, in a temporary directory. It lists the tree, checks out an earlier version, compares two versions and deletes a file.

main.go
package main

import (
	"context"
	"fmt"
	"io/fs"
	"log"
	"os"
	"path/filepath"
	"strings"

	kolofiles "github.com/ciranda-dev/kolo-files"
	koloid "github.com/ciranda-dev/kolo-id"
	kolostore "github.com/ciranda-dev/kolo-store"
)

func main() {
	dir, err := os.MkdirTemp("", "kolo-files")
	if err != nil {
		log.Fatal(err)
	}
	defer os.RemoveAll(dir)
	ctx := context.Background()

	// Two kolo-store logs of one owner on disk: one for names and versions, one for content.
	db, err := kolostore.OpenPebble(filepath.Join(dir, "store"))
	if err != nil {
		log.Fatal(err)
	}
	st := kolostore.New(db)
	defer st.Close()
	me, err := koloid.Generate()
	if err != nil {
		log.Fatal(err)
	}
	meta, err := st.Create(me)
	if err != nil {
		log.Fatal(err)
	}
	content, err := st.Create(me)
	if err != nil {
		log.Fatal(err)
	}
	v, err := kolofiles.Create(meta, content)
	if err != nil {
		log.Fatal(err)
	}
	defer v.Close()

	must := func(err error) {
		if err != nil {
			log.Fatal(err)
		}
	}
	// Every call below is one op, so every call makes a version.
	must(v.Mkdir("notes"))
	must(v.WriteFile(ctx, "notes/todo.txt", strings.NewReader("buy bread")))
	first := v.Len()
	must(v.WriteFile(ctx, "notes/todo.txt", strings.NewReader("buy bread and milk")))
	must(v.Move("notes/todo.txt", "notes/shopping.txt"))
	must(v.WriteFile(ctx, "draft.txt", strings.NewReader("not for keeping")))

	list := func(name string, fsys fs.FS) {
		fmt.Println(name)
		must(fs.WalkDir(fsys, ".", func(p string, d fs.DirEntry, err error) error {
			if err != nil || d.IsDir() {
				return err
			}
			b, err := fs.ReadFile(fsys, p)
			fmt.Printf("  %s: %s\n", p, b)
			return err
		}))
	}
	list("now:", v)
	old, err := v.Snapshot(first)
	must(err)
	list(fmt.Sprintf("at version %d:", first), old)

	changes, err := v.Diff(first, v.Len())
	must(err)
	for _, c := range changes {
		fmt.Printf("changed: %q -> %q, content modified: %v\n", c.Old, c.New, c.Modified)
	}

	// Delete destroys the name, the key and every object of the file; no version shows it again.
	must(v.Delete("draft.txt"))
	_, err = v.Stat("draft.txt")
	fmt.Println("draft.txt after delete:", err)
}

It prints:

Text
now:
  draft.txt: not for keeping
  notes/shopping.txt: buy bread and milk
at version 3:
  notes/todo.txt: buy bread
changed: "" -> "draft.txt", content modified: false
changed: "notes/todo.txt" -> "notes/shopping.txt", content modified: true
draft.txt after delete: kolofiles: stat draft.txt: file does not exist

Create writes entry 0, so an empty volume is version 1. Mkdir makes version 2 and the first WriteFile version 3, which is why first is 3. Between version 3 and now there are two changes: draft.txt was added, and todo.txt was renamed and edited, which is one change because the node is the same.

API reference

The API reference is generated from the Go source and is in English.

Package kolofiles · github.com/ciranda-dev/kolo-files · v0.1.0

Package kolofiles keeps a tree of files and folders with its whole history in two Kolo logs: names, folders and versions on top of kolo-blobs.

A Volume is a metadata log of ops (one entry per change: mkdir, create, write, move, remove, purge) and a content log of kolo-blobs objects, both kolo-store logs of one owner. The tree is rebuilt by replaying the metadata log; every op is a version, so any earlier tree can be checked out with Snapshot and compared with Diff. Each file has its own kolo-blobs key, kept inside its create op, so a pinning peer that holds both logs sees neither names, keys nor content. Delete destroys every name, key and object of a file or folder with kolo-log's deletion. Volume and Snapshot implement io/fs. A reader opens a replica of the metadata log and reads content from a content replica or, entry by entry, from another peer; marks choose per path which files a device keeps.

Experimental: the API may change before v1.0.0.

Constants

const (
	// DefaultMaxDepth is the default of WithMaxDepth: the deepest tree one Linux path argument can name.
	DefaultMaxDepth = 2048
	// DefaultMaxNameLength is the default of WithMaxNameLength, in bytes: 255 UTF-16 units of at most 3 bytes each.
	DefaultMaxNameLength = 765
)

Variables

var (
	// ErrNotDir means an element of a path that should be a directory is a file.
	ErrNotDir = errors.New("kolofiles: not a directory")
	// ErrIsDir means a file operation on a directory.
	ErrIsDir = errors.New("kolofiles: is a directory")
	// ErrLimit means a path or a log over MaxDepth or MaxNameLength, or an op to append over the largest entry kolo-sync carries.
	ErrLimit = errors.New("kolofiles: over a bound")
	// ErrReadOnly means a write on a reader.
	ErrReadOnly = errors.New("kolofiles: read-only volume")
	// ErrClosed means a call on a closed Volume, or on a Snapshot or file of one.
	ErrClosed = errors.New("kolofiles: volume closed")
	// ErrCorrupt means the log is not a kolo-files log, or an op or an object table breaks a rule of the format.
	ErrCorrupt = errors.New("kolofiles: corrupt volume")
	// ErrUnsafeName means LocalPath refused a name the host reserves for a device (some versions open the device
	// instead) or cannot create.
	ErrUnsafeName = errors.New("kolofiles: name not allowed on this system")
	// ErrInvalidArgument means a nil, out of range or mismatched argument, or a mark where marks do not apply.
	ErrInvalidArgument = errors.New("kolofiles: invalid argument")
)

Functions

func ContentLog

func ContentLog(meta *kolostore.Replica) (kololog.LogID, error)

ContentLog returns the ID of the content log that the volume in meta names in its first entry.

func LocalPath

func LocalPath(name string) (string, error)

LocalPath converts a path of a volume into a path for the host operating system. On Windows it refuses with ErrUnsafeName a name Windows reserves for a device (some versions open the device instead) or cannot create: CON, NUL.txt, COM1 and the other device names, one of < > : " | ? * \, a trailing dot or space, a control character, or more than 255 UTF-16 units. A path that is not an io/fs path, or holds a NUL, is fs.ErrInvalid. Errors are *fs.PathError.

LocalPath checks names only and never looks at the disk. To write a volume to disk, create each folder and file under an os.Root with Mkdir and O_CREATE|O_EXCL, so a name aliasing another (case, a Windows 8.3 short name) fails with fs.ErrExist instead of overwriting, and no link leads outside the directory, as the example does.

Types

type Change

type Change struct {
	Node     NodeID
	Old, New string // the path at the older and at the newer version; empty where the node is not in the tree
	Modified bool   // the file's content differs
	Deleted  bool   // the node was deleted between the versions; its paths are destroyed with it
}

Change is one node that differs between two versions, as Diff reports it.

type Mark

type Mark uint8

Mark says what a reader device keeps of a file; a directory's mark applies to everything below it.

const (
	// MarkInherit clears a mark: the node takes its nearest marked ancestor's mark.
	MarkInherit Mark = iota
	// MarkFetchOnDemand fetches only the objects a read needs, when it needs them, and keeps none. The root's default.
	MarkFetchOnDemand
	// MarkKeepLocal keeps every object of the file's current version on the device, so it reads offline.
	MarkKeepLocal
)
func (Mark) String
func (m Mark) String() string

String returns the mark's name.

type NodeID

type NodeID [16]byte

NodeID identifies a file or directory for its whole life; moves and renames keep it. The root is the zero ID.

func (NodeID) String
func (id NodeID) String() string

String returns the ID as lowercase hex.

type NodeInfo

type NodeInfo struct {
	ID      NodeID
	Version uint64       // the version in which the node last changed: the length of the log after its last op
	Blob    koloblobs.ID // the blob of a file's content; zero for a directory
}

NodeInfo is what fs.FileInfo.Sys returns for a file or directory of a Volume or Snapshot.

type Option

type Option func(*options)

Option configures Create, Open and the readers.

func WithMaxDepth
func WithMaxDepth(n int) Option

WithMaxDepth sets how many names below the root a path may have, at least 1; the default is DefaultMaxDepth.

func WithMaxNameLength
func WithMaxNameLength(n int) Option

WithMaxNameLength sets the longest name in bytes after NFC, 1 to the largest op; the default is DefaultMaxNameLength.

type Snapshot

type Snapshot struct {
	// contains filtered or unexported fields
}

Snapshot is the tree at one version of a volume; it is immutable, safe for concurrent use and needs no closing. Its reads go through the volume, so they fail with ErrClosed once the volume is closed.

func (Snapshot) Open
func (s *Snapshot) Open(name string) (fs.File, error)

Open opens a file or directory of the snapshot for reading, as Volume.Open.

func (Snapshot) OpenContext
func (s *Snapshot) OpenContext(ctx context.Context, name string) (fs.File, error)

OpenContext is Open with a context that bounds every content read of the file it returns.

func (Snapshot) ReadDir
func (s *Snapshot) ReadDir(name string) ([]fs.DirEntry, error)

ReadDir lists a directory of the snapshot, as Volume.ReadDir.

func (Snapshot) Stat
func (s *Snapshot) Stat(name string) (fs.FileInfo, error)

Stat describes a file or directory of the snapshot, as Volume.Stat.

func (Snapshot) Version
func (s *Snapshot) Version() uint64

Version returns the snapshot's version: the length of the metadata log it was replayed from.

type Stats

type Stats struct {
	Ops            uint64 // ops appended to the metadata log
	Replayed       uint64 // entries replayed by opens, refreshes, snapshots and diffs
	BytesWritten   uint64 // file content read by WriteFile
	BytesRead      uint64 // file content returned by files
	ObjectsFetched uint64 // objects from the content backend
	ObjectsLocal   uint64 // objects served by the local backend of a reader
}

Stats are a Volume's totals since it was opened.

type Volume

type Volume struct {
	// contains filtered or unexported fields
}

Volume is a tree of files and folders kept in a metadata log and a content log. It is safe for concurrent use. A Volume from Create or Open writes; one from a reader constructor reads a replica.

func Create
func Create(meta, content *kolostore.Log, opts ...Option) (*Volume, error)

Create starts a new volume in meta and content, two empty logs of one owner with the kololog.Bytes encoding, and takes them over: Close closes them. If Create fails, the logs stay open.

func Open
func Open(meta, content *kolostore.Log, opts ...Option) (*Volume, error)

Open rebuilds the volume in meta, your own metadata log, with content, the log its first entry names, and takes them over. It finishes a Delete that a crash interrupted. If Open fails, the logs stay open.

func OpenFetchReader
func OpenFetchReader(meta *kolostore.Replica, fetch koloblobs.FetchFunc, local kolostore.Backend, opts ...Option) (*Volume, error)

OpenFetchReader is OpenReader with content read from another peer, one content log entry per fetch call.

func OpenReader
func OpenReader(meta *kolostore.Replica, content koloblobs.Backend, local kolostore.Backend, opts ...Option) (*Volume, error)

OpenReader rebuilds a read-only volume from meta, a replica of the owner's metadata log, and takes it over. content holds the objects, local (may be nil) the marks and kept objects. If it fails, the replica stays open.

func (Volume) Close
func (v *Volume) Close() error

Close closes the volume and the logs or the replica it took over, not the backends. It waits for a running WriteFile, Prefetch or Refresh, for example on a stalled fetch, until that call's context ends. A second Close returns nil.

func (Volume) Delete
func (v *Volume) Delete(name string) error

Delete removes the file or directory name with everything below it and destroys every name, file key and object it held. The root cannot be deleted; a delete that fails after its REMOVE op is finished by the next Open.

func (Volume) Diff
func (v *Volume) Diff(a, b uint64) ([]Change, error)

Diff returns one Change per node added, moved, modified or deleted between versions a and b, 1 <= a <= b <= Len, sorted by path, then node ID. A node deeper than MaxDepth, which only a hostile log has, has empty paths.

func (Volume) ID
func (v *Volume) ID() kololog.LogID

ID returns the volume's ID, the ID of its metadata log.

func (Volume) Len
func (v *Volume) Len() uint64

Len returns the number of metadata log entries the tree is built from, which is the current version. On a reader it is at most the size of the replica's last verified head, and less while an entry key or a purge op is missing.

func (Volume) Mark
func (v *Volume) Mark(name string) (Mark, error)

Mark returns the mark set on a path itself, MarkInherit when it has none.

func (Volume) Mkdir
func (v *Volume) Mkdir(name string) error

Mkdir creates the directory name; its parent must exist, and an existing name is fs.ErrExist.

func (Volume) Move
func (v *Volume) Move(from, to string) error

Move renames or moves from to the path to, whose parent must exist; a directory moves with everything below it. An existing to is fs.ErrExist; the root, or a move into from's own subtree, is fs.ErrInvalid.

func (Volume) Open
func (v *Volume) Open(name string) (fs.File, error)

Open opens a file or directory for reading, as fs.FS: a file is also an io.ReaderAt and io.Seeker, a directory an fs.ReadDirFile. Reads of remote content cannot be cancelled; use OpenContext for that.

func (Volume) OpenContext
func (v *Volume) OpenContext(ctx context.Context, name string) (fs.File, error)

OpenContext is Open with a context that bounds every content read of the file it returns.

func (Volume) Prefetch
func (v *Volume) Prefetch(ctx context.Context) error

Prefetch reads every MarkKeepLocal file's current version through the local backend, keeping its objects and dropping those of older versions. It goes on after a file fails and returns every error. It holds the change lock, so Refresh, SetMark and Close wait for it: end a long one with ctx.

func (Volume) ReadDir
func (v *Volume) ReadDir(name string) ([]fs.DirEntry, error)

ReadDir lists a directory sorted by name in byte order, as fs.ReadDirFS.

func (Volume) Refresh
func (v *Volume) Refresh() error

Refresh brings a reader up to its replica's latest verified head, in place when it can, else by a full replay. Len may go down. On a writer it does nothing. On an error before the new tree is installed it stays as it was; an error from the clean-up of the local backend after that leaves the new tree.

func (Volume) SetMark
func (v *Volume) SetMark(name string, m Mark) error

SetMark sets the mark of a path on this reader device, MarkInherit clears it; a directory's mark covers everything below it. Marks need a reader opened with a local backend.

func (Volume) Snapshot
func (v *Volume) Snapshot(ver uint64) (*Snapshot, error)

Snapshot returns the tree after the first ver entries, 1 to Len. A deleted node is in no snapshot.

func (Volume) Stat
func (v *Volume) Stat(name string) (fs.FileInfo, error)

Stat describes a file or directory, as fs.StatFS; its Sys is a NodeInfo.

func (Volume) Stats
func (v *Volume) Stats() Stats

Stats returns the volume's totals; it is safe to call at any time.

func (Volume) WriteFile
func (v *Volume) WriteFile(ctx context.Context, name string, src io.Reader) error

WriteFile creates the file name, whose parent must exist, or stores new content for an existing file, reading src to its end. Only changed chunks are stored; a write killed part way is resumed by calling it again.

Good to know

A path follows io/fs.ValidPath: slash-separated, relative to the volume root, with . as the root. A path that breaks a rule is fs.ErrInvalid inside an *fs.PathError:

RuleDecision
EncodingValid UTF-8
NormalizationNames are stored in NFC, and every path given to the API is normalized first, so e plus a combining acute accent and the precomposed é are one name
Unassigned code pointsRefused when a name is created (in the Unicode version of the Go release); a reader accepts them, since a newer writer's tables may know more
Control charactersRefused (general category Cc)
., .., empty, / in a nameRefused
CaseNames compare byte for byte: README and readme may both exist, and a case-only rename is an ordinary move
Windows-only rules (<>:"|?*, CON and the other device names with or without an extension, a trailing dot or space, over 255 UTF-16 units)Allowed, since other systems create such names; code that writes a volume to disk converts each path with LocalPath, which on Windows refuses them with ErrUnsafeName (filepath.Localize alone accepts NUL.txt and <>|?*"; os.CopyFS does not use these rules). See Writing a volume to disk
Name lengthAt most MaxNameLength bytes after NFC, default 765 (DefaultMaxNameLength)
DepthAt most MaxDepth names below the root, default 2048 (DefaultMaxDepth)

Every file and folder has a random 16-byte NodeID, chosen when it is created and never changed by a move or a rename. Ops refer to nodes, never to paths, so a rename is one op, a moved folder takes its subtree along and history follows a file across renames. FileInfo.Sys() is a NodeInfo: the node's ID, the version in which it last changed and the blob ID. Stat reports files as 0444 and folders as fs.ModeDir | 0555; the modification time is the writer's clock at the op, for display only. Listings are sorted by name in byte order. A file is also an io.ReaderAt and an io.Seeker, and a range read fetches one node per tree level and the chunks that cover it. OpenContext gives a file a context that ends its reads.

All steps

Files on top of blobs

kolo-files decides names, folders and versions. kolo-blobs stores and checks the bytes of each version.

  1. A volume is two kolo-store logs of one owner: the metadata log, one op per change, and the content log, which holds kolo-blobs objects. Entry 0 of the metadata log names the content log.
  2. WriteFile gives the file's bytes to kolo-blobs under the file's own key. Its chunks and its root become entries of the content log, and the root's hash is the blob ID.
  3. It also writes an object table, a second blob under the same key, that lists the content log entry of each object, so a reader finds them without scanning the log.
  4. The CREATE op holds the name, the file key and the version: the blob ID, the table ID and where the table's own objects are. kolo-log encrypts every op.
  5. A new version is a WRITE op with a new blob and table. The key stays the same, so unchanged chunks are not stored again.

WriteFile creates a file (its parent must exist) or stores new content for an existing one, reading the source to its end. Only the chunks that changed since earlier versions are stored, because kolo-blobs deduplicates within one key, and a killed write is resumed by calling it again with the same content. ctx is checked between chunks. Mkdir needs its parent and fails with fs.ErrExist on an existing name. Move renames or moves a file or a whole folder; the destination's parent must exist and the destination must not (fs.ErrExist), a move into the node's own subtree or of the root is fs.ErrInvalid, and a move to the same path does nothing. A Volume and a Snapshot are safe for concurrent use. Writes are serialized from the start of a blob write to the op's append, reads never wait for a blob write, and a failed append leaves the tree as it was.

Names

A name is a UTF-8 string in NFC. A path given in another form is normalized first, so é typed as e plus a combining accent and the precomposed é are one name, and the original bytes are not kept. Names compare byte for byte, so case is kept and README and readme can coexist. Control characters, empty names, . and .. are refused.

Writing a volume to disk

LocalPath converts a volume path into a path for the host system. On Windows it refuses with ErrUnsafeName a name Windows reserves for a device in any ASCII case (CON, NUL.txt, COM1 and the others, with or without an extension; some Windows versions open the device instead of creating a file), one of <>:"|?*\, a trailing dot or space, a control character and a name over 255 UTF-16 units. It checks names only and never looks at the disk.

A disk can give two volume names one entry: names that differ only in case on a case-insensitive disk (the macOS and Windows defaults), or a Windows 8.3 short name such as LONGFI~1.TXT for longfilename.txt or PROGRA~1 for a folder. A link already in the destination can also lead outside it. So create each folder and file under an os.Root, with Mkdir and O_CREATE|O_EXCL: an alias then fails with fs.ErrExist instead of overwriting another entry, and os.Root refuses a path that leaves the directory. This program is the LocalPath example of the library, compiled and run by go test. It writes the volume into a temporary directory and checks out the same tree twice, and the second checkout overwrites nothing:

main.go
package main

import (
	"context"
	"errors"
	"fmt"
	"io"
	"io/fs"
	"log"
	"os"
	"path/filepath"
	"strings"

	kolofiles "github.com/ciranda-dev/kolo-files"
	koloid "github.com/ciranda-dev/kolo-id"
	kolostore "github.com/ciranda-dev/kolo-store"
)

// checkout writes the tree of v into dir, which should be empty. Each folder and file is created under an
// os.Root, with Mkdir and O_EXCL, so a name that aliases one already written (a case-only pair on a
// case-insensitive disk, a Windows 8.3 short name) fails with fs.ErrExist instead of overwriting it, and no link
// leads outside dir.
func checkout(v fs.FS, dir string) error {
	root, err := os.OpenRoot(dir)
	if err != nil {
		return err
	}
	defer root.Close()
	return fs.WalkDir(v, ".", func(name string, d fs.DirEntry, err error) error {
		if err != nil || name == "." {
			return err
		}
		local, err := kolofiles.LocalPath(name)
		if err != nil {
			return err
		}
		if d.IsDir() {
			return root.Mkdir(local, 0o777)
		}
		src, err := v.Open(name)
		if err != nil {
			return err
		}
		defer src.Close()
		dst, err := root.OpenFile(local, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o666)
		if err != nil {
			return err
		}
		if _, err := io.Copy(dst, src); err != nil {
			dst.Close()
			return err
		}
		return dst.Close()
	})
}

func main() {
	st := kolostore.New(kolostore.NewMemory())
	defer st.Close()
	me, err := koloid.Generate()
	if err != nil {
		log.Fatal(err)
	}
	meta, err := st.Create(me)
	if err != nil {
		log.Fatal(err)
	}
	content, err := st.Create(me)
	if err != nil {
		log.Fatal(err)
	}
	v, err := kolofiles.Create(meta, content)
	if err != nil {
		log.Fatal(err)
	}
	defer v.Close()
	if err := v.Mkdir("notes"); err != nil {
		log.Fatal(err)
	}
	if err := v.WriteFile(context.Background(), "notes/todo.txt", strings.NewReader("buy bread")); err != nil {
		log.Fatal(err)
	}

	dir, err := os.MkdirTemp("", "checkout")
	if err != nil {
		log.Fatal(err)
	}
	defer os.RemoveAll(dir)
	if err := checkout(v, dir); err != nil {
		log.Fatal(err)
	}
	b, err := os.ReadFile(filepath.Join(dir, "notes", "todo.txt"))
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(string(b))
	// A second checkout into the same directory overwrites nothing.
	fmt.Println(errors.Is(checkout(v, dir), fs.ErrExist))
}

It prints:

Text
buy bread
true

os.CopyFS is not a substitute: it follows links already in the destination, merges folders whose names differ only in case, and on Windows it renames a trailing dot or space and writes device names with an extension such as NUL.txt, all without an error. filepath.Localize alone is not enough either: it accepts NUL.txt and <>|?*".

Versions

All steps

Versions, checkout and diff

Every op is a version, so history has the granularity of single changes.

  1. A version is the length of the metadata log: the tree after its first entries. Version 1 is the empty root, and Len is the current version.
  2. Snapshot(3) replays the first 3 entries into a tree of its own. It is immutable, needs no closing, and is an io/fs file system.
  3. Snapshot(Len) is the tree now. Here the log has 6 entries, so the current version is 6.
  4. Diff(3, 6) returns one Change per node added, moved, modified or deleted in between, sorted by path. A moved folder is one change.
  5. Deletion wins over history: a deleted node is in no snapshot, as if it had never been there. A diff shows it only as a Change with Deleted set and no paths.

Len is the current version: the number of metadata entries the tree is built from. Snapshot(version), for 1 to Len, returns the tree after the first version entries as an immutable Snapshot with the same Open, OpenContext, ReadDir and Stat, and Version. Version 1 is the empty root. Diff(a, b), for 1 <= a <= b <= Len, returns one Change per node added, moved, modified or deleted in between, sorted by path and then node ID; a moved folder is one change. Deletion wins over history: a deleted node is in no snapshot, as if it had never been there, and it appears in a diff only as a Change with Deleted set and no paths. Overwritten content is kept, so every earlier version of a live file stays readable; only Delete removes history. A Snapshot costs one pass over the first version entries.

Reading from another device

OpenReader(meta, content, local) builds a read-only volume from a replica of the metadata log. content is a kolo-blobs Backend, such as koloblobs.NewLogBackend on a replica of the content log. OpenFetchReader(meta, fetch, local) reads the content from another peer instead, where fetch(ctx, index) gets one content log entry, for example with kolo-sync's Fetch, or with FetchWithKeys from a kolo-pin pinning peer that serves ciphertext only while the reader holds the entry keys. The pinning peer never receives a key. The README has the code, and kolofiles.ContentLog(meta) gives the content log's ID from the first entry.

A reader replays the longest prefix it can replay exactly: it stops at an entry held without its key and before a destroyed entry that no complete purge lists, and Len says which version that is. Refresh brings it up to the replica's latest verified head, incrementally when it can, and Len may go down. An incremental refresh checks every entry below Len for a destroy marker, so it costs time in Len. When the owner changes the volume, pull the metadata log again and call Refresh; a version written after the head you saved needs a new head and its keys. A reader device keeps the metadata log with its entry keys, which come from the owner or from any peer with kolo-sync Keys access, and Close leaves the local and content backends open.

Selective sync

All steps

Selective sync

Marks are one device's choice. Disk use never grows on its own: the default keeps nothing.

  1. A reader keeps a replica of the metadata log, so it has every name and every version and can list and walk the whole tree.
  2. Content is not copied by default. With MarkFetchOnDemand, the root's default, a read fetches only the objects it needs from another peer, one node per tree level and the chunks that cover the range, and keeps none.
  3. MarkKeepLocal keeps every object of the current version of the files it covers. A node without a mark takes the mark of its nearest marked ancestor, and a folder's mark covers files added later.
  4. Prefetch fetches the objects of each kept file that the device lacks and removes its kept objects of older versions. Every kept object is checked against its hash.
  5. Marks are stored by node ID in the local backend of this device only, so they follow moves and are in no shared log.

SetMark, Mark and Prefetch work on a reader opened with a local backend. MarkKeepLocal keeps every object of the file's current version on the device, so it reads offline. MarkFetchOnDemand fetches only the objects a read needs and keeps nothing. MarkInherit clears a mark. A node without a mark takes its nearest marked ancestor's, and the root's default is MarkFetchOnDemand, so disk use never grows on its own. Marks are stored by node ID, so they follow moves, and they are dropped with the node. Prefetch reads every MarkKeepLocal file's current version through the local backend, removes its kept objects of older versions, goes on after a file fails and returns every error. A kept object is checked against its hash before it is kept and before it is served; one that fails is dropped. A move that puts a file under a node with another mark, seen by Refresh, removes the objects kept for it. A mark is one device's choice, so it is never in the shared logs. Marks on a writer, or on a reader without a local backend, are ErrInvalidArgument.

Stats returns totals since open, read with atomics: Ops, Replayed, BytesWritten, BytesRead, ObjectsFetched and ObjectsLocal.

Delete

Delete removes a file or a folder with everything below it and destroys what it held. It appends a REMOVE op, which holds only the node ID, and PURGE ops that list the metadata and content entries of the subtree, each under the largest entry kolo-sync carries, and then destroys those entries with kolo-log's Delete. What is destroyed: every name the node ever had, its file key (so every version becomes unreadable at once on peers that follow the protocol) and every object. What stays: the REMOVE and PURGE ops (a node ID and entry indexes), kolo-log's signed records and markers, and the fact that a delete happened. A crash at any step is finished by the next Open. The root cannot be deleted. See Deletion.

Limits

The two bounds below are safety settings, not product limits, and each has an option.

  • MaxDepth, 2048 by default (DefaultMaxDepth), set with WithMaxDepth(n), at least 1: the deepest tree one Linux path argument can name (PATH_MAX 4096 with the NUL, one-byte names).
  • MaxNameLength, 765 bytes by default (DefaultMaxNameLength), set with WithMaxNameLength(n), from 1 up to the largest op, kolo-blobs MaxObjectSize (4,128,764 bytes), since a name must fit in one op. 765 bytes holds every name that ext4 (255 bytes) and Windows allow: Windows names are at most 255 UTF-16 units, and a UTF-16 unit takes at most 3 bytes of UTF-8.
  • A path over a bound is ErrLimit. A writer enforces the bounds on every create and move, so a reader with the same settings opens everything the writer wrote. A reader checks a replayed create or move only for the depth of the node it creates or moves, not for the nodes below a moved folder, so a hostile log can put nodes deeper than MaxDepth; such a node has no path (a lookup is ErrLimit, and Diff reports it with empty paths).
  • Everything else is bounded by what kolo-store has accepted: there is no limit on files, folders, versions or file size.
  • Walks up the tree (moves, paths, marks) take at most MaxDepth steps. Walks down it (a delete, a check before a move) take at most the number of nodes. Every walk is iterative.
  • An object table is read as a stream: its count is checked against the objects a blob of that size can hold and against the table's size, and memory grows with the bytes the table blob really holds, not with the size a writer claims.
  • Memory: the current tree is kept in memory, about 211 bytes per live node with a 10-byte name (BenchmarkStateNode, 64-bit, Apple M5, Go 1.27.3), plus the ID of every node a delete removed, 8 bytes per destroyed entry, and kolo-store's own cost per entry. A replay of a whole log holds every decoded op until it has applied them, about 344 bytes per entry with a 10-byte name (BenchmarkReplayEntry, same machine) besides the tree, so open a very large volume with GOMEMLIMIT set. Snapshots and diffs hold their own tree while they run and nothing after.

Security

  • A peer that serves entries to a reader with Fetch or FetchWithKeys sees which content entries are fetched and when. From the order (a table, the root, one node per level, then chunks) it learns which file is opened and which ranges are read, and the size of the head the reader checks against.
  • Under one file key, equal chunks are equal ciphertext, so two versions of a file show which chunks they share. Different files never share an object, because each file has its own key.
  • Each file gets a kolo-blobs key when it is created, inside its create op. Keys are not kept in memory and are never returned, printed or logged; a copy read from the log is cleared after use. No error carries a key, file content or entry bytes.
  • Each version also writes an object table, a second blob under the file key that lists every object of the content blob and the content log entry that holds it. A reader without the content replica finds each entry from it with no scan of the log, and Delete can list the entries of every version.
  • Delete is a best-effort tombstone: it holds on every peer that follows the protocol and applies the delete markers, and a peer that already received a key can keep the data. A reader that sees destroyed entries before the PURGE op that lists them steps back to just before them until the op arrives, and drops the marks and kept objects of removed nodes on its next open or refresh. Removing a kept object from the local Pebble store is a tombstone too, so old bytes stay until compaction.
  • A write killed before its op was appended leaves unreferenced objects. For a new file nobody can read them, since its key was never stored; for an existing file they are under that file's key and are destroyed when the file is deleted.
  • A volume has one writer, its owner. A log with dependencies on another writer is ErrCorrupt. A reader refuses every op that breaks a rule of the format with ErrCorrupt; the README lists the rules and the formats.

Building an update system on it

These are facts about the library, not a feature: kolo-files has no update system.

  • A version is a length of the owner's signed metadata log, and Snapshot and Diff read any version.
  • Each file's blob ID is in NodeInfo, from Stat().Sys().
  • Prefetch fetches only the objects the device does not hold yet, and an interrupted Prefetch resumes with the objects still missing.
  • kolo-files keeps no file modes, executable bit or symlinks, and Delete also removes a node from older snapshots. A layer that installs applications keeps the modes, symlinks and expected blob IDs in its own signed manifest.

Known limits

  • One writer only.
  • Stat reports no permissions beyond 0444 and 0555, and the modification time is the writer's clock.
  • Windows-only name rules are not applied inside a volume: write a volume to disk with LocalPath and the recipe in Writing a volume to disk. LocalPath checks names only.
  • All tests run on 127.0.0.1. Experimental. The API may change before v1.0.0.

Errors

ErrorMessage
ErrClosedkolofiles: volume closed
ErrCorruptkolofiles: corrupt volume
ErrInvalidArgumentkolofiles: invalid argument
ErrIsDirkolofiles: is a directory
ErrLimitkolofiles: over a bound
ErrNotDirkolofiles: not a directory
ErrReadOnlykolofiles: read-only volume
ErrUnsafeNamekolofiles: name not allowed on this system

Test errors with errors.Is; none carries a key, file content or entry bytes. Path errors are *fs.PathError with Op starting kolofiles:, so fs.ErrNotExist, fs.ErrExist and fs.ErrInvalid match; names appear only in PathError.Path, as the caller gave them. Errors from kolo-blobs, kolo-store and kolo-log pass through wrapped. A file whose entries were destroyed reads as kololog.ErrDeleted (and fs.ErrNotExist). ErrLimit also covers an op to append over the largest entry kolo-sync carries, ErrClosed also covers a read of a Snapshot or a file of a closed volume, and ErrInvalidArgument also covers a version out of range, logs of different owners and a non-empty log given to Create.

References

  • The io/fs package of the Go standard library.
    • Takes: path syntax (ValidPath), the FS, ReadDirFS, StatFS, File and ReadDirFile interfaces, PathError and its sentinels; path/filepath.Localize and os.Root for writing a volume to disk.
    • Not implemented: nothing left out.
  • Unicode Standard Annex #15, Unicode Normalization Forms (revision 58, Unicode 18.0.0, 12 August 2026).
    • Takes: NFC for names, and its stability policy for assigned characters.
    • Not implemented: NFKC and NFKD.
  • The file name rules of POSIX file systems, in plain words.
    • Takes: no NUL byte and no / in a name.
    • Not implemented: nothing left out.
  • The references of kolo-blobs.
    • Takes: file keys, objects and the tree under every file, through kolo-blobs.
    • Not implemented: see the References of kolo-blobs.
  • RFC 9562, Universally Unique IDentifiers (UUIDs) (Proposed Standard, May 2024).
    • Takes: nothing.
    • Not implemented: UUIDs. Node IDs are 16 random bytes, deliberately not UUIDs.