kolo docsbeta

Share files between two terminals

A step-by-step tutorial. One terminal shares a file with kolo-blobs, the other receives it with a share code and checks every byte; then the same for a folder and its versions with kolo-files.

On this page

In this tutorial you build two small programs and run each in two terminal windows on your computer. The first shares one file: Terminal 1 stores it and prints a share code, and Terminal 2 uses the code to fetch the file, check every chunk of it and save it. The second shares a whole folder and keeps the copy up to date: each time you change the folder and press Enter, the other terminal saves the new version and prints what changed.

The first program uses kolo-blobs, which stores the bytes of one file. The second uses kolo-files, which keeps files and folders, their names and every version on top of kolo-blobs. Everything stays on your computer, on the address 127.0.0.1. The finished code is in the kolo-share-example repository.

1Check what you need

You need Go 1.26 or newer and two terminal windows. If Go is not installed yet, install it for your system:

Download the macOS installer (the .pkg file) from go.dev/dl and run it. If you use Homebrew, this works too:

bash
brew install go

Check your Go version:

Terminal
go version

You should see 1.26 or newer, for example:

Text
go version go1.27.3 darwin/arm64

Now open two terminal windows and keep them side by side:

Open the Terminal app and press Cmd+N for a second window.

This tutorial calls them Terminal 1 (the peer that shares) and Terminal 2 (the peer that receives).

2Create a folder and a module

In Terminal 1, make a folder for the project and go into it:

bash
mkdir kolo-share && cd kolo-share

Then create a Go module in it:

Terminal
go mod init example.com/kolo-share

Go answers:

Text
go: creating new go.mod: module example.com/kolo-share

3Get the libraries

Still in Terminal 1, add the Kolo libraries the two programs use: kolo-blobs and kolo-files for the content, kolo-store to keep it in logs on disk, kolo-sync to copy those logs to the other peer, kolo-secure for the encrypted connection, kolo-id for each peer's key and kolo-log for the log IDs.

Terminal
go get github.com/ciranda-dev/kolo-blobs@v0.1.0 github.com/ciranda-dev/kolo-files@v0.1.0 github.com/ciranda-dev/kolo-sync@v0.1.0 github.com/ciranda-dev/kolo-store@v0.1.0 github.com/ciranda-dev/kolo-secure@v0.1.0 github.com/ciranda-dev/kolo-id@v0.1.0 github.com/ciranda-dev/kolo-log@v0.1.0

Go downloads them with what they need and prints one go: added line for each module it adds to go.mod.

4Write the program that shares one file

Here is what the program does, in five steps:

All steps

One file, two terminals

The share code is the only thing you pass by hand. Whoever has it can read the file.

  1. file send writes the file as a blob into a new kolo-store log, under a new file key, and prints a share code.
  2. You copy the code to the other terminal: the sender's address and key, the log ID, a seed for the receiver's identity, the blob ID and the file key.
  3. file get turns the seed into its identity and connects with kolo-secure, which checks that the other side holds the sender's key.
  4. kolo-sync pulls the log. The sender serves it only to the identity from the code, and every entry is checked against the sender's signed head.
  5. kolo-blobs opens the blob with the ID and key from the code and checks every chunk while it saves the file. A file that fails a check is removed.

Make a folder named file inside kolo-share, create a file named main.go in it and paste this program. It has two commands: send FILE stores the file and serves it, and get CODE OUT fetches it with the share code and saves it as OUT.

file/main.go
// Command file shares one file between two peers with kolo-blobs.
//
//	file send FILE      stores FILE as a blob, prints a share code and serves the blob
//	file get CODE OUT   fetches the blob with the code, checks every chunk and saves it as OUT
package main

import (
	"bufio"
	"context"
	"crypto/ed25519"
	"crypto/rand"
	"crypto/sha256"
	"encoding/base64"
	"encoding/binary"
	"errors"
	"flag"
	"fmt"
	"io"
	"io/fs"
	"log"
	"net/netip"
	"os"
	"os/signal"
	"path/filepath"
	"strings"
	"sync"
	"syscall"
	"time"

	koloblobs "github.com/ciranda-dev/kolo-blobs"
	koloid "github.com/ciranda-dev/kolo-id"
	kololog "github.com/ciranda-dev/kolo-log"
	kolosecure "github.com/ciranda-dev/kolo-secure"
	kolostore "github.com/ciranda-dev/kolo-store"
	kolosync "github.com/ciranda-dev/kolo-sync"
)

const dialTimeout = 10 * time.Second

func main() {
	listen := flag.String("listen", "127.0.0.1:0", "IPv4 IP:PORT that send listens on; the share code carries it")
	maxBytes := flag.Int64("max-bytes", 1<<30, "most bytes get stores for a file")
	flag.Usage = func() {
		fmt.Fprintln(os.Stderr, "usage: file [-listen IP:PORT] send FILE\n       file [-max-bytes N] get CODE OUT   (CODE - reads the code from standard input)")
		flag.PrintDefaults()
	}
	flag.Parse()
	if *maxBytes <= 0 {
		flag.Usage()
		os.Exit(2)
	}
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM, syscall.SIGHUP)
	defer stop()
	log.SetFlags(log.Ltime)

	var err error
	switch args := flag.Args(); {
	case len(args) == 2 && args[0] == "send":
		err = send(ctx, *listen, args[1])
	case len(args) == 3 && args[0] == "get":
		err = get(ctx, args[1], args[2], *maxBytes)
	default:
		flag.Usage()
		os.Exit(2)
	}
	if err != nil && ctx.Err() == nil {
		log.Fatal(err)
	}
}

// send stores the file as a blob in a new log and serves that log to whoever holds the share code.
func send(ctx context.Context, listen, name string) error {
	addr, err := netip.ParseAddrPort(listen)
	if err != nil || !addr.Addr().Is4() || addr.Addr().IsUnspecified() {
		return fmt.Errorf("-listen needs an IPv4 address the receiver can reach, such as 127.0.0.1:0, not %q", listen)
	}
	f, err := os.Open(name)
	if err != nil {
		return err
	}
	defer f.Close()
	if info, err := f.Stat(); err != nil {
		return err
	} else if !info.Mode().IsRegular() {
		return fmt.Errorf("%s is not a regular file", name)
	}

	st, _, cleanup, err := openStore()
	if err != nil {
		return err
	}
	defer cleanup()
	me, err := koloid.Generate()
	if err != nil {
		return err
	}
	l, err := st.Create(me)
	if err != nil {
		return err
	}
	key := koloblobs.NewKey()
	sum := sha256.New()
	id, err := koloblobs.New(koloblobs.NewLogBackend(l)).Write(ctx, key, io.TeeReader(f, sum))
	if err != nil {
		return fmt.Errorf("store %s: %w", name, err)
	}
	log.Printf("stored %s as blob %s in %d log entries", name, id.String()[:16], l.Len())
	log.Printf("sha256 %x", sum.Sum(nil))

	// The receiver's identity comes from a seed in the share code, so only the code's holder gets the log.
	var seed [32]byte
	rand.Read(seed[:])
	reader, err := koloid.FromSeed(seed[:])
	if err != nil {
		return err
	}
	srv := kolosync.NewServer(func(peer, _ ed25519.PublicKey, _ kololog.LogID) kolosync.Access {
		if peer.Equal(reader.PublicKey()) {
			return kolosync.Keys
		}
		return kolosync.NoAccess
	})
	if err := srv.Offer(l); err != nil {
		return err
	}
	ln, err := kolosecure.Listen(ctx, addr.String(), me, nil)
	if err != nil {
		return err
	}
	defer ln.Close()
	if addr, err = netip.ParseAddrPort(ln.Addr().String()); err != nil { // the port the system chose
		return err
	}

	c := shareCode{addr: addr, sender: me.PublicKey(), log: l.ID(), seed: seed, blob: id, key: key}
	fmt.Printf("share code:\n%s\n", c)
	log.Print("waiting for the receiver, press Ctrl-C to stop")
	var wg sync.WaitGroup
	defer wg.Wait()
	ctx, cancel := context.WithCancel(ctx) // ends the connections below whenever send returns
	defer cancel()
	for {
		conn, err := ln.Accept(ctx)
		if err != nil {
			return err
		}
		log.Printf("peer %x connected", conn.RemotePublicKey()[:4])
		wg.Go(func() {
			defer conn.Close()
			srv.Serve(ctx, conn)
			log.Printf("peer %x left", conn.RemotePublicKey()[:4])
		})
	}
}

// get copies the sender's log, then reads the blob from it and saves it, checking every chunk on the way.
// It stops with errTooBig when the copy would hold more than maxBytes.
func get(ctx context.Context, code, out string, maxBytes int64) error {
	code, err := readCode(code)
	if err != nil {
		return err
	}
	c, err := parseCode(code)
	if err != nil {
		return err
	}
	if _, err := os.Lstat(out); err == nil {
		return fmt.Errorf("%s already exists; choose a new name", out)
	}
	me, err := koloid.FromSeed(c.seed[:])
	if err != nil {
		return err
	}
	st, dir, cleanup, err := openStore()
	if err != nil {
		return err
	}
	defer cleanup()

	dctx, cancel := context.WithTimeout(ctx, dialTimeout)
	defer cancel()
	conn, err := kolosecure.Dial(dctx, c.addr.String(), me, c.sender, nil)
	if err != nil {
		return fmt.Errorf("cannot reach the sender at %s: %w", c.addr, err)
	}
	defer conn.Close()
	header, err := kolosync.FetchHeader(ctx, conn, c.sender, c.log)
	if errors.Is(err, kolosync.ErrNotFound) {
		return fmt.Errorf("the sender does not share this file with this code: %w", err)
	} else if err != nil {
		return fmt.Errorf("ask the sender for its log: %w", err)
	}
	rep, err := st.OpenReplica(c.sender, c.log, header)
	if err != nil {
		return err
	}
	log.Print("downloading")
	lctx, stopLimit := limit(ctx, dir, maxBytes)
	defer stopLimit()
	if err := kolosync.Pull(lctx, conn, rep); err != nil {
		return fmt.Errorf("download: %w", tooBig(lctx, err))
	}
	log.Printf("received %d log entries, each checked against the sender's signed head", rep.Len())

	r, err := koloblobs.New(koloblobs.NewLogBackend(rep)).Open(ctx, c.blob, c.key)
	if err != nil {
		return explain(err)
	}
	defer r.Close()
	if r.Size() > maxBytes {
		return errTooBig
	}
	f, err := os.OpenFile(out, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o600) // never replaces a file
	if err != nil {
		return err
	}
	sum := sha256.New()
	_, err = io.Copy(io.MultiWriter(f, sum), r)
	if cerr := f.Close(); err == nil {
		err = cerr
	}
	if err != nil {
		os.Remove(out) // no half-written or unverified file is left behind
		return explain(err)
	}
	log.Printf("saved %s: %d bytes, every chunk checked against the blob ID", out, r.Size())
	log.Printf("sha256 %x", sum.Sum(nil))
	return nil
}

// explain turns the errors of kolo-blobs into a sentence a user can act on.
func explain(err error) error {
	switch {
	case errors.Is(err, koloblobs.ErrWrongKey):
		return fmt.Errorf("the key in the share code does not open this file: %w", err)
	case errors.Is(err, koloblobs.ErrCorrupt):
		return fmt.Errorf("the file failed verification, nothing was saved: %w", err)
	case errors.Is(err, koloblobs.ErrNotFound):
		return fmt.Errorf("the sender's log does not hold this file: %w", err)
	}
	return err
}

// openStore opens a kolo-store in a new temporary folder, removed by the returned function.
func openStore() (*kolostore.Store, string, func(), error) {
	dir, err := os.MkdirTemp("", "kolo-share-")
	if err != nil {
		return nil, "", nil, err
	}
	db, err := kolostore.OpenPebble(dir)
	if err != nil {
		os.RemoveAll(dir)
		return nil, "", nil, err
	}
	st := kolostore.New(db)
	return st, dir, func() {
		st.Close()
		os.RemoveAll(dir)
	}, nil
}

// errTooBig ends a get that would store more than -max-bytes.
var errTooBig = errors.New("the sender offers more than -max-bytes allows; only get codes from people you trust")

// limit returns a context that ends with errTooBig once the folder dir holds more than max bytes.
func limit(ctx context.Context, dir string, max int64) (context.Context, context.CancelFunc) {
	ctx, cancel := context.WithCancelCause(ctx)
	go func() {
		for {
			select {
			case <-ctx.Done():
				return
			case <-time.After(250 * time.Millisecond):
			}
			var n int64
			filepath.WalkDir(dir, func(_ string, d fs.DirEntry, err error) error {
				if err != nil || d.IsDir() {
					return nil
				}
				if info, err := d.Info(); err == nil {
					n += info.Size()
				}
				return nil
			})
			if n > max {
				cancel(errTooBig)
				return
			}
		}
	}()
	return ctx, func() { cancel(nil) }
}

// tooBig returns errTooBig when ctx ended because of limit, and err otherwise.
func tooBig(ctx context.Context, err error) error {
	if errors.Is(context.Cause(ctx), errTooBig) {
		return errTooBig
	}
	return err
}

// readCode returns code, or the first line of standard input when code is "-", so that it stays out of ps and the shell history.
func readCode(code string) (string, error) {
	if code != "-" {
		return code, nil
	}
	line, _ := bufio.NewReader(io.LimitReader(os.Stdin, 4096)).ReadString('\n')
	if line == "" {
		return "", errors.New("no share code on standard input")
	}
	return line, nil
}

// checkAddr refuses the addresses nobody can listen on, so that a share code cannot make get send packets to them.
func checkAddr(a netip.AddrPort) bool {
	ip := a.Addr()
	return a.Port() != 0 && !ip.IsUnspecified() && !ip.IsMulticast() && ip != netip.AddrFrom4([4]byte{255, 255, 255, 255})
}

// shareCode is all the receiver needs: where the sender listens, its public key, the log holding the blob,
// the seed of the receiver's identity, and the blob's ID and key. Whoever has it can read the file.
type shareCode struct {
	addr   netip.AddrPort
	sender ed25519.PublicKey
	log    kololog.LogID
	seed   [32]byte
	blob   koloblobs.ID
	key    koloblobs.Key
}

const (
	codePrefix = "kolo-file:"
	codeSize   = 6 + 32 + kololog.LogIDSize + 32 + koloblobs.HashSize + koloblobs.KeySize
)

func (c shareCode) String() string {
	ip := c.addr.Addr().As4()
	b := binary.BigEndian.AppendUint16(ip[:], c.addr.Port())
	b = append(b, c.sender...)
	b = append(b, c.log[:]...)
	b = append(b, c.seed[:]...)
	b = append(b, c.blob[:]...)
	b = append(b, c.key[:]...)
	return codePrefix + base64.RawURLEncoding.EncodeToString(b)
}

func parseCode(s string) (shareCode, error) {
	rest, ok := strings.CutPrefix(strings.TrimSpace(s), codePrefix)
	b, err := base64.RawURLEncoding.DecodeString(rest)
	if !ok || err != nil || len(b) != codeSize {
		return shareCode{}, fmt.Errorf("this is not a file share code; copy the whole line, it starts with %q", codePrefix)
	}
	var c shareCode
	c.addr = netip.AddrPortFrom(netip.AddrFrom4([4]byte(b[:4])), binary.BigEndian.Uint16(b[4:6]))
	if !checkAddr(c.addr) {
		return shareCode{}, fmt.Errorf("the share code names %s, an address nobody can listen on", c.addr)
	}
	b = b[6:]
	c.sender = ed25519.PublicKey(b[:32])
	b = b[32:]
	b = b[copy(c.log[:], b):]
	b = b[copy(c.seed[:], b):]
	b = b[copy(c.blob[:], b):]
	copy(c.key[:], b)
	return c, nil
}

What each part does:

  • send creates a new identity for this run with koloid.Generate and a kolo-store log in a new temporary folder (openStore), which is removed when the program stops.
  • koloblobs.NewKey makes a random key for this file, and Write cuts the file into chunks, encrypts each one and appends them to the log as entries. It returns the blob ID. The program also computes the file's SHA-256 while it reads, with io.TeeReader, so you can compare it with the copy later.
  • The share code holds everything the receiver needs: the address and public key of the sender, the ID of the log, the blob ID, the file key, and a random seed. The receiver turns the seed into its identity with koloid.FromSeed. The kolo-sync server gives Keys access only to that identity, so a peer without the code gets nothing.
  • get connects with kolosecure.Dial, which checks that the other side holds the sender's key. kolosync.FetchHeader and OpenReplica make a local copy of the sender's log, and kolosync.Pull fills it, checking every entry against the sender's signed head.
  • Open checks the blob's root against the ID and the key, and io.Copy reads the file chunk by chunk; each chunk is checked against the tree before its bytes are written. If anything fails, the program removes the output file, also when you interrupt it, so a half-written or unverified file is never left behind. The file is created with mode 0600: only you can read it.
  • get also keeps itself safe against a code that comes from someone else. It stops with the sender offers more than -max-bytes allows; only get codes from people you trust when what it downloads (the temporary store) or what it writes is more than -max-bytes, 1 GiB by default (for example go run ./file -max-bytes 100000000 get CODE copy.jpg). A code that names 0.0.0.0, a multicast address, the broadcast address or port 0 is refused (the share code names ADDR, an address nobody can listen on). A code is an address plus a secret, so use only codes from people you trust. The code can be read from standard input with -, so it stays out of ps and the shell history: echo CODE | go run ./file get - copy.txt.
  • explain turns the errors of kolo-blobs into sentences: a wrong key (ErrWrongKey), an object that fails a check (ErrCorrupt) or a missing one (ErrNotFound).
  • -listen is the address the sender listens on, 127.0.0.1:0 by default: your computer only, on a port the system picks. The code carries that address, so it must be an IPv4 address the receiver can reach.

5Make a file to share

In Terminal 1, in the kolo-share folder, make a small text file:

bash
echo "hello from terminal 1" > hello.txt

Any file works: a photo, a video, an archive. A file of any size is stored as many chunks as it needs.

6Share it from Terminal 1

In Terminal 1, run:

Terminal
go run ./file send hello.txt

The first run compiles for a moment. Then it stores the file, prints its SHA-256 and the share code, and waits. Your times, IDs, hash and code will differ:

Text
21:06:38 stored hello.txt as blob a9d85e576f8d1489 in 2 log entries
21:06:38 sha256 2b3bbe311b98a8d367296b62fcb806cb076ccddb893f9f02dbd0c327c98e2385
share code:
kolo-file:fwAAAfOct7bUTbr73o5USjYMV_QoCysEQPbcwAuLL6DkJGdBSUQglI3Xz74siUO8VhQOnKqBB_BVw5FKd-VlHrXEjaeUYdYkEK2038sz_yilgROh560Mz7emkvCv0wzsYlDWEg5KqdheV2-NFImL7UW_f-0M4Q-5I9a4D2Tiy1BSEnQPJUFePQEC6hq_trv_ybEglewb2IyH6o9exzlkNklfTSSOAw
21:06:38 waiting for the receiver, press Ctrl-C to stop

A small file takes 2 log entries: one chunk and the root of its tree. Select the whole line that starts with kolo-file: and copy it. Leave Terminal 1 running.

7Receive it in Terminal 2

In Terminal 2, go to the same kolo-share folder (use the path where you created it):

Terminal
cd kolo-share

Then run get with the code you copied, in single quotes, and a name for the copy:

Terminal
go run ./file get '<share code>' copy.txt

Replace <share code> with the whole code, kolo-file: included. Terminal 2 prints:

Text
21:06:38 downloading
21:06:38 received 2 log entries, each checked against the sender's signed head
21:06:38 saved copy.txt: 22 bytes, every chunk checked against the blob ID
21:06:38 sha256 2b3bbe311b98a8d367296b62fcb806cb076ccddb893f9f02dbd0c327c98e2385

And Terminal 1 shows the receiver come and go, under the name of its key:

Text
21:06:38 peer db309a79 connected
21:06:38 peer db309a79 left

8Check that the copy is the same

The program already checked every chunk against the blob ID, and the two sha256 lines, one in each terminal, are the same. You can also ask your system for the SHA-256 of both files, in Terminal 2:

bash
shasum -a 256 hello.txt copy.txt

Both lines carry the same hash:

Text
2b3bbe311b98a8d367296b62fcb806cb076ccddb893f9f02dbd0c327c98e2385  hello.txt
2b3bbe311b98a8d367296b62fcb806cb076ccddb893f9f02dbd0c327c98e2385  copy.txt

Get-FileHash on Windows prints a table with the hash in capitals. There, Set-Content also ends the line with two characters, so the file has 23 bytes and another hash; the two files still match each other.

9See what happens when something goes wrong

Run get again with the same name. It never replaces a file, and it removes the partial file when it fails or is interrupted:

Terminal
go run ./file get '<share code>' copy.txt
Text
21:06:39 copy.txt already exists; choose a new name
exit status 1

The last line comes from go run: the program exited with an error. Now give it something that is not a code:

Terminal
go run ./file get 'kolo-file:abc' other.txt
Text
21:06:39 this is not a file share code; copy the whole line, it starts with "kolo-file:"
exit status 1

Press Ctrl-C in Terminal 1 to stop the sender, then run get once more with the real code and a new name, in Terminal 2:

Terminal
go run ./file get '<share code>' late.txt

After a few seconds:

Text
21:06:53 cannot reach the sender at 127.0.0.1:62364: kolotransport: timeout
exit status 1

late.txt is not created. A code whose key was changed fails with the key in the share code does not open this file, and a code whose seed was changed with the sender does not share this file with this code: the repository's tests check both.

10Write the program that shares a folder

The second program shares a folder with kolo-files. The folder becomes a volume: a metadata log with one op per change, and a content log that holds each file's content as a kolo-blobs blob. Each op is a new version.

All steps

A folder and its versions

The receiver saves each version in a folder of its own, so older versions stay as they were.

  1. folder send puts every file and folder in a new kolo-files volume and prints a share code. No file key is in the code: the keys travel inside the metadata log.
  2. You copy the code to the other terminal: the sender's address and key, the metadata log ID and a seed for the receiver's identity.
  3. folder get pulls the metadata log, finds the content log in its first entry and pulls that too, then saves the volume with os.CopyFS in OUT/v and the version number, such as OUT/v5.
  4. Press Enter in Terminal 1: new files and folders become CREATE and MKDIR ops, changed files WRITE ops, deleted ones REMOVE and PURGE ops. Each op is a new version.
  5. Every 2 seconds the receiver pulls the new entries only, refreshes the volume, prints Diff between the saved version and the new one, and saves the new one.

Make a folder named folder inside kolo-share, create main.go in it and paste this program:

folder/main.go
// Command folder shares a folder between two peers with kolo-files and keeps the copy up to date.
//
//	folder send DIR    puts DIR in a volume, prints a share code and publishes changes each time you press Enter
//	folder get CODE OUT   saves the folder into OUT/v<version>, then each new version the sender publishes
package main

import (
	"bufio"
	"context"
	"crypto/ed25519"
	"crypto/rand"
	"encoding/base64"
	"encoding/binary"
	"errors"
	"flag"
	"fmt"
	"io"
	"io/fs"
	"log"
	"net/netip"
	"os"
	"os/signal"
	"path/filepath"
	"strings"
	"sync"
	"syscall"
	"time"

	koloblobs "github.com/ciranda-dev/kolo-blobs"
	kolofiles "github.com/ciranda-dev/kolo-files"
	koloid "github.com/ciranda-dev/kolo-id"
	kololog "github.com/ciranda-dev/kolo-log"
	kolosecure "github.com/ciranda-dev/kolo-secure"
	kolostore "github.com/ciranda-dev/kolo-store"
	kolosync "github.com/ciranda-dev/kolo-sync"
)

const dialTimeout = 10 * time.Second

func main() {
	listen := flag.String("listen", "127.0.0.1:0", "IPv4 IP:PORT that send listens on; the share code carries it")
	every := flag.Duration("every", 2*time.Second, "how often get asks the sender for a new version")
	maxBytes := flag.Int64("max-bytes", 1<<30, "most bytes get stores, and writes into OUT in all")
	flag.Usage = func() {
		fmt.Fprintln(os.Stderr, "usage: folder [-listen IP:PORT] send DIR\n       folder [-every 2s] [-max-bytes N] get CODE OUT   (CODE - reads the code from standard input)")
		flag.PrintDefaults()
	}
	flag.Parse()
	if *every <= 0 || *maxBytes <= 0 {
		flag.Usage()
		os.Exit(2)
	}
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM, syscall.SIGHUP)
	defer stop()
	log.SetFlags(log.Ltime)

	var err error
	switch args := flag.Args(); {
	case len(args) == 2 && args[0] == "send":
		err = send(ctx, *listen, args[1])
	case len(args) == 3 && args[0] == "get":
		err = get(ctx, args[1], args[2], *every, *maxBytes)
	default:
		flag.Usage()
		os.Exit(2)
	}
	if err != nil && ctx.Err() == nil {
		log.Fatal(err)
	}
}

// send keeps dir in a volume, serves its two logs to whoever holds the share code and adds a version on Enter.
func send(ctx context.Context, listen, dir string) error {
	addr, err := netip.ParseAddrPort(listen)
	if err != nil || !addr.Addr().Is4() || addr.Addr().IsUnspecified() {
		return fmt.Errorf("-listen needs an IPv4 address the receiver can reach, such as 127.0.0.1:0, not %q", listen)
	}
	if info, err := os.Lstat(dir); err != nil {
		return err
	} else if !info.IsDir() {
		return fmt.Errorf("%s is not a folder", dir)
	}
	st, _, cleanup, err := openStore()
	if err != nil {
		return err
	}
	defer cleanup()
	me, err := koloid.Generate()
	if err != nil {
		return err
	}
	meta, err := st.Create(me)
	if err != nil {
		return err
	}
	content, err := st.Create(me)
	if err != nil {
		return err
	}
	v, err := kolofiles.Create(meta, content)
	if err != nil {
		return err
	}
	defer v.Close()
	seen := map[string]stamp{}
	if err := update(ctx, v, dir, seen); err != nil {
		return err
	}
	log.Printf("version %d holds %s", v.Len(), dir)

	var seed [32]byte
	rand.Read(seed[:])
	reader, err := koloid.FromSeed(seed[:])
	if err != nil {
		return err
	}
	srv := kolosync.NewServer(func(peer, _ ed25519.PublicKey, _ kololog.LogID) kolosync.Access {
		if peer.Equal(reader.PublicKey()) {
			return kolosync.Keys
		}
		return kolosync.NoAccess
	})
	for _, l := range []*kolostore.Log{meta, content} {
		if err := srv.Offer(l); err != nil {
			return err
		}
	}
	ln, err := kolosecure.Listen(ctx, addr.String(), me, nil)
	if err != nil {
		return err
	}
	defer ln.Close()
	if addr, err = netip.ParseAddrPort(ln.Addr().String()); err != nil { // the port the system chose
		return err
	}

	c := shareCode{addr: addr, sender: me.PublicKey(), meta: v.ID(), seed: seed}
	fmt.Printf("share code:\n%s\n", c)
	log.Print("change files in the folder, then press Enter to publish them; Ctrl-C stops")
	var wg sync.WaitGroup
	defer wg.Wait()
	ctx, cancel := context.WithCancel(ctx) // ends the goroutines below whenever send returns
	defer cancel()
	wg.Go(func() {
		for {
			conn, err := ln.Accept(ctx)
			if err != nil {
				return
			}
			log.Printf("peer %x connected", conn.RemotePublicKey()[:4])
			wg.Go(func() {
				defer conn.Close()
				srv.Serve(ctx, conn)
				log.Printf("peer %x left", conn.RemotePublicKey()[:4])
			})
		}
	})
	enter := make(chan struct{})
	go func() {
		sc := bufio.NewScanner(os.Stdin)
		for sc.Scan() {
			enter <- struct{}{}
		}
	}()
	for {
		select {
		case <-ctx.Done():
			return nil
		case <-enter:
			before := v.Len()
			if err := update(ctx, v, dir, seen); err != nil {
				return err
			}
			if v.Len() == before {
				log.Print("nothing changed")
			} else {
				log.Printf("published version %d", v.Len())
			}
		}
	}
}

// stamp is what update compares to find a changed file: its size and modification time.
type stamp struct {
	dir  bool
	size int64
	mod  time.Time
}

// update makes the volume match dir: it adds new folders and files, writes changed files and deletes what is gone.
// seen holds the stamps of the last update and is brought up to date.
func update(ctx context.Context, v *kolofiles.Volume, dir string, seen map[string]stamp) error {
	now := map[string]stamp{}
	err := filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error {
		if errors.Is(err, fs.ErrNotExist) && p != dir {
			return nil // removed while we walked: the next update sees it gone
		}
		if err != nil || p == dir {
			return err
		}
		if !d.IsDir() && !d.Type().IsRegular() {
			return nil // links and devices are not shared
		}
		rel, err := filepath.Rel(dir, p)
		if err != nil {
			return err
		}
		name := filepath.ToSlash(rel)
		info, err := d.Info()
		if errors.Is(err, fs.ErrNotExist) {
			return nil
		} else if err != nil {
			return err
		}
		s := stamp{dir: true}
		if !d.IsDir() {
			s = stamp{size: info.Size(), mod: info.ModTime()}
		}
		old, ok := seen[name]
		if ok && old.dir != s.dir { // a file became a folder or the other way round
			if err := v.Delete(name); err != nil {
				return err
			}
			ok = false
		}
		switch {
		case ok && (s.dir || old == s):
		case s.dir:
			err = v.Mkdir(name)
		default:
			var f *os.File
			if f, err = os.Open(p); err == nil {
				err = v.WriteFile(ctx, name, f)
				f.Close()
			}
		}
		if err != nil {
			log.Printf("skipped %s: %v", name, err)
			if ok {
				now[name] = old // keep the published version; the next update tries again
			}
			if s.dir {
				return fs.SkipDir
			}
			return nil
		}
		now[name] = s
		return nil
	})
	if err != nil {
		return err
	}
	for name := range seen {
		if _, ok := now[name]; !ok {
			// A folder's delete takes everything below it, so a later delete of a child finds nothing,
			// or finds a file where its folder was.
			if err := v.Delete(name); err != nil && !errors.Is(err, fs.ErrNotExist) && !errors.Is(err, kolofiles.ErrNotDir) {
				return err
			}
		}
	}
	clear(seen)
	for name, s := range now {
		seen[name] = s
	}
	return nil
}

// get copies the sender's two logs, saves the folder, then pulls again every few seconds and saves each new version.
// It stops with errTooBig when the copy, or all the versions saved in out together, would hold more than maxBytes.
func get(ctx context.Context, code, out string, every time.Duration, maxBytes int64) error {
	code, err := readCode(code)
	if err != nil {
		return err
	}
	c, err := parseCode(code)
	if err != nil {
		return err
	}
	if _, err := os.Lstat(out); err == nil {
		return fmt.Errorf("%s already exists; choose a new name", out)
	}
	me, err := koloid.FromSeed(c.seed[:])
	if err != nil {
		return err
	}
	st, dir, cleanup, err := openStore()
	if err != nil {
		return err
	}
	defer cleanup()
	dctx, cancel := context.WithTimeout(ctx, dialTimeout)
	defer cancel()
	conn, err := kolosecure.Dial(dctx, c.addr.String(), me, c.sender, nil)
	if err != nil {
		return fmt.Errorf("cannot reach the sender at %s: %w", c.addr, err)
	}
	defer conn.Close()

	lctx, stopLimit := limit(ctx, dir, maxBytes)
	defer stopLimit()
	meta, err := replica(lctx, st, conn, c.sender, c.meta)
	if err != nil {
		return err
	}
	contentID, err := kolofiles.ContentLog(meta)
	if err != nil {
		return err
	}
	content, err := replica(lctx, st, conn, c.sender, contentID)
	if err != nil {
		return err
	}
	v, err := kolofiles.OpenReader(meta, koloblobs.NewLogBackend(content), nil)
	if err != nil {
		return err
	}
	defer v.Close()

	if err := os.MkdirAll(out, 0o700); err != nil { // the folders inside are private too
		return err
	}
	var saved uint64
	var written int64
	defer func() {
		if saved == 0 {
			os.Remove(out) // leaves nothing behind when no version was saved
		}
	}()
	for {
		if n := v.Len(); n > saved {
			if saved > 0 {
				changes, err := v.Diff(saved, n)
				if err != nil {
					return err
				}
				printChanges(changes)
			}
			size, err := treeSize(v)
			if err != nil {
				return err
			}
			if written += size; written > maxBytes {
				return errTooBig
			}
			dst := filepath.Join(out, fmt.Sprintf("v%d", n))
			if err := os.CopyFS(dst, v); err != nil {
				os.RemoveAll(dst) // no half-saved version is left behind
				return fmt.Errorf("save version %d: %w", n, err)
			}
			log.Printf("saved version %d in %s, every file checked against its blob ID", n, dst)
			saved = n
		}
		select {
		case <-ctx.Done():
			return nil
		case <-time.After(every):
		}
		// The metadata first: content pulled after it holds every object its ops name.
		if err := kolosync.Pull(lctx, conn, meta); err != nil {
			return fmt.Errorf("lost the sender: %w", tooBig(lctx, err))
		}
		if err := kolosync.Pull(lctx, conn, content); err != nil {
			return fmt.Errorf("lost the sender: %w", tooBig(lctx, err))
		}
		if err := v.Refresh(); err != nil {
			return err
		}
	}
}

// treeSize adds up the sizes of the files in fsys.
func treeSize(fsys fs.FS) (n int64, err error) {
	err = fs.WalkDir(fsys, ".", func(_ string, d fs.DirEntry, err error) error {
		if err != nil || d.IsDir() {
			return err
		}
		info, err := d.Info()
		if err == nil {
			n += info.Size()
		}
		return err
	})
	return n, err
}

// replica opens a local replica of the sender's log logID and pulls it.
func replica(ctx context.Context, st *kolostore.Store, conn *kolosecure.Conn, owner ed25519.PublicKey, logID kololog.LogID) (*kolostore.Replica, error) {
	header, err := kolosync.FetchHeader(ctx, conn, owner, logID)
	if errors.Is(err, kolosync.ErrNotFound) {
		return nil, fmt.Errorf("the sender does not share this folder with this code: %w", err)
	} else if err != nil {
		return nil, fmt.Errorf("ask the sender for its log: %w", err)
	}
	rep, err := st.OpenReplica(owner, logID, header)
	if err != nil {
		return nil, err
	}
	if err := kolosync.Pull(ctx, conn, rep); err != nil {
		return nil, fmt.Errorf("download: %w", err)
	}
	return rep, nil
}

// printChanges prints what Diff found between two versions.
func printChanges(changes []kolofiles.Change) {
	for _, c := range changes {
		switch {
		case c.Deleted:
			log.Print("  deleted a file or folder (its names are gone with it)")
		case c.Old == "":
			log.Printf("  added %s", c.New)
		case c.Old != c.New && c.Modified:
			log.Printf("  moved %s to %s and changed it", c.Old, c.New)
		case c.Old != c.New:
			log.Printf("  moved %s to %s", c.Old, c.New)
		default:
			log.Printf("  changed %s", c.New)
		}
	}
}

// openStore opens a kolo-store in a new temporary folder, removed by the returned function.
func openStore() (*kolostore.Store, string, func(), error) {
	dir, err := os.MkdirTemp("", "kolo-share-")
	if err != nil {
		return nil, "", nil, err
	}
	db, err := kolostore.OpenPebble(dir)
	if err != nil {
		os.RemoveAll(dir)
		return nil, "", nil, err
	}
	st := kolostore.New(db)
	return st, dir, func() {
		st.Close()
		os.RemoveAll(dir)
	}, nil
}

// errTooBig ends a get that would store more than -max-bytes.
var errTooBig = errors.New("the sender offers more than -max-bytes allows; only get codes from people you trust")

// limit returns a context that ends with errTooBig once the folder dir holds more than max bytes.
func limit(ctx context.Context, dir string, max int64) (context.Context, context.CancelFunc) {
	ctx, cancel := context.WithCancelCause(ctx)
	go func() {
		for {
			select {
			case <-ctx.Done():
				return
			case <-time.After(250 * time.Millisecond):
			}
			var n int64
			filepath.WalkDir(dir, func(_ string, d fs.DirEntry, err error) error {
				if err != nil || d.IsDir() {
					return nil
				}
				if info, err := d.Info(); err == nil {
					n += info.Size()
				}
				return nil
			})
			if n > max {
				cancel(errTooBig)
				return
			}
		}
	}()
	return ctx, func() { cancel(nil) }
}

// tooBig returns errTooBig when ctx ended because of limit, and err otherwise.
func tooBig(ctx context.Context, err error) error {
	if errors.Is(context.Cause(ctx), errTooBig) {
		return errTooBig
	}
	return err
}

// readCode returns code, or the first line of standard input when code is "-", so that it stays out of ps and the shell history.
func readCode(code string) (string, error) {
	if code != "-" {
		return code, nil
	}
	line, _ := bufio.NewReader(io.LimitReader(os.Stdin, 4096)).ReadString('\n')
	if line == "" {
		return "", errors.New("no share code on standard input")
	}
	return line, nil
}

// checkAddr refuses the addresses nobody can listen on, so that a share code cannot make get send packets to them.
func checkAddr(a netip.AddrPort) bool {
	ip := a.Addr()
	return a.Port() != 0 && !ip.IsUnspecified() && !ip.IsMulticast() && ip != netip.AddrFrom4([4]byte{255, 255, 255, 255})
}

// shareCode is all the receiver needs: where the sender listens, its public key, the volume's metadata log and
// the seed of the receiver's identity. The file keys travel inside the metadata log. Whoever has it can read the folder.
type shareCode struct {
	addr   netip.AddrPort
	sender ed25519.PublicKey
	meta   kololog.LogID
	seed   [32]byte
}

const (
	codePrefix = "kolo-folder:"
	codeSize   = 6 + 32 + kololog.LogIDSize + 32
)

func (c shareCode) String() string {
	ip := c.addr.Addr().As4()
	b := binary.BigEndian.AppendUint16(ip[:], c.addr.Port())
	b = append(b, c.sender...)
	b = append(b, c.meta[:]...)
	b = append(b, c.seed[:]...)
	return codePrefix + base64.RawURLEncoding.EncodeToString(b)
}

func parseCode(s string) (shareCode, error) {
	rest, ok := strings.CutPrefix(strings.TrimSpace(s), codePrefix)
	b, err := base64.RawURLEncoding.DecodeString(rest)
	if !ok || err != nil || len(b) != codeSize {
		return shareCode{}, fmt.Errorf("this is not a folder share code; copy the whole line, it starts with %q", codePrefix)
	}
	var c shareCode
	c.addr = netip.AddrPortFrom(netip.AddrFrom4([4]byte(b[:4])), binary.BigEndian.Uint16(b[4:6]))
	if !checkAddr(c.addr) {
		return shareCode{}, fmt.Errorf("the share code names %s, an address nobody can listen on", c.addr)
	}
	b = b[6:]
	c.sender = ed25519.PublicKey(b[:32])
	b = b[32:]
	b = b[copy(c.meta[:], b):]
	copy(c.seed[:], b)
	return c, nil
}

What is new compared with the first program:

  • send creates two logs and kolofiles.Create makes a volume from them. update walks the folder: it calls Mkdir for each new folder, WriteFile for each new or changed file, and Delete for what is gone. A file counts as changed when its size or modification time differs from the last pass, because WriteFile always adds a version, even for the same content.
  • Each time you press Enter, update runs again and the sender prints the new version, or nothing changed.
  • The share code is shorter: it names only the metadata log. The file keys are inside that log, encrypted by kolo-log, and travel with it.
  • get pulls the metadata log, finds the content log with kolofiles.ContentLog and pulls it too, then opens the volume with OpenReader. A Volume is an io/fs file system, so os.CopyFS saves it to disk, and every file it reads is checked against its blob ID. Like the first program, it refuses an OUT that already exists, and removes a version it could not save whole. OUT and the folders in it are created with mode 0700, so only you can read them.
  • get stops with the sender offers more than -max-bytes allows; only get codes from people you trust when what it downloads, or all the versions it wrote in OUT together, are more than -max-bytes (1 GiB by default; every version is a full copy). The code can be read from standard input with -, and a code that names an address nobody can listen on is refused, as in the first program.
  • Every 2 seconds (-every), get pulls both logs again, metadata first, and calls Refresh. When the version grew, Diff lists what changed and the new version is saved in a folder of its own, next to the older ones.
  • This program compares paths, so a renamed file reaches the receiver as a delete and an add. kolo-files has Move, which keeps the file's node and its history; an app that knows a file was renamed calls it.

11Share a folder and change it

In Terminal 1, make a folder with a few files:

bash
mkdir -p shared/notes && echo "buy bread" > shared/todo.txt && echo "first draft" > shared/notes/ideas.txt && cp hello.txt shared/

Share it:

Terminal
go run ./folder send shared
Text
21:07:02 version 5 holds shared
share code:
kolo-folder:fwAAAdrwpr17aqfOwmDq_IOx4WKK4QJkgD3tpVO6DthM9tPXLrBvR-B9CcWI5GObQ0v95GHhjronYzB1xx6FQDd5z9F-VhVeZaSEJgNRBoAf-ur59bcaY70ePv58tCTZpJtRxOdI
21:07:02 change files in the folder, then press Enter to publish them; Ctrl-C stops

Version 5 is the empty volume plus four ops: the files hello.txt, todo.txt and notes/ideas.txt, and the folder notes. In Terminal 2, receive it into a folder named copy:

Terminal
go run ./folder get '<share code>' copy
Text
21:07:02 saved version 5 in copy/v5, every file checked against its blob ID

Leave both running. Open a third terminal window in the kolo-share folder, or use your file manager, and change the shared folder: edit a file, add one and delete one.

bash
echo "buy bread and milk" > shared/todo.txt && echo "plans for the weekend" > shared/plans.txt && rm shared/hello.txt

Then press Enter in Terminal 1:

Text
21:07:06 published version 9

Four ops this time: a WRITE for todo.txt, a CREATE for plans.txt, and a REMOVE and a PURGE for the deleted file. Within 2 seconds, Terminal 2 prints what changed and saves the new version:

Text
21:07:06   deleted a file or folder (its names are gone with it)
21:07:06   added plans.txt
21:07:06   changed todo.txt
21:07:06 saved version 9 in copy/v9, every file checked against its blob ID

copy/v5 still holds the first version, with todo.txt saying buy bread, and copy/v9 holds the folder as it is now. A deleted file shows no name in the diff: its names, its key and its content are destroyed with it. Press Enter again in Terminal 1 without changing anything, and it prints nothing changed.

12Stop everything

Press Ctrl-C in Terminal 1. The sender stops and removes its temporary store. Terminal 2 notices at its next pull and stops too, keeping what it saved:

Text
21:07:18 lost the sender: kolosync: timeout: opening a stream
exit status 1

Between two computers

The two programs connect directly: the share code carries the sender's IPv4 address and port, and the receiver dials it. They do not use the DHT or the swarm, so there is no bootstrap to configure.

  • On one computer it works as it is: -listen defaults to 127.0.0.1:0.
  • On the same local network, the sender passes -listen before send, with the IPv4 address of its network card and a port, such as -listen 192.168.1.20:7450, and the receiver reaches it with the code. The address must be IPv4 and not 0.0.0.0; otherwise send stops with -listen needs an IPv4 address the receiver can reach.
  • Between two different networks it works only when the sender has a public IPv4 address that the receiver can reach, with the UDP port open in the firewall.
  • Two computers that are both behind home routers will not connect: these programs do not punch through NAT and use no relay.

For that case use the Kolo examples (chat, share, meet, board) or the kolo-swarm library, which find peers through the public entry nodes by default.

One file or a folder

Aspectfile (kolo-blobs)folder (kolo-files)
What is sharedthe bytes of one filea tree of files and folders, with their names
What the code carriesthe blob ID and the file keythe metadata log's ID; the file keys travel inside that log
A changea new file is a new blob and a new codeeach change is a new version of the same volume
Historynoneevery version stays readable, with Snapshot and Diff
Logs copiedonetwo: metadata, then content

Use kolo-blobs alone when your app already knows its files and only needs to store, send and check bytes, such as one attachment or one download. Use kolo-files when people share a folder that changes, need names and folders, or want to go back to an earlier version.

What just happened

All steps

Files on top of blobs

kolo-files decides names, folders and versions. kolo-blobs stores and checks the bytes of each version.

  1. A volume is two kolo-store logs of one owner: the metadata log, one op per change, and the content log, which holds kolo-blobs objects. Entry 0 of the metadata log names the content log.
  2. WriteFile gives the file's bytes to kolo-blobs under the file's own key. Its chunks and its root become entries of the content log, and the root's hash is the blob ID.
  3. It also writes an object table, a second blob under the same key, that lists the content log entry of each object, so a reader finds them without scanning the log.
  4. The CREATE op holds the name, the file key and the version: the blob ID, the table ID and where the table's own objects are. kolo-log encrypts every op.
  5. A new version is a WRITE op with a new blob and table. The key stays the same, so unchanged chunks are not stored again.
  • kolo-blobs cut each file into chunks, encrypted each chunk under keys derived from the file key, and named the file by the hash of its tree's root: the blob ID. See kolo-blobs.
  • The chunks and the root were entries of a kolo-store log. kolo-sync copied that log to the receiver, checking each entry against the owner's signed head. See Logs and Replicas.
  • The receiver then checked every chunk against the blob ID while it read, so a wrong byte would have stopped the read instead of reaching your disk.
  • For the folder, kolo-files kept one op per change in a metadata log and each file version as a blob in a content log, under one key per file. A version is the length of the metadata log, so the receiver could save each version and compare two with Diff. See kolo-files.
  • The connection was a kolo-secure channel tied to the sender's key, and the share code was the only thing passed by hand.

Next steps

  • Compare your files with the finished code in the kolo-share-example repository. Its tests run both programs on 127.0.0.1 and check each error shown here.
  • Read the kolo-blobs page: range reads, the formats, the limits, and how to read a blob from a pinning peer.
  • Read the kolo-files page: names, versions, selective sync and how Delete destroys a file.
  • The logs of both programs can also rest on a kolo-pin pinning peer, which holds only ciphertext, so a file stays reachable while your computer is off. The kolo-blobs and kolo-files pages explain how a reader reads from one.