In this tutorial you build two small programs and run each in two terminal windows on your computer. The first shares one file: Terminal 1 stores it and prints a share code, and Terminal 2 uses the code to fetch the file, check every chunk of it and save it. The second shares a whole folder and keeps the copy up to date: each time you change the folder and press Enter, the other terminal saves the new version and prints what changed.
The first program uses kolo-blobs, which stores
the bytes of one file. The second uses kolo-files,
which keeps files and folders, their names and every version on top of
kolo-blobs. Everything stays on your computer, on the address 127.0.0.1.
The finished code is in the
kolo-share-example
repository.
1Check what you need
You need Go 1.26 or newer and two terminal windows. If Go is not installed yet, install it for your system:
Download the macOS installer (the .pkg file) from
go.dev/dl and run it. If you use Homebrew, this works
too:
brew install goDownload the Windows installer (the .msi file) from
go.dev/dl and run it. It adds Go to your PATH. Then
open PowerShell.
Use your distribution's golang package if it is version 1.26 or newer.
Otherwise download the .tar.gz file from go.dev/dl
and follow the Linux steps on go.dev/doc/install.
Check your Go version:
go versionYou should see 1.26 or newer, for example:
go version go1.27.3 darwin/arm64Now open two terminal windows and keep them side by side:
Open the Terminal app and press Cmd+N for a second window.
Open PowerShell twice, or open Windows Terminal and press Ctrl+Shift+T for a second tab.
Open your terminal app and press Ctrl+Shift+N for a second window (Ctrl+Shift+T opens a tab instead in most terminals).
This tutorial calls them Terminal 1 (the peer that shares) and Terminal 2 (the peer that receives).
2Create a folder and a module
In Terminal 1, make a folder for the project and go into it:
mkdir kolo-share && cd kolo-sharemkdir kolo-share; cd kolo-sharemkdir kolo-share && cd kolo-shareThen create a Go module in it:
go mod init example.com/kolo-shareGo answers:
go: creating new go.mod: module example.com/kolo-share3Get the libraries
Still in Terminal 1, add the Kolo libraries the two programs use: kolo-blobs and kolo-files for the content, kolo-store to keep it in logs on disk, kolo-sync to copy those logs to the other peer, kolo-secure for the encrypted connection, kolo-id for each peer's key and kolo-log for the log IDs.
go get github.com/ciranda-dev/kolo-blobs@v0.1.0 github.com/ciranda-dev/kolo-files@v0.1.0 github.com/ciranda-dev/kolo-sync@v0.1.0 github.com/ciranda-dev/kolo-store@v0.1.0 github.com/ciranda-dev/kolo-secure@v0.1.0 github.com/ciranda-dev/kolo-id@v0.1.0 github.com/ciranda-dev/kolo-log@v0.1.0Go downloads them with what they need and prints one go: added line for
each module it adds to go.mod.
4Write the program that shares one file
Here is what the program does, in five steps:
One file, two terminals
The share code is the only thing you pass by hand. Whoever has it can read the file.
- file send writes the file as a blob into a new kolo-store log, under a new file key, and prints a share code.
- You copy the code to the other terminal: the sender's address and key, the log ID, a seed for the receiver's identity, the blob ID and the file key.
- file get turns the seed into its identity and connects with kolo-secure, which checks that the other side holds the sender's key.
- kolo-sync pulls the log. The sender serves it only to the identity from the code, and every entry is checked against the sender's signed head.
- kolo-blobs opens the blob with the ID and key from the code and checks every chunk while it saves the file. A file that fails a check is removed.
Make a folder named file inside kolo-share, create a file named main.go
in it and paste this program. It has two commands: send FILE stores the file
and serves it, and get CODE OUT fetches it with the share code and saves it
as OUT.
// Command file shares one file between two peers with kolo-blobs.
//
// file send FILE stores FILE as a blob, prints a share code and serves the blob
// file get CODE OUT fetches the blob with the code, checks every chunk and saves it as OUT
package main
import (
"bufio"
"context"
"crypto/ed25519"
"crypto/rand"
"crypto/sha256"
"encoding/base64"
"encoding/binary"
"errors"
"flag"
"fmt"
"io"
"io/fs"
"log"
"net/netip"
"os"
"os/signal"
"path/filepath"
"strings"
"sync"
"syscall"
"time"
koloblobs "github.com/ciranda-dev/kolo-blobs"
koloid "github.com/ciranda-dev/kolo-id"
kololog "github.com/ciranda-dev/kolo-log"
kolosecure "github.com/ciranda-dev/kolo-secure"
kolostore "github.com/ciranda-dev/kolo-store"
kolosync "github.com/ciranda-dev/kolo-sync"
)
const dialTimeout = 10 * time.Second
func main() {
listen := flag.String("listen", "127.0.0.1:0", "IPv4 IP:PORT that send listens on; the share code carries it")
maxBytes := flag.Int64("max-bytes", 1<<30, "most bytes get stores for a file")
flag.Usage = func() {
fmt.Fprintln(os.Stderr, "usage: file [-listen IP:PORT] send FILE\n file [-max-bytes N] get CODE OUT (CODE - reads the code from standard input)")
flag.PrintDefaults()
}
flag.Parse()
if *maxBytes <= 0 {
flag.Usage()
os.Exit(2)
}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM, syscall.SIGHUP)
defer stop()
log.SetFlags(log.Ltime)
var err error
switch args := flag.Args(); {
case len(args) == 2 && args[0] == "send":
err = send(ctx, *listen, args[1])
case len(args) == 3 && args[0] == "get":
err = get(ctx, args[1], args[2], *maxBytes)
default:
flag.Usage()
os.Exit(2)
}
if err != nil && ctx.Err() == nil {
log.Fatal(err)
}
}
// send stores the file as a blob in a new log and serves that log to whoever holds the share code.
func send(ctx context.Context, listen, name string) error {
addr, err := netip.ParseAddrPort(listen)
if err != nil || !addr.Addr().Is4() || addr.Addr().IsUnspecified() {
return fmt.Errorf("-listen needs an IPv4 address the receiver can reach, such as 127.0.0.1:0, not %q", listen)
}
f, err := os.Open(name)
if err != nil {
return err
}
defer f.Close()
if info, err := f.Stat(); err != nil {
return err
} else if !info.Mode().IsRegular() {
return fmt.Errorf("%s is not a regular file", name)
}
st, _, cleanup, err := openStore()
if err != nil {
return err
}
defer cleanup()
me, err := koloid.Generate()
if err != nil {
return err
}
l, err := st.Create(me)
if err != nil {
return err
}
key := koloblobs.NewKey()
sum := sha256.New()
id, err := koloblobs.New(koloblobs.NewLogBackend(l)).Write(ctx, key, io.TeeReader(f, sum))
if err != nil {
return fmt.Errorf("store %s: %w", name, err)
}
log.Printf("stored %s as blob %s in %d log entries", name, id.String()[:16], l.Len())
log.Printf("sha256 %x", sum.Sum(nil))
// The receiver's identity comes from a seed in the share code, so only the code's holder gets the log.
var seed [32]byte
rand.Read(seed[:])
reader, err := koloid.FromSeed(seed[:])
if err != nil {
return err
}
srv := kolosync.NewServer(func(peer, _ ed25519.PublicKey, _ kololog.LogID) kolosync.Access {
if peer.Equal(reader.PublicKey()) {
return kolosync.Keys
}
return kolosync.NoAccess
})
if err := srv.Offer(l); err != nil {
return err
}
ln, err := kolosecure.Listen(ctx, addr.String(), me, nil)
if err != nil {
return err
}
defer ln.Close()
if addr, err = netip.ParseAddrPort(ln.Addr().String()); err != nil { // the port the system chose
return err
}
c := shareCode{addr: addr, sender: me.PublicKey(), log: l.ID(), seed: seed, blob: id, key: key}
fmt.Printf("share code:\n%s\n", c)
log.Print("waiting for the receiver, press Ctrl-C to stop")
var wg sync.WaitGroup
defer wg.Wait()
ctx, cancel := context.WithCancel(ctx) // ends the connections below whenever send returns
defer cancel()
for {
conn, err := ln.Accept(ctx)
if err != nil {
return err
}
log.Printf("peer %x connected", conn.RemotePublicKey()[:4])
wg.Go(func() {
defer conn.Close()
srv.Serve(ctx, conn)
log.Printf("peer %x left", conn.RemotePublicKey()[:4])
})
}
}
// get copies the sender's log, then reads the blob from it and saves it, checking every chunk on the way.
// It stops with errTooBig when the copy would hold more than maxBytes.
func get(ctx context.Context, code, out string, maxBytes int64) error {
code, err := readCode(code)
if err != nil {
return err
}
c, err := parseCode(code)
if err != nil {
return err
}
if _, err := os.Lstat(out); err == nil {
return fmt.Errorf("%s already exists; choose a new name", out)
}
me, err := koloid.FromSeed(c.seed[:])
if err != nil {
return err
}
st, dir, cleanup, err := openStore()
if err != nil {
return err
}
defer cleanup()
dctx, cancel := context.WithTimeout(ctx, dialTimeout)
defer cancel()
conn, err := kolosecure.Dial(dctx, c.addr.String(), me, c.sender, nil)
if err != nil {
return fmt.Errorf("cannot reach the sender at %s: %w", c.addr, err)
}
defer conn.Close()
header, err := kolosync.FetchHeader(ctx, conn, c.sender, c.log)
if errors.Is(err, kolosync.ErrNotFound) {
return fmt.Errorf("the sender does not share this file with this code: %w", err)
} else if err != nil {
return fmt.Errorf("ask the sender for its log: %w", err)
}
rep, err := st.OpenReplica(c.sender, c.log, header)
if err != nil {
return err
}
log.Print("downloading")
lctx, stopLimit := limit(ctx, dir, maxBytes)
defer stopLimit()
if err := kolosync.Pull(lctx, conn, rep); err != nil {
return fmt.Errorf("download: %w", tooBig(lctx, err))
}
log.Printf("received %d log entries, each checked against the sender's signed head", rep.Len())
r, err := koloblobs.New(koloblobs.NewLogBackend(rep)).Open(ctx, c.blob, c.key)
if err != nil {
return explain(err)
}
defer r.Close()
if r.Size() > maxBytes {
return errTooBig
}
f, err := os.OpenFile(out, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o600) // never replaces a file
if err != nil {
return err
}
sum := sha256.New()
_, err = io.Copy(io.MultiWriter(f, sum), r)
if cerr := f.Close(); err == nil {
err = cerr
}
if err != nil {
os.Remove(out) // no half-written or unverified file is left behind
return explain(err)
}
log.Printf("saved %s: %d bytes, every chunk checked against the blob ID", out, r.Size())
log.Printf("sha256 %x", sum.Sum(nil))
return nil
}
// explain turns the errors of kolo-blobs into a sentence a user can act on.
func explain(err error) error {
switch {
case errors.Is(err, koloblobs.ErrWrongKey):
return fmt.Errorf("the key in the share code does not open this file: %w", err)
case errors.Is(err, koloblobs.ErrCorrupt):
return fmt.Errorf("the file failed verification, nothing was saved: %w", err)
case errors.Is(err, koloblobs.ErrNotFound):
return fmt.Errorf("the sender's log does not hold this file: %w", err)
}
return err
}
// openStore opens a kolo-store in a new temporary folder, removed by the returned function.
func openStore() (*kolostore.Store, string, func(), error) {
dir, err := os.MkdirTemp("", "kolo-share-")
if err != nil {
return nil, "", nil, err
}
db, err := kolostore.OpenPebble(dir)
if err != nil {
os.RemoveAll(dir)
return nil, "", nil, err
}
st := kolostore.New(db)
return st, dir, func() {
st.Close()
os.RemoveAll(dir)
}, nil
}
// errTooBig ends a get that would store more than -max-bytes.
var errTooBig = errors.New("the sender offers more than -max-bytes allows; only get codes from people you trust")
// limit returns a context that ends with errTooBig once the folder dir holds more than max bytes.
func limit(ctx context.Context, dir string, max int64) (context.Context, context.CancelFunc) {
ctx, cancel := context.WithCancelCause(ctx)
go func() {
for {
select {
case <-ctx.Done():
return
case <-time.After(250 * time.Millisecond):
}
var n int64
filepath.WalkDir(dir, func(_ string, d fs.DirEntry, err error) error {
if err != nil || d.IsDir() {
return nil
}
if info, err := d.Info(); err == nil {
n += info.Size()
}
return nil
})
if n > max {
cancel(errTooBig)
return
}
}
}()
return ctx, func() { cancel(nil) }
}
// tooBig returns errTooBig when ctx ended because of limit, and err otherwise.
func tooBig(ctx context.Context, err error) error {
if errors.Is(context.Cause(ctx), errTooBig) {
return errTooBig
}
return err
}
// readCode returns code, or the first line of standard input when code is "-", so that it stays out of ps and the shell history.
func readCode(code string) (string, error) {
if code != "-" {
return code, nil
}
line, _ := bufio.NewReader(io.LimitReader(os.Stdin, 4096)).ReadString('\n')
if line == "" {
return "", errors.New("no share code on standard input")
}
return line, nil
}
// checkAddr refuses the addresses nobody can listen on, so that a share code cannot make get send packets to them.
func checkAddr(a netip.AddrPort) bool {
ip := a.Addr()
return a.Port() != 0 && !ip.IsUnspecified() && !ip.IsMulticast() && ip != netip.AddrFrom4([4]byte{255, 255, 255, 255})
}
// shareCode is all the receiver needs: where the sender listens, its public key, the log holding the blob,
// the seed of the receiver's identity, and the blob's ID and key. Whoever has it can read the file.
type shareCode struct {
addr netip.AddrPort
sender ed25519.PublicKey
log kololog.LogID
seed [32]byte
blob koloblobs.ID
key koloblobs.Key
}
const (
codePrefix = "kolo-file:"
codeSize = 6 + 32 + kololog.LogIDSize + 32 + koloblobs.HashSize + koloblobs.KeySize
)
func (c shareCode) String() string {
ip := c.addr.Addr().As4()
b := binary.BigEndian.AppendUint16(ip[:], c.addr.Port())
b = append(b, c.sender...)
b = append(b, c.log[:]...)
b = append(b, c.seed[:]...)
b = append(b, c.blob[:]...)
b = append(b, c.key[:]...)
return codePrefix + base64.RawURLEncoding.EncodeToString(b)
}
func parseCode(s string) (shareCode, error) {
rest, ok := strings.CutPrefix(strings.TrimSpace(s), codePrefix)
b, err := base64.RawURLEncoding.DecodeString(rest)
if !ok || err != nil || len(b) != codeSize {
return shareCode{}, fmt.Errorf("this is not a file share code; copy the whole line, it starts with %q", codePrefix)
}
var c shareCode
c.addr = netip.AddrPortFrom(netip.AddrFrom4([4]byte(b[:4])), binary.BigEndian.Uint16(b[4:6]))
if !checkAddr(c.addr) {
return shareCode{}, fmt.Errorf("the share code names %s, an address nobody can listen on", c.addr)
}
b = b[6:]
c.sender = ed25519.PublicKey(b[:32])
b = b[32:]
b = b[copy(c.log[:], b):]
b = b[copy(c.seed[:], b):]
b = b[copy(c.blob[:], b):]
copy(c.key[:], b)
return c, nil
}What each part does:
sendcreates a new identity for this run withkoloid.Generateand a kolo-store log in a new temporary folder (openStore), which is removed when the program stops.koloblobs.NewKeymakes a random key for this file, andWritecuts the file into chunks, encrypts each one and appends them to the log as entries. It returns the blob ID. The program also computes the file's SHA-256 while it reads, withio.TeeReader, so you can compare it with the copy later.- The share code holds everything the receiver needs: the address and public
key of the sender, the ID of the log, the blob ID, the file key, and a random
seed. The receiver turns the seed into its identity with
koloid.FromSeed. The kolo-sync server givesKeysaccess only to that identity, so a peer without the code gets nothing. getconnects withkolosecure.Dial, which checks that the other side holds the sender's key.kolosync.FetchHeaderandOpenReplicamake a local copy of the sender's log, andkolosync.Pullfills it, checking every entry against the sender's signed head.Openchecks the blob's root against the ID and the key, andio.Copyreads the file chunk by chunk; each chunk is checked against the tree before its bytes are written. If anything fails, the program removes the output file, also when you interrupt it, so a half-written or unverified file is never left behind. The file is created with mode 0600: only you can read it.getalso keeps itself safe against a code that comes from someone else. It stops withthe sender offers more than -max-bytes allows; only get codes from people you trustwhen what it downloads (the temporary store) or what it writes is more than-max-bytes, 1 GiB by default (for examplego run ./file -max-bytes 100000000 get CODE copy.jpg). A code that names0.0.0.0, a multicast address, the broadcast address or port 0 is refused (the share code names ADDR, an address nobody can listen on). A code is an address plus a secret, so use only codes from people you trust. The code can be read from standard input with-, so it stays out ofpsand the shell history:echo CODE | go run ./file get - copy.txt.explainturns the errors of kolo-blobs into sentences: a wrong key (ErrWrongKey), an object that fails a check (ErrCorrupt) or a missing one (ErrNotFound).-listenis the address the sender listens on,127.0.0.1:0by default: your computer only, on a port the system picks. The code carries that address, so it must be an IPv4 address the receiver can reach.
5Make a file to share
In Terminal 1, in the kolo-share folder, make a small text file:
echo "hello from terminal 1" > hello.txtSet-Content hello.txt "hello from terminal 1"echo "hello from terminal 1" > hello.txtAny file works: a photo, a video, an archive. A file of any size is stored as many chunks as it needs.
6Share it from Terminal 1
In Terminal 1, run:
go run ./file send hello.txtThe first run compiles for a moment. Then it stores the file, prints its SHA-256 and the share code, and waits. Your times, IDs, hash and code will differ:
21:06:38 stored hello.txt as blob a9d85e576f8d1489 in 2 log entries
21:06:38 sha256 2b3bbe311b98a8d367296b62fcb806cb076ccddb893f9f02dbd0c327c98e2385
share code:
kolo-file:fwAAAfOct7bUTbr73o5USjYMV_QoCysEQPbcwAuLL6DkJGdBSUQglI3Xz74siUO8VhQOnKqBB_BVw5FKd-VlHrXEjaeUYdYkEK2038sz_yilgROh560Mz7emkvCv0wzsYlDWEg5KqdheV2-NFImL7UW_f-0M4Q-5I9a4D2Tiy1BSEnQPJUFePQEC6hq_trv_ybEglewb2IyH6o9exzlkNklfTSSOAw
21:06:38 waiting for the receiver, press Ctrl-C to stopA small file takes 2 log entries: one chunk and the root of its tree. Select
the whole line that starts with kolo-file: and copy it. Leave Terminal 1
running.
7Receive it in Terminal 2
In Terminal 2, go to the same kolo-share folder (use the path where you
created it):
cd kolo-shareThen run get with the code you copied, in single quotes, and a name for the
copy:
go run ./file get '<share code>' copy.txtReplace <share code> with the whole code, kolo-file: included. Terminal 2
prints:
21:06:38 downloading
21:06:38 received 2 log entries, each checked against the sender's signed head
21:06:38 saved copy.txt: 22 bytes, every chunk checked against the blob ID
21:06:38 sha256 2b3bbe311b98a8d367296b62fcb806cb076ccddb893f9f02dbd0c327c98e2385And Terminal 1 shows the receiver come and go, under the name of its key:
21:06:38 peer db309a79 connected
21:06:38 peer db309a79 left8Check that the copy is the same
The program already checked every chunk against the blob ID, and the two
sha256 lines, one in each terminal, are the same. You can also ask your
system for the SHA-256 of both files, in Terminal 2:
shasum -a 256 hello.txt copy.txtGet-FileHash hello.txt, copy.txtsha256sum hello.txt copy.txtBoth lines carry the same hash:
2b3bbe311b98a8d367296b62fcb806cb076ccddb893f9f02dbd0c327c98e2385 hello.txt
2b3bbe311b98a8d367296b62fcb806cb076ccddb893f9f02dbd0c327c98e2385 copy.txtGet-FileHash on Windows prints a table with the hash in capitals. There,
Set-Content also ends the line with two characters, so the file has 23 bytes
and another hash; the two files still match each other.
9See what happens when something goes wrong
Run get again with the same name. It never replaces a file, and it removes the
partial file when it fails or is interrupted:
go run ./file get '<share code>' copy.txt21:06:39 copy.txt already exists; choose a new name
exit status 1The last line comes from go run: the program exited with an error. Now give
it something that is not a code:
go run ./file get 'kolo-file:abc' other.txt21:06:39 this is not a file share code; copy the whole line, it starts with "kolo-file:"
exit status 1Press Ctrl-C in Terminal 1 to stop the sender, then run get once more
with the real code and a new name, in Terminal 2:
go run ./file get '<share code>' late.txtAfter a few seconds:
21:06:53 cannot reach the sender at 127.0.0.1:62364: kolotransport: timeout
exit status 1late.txt is not created. A code whose key was changed fails with the key in the share code does not open this file, and a code whose seed was changed with
the sender does not share this file with this code: the repository's tests
check both.
10Write the program that shares a folder
The second program shares a folder with kolo-files. The folder becomes a volume: a metadata log with one op per change, and a content log that holds each file's content as a kolo-blobs blob. Each op is a new version.
A folder and its versions
The receiver saves each version in a folder of its own, so older versions stay as they were.
- folder send puts every file and folder in a new kolo-files volume and prints a share code. No file key is in the code: the keys travel inside the metadata log.
- You copy the code to the other terminal: the sender's address and key, the metadata log ID and a seed for the receiver's identity.
- folder get pulls the metadata log, finds the content log in its first entry and pulls that too, then saves the volume with os.CopyFS in OUT/v and the version number, such as OUT/v5.
- Press Enter in Terminal 1: new files and folders become CREATE and MKDIR ops, changed files WRITE ops, deleted ones REMOVE and PURGE ops. Each op is a new version.
- Every 2 seconds the receiver pulls the new entries only, refreshes the volume, prints Diff between the saved version and the new one, and saves the new one.
Make a folder named folder inside kolo-share, create main.go in it and
paste this program:
// Command folder shares a folder between two peers with kolo-files and keeps the copy up to date.
//
// folder send DIR puts DIR in a volume, prints a share code and publishes changes each time you press Enter
// folder get CODE OUT saves the folder into OUT/v<version>, then each new version the sender publishes
package main
import (
"bufio"
"context"
"crypto/ed25519"
"crypto/rand"
"encoding/base64"
"encoding/binary"
"errors"
"flag"
"fmt"
"io"
"io/fs"
"log"
"net/netip"
"os"
"os/signal"
"path/filepath"
"strings"
"sync"
"syscall"
"time"
koloblobs "github.com/ciranda-dev/kolo-blobs"
kolofiles "github.com/ciranda-dev/kolo-files"
koloid "github.com/ciranda-dev/kolo-id"
kololog "github.com/ciranda-dev/kolo-log"
kolosecure "github.com/ciranda-dev/kolo-secure"
kolostore "github.com/ciranda-dev/kolo-store"
kolosync "github.com/ciranda-dev/kolo-sync"
)
const dialTimeout = 10 * time.Second
func main() {
listen := flag.String("listen", "127.0.0.1:0", "IPv4 IP:PORT that send listens on; the share code carries it")
every := flag.Duration("every", 2*time.Second, "how often get asks the sender for a new version")
maxBytes := flag.Int64("max-bytes", 1<<30, "most bytes get stores, and writes into OUT in all")
flag.Usage = func() {
fmt.Fprintln(os.Stderr, "usage: folder [-listen IP:PORT] send DIR\n folder [-every 2s] [-max-bytes N] get CODE OUT (CODE - reads the code from standard input)")
flag.PrintDefaults()
}
flag.Parse()
if *every <= 0 || *maxBytes <= 0 {
flag.Usage()
os.Exit(2)
}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM, syscall.SIGHUP)
defer stop()
log.SetFlags(log.Ltime)
var err error
switch args := flag.Args(); {
case len(args) == 2 && args[0] == "send":
err = send(ctx, *listen, args[1])
case len(args) == 3 && args[0] == "get":
err = get(ctx, args[1], args[2], *every, *maxBytes)
default:
flag.Usage()
os.Exit(2)
}
if err != nil && ctx.Err() == nil {
log.Fatal(err)
}
}
// send keeps dir in a volume, serves its two logs to whoever holds the share code and adds a version on Enter.
func send(ctx context.Context, listen, dir string) error {
addr, err := netip.ParseAddrPort(listen)
if err != nil || !addr.Addr().Is4() || addr.Addr().IsUnspecified() {
return fmt.Errorf("-listen needs an IPv4 address the receiver can reach, such as 127.0.0.1:0, not %q", listen)
}
if info, err := os.Lstat(dir); err != nil {
return err
} else if !info.IsDir() {
return fmt.Errorf("%s is not a folder", dir)
}
st, _, cleanup, err := openStore()
if err != nil {
return err
}
defer cleanup()
me, err := koloid.Generate()
if err != nil {
return err
}
meta, err := st.Create(me)
if err != nil {
return err
}
content, err := st.Create(me)
if err != nil {
return err
}
v, err := kolofiles.Create(meta, content)
if err != nil {
return err
}
defer v.Close()
seen := map[string]stamp{}
if err := update(ctx, v, dir, seen); err != nil {
return err
}
log.Printf("version %d holds %s", v.Len(), dir)
var seed [32]byte
rand.Read(seed[:])
reader, err := koloid.FromSeed(seed[:])
if err != nil {
return err
}
srv := kolosync.NewServer(func(peer, _ ed25519.PublicKey, _ kololog.LogID) kolosync.Access {
if peer.Equal(reader.PublicKey()) {
return kolosync.Keys
}
return kolosync.NoAccess
})
for _, l := range []*kolostore.Log{meta, content} {
if err := srv.Offer(l); err != nil {
return err
}
}
ln, err := kolosecure.Listen(ctx, addr.String(), me, nil)
if err != nil {
return err
}
defer ln.Close()
if addr, err = netip.ParseAddrPort(ln.Addr().String()); err != nil { // the port the system chose
return err
}
c := shareCode{addr: addr, sender: me.PublicKey(), meta: v.ID(), seed: seed}
fmt.Printf("share code:\n%s\n", c)
log.Print("change files in the folder, then press Enter to publish them; Ctrl-C stops")
var wg sync.WaitGroup
defer wg.Wait()
ctx, cancel := context.WithCancel(ctx) // ends the goroutines below whenever send returns
defer cancel()
wg.Go(func() {
for {
conn, err := ln.Accept(ctx)
if err != nil {
return
}
log.Printf("peer %x connected", conn.RemotePublicKey()[:4])
wg.Go(func() {
defer conn.Close()
srv.Serve(ctx, conn)
log.Printf("peer %x left", conn.RemotePublicKey()[:4])
})
}
})
enter := make(chan struct{})
go func() {
sc := bufio.NewScanner(os.Stdin)
for sc.Scan() {
enter <- struct{}{}
}
}()
for {
select {
case <-ctx.Done():
return nil
case <-enter:
before := v.Len()
if err := update(ctx, v, dir, seen); err != nil {
return err
}
if v.Len() == before {
log.Print("nothing changed")
} else {
log.Printf("published version %d", v.Len())
}
}
}
}
// stamp is what update compares to find a changed file: its size and modification time.
type stamp struct {
dir bool
size int64
mod time.Time
}
// update makes the volume match dir: it adds new folders and files, writes changed files and deletes what is gone.
// seen holds the stamps of the last update and is brought up to date.
func update(ctx context.Context, v *kolofiles.Volume, dir string, seen map[string]stamp) error {
now := map[string]stamp{}
err := filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error {
if errors.Is(err, fs.ErrNotExist) && p != dir {
return nil // removed while we walked: the next update sees it gone
}
if err != nil || p == dir {
return err
}
if !d.IsDir() && !d.Type().IsRegular() {
return nil // links and devices are not shared
}
rel, err := filepath.Rel(dir, p)
if err != nil {
return err
}
name := filepath.ToSlash(rel)
info, err := d.Info()
if errors.Is(err, fs.ErrNotExist) {
return nil
} else if err != nil {
return err
}
s := stamp{dir: true}
if !d.IsDir() {
s = stamp{size: info.Size(), mod: info.ModTime()}
}
old, ok := seen[name]
if ok && old.dir != s.dir { // a file became a folder or the other way round
if err := v.Delete(name); err != nil {
return err
}
ok = false
}
switch {
case ok && (s.dir || old == s):
case s.dir:
err = v.Mkdir(name)
default:
var f *os.File
if f, err = os.Open(p); err == nil {
err = v.WriteFile(ctx, name, f)
f.Close()
}
}
if err != nil {
log.Printf("skipped %s: %v", name, err)
if ok {
now[name] = old // keep the published version; the next update tries again
}
if s.dir {
return fs.SkipDir
}
return nil
}
now[name] = s
return nil
})
if err != nil {
return err
}
for name := range seen {
if _, ok := now[name]; !ok {
// A folder's delete takes everything below it, so a later delete of a child finds nothing,
// or finds a file where its folder was.
if err := v.Delete(name); err != nil && !errors.Is(err, fs.ErrNotExist) && !errors.Is(err, kolofiles.ErrNotDir) {
return err
}
}
}
clear(seen)
for name, s := range now {
seen[name] = s
}
return nil
}
// get copies the sender's two logs, saves the folder, then pulls again every few seconds and saves each new version.
// It stops with errTooBig when the copy, or all the versions saved in out together, would hold more than maxBytes.
func get(ctx context.Context, code, out string, every time.Duration, maxBytes int64) error {
code, err := readCode(code)
if err != nil {
return err
}
c, err := parseCode(code)
if err != nil {
return err
}
if _, err := os.Lstat(out); err == nil {
return fmt.Errorf("%s already exists; choose a new name", out)
}
me, err := koloid.FromSeed(c.seed[:])
if err != nil {
return err
}
st, dir, cleanup, err := openStore()
if err != nil {
return err
}
defer cleanup()
dctx, cancel := context.WithTimeout(ctx, dialTimeout)
defer cancel()
conn, err := kolosecure.Dial(dctx, c.addr.String(), me, c.sender, nil)
if err != nil {
return fmt.Errorf("cannot reach the sender at %s: %w", c.addr, err)
}
defer conn.Close()
lctx, stopLimit := limit(ctx, dir, maxBytes)
defer stopLimit()
meta, err := replica(lctx, st, conn, c.sender, c.meta)
if err != nil {
return err
}
contentID, err := kolofiles.ContentLog(meta)
if err != nil {
return err
}
content, err := replica(lctx, st, conn, c.sender, contentID)
if err != nil {
return err
}
v, err := kolofiles.OpenReader(meta, koloblobs.NewLogBackend(content), nil)
if err != nil {
return err
}
defer v.Close()
if err := os.MkdirAll(out, 0o700); err != nil { // the folders inside are private too
return err
}
var saved uint64
var written int64
defer func() {
if saved == 0 {
os.Remove(out) // leaves nothing behind when no version was saved
}
}()
for {
if n := v.Len(); n > saved {
if saved > 0 {
changes, err := v.Diff(saved, n)
if err != nil {
return err
}
printChanges(changes)
}
size, err := treeSize(v)
if err != nil {
return err
}
if written += size; written > maxBytes {
return errTooBig
}
dst := filepath.Join(out, fmt.Sprintf("v%d", n))
if err := os.CopyFS(dst, v); err != nil {
os.RemoveAll(dst) // no half-saved version is left behind
return fmt.Errorf("save version %d: %w", n, err)
}
log.Printf("saved version %d in %s, every file checked against its blob ID", n, dst)
saved = n
}
select {
case <-ctx.Done():
return nil
case <-time.After(every):
}
// The metadata first: content pulled after it holds every object its ops name.
if err := kolosync.Pull(lctx, conn, meta); err != nil {
return fmt.Errorf("lost the sender: %w", tooBig(lctx, err))
}
if err := kolosync.Pull(lctx, conn, content); err != nil {
return fmt.Errorf("lost the sender: %w", tooBig(lctx, err))
}
if err := v.Refresh(); err != nil {
return err
}
}
}
// treeSize adds up the sizes of the files in fsys.
func treeSize(fsys fs.FS) (n int64, err error) {
err = fs.WalkDir(fsys, ".", func(_ string, d fs.DirEntry, err error) error {
if err != nil || d.IsDir() {
return err
}
info, err := d.Info()
if err == nil {
n += info.Size()
}
return err
})
return n, err
}
// replica opens a local replica of the sender's log logID and pulls it.
func replica(ctx context.Context, st *kolostore.Store, conn *kolosecure.Conn, owner ed25519.PublicKey, logID kololog.LogID) (*kolostore.Replica, error) {
header, err := kolosync.FetchHeader(ctx, conn, owner, logID)
if errors.Is(err, kolosync.ErrNotFound) {
return nil, fmt.Errorf("the sender does not share this folder with this code: %w", err)
} else if err != nil {
return nil, fmt.Errorf("ask the sender for its log: %w", err)
}
rep, err := st.OpenReplica(owner, logID, header)
if err != nil {
return nil, err
}
if err := kolosync.Pull(ctx, conn, rep); err != nil {
return nil, fmt.Errorf("download: %w", err)
}
return rep, nil
}
// printChanges prints what Diff found between two versions.
func printChanges(changes []kolofiles.Change) {
for _, c := range changes {
switch {
case c.Deleted:
log.Print(" deleted a file or folder (its names are gone with it)")
case c.Old == "":
log.Printf(" added %s", c.New)
case c.Old != c.New && c.Modified:
log.Printf(" moved %s to %s and changed it", c.Old, c.New)
case c.Old != c.New:
log.Printf(" moved %s to %s", c.Old, c.New)
default:
log.Printf(" changed %s", c.New)
}
}
}
// openStore opens a kolo-store in a new temporary folder, removed by the returned function.
func openStore() (*kolostore.Store, string, func(), error) {
dir, err := os.MkdirTemp("", "kolo-share-")
if err != nil {
return nil, "", nil, err
}
db, err := kolostore.OpenPebble(dir)
if err != nil {
os.RemoveAll(dir)
return nil, "", nil, err
}
st := kolostore.New(db)
return st, dir, func() {
st.Close()
os.RemoveAll(dir)
}, nil
}
// errTooBig ends a get that would store more than -max-bytes.
var errTooBig = errors.New("the sender offers more than -max-bytes allows; only get codes from people you trust")
// limit returns a context that ends with errTooBig once the folder dir holds more than max bytes.
func limit(ctx context.Context, dir string, max int64) (context.Context, context.CancelFunc) {
ctx, cancel := context.WithCancelCause(ctx)
go func() {
for {
select {
case <-ctx.Done():
return
case <-time.After(250 * time.Millisecond):
}
var n int64
filepath.WalkDir(dir, func(_ string, d fs.DirEntry, err error) error {
if err != nil || d.IsDir() {
return nil
}
if info, err := d.Info(); err == nil {
n += info.Size()
}
return nil
})
if n > max {
cancel(errTooBig)
return
}
}
}()
return ctx, func() { cancel(nil) }
}
// tooBig returns errTooBig when ctx ended because of limit, and err otherwise.
func tooBig(ctx context.Context, err error) error {
if errors.Is(context.Cause(ctx), errTooBig) {
return errTooBig
}
return err
}
// readCode returns code, or the first line of standard input when code is "-", so that it stays out of ps and the shell history.
func readCode(code string) (string, error) {
if code != "-" {
return code, nil
}
line, _ := bufio.NewReader(io.LimitReader(os.Stdin, 4096)).ReadString('\n')
if line == "" {
return "", errors.New("no share code on standard input")
}
return line, nil
}
// checkAddr refuses the addresses nobody can listen on, so that a share code cannot make get send packets to them.
func checkAddr(a netip.AddrPort) bool {
ip := a.Addr()
return a.Port() != 0 && !ip.IsUnspecified() && !ip.IsMulticast() && ip != netip.AddrFrom4([4]byte{255, 255, 255, 255})
}
// shareCode is all the receiver needs: where the sender listens, its public key, the volume's metadata log and
// the seed of the receiver's identity. The file keys travel inside the metadata log. Whoever has it can read the folder.
type shareCode struct {
addr netip.AddrPort
sender ed25519.PublicKey
meta kololog.LogID
seed [32]byte
}
const (
codePrefix = "kolo-folder:"
codeSize = 6 + 32 + kololog.LogIDSize + 32
)
func (c shareCode) String() string {
ip := c.addr.Addr().As4()
b := binary.BigEndian.AppendUint16(ip[:], c.addr.Port())
b = append(b, c.sender...)
b = append(b, c.meta[:]...)
b = append(b, c.seed[:]...)
return codePrefix + base64.RawURLEncoding.EncodeToString(b)
}
func parseCode(s string) (shareCode, error) {
rest, ok := strings.CutPrefix(strings.TrimSpace(s), codePrefix)
b, err := base64.RawURLEncoding.DecodeString(rest)
if !ok || err != nil || len(b) != codeSize {
return shareCode{}, fmt.Errorf("this is not a folder share code; copy the whole line, it starts with %q", codePrefix)
}
var c shareCode
c.addr = netip.AddrPortFrom(netip.AddrFrom4([4]byte(b[:4])), binary.BigEndian.Uint16(b[4:6]))
if !checkAddr(c.addr) {
return shareCode{}, fmt.Errorf("the share code names %s, an address nobody can listen on", c.addr)
}
b = b[6:]
c.sender = ed25519.PublicKey(b[:32])
b = b[32:]
b = b[copy(c.meta[:], b):]
copy(c.seed[:], b)
return c, nil
}What is new compared with the first program:
sendcreates two logs andkolofiles.Createmakes a volume from them.updatewalks the folder: it callsMkdirfor each new folder,WriteFilefor each new or changed file, andDeletefor what is gone. A file counts as changed when its size or modification time differs from the last pass, becauseWriteFilealways adds a version, even for the same content.- Each time you press Enter,
updateruns again and the sender prints the new version, ornothing changed. - The share code is shorter: it names only the metadata log. The file keys are inside that log, encrypted by kolo-log, and travel with it.
getpulls the metadata log, finds the content log withkolofiles.ContentLogand pulls it too, then opens the volume withOpenReader. AVolumeis anio/fsfile system, soos.CopyFSsaves it to disk, and every file it reads is checked against its blob ID. Like the first program, it refuses anOUTthat already exists, and removes a version it could not save whole.OUTand the folders in it are created with mode 0700, so only you can read them.getstops withthe sender offers more than -max-bytes allows; only get codes from people you trustwhen what it downloads, or all the versions it wrote inOUTtogether, are more than-max-bytes(1 GiB by default; every version is a full copy). The code can be read from standard input with-, and a code that names an address nobody can listen on is refused, as in the first program.- Every 2 seconds (
-every),getpulls both logs again, metadata first, and callsRefresh. When the version grew,Difflists what changed and the new version is saved in a folder of its own, next to the older ones. - This program compares paths, so a renamed file reaches the receiver as a
delete and an add. kolo-files has
Move, which keeps the file's node and its history; an app that knows a file was renamed calls it.
11Share a folder and change it
In Terminal 1, make a folder with a few files:
mkdir -p shared/notes && echo "buy bread" > shared/todo.txt && echo "first draft" > shared/notes/ideas.txt && cp hello.txt shared/mkdir shared\notes; Set-Content shared\todo.txt "buy bread"; Set-Content shared\notes\ideas.txt "first draft"; Copy-Item hello.txt shared\mkdir -p shared/notes && echo "buy bread" > shared/todo.txt && echo "first draft" > shared/notes/ideas.txt && cp hello.txt shared/Share it:
go run ./folder send shared21:07:02 version 5 holds shared
share code:
kolo-folder:fwAAAdrwpr17aqfOwmDq_IOx4WKK4QJkgD3tpVO6DthM9tPXLrBvR-B9CcWI5GObQ0v95GHhjronYzB1xx6FQDd5z9F-VhVeZaSEJgNRBoAf-ur59bcaY70ePv58tCTZpJtRxOdI
21:07:02 change files in the folder, then press Enter to publish them; Ctrl-C stopsVersion 5 is the empty volume plus four ops: the files hello.txt, todo.txt
and notes/ideas.txt, and the folder notes. In Terminal 2, receive it
into a folder named copy:
go run ./folder get '<share code>' copy21:07:02 saved version 5 in copy/v5, every file checked against its blob IDLeave both running. Open a third terminal window in the kolo-share folder,
or use your file manager, and change the shared folder: edit a file, add one
and delete one.
echo "buy bread and milk" > shared/todo.txt && echo "plans for the weekend" > shared/plans.txt && rm shared/hello.txtSet-Content shared\todo.txt "buy bread and milk"; Set-Content shared\plans.txt "plans for the weekend"; Remove-Item shared\hello.txtecho "buy bread and milk" > shared/todo.txt && echo "plans for the weekend" > shared/plans.txt && rm shared/hello.txtThen press Enter in Terminal 1:
21:07:06 published version 9Four ops this time: a WRITE for todo.txt, a CREATE for plans.txt, and a
REMOVE and a PURGE for the deleted file. Within 2 seconds, Terminal 2
prints what changed and saves the new version:
21:07:06 deleted a file or folder (its names are gone with it)
21:07:06 added plans.txt
21:07:06 changed todo.txt
21:07:06 saved version 9 in copy/v9, every file checked against its blob IDcopy/v5 still holds the first version, with todo.txt saying buy bread,
and copy/v9 holds the folder as it is now. A deleted file shows no name in
the diff: its names, its key and its content are destroyed with it. Press Enter
again in Terminal 1 without changing anything, and it prints
nothing changed.
12Stop everything
Press Ctrl-C in Terminal 1. The sender stops and removes its temporary store. Terminal 2 notices at its next pull and stops too, keeping what it saved:
21:07:18 lost the sender: kolosync: timeout: opening a stream
exit status 1Between two computers
The two programs connect directly: the share code carries the sender's IPv4 address and port, and the receiver dials it. They do not use the DHT or the swarm, so there is no bootstrap to configure.
- On one computer it works as it is:
-listendefaults to127.0.0.1:0. - On the same local network, the sender passes
-listenbeforesend, with the IPv4 address of its network card and a port, such as-listen 192.168.1.20:7450, and the receiver reaches it with the code. The address must be IPv4 and not0.0.0.0; otherwisesendstops with-listen needs an IPv4 address the receiver can reach. - Between two different networks it works only when the sender has a public IPv4 address that the receiver can reach, with the UDP port open in the firewall.
- Two computers that are both behind home routers will not connect: these programs do not punch through NAT and use no relay.
For that case use the Kolo examples (chat, share, meet, board) or the kolo-swarm library, which find peers through the public entry nodes by default.
One file or a folder
| Aspect | file (kolo-blobs) | folder (kolo-files) |
|---|---|---|
| What is shared | the bytes of one file | a tree of files and folders, with their names |
| What the code carries | the blob ID and the file key | the metadata log's ID; the file keys travel inside that log |
| A change | a new file is a new blob and a new code | each change is a new version of the same volume |
| History | none | every version stays readable, with Snapshot and Diff |
| Logs copied | one | two: metadata, then content |
Use kolo-blobs alone when your app already knows its files and only needs to store, send and check bytes, such as one attachment or one download. Use kolo-files when people share a folder that changes, need names and folders, or want to go back to an earlier version.
What just happened
Files on top of blobs
kolo-files decides names, folders and versions. kolo-blobs stores and checks the bytes of each version.
- A volume is two kolo-store logs of one owner: the metadata log, one op per change, and the content log, which holds kolo-blobs objects. Entry 0 of the metadata log names the content log.
- WriteFile gives the file's bytes to kolo-blobs under the file's own key. Its chunks and its root become entries of the content log, and the root's hash is the blob ID.
- It also writes an object table, a second blob under the same key, that lists the content log entry of each object, so a reader finds them without scanning the log.
- The CREATE op holds the name, the file key and the version: the blob ID, the table ID and where the table's own objects are. kolo-log encrypts every op.
- A new version is a WRITE op with a new blob and table. The key stays the same, so unchanged chunks are not stored again.
- kolo-blobs cut each file into chunks, encrypted each chunk under keys derived from the file key, and named the file by the hash of its tree's root: the blob ID. See kolo-blobs.
- The chunks and the root were entries of a kolo-store log. kolo-sync copied that log to the receiver, checking each entry against the owner's signed head. See Logs and Replicas.
- The receiver then checked every chunk against the blob ID while it read, so a wrong byte would have stopped the read instead of reaching your disk.
- For the folder, kolo-files kept one op per change in a metadata log and each
file version as a blob in a content log, under one key per file. A version is
the length of the metadata log, so the receiver could save each version and
compare two with
Diff. See kolo-files. - The connection was a kolo-secure channel tied to the sender's key, and the share code was the only thing passed by hand.
Next steps
- Compare your files with the finished code in the
kolo-share-example
repository. Its tests run both programs on
127.0.0.1and check each error shown here. - Read the kolo-blobs page: range reads, the formats, the limits, and how to read a blob from a pinning peer.
- Read the kolo-files page: names, versions,
selective sync and how
Deletedestroys a file. - The logs of both programs can also rest on a kolo-pin pinning peer, which holds only ciphertext, so a file stays reachable while your computer is off. The kolo-blobs and kolo-files pages explain how a reader reads from one.