In the previous step you built a desktop chat. Now you add a pin, so messages reach people who are offline.
A pin is a peer that keeps encrypted data for others without being able to read it. Direct delivery does not change: when both peers are online, a message still goes straight from one to the other. What you add is a second path: every message is also written to your own signed log, and the pin keeps a copy of that log, so a peer who was offline can fetch what it missed when it opens the app.
This step is backend only: you paste the Go code below into the project of the previous step. The window of this step is already built, and you bring it in with one command.
1Check what you need
You need what step 2 needed: Go 1.26 or newer, Node.js, wails3, the bootstrap node, and openssl for a random key. Start from the project you finished in step 2. The pin itself is a program from kolo-pin that you run with go run, so there is nothing more to install.
2Bring the window and the libraries
The window of this step has a new status line and a mark for messages that came from the pin. It is already built: bring it into your project from the tag step-3, which also adds .data to .gitignore. Then add the libraries this step uses: kolo-log for the signed log, kolo-store to keep it on disk, kolo-sync to replicate it, and kolo-pin for the pin.
git fetch origin --tags
git checkout step-3 -- frontend .gitignorego get github.com/ciranda-dev/kolo-log@v0.1.0 github.com/ciranda-dev/kolo-store@v0.1.0 github.com/ciranda-dev/kolo-sync@v0.1.0 github.com/ciranda-dev/kolo-pin@v0.1.0The tag step-3 also holds the finished version of every file below, in case you want to compare yours with it.
3Read the settings
Nothing about the pin is written into the code. Every setting is a flag or an environment variable, and a flag wins. A setting that is missing or malformed stops Start with a typed error: errors.Is(err, ErrConfig) is true for all of them, and ErrMissing or ErrMalformed tell why. A pin token that cannot work gets the same treatment, wrapping the error of kolo-pin that says why.
Create config.go with:
import (
"crypto/ed25519"
"encoding/hex"
"errors"
"flag"
"fmt"
"io"
"net/netip"
"os"
"path/filepath"
"strconv"
"time"
)Add to config.go:
// Defaults of the settings that have one. Everything else must be configured.
const (
defaultBootstrap = "127.0.0.1:4000"
defaultListen = "127.0.0.1:0"
defaultPinAddr = "127.0.0.1:7450" // where `kolo-pin run` listens unless told otherwise
defaultPinTTL = 30 * 24 * time.Hour
defaultPinQuota = 16 << 20 // bytes
defaultLookupInterval = 5 * time.Second
)
// ErrConfig matches every ConfigError, whatever setting it names.
var ErrConfig = errors.New("invalid configuration")
// Causes of a ConfigError, besides the errors of kolo-pin for a bad token.
var (
ErrMissing = errors.New("is required")
ErrMalformed = errors.New("is malformed")
)
// ConfigError names the setting that is wrong and why. errors.Is(err, ErrConfig) is true for every one, and
// errors.Is(err, ErrMissing), pin.ErrExpired and the like tell the cause.
type ConfigError struct {
Setting string // the flag name, such as "pin-key"
Err error
}
func (e *ConfigError) Error() string { return fmt.Sprintf("kolo-chat: -%s %v", e.Setting, e.Err) }
func (e *ConfigError) Unwrap() error { return e.Err }
func (e *ConfigError) Is(target error) bool {
return target == ErrConfig
}
// Config is everything the chat needs from outside. A zero field takes its default, if it has one.
// LoadConfig fills it from flags and environment variables.
type Config struct {
DataDir string // -data, KOLO_CHAT_DATA: identity, Pebble store, members. Default: <user config dir>/kolo-chat
Bootstrap string // -bootstrap, KOLO_CHAT_BOOTSTRAP: IP:PORT of a bootstrap node. Default 127.0.0.1:4000
Listen string // -listen, KOLO_CHAT_LISTEN: local IP:PORT. Default 127.0.0.1:0
ConversationKey string // -conversation-key, KOLO_CHAT_CONVERSATION_KEY: 64 hex characters (32 bytes). Required
PinAddr string // -pin, KOLO_CHAT_PIN: IP:PORT of the pin. Default 127.0.0.1:7450
PinKey string // -pin-key, KOLO_CHAT_PIN_KEY: the pin's public key in hex (`kolo-pin id`). Required
PinToken string // -pin-token, KOLO_CHAT_PIN_TOKEN: a capability in hex, issued by this app's identity. Default: the app issues one
PinTTL time.Duration // -pin-ttl, KOLO_CHAT_PIN_TTL: lifetime of the capability the app issues. Default 720h
PinQuota uint64 // -pin-quota, KOLO_CHAT_PIN_QUOTA: bytes the pin may keep for you, in the capability the app issues. Default 16777216
LookupInterval time.Duration // -lookup-interval, KOLO_CHAT_LOOKUP_INTERVAL: time between topic lookups. Default 5s
}
// LoadConfig reads the flags in args. A setting that is not given as a flag comes from its environment variable
// (read with getenv, such as os.Getenv), and a flag wins over its variable.
func LoadConfig(args []string, getenv func(string) string, stderr io.Writer) (Config, error) {
var c Config
fs := flag.NewFlagSet("kolo-chat", flag.ContinueOnError)
fs.SetOutput(stderr)
text := func(p *string) func(string) error {
return func(v string) error { *p = v; return nil }
}
dur := func(p *time.Duration) func(string) error {
return func(v string) (err error) { *p, err = time.ParseDuration(v); return err }
}
num := func(p *uint64) func(string) error {
return func(v string) (err error) { *p, err = strconv.ParseUint(v, 10, 64); return err }
}
for _, s := range []struct {
name, env, help string
set func(string) error
}{
{"data", "KOLO_CHAT_DATA", "data directory (default <user config dir>/kolo-chat)", text(&c.DataDir)},
{"bootstrap", "KOLO_CHAT_BOOTSTRAP", "IP:PORT of the bootstrap node (default " + defaultBootstrap + ")", text(&c.Bootstrap)},
{"listen", "KOLO_CHAT_LISTEN", "local IP:PORT to listen on (default " + defaultListen + ")", text(&c.Listen)},
{"conversation-key", "KOLO_CHAT_CONVERSATION_KEY", "32-byte key of the conversation, in hex (required)", text(&c.ConversationKey)},
{"pin", "KOLO_CHAT_PIN", "IP:PORT of the pin (default " + defaultPinAddr + ")", text(&c.PinAddr)},
{"pin-key", "KOLO_CHAT_PIN_KEY", "public key of the pin, in hex (required)", text(&c.PinKey)},
{"pin-token", "KOLO_CHAT_PIN_TOKEN", "capability for the pin, in hex (default: issued by the app)", text(&c.PinToken)},
{"pin-ttl", "KOLO_CHAT_PIN_TTL", "lifetime of the capability the app issues (default 720h)", dur(&c.PinTTL)},
{"pin-quota", "KOLO_CHAT_PIN_QUOTA", "bytes the pin may keep for you (default 16777216)", num(&c.PinQuota)},
{"lookup-interval", "KOLO_CHAT_LOOKUP_INTERVAL", "time between lookups of the topic (default 5s)", dur(&c.LookupInterval)},
} {
if v := getenv(s.env); v != "" {
if err := s.set(v); err != nil {
return c, &ConfigError{Setting: s.name, Err: fmt.Errorf("%w: %s: %w", ErrMalformed, s.env, err)}
}
}
fs.Func(s.name, s.help+" ("+s.env+")", s.set)
}
return c, fs.Parse(args)
}
// settings is a Config after defaults and checks: values the code can use as they are.
type settings struct {
Config
conversationKey [32]byte
pinKey ed25519.PublicKey
token []byte // nil: the app issues one
bootstrap netip.AddrPort
}
// resolve applies the defaults and checks every setting, returning a *ConfigError for the first bad one.
// A token is checked later, in capability, because it needs the identity.
func (c Config) resolve() (settings, error) {
s := settings{Config: c}
if s.DataDir == "" {
dir, err := os.UserConfigDir()
if err != nil {
return s, &ConfigError{Setting: "data", Err: fmt.Errorf("%w: no user config directory: %w", ErrMissing, err)}
}
s.DataDir = filepath.Join(dir, "kolo-chat")
}
s.Bootstrap = orDefault(s.Bootstrap, defaultBootstrap)
s.Listen = orDefault(s.Listen, defaultListen)
s.PinAddr = orDefault(s.PinAddr, defaultPinAddr)
if s.PinTTL == 0 {
s.PinTTL = defaultPinTTL
}
if s.PinQuota == 0 {
s.PinQuota = defaultPinQuota
}
if s.LookupInterval == 0 {
s.LookupInterval = defaultLookupInterval
}
boot, err := netip.ParseAddrPort(s.Bootstrap)
if err != nil {
return s, &ConfigError{Setting: "bootstrap", Err: fmt.Errorf("%w: %w", ErrMalformed, err)}
}
s.bootstrap = boot
if _, err := netip.ParseAddrPort(s.PinAddr); err != nil {
return s, &ConfigError{Setting: "pin", Err: fmt.Errorf("%w: %w", ErrMalformed, err)}
}
if err := parseHex("conversation-key", c.ConversationKey, s.conversationKey[:]); err != nil {
return s, err
}
s.pinKey = make(ed25519.PublicKey, ed25519.PublicKeySize)
if err := parseHex("pin-key", c.PinKey, s.pinKey); err != nil {
return s, err
}
if c.PinToken != "" {
if s.token, err = hex.DecodeString(c.PinToken); err != nil {
return s, &ConfigError{Setting: "pin-token", Err: fmt.Errorf("%w: %w", ErrMalformed, err)}
}
}
return s, nil
}
func orDefault(v, def string) string {
if v == "" {
return def
}
return v
}
// parseHex decodes the hex setting value into dst, which has the exact size wanted.
func parseHex(setting, value string, dst []byte) error {
if value == "" {
return &ConfigError{Setting: setting, Err: ErrMissing}
}
b, err := hex.DecodeString(value)
if err != nil || len(b) != len(dst) {
return &ConfigError{Setting: setting, Err: fmt.Errorf("%w: want %d bytes in hex", ErrMalformed, len(dst))}
}
copy(dst, b)
return nil
}4Describe the state of the service
The service now needs more than a map of peers. A message between peers has a small wire format, a member remembers the log of a peer, and the pin has a state the window can show. The service is also built with its settings and with a function that sends events to the window, so the logic can be tested without a window.
In chatservice.go, replace the import block with:
import (
"context"
"crypto/cipher"
"crypto/ed25519"
"crypto/hmac"
"crypto/rand"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"io/fs"
"log"
"net/netip"
"os"
"path/filepath"
"slices"
"sync"
"time"
koloid "github.com/ciranda-dev/kolo-id"
kololog "github.com/ciranda-dev/kolo-log"
kolosecure "github.com/ciranda-dev/kolo-secure"
kolostore "github.com/ciranda-dev/kolo-store"
koloswarm "github.com/ciranda-dev/kolo-swarm"
kolosync "github.com/ciranda-dev/kolo-sync"
"golang.org/x/crypto/chacha20poly1305"
)Add to chatservice.go:
const (
maxMessage = 64 * 1024
maxWire = 128 * 1024 // a message as JSON, with its sealed text in base64
sendTimeout = 10 * time.Second
codeTooLong = 1
)In chatservice.go, replace ChatMsg with:
// ChatMsg is the payload of the "chat:msg" event. From is the sender's kolo-id (hex public key) and Via says how
// the message arrived: "direct" from the sender, "pin" when its ciphertext came from the pin, "peer" when it was
// pulled from the sender after the direct delivery was missed.
type ChatMsg struct {
From string `json:"from"`
Text string `json:"text"`
Via string `json:"via"`
}Add to chatservice.go:
// PinStatus is the payload of the "chat:pin" event. Error is the sentence to show when the pin cannot be used,
// and Locked counts the pinned messages that wait for their sender's key.
type PinStatus struct {
Error string `json:"error"`
Locked int `json:"locked"`
}
// wireMsg is what peers send each other on the direct connection, one unidirectional stream each.
type wireMsg struct {
Kind string `json:"kind"` // "hello" or "msg"
Header []byte `json:"header,omitempty"` // hello: the sender's signed log header
Proof []byte `json:"proof,omitempty"` // hello: proves the sender knows the conversation key
Reply bool `json:"reply,omitempty"` // hello: answers a hello, so the sender has registered us
Index uint64 `json:"index,omitempty"` // msg: the entry's index in the sender's log
Sealed []byte `json:"sealed,omitempty"` // msg: the entry's value, the text sealed with the conversation key
}
// member is a peer of the conversation: its signed log header, which names its public key and log ID, and how
// many of its messages the window has shown. Both are kept in the data directory.
type member struct {
Header []byte `json:"header"`
Shown uint64 `json:"shown"`
}
type pinState struct {
Push, Pull string // what to tell the user, "" when it works
Entries, Used, Quota uint64
}In chatservice.go, replace ChatService with:
type ChatService struct {
cfg Config
emit func(name string, data any)
startMu sync.Mutex
set settings
id *koloid.Identity
pub ed25519.PublicKey
aead cipher.AEAD
token []byte
store *kolostore.Store
log *kolostore.Log // our own log, the only one we write
srv *kolosync.Server
sw *koloswarm.Swarm
ctx context.Context
cancel context.CancelFunc
wg sync.WaitGroup
push chan struct{} // a push to the pin is wanted
pinDone chan struct{} // closed when the pull from the pin at start is over
mu sync.Mutex // guards the fields below
peers map[string]*koloswarm.Peer
members map[string]*member
replicas map[string]*kolostore.Replica
pinned map[string]uint64 // per author: entries below this index came from the pin
locked map[string]int // per author: entries that wait for a key
pin pinState
showMu sync.Mutex // one message is shown at a time, so they keep their order
pullMu sync.Mutex // one pull at a time, since a replica takes one
pushMu sync.Mutex // one push at a time
}Add to chatservice.go:
// NewChatService returns a chat that reads cfg when it starts and tells the window what happens through emit.
func NewChatService(cfg Config, emit func(name string, data any)) *ChatService {
return &ChatService{cfg: cfg, emit: emit}
}Delete maxMessage from chatservice.go.
5Start and stop the service
Start opens the store, loads the identity, joins the swarm and starts the loops, and ServiceShutdown closes everything when the app quits. main.go builds the service from the settings and wires its events to the window.
In chatservice.go, replace Start with:
// Start joins the conversation and returns this user's kolo-id (hex public key). It opens the data directory,
// the log and the pin, so it fails with a *ConfigError for a bad setting and with the errors of kolo-pin
// (pin.ErrExpired, pin.ErrUnauthorized) for a bad token. A pin that cannot be reached is not an error: the chat
// works directly and tells the window.
func (c *ChatService) Start() (string, error) {
c.startMu.Lock()
defer c.startMu.Unlock()
if c.id != nil {
return c.id.PublicKeyString(), nil
}
if err := c.start(); err != nil {
c.shutdown()
return "", err
}
return c.id.PublicKeyString(), nil
}Add to chatservice.go:
func (c *ChatService) start() (err error) {
if c.set, err = c.cfg.resolve(); err != nil {
return err
}
if err = os.MkdirAll(c.set.DataDir, 0o700); err != nil {
return err
}
if c.id, err = loadIdentity(c.set.DataDir); err != nil {
return err
}
c.pub = c.id.PublicKey()
if c.aead, err = chacha20poly1305.NewX(c.set.conversationKey[:]); err != nil {
return err
}
backend, err := kolostore.OpenPebble(filepath.Join(c.set.DataDir, "store"))
if err != nil {
return err
}
c.store = kolostore.New(backend)
if c.log, err = openOwnLog(c.store, c.id); err != nil {
return err
}
if c.token, err = capability(c.set, c.id, c.log.ID()); err != nil {
return err
}
c.peers = map[string]*koloswarm.Peer{}
c.replicas = map[string]*kolostore.Replica{}
c.pinned = map[string]uint64{}
c.locked = map[string]int{}
if c.members, err = loadMembers(c.set.DataDir); err != nil {
return err
}
c.srv = kolosync.NewServer(c.authorize)
if err = c.srv.Offer(c.log); err != nil {
return err
}
c.ctx, c.cancel = context.WithCancel(context.Background())
c.push = make(chan struct{}, 1)
c.pinDone = make(chan struct{})
if c.sw, err = koloswarm.New(c.ctx, c.set.Listen, c.id, &koloswarm.Config{LookupInterval: c.set.LookupInterval}); err != nil {
return err
}
if err = c.sw.Bootstrap(c.ctx, []netip.AddrPort{c.set.bootstrap}); err != nil {
return err
}
log.Print("kolo-id ", c.id.PublicKeyString())
c.wg.Go(c.swarmLoop)
c.wg.Go(c.pushLoop)
c.wg.Go(func() { // on open: the pin first, then the peers
defer close(c.pinDone)
if err := c.pullFromPin(c.ctx); err != nil {
log.Print("pin pull: ", err)
}
})
c.pushSoon()
c.wg.Go(func() { // join the conversation's topic
err := c.sw.Join(c.ctx, c.topic(), &koloswarm.JoinOptions{Announce: true, Lookup: true})
if err != nil && c.ctx.Err() == nil {
log.Print("join: ", err)
}
})
return nil
}
// ServiceShutdown stops the chat and closes the store. Wails calls it when the app quits.
func (c *ChatService) ServiceShutdown() error {
c.startMu.Lock()
defer c.startMu.Unlock()
return c.shutdown()
}
func (c *ChatService) shutdown() error {
if c.cancel != nil {
c.cancel()
}
var errs []error
if c.sw != nil {
errs = append(errs, c.sw.Close())
}
c.wg.Wait()
if c.store != nil {
errs = append(errs, c.store.Close()) // also closes the log and the replicas
}
c.id, c.store, c.sw, c.cancel = nil, nil, nil, nil
return errors.Join(errs...)
}
// topic is where the conversation's peers find each other: a hash of the conversation key, so only those who
// know the key look at it.
func (c *ChatService) topic() koloswarm.Topic {
return sha256.Sum256(append([]byte("kolochat/topic/v1"), c.set.conversationKey[:]...))
}
// swarmLoop connects to every peer the swarm finds.
func (c *ChatService) swarmLoop() {
for {
ev, err := c.sw.Next(c.ctx)
if err != nil {
if c.ctx.Err() == nil {
log.Print("swarm: ", err)
}
return
}
if ev.Kind == koloswarm.Connected && ev.Peer != nil {
c.addPeer(ev.Peer)
}
}
}In main.go, replace the import block with:
import (
"embed"
"errors"
"flag"
"log"
"os"
"github.com/wailsapp/wails/v3/pkg/application"
)In main.go, replace main with:
func main() {
log.Printf("Kolo Chat %s", version)
applyMemoryLimit()
cfg, err := LoadConfig(os.Args[1:], os.Getenv, os.Stderr)
if errors.Is(err, flag.ErrHelp) {
return
}
if err != nil {
log.Fatal(err)
}
// The chat tells the window what happens by emitting events; the Wails app is only needed once one is.
emit := func(name string, data any) { application.Get().Event.Emit(name, data) }
app := application.New(application.Options{
Name: "Kolo Chat",
Description: "Peer-to-peer desktop chat built with Kolo",
Icon: appIcon,
Services: []application.Service{
application.NewService(NewChatService(cfg, emit)),
},
Assets: application.AssetOptions{
Handler: application.AssetFileServerFS(assets),
},
Mac: application.MacOptions{
ApplicationShouldTerminateAfterLastWindowClosed: true,
},
})
app.Window.NewWithOptions(application.WebviewWindowOptions{
Title: "Kolo Chat",
// Window sized to the golden ratio (1000 / 618 ≈ 1.618).
Width: 1000,
Height: 618,
Mac: application.MacWindow{
InvisibleTitleBarHeight: 50,
Backdrop: application.MacBackdropTranslucent,
TitleBar: application.MacTitleBarHiddenInset,
},
BackgroundColour: application.NewRGB(7, 11, 18),
URL: "/",
})
if err := app.Run(); err != nil {
log.Fatal(err)
}
}6Keep an identity and a log
An offline peer can only catch up from a log that is still yours after a restart, so the identity has to survive it. The app loads its key from the data directory and makes one the first time. The log lives in a kolo-store, which keeps it on disk, and the app opens its own log or creates it.
Add to chatservice.go:
func loadIdentity(dir string) (*koloid.Identity, error) {
path := filepath.Join(dir, "identity.key")
id, err := koloid.Load(path)
if !errors.Is(err, fs.ErrNotExist) {
return id, err
}
if id, err = koloid.Generate(); err != nil {
return nil, err
}
if err = id.SaveNew(path); err != nil && !errors.Is(err, koloid.ErrSavedNotSynced) {
return nil, err
}
return id, nil
}
// openOwnLog opens our log in the store, or creates it the first time.
func openOwnLog(st *kolostore.Store, id *koloid.Identity) (*kolostore.Log, error) {
ids, err := st.List()
if err != nil {
return nil, err
}
for _, logID := range ids {
pub, replica, err := st.Owner(logID)
if err != nil {
return nil, err
}
if !replica && pub.Equal(id.PublicKey()) {
return st.Open(id, logID)
}
}
return st.Create(id)
}7Meet the other peers
When two peers connect they say hello: each sends its signed log header, which names its key and log ID, and a proof that it knows the conversation key, bound to both keys so it cannot be replayed to another peer. authorize is what kolo-sync asks before it serves our log: only a peer that proved it knows the conversation key may read it, and only with entry keys if it is a member. Members are saved, so a restart remembers who we met.
In chatservice.go, replace addPeer with:
func (c *ChatService) addPeer(p *koloswarm.Peer) {
key := hex.EncodeToString(p.PublicKey())
c.mu.Lock()
c.peers[key] = p
c.mu.Unlock()
log.Print("peer connected: ", key[:8])
c.emitPeers()
if conn, ok := p.Conn().(*kolosecure.Conn); ok { // not for a relayed connection
c.wg.Go(func() { c.srv.Serve(c.ctx, conn) })
}
c.wg.Go(func() {
defer func() {
c.mu.Lock()
if c.peers[key] == p {
delete(c.peers, key)
}
c.mu.Unlock()
log.Print("peer gone: ", key[:8])
c.emitPeers()
}()
if err := c.sendTo(p, c.hello(p, false)); err != nil {
log.Print("hello: ", err)
}
c.receive(p)
})
}Add to chatservice.go:
// receive handles what one peer sends, one stream at a time.
func (c *ChatService) receive(p *koloswarm.Peer) {
for {
st, err := p.Conn().AcceptUniStream(c.ctx)
if err != nil {
return
}
st.SetReadDeadline(time.Now().Add(sendTimeout))
data, err := io.ReadAll(io.LimitReader(st, maxWire+1))
if err != nil || len(data) > maxWire {
st.CancelRead(codeTooLong)
continue
}
var m wireMsg
if json.Unmarshal(data, &m) != nil {
continue
}
switch m.Kind {
case "hello":
c.onHello(p, m)
case "msg":
c.onMessage(p, m)
}
}
}
func (c *ChatService) sendTo(p *koloswarm.Peer, m wireMsg) error {
data, err := json.Marshal(m)
if err != nil {
return err
}
ctx, cancel := context.WithTimeout(c.ctx, sendTimeout)
defer cancel()
st, err := p.Conn().OpenUniStream(ctx)
if err != nil {
return err
}
st.SetWriteDeadline(time.Now().Add(sendTimeout))
if _, err := st.Write(data); err != nil {
st.CancelWrite(codeTooLong)
return err
}
return st.Close()
}Add to chatservice.go:
// hello introduces us: our signed log header, which names our key and log ID, and a proof that we know the
// conversation key, bound to both keys so it cannot be replayed to another peer.
func (c *ChatService) hello(p *koloswarm.Peer, reply bool) wireMsg {
return wireMsg{Kind: "hello", Header: c.log.Header().Encode(), Proof: c.proof(c.pub, p.PublicKey()), Reply: reply}
}
func (c *ChatService) proof(from, to ed25519.PublicKey) []byte {
mac := hmac.New(sha256.New, c.set.conversationKey[:])
mac.Write([]byte("kolochat/hello/v1"))
mac.Write(from)
mac.Write(to)
return mac.Sum(nil)
}
// onHello registers a peer that proves it knows the conversation key and answers with our own hello. The
// answer tells the peer that we registered it, so it may pull our log; a hello that is an answer tells us
// the same, so we may pull its log.
func (c *ChatService) onHello(p *koloswarm.Peer, m wireMsg) {
pub := p.PublicKey()
h, err := kololog.DecodeLogHeader(m.Header)
if err != nil || kololog.VerifyLogHeader(pub, h) != nil || !hmac.Equal(m.Proof, c.proof(pub, c.pub)) {
log.Print("hello refused from ", hex.EncodeToString(pub)[:8])
return
}
if err := c.addMember(pub, m.Header); err != nil {
log.Print("member: ", err)
return
}
if !m.Reply {
if err := c.sendTo(p, c.hello(p, true)); err != nil {
log.Print("hello: ", err)
}
return
}
c.wg.Go(func() { c.syncFromPeer(p) })
}
// authorize lets the members of the conversation read our log with its entry keys, and nobody else.
func (c *ChatService) authorize(peer, owner ed25519.PublicKey, logID kololog.LogID) kolosync.Access {
c.mu.Lock()
_, ok := c.members[hex.EncodeToString(peer)]
c.mu.Unlock()
if ok && owner.Equal(c.pub) && logID == c.log.ID() {
return kolosync.Keys
}
return kolosync.NoAccess
}Add to chatservice.go:
func (c *ChatService) addMember(pub ed25519.PublicKey, header []byte) error {
author := hex.EncodeToString(pub)
c.mu.Lock()
defer c.mu.Unlock()
if _, ok := c.members[author]; ok {
return nil
}
c.members[author] = &member{Header: header}
return saveMembers(c.set.DataDir, c.members)
}
func loadMembers(dir string) (map[string]*member, error) {
members := map[string]*member{}
data, err := os.ReadFile(filepath.Join(dir, "members.json"))
if errors.Is(err, fs.ErrNotExist) {
return members, nil
}
if err != nil {
return nil, err
}
return members, json.Unmarshal(data, &members)
}
// saveMembers replaces members.json in one step, so a crash leaves the old file or the new one.
func saveMembers(dir string, members map[string]*member) error {
data, err := json.Marshal(members)
if err != nil {
return err
}
tmp := filepath.Join(dir, "members.json.tmp")
if err := os.WriteFile(tmp, data, 0o600); err != nil {
return err
}
return os.Rename(tmp, filepath.Join(dir, "members.json"))
}8Seal and send a message
Before a message is appended it is sealed with the conversation key. kolo-log already encrypts every entry under a key of its own, but those keys can reach people who must not read the conversation, so this second layer is the one that stays with the group. Send appends the sealed text to the log, still sends it straight to every connected peer, and asks for a push to the pin.
In chatservice.go, replace Send with:
// Send seals the text with the conversation key and appends it to our log. Then it hands it to every peer
// that is connected, and to the pin for those who are not. A pin that fails never fails the Send.
func (c *ChatService) Send(text string) error {
if len(text) > maxMessage {
return errors.New("message too long")
}
if c.log == nil {
return errors.New("not started")
}
sealed, err := c.seal(text)
if err != nil {
return err
}
index, err := c.log.Append(sealed)
if err != nil {
return err
}
c.mu.Lock()
peers := make([]*koloswarm.Peer, 0, len(c.peers))
for _, p := range c.peers {
peers = append(peers, p)
}
c.mu.Unlock()
for _, p := range peers {
if err := c.sendTo(p, wireMsg{Kind: "msg", Index: index, Sealed: sealed}); err != nil {
log.Print("direct send: ", err) // the pin and the next sync still carry it
}
}
c.pushSoon()
return nil
}Add to chatservice.go:
// seal and open protect the text with the conversation key (XChaCha20-Poly1305), bound to the author's key.
// This is the layer the pin and anyone with entry keys cannot remove.
func (c *ChatService) seal(text string) ([]byte, error) {
nonce := make([]byte, c.aead.NonceSize())
if _, err := rand.Read(nonce); err != nil {
return nil, err
}
return c.aead.Seal(nonce, nonce, []byte(text), c.pub), nil
}
func (c *ChatService) open(author ed25519.PublicKey, sealed []byte) (string, error) {
n := c.aead.NonceSize()
if len(sealed) < n {
return "", errors.New("sealed text too short")
}
text, err := c.aead.Open(nil, sealed[:n], sealed[n:], author)
return string(text), err
}In chatservice.go, replace emitPeers with:
func (c *ChatService) emitPeers() {
c.mu.Lock()
n := len(c.peers)
c.mu.Unlock()
c.emit("chat:peers", n)
}Delete emit from chatservice.go.
9Receive and show messages
A message that arrives directly is checked and shown. For everything else, syncFromPeer pulls the peer's log with kolo-sync, which verifies every entry against the owner's signed head. show gives the window the entries it has not shown yet, in the order of the sender's log. The count of what was already shown is kept per sender, which is what prevents duplicates with the messages that arrived directly. An entry whose key has not come yet waits, and nothing after it jumps ahead.
Add to chatservice.go:
// onMessage shows a message that arrives directly if it is the next one of its sender, and drops it if it was
// shown already. A gap means we missed some: a sync from the sender shows them all, in order.
func (c *ChatService) onMessage(p *koloswarm.Peer, m wireMsg) {
author := hex.EncodeToString(p.PublicKey())
c.showMu.Lock()
defer c.showMu.Unlock()
next, known := c.shown(author)
switch {
case !known: // its hello is not here yet, and the sync that follows it brings this message too
case m.Index > next:
c.wg.Go(func() { c.syncFromPeer(p) })
case m.Index == next:
text, err := c.open(p.PublicKey(), m.Sealed)
if err != nil {
log.Print("direct message refused: ", err)
return
}
c.setShown(author, next+1)
c.emit("chat:msg", ChatMsg{From: author, Text: text, Via: "direct"})
}
}
// syncFromPeer pulls the sender's log from the sender, with its entry keys, and shows what is new. It waits
// for the pull from the pin at start, so what the pin has is taken from the pin.
func (c *ChatService) syncFromPeer(p *koloswarm.Peer) {
select {
case <-c.pinDone:
case <-c.ctx.Done():
return
}
conn, ok := p.Conn().(*kolosecure.Conn)
if !ok {
return
}
author := hex.EncodeToString(p.PublicKey())
rep, err := c.replica(author)
if err != nil {
log.Print("replica: ", err)
return
}
ctx, cancel := context.WithTimeout(c.ctx, pinTimeout)
defer cancel()
c.pullMu.Lock()
err = kolosync.Pull(ctx, conn, rep)
c.pullMu.Unlock()
if err != nil && c.ctx.Err() == nil {
log.Print("sync from peer: ", err)
return
}
c.show(author)
}
// show gives the window the messages of an author that it has not seen, in the order of the author's log. It
// stops at the first entry whose key is missing: the pin sent the ciphertext, and the key comes only from the
// author, so the entry waits and nothing after it jumps ahead.
func (c *ChatService) show(author string) {
c.showMu.Lock()
defer c.showMu.Unlock()
rep, err := c.replica(author)
if err != nil {
return
}
pub := rep.Header().LogKey
for {
index, _ := c.shown(author)
if index >= rep.Len() {
break
}
sealed, err := rep.Get(index)
if errors.Is(err, kololog.ErrKeyUnavailable) {
break
}
if err == nil {
var text string
if text, err = c.open(pub, sealed); err == nil {
c.mu.Lock()
via := "peer"
if index < c.pinned[author] {
via = "pin"
}
c.mu.Unlock()
c.emit("chat:msg", ChatMsg{From: author, Text: text, Via: via})
}
}
if err != nil && !errors.Is(err, kololog.ErrDeleted) {
log.Printf("entry %d of %s skipped: %v", index, author[:8], err)
}
c.setShown(author, index+1)
}
locked := 0
if next, _ := c.shown(author); rep.Len() > next {
locked = int(rep.Len() - next)
}
c.setPin(func(*pinState) { c.locked[author] = locked })
}Add to chatservice.go:
// replica opens, once, the read-only copy of a member's log that we keep in our store.
func (c *ChatService) replica(author string) (*kolostore.Replica, error) {
c.mu.Lock()
defer c.mu.Unlock()
if rep, ok := c.replicas[author]; ok {
return rep, nil
}
m, ok := c.members[author]
if !ok {
return nil, fmt.Errorf("unknown member %s", author[:8])
}
h, err := kololog.DecodeLogHeader(m.Header)
if err != nil {
return nil, err
}
rep, err := c.store.OpenReplica(h.LogKey, h.LogID, m.Header)
if err != nil {
return nil, err
}
c.replicas[author] = rep
return rep, nil
}
func (c *ChatService) markPinned(author string, n uint64) {
c.mu.Lock()
c.pinned[author] = n
c.mu.Unlock()
}
// authors lists the members, as hex public keys.
func (c *ChatService) authors() []string {
c.mu.Lock()
defer c.mu.Unlock()
keys := make([]string, 0, len(c.members))
for k := range c.members {
keys = append(keys, k)
}
slices.Sort(keys)
return keys
}
func (c *ChatService) shown(author string) (uint64, bool) {
c.mu.Lock()
defer c.mu.Unlock()
m, ok := c.members[author]
if !ok {
return 0, false
}
return m.Shown, true
}
func (c *ChatService) setShown(author string, n uint64) {
c.mu.Lock()
c.members[author].Shown = n
err := saveMembers(c.set.DataDir, c.members)
c.mu.Unlock()
if err != nil {
log.Print("members: ", err)
}
}10Push the log to the pin
The pin accepts a log only under a capability: a token signed by the log's owner, with an expiry and a quota. The app issues its own from its identity, keeps it in the data directory and reuses it while it still has more than half of its life. It checks a token before using it, so that a wrong or expired one fails at start with a clear error, not later at the pin. pin.Add gives the pin the log and the pin pulls and verifies what it does not have yet; it can be repeated at any time, so every message just asks for another push, and requests that come while a push runs merge into one.
Create pin.go with:
import (
"context"
"crypto/ed25519"
"errors"
"fmt"
"os"
"path/filepath"
"time"
koloid "github.com/ciranda-dev/kolo-id"
kololog "github.com/ciranda-dev/kolo-log"
"github.com/ciranda-dev/kolo-pin"
kolosecure "github.com/ciranda-dev/kolo-secure"
kolosync "github.com/ciranda-dev/kolo-sync"
)Add to pin.go:
// pinTimeout bounds one conversation with the pin: connecting, pushing or pulling.
const pinTimeout = 30 * time.Second
// capability returns the signed token that lets the pin keep this log. A token from the configuration is
// checked and used as it is. Otherwise the app issues one with its own identity, for the pin and this one log,
// keeps it in the data directory and reuses it until half of its lifetime has passed, since the pin refuses a
// second capability issued in the same second as the first.
func capability(set settings, id *koloid.Identity, logID kololog.LogID) ([]byte, error) {
if set.token != nil {
return set.token, checkToken(set.token, id, set.pinKey, logID)
}
path := filepath.Join(set.DataDir, "pin-token")
if token, err := os.ReadFile(path); err == nil {
c, err := pin.ParseCapability(token)
if err == nil && c.Quota == set.PinQuota && time.Until(c.Expires) > set.PinTTL/2 &&
checkToken(token, id, set.pinKey, logID) == nil {
return token, nil
}
}
token, err := pin.Issue(id, pin.Capability{
Peer: set.pinKey,
Logs: []kololog.LogID{logID},
Expires: time.Now().Add(set.PinTTL),
Quota: set.PinQuota,
})
if err != nil {
return nil, &ConfigError{Setting: "pin-ttl", Err: err}
}
if err := os.WriteFile(path, token, 0o600); err != nil {
return nil, err
}
return token, nil
}
// checkToken says why a token cannot work, with the errors of kolo-pin: ErrInvalidCapability for bytes that are
// not a capability signed by its owner, ErrUnauthorized for one that is not from this identity, for this pin and
// for this log, and ErrExpired for one that has expired. The pin checks all of it again when it is used.
func checkToken(token []byte, id *koloid.Identity, pinKey ed25519.PublicKey, logID kololog.LogID) error {
c, err := pin.ParseCapability(token)
switch {
case err != nil:
err = fmt.Errorf("%w: %w", pin.ErrInvalidCapability, err)
case !c.Owner.Equal(id.PublicKey()):
err = fmt.Errorf("%w: issued by another identity", pin.ErrUnauthorized)
case !c.Peer.Equal(pinKey):
err = fmt.Errorf("%w: issued for another pin", pin.ErrUnauthorized)
case !c.Covers(logID):
err = fmt.Errorf("%w: does not cover this conversation's log", pin.ErrUnauthorized)
case !time.Now().Before(c.Expires):
err = fmt.Errorf("%w: at %s", pin.ErrExpired, c.Expires.Format(time.RFC3339))
}
if err != nil {
return &ConfigError{Setting: "pin-token", Err: err}
}
return nil
}
// pushToPin gives the pin our log, which pulls and verifies what it does not have yet. It is how every message
// reaches the pin, and it can be repeated at any time. The errors are those of kolo-pin: pin.ErrExpired,
// pin.ErrUnauthorized, pin.ErrRevoked, pin.ErrQuotaExceeded, pin.ErrDiskFull, pin.ErrUnreachable.
func (c *ChatService) pushToPin(ctx context.Context) error {
c.pushMu.Lock() // the pin answers busy to a second Add of the same log
defer c.pushMu.Unlock()
ctx, cancel := context.WithTimeout(ctx, pinTimeout)
defer cancel()
peer := pin.Peer{ID: c.set.pinKey, Addr: c.set.PinAddr}
info, err := pin.Add(ctx, c.id, peer, c.log, c.token)
c.setPin(func(s *pinState) {
s.Push = describe(err, c.set)
s.Entries, s.Used, s.Quota = info.Entries, info.Used, info.Quota
})
return err
}Add to chatservice.go:
// pushLoop pushes to the pin whenever one is wanted; wishes that come while it works are merged into one.
func (c *ChatService) pushLoop() {
for {
select {
case <-c.ctx.Done():
return
case <-c.push:
}
if err := c.pushToPin(c.ctx); err != nil && c.ctx.Err() == nil {
log.Print("pin push: ", err)
}
}
}
func (c *ChatService) pushSoon() {
select {
case c.push <- struct{}{}:
default:
}
}Add to chatservice.go:
// setPin changes the pin's state under the lock and tells the window.
func (c *ChatService) setPin(change func(*pinState)) {
c.mu.Lock()
change(&c.pin)
st := PinStatus{Error: c.pin.Push}
if st.Error == "" {
st.Error = c.pin.Pull
}
for _, n := range c.locked {
st.Locked += n
}
c.mu.Unlock()
c.emit("chat:pin", st)
}11Pull what you missed
On open, the app asks the pin first, then the peers. For each peer it knows, it keeps a read-only replica of that peer's log in the store, and PullCiphertext brings it up to date from the pin. A pin never has entry keys, so what arrives is ciphertext, and describe turns the errors of the pin into sentences the window can show.
Add to pin.go:
// pullFromPin brings the replica of every known peer's log up to date from the pin. The pin sends ciphertext
// only, so an entry is stored but stays unreadable until its key arrives from the sender (see syncFromPeer).
func (c *ChatService) pullFromPin(ctx context.Context) error {
authors := c.authors()
if len(authors) == 0 {
return nil
}
ctx, cancel := context.WithTimeout(ctx, pinTimeout)
defer cancel()
conn, err := kolosecure.Dial(ctx, c.set.PinAddr, c.id, c.set.pinKey, nil)
if err != nil {
err = fmt.Errorf("%w: %w", pin.ErrUnreachable, err)
c.setPin(func(s *pinState) { s.Pull = describe(err, c.set) })
return err
}
defer conn.Close()
var errs []error
for _, author := range authors {
rep, err := c.replica(author)
if err == nil {
c.pullMu.Lock()
err = kolosync.PullCiphertext(ctx, conn, rep)
c.pullMu.Unlock()
}
if errors.Is(err, kolosync.ErrNotFound) {
continue // the author has not pinned anything yet
}
if err != nil {
errs = append(errs, fmt.Errorf("pin pull: %w", err))
continue
}
c.markPinned(author, rep.Len())
c.show(author)
}
err = errors.Join(errs...)
c.setPin(func(s *pinState) { s.Pull = describe(err, c.set) })
return err
}
// describe turns an error of the pin into the sentence the window shows, and "" for no error.
func describe(err error, set settings) string {
switch {
case err == nil:
return ""
case errors.Is(err, pin.ErrExpired):
return "The pin token has expired, so new messages are not pinned. Restart the app to issue a new one, or set a new -pin-token. Messages still go directly to peers who are online."
case errors.Is(err, pin.ErrQuotaExceeded):
return "The pin quota is full, so new messages are not pinned. Raise -pin-quota and restart. Messages still go directly to peers who are online."
case errors.Is(err, pin.ErrDiskFull):
return "The pin has no room left, so new messages are not pinned. Ask its operator for more space."
case errors.Is(err, pin.ErrUnauthorized):
return "The pin refused this identity or token. Check -pin-key and -pin-token, and ask the operator whether your key is allowed."
case errors.Is(err, pin.ErrRevoked):
return "The pin token was revoked or replaced by a newer one."
case errors.Is(err, pin.ErrUnreachable):
return "The pin at " + set.PinAddr + " is unreachable. Messages still go directly to peers who are online."
}
return "Pin error: " + err.Error()
}12What the window does with it
The window gets one more event, chat:pin, with the last pin error and how many messages wait for their key:
const offPin = Events.On("chat:pin", (e) => setPin({ error: e.data?.error ?? '', locked: e.data?.locked ?? 0 }))A message that came from the pin is marked:
{msg.via === 'pin' ? <span className="msg__via"> · synced from pin</span> : null}And an error of the pin, with what to do about it, is shown in plain words:
{pin.error ? <p className="error">{pin.error}</p> : null}
{!pin.error && pin.locked > 0 ? (
<p className="note">
{pin.locked} pinned {pin.locked === 1 ? 'message waits' : 'messages wait'} for the sender to come online.
</p>
) : null}13Run two instances and close one
Start the bootstrap node and a local pin that keeps at most 100 MB, and give the app the pin's public key and a key for the conversation. Everyone who chats gets the same conversation key.
# 1. The bootstrap node (keep it running)
go run ./cmd/bootstrap
# 2. A local pin, with its data in .data/pin and at most 100 MB (keep it running)
go run github.com/ciranda-dev/kolo-pin/cmd/kolo-pin run -data .data/pin -max-bytes 100MB
# 3. The pin's public key, and a key for the conversation
export KOLO_CHAT_PIN_KEY=$(go run github.com/ciranda-dev/kolo-pin/cmd/kolo-pin id -data .data/pin)
export KOLO_CHAT_CONVERSATION_KEY=$(openssl rand -hex 32)Then build the app and start two instances, each with its own data directory and the same two variables:
# 4. Build the app
wails3 build
# 5. Two instances, each with its own data directory (the same variables in both terminals)
./bin/kolo-chat -data .data/alice
./bin/kolo-chat -data .data/bobType in one window and the message shows up in the other. Now close Bob's window, send two messages from Alice, and open Bob again: after a moment the two messages appear in Bob's window, marked "synced from pin", once each and in order. If Alice is not online when Bob opens, Bob gets the ciphertext from the pin and sees "pinned messages wait for the sender to come online" until Alice returns. Start an instance with a wrong or expired -pin-token and it stops with an error that names the setting and the reason.
14What the pin can and cannot see
The pin sees which public keys and log IDs it keeps, how many entries each has and their sizes (kolo-log pads every entry), when a log grows, the IP address and timing of each connection, and your capability: its expiry, its quota and the log it covers. It cannot read a message: it never gets entry keys, and even someone who holds entry keys finds only text sealed with the conversation key.
When the token expires, the pin refuses new pushes and deletes your log; the window says so, and direct delivery goes on. When the quota is full, the pin refuses a push that would go past it, keeps what it has and does not evict anything; new messages are not pinned until you raise -pin-quota and restart.
Know the limits of this step. Entry keys come only from the sender, so a message written while you were offline is readable once its sender is online. The finished example, Kolo Chat, lifts this one: every member derives the entry keys from the conversation key with kolo-log's derived entry keys, so a message is read from a pin while its sender is offline. There is no delegation: each member issues its own capability, and the pin's operator lists the members it accepts. First contact is direct, because a peer's key and log ID come from a hello on the direct connection. Messages are ordered per sender. And there is no way yet to remove a member or rotate the conversation key. The finished example shows one list of rooms, one for each conversation, and its pin is optional: without one, messages reach people only while they are online.
Next steps
- Read how to run a pin for real: Run a pinning peer on a VPS, Monitoring a pinning peer and Choosing pinning peers.
- To compare your files with the finished ones, check out the tag
step-3of the starter repository, or see the kolo-chat-wails3-example repository of the ciranda-dev organization.