kolo docsbeta

Choosing pinning peers

Keep each log on two or more pinning peers in different places, picked the same way by every device.

On this page

One pinning peer can go offline or lose its disk, so an application usually keeps each log on two or more of them, run by different people or in different places. kolo-pin picks them with Select and adds the log to them with AddMany.

Give each peer a group

The application keeps a list of pinning peers, each a pin.Peer with its key, its IPv4 address and an optional Group, such as its provider or region. Peers in one group can fail together, so Select takes chosen peers from different groups when it can.

How Select picks

All steps

How a pin is chosen

Rendezvous hashing: every device computes the same choice for a log, with no coordination. Weights here are shortened.

  1. For each pin, the client computes a weight: SHA-256 of the pin's ID and the log ID.
  2. The pins are sorted by weight, heaviest first.
  3. Select takes the heaviest pin of each group until it has enough copies, 2 by default. Pin 1 is skipped: its group already has pin 3. With too few groups, the next heaviest fill in.
  4. Add pin 5, in a new group. For this log it ranks second, so it takes the place of pin 4. A log where it does not rank stays where it is.

Select(peers, logID, n) uses rendezvous hashing (Thaler and Ravishankar, 1998, the original paper on it). Every pair of a peer and a log gets a weight, the SHA-256 hash of the peer's ID followed by the log ID. The peers with the highest weights for that log win.

  • Every device that has the same list picks the same peers for a log, with no coordination. The order of the list does not matter.
  • When a peer joins or leaves, only the logs for which that peer is among the chosen ones move. Every other log stays where it is.
  • Each log gets its own order, so the logs spread evenly over the peers.

Going down that order, Select takes the first peer of each group until it has n. If there are fewer groups than n, the next heaviest peers fill in and SharedGroups is true. A peer without a group counts as its own group, peers with an invalid ID are skipped, and of repeated IDs one is kept. n of 0 or less means DefaultCopies, 2.

Add the log with AddMany

A capability names one pinning peer, so IssueMany signs one for each peer, all with the same issue time. AddMany(ctx, id, peers, log, capabilities, n) adds the log to the n peers Select chooses, using the capability that names each one.

When a chosen peer is unreachable, AddMany tries the next peer in the same order that was not chosen, at most MaxFallback (2) times. That next peer can be in a group already used. A peer that answers with an error is Refused and is not replaced. The Report lists every try with its Outcome (Accepted, Refused or Unreachable) and counts how many accepted. AddMany returns ErrNoPeer only if none did, and there is no retry in the background: call it again later.

This program runs three pinning peers on loopback in two groups, with their data in a temporary directory, and pins one log to two of them:

main.go
package main

import (
	"context"
	"fmt"
	"log"
	"os"
	"path/filepath"
	"sync"
	"time"

	koloid "github.com/ciranda-dev/kolo-id"
	"github.com/ciranda-dev/kolo-pin"
	kolosecure "github.com/ciranda-dev/kolo-secure"
	kolostore "github.com/ciranda-dev/kolo-store"
)

func main() {
	dir, err := os.MkdirTemp("", "kolo-pin-choose")
	if err != nil {
		log.Fatal(err)
	}
	defer os.RemoveAll(dir)
	var wg sync.WaitGroup
	defer wg.Wait()
	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
	defer cancel()

	// Three pinning peers in two groups; an application reads this list from its configuration.
	var peers []pin.Peer
	for i, group := range []string{"provider-a", "provider-a", "provider-b"} {
		d := filepath.Join(dir, fmt.Sprint("pin", i))
		id, err := pin.LoadIdentity(d)
		if err != nil {
			log.Fatal(err)
		}
		ln, err := kolosecure.Listen(ctx, "127.0.0.1:0", id, nil)
		if err != nil {
			log.Fatal(err)
		}
		defer ln.Close()
		wg.Go(func() { pin.Serve(ctx, ln, pin.Options{Dir: d}) })
		peers = append(peers, pin.Peer{ID: id.PublicKey(), Addr: ln.Addr().String(), Group: group})
	}

	alice, err := koloid.Generate()
	if err != nil {
		log.Fatal(err)
	}
	st := kolostore.New(kolostore.NewMemory())
	defer st.Close()
	l, err := st.Create(alice)
	if err != nil {
		log.Fatal(err)
	}
	if _, err := l.Append([]byte("chat history")); err != nil {
		log.Fatal(err)
	}

	sel := pin.Select(peers, l.ID(), 2)
	fmt.Printf("chosen: %s and %s, shared group: %v\n", sel.Peers[0].Group, sel.Peers[1].Group, sel.SharedGroups)

	// One capability per peer, so a fallback peer has one too.
	caps, err := pin.IssueMany(alice, peers, pin.Capability{Expires: time.Now().Add(30 * 24 * time.Hour)})
	if err != nil {
		log.Fatal(err)
	}
	report, err := pin.AddMany(ctx, alice, peers, l, caps, 2)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("accepted by %d peers\n", report.Accepted)
	for _, r := range report.Results {
		fmt.Printf("  %s: %v, %d entries\n", r.Peer.Group, r.Outcome, r.Info.Entries)
	}
}

It prints something like this. The two groups can come in the other order, since the keys are new each run:

Text
chosen: provider-a and provider-b, shared group: false
accepted by 2 peers
  provider-a: accepted, 1 entries
  provider-b: accepted, 1 entries

The program signs a capability for every peer in the list, not only the chosen ones. A fallback peer without a capability that names it is Refused.

Next steps