Posts

Adding replication to the Key-Value Store

During the past few weeks I’ve built a basic key-value store that persists data to disk. It works, but only as a single node, so there is a fixed limit on the amount of data it can store, and the througput it can sustain.

We can make the store scale horizontally in two ways:

• Replication, which keeps additional copies of the data in other nodes, improving throughput.
• Sharding, which partitions the keys across different nodes, so that we can store more data.

In this post I’m going to tackle the first one, replication, refactoring the storage layer so that we can have as many replicas as we want serving reads. All the code is available in GitHub.

Notes on Eio

This weekend I took a step back from working on my Key-Value store to make sure I understand the concurrency model of Eio, the OCaml 5 library that I’m using for concurrency and parallelism.

I decided to turn my notes into a post to force myself to address any gaps in my understanding. I’m sure that there are several errors, or things to improve, so if you are reading this and find some, please, ping me on Twitter!

A basic Write Ahead Log

This weekend I decided to add some basic persistence to my Key-Value store. I considered going directly into B-Trees or LSM, but they are quite involved. Moving, forward I want to focus more on the distributed side of the store rather than low level storage details (saving those for later!), so for now I decided to implement something simpler, a basic Write Ahead Log (WAL, for short).

What is a Write Ahead Log?

This post from Oskar Dudycz does a much better job than I could explaining the theory, so I will just give the high level summary.

Giving OCaml a try

During the last couple of weeks I’ve gone on a completely different direction. I happened to hear a few people saying good things about OCaml, and I was kind of missing coding with strong types, so I decided to reimplement my basic key value store (see the post), and see if it was fun to write.

First of all, here is the result. The functionality is the same, a hash map over a network, using an “official” async library, and a basic client and server that receive and send S-Expressions over the wire.

Using Zig from Common Lisp

2025-03-12 Update, as people on Twitter, Lobsters, and Reddit pointed out, I was missing a extern in the struct, see the last section!

Last week I started playing with my own toy key-value store (see the previous post). At the end I got to a hashtable exposed over the network, using a protocol based on S-Expressions. For the next steps, I have two alternatives, I can work on the low level representation of the data, maybe implement B-Trees, and some storage, or I can go up instead, and see how can I make it distributed, and play with some nice algorithms.

Redis from scratch, or Cledis

This week I was looking for something new, and I realised that I’ve never tried to do any network programming using Common Lisp. Also, some time ago I came across the Build your own Redis book, but never tried to go through it. The project is perfect, it contains threading, networking, and even some async programming. I had no idea what the Common Lisp ecosystem for those things looks like, so it was time to find out.

Base64 Encoding & Decoding

Looking for a first project to play with, I remembered CryptoPals. I did some of it several years ago, and it was fun, so I thought it could be a good way to get familiar with Common Lisp, at least the first set. With that in mind, I had a look, and hear we are!

The very first challenge is about hex and base64 encoding. The first time around I just used available libraries for both (I used Rust at the time), but the point of this was to open some black boxes, so I decided to implement everything from scratch!

Back to basics

During the past few years my work as a software engineer has involved less low level, algorithmic coding, and more high level strategy, architecture, dependency and team interactions. While that makes sense as careen progresion (I have more impact that way), the truth is that I miss coding things from scratch.

Also, with the coming of our AI overlords, I keep on hearing that these skills, that I’ve spent years improving, are going to, if not disappear, become less relevant. In the not so far future, we will be able to pay a fee to one of the remaining AI providers, and ask an agent to generate our app, and set up a new business from scratch. In that world nobody will need to look at code anymore!