🐫 Songs of the OCaml Compiler Part I: A Proxy Project

I recently found myself with some time and the chance to finally pick up OCaml. Its use in correctness-first systems is what had first caught my eye back in my University days If I remember right, the earliest glimpse I had of OCaml in industry came from this talk, hosted by the school’s hacker community where an alumni from a prop-trading shop came down to share what they were building. ; I never had the chance to explore it deeply. Happily, this opportunity also aligned with a long-term personal goal: to build meaningful pedagogical artefacts An example: meaningful guided projects with an effortful learning curve. for FOSS ecosystems. Teaching people how to teach themselves is difficult because learning is a deeply personal habit and not every skill is learned the same, but a reliable start to exploring that is to make my own learning style visible.

To that end, I set an impossible goal for myself: to spend 2 weeks learning OCaml’s core patterns-of-thought and to gain some intuition around the expressiveness that it affords. To surface as much friction as fast as possible, I designed a proxy project A proxy project is a deliberately constructed side project whose primary purpose is to catalyse learning or exploration in a new domain. Instead of pursuing the project’s outcome as an end in itself, the creator uses it as a proxy for skill-building, intuition-forming, or concept-testing under realistic conditions. Disclaimer: I’m coining this term, it doesn’t have a formal definition outside of this series (yet?) to act as the learning substrate: a deterministic, dependency-free event-simulator for illustrating the classic Paxos protocol. ~4 weeks into it scope-creep is inevitable for personal projects and the project is at a stage where I can focus on talking about it and getting some feedback to improve on the goals that it chases.

This forms the backdrop of a 3-part series, written primarily for a technical audience A little programming experience goes a long way here. I won’t focus heavily on syntax, though OCaml’s consistency makes it very readable. For code-examples, I’ll describe intent wherever possible so that unfamiliar syntax never gets in the way. Part 2 will be a little more involved at certain places because it will focus the value that the type system unlocks, but non-technical folks may just gloss over them. and occasionally wandering into personal narrative musings.

• Part I covers the outcomes of proxy project, the didactic motivations behind the challenge and establishes the foundational design grammar that anchors the architecture

• Part II (🚧 wip, publishing soon 🚧 ) examines implementation; key subsystems, design trade-offs and some early commentary on the ergonomics around the OCaml language, by way of example

• Part III (🚧 wip, publishing soon 🚧 ) closes with how this project may evolve and some retrospectives on this challenge

Tuaregs & Trans-Saharan Trade Caravans

Back in the 8^th Century, communities such as the Tuareg It’s fascinating how we in the tech community name things. The major Emacs mode for OCaml is called “Tuareg”. The Tuareg people are still here today: “The caravans, albeit on a much smaller scale than in their heyday, are still going today. Saharan salt from Taoudenni is still transported by Tuareg camel caravans, the 90-kilo slabs now ultimately destined for the refineries of Bamako in Mali.” Read more here. shepherded vast caravans across the Sahara – long, shifting formations of camels carrying salt, spices and stories. Without having any instruments other than the stars and their memory to guide them, the caravans held loose formations, sometimes hundreds of meters apart, moving in parallel across an indifferent sea of dunes. To coordinate anything between formations – navigation, water, trade-routes – they relied on human couriers that rode the fastest dromedaries “the Moroccans were successfully breeding camels on a huge scale and they even created a cross-breed between the dromedary camel and the two-humped Bactrian camel of Asia (Camelus bactrianus). The result of these experiments produced two variants of dromedaries: a sleek, fast-running camel useful for messenger services and a heavier, slower camel that could carry more weight than the pure dromedary.” Read more here. the desert could produce in order to send messages. This worked at the mercy of the desert, which posed a variety of challenges. Couriers might vanish behind a dune, or return dazed from the heat. Camels might collapse. Sandstormes might roll in and, for hours or days, erase all paths between the formations without any way to communicate between them…

This makes a rich narrative backdrop for illustrating the Paxos protocol. Each caravan formation becomes a participant in our distributed system; each runner (courier) a fragile network packet; every fainting camel, a node crash; every sandstorm a network partition. Even amongst these sources of chaos, it is possible to achieve stable consensus provided the rules are right.

Running the Simulator

Picture this scenario: there are five caravans These are “formations” that need to coordinate with each other. In our distributed system of caravans, these are the peer nodes that have to collaborate. In the simulator, you’ll see “Nodes” sending and receiving messages, making decisions and reacting to situations. Every node is equal in standing, no node is superior to another. There is no king. — Aghilas, Taziri, Amastan, Itri, and Tassadit — which will cross the Sahara by relying on human messengers to carry parchment slips For our distributed system, these are messages that will be sent over a communication channel. For now, we’ll just call the channel a message bus. A particular formation (node) has the ability to send a message to another formation (i.e. a unicast message) or they may send a message to ALL other formations (i.e. a broadcast message). between groups. We will witness Keep an eye out for “Narration” blocks displayed within the simulation! the messenger from Tassadit collapse early making Tassadit unable to dispatch messengers and so they will be unreacheable. A storm will then reshape the formations such that two-subgroups form and any formation from one subgroup will be unable to communicate to a formation from the other sub-group Essentially, a subset of nodes falling into a separate network partition, thereby preventing communication between peer nodes across network partitions. Nodes within the same partition would will be able to communicate with each other. because no messengers will be able to go through the storm with huge dust-clouds. Caravan Aghilas will attempt This node will kickstart the paxos process by seeking permission to make a proposal to propose something but that will fail to gather a quorum because of the partitioning. After the storm clears, Tassadit’s messenger would have recovered and a fresh proposal from Itri will go through the entire paxos process and the group will agree on which campsite to head towards.

We shall let the simulator show us this. Running a single make magic command after cloning the repo from here spawns the simulator This assumes that you are on a unix machine and have OCaml setup locally. Otherwise, the README::Quickstart offers setup-help, if needed. There’s a convenience setup script that you can consider running by just calling make setup. Please read all scripts before running them!

Under the Hood

The rest of this post is about motivations and some thoughts on abstract design. If you’re more excited by code, please jump straight to Part 2 (🚧 wip, publishing soon 🚧 ) to see how the simulator is built.

Didactic Motivations Behind the Proxy Project

For me, this project has elements of what a good proxy-project may look like. This section attempts to articulate such elements and why specifically a Paxos Simulator.

Elements of a Good Proxy Project

• Offers Productive Friction that Fast-Forwards Understanding [Desirable Difficulty]

  Proxy projects thrive on desirable difficulty A concept introduced by psychologist Robert A. Bjork in 1994, it describes learning challenges that improve long-term retention and understanding. Bjork suggests that this outcome requires (1) learners perceiving success as achievable, (2) steady progress toward goals, and (3) calibrated difficulty levels that avoid overwhelming learners. This principle is supported by frameworks like the New Theory of Disuse, Challenge Point Framework, and Cognitive Load Theory, explaining why strategies such as testing, spacing, and interleaving strengthen durable learning despite initial effortful experiences. For more detail, see Desirable Difficulty (wiki) and Bjork Lab Paper. , a principle from learning science where effortful challenges forge lasting understanding. The productive friction that desirable difficulty unlocks helps hone our insight by spotlighting mismatches between assumptions and reality; mistakes that serve a diagnostic role to refine our mental models.

  It’s also a skill to be able to judge when to tolerate these mismatches. Not every rabbit-hole needs exploring, not every instance of friction needs attention else we’d end up trying to boil every ocean we encounter – it’s sometimes sufficient to just know it exists.

• Gives Good Analogies for Cognitive Scaffolding

  High-friction tasks build on what we already know, helping us quickly spot patterns and make sense of new domains. Patterns are like analogies This has to be an example of a meta-analogy, right?? — they map something we already understand onto something unfamiliar.

  From a constructivist view, we don’t just internalise new ideas whole; we piece them together by hooking onto things we already understand. Good analogies give us “hooks” to hang new ideas on. Analogy ≠ surface similarity. This comes from Dedre Gentner’s Structure-Mapping Theory, which shows that good analogies aren’t about looking alike on the surface but about matching the deeper relationships. The trick is that a good analogy helps us think differently by highlighting how things connect below the surface and in which areas they may not map to reality. Gentner’s 1983 paper breaks this down nicely.

  Practically, a good analogy To me, a good analogy is intentionally reductionist: it makes the essential dynamics legible while inviting us to question the edges where the mapping breaks. I’d say that boundary-defining is bread and butter for any engineer. cuts through the noise: it narrows down where we should focus, so we put our effort in the right spots. That’s why analogies aren’t crutches – they’re scaffolds holding up an idea just long enough for us to really get it.

• Has Opinionated Guardrails to Inherit / Reject Technical Taste

  Personally, crafting code taps more onto my intuition than my logic. Heuristics that we gain from exposing ourselves drive this intuition. These are the rules of thumb that people internalise over years – patterns like “favour composition over inheritance” or “write shallow interfaces, for deep modules”. They’re not exactly laws, but they’re the instincts that separate good design from brittle hacks.

  In this proxy challenge, when working with a language like OCaml (strong type system, has a bias for functional code) we start feeling the pull of its idioms: e.g. immutable data, pattern matching, error-handling primitives. We might love and internalise some of them while rejecting others because they don’t fit our mental model well. That’s fine; either way we’re taste-making This is why I gravitate toward gaining exposure to problems over other tasks like formal verification or micro-optimizations. I want to absorb the rhythm of a language – how it makes certain problems feel elegant and others feel clunky. That’s the real signal for whether a tool deserves space in our mental toolbox. by training our intuition – learning to “sense” when a design “feels right” versus when it’s fighting the language.

Why a Paxos Simulator ?

To be honest, I didn’t begin with a Paxos simulator in mind for the proxy project. A showerthought (showerquestion?) about “what’s the best technical inbound marketing I’ve seen” reminded me of TigerBeetle’s Simulator (please read their fascinating writeup). Beneath its cutesy surface, it’s a carefully constructed analogy that maps core concepts of a financial ledger onto little animated animals. The brilliance is that this analogy travels: it gives engineers, PMs, procurement folks – anyone adjacent to systems work – a way to feel the underlying rules of the product. A good simulator invites “what if?” questions from people who normally never ask them. Suddenly someone in sales is asking about throughput collapse under backpressure or failure cascades. That’s the power of a well-designed epistemic engine: it lowers the barrier to intuition An epistemic engine is a tool, like a simulator or model, designed to make complex concepts tangible and intuitively understandable without oversimplifying. It works as a scaffold for learning—similar to Wittgenstein’s ladder metaphor – helping users climb toward deeper insight by providing a temporary framework that can be “discarded” once true understanding is reached. For further reading, see Wittgenstein’s ladder and Stanford Encyclopedia on Wittgenstein. without diluting the complexity underneath.

Distributed Systems are especially suited to this kind of treatment because they are inherently invisible. It’s hard to learn because it’s hard to visualise. A simulator brings those hidden dynamics to the foreground. With a deterministic engine, you can replay failures, slow down the world, poke at quorums, watch messages reorder, and actually see why certain invariants matter. It turns reasoning about the system from “remember the rules” into “observe the behaviour,” which is a more native way to internalise distributed thinking.

Enter Paxos: a classic algorithm; “challenging” but sharply defined, famously subtle yet fundamentally simple once the invariants click. It forces us to inhabit the mindset of actors exchanging messages in an unreliable world. It’s rewarding when we connect the dots right – e.g. when we see an acceptor broadcast to itself and realise that this tiny behavioural choice collapses a bunch of edge-cases which we’d no longer need to account for. I think people describe Paxos as difficult I think that Alex Kladov’s (TigerBeetle Engineer) post on this is probably the typical learning journey for this. Funnily, I only saw this post after I was done, at the time of writing this post instead of when I was bridging my own knowledge gaps because its rules are easy to memorise but hard to feel.

A simulator reverses that: it makes the feeling primary and the formalism derivative.

Linguistic Relativity: What language do you think in?

Behind all these motivations sits a deeper question actually: “What does it mean for a language to shape the way we think?” Before we move onto the abstract design for the simulator, it’s worth stepping back and examining how languages – both natural and programming – shape habits of attention.

There is a long-standing idea in linguistics that argues that our worldview is greatly influenced by the perspectives that we get from the languages that we speak: linguistic relativity Prussian philosophers, Wilhelm von Humbolt and Johann Gottfried Herder, were the first to articulate this. They considered language as the expression of the spirit of a nation – presenting it to be “[the] forming organ of thought” and the link between language and thought as a continuous feedback loop where our thoughts shape our words and words shape our thoughts. The history of this school of thought is fascinating and somewhat controversial. If you’re interested in exploring this rabbit-hole, I would recommend starting with this Aeon essay by the linguist James McElvenny that looks at this with a historical lens. . You see this more clearly if you grow up in a multilingual/multicultural place, that each language brings its own habits of attention.

Natural Languages and Habits of Attention

Early in my childhood, I used to think in Odia A language that hails from the East of India; one of the 22 officially recognised Indian languages. India is huge and the number of languages & dialects are in the hundreds. with some Hindi and English that I picked up from watching TV. Since my formative years were spent in Singapore, it was easier to see how people’s cultural background influenced their use of a common medium, English. Some curious observations:

• I have the habit of anthropomorphising things – instead of saying “that bottle dropped from the table”, I say “that bottle fell from the table” – which likely comes from a Hindu upbringing with concepts such as all things have a soul (ātman).

• My Chinese-speaking friends lean heavily into concepts packaged as imagery in the forms of idioms, metaphors and compositional memory. It likely comes from their language being based on symbols instead of an alphabet so Chinese speakers develop strong cognitive associations between images and abstract ideas. Here’s a surprisingly ubiquitous reference that involves a parable of an old man losing his horse.

• Most Malay speakers I know have a strong preference for simplicity and clarity when they use English, which is likely influenced by Bahasa’s direct, economical grammar. The clarity that comes from this is best seen when people play team-sports or conduct military drill.

Over time, English became the primary language that I “think in”, but the earlier layers of influence will never disappear. What has emerged instead is a kind of cognitive code-switching: not just between parts of different languages but between habits of using English that I picked up from others around me. It’s one of the quiet privileges of growing up multilingual/multicultural: you get to choose the mental lens that best fits the moment.

Programming Languages and Habits of Thought

I see Programming Languages in a similar vein – they are languages that cultivate habits of thought. Each one leaves cognitive residues: mental models, instincts, and ideals for what “good” looks like. These contribute to “technical taste”: the instinct for sensing when a design is elegant, when an abstraction is honest or whether a module is at the wrong conceptual depth. For me, (attempting to) hear the compiler sing is more about drawing from a deeper well of inspiration rather than being dogmatic about patterns In fact, the more we get exposed to things, the more accurately we are able to modulate how strongly to voice a technical judgement. The more we learn, the less dogmatic we become.

Before we start thinking about the architecture (Part II) we need to lay some foundations by focusing on the abstract design of our system – especially how defining what NOT to care about clarifies what matters most.

Abstract Design Grammar: Pressure before Shape

In this phase, no code is written.

We don’t reach for data-structures, or types, or “The OCaml Way” of doing things. We just listen – observe, model and articulate the shape of the problem until the architecture begins to reveal itself.

This phase is spacious on purpose. Mistakes are cheap here. Analogies can be stretched, redrawn, or tossed. We can identify which contracts actually matter and which ones only feel important because a particular tool nudges us in that direction. By the time any code is crafted, the architecture should feel like something we uncovered, not something we forced.

There’s a sobriety to this style of design: we decide what NOT to care about first – constraints before constructs.

Constraints before Constructs

Designing any system begins with an inventory of boundaries – the agreements we will honour, and the things we will deliberately ignore. Constraints are not limitations; they are the material of good abstractions. Before we know what to build, we need to know what must never break (system invariants), what can be safely elided, and what correctness even means.

For this challenge, we’re building a Paxos simulator, not a proper distributed system. Those are different ambitions. One teaches; the other survives the real world. A simulator doesn't immediately need to handle It’s very important to know that the project will naturally need to evolve into one where the simulator simulates this and actually handles this. Otherwise, what would be the point of faking a distributed system and writing this code? clock drift, node discovery, or complex failure modes. It needs to let us pause the world, zoom in, slow down time, and observe causality with surgical clarity. It must be deterministic, replayable, and enjoyable to explore. These constraints simplify everything for the first version (v0) of this project. They also sharpen the didactic edge of the project.

Listening for Structure

If you freeze the world of our camel caravans – holding the wind still, pausing each formation mid-stride – four kinds of structures make themselves known:

• caravans with private commitments \(\implies\) local state
• couriers shuttling parchment \(\implies\) messages
• desert sun rising and falling \(\implies\) logical time
• chronicler observing from afar \(\implies\) simulator interface

But notice: we don’t jump to modules or types yet. At this level, these aren’t subsystems; they’re forces. They’re the pressures that will eventually shape subsystems. They describe “what must exist” but not “how it should be implemented.”

This is the most underrated part of software design. The system is already whispering its structure – you just have to be attentive enough to hear it.

What we Choose Not to Model

A good amount of clarity comes from omission.

For v0, we shall omit:

• failure detection
• node catch-up
• real-world liveness guarantees
• true concurrent execution

They’re important, but they’re important later. They don’t reshape the fundamental architecture; they only layer more behaviour onto it.

Leaving them out lets the core problem – consensus emerging from unreliable communication – speak more loudly.

This is the part of design work that feels like taste. You protect the essence of the problem by removing distractions.

The Grammar

Before anything becomes code, an abstract grammar emerges – not a diagram, just rules of interaction:

• Nodes talk only by exchanging messages
• Time moves in discrete, inspectable steps; decoupled from the wall-clock
• Events are deterministic and replayable
• No one reaches into anyone else’s state
• The world (scheduler) can be paused, fast-forwarded, and rewound
• The story of execution must be readable – a chronicle you can follow

For instance: “Nodes talk only by exchanging messages” means that in Part II, you’ll never see one node directly calling another node’s method or reading its internal state. The Bus is the only communication channel – a constraint that makes reasoning about causality trivial.

This grammar is what Part II is built on. It shaped the OCaml implementation more than OCaml shaped it.

Onward to Part II

Part II takes us from forces to forms: how the abstract grammar manifests as OCaml modules, how the caravan metaphor becomes a type-safe Node signature, how message-passing becomes a Bus with routing logic, how logical time becomes a Scheduler with event queues. We’ll see how OCaml’s type system enforces the contracts we’ve just described, how functors let us parameterize behavior without sacrificing safety, and how the simulator harness makes the invisible visible.

If Part I is the why and the what, Part II is the how – the architectural choices, the trade-offs, the moments where the language sang and the moments where it fought back. The grammar is set. Now we build the instrument.