Cesar Petrescu

Lab · Infrastructure

  • Production
  • Mail Infra

Distributed Mail Services

Production mail stack, thousands of emails/month with zero-downtime upgrades

Distributed mail infrastructure handling thousands of emails per month across active-active Stalwart nodes. Survives rolling upgrades and node failures without taking mail offline. An automation layer on top handles domain onboarding, DNS configuration, and mailbox deployment in one step.

  • Stalwart
  • FoundationDB
  • MinIO
  • NATS
  • Email Hosting Platform
  • Linux
  • Docker

Open the project

How it works

How a message crosses the distributed mail fabric

The mail path separates protocol handling, coordination metadata, large blobs, and asynchronous events so a node can be upgraded or lost without making every mailbox unavailable.

  1. Accept
  2. Coordinate
  3. Store
  4. Recover

Step 1 of 4 · Stalwart · SMTP · IMAP

An active Stalwart node accepts SMTP or serves IMAP

Clients reach one of the active-active mail nodes for submission, relay, or mailbox access. Domain and mailbox configuration is created by the automation layer instead of repeated manual setup.

What moves forward An authenticated mail operation that is not pinned to one server.

Step 2 of 4 · FoundationDB

Shared metadata keeps nodes consistent

Stalwart uses FoundationDB for mailbox metadata, coordination, and small state. Any healthy node can read the same logical mailbox view, avoiding a single-node metadata dependency.

What moves forward Consistent message and mailbox metadata across active nodes.

Step 3 of 4 · MinIO · NATS

Large message bodies and attachments live in object storage

MinIO holds durable blobs separately from coordination data, while NATS publishes routing, automation, and audit events to asynchronous workers without blocking the protocol-facing node.

What moves forward Durable content plus decoupled events for downstream processing.

Step 4 of 4 · Linux · Docker · Backups

Traffic continues while a node changes or fails

Rolling upgrades remove one node at a time while peers continue serving mail. Backups and restore checks cover shared state and blobs, and failed nodes can be rebuilt without relocating every mailbox.

What moves forward Continuous mail availability with explicit failure domains.

System map

  • Request
  • Engines
  • Data
ClientsClientsCoreCoreDataDataSMTP clientssubmission + relaySMTP clients — submission + relayIMAP clientsmailbox accessIMAP clients — mailbox accessStalwart nodesactive-activeStalwart nodes — active-activeRouting workersasync processingRouting workers — async processingFoundationDBmetadata + coordinationFoundationDB — metadata + coordinationMinIO clusterattachments + blobsMinIO cluster — attachments + blobsNATSpub/sub event busNATS — pub/sub event bus

Tap a service to see what it calls and what calls it.

Clients

Core

Data

Components

Goals

  • Eliminate single-node mailbox dependency
  • Keep mail available during rolling upgrades
  • Event-driven hooks for routing, automation, and audit
  • Production volume: thousands of emails/month

Automation layer

  • Internal email hosting platform for faster rollout and lower operational overhead
  • Automates domain onboarding, DNS configuration, and mailbox deployment on top of the distributed stack

Data model

  • FoundationDB for coordination + small metadata
  • MinIO for large blobs and attachments

Ops

  • Node failure tolerance
  • Backups + restore verification
  • Clear failure domains and fast rebuilds

Architecture

Mail Nodes
  • Stalwart (active-active)
  • SMTP/Submission
  • IMAP access
State
  • FoundationDB (metadata + coordination)
Storage
  • MinIO cluster (blobs + attachments)
Event Bus
  • NATS pub/sub (routing + async workers)