The problem

The ERP ran on one machine. Any hardware fault, any kernel update, any disk filling up meant the whole distribution network stopped until someone fixed it by hand. Managed database services were not an option, so redundancy had to be built on hardware we already owned.

Approach

Three bare-metal servers, with every layer given its own way of surviving a node loss rather than relying on one clever trick.

• Database. Patroni holds an election through etcd and promotes a replica. PgBouncer sits in front, so the application reconnects without knowing anything moved.
• Entry point. Keepalived carries a floating IP that follows whichever machine is actually serving, rather than always sitting on the first one.
• Sessions. Redis Sentinel across three nodes, so a logged-in user stays logged in through a failover.
• Filestore. A three-node MinIO quorum, which keeps serving with one node gone.
• Application. Odoo replicas on Docker Swarm behind Traefik, with health checks removing a dead replica from the pool.

Outcome

Failover was tested by killing machines rather than by reading documentation. Any single server can go down and the cluster keeps serving. Approved for production rollout.

The repository is a documentation repo first: fifteen documents covering every layer, the reasoning behind each choice, and the failure modes, alongside the tuned configs themselves.

The problem

Over a hundred Odoo instances spread across several servers, each one needing backups, restarts, module upgrades and health checks. Doing that by hand does not scale, and doing it inconsistently is worse than not doing it.

Approach

Ansible playbooks that treat the fleet as one target instead of many.

• Backups run nightly, dumping the database and packing the filestore in a format the Odoo web interface can restore directly.
• Health checks curl every domain and stay quiet unless something is actually down, so the alert channel keeps meaning something.
• Upgrades pull from git, run the module upgrade, and report the real Odoo error lines when one fails rather than a generic failure.
• Everything runs async and in parallel rather than one server at a time, which turned a fleet-wide operation from hours into minutes.
• Alerts are batched into one message per server instead of one per instance, which is the difference between a useful notification and a flooded channel.

Outcome

Running in production. Metrics land in Prometheus and Grafana, and failures reach Discord with enough context to act on without opening a terminal first.

The problem

An ERP enforces most of its rules in application code. When another system writes to the same tables directly, those rules never run: the check exists, it just does not fire. The result is records that look fine to whoever wrote them and wrong to everyone downstream.

Approach

A watchdog that verifies the result instead of trusting the writer, running outside the application entirely.

• Push to register, poll to verify. The ERP notifies RIO when a record is created. RIO holds that id on a short watch list and reads it back a few minutes later, so it sees the row as it ended up rather than as it was intended.
• One batched read covers every watched id, so the cost does not grow with the size of the list.
• Alerts are coalesced into one Discord and one email per tick, not one per anomaly.
• Known corruption is repaired automatically, and the rest is reported.
• A dead-man switch sends a heartbeat to an external receiver while healthy, because a dead process cannot alert anyone that it died.

Outcome

A single static Go binary carrying its own scheduler, web dashboard, and state store. It also watches for scheduled jobs that quietly stop running, which is a failure with no error message and no crash, only work silently piling up.

The problem

Three separate processes still ran on Word documents and mail merge: onboarding a new distributor, issuing the distributor agreement, and circulating internal memos. Each one produced files that lived on somebody’s machine, with approval happening over chat and signatures added by hand. Nothing connected the document to the approval that authorised it.

Approach

One system covering all three, each keeping its own flow.

• Partner onboarding. The master data form became a record with a multi-level approval chain, and the partner is created from the approved form rather than typed in again afterwards.
• Agreements. Contracts are generated as PDFs from templates, so the terms come from the record instead of from whichever copy of the document someone started from.
• Memos. Generated per recipient and stamped with the approval signatures that authorised them, replacing the mail-merge step entirely.

Outcome

The document and the approval that produced it are now the same record. Reprinting is deterministic, and there is no version of a contract circulating that the system does not know about.

The problem

A factory running its production on spreadsheets and a general-purpose ERP that did not model how the plant actually worked. Costing in particular was the weak point: what a finished product truly cost, once waste, repacking and conversion were accounted for, was not something the system could answer.

Approach

A full Odoo 16 build shaped around the plant rather than around the software.

• Manufacturing orders carrying the real bill of materials, including by-products.
• Waste tolerance modelled explicitly, so expected loss is planned rather than discovered at month end.
• Conversion cost applied per operation, so cost accumulates along the route.
• Repacking handled as its own flow rather than forced into a standard transfer.
• Cost structure reporting that shows how a finished unit arrived at its cost.

Outcome

Production, inventory and accounting run off the same records, and cost structure is answerable per manufacturing order instead of reconstructed afterwards.

The problem

The unit ran on Microsoft Dynamics AX. Moving to Odoo 16 meant that every workflow the business depended on had to exist there before anyone could switch, and standard Odoo covers maybe half of them. The other half were the ones the business actually argues about: how a credit limit is decided, when an invoice may be reprinted, what counts as a valid load.

Approach

Not one migration project, but a running list of workflows rebuilt one at a time, each shipped and used before the next was started. Eighteen so far, including:

• Order to cash. Delivery-driven invoicing on a strict one-delivery-one-invoice basis, sale order validation on processing, and cancellation reasons captured per line.
• Credit control. Credit limits generated from a configurable formula master rather than set by hand, with an approval flow above them.
• Receivables. AR and AP aging reported per branch, and cheque status monitored from collection through to clearing so an invoice cannot read as paid before the money exists.
• Warehouse. Available stock exposed next to on-hand, a multi-line scrap journal, and load matrix validation on purchase orders.
• Expenses. Petty cash handled on top of the native expense flow, routed to the right branch account.
• Documents. Invoice, delivery and monitoring reports rebuilt to the business templates, with reprints watermarked so the original stays distinguishable.

Outcome

The unit runs its day-to-day on Odoo rather than AX, and the workflows that were specific to how this business operates survived the move instead of being flattened into whatever the software did by default.

The setup

Every distributor runs their own instance, isolated from the others. I work across that estate: building what the business needed on top of it, and keeping it running.

What I built

• Realtime field visibility. Visit activity streams out of every instance through middleware into one dashboard, with per-instance isolation preserved through the pipeline so a distributor’s data stays theirs. Reporting never queries a production instance directly.
• A flat replica instead of live queries. Visit lines are replicated into a flat table with daily aggregates, because a dashboard reading transactional tables gets slower every month it runs.
• Exports finance asked for. Invoice line data out to CSV by month range or in full, from a menu next to the invoices themselves rather than a report someone has to request.
• Containerised, so the fleet is operable. Which is what makes running this many instances possible at all, and is covered separately by the automation that maintains them.

Outcome

One live view across the network, and instances that can be operated as a fleet rather than individually.

The problem

Chasing overdue invoices was manual. Someone had to pull the aging report, work out which customer owed what and for how long, then message each of them individually. It happened when there was time for it, which is not the same as when it was needed, and there was no record of who had actually been contacted.

Approach

Compute the aging, generate the message, send it, and keep the receipt.

• Aging is computed per customer from invoice data into its own staging table, rather than read off a report.
• Each customer gets their own message, listing their own outstanding amounts by age bracket.
• Delivery over WhatsApp, which is where these conversations already happen.
• An audit trail of what was sent to whom and when, so the follow-up conversation can start from a fact rather than a guess.

Outcome

Reminders go out on schedule instead of when someone remembers, and collections has a record of every notice sent.

The problem

Field sales activity lived inside each distributor’s own instance. Seeing what was happening across the network meant asking for exports, and by the time they arrived the day was over. There was no single view of visits actually being completed.

Approach

Stream visit activity out of every instance into one dashboard.

• Middleware between the instances and the dashboard, so the reporting side never queries a distributor’s production database directly.
• Per-instance isolation preserved through the pipeline, so a distributor’s data stays attributable to them.
• A flat replica of visit lines plus daily aggregates, because a dashboard reading raw transactional tables gets slower as the data grows.
• Checkout events captured as they happen rather than reconstructed later.

Outcome

One live view across the network, refreshed continuously, without any instance being exposed to the reporting layer.

The problem

Every external API call authenticated first, and every authentication hit the same small table through the ORM. The table was tiny and the answer barely changed, but the lookup ran on every single request, so the database spent a meaningful share of its time answering the same question.

Approach

A caching layer at the ORM level rather than in each caller.

• Intercepts search and read on the auth model, so nothing calling it needed to change.
• Redis holds the result, keyed so a credential change invalidates cleanly.
• Failure falls through to the database rather than failing the request, since an unavailable cache should slow things down, not break them.

Outcome

Repeated auth lookups stopped reaching PostgreSQL. The work happens once and is answered from memory afterwards.

The problem

At peak sales force traffic the database server ran hot enough to threaten everything else on it. The instinct in that situation is to add hardware, which buys time without answering why the load exists.

Approach

Find the queries responsible, then remove the reason they were expensive.

• Read the slow queries first, rather than guessing which ones mattered.
• Redis in front of Odoo’s read paths, so the repeated reads that dominated the load stopped reaching the database at all.
• Indexes rebuilt behind the queries that remained, designed around how they actually filter rather than around the columns that looked obvious.

Outcome

Database CPU and query load dropped by roughly 80% at peak, on the same hardware.

The problem

Field staff attach photos to every visit, and those images are read back constantly by dashboards and checks. Each read went to the filestore, so a page showing many visits meant many file reads for images that had not changed since they were uploaded.

Approach

Cache what does not change.

• Visit photos served from Redis after their first read.
• Inventory check and visit route data cached on the same layer, since they follow the same pattern of being written once and read repeatedly.
• Invalidation tied to the record, so an updated attachment is not served stale.

Outcome

Repeat reads stopped touching the filestore, which took load off the parts of the day when field staff are most active.

The problem

Customer records are written by more than one system, and the path that carried them outward had grown organically. The same customer was staged again and again, records could be sent with references that pointed nowhere useful, and a failed send was silent, so a record could simply never arrive with nobody the wiser.

Approach

Rebuild the models on both sides of that path.

• The partner model was rewritten, with the previous version kept alongside during the change rather than replaced blind.
• Salesperson lookup was fixed to resolve employees whose records had been deactivated.
• The staging pipeline was split into deciding whether a change is worth staging, preparing the payload, and inserting it, with a separate path for updates.
• Duplicate staging was cut off at the source: a write that had already been staged no longer runs the whole path again.
• Failures now alert over Discord and email instead of passing silently.

Outcome

Monthly volume fell by around 86%, from 335,000 staging records in December to 46,000 by February. Each record is one outbound request, so most of that traffic was the same customer being staged repeatedly: a write that had already been staged triggering the whole path again.

The first fix went in on 8 January and stopped the duplicate creation. The rest followed through February, and volume has stayed flat since.

Records that used to fail quietly now either arrive or announce that they did not. The same records are checked independently from outside by the integrity watchdog, listed separately here.

The cards above are the work worth telling a story about. This is the rest of it. Each line is a workstream in my notes, with its own scope, decisions, and commands behind it. No detail here, just the shape of the ground covered.

Sales and distribution

• Receivables and payables aging report grouped per branch
• Customer credit limits generated daily from a configurable formula
• Credit limit approvals moved onto the shared approval engine
• Delivery-driven invoice automation, one delivery order per invoice
• Business unit dimensions carried from product master into orders and invoices
• Order validation rules applied at the point an order is processed
• Sale order form rebuilt around delivery and invoice addresses
• Cancellation reasons captured per order and per line, with stale order reminders
• Self-pickup orders with their own transporter master data
• Delivery cost calculated from partner address and product unit mapping
• Load matrix capacity validation on sales orders
• Load matrix validation for purchase orders
• Branch requisitions in the central ERP raising purchase orders in the branch system
• Available quantity exposed next to on-hand on product forms
• Multi-line scrap journal for stock written off
• Cheque monitoring from collection to clearing, blocking premature paid status
• Petty cash routed through expense sheets against the branch journal
• Pricelists and their items synced from the central ERP to store systems
• Invoice journal, delivery order and monitoring documents rebuilt to business templates
• First invoice print clean, every reprint marked as a copy
• Claim document receipt export driven from a read-only view
• Invoice line export to CSV by month range or full history

Sales force

• Outlet visit surveys with a master question bank and per-visit answers
• Points awarded per visit activity as a gamification layer
• Flat replica and daily aggregates behind the visit dashboard
• Realtime visit dashboard brought into the branch system
• Visit dashboard logic migrated to the store system
• Dashboard writes moved to async processing through middleware

Claims, agreements, documents

• Multi-claim memos claimable by different distributors and regions
• Claim notification and redirect widget behind single sign-on
• Document send flow reworked for agreement settlement
• Distributor agreements generated as PDF, replacing Word documents
• Distributor onboarding form with multi-level approval
• Memos generated per recipient with approval signatures, replacing mail merge
• Customer money flow: deposits, bank mutation grouping, bulk receipts

Manufacturing

• Multi-level bills of material, work order types and quality checkpoints
• Bill of material structure and cost rollup report
• Cost structure report backported from a later Odoo edition
• Barcode scanning for material picking and finished goods registration
• Expected against actual waste on bills of material, with quality classification
• Work orders with visual instructions and checklists

Data integrity, integrations, alerts

• Staging sync hardened with validation and alerting
• Every data export raising an alert with who exported what
• Generic on/off switch table and email parameter modules, reused across projects

Infrastructure

• Central ERP containerised
• Store system containerised
• VPN client containerised with GUI forwarding
• Prometheus and Grafana monitoring across the distribution instances