Learning Scala: Effect Correctness Is a Modeling Problem
I have asked all of my neighbors to call the fire department if they see smoke billowing out of my house. If they send too many trucks because too many called, my neighbors will have something to laugh about while they watch my house burn down. We probably do not want them sending all the trucks to one side of town in case a fire breaks out elsewhere, or shelling out more diesel than they need to rolling unneeded trucks, but as a homeowner, I do not worry about idempotency. That is the fire department's problem.
Catch up on earlier posts to follow along with the Functional Programming Isn’t Just for Academics series:
- Post 1: Why Functional Programming Matters for the Systems We Build Today Post 2: Immutability by Default and the Foundation of Reliable Systems Post 3: Pure Functions: Your First Step Toward Bug-Free Concurrency
Each post in this series explores how teams use Scala to build applications that stay clean, testable, and easy to scale.
Why Retry Mechanisms Guarantee Duplicate Work
Redundancy, fail-over, redelivery, supervision, automatic retry: each of them exists so that when something goes wrong the work still happens, and the way each of them accomplishes that is by attempting it another time. A system that reroutes around a dead node, redelivers an unacknowledged message, and replays an interrupted job is a system that does important things more than once. Not as a malfunction. On purpose, as a sign of health. Every mechanism you bought to keep work from being lost is a mechanism for doing it again. Which means the better your infrastructure gets, the more correctness you owe your model.
When a customer is charged twice on your system and your infrastructure, you go looking for the broken thing, and you may not find one. The queue delivered the message. The worker that crashed mid-charge was restarted by its supervisor in under a second, exactly as designed, and the restarted worker did what it was told: it charged the card. Every recovery mechanism worked. The transaction is what failed. Every component behaved the way you paid for it to behave. The second charge is not the price of a fragile system. It is the price of a robust one.
We spent a great deal of money making the work happen no matter what. We did not, by spending that money, make the work safe to happen more than once. Those are two different purchases, and most systems only make the first one.
What Your Infrastructure Can and Cannot See
The infrastructure answers one question extraordinarily well: is the system up? Is it responding, is it draining the queue, is it within its latency budget, did it survive the node we just killed. Availability. Liveness. The whole apparatus of health checks and supervisors and elastic capacity exists to keep that answer "yes," and after thirty years of practice it keeps it "yes" remarkably cheaply.
There is a second question it cannot answer at all: did the right thing happen exactly the right number of times? Call it effect correctness. And it cannot answer it because of where it stands. The supervisor watching the worker that charged the card and died a moment later sees a dead process. It does not see an escaped side effect. It cannot see that money already moved, because money moving is not visible from outside the operation. It is a fact that lives inside the operation's own understanding of what it did. So the supervisor does the one correct thing available to a thing standing outside: it starts the work again. It cannot un-charge a card. It was never holding the information that would tell it not to.
This is not a failure of the infrastructure. It is a boundary. Infrastructure can see that a process lived or died. It cannot see what that process meant. Meaning is the model's job, and the model is the only thing standing in the right place to do it.
The Same Repair Shows Up in Commits, Config, and Commerce
Once you see the boundary, you start seeing the same repair everywhere, and it is always the same repair. Some fact that everyone treated as the infrastructure's responsibility gets pulled inside the operation itself, where it becomes something the program can see and reason about.
A commit that runs twice stops being a disaster the moment the operation is keyed to a specific intent and is built to treat "I have already done this" as an ordinary answer it can return, rather than a surprise it stumbles into. The duplicate is still delivered. It is simply no longer harmful, because the operation recognizes it.
Configured behavior stops being a gamble the moment the configuration is a typed value checked at compile time, rather than a raw string injected into a running process and discovered to be wrong only when it misbehaves in production. Validity moves from something operations hopes is true to something that was established before the value could become behavior.
Commerce stops losing its own history the moment the decisions someone actually made (authorize, capture, ship, inspect, decide, refund, restock, destroy) become the durable thing, and an order is understood as what those decisions add up to rather than a row updated in place. Durability stops being the database's problem and becomes a property of how the domain was described.
Three instances, one move. Each takes a property we are used to handing to someone else's infrastructure, reliability and validity and durability, and makes it a question the domain answers about itself. The pattern is not a set of techniques. It is one conviction applied over and over: the thing you externalized belongs inside. What follows is what happens when you apply it to the one property this article is about.
Why Supervision Trees and Durable Execution Engines Do Not Solve This
The obvious objection is the correct one and deserves a straight answer. There is a mature, battle-tested style of building systems precisely for this kind of failure: make everything message-driven, keep state isolated, supervise everything, let the broken parts crash and be reborn clean, push back when overloaded, scale out when not. That is the actor model with its supervision trees, an idea that runs from Erlang through Akka and Pekko. It works. It has kept enormous systems alive through failures that would have flattened a more naive design. And, importantly, it is already aligned with the discipline I am describing. Immutable messages. Isolated state. No shared mutable memory between the parts. The reason those systems are so resilient is, in no small part, that they refuse to share mutable state, which is the same refusal at the heart of functional programming.
Durable execution engines record each step of a workflow and replay it deterministically after a failure, which removes a large class of this problem and is a good thing to reach for. But the guarantee stops where determinism does. Replaying a workflow is safe because the engine knows what it already did. The charge is the part it cannot replay, because the card is outside the engine, and something still has to recognize that this intent has been seen before. The engine can make the workflow's own state safe to replay. It cannot make the activity at the edge safe to attempt twice, which is why those engines ask you to write idempotent activities rather than promising to make yours idempotent for you.
So none of what follows is an argument against any of that. I am saying those tools solve a real problem completely, and that there is a second problem sitting right next to it that they, by their nature, cannot reach. Look at what those tools actually guarantee. At-least-once delivery. Redelivery on missing acknowledgment. Replay after a restart. Those are the features, not the failure modes: they are how the system stays alive, and every one of them is a promise to attempt the work again. So the retries are coming, and they are coming in proportion to how well you have built. They are not a malfunction to be stamped out; they are the designed behavior of a system that refuses to lose your work.
Why an Idempotency Key Does Not Know What It Is Protecting
Plenty of things in the stack can make an arriving duplicate harmless. A payment provider will honor an idempotency key. A unique constraint will refuse the second row. A workflow engine will remember a step it has already completed. Every one of those works, and not one of them decides what it is working on. They compare identifiers. They cannot invent the business identity an identifier stands for. Whether a resubmitted cart is the same purchase or a second one, whether a corrected address makes a new authorization, whether an ambiguous payment response entitles you to charge again: none of those are questions about keys. The broker can guarantee the message arrives. It cannot guarantee that arriving twice is safe. It has no idea what the message means. The supervisor can keep the system alive. It cannot keep the transaction honest.
Consider the shape most charge operations actually have. They take a card and an amount, they do the thing, and they hand back nothing. That operation has nowhere to stand. There is no seam in it that could carry "I have seen this exact charge before," because it was never given anything to recognize and nothing to say if it did. Run it twice and it charges twice, and no amount of supervision around it changes that, because the duplication is invisible from outside and the operation itself was given no way to notice.
Giving the Operation an Intent It Can Recognize
We built this exact shape once already in the intent ID and typed outcome pattern from Part 17, in the context of a generic distributed operation. Here it is worth walking through again in the specific case this whole piece has been circling: a charge that must not happen twice.
Now give that boundary two things it did not have. First, an intent with a name of its own: not a card and an amount, but an identifier for the commercial act those belong to. The caller chooses that name before trying and reuses it on every retry of the same act, which means somebody had to decide what makes two attempts the same act. That decision is not a technical one. Whether a resubmitted cart is the same purchase, whether adding a line item makes it a different one, whether a corrected shipping address starts a new authorization: those are answers the business gives, and the boundary is where the answer gets written down.
Giving the Operation an Outcome That Can Tell the Difference
Second, an outcome that can tell the difference between having done the work and having already done it. Not success or failure, which collapses the exact case this is all about, but charged and already charged, both of them ordinary answers, both carrying the receipt the customer is owed.
The retry calls chargeCard with the same ChargeIntent it used the first time. The ledger lookup is what turns a second attempt from a second charge into a recognized repeat. Nothing here is exotic. It is a lookup, a branch, and two cases in an enum. The discipline is entirely in deciding, up front, what the intent identifier means and insisting the type signature carry both outcomes instead of hiding the retry path inside a try block.
What that shape does not do is make any of it true. It makes the requirement expressible. Something durable still has to remember the name and what it produced. Concurrent attempts still have to be serialized. A repeat of the same name carrying a different amount has to be refused rather than quietly accepted. The first result has to be recoverable after the process holding it dies. That work can live in your own store, or you can hand it to a payment provider that honors the key, or to a workflow engine that remembers the step. All of those are fine, and the choice between them is an engineering one. None of them tells you what the name means.
With that in place the retry has somewhere to land. The second attempt is recognized, the work is not repeated, and the caller is handed the result the first attempt produced. The infrastructure around both versions is identical. What changed is that the operation was finally told what it was for.
Why Modeling the Intent Costs Less Than Patching Around It
A skeptic might claim we are just kicking the can down the road, or if being polite, ask, "why should I stretch my brain to model all this, when there are a dozen mature ways to handle it at the infrastructure level?"
Because the brain-stretch does not disappear in the infrastructure-only path. It relocates. You will still pay for correctness: in dedupe tables bolted in front of endpoints, in dead-letter queues, in idempotency middleware holding a key whose meaning lives somewhere else, in nightly reconciliation jobs that hunt down the double-charges after the fact and refund them. That is all cognitive load too. The difference is when it is paid and by whom. The modeling cost is paid once, at design time, by the person who actually knows what "the same intent" means: the author, who has the most context anyone will ever have about this operation. The infrastructure-patch cost is paid again and again, at runtime, at two in the morning, by whoever is on call, the person with the least context, reconstructing from the outside what the operation should have known from the inside.
A Batch Job That Charged Fifty Customers Twice
I have watched this go wrong in the most ordinary way there is. A batch of orders went in for processing. The batch was larger than the address validation service would take, and that service was throttled, so partway through the run it stopped answering. Every order after that point failed. The first fifty had already gone through. The process then did exactly what it was designed to do: the failures were kicked up to a human channel. What the process was not designed to do was fix them in bulk. Correcting a bad order was a one at a time job through customer service, which is a perfectly reasonable design right up until the day the problem arrives all at once. So someone dealt with the validation limit, took the batch, and reprocessed it, which is what anybody would have done. Fifty customers were charged twice and received the goods twice.
Look at what that operator was actually given. A batch that failed, with no message saying that the first fifty of it had not, so the operator assumed the batch failed atomically. From outside the run, a job that died partway through and a job that never started are the same shape. Assuming the whole thing failed was the only assumption available. And the check that would have caught it, going back through the history to find which orders had already completed, is precisely the work the operation should be doing for itself, on every call, without anyone having to remember. A person at an API at the wrong end of a bad week is the least reliable place in the whole system to put that check. It is also where we put it, every single time an operation cannot answer the question on its own. The payment token was no help, because each batch item looked like a new order. We had failed to model the batch correctly, functionally or otherwise.
When You Actually Owe an Operation Its Own Identity
Which raises the fair question of where this stops. Not everywhere. Most code has no business carrying an identity, and a function that formats an address certainly does not need one. Not simply wherever there are retries, either. Those fifty orders were not duplicated by a retry in any technical sense; they were duplicated by a person with a script on a bad afternoon. A partner replaying a webhook is not a retry. A customer clicking twice is not a retry. Scope this to retries and you will miss every one of the cases that actually bite. How would that batch have gone differently if we had first registered the intent to create each order, then processed against that registration? Same intent, same order. No registered intent, a new order. Even a known endpoint that only ever answers with a plain pass or fail beats an endpoint with no way to check at all.
Three conditions, and you owe an identity only where all three hold. The effect is irreversible or expensive to reverse: money moved, goods shipped, a message sent, an obligation created. The operation can be invoked more than once for the same real-world intent, by anything at all, machinery or a person or a partner or an agent. And nothing outside the operation can tell whether it already happened. Where all three are true you are on the hook, and no vendor can take you off it, because the mechanism is theirs and the consequence is yours. Where any one of them is false, do not bother. Reads are free, reversible effects are cheap, and if something outside can genuinely see whether the thing happened, let it.
Notice what that does and does not ask of you. Choosing a queue does not make you responsible for its internals, any more than calling the fire department makes you responsible for their dispatch. It makes you responsible for the boundary where your intent crosses into it. They own the mechanism. You own the name.
Effect Correctness Is the One Thing You Cannot Outsource
Effect correctness under retry and replay is the one thing that cannot be outsourced. Whether the card was charged once or twice. Whether the order exists once or three times. Whether replaying the log rebuilds the same truth or a corrupted one. That specific question lives inside the operation, and it can only be answered by the operation, and the operation can only answer it if it was modeled to. You can buy the mechanism that remembers a key. You still have to decide what the key means and how that relates to the intent, be it the operator's, the seller's, the customer's, or an authorized agent's.
Infrastructure can keep the system alive. Only the model can keep the transaction honest. Those are different jobs, and the long habit of our industry has been to fund the first one lavishly and assume the second one comes free with it. It does not. It never did. The reconciliation jobs and the 2am pages are the invoice for assuming it did.
This matters more now than it used to, because software is being asked to act with more discretion, at higher speed, in higher volume, than any human-paced oversight was built to check. When a system merely described options, a duplicate was an annoyance. When a system acts (moves money, commits inventory, creates obligations) and does so thousands of times a second on behalf of callers who will cheerfully retry the instant a response goes missing, the duplicate is the whole game. The part you cannot buy from your infrastructure has become the part that matters most. It is worth building. It was always worth building. Now it is not optional.
This is Part 22 in an ongoing series. If you found this useful, Part 21 covers why commerce is a sequence of verbs and not a set of nouns, using a product return to make the case for modeling decisions instead of reversing a row. Read "Why Commerce Is Verbs, Not Nouns"
Frequently Asked Questions
What is effect correctness in distributed systems?
Effect correctness is the guarantee that an operation with a real-world side effect, like charging a card or creating an order, happens exactly the number of times it was actually intended to happen, even when the underlying infrastructure retries, redelivers, or replays the request. It is separate from availability, which only measures whether a system is up and responding.
Why do retries cause duplicate charges even when nothing is broken?
Redundancy, supervision, and automatic retry all work by attempting an action again after a failure. If a worker charges a card and crashes before confirming, a healthy supervisor restarts it, and the restarted worker charges the card again because it has no way to know the first attempt succeeded. Every component behaved correctly. The duplicate charge is the cost of that correctness, not a sign something failed.
What is the difference between idempotency and effect correctness?
Idempotency is a property of an operation: running it more than once produces the same result as running it once. Effect correctness is the broader outcome that idempotency is meant to protect, that a real-world effect like a charge or a shipment happens the right number of times. Idempotency keys and unique constraints can help enforce it, but only if the operation itself knows what counts as a repeat.
How do you model an idempotent charge operation in Scala?
Give the operation two things it would not have otherwise. First, an intent identifier chosen by the caller and reused on every retry of the same commercial act. Second, a typed outcome with at least two cases, one for a new attempt and one for a recognized repeat, so the retry path is explicit in the type signature rather than hidden inside a boolean or an exception.
When does an operation actually need its own identity?
An operation needs an identity when three conditions are all true: the effect is irreversible or expensive to reverse, the operation can be invoked more than once for the same real-world intent by anything at all, and nothing outside the operation can tell whether it already happened. If any of those three is false, modeling an identity is not worth the effort.
Can infrastructure alone guarantee exactly once processing?
No. Infrastructure like message brokers, durable execution engines, and supervisors can guarantee that work is attempted and that state is replayed consistently, but none of them can see what a business intent means. A payment gateway can honor an idempotency key, but only the domain model can decide whether two requests represent the same purchase or two different ones.