The first time an auditor asks for your audit trail, you will probably export a CSV from whatever logging system you already run, and it will look fine. Timestamps, usernames, actions, resource IDs, all in order. Then someone asks the only question that matters: how do we know this file matches what actually happened? An immutable audit log that nobody outside your team can independently check is not evidence. It is a claim about your own behaviour, produced by systems you control, in a format you chose. Auditors are polite about this. Regulators are getting less so.
"Immutable" in most infrastructure stacks means something weaker than the word suggests. It usually means append-only by convention - the application only ever inserts, and everyone agrees not to run an UPDATE. Sometimes it means a retention policy on an object store, or a WORM bucket, which protects against deletion but says nothing about what was written in the first place. All of these are real controls. None of them lets a third party detect a change after the fact. That gap - between nobody is supposed to edit this and you can prove nobody did - is the whole subject.
How a tamper-evident audit trail works
Take every field of a log entry that matters - the action, who did it, which organisation they were acting for, the resource, the changed values, the correlation ID, the risk level, the client address - serialise them in a fixed, agreed order, append the previous entry's hash, and run SHA-256 over the result. Store that digest on the row as entry_hash, and store the predecessor's digest as prev_hash. The first row has no predecessor, so its prev_hash is null. Everything after it is welded to what came before it.
The consequence is the useful part. Change a single character in a three-month-old entry and its recomputed hash no longer matches the one stored on the row. Delete a row entirely and the next row's prev_hash points at a predecessor that is not there. Insert a fabricated entry and it has no valid place in the sequence at all. You do not need to trust the storage layer, the operator, or the vendor. You need the entries, the hashing rules, and a script. That is what makes it a tamper-evident audit trail rather than a tidy log.
Be precise about what this does not give you. A hash chain is tamper-evident, not tamper-proof. An attacker who holds write access to the database and knows the hashing rules can rewrite an entry and then recompute every hash after it, producing a chain that verifies perfectly. What defeats that is publishing the chain head somewhere the attacker does not control: signing the current entry_hash with a key held outside the database, exporting it, sending it to an auditor, writing it into a separate system. Any of those freezes history up to that moment.
Why a single writer matters more than you think
A hash chain has exactly one head, and every new entry has to read it before it can link to it. Run two writers against the same table and they will race - both read the same head, both link to it, and one of them is now wrong. The chain reports itself broken, and you spend a day looking for an attacker who does not exist. We treat this as an invariant rather than a preference: one process consumes the audit queue and writes the table, and turning on a second writer is a deliberate cutover, not a scaling knob.
Verification failures also need to be legible, because operators react to the first word they read. Our verifier reports three distinct reasons. chain_broken means the linkage is wrong, that prev_hash does not match the predecessor. content_tampered means the linkage is intact but recomputing the hash from the stored fields produces something different. entry_never_hashed means the row was written without a hash at all. Only the second is evidence of tampering, and collapsing the three into one message is how you start a security incident over a bug in a test.
That third case is not hypothetical, and it is worth telling on ourselves. A handful of rows in our own chain were written by a test that inserted directly into the table with raw SQL, bypassing the writer, leaving both hash columns empty. The database triggers that make the table append-only refuse deletes as well as updates, which means those rows are permanently unrepairable. They will sit in the chain forever, and every verification run reaches them. The fix was a helper that tests are required to use, plus a guard test that fails the moment anyone reaches for raw SQL again.
What an auditor should actually receive
The deliverable is not a screenshot of a dashboard saying "chain valid". It is a CSV containing the entries themselves with both hash columns included - id, action, actor, organisation, customer scope, resource type and ID, risk level, correlation ID, address, user agent, prev_hash, entry_hash and timestamp - plus a written specification of exactly which fields go into the digest and in what order. With those two things an auditor writes twenty lines of Python and checks your work without asking you for anything else. That is evidence: something you can hand over and then lose control of.
A small detail with real consequences: exported audit fields are attacker-influenced strings, and spreadsheets will execute anything beginning with an equals, plus, minus or at sign. An audit export that opens a shell on the auditor's laptop is a memorable way to fail an audit. Every field in our export is escaped and formula-prefixed values are neutralised before they reach the file. It is the kind of control that looks like fussiness right up until the first person opens the evidence pack in Excel.
Underneath all of this, the table itself has to refuse the dangerous operations. Append-only enforced in application code is a promise that survives exactly as long as nobody writes a migration, a cleanup script, or a well-meaning fix at two in the morning. Database triggers that reject UPDATE and DELETE on the audit table turn that promise into a constraint. It also means the application and the storage can only disagree in one direction: the database may refuse a write the application wanted, but the application can never quietly rewrite the database.
Verify your immutable audit log on a schedule
Most teams that have a verify endpoint call it once, during the demo. That is the wrong cadence. A chain break discovered when the auditor asks is a forensic problem covering however many months have passed; the same break discovered by a scheduled run is a bug report with a timestamp. Verification over a large table has to be built for that. Ours walks the entries in ordered pages rather than loading everything into memory, because the table grows with every mutation on the platform, and a verifier that only works on small tables is a verifier that will stop working.
The other habit is coverage. It is easy to put audit logging in the HTTP middleware, watch every API mutation appear in the trail, and declare it done. Then a cron job archives a customer's data, a queue consumer revokes an agent, a background worker rotates a credential, and none of it is in the trail, because none of it was an HTTP request. Those paths have to log explicitly. The uncomfortable truth is that the actions least likely to be logged are exactly the ones an investigator most wants to see.
NIS2 and DORA push in the same direction: controls you can demonstrate rather than assert. Nobody is going to hand you a certificate for using SHA-256, and we are not claiming one. But when someone asks who changed the firewall rule on the third of March, there is a large difference between a query result and a query result whose integrity a stranger can check. Three questions worth asking anyone selling you an immutable audit log: can I export the hashes, can I recompute them without your software, and what happens when the chain reports itself broken?