Unix file publication: replace names without exposing partial bytes

First principles

A pathname and an open file descriptor are different kinds of reference. The pathname is a directory entry used to locate an object. A descriptor refers to the object already opened. Replacing a directory entry does not force existing descriptors to switch objects. This distinction explains why a reader can continue reading an old configuration while a later reader opens its replacement. It also explains why updating a name can be atomic without synchronizing every reader's view.

Writing directly into the published file exposes intermediate state. A reader can open it after truncation but before the new bytes finish arriving. JSON parsing fails, an empty configuration disables a feature, or a backup captures half a report. Instead, construct the complete replacement under a temporary name and rename it into place. On supported local filesystems, replacement within the same filesystem changes the name atomically: a lookup sees the old object or the new object, not the partially written temporary content.

Atomic visibility is only one guarantee. After a power loss, storage may not preserve the latest data or directory update unless the required synchronization has completed and the filesystem honors it. File synchronization requests durability of file contents; directory synchronization requests durability of the naming change. Different operating systems and filesystems vary in support. A robust design states which guarantee it needs instead of treating the word atomic as a synonym for permanent.

Runnable example

Run this Python 3 example on a local Unix filesystem supporting directory fsync. It publishes a synthetic configuration in a disposable directory. Using mkstemp in the target directory avoids a predictable temporary filename and keeps replacement on the same filesystem.

import json
import os
import tempfile
from pathlib import Path

def publish(target, value):
    descriptor, temporary = tempfile.mkstemp(
        prefix=".pending-", dir=target.parent
    )
    try:
        with os.fdopen(descriptor, "w", encoding="utf-8") as stream:
            json.dump(value, stream)
            stream.flush()
            os.fsync(stream.fileno())
        os.replace(temporary, target)
        directory = os.open(target.parent, os.O_RDONLY)
        try:
            os.fsync(directory)
        finally:
            os.close(directory)
    finally:
        if os.path.exists(temporary):
            os.unlink(temporary)

with tempfile.TemporaryDirectory() as folder:
    target = Path(folder) / "settings.json"
    publish(target, {"revision": 1})
    with target.open(encoding="utf-8") as old_reader:
        publish(target, {"revision": 2})
        assert json.load(old_reader)["revision"] == 1
    assert json.loads(target.read_text(encoding="utf-8"))["revision"] == 2
    print("old descriptor: 1; new pathname: 2")

The old reader deliberately stays open across publication. Its result is not stale cache behavior; it is the expected identity of the object referenced by that descriptor. Applications that require the new configuration must reopen the path at an appropriate boundary. A watcher notification can prompt that reload, but the loader should still parse and validate the complete new configuration before replacing its in-memory state.

Contextual failure case

A maintenance script rewrites a service configuration using output redirection into the live file. The shell truncates the file before the generator finishes. A concurrent reload sees an empty document, and the service falls back to unsafe defaults. Retrying the reload after a delay masks the race but does not remove it. Build the new configuration separately, validate it, and publish only after successful generation. Failed generation should leave the existing published file unchanged.

Another script writes its temporary file in a general temporary directory and moves it to a mounted data volume. Crossing filesystems may fail or require a tool to copy and delete rather than perform an atomic rename. The publication algorithm must control where the temporary object is created. Even correct rename does not solve concurrent writers: the last successful replacement wins, which can lose an earlier edit unless writers coordinate or check expected revisions.

Verification

The demonstration prints old descriptor: 1; new pathname: 2. Change the publisher to raise before replacement and confirm the old pathname still contains revision one. Test malformed input before publication rather than relying on readers to discover it afterward. Inspect a synthetic configuration with payload tools, but avoid pasting real credentials. For large reports, compare byte counts and checksums between the validated temporary artifact and the published object.

Crash durability cannot be proven by a normal unit test that exits cleanly. It requires platform-specific fault testing and knowledge of the filesystem and storage stack. If directory fsync fails after replacement, the new name may already be visible even though durability is uncertain. Report that ambiguous outcome clearly; do not assume a failed function means the previous file remains intact.

Exercise

Add a schema check requiring an integer revision and a nonempty service name. Attempt to publish an invalid document and prove the old configuration remains readable. Then design a strategy for two writers both editing revision two. Explain whether you need a single owner, an advisory lock respected by all writers, or a database-backed revision protocol. Distinguish protection from partial bytes from protection against lost updates.

Safety and caveats

Use a directory controlled by the service account, not an attacker-writable path. Secure creation of the temporary file does not eliminate races involving malicious replacement of parent directories. Define target permissions and ownership explicitly; replacement publishes the temporary object's metadata rather than automatically preserving the old file's mode. Network filesystems can have different consistency behavior. Synchronization costs can be substantial, so batch carefully without claiming durability before the batch has actually completed.