A distributed lock is a lease on a shared resource, and every lease can expire while its holder is still running. The recurring mistake is treating the lock itself as the safety property. The safety property is that the resource rejects stale holders, which requires a monotonic fencing token checked by the resource, not by the lock service. Redis and etcd can both hand out ordered tokens, but neither makes the token part of the write path on its own. This article covers where each stops being safe and how to build the piece that makes either usable.

The lease that outlives its holder

The documented behaviour of every distributed lock is the same: acquire returns a token and a time-to-live; the holder is expected to renew before the TTL elapses; if renewal fails, the lock is released and another client can acquire it. The gap between that sentence and production is the pause. A process can be alive, holding a token it believes is valid, and not executing: stopped at a GC safepoint (the JVM's stop-the-world pauses are the canonical example), descheduled by the hypervisor, blocked on a full TCP send buffer, or frozen by a container runtime during a live migration. When it wakes, it resumes the critical section with a token the lock service has already expired and reissued.

This is not a Redis bug or an etcd bug. It is the definition of a lease. The counter-argument from the Redis side is that for efficiency, meaning avoiding duplicated work, a lock that occasionally allows two holders is acceptable. Both positions are correct about different problems. The failure mode is teams using an efficiency lock where they need a correctness lock.

The distinction changes what you build:

• An efficiency lock (one cron leader, one cache warmer) only needs to be mostly right. A duplicated run wastes CPU; nothing corrupts.
• A correctness lock (one writer to a file, one allocator of a sequence range, one processor of a payment) needs a guarantee that the resource will not accept a write from a holder whose lease has expired.

That guarantee cannot come from the lock service alone. It has to come from the resource.

Fencing tokens: the part that makes a lock a lock

A fencing token is a number that increases every time the lock is granted. The lock service hands it to the holder; the holder includes it in every write to the protected resource; the resource remembers the highest token it has seen and rejects anything lower. If client A holds token 33, pauses, loses the lease, and client B acquires token 34 and writes, then A wakes and tries to write with 33, the resource rejects it. The lock service's timing no longer has to be perfect. The resource enforces order.

This is the mechanism that turns a lease into a safety property, and it is the mechanism most Redis-lock tutorials omit. A typical implementation uses SET key value NX PX 30000 and a Lua compare-and-delete on release, which is correct as far as it goes: the value is a random client ID, so a client cannot delete another client's lock. But the lock value is not a fencing token. It does not increase monotonically, and the resource has no way to compare it against the last write it accepted.

etcd's concurrency package makes the token part of the API. A concurrency.Mutex is backed by a lease and a key; the key is created under a prefix with a revision, and etcd's global revision is monotonic across the cluster. The mutex's key encodes the revision at which it was created, and that revision can be used as a fencing token. The etcd API guarantees that revisions increase monotonically. The lock service is still a lease, and the lease can still expire, but the token it hands out is ordered, and a resource that checks it can reject stale writes.

Redis can produce an ordered token too, via INCR on a separate key, but nothing in Redis enforces that the resource checks it. That is application code. The same is true of etcd: the revision is only useful if the resource compares it. The difference is that etcd's client library hands you the revision as part of the mutex, while Redis's SET NX PX hands you an opaque value that must be turned into a token separately.

Implementing a lock with a fencing check

The following example uses etcd's Go client. It is a complete, runnable program that acquires a mutex, uses the revision as a fencing token, and writes to a simulated resource that rejects stale tokens. The resource is an in-memory integer; in a real system it would be a row in Postgres, an object in S3 with a version condition, or a file with an atomic rename.

package main

import (
    "context"
    "fmt"
    "log"
    "time"

    clientv3 "go.etcd.io/etcd/client/v3"
    "go.etcd.io/etcd/client/v3/concurrency"
)

// Resource enforces monotonic fencing tokens.
type Resource struct {
    lastToken int64
    value     string
}

func (r *Resource) Write(token int64, value string) error {
    if token <= r.lastToken {
        return fmt.Errorf("stale token %d, last accepted %d", token, r.lastToken)
    }
    r.lastToken = token
    r.value = value
    return nil
}

func main() {
    cli, err := clientv3.New(clientv3.Config{
        Endpoints:   []string{"127.0.0.1:2379"},
        DialTimeout: 5 * time.Second,
    })
    if err != nil {
        log.Fatal(err)
    }
    defer cli.Close()

    session, err := concurrency.NewSession(cli, concurrency.WithTTL(10))
    if err != nil {
        log.Fatal(err)
    }
    defer session.Close()

    mutex := concurrency.NewMutex(session, "/locks/my-resource")
    ctx := context.Background()
    if err := mutex.Lock(ctx); err != nil {
        log.Fatal(err)
    }
    defer mutex.Unlock(ctx)

    // The mutex key's revision is the fencing token.
    token := mutex.Header().Revision

    res := &Resource{}
    if err := res.Write(token, "first write"); err != nil {
        log.Fatal(err)
    }
    fmt.Printf("wrote with token %d\n", token)

    // Simulate a stale holder trying to write with an older token.
    if err := res.Write(token-1, "stale write"); err != nil {
        fmt.Printf("rejected as expected: %v\n", err)
    }
}

The key detail is mutex.Header().Revision. That revision is the etcd cluster's global revision at the moment the mutex key was created. It is monotonic across all keys, not just this lock, which means two different locks cannot produce the same token. The resource's Write method is the safety property: it compares the incoming token to the last one it accepted and rejects anything that is not strictly greater. If the holder pauses past its lease, another client acquires the mutex with a higher revision, and the paused holder's write is rejected.

The same pattern works with Redis, but the token has to be constructed. A minimal approach uses a separate INCR key as the token source and the lock key as the lease:

-- KEYS[1] = lock key, KEYS[2] = token counter
-- ARGV[1] = client id, ARGV[2] = ttl in ms
local acquired = redis.call('SET', KEYS[1], ARGV[1], 'NX', 'PX', ARGV[2])
if not acquired then
  return nil
end
local token = redis.call('INCR', KEYS[2])
return token

This returns a monotonically increasing token if the script is run against a single Redis primary. Under Redis Cluster, the two keys must be in the same hash slot (use a hash tag such as {lock}:counter and {lock}:key) or the script fails with a CROSSSLOT error. The token is also only monotonic as long as the primary does not fail over to a replica that has not yet replicated the INCR, a real risk with asynchronous replication, which is Redis's default. That is why the etcd revision is a better token: etcd commits through Raft, so a revision that has been returned to a client has been replicated to a quorum.

Measuring whether the lock is actually holding

The metric that matters is not lock acquisition latency. It is the rate of rejected writes at the resource. If the fencing check is working, every stale holder produces a rejection, and that rejection is a signal that a lease expired while a holder was still alive. A dashboard that shows zero rejections over a long period is ambiguous: either the lock is working perfectly, or the resource is not checking the token. A steady trickle of rejections is healthier than none, because it proves the check is live.

A second metric is the gap between lease renewal and lease expiry. etcd's concurrency.Session sends keepalives at a configurable interval; the documented default is every TTL/3. If the keepalive round-trip time approaches the TTL, the session will expire under load, and the lock will churn. Watching the p99 of keepalive latency against the TTL is more useful than watching lock wait time, because it predicts the failure before it happens. For Redis, the equivalent is the latency of the PEXPIRE or SET ... PX renewal call; if the primary is busy with a slow command (a large KEYS, a SAVE, a replica sync), the renewal can be delayed past the TTL.

To measure this rather than guess, instrument three things and compare them over the same window:

1. Rejection counter at the resource, incremented in the same transaction that rejects a stale token. Alert on a sudden spike, and audit if it stays at exactly zero for weeks.
2. Renewal latency histogram in the lock client, recorded around the keepalive or PEXPIRE call. Compare p99 to the TTL; a p99 above roughly one third of the TTL means the lease is at risk under load.
3. Lease-loss counter, incremented whenever a keepalive fails or a renewal returns false. A lease loss followed by a successful re-acquire with a new token is the exact sequence fencing is designed to catch.

A third check is a periodic audit: for each lock key, compare the holder's token to the resource's last accepted token. If the holder's token is less than or equal to the resource's last token, the lock is already broken, because the holder believes it holds the lock but the resource has moved on. This audit can be a background job that runs every few minutes; it does not need to be in the hot path.

The failure modes that fencing does not fix

Fencing fixes stale holders. It does not fix a resource that is not the single source of truth. If the protected resource is replicated with asynchronous replication (a Redis replica, a Postgres read replica, an object store bucket in a different region), the fencing check can be bypassed by writing to a replica that has not seen the latest token. The check has to happen at the primary, or the resource has to use a conditional write that the storage layer enforces (S3's If-Match with an ETag, Postgres's UPDATE ... WHERE version = $1). A fencing token checked by application code in front of an eventually consistent store is not a safety property; it is a suggestion.

Another failure mode is clock skew. Redis's PX expiry is based on the primary's monotonic clock, which is fine, but the client's decision to renew is based on the client's clock. If the client's clock is behind the primary's, it will renew too late; if it is ahead, it will renew too early and waste TTL. etcd's leases are server-side, so the client does not need a synchronized clock for the lease itself, but the client's keepalive loop still uses a timer. NTP drift on a VM is usually small, but a container that has been suspended and resumed can have a clock that jumps by seconds. The fix is to use monotonic time for renewal scheduling, not wall-clock time, and to treat any keepalive failure as an immediate lease loss rather than retrying with the same token.

A third failure mode is lock churn under partition. If the network between the client and the lock service is partitioned, the client cannot renew, the lease expires, and another client acquires. When the partition heals, the original client may still believe it holds the lock. Fencing catches this at the resource, but only if the resource is reachable from both clients. If the resource is also partitioned, the two clients may write to different partitions of the resource, and the fencing check never sees the conflict. This is the case where a distributed lock is the wrong tool: the resource itself needs to be a consensus system, not a lock in front of one.

A fourth failure mode is the token that resets. If the token counter is rebuilt (a Redis key evicted, a database sequence recreated, a migration that copies rows without the counter), tokens can go backwards and the resource will reject every legitimate write, or worse, accept a stale one if the counter restarts above the old maximum. Tokens should live in the same durable store as the data they protect, or be derived from a source that cannot regress, such as etcd's revision.

When a lock is the wrong answer

The most common mistake is using a distributed lock to protect a resource that already has a compare-and-swap primitive. Postgres has UPDATE ... WHERE version = $1; S3 has conditional writes with If-Match; etcd has transactions with Compare; Redis has WATCH/MULTI/EXEC (with caveats under cluster). If the resource can reject a stale write on its own, the lock is redundant. The lock adds a second system that can fail, a lease that can expire, and a token that has to be plumbed through. The compare-and-swap is one round trip to the resource and no separate failure domain.

The second case is a lock used for mutual exclusion of a long-running job. If the job takes longer than the lease TTL, the holder has to renew, and every renewal is a chance to lose the lease. A better pattern is a work queue with visibility timeouts (SQS, a lease-based queue, a database-backed queue with FOR UPDATE SKIP LOCKED). The queue hands out a message with a visibility timeout; if the worker does not delete the message before the timeout, the message becomes visible again. The worker does not need to hold a lock for the duration of the job; it needs to finish before the timeout, and if it does not, the job is retried. This is the same lease mechanism, but the retry semantics are built in, and the resource (the queue) enforces them.

The third case is a lock used to coordinate a read-modify-write across multiple services. If two services read a value, compute a new value, and write it back, a lock serializes them, but it does not make the computation correct if the value changes between the read and the write. The fix is optimistic concurrency at the resource: read the version, compute, write with a WHERE version = $1 condition, retry on conflict. The lock is a pessimistic version of the same thing, with more failure modes.

Where each service actually fits

The table makes the choice look like a latency-versus-correctness trade-off, and in practice that is what it is. Redis is the right choice when the lock is an efficiency lock and the resource tolerates rare double-execution: a cache warmer, a leader election for a stateless worker, a rate limiter that can occasionally over-admit. etcd is the right choice when the lock is a correctness lock and the resource can check a fencing token: a sequence allocator, a single-writer file system, a controller that must not have two active instances. In both cases, the resource's check is what makes it safe. The lock service only decides who gets to try.

The counterintuitive part is that the lock service's consistency model matters less than the resource's. A team can run Redis with a single primary and no replication and get a perfectly safe lock for a correctness use case, as long as the resource checks the token. A team can run a three-node etcd cluster and still have an unsafe lock if the resource writes to a replica that has not seen the latest revision. The lock is a coordination hint; the resource is the enforcement point.

A worked example: what goes wrong without the check

Consider a sequence allocator that hands out blocks of IDs. Service A acquires the lock with a 10-second TTL, reads the stored counter (say 1,000), and plans to write back 1,100 after a short computation. Before the write, A is paused for 12 seconds by a stop-the-world GC pause. The lease expires; service B acquires the lock, reads the counter (still 1,000), and writes back 1,100. A wakes and writes back 1,100 as well. Both services believe they own the range 1,000–1,099, and one range is silently lost or duplicated.

Now add fencing. The lock hands out token 33 to A and token 34 to B. A's write carries 33, B's carries 34. The resource stores the last accepted token alongside the counter. B's write with 34 succeeds and sets the last token to 34. A's write with 33 arrives later and is rejected because 33 is not greater than 34. A can then re-read the counter, recompute a fresh range, and retry under a new lock acquisition. The pause still happened, the lease still expired, and the lock still churned, but no range was lost. That is the entire value of the token: it converts a timing failure into a detectable, retryable error.

The same reasoning applies to a file writer. Without fencing, a paused writer can overwrite a newer version with an older one. With fencing, the write is a conditional put against a stored token, and the older writer's request fails with a conflict the application can handle.

The check to run before the next deploy

The next time a lock is added to a codebase, the first question is not which service to use. It is what the resource does when it receives a write with a token lower than the last one it accepted. If the answer is "nothing, it just writes," the lock is not a lock; it is a lease with a hope. The fix is a single conditional in the write path: compare the incoming token to the stored token, reject if it is not greater, and update the stored token in the same transaction. In Postgres that is one UPDATE ... WHERE last_token < $1 RETURNING ...; in S3 it is a conditional PutObject with If-Match on the token object; in a file system it is an atomic rename with a token-named file. Pick the resource that holds the data, add the check, and then the choice between Redis and etcd becomes a latency question rather than a safety question.

Action for the next 30 minutes: find every place in the codebase that calls a lock's acquire, and for each one write down the resource it protects and whether that resource compares a stored token to an incoming one. Any lock whose resource does not compare tokens is an efficiency lock whether or not the team knows it; either add the conditional write or document the accepted risk.

Related reading

• MCP hidden cost traps
• Thundering Herd: The Hidden Scaling Trap
• Monolith to services without a rewrite

Official documentation

• PostgreSQL documentation
• Redis documentation