RedisLockService

Redis is an excellent in-memory key-value database which can be used to implement a distributed lock. Distributed locks are useful when you want to run some protected code which must only be run once in your entire cluster.

The included RedisLockService provides distributed locks, which allow threads to coordinate across JVM instances via Redis.

Its features:

• Several locks can be acquired atomically: either all of them, or none.
• Read (shared) and write (exclusive) locks. See Read and write locks.
• Locks are reentrant per thread: a thread that already holds a lock can acquire it again.
• The lease of held locks is renewed automatically while the protected code runs.
• Deadlocks between threads (across JVMs) are detected.
• Redis' clock (TIME) is used as the single time source for all JVMs, so their clocks need not be in sync.
• Protection against Redis restarts / data loss. See Redis restarts.
• Works with plain Redis 6.2 or newer, no Lua scripts are used.

See here for a comparison to Redisson's locks.

Things to be aware of:

• The lock records do not survive a Redis restart. See Redis restarts.
• It is meant for a single Redis instance, and should probably not be used with Redis Cluster or replicas with failover (Sentinel). See Replicas, failover and Redis Cluster.
• The calling thread should preferably be a virtual thread, as waiting for a lock blocks it.

Set up a shared RedisLockService

The lock service stores its records as JSON and leaves the choice of the JSON library to you. Implement LockServiceJsonHandler, for example with Jackson 3 (kotlin module):

import tools.jackson.databind.ObjectMapper
import tools.jackson.module.kotlin.jacksonObjectMapper

class JacksonLockServiceJsonHandler(
    private val jsonMapper: ObjectMapper = jacksonObjectMapper(),
) : LockServiceJsonHandler {
    override fun <T> deserialize(json: String, clazz: Class<T>): T = jsonMapper.readValue(json, clazz)
    override fun <T> serialize(dto: T): String = jsonMapper.writeValueAsString(dto)
}

Then create the service. It also needs a started RedisPubSubService, which is used to wake up waiting threads as soon as a lock is released (also in other JVMs), and to share the waiting threads across JVMs for deadlock detection:

// share this as a bean/singleton
val redisLockService = RedisLockService(
    connectionPool = connectionPool,
    redisPubSubService = redisPubSubService,
    lockServiceJsonHandler = JacksonLockServiceJsonHandler(),
)

Optional parameters:

Parameter	Default	Description
expireLocksWhichShallNotBeKeptSeconds	30 (seconds)	How long the record of a released lock is kept, unless keepLockDataAfterUse=true is used.
pollDelayWhenWaitingForLockMs	5000 (milliseconds)	How often a waiting thread retries, as a fallback in case a release notification was missed. Also determines how fast a deadlock is detected.
connectionPoolTimeout	5 seconds	Max time to wait for a connection from the pool.
lockKeyPrefix	the class name	Prefix of all keys and pub/sub-channels used in Redis. Services sharing locks must use the same prefix.
lockLeaseTimeSeconds	60 (seconds)	For how long a lock is held without renewal. After this time the lock is considered to be available. Must be the same for all services sharing locks.
lockLeaseRenewalTimeSeconds	lockLeaseTimeSeconds / 2	How often the leases of held locks are renewed. Must be lower than lockLeaseTimeSeconds. Each acquisition is renewed by its own virtual thread; a failed renewal is retried every second (or every lockLeaseRenewalTimeSeconds, if lower) until the lease runs out.
onNewThreadHandler	{ it }	Each time a new virtual thread is started for renewing the leases of an acquisition, this handler is used to wrap the Runnable. It is called on the thread calling tryLock, so it can be used to transfer the MDC to the new thread, or to apply a custom exception handler to handle logging for example.

Each renewal briefly borrows a connection from the pool. Make sure the pool has connections to spare beyond those the protected code keeps busy, otherwise renewals have to wait and the leases might run out.

Acquire locks and run protected code

val (acquired, result) = redisLockService.tryWriteLock(listOf("customer-42", "invoice-7"), timeoutMs = 10_000) { hasLock ->
    // all locks are held here
    doWork()
    if (!hasLock()) return@tryWriteLock null // no longer safe to proceed
    "done"
}
if (!acquired) {
    // the locks could not be acquired within 10 seconds, the protected code did not run
}

tryWriteLock (and tryReadLock, tryLock, see Read and write locks) waits up to timeoutMs for all locks to become available (default 0: a single attempt). If it succeeds, it runs the protected code and releases the locks afterwards, also if the code throws. It returns true and the result of the protected code, or false and null if the locks could not be acquired.

The protected code should check hasLock() regularly, for example between steps of loop iterations, and abort if it returns false. That happens if the lock was lost: the lease could not be renewed in time (failed renewals are retried until the lease runs out), the record was taken over by someone else, or Redis restarted.

Nested calls on the same thread acquire only the locks the thread does not hold yet. A lock is released when the call that acquired it returns:

redisLockService.tryWriteLock("a") {
    redisLockService.tryWriteLock(listOf("a", "b")) {
        // "a" is reused, "b" is acquired
    }
    // "b" is released, "a" is still held
}

If the thread has lost one of the locks it already holds (hasLock() returned false), a nested call on that lock returns false and null without running the protected code. The lost lock is not acquired again.

Keep lock data after use

By default, the record of a released lock expires after expireLocksWhichShallNotBeKeptSeconds. With keepLockDataAfterUse = true, it is kept, so that its usage data can be looked up later:

redisLockService.tryWriteLock("nightly-report-job", keepLockDataAfterUse = true) {
    runNightlyReport()
}

Only use this for a stable set of lock names, such as job names. Dynamic names, like customer ids, would pile up in Redis. For a lock the thread already holds, the value from the call that acquired it wins.

Look up lock data

val infos: List<LockInfo> = redisLockService.getLockInfo(listOf("nightly-report-job"))
infos.forEach {
    println("${it.lockName}: writer ${it.writeOwner}, readers ${it.readOwners}, " +
            "last updated at ${it.updatedAtEpochMillis}")
}

Only the locks with a record in Redis are returned. A LockInfo holds the write owner (writeOwner) and the read owners (readOwners) as LockHoldings. Its owner is the current owner while the lock is held (expiresAtEpochMillis is not null), and the previous owner otherwise.

Read and write locks

Read locks of the same name are shared between threads, a write lock is exclusive. Use tryReadLock or tryWriteLock to request locks of one type, or tryLock with "a".readLock / "a".writeLock to mix them:

redisLockService.tryLock(listOf("config".readLock, "report".writeLock)) {
    // reading "config" while other threads may read it too, writing "report" exclusively
}

Requesting both the read and the write lock of the same name in one call requests the write lock only. A thread holding a read lock can request the write lock of the same name in a nested call (granted once no other thread holds the read lock); a thread holding the write lock reuses it for nested read lock requests.

Writers are preferred: while a thread waits for a write lock, no new read lock of the same name is granted (also not for timeoutMs = 0). The current readers finish, then the writer gets the lock, so overlapping readers cannot starve it. Read locks the thread already holds are reused by nested calls as usual. If the writer gives up, the readers which stepped down for it are woken right away.

Deadlock detection

A deadlock happens when threads wait for locks that the other ones hold, for example:

• thread 1 holds lock a and waits for lock b
• thread 2 holds lock b and waits for lock a

Or:

• thread 1 holds lock a and waits for lock b
• thread 2 holds lock b and waits for lock c
• thread 3 holds lock c and waits for lock a

Or, because writers are preferred (see Read and write locks):

• thread 1 holds the read lock b and waits for the read lock a
• thread 2 waits for the write locks a and b

(where threads can either be on the local JVM or across multiple JVMs)

Deadlock detection is always on. Waiting threads broadcast which locks they hold and wait for via the RedisPubSubService, so that every JVM knows all waiting threads. After each failed attempt, a waiting thread looks for a cycle of waiting threads leading back to itself, across all JVMs. Every thread in the cycle finds it, but only one of them gives up: its tryLock throws RedisLockServiceDeadlockException, which releases its outer locks as the exception propagates. The other threads then acquire their locks.

A deadlock is therefore detected within about pollDelayWhenWaitingForLockMs.

Redis restarts

When Redis restarts, it might lose the lock records. Two measures keep the locks safe:

• After a (re)start, no lock is handed out until Redis has been up for lockLeaseTimeSeconds. This waiting time counts toward timeoutMs.
• Within that time, the lease renewal of the previous owners notices the restart. They back down, also if their records survived the restart (persistence): the locks are marked as lost and not renewed, so hasLock() returns false and their protected code should wind down. If the renewal does not get through at all, their leases run out within that time as well.

Replicas, failover and Redis Cluster

Both measures rely on noticing the restart. After a failover (e.g. with Sentinel), the promoted replica has been up for long already, so there is no startup wait. Lock records that had not been replicated yet are lost and can be taken right away, while the previous owners only notice on their next lease renewal. Redis Cluster adds to that: the keys of several locks usually live in different slots, so acquiring them atomically is not possible there. The RedisLockService should therefore probably not be used with either.