RedisLockService

Redis is an excellent in-memory key-value database which can be used to implement a distributed lock. Distributed locks are useful when you want to run some protected code which must only be run once in your entire cluster.

The included RedisLockService provides distributed locks, which allow threads to coordinate across JVM instances via Redis.

It's features:

• Several locks can be acquired atomically: either all of them, or none.
• Locks are reentrant per thread: a thread that already holds a lock can acquire it again.
• The lease of held locks is renewed automatically while the protected code runs.
• Deadlocks between threads (across JVMs) can be detected.
• Redis' clock (TIME) is used as the single time source for all JVMs, so their clocks need not be in sync.
• Protection against Redis restarts / data loss. See Redis restarts.
• Works with plain Redis 6.2 or newer, no Lua scripts are used.

Things to be aware of:

• The lock records do not survive a Redis restart. See Redis restarts.
• It is meant for a single Redis instance, and should probably not be used with Redis Cluster or replicas with failover (Sentinel). See Replicas, failover and Redis Cluster.
• The calling thread should preferably be a virtual thread, as waiting for a lock blocks it.

Set up a shared RedisLockService

The lock service stores its records as JSON and leaves the choice of the JSON library to you. Implement LockServiceJsonHandler, for example with Jackson (kotlin module):

class JacksonLockServiceJsonHandler(
    private val jsonMapper: ObjectMapper = jacksonObjectMapper(),
) : RedisLockService.LockServiceJsonHandler {
    override fun <T> deserialize(json: String, clazz: Class<T>): T = jsonMapper.readValue(json, clazz)
    override fun <T> serialize(dto: T): String = jsonMapper.writeValueAsString(dto)
}

Then create the service:

// share this as a bean/singleton
val redisLockService = RedisLockService(
    connectionPool = connectionPool,
    lockServiceJsonHandler = JacksonLockServiceJsonHandler(),
).apply { start() }

// on shutdown
redisLockService.stop()

Optional parameters:

Parameter	Default	Description
redisPubSubService	null	A RedisPubSubService. Waiters are then woken up as soon as a lock is released in another JVM, instead of on their next poll.
deadlockDetection	true	See Deadlock detection.
expireLocksWhichShallNotBeKeptSeconds	30 (seconds)	How long the record of a released lock is kept, unless keepLockDataAfterUse=true is used.
pollDelayWhenWaitingForLockMs	5000 (milliseconds)	How often a waiting thread retries. Can/should be increased when redisPubSubService is used because then it only is required as a fallback.
lockKeyPrefix	the class name	Prefix of all keys and pub/sub-channels used in Redis. Services sharing locks must use the same prefix.
lockLeaseTimeSeconds	60 (seconds)	For how long a lock is held without renewal. After this time the lock is considered to be available.
lockLeaseRenewalTimeSeconds	lockLeaseTimeSeconds / 2	How often the leases of held locks are renewed. Must be lower than lockLeaseTimeSeconds. Each acquisition is renewed by its own virtual thread; a failed renewal is retried every second (or every lockLeaseRenewalTimeSeconds, if lower) until the lease runs out.
onNewThreadHandler	{ it }	Each time a new virtual thread is started for renewing the leases of an acquisition, this handler is used to wrap the Runnable. It is called on the thread calling tryLock, so it can be used to transfer the MDC to the new thread, or to apply a custom exception handler to handle logging for example.

Each renewal briefly borrows a connection from the pool. Make sure the pool has connections to spare beyond those the protected code keeps busy, otherwise renewals have to wait and the leases might run out.

stop() makes hasLock() return false for all held locks and makes waiting threads give up. It does not release the lock records: they are released when the protected code returns, or expire with their lease.

Acquire locks and run protected code

val (acquired, result) = redisLockService.tryLock(listOf("customer-42", "invoice-7"), timeoutMs = 10_000) { hasLock ->
    // all locks are held here
    doWork()
    if (!hasLock()) return@tryLock null // no longer safe to proceed
    "done"
}
if (!acquired) {
    // the locks could not be acquired within 10 seconds, the protected code did not run
}

tryLock waits up to timeoutMs for all locks to become available (default 0: a single attempt). If it succeeds, it runs the protected code and releases the locks afterwards, also if the code throws. It returns true and the result of the protected code, or false and null if the locks could not be acquired.

The protected code should check hasLock() regularly, for example between steps of loop iterations, and abort if it returns false. That happens if the lock was lost: the lease could not be renewed in time, the record was taken over by someone else, Redis restarted, or the service was stopped.

Nested calls on the same thread acquire only the locks the thread does not hold yet. A lock is released when the call that acquired it returns:

redisLockService.tryLock(listOf("a")) {
    redisLockService.tryLock(listOf("a", "b")) {
        // "a" is reused, "b" is acquired
    }
    // "b" is released, "a" is still held
}

If the thread has lost one of the locks it already holds (hasLock() returned false), a nested call on that lock returns false and null without running the protected code. The lost lock is not acquired again.

Keep lock data after use

By default, the record of a released lock expires after expireLocksWhichShallNotBeKeptSeconds. With keepLockDataAfterUse = true, it is kept, so that its usage data can be looked up later:

redisLockService.tryLock(listOf("nightly-report-job"), keepLockDataAfterUse = true) {
    runNightlyReport()
}

Only use this for a stable set of lock names, such as job names. Dynamic names, like customer ids, would pile up in Redis. For a lock the thread already holds, the value from the call that acquired it wins.

Look up lock data

val infos: List<RedisLockService.LockInfo> = redisLockService.getLockInfo(listOf("nightly-report-job"))
infos.forEach {
    println("${it.lockName}: last owner ${it.owner}, held: ${it.expiresAtEpochMillis != null}, " +
            "last updated at ${it.updatedAtEpochMillis}")
}

Only the locks with a record in Redis are returned. owner is the current owner while the lock is held (expiresAtEpochMillis is not null), and the previous owner otherwise.

Deadlock detection

A deadlock happens when threads wait for locks that the other ones hold, for example:

• thread 1 holds lock a and waits for lock b
• thread 2 holds lock b and waits for lock a

Or:

• thread 1 holds lock a and waits for lock b
• thread 2 holds lock b and waits for lock c
• thread 3 holds lock c and waits for lock a

(where threads can either be on the local JVM of across multiple JVMs)

With deadlockDetection = true (the default), threads that wait while holding locks register themselves in Redis. After each failed attempt, they look for a cycle of waiting threads leading back to themselves, across all JVMs. Every thread in the cycle finds it, but only one of them gives up: its tryLock throws RedisLockServiceDeadlockException, which releases its outer locks as the exception propagates. The other threads then acquire their locks.

A deadlock is therefore detected within about pollDelayWhenWaitingForLockMs.

Redis restarts

When Redis restarts, it might lose the lock records. Two measures keep the locks safe:

• After a (re)start, no lock is handed out until Redis has been up for lockLeaseTimeSeconds. This waiting time counts toward timeoutMs.
• Within that time, the lease renewal of the previous owners restores their records, so that nobody else takes their locks. It also makes their hasLock() return false, as the locks might have been taken over in between, so their protected code should wind down.

Replicas, failover and Redis Cluster

Both measures rely on noticing the restart. After a failover (e.g. with Sentinel), the promoted replica has been up for long already, so there is no startup wait. Lock records that had not been replicated yet are lost and can be taken right away, while the previous owners only notice on their next lease renewal. Redis Cluster adds to that: the keys of several locks usually live in different slots, so acquiring them atomically is not possible there. The RedisLockService should therefore probably not be used with either.