After I was greeted by problems with my DNS setup right after I got up from my monitoring, the debugging started right away.
In the end, two nodes were not really reachable anymore, apparently because they received too much traffic.
The only way to get the nodes at least manageable via SSH was to disable the Revolver IPv4 address. After that the delay in the SSH sessions was gone, turning off the dnsdist was not enough.