After fixing SKARA-1012, a new similar problem has appeared. Now we frequently get "uncleanable" repos in the bot scratch dirs. The error messages look like this:

Invalid local repository detected (uncleanable) - preserved in: /home/bots/scratch/scratch-26/mlbridge-mergebase/openjdk/jfx-uncleanable-859b698e-e593-4621-ad94-5de33032ba9c

The failing git command is "git reset --hard" which fails with:

fatal: Unable to create '/home/bots/scratch/scratch-58/mlbridge-mergebase/openjdk/panama-foreign-uncleanable-6863adc2-f779-4910-9e5c-19e56ebc881d/.git/index.lock': File exists.

Tracing back logs when a repo like this was last used typically finds the initial clone of the repo, which is then interrupted by a bot runner restart. My theory here is that since we added --dissociate, cloning takes much longer, so the probability of getting interrupted by a restart goes up quite a bit.

One way of mitigating this would be to always clone to a different dir name (add .tmp or something), then move the new repo to the final name. By doing that we know that any existing dir with a .tmp suffix is known to be unfinished and can just be deleted, with no need to send any alarms or preserve the contents.