Recently, we have seen the PR bot getting slowed down to a crawl, taking hours responding to updates in PRs (e.g. SKARA-1178). I believe I have finally found the cause. The scheduler is slowly getting filled up with CommitCommentWorkItems.
A CommitCommentWorkItem is scheduled for each repository every time PullRequestBot::getPeriodicItems is called, which is every 10 seconds. So unless we are able to process these for every repository in 10 seconds, they will start to pile up. Until recently, we probably could, but the more repositories we create, the more of these get scheduled.
The defense for this is the WorkItem::concurrentWith method, which is supposed to protect both WorkItems that touch the same data from getting into races, but also to avoid keeping duplicate WorkItems in the pending queue. In the case of CommitCommentWorkItem, ::concurrentWith just returns true, so there is no protection against flooding.
A CommitCommentWorkItem is scheduled for each repository every time PullRequestBot::getPeriodicItems is called, which is every 10 seconds. So unless we are able to process these for every repository in 10 seconds, they will start to pile up. Until recently, we probably could, but the more repositories we create, the more of these get scheduled.
The defense for this is the WorkItem::concurrentWith method, which is supposed to protect both WorkItems that touch the same data from getting into races, but also to avoid keeping duplicate WorkItems in the pending queue. In the case of CommitCommentWorkItem, ::concurrentWith just returns true, so there is no protection against flooding.
- relates to
-
SKARA-1178 PR bot is slow
-
- Resolved
-
