P2
A user reported that /backport commands are taking a long time to process, up to 30 minutes. Here is an example:
https://github.com/openjdk/jdk/commit/53bb7cd415f1d2e87d8f06ad5eb611bfdf8ef0ad
Investigating this, I noticed that the CommitCommandWorkItem processing this was getting several failed REST calls to GitHub due to hitting a secondary rate limit. The message looks like this:
{
"documentation_url": "https://docs.github.com/en/free-pro-team@latest/rest/overview/rate-limits-for-the-rest-api#about-secondary-rate-limits",
"message": "You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID 3264:936D:123F68F:25A489C:6557D2AC."
}
Looking through logs, we are getting these specifically for REST calls to this endpoint on the openjdk-bots/jdk17u-dev repo:
https://api.github.com/repos/openjdk-bots/jdk17u-dev/branches
It should be noted that the number of branches on this repo is now large enough to require at least 5 pages to retrieve them all and the rate limit seems to hit randomly on any of the pagination calls. It's still unclear what the correct fix is here. I'm guessing that we started to hit the limit when the number of branches hit a certain threshold as that in turn caused a certain number of pagination calls to be needed. The reported /backport invocation was targeting jdk21u, so this isn't limited to jdk17u-dev, but I'm guessing it's primarily that repo that is causing the rate limit to be triggered, and when that happens, it also affects other repos under openjdk-bots.
https://github.com/openjdk/jdk/commit/53bb7cd415f1d2e87d8f06ad5eb611bfdf8ef0ad
Investigating this, I noticed that the CommitCommandWorkItem processing this was getting several failed REST calls to GitHub due to hitting a secondary rate limit. The message looks like this:
{
"documentation_url": "https://docs.github.com/en/free-pro-team@latest/rest/overview/rate-limits-for-the-rest-api#about-secondary-rate-limits",
"message": "You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID 3264:936D:123F68F:25A489C:6557D2AC."
}
Looking through logs, we are getting these specifically for REST calls to this endpoint on the openjdk-bots/jdk17u-dev repo:
https://api.github.com/repos/openjdk-bots/jdk17u-dev/branches
It should be noted that the number of branches on this repo is now large enough to require at least 5 pages to retrieve them all and the rate limit seems to hit randomly on any of the pagination calls. It's still unclear what the correct fix is here. I'm guessing that we started to hit the limit when the number of branches hit a certain threshold as that in turn caused a certain number of pagination calls to be needed. The reported /backport invocation was targeting jdk21u, so this isn't limited to jdk17u-dev, but I'm guessing it's primarily that repo that is causing the rate limit to be triggered, and when that happens, it also affects other repos under openjdk-bots.
