Rate limit requests for addons.xml.gz

[MoojMidge]

Rate limiting has been added to mirrors.kodi.tv which the multiple requests for the various repository addons.xml.gz will trigger.

This adds some rate limiting, error handling and retry backoff to avoid the rate limit resulting in the addon checker failing

[MoojMidge]

Maybe this line should be moved back to where it was, so that the checker will fail if the versioned repository download fails due to some other server error that cannot be resolved by the allowed retries?

[MoojMidge]

One issue with this PR is that, while it does prevent the rate limiting, it may allow for the possibility of the checker appearing to complete successfully, if some other server error occurs.

In this instance the checker would just run with an empty list of addons for the versioned repository that failed to download. Perhaps this should still raise an exception?

It was also suggested that not all these repositories (currently Gotham and up) should be checked anymore. Maybe only Leia and above?

[basrieter]

Could we not just send a specific header that will disable the rate limit for that call? The main reason we limit the rate is the normal repository fetching of addon.xml(.md5) that floods the servers. So we could perhaps differentiate on the agent header in the rate limiter?

[MoojMidge]

Yes, that is also an option, but would only apply to mirrors.kodi.tv itself.

The actual mirrors themselves would not respond in the same manner.

I updated the PR:

• removed the explicit time delays between requests, while still keeping the retry back-off
• instead limit the number of connections per host to 3
• applies to mirrors.kodi.tv and any mirrors used, up to the default connection pool limit of 10
• will block any further connections per host until the existing 3 connections to the host have finished

Should be a little bit more friendly to all the servers involved.

Also moved the self.addons = [] line back to where it was originally, so that the checker would fail if the list of addons still failed to download for some reason.

[razzeee]

Thanks for working on this

[MoojMidge]

Just realised there is an issue with this PR if used with older distributions that have old versions of requests/urllib3 e.g. Ubuntu 20.04

requests.adapters.Retry is using the allowed_methods parameter, which was introduced in v1.26 of urllib3 a bit over 3 years ago.

Shall I create a PR for https://github.com/xbmc/addon-check/blob/master/requirements.txt to use this as the minimum version required?

[basrieter]

Thay makes sense.