`url_exists` related improvements (#34095)

For reasons beyond me Python thinks it's a great idea to upgrade HEAD
requests to GET requests when following redirects. So, this PR adds a
better `HTTPRedirectHandler`, and also moves some ad-hoc logic around
for dealing with disabling SSL certs verification.

Also, I'm stumped by the fact that Spack's `url_exists` does not use
HEAD requests at all, so in certain cases Spack awkwardly downloads
something first to see if it can download it, and then downloads it
again because it knows it can download it. So, this PR ensures that both
urllib and botocore use HEAD requests.

Finally, it also removes some things that were there to support currently
unsupported Python versions.

Notice that the HTTP spec [section 10.3.2](https://datatracker.ietf.org/doc/html/rfc2616.html#section-10.3.2) just talks about how to deal
with POST request on redirect (whether to follow or not):

>   If the 301 status code is received in response to a request other
>   than GET or HEAD, the user agent MUST NOT automatically redirect the
>   request unless it can be confirmed by the user, since this might
>   change the conditions under which the request was issued.

>   Note: When automatically redirecting a POST request after
>   receiving a 301 status code, some existing HTTP/1.0 user agents
>   will erroneously change it into a GET request.

Python has a comment about this, they choose to go with the "erroneous change".
But they then mess up the HEAD request while following the redirect, probably
because they were too busy discussing how to deal with POST.

See https://github.com/python/cpython/pull/99731

0 files changed, 0 insertions, 0 deletions