Helpers¶
Useful methods for working with httplib
, completely decoupled from
code specific to urllib3.
Timeouts¶
-
class
urllib3.util.timeout.
Timeout
(total=None, connect=<object object>, read=<object object>)¶ Timeout configuration.
Timeouts can be defined as a default for a pool:
timeout = Timeout(connect=2.0, read=7.0) http = PoolManager(timeout=timeout) response = http.request('GET', 'http://example.com/')
Or per-request (which overrides the default for the pool):
response = http.request('GET', 'http://example.com/', timeout=Timeout(10))
Timeouts can be disabled by setting all the parameters to
None
:no_timeout = Timeout(connect=None, read=None) response = http.request('GET', 'http://example.com/, timeout=no_timeout)
Parameters: - total (integer, float, or None) –
This combines the connect and read timeouts into one; the read timeout will be set to the time leftover from the connect attempt. In the event that both a connect timeout and a total are specified, or a read timeout and a total are specified, the shorter timeout will be applied.
Defaults to None.
- connect (integer, float, or None) – The maximum amount of time to wait for a connection attempt to a server to succeed. Omitting the parameter will default the connect timeout to the system default, probably the global default timeout in socket.py. None will set an infinite timeout for connection attempts.
- read (integer, float, or None) –
The maximum amount of time to wait between consecutive read operations for a response from the server. Omitting the parameter will default the read timeout to the system default, probably the global default timeout in socket.py. None will set an infinite timeout.
Note
Many factors can affect the total amount of time for urllib3 to return an HTTP response.
For example, Python’s DNS resolver does not obey the timeout specified on the socket. Other factors that can affect total request time include high CPU load, high swap, the program running at a low priority level, or other behaviors.
In addition, the read and total timeouts only measure the time between read operations on the socket connecting the client and the server, not the total amount of time for the request to return a complete response. For most requests, the timeout is raised because the server has not sent the first byte in the specified time. This is not always the case; if a server streams one byte every fifteen seconds, a timeout of 20 seconds will not trigger, even though the request will take several minutes to complete.
If your goal is to cut off any request after a set amount of wall clock time, consider having a second “watcher” thread to cut off a slow request.
-
DEFAULT_TIMEOUT
= <object object>¶ A sentinel object representing the default timeout value
-
clone
()¶ Create a copy of the timeout object
Timeout properties are stored per-pool but each request needs a fresh Timeout object to ensure each one has its own start/stop configured.
Returns: a copy of the timeout object Return type: Timeout
-
connect_timeout
¶ Get the value to use when setting a connection timeout.
This will be a positive float or integer, the value None (never timeout), or the default system timeout.
Returns: Connect timeout. Return type: int, float, Timeout.DEFAULT_TIMEOUT
or None
-
classmethod
from_float
(timeout)¶ Create a new Timeout from a legacy timeout value.
The timeout value used by httplib.py sets the same timeout on the connect(), and recv() socket requests. This creates a
Timeout
object that sets the individual timeouts to thetimeout
value passed to this function.Parameters: timeout (integer, float, sentinel default object, or None) – The legacy timeout value. Returns: Timeout object Return type: Timeout
-
get_connect_duration
()¶ Gets the time elapsed since the call to
start_connect()
.Returns: Elapsed time. Return type: float Raises urllib3.exceptions.TimeoutStateError: if you attempt to get duration for a timer that hasn’t been started.
-
read_timeout
¶ Get the value for the read timeout.
This assumes some time has elapsed in the connection timeout and computes the read timeout appropriately.
If self.total is set, the read timeout is dependent on the amount of time taken by the connect timeout. If the connection time has not been established, a
TimeoutStateError
will be raised.Returns: Value to use for the read timeout. Return type: int, float, Timeout.DEFAULT_TIMEOUT
or NoneRaises urllib3.exceptions.TimeoutStateError: If start_connect()
has not yet been called on this object.
-
start_connect
()¶ Start the timeout clock, used during a connect() attempt
Raises urllib3.exceptions.TimeoutStateError: if you attempt to start a timer that has been started already.
- total (integer, float, or None) –
-
urllib3.util.timeout.
current_time
()¶ Retrieve the current time. This function is mocked out in unit testing.
Retries¶
-
class
urllib3.util.retry.
Retry
(total=10, connect=None, read=None, redirect=None, method_whitelist=frozenset(['HEAD', 'TRACE', 'GET', 'PUT', 'OPTIONS', 'DELETE']), status_forcelist=None, backoff_factor=0, raise_on_redirect=True, _observed_errors=0)¶ Retry configuration.
Each retry attempt will create a new Retry object with updated values, so they can be safely reused.
Retries can be defined as a default for a pool:
retries = Retry(connect=5, read=2, redirect=5) http = PoolManager(retries=retries) response = http.request('GET', 'http://example.com/')
Or per-request (which overrides the default for the pool):
response = http.request('GET', 'http://example.com/', retries=Retry(10))
Retries can be disabled by passing
False
:response = http.request('GET', 'http://example.com/', retries=False)
Errors will be wrapped in
MaxRetryError
unless retries are disabled, in which case the causing exception will be raised.Parameters: - total (int) –
Total number of retries to allow. Takes precedence over other counts.
Set to
None
to remove this constraint and fall back on other counts. It’s a good idea to set this to some sensibly-high value to account for unexpected edge cases and avoid infinite retry loops.Set to
0
to fail on the first retry.Set to
False
to disable and implyraise_on_redirect=False
. - connect (int) –
How many connection-related errors to retry on.
These are errors raised before the request is sent to the remote server, which we assume has not triggered the server to process the request.
Set to
0
to fail on the first retry of this type. - read (int) –
How many times to retry on read errors.
These errors are raised after the request was sent to the server, so the request may have side-effects.
Set to
0
to fail on the first retry of this type. - redirect (int) –
How many redirects to perform. Limit this to avoid infinite redirect loops.
A redirect is a HTTP response with a status code 301, 302, 303, 307 or 308.
Set to
0
to fail on the first retry of this type.Set to
False
to disable and implyraise_on_redirect=False
. - method_whitelist (iterable) –
Set of uppercased HTTP method verbs that we should retry on.
By default, we only retry on methods which are considered to be indempotent (multiple requests with the same parameters end with the same state). See
Retry.DEFAULT_METHOD_WHITELIST
. - status_forcelist (iterable) –
A set of HTTP status codes that we should force a retry on.
By default, this is disabled with
None
. - backoff_factor (float) –
A backoff factor to apply between attempts. urllib3 will sleep for:
{backoff factor} * (2 ^ ({number of total retries} - 1))
seconds. If the backoff_factor is 0.1, then
sleep()
will sleep for [0.1s, 0.2s, 0.4s, ...] between retries. It will never be longer thanRetry.MAX_BACKOFF
.By default, backoff is disabled (set to 0).
- raise_on_redirect (bool) – Whether, if the number of redirects is exhausted, to raise a MaxRetryError, or to return a response with a response code in the 3xx range.
-
BACKOFF_MAX
= 120¶ Maximum backoff time.
-
classmethod
from_int
(retries, redirect=True, default=None)¶ Backwards-compatibility for the old retries format.
-
increment
(method=None, url=None, response=None, error=None, _pool=None, _stacktrace=None)¶ Return a new Retry object with incremented retry counters.
Parameters: - response (
HTTPResponse
) – A response object, or None, if the server did not return a response. - error (Exception) – An error encountered during the request, or None if the response was received successfully.
Returns: A new
Retry
object.- response (
-
is_exhausted
()¶ Are we out of retries?
-
is_forced_retry
(method, status_code)¶ Is this method/status code retryable? (Based on method/codes whitelists)
-
sleep
()¶ Sleep between retry attempts using an exponential backoff.
By default, the backoff factor is 0 and this method will return immediately.
- total (int) –
URL Helpers¶
-
class
urllib3.util.url.
Url
¶ Datastructure for representing an HTTP URL. Used as a return value for
parse_url()
.-
hostname
¶ For backwards-compatibility with urlparse. We’re nice like that.
-
netloc
¶ Network location including host and port
-
request_uri
¶ Absolute path including the query string.
-
url
¶ Convert self into a url
This function should more or less round-trip with
parse_url()
. The returned url may not be exactly the same as the url inputted toparse_url()
, but it should be equivalent by the RFC (e.g., urls with a blank port will have : removed).Example:
>>> U = parse_url('http://google.com/mail/') >>> U.url 'http://google.com/mail/' >>> Url('http', 'username:password', 'host.com', 80, ... '/path', 'query', 'fragment').url 'http://username:password@host.com:80/path?query#fragment'
-
-
urllib3.util.url.
get_host
(url)¶ Deprecated. Use
parse_url()
instead.
-
urllib3.util.url.
parse_url
(url)¶ Given a url, return a parsed
Url
namedtuple. Best-effort is performed to parse incomplete urls. Fields not provided will be None.Partly backwards-compatible with
urlparse
.Example:
>>> parse_url('http://google.com/mail/') Url(scheme='http', host='google.com', port=None, path='/mail/', ...) >>> parse_url('google.com:80') Url(scheme=None, host='google.com', port=80, path=None, ...) >>> parse_url('/foo?bar') Url(scheme=None, host=None, port=None, path='/foo', query='bar', ...)
-
urllib3.util.url.
split_first
(s, delims)¶ Given a string and an iterable of delimiters, split on the first found delimiter. Return two split parts and the matched delimiter.
If not found, then the first part is the full input string.
Example:
>>> split_first('foo/bar?baz', '?/=') ('foo', 'bar?baz', '/') >>> split_first('foo/bar?baz', '123') ('foo/bar?baz', '', None)
Scales linearly with number of delims. Not ideal for large number of delims.
Filepost¶
-
urllib3.filepost.
choose_boundary
()¶ Our embarassingly-simple replacement for mimetools.choose_boundary.
-
urllib3.filepost.
encode_multipart_formdata
(fields, boundary=None)¶ Encode a dictionary of
fields
using the multipart/form-data MIME format.Parameters: - fields – Dictionary of fields or list of (key,
RequestField
). - boundary – If not specified, then a random boundary will be generated using
mimetools.choose_boundary()
.
- fields – Dictionary of fields or list of (key,
-
urllib3.filepost.
iter_field_objects
(fields)¶ Iterate over fields.
Supports list of (k, v) tuples and dicts, and lists of
RequestField
.
-
urllib3.filepost.
iter_fields
(fields)¶ Deprecated since version 1.6.
Iterate over fields.
The addition of
RequestField
makes this function obsolete. Instead, useiter_field_objects()
, which returnsRequestField
objects.Supports list of (k, v) tuples and dicts.
-
class
urllib3.fields.
RequestField
(name, data, filename=None, headers=None)¶ A data container for request body parameters.
Parameters: - name – The name of this request field.
- data – The data/value body.
- filename – An optional filename of the request field.
- headers – An optional dict-like object of headers to initially use for the field.
-
classmethod
from_tuples
(fieldname, value)¶ A
RequestField
factory from old-style tuple parameters.Supports constructing
RequestField
from parameter of key/value strings AND key/filetuple. A filetuple is a (filename, data, MIME type) tuple where the MIME type is optional. For example:'foo': 'bar', 'fakefile': ('foofile.txt', 'contents of foofile'), 'realfile': ('barfile.txt', open('realfile').read()), 'typedfile': ('bazfile.bin', open('bazfile').read(), 'image/jpeg'), 'nonamefile': 'contents of nonamefile field',
Field names and filenames must be unicode.
-
make_multipart
(content_disposition=None, content_type=None, content_location=None)¶ Makes this request field into a multipart request field.
This method overrides “Content-Disposition”, “Content-Type” and “Content-Location” headers to the request parameter.
Parameters: - content_type – The ‘Content-Type’ of the request body.
- content_location – The ‘Content-Location’ of the request body.
-
render_headers
()¶ Renders the headers for this request field.
-
urllib3.fields.
format_header_param
(name, value)¶ Helper function to format and quote a single header parameter.
Particularly useful for header parameters which might contain non-ASCII values, like file names. This follows RFC 2231, as suggested by RFC 2388 Section 4.4.
Parameters: - name – The name of the parameter, a string expected to be ASCII only.
- value – The value of the parameter, provided as a unicode string.
Request¶
-
class
urllib3.request.
RequestMethods
(headers=None)¶ Convenience mixin for classes who implement a
urlopen()
method, such asHTTPConnectionPool
andPoolManager
.Provides behavior for making common types of HTTP request methods and decides which type of request field encoding to use.
Specifically,
request_encode_url()
is for sending requests whose fields are encoded in the URL (such as GET, HEAD, DELETE).request_encode_body()
is for sending requests whose fields are encoded in the body of the request using multipart or www-form-urlencoded (such as for POST, PUT, PATCH).request()
is for making any kind of request, it will look up the appropriate encoding format and use one of the above two methods to make the request.Initializer parameters:
Parameters: headers – Headers to include with all requests, unless other headers are given explicitly. -
request
(method, url, fields=None, headers=None, **urlopen_kw)¶ Make a request using
urlopen()
with the appropriate encoding offields
based on themethod
used.This is a convenience method that requires the least amount of manual effort. It can be used in most situations, while still having the option to drop down to more specific methods when necessary, such as
request_encode_url()
,request_encode_body()
, or even the lowest levelurlopen()
.
-
request_encode_body
(method, url, fields=None, headers=None, encode_multipart=True, multipart_boundary=None, **urlopen_kw)¶ Make a request using
urlopen()
with thefields
encoded in the body. This is useful for request methods like POST, PUT, PATCH, etc.When
encode_multipart=True
(default), thenurllib3.filepost.encode_multipart_formdata()
is used to encode the payload with the appropriate content type. Otherwiseurllib.urlencode()
is used with the ‘application/x-www-form-urlencoded’ content type.Multipart encoding must be used when posting files, and it’s reasonably safe to use it in other times too. However, it may break request signing, such as with OAuth.
Supports an optional
fields
parameter of key/value strings AND key/filetuple. A filetuple is a (filename, data, MIME type) tuple where the MIME type is optional. For example:fields = { 'foo': 'bar', 'fakefile': ('foofile.txt', 'contents of foofile'), 'realfile': ('barfile.txt', open('realfile').read()), 'typedfile': ('bazfile.bin', open('bazfile').read(), 'image/jpeg'), 'nonamefile': 'contents of nonamefile field', }
When uploading a file, providing a filename (the first parameter of the tuple) is optional but recommended to best mimick behavior of browsers.
Note that if
headers
are supplied, the ‘Content-Type’ header will be overwritten because it depends on the dynamic random boundary string which is used to compose the body of the request. The random boundary string can be explicitly set with themultipart_boundary
parameter.
-
request_encode_url
(method, url, fields=None, **urlopen_kw)¶ Make a request using
urlopen()
with thefields
encoded in the url. This is useful for request methods like GET, HEAD, DELETE, etc.
-
-
urllib3.util.request.
make_headers
(keep_alive=None, accept_encoding=None, user_agent=None, basic_auth=None, proxy_basic_auth=None, disable_cache=None)¶ Shortcuts for generating request headers.
Parameters: - keep_alive – If
True
, adds ‘connection: keep-alive’ header. - accept_encoding – Can be a boolean, list, or string.
True
translates to ‘gzip,deflate’. List will get joined by comma. String will be used as provided. - user_agent – String representing the user-agent you want, such as “python-urllib3/0.6”
- basic_auth – Colon-separated username:password string for ‘authorization: basic ...’ auth header.
- proxy_basic_auth – Colon-separated username:password string for ‘proxy-authorization: basic ...’ auth header.
- disable_cache – If
True
, adds ‘cache-control: no-cache’ header.
Example:
>>> make_headers(keep_alive=True, user_agent="Batman/1.0") {'connection': 'keep-alive', 'user-agent': 'Batman/1.0'} >>> make_headers(accept_encoding=True) {'accept-encoding': 'gzip,deflate'}
- keep_alive – If
Response¶
-
class
urllib3.response.
HTTPResponse
(body='', headers=None, status=0, version=0, reason=None, strict=0, preload_content=True, decode_content=True, original_response=None, pool=None, connection=None)¶ HTTP Response container.
Backwards-compatible to httplib’s HTTPResponse but the response
body
is loaded and decoded on-demand when thedata
property is accessed. This class is also compatible with the Python standard library’sio
module, and can hence be treated as a readable object in the context of that framework.Extra parameters for behaviour not present in httplib.HTTPResponse:
Parameters: - preload_content – If True, the response’s body will be preloaded during construction.
- decode_content – If True, attempts to decode specific content-encoding’s based on headers (like ‘gzip’ and ‘deflate’) will be skipped and raw data will be used instead.
- original_response – When this HTTPResponse wrapper is generated from an httplib.HTTPResponse object, it’s convenient to include the original for debug purposes. It’s otherwise unused.
-
CONTENT_DECODERS
= ['gzip', 'deflate']¶
-
REDIRECT_STATUSES
= [301, 302, 303, 307, 308]¶
-
close
()¶
-
closed
¶
-
data
¶
-
fileno
()¶
-
flush
()¶
-
classmethod
from_httplib
(ResponseCls, r, **response_kw)¶ Given an
httplib.HTTPResponse
instancer
, return a correspondingurllib3.response.HTTPResponse
object.Remaining parameters are passed to the HTTPResponse constructor, along with
original_response=r
.
-
get_redirect_location
()¶ Should we redirect and where to?
Returns: Truthy redirect location string if we got a redirect status code and valid location. None
if redirect status and no location.False
if not a redirect status code.
-
getheader
(name, default=None)¶
-
getheaders
()¶
-
read
(amt=None, decode_content=None, cache_content=False)¶ Similar to
httplib.HTTPResponse.read()
, but with two additional parameters:decode_content
andcache_content
.Parameters: - amt – How much of the content to read. If specified, caching is skipped because it doesn’t make sense to cache partial content as the full response.
- decode_content – If True, will attempt to decode the body based on the ‘content-encoding’ header.
- cache_content – If True, will save the returned data such that the same result is
returned despite of the state of the underlying file object. This
is useful if you want the
.data
property to continue working after having.read()
the file object. (Overridden ifamt
is set.)
-
read_chunked
(amt=None, decode_content=None)¶ Similar to
HTTPResponse.read()
, but with an additional parameter:decode_content
.Parameters: decode_content – If True, will attempt to decode the body based on the ‘content-encoding’ header.
-
readable
()¶
-
readinto
(b)¶
-
release_conn
()¶
-
stream
(amt=65536, decode_content=None)¶ A generator wrapper for the read() method. A call will block until
amt
bytes have been read from the connection or until the connection is closed.Parameters: - amt – How much of the content to read. The generator will return up to much data per iteration, but may return less. This is particularly likely when using compressed data. However, the empty string will never be returned.
- decode_content – If True, will attempt to decode the body based on the ‘content-encoding’ header.
-
tell
()¶ Obtain the number of bytes pulled over the wire so far. May differ from the amount of content returned by :meth:
HTTPResponse.read
if bytes are encoded on the wire (e.g, compressed).
SSL/TLS Helpers¶
-
urllib3.util.ssl_.
assert_fingerprint
(cert, fingerprint)¶ Checks if given fingerprint matches the supplied certificate.
Parameters: - cert – Certificate as bytes object.
- fingerprint – Fingerprint as string of hexdigits, can be interspersed by colons.
-
urllib3.util.ssl_.
create_urllib3_context
(ssl_version=None, cert_reqs=None, options=None, ciphers=None)¶ All arguments have the same meaning as
ssl_wrap_socket
.By default, this function does a lot of the same work that
ssl.create_default_context
does on Python 3.4+. It:- Disables SSLv2, SSLv3, and compression
- Sets a restricted set of server ciphers
If you wish to enable SSLv3, you can do:
from urllib3.util import ssl_ context = ssl_.create_urllib3_context() context.options &= ~ssl_.OP_NO_SSLv3
You can do the same to enable compression (substituting
COMPRESSION
forSSLv3
in the last line above).Parameters: - ssl_version – The desired protocol version to use. This will default to PROTOCOL_SSLv23 which will negotiate the highest protocol that both the server and your installation of OpenSSL support.
- cert_reqs – Whether to require the certificate verification. This defaults to
ssl.CERT_REQUIRED
. - options – Specific OpenSSL options. These default to
ssl.OP_NO_SSLv2
,ssl.OP_NO_SSLv3
,ssl.OP_NO_COMPRESSION
. - ciphers – Which cipher suites to allow the server to select.
Returns: Constructed SSLContext object with specified options
Return type: SSLContext
-
urllib3.util.ssl_.
resolve_cert_reqs
(candidate)¶ Resolves the argument to a numeric constant, which can be passed to the wrap_socket function/method from the ssl module. Defaults to
ssl.CERT_NONE
. If given a string it is assumed to be the name of the constant in thessl
module or its abbrevation. (So you can specify REQUIRED instead of CERT_REQUIRED. If it’s neither None nor a string we assume it is already the numeric constant which can directly be passed to wrap_socket.
-
urllib3.util.ssl_.
resolve_ssl_version
(candidate)¶ like resolve_cert_reqs
-
urllib3.util.ssl_.
ssl_wrap_socket
(sock, keyfile=None, certfile=None, cert_reqs=None, ca_certs=None, server_hostname=None, ssl_version=None, ciphers=None, ssl_context=None)¶ All arguments except for server_hostname and ssl_context have the same meaning as they do when using
ssl.wrap_socket()
.Parameters: - server_hostname – When SNI is supported, the expected hostname of the certificate
- ssl_context – A pre-made
SSLContext
object. If none is provided, one will be created usingcreate_urllib3_context()
. - ciphers – A string of ciphers we wish the client to support. This is not supported on Python 2.6 as the ssl module does not support it.