PoolManager

A pool manager is an abstraction for a collection of ConnectionPools.

If you need to make requests to multiple hosts, then you can use a PoolManager, which takes care of maintaining your pools so you don’t have to.

>>> from urllib3 import PoolManager
>>> manager = PoolManager(10)
>>> r = manager.request('GET', 'http://example.com')
>>> r.headers['server']
'ECS (iad/182A)'
>>> r = manager.request('GET', 'http://httpbin.org/')
>>> r.headers['server']
'gunicorn/18.0'
>>> r = manager.request('POST', 'http://httpbin.org/headers')
>>> r = manager.request('HEAD', 'http://httpbin.org/cookies')
>>> len(manager.pools)
2
>>> conn = manager.connection_from_host('httpbin.org')
>>> conn.num_requests
3

The API of a PoolManager object is similar to that of a ConnectionPool, so they can be passed around interchangeably.

The PoolManager uses a Least Recently Used (LRU) policy for discarding old pools. That is, if you set the PoolManager num_pools to 10, then after making requests to 11 or more different hosts, the least recently used pools will be cleaned up eventually.

Cleanup of stale pools does not happen immediately. You can read more about the implementation and the various adjustable variables within RecentlyUsedContainer.

API

class urllib3.poolmanager.PoolManager(num_pools=10, headers=None, **connection_pool_kw)

Allows for arbitrary requests while transparently keeping track of necessary connection pools for you.

Parameters:
  • num_pools – Number of connection pools to cache before discarding the least recently used pool.
  • headers – Headers to include with all requests, unless other headers are given explicitly.
  • **connection_pool_kw – Additional parameters are used to create fresh urllib3.connectionpool.ConnectionPool instances.

Example:

>>> manager = PoolManager(num_pools=2)
>>> r = manager.request('GET', 'http://google.com/')
>>> r = manager.request('GET', 'http://google.com/mail')
>>> r = manager.request('GET', 'http://yahoo.com/')
>>> len(manager.pools)
2
clear()

Empty our store of pools and direct them all to close.

This will not affect in-flight connections, but they will not be re-used after completion.

connection_from_host(host, port=None, scheme='http')

Get a ConnectionPool based on the host, port, and scheme.

If port isn’t given, it will be derived from the scheme using urllib3.connectionpool.port_by_scheme.

connection_from_url(url)

Similar to urllib3.connectionpool.connection_from_url() but doesn’t pass any additional parameters to the urllib3.connectionpool.ConnectionPool constructor.

Additional parameters are taken from the PoolManager constructor.

request(method, url, fields=None, headers=None, **urlopen_kw)

Make a request using urlopen() with the appropriate encoding of fields based on the method used.

This is a convenience method that requires the least amount of manual effort. It can be used in most situations, while still having the option to drop down to more specific methods when necessary, such as request_encode_url(), request_encode_body(), or even the lowest level urlopen().

request_encode_body(method, url, fields=None, headers=None, encode_multipart=True, multipart_boundary=None, **urlopen_kw)

Make a request using urlopen() with the fields encoded in the body. This is useful for request methods like POST, PUT, PATCH, etc.

When encode_multipart=True (default), then urllib3.filepost.encode_multipart_formdata() is used to encode the payload with the appropriate content type. Otherwise urllib.urlencode() is used with the ‘application/x-www-form-urlencoded’ content type.

Multipart encoding must be used when posting files, and it’s reasonably safe to use it in other times too. However, it may break request signing, such as with OAuth.

Supports an optional fields parameter of key/value strings AND key/filetuple. A filetuple is a (filename, data, MIME type) tuple where the MIME type is optional. For example:

fields = {
    'foo': 'bar',
    'fakefile': ('foofile.txt', 'contents of foofile'),
    'realfile': ('barfile.txt', open('realfile').read()),
    'typedfile': ('bazfile.bin', open('bazfile').read(),
                  'image/jpeg'),
    'nonamefile': 'contents of nonamefile field',
}

When uploading a file, providing a filename (the first parameter of the tuple) is optional but recommended to best mimick behavior of browsers.

Note that if headers are supplied, the ‘Content-Type’ header will be overwritten because it depends on the dynamic random boundary string which is used to compose the body of the request. The random boundary string can be explicitly set with the multipart_boundary parameter.

request_encode_url(method, url, fields=None, **urlopen_kw)

Make a request using urlopen() with the fields encoded in the url. This is useful for request methods like GET, HEAD, DELETE, etc.

urlopen(method, url, redirect=True, **kw)

Same as urllib3.connectionpool.HTTPConnectionPool.urlopen() with custom cross-host redirect logic and only sends the request-uri portion of the url.

The given url parameter must be absolute, such that an appropriate urllib3.connectionpool.ConnectionPool can be chosen for it.

ProxyManager

ProxyManager is an HTTP proxy-aware subclass of PoolManager. It produces a single HTTPConnectionPool instance for all HTTP connections and individual per-server:port HTTPSConnectionPool instances for tunnelled HTTPS connections.

API

class urllib3.poolmanager.ProxyManager(proxy_url, num_pools=10, headers=None, proxy_headers=None, **connection_pool_kw)

Behaves just like PoolManager, but sends all requests through the defined proxy, using the CONNECT method for HTTPS URLs.

Parameters:
  • proxy_url – The URL of the proxy to be used.
  • proxy_headers – A dictionary contaning headers that will be sent to the proxy. In case of HTTP they are being sent with each request, while in the HTTPS/CONNECT case they are sent only once. Could be used for proxy authentication.
Example:
>>> proxy = urllib3.ProxyManager('http://localhost:3128/')
>>> r1 = proxy.request('GET', 'http://google.com/')
>>> r2 = proxy.request('GET', 'http://httpbin.org/')
>>> len(proxy.pools)
1
>>> r3 = proxy.request('GET', 'https://httpbin.org/')
>>> r4 = proxy.request('GET', 'https://twitter.com/')
>>> len(proxy.pools)
3