Developers¶
Interfaces¶
-
class
tooz.coordination.
CoordinationDriver
[source]¶ Bases:
object
-
__weakref__
¶ list of weak references to the object (if defined)
-
static
create_group
(group_id)¶ Request the creation of a group asynchronously.
Parameters: group_id (str) – the id of the group to create Returns: None Return type: CoordAsyncResult
-
static
delete_group
(group_id)¶ Delete a group asynchronously.
Parameters: group_id (str) – the id of the group to leave Returns: Result Return type: CoordAsyncResult
-
static
get_groups
()¶ Return the list composed by all groups ids asynchronously.
Returns: the list of all created group ids Return type: CoordAsyncResult
-
static
get_leader
(group_id)¶ Return the leader for a group.
Parameters: group_id – the id of the group: Returns: the leader Return type: CoordAsyncResult
-
static
get_lock
(name)¶ Return a distributed lock.
This is a exclusive lock, a second call to acquire() will block or return False.
Parameters: name – The lock name that is used to identify it across all nodes.
-
static
get_member_capabilities
(group_id, member_id)¶ Return the capabilities of a member asynchronously.
Parameters: - group_id (str) – the id of the group of the member
- member_id (str) – the id of the member
Returns: capabilities of a member
Return type: CoordAsyncResult
-
static
get_members
(group_id)¶ Return the list of all members ids of the specified group asynchronously.
Returns: list of all created group ids Return type: CoordAsyncResult
-
static
heartbeat
()¶ Method to run once in a while to be sure that the member is not dead and is still an active member of a group.
-
static
join_group
(group_id, capabilities='')¶ Join a group and establish group membership asynchronously.
Parameters: - group_id (str) – the id of the group to join
- capabilities (object (typically str)) – the capabilities of the joined member
Returns: None
Return type: CoordAsyncResult
-
static
leave_group
(group_id)¶ Leave a group asynchronously.
Parameters: group_id (str) – the id of the group to leave Returns: None Return type: CoordAsyncResult
-
static
run_elect_coordinator
()¶ Try to leader elect this coordinator & activate hooks on success.
-
static
run_watchers
(timeout=None)¶ Run the watchers callback.
This may also activate
run_elect_coordinator()
(depending on driver implementation).
-
static
stand_down_group_leader
(group_id)¶ Stand down as the group leader if we are.
Parameters: group_id – The group where we don’t want to be a leader anymore
-
start
()¶ Start the service engine.
If needed, the establishment of a connection to the servers is initiated.
-
stop
()¶ Stop the service engine.
If needed, the connection to servers is closed and the client will disappear from all joined groups.
-
unwatch_elected_as_leader
(group_id, callback)¶ Call a function when member gets elected as leader.
The callback functions will be executed when run_watchers is called.
Parameters: - group_id – The group id to watch
- callback – The function to execute when a member leaves this group
-
unwatch_join_group
(group_id, callback)¶ Stop executing a function when a group_id sees a new member joined.
Parameters: - group_id – The group id to unwatch
- callback – The function that was executed when a member joined this group
-
unwatch_leave_group
(group_id, callback)¶ Stop executing a function when a group_id sees a new member leaving.
Parameters: - group_id – The group id to unwatch
- callback – The function that was executed when a member left this group
-
static
update_capabilities
(group_id, capabilities)¶ Update capabilities of the caller in the specified group asynchronously.
Parameters: - group_id (str) – the id of the group of the current member
- capabilities (object (typically str)) – the capabilities of the updated member
Returns: None
Return type: CoordAsyncResult
-
watch_elected_as_leader
(group_id, callback)¶ Call a function when member gets elected as leader.
The callback functions will be executed when run_watchers is called.
Parameters: - group_id – The group id to watch
- callback – The function to execute when a member leaves this group
-
watch_join_group
(group_id, callback)¶ Call a function when group_id sees a new member joined.
The callback functions will be executed when run_watchers is called.
Parameters: - group_id – The group id to watch
- callback – The function to execute when a member joins this group
-
watch_leave_group
(group_id, callback)¶ Call a function when group_id sees a new member leaving.
The callback functions will be executed when run_watchers is called.
Parameters: - group_id – The group id to watch
- callback – The function to execute when a member leaves this group
-
File¶
-
class
tooz.drivers.file.
FileDriver
(member_id, parsed_url, options)[source]¶ Bases:
tooz.coordination._RunWatchersMixin
,tooz.coordination.CoordinationDriver
A file based driver.
This driver uses files and directories (and associated file locks) to provide the coordination driver semantics and required API(s). It is missing some functionality but in the future these not implemented API(s) will be filled in.
NOTE(harlowja): it does not automatically delete members from groups of processes that have died, manual cleanup will be needed for those types of failures.
-
HASH_ROUTINE
= 'sha1'¶ This routine is used to hash a member (or group) id into a filesystem safe name that can be used for member lookup and group joining.
-
IPC¶
-
class
tooz.drivers.ipc.
IPCDriver
(member_id, parsed_url, options)[source]¶ Bases:
tooz.coordination.CoordinationDriver
A IPC based driver.
This driver uses IPC concepts to provide the coordination driver semantics and required API(s). It is missing some functionality but in the future these not implemented API(s) will be filled in.
Memcached¶
-
class
tooz.drivers.memcached.
MemcachedDriver
(member_id, parsed_url, options)[source]¶ Bases:
tooz.coordination._RunWatchersMixin
,tooz.coordination.CoordinationDriver
A memcached based driver.
This driver users memcached concepts to provide the coordination driver semantics and required API(s). It is fully functional and implements all of the coordination driver API(s). It stores data into memcache using expiries and msgpack encoded values.
-
DEFAULT_TIMEOUT
= 30¶ Default socket/lock/member/leader timeout used when none is provided.
-
GROUP_LEADER_PREFIX
= '_TOOZ_GROUP_LEADER_'¶ Key prefix attached to leaders of groups (used in name-spacing keys)
-
GROUP_LIST_KEY
= '_TOOZ_GROUP_LIST'¶ Key where all groups ‘known’ are stored.
-
GROUP_PREFIX
= '_TOOZ_GROUP_'¶ Key prefix attached to groups (used in name-spacing keys)
-
MEMBER_PREFIX
= '_TOOZ_MEMBER_'¶ Key prefix attached to members of groups (used in name-spacing keys)
-
STILL_ALIVE
= "It's alive!"¶ String used to keep a key/member alive (until it next expires).
-
Mysql¶
-
class
tooz.drivers.mysql.
MySQLDriver
(member_id, parsed_url, options)[source]¶ Bases:
tooz.coordination.CoordinationDriver
A MySQL based driver.
This driver users MySQL database tables to provide the coordination driver semantics and required API(s). It is missing some functionality but in the future these not implemented API(s) will be filled in.
PostgreSQL¶
-
class
tooz.drivers.pgsql.
PostgresDriver
(member_id, parsed_url, options)[source]¶ Bases:
tooz.coordination.CoordinationDriver
A PostgreSQL based driver.
This driver users PostgreSQL database tables to provide the coordination driver semantics and required API(s). It is missing some functionality but in the future these not implemented API(s) will be filled in.
Redis¶
-
class
tooz.drivers.redis.
RedisDriver
(member_id, parsed_url, options)[source]¶ Bases:
tooz.coordination._RunWatchersMixin
,tooz.coordination.CoordinationDriver
Redis provides a few nice benefits that act as a poormans zookeeper.
It is fully functional and implements all of the coordination driver API(s). It stores data into redis using the provided redis API(s) using msgpack encoded values as needed.
- Durability (when setup with AOF mode).
- Consistent, note that this is still restricted to only one redis server, without the recently released redis (alpha) clustering > 1 server will not be consistent when partitions or failures occur (even redis clustering docs state it is not a fully AP or CP solution, which means even with it there will still be potential inconsistencies).
- Master/slave failover (when setup with redis sentinel), giving some notion of HA (values can be lost when a failover transition occurs).
To use a sentinel the connection URI must point to the sentinel server. At connection time the sentinel will be asked for the current IP and port of the master and then connect there. The connection URI for sentinel should be written as follows:
redis://<sentinel host>:<sentinel port>?sentinel=<master name>
Additional sentinel hosts are listed with multiple
sentinel_fallback
parameters as follows:redis://<sentinel host>:<sentinel port>?sentinel=<master name>& sentinel_fallback=<other sentinel host>:<sentinel port>& sentinel_fallback=<other sentinel host>:<sentinel port>& sentinel_fallback=<other sentinel host>:<sentinel port>
Further resources/links:
Note that this client will itself retry on transaction failure (when they keys being watched have changed underneath the current transaction). Currently the number of attempts that are tried is infinite (this might be addressed in https://github.com/andymccurdy/redis-py/issues/566 when that gets worked on). See http://redis.io/topics/transactions for more information on this topic.
-
CLIENT_ARGS
= frozenset(['ssl_certfile', 'encoding', 'socket_timeout', 'db', 'retry_on_timeout', 'sentinel_fallback', 'ssl', 'ssl_keyfile', 'socket_keepalive', 'sentinel'])¶ Keys that we allow to proxy from the coordinator configuration into the redis client (used to configure the redis client internals so that it works as you expect/want it to).
See: http://redis-py.readthedocs.org/en/latest/#redis.Redis
See: https://github.com/andymccurdy/redis-py/blob/2.10.3/redis/client.py
-
CLIENT_BOOL_ARGS
= frozenset(['ssl', 'retry_on_timeout'])¶ Client arguments that are expected to be boolean convertible.
-
CLIENT_DEFAULT_SOCKET_TO
= 30¶ Default socket timeout to use when none is provided.
-
CLIENT_INT_ARGS
= frozenset(['socket_keepalive', 'socket_timeout', 'db'])¶ Client arguments that are expected to be int convertible.
-
CLIENT_LIST_ARGS
= frozenset(['sentinel_fallback'])¶ Client arguments that are expected/allowed to be lists.
-
DEFAULT_ENCODING
= 'utf8'¶ This is for python3.x; which will behave differently when returned binary types or unicode types (redis uses binary internally it appears), so to just stick with a common way of doing this, make all the things binary (with this default encoding if one is not given and a unicode string is provided).
-
DEFAULT_NAMESPACE
= '_tooz'¶ Default namespace for keys when none is provided.
-
GROUP_EXISTS
= '__created__'¶ Redis deletes dictionaries that have no keys in them, which means the key will disappear which means we can’t tell the difference between a group not existing and a group being empty without this key being saved...
-
GROUP_EXISTS_VALUE
= '1'¶ Value used (with group exists key) to keep a group from disappearing.
-
MIN_VERSION
= LooseVersion ('2.6.0')¶ The min redis version that this driver requires to operate with...
-
NAMESPACE_SEP
= ':'¶ Separator that is used to combine a key with the namespace (to get the actual key that will be used).
-
SCRIPTS
= {'delete_group': '\n-- Extract *all* the variables (so we can easily know what they are)...\nlocal namespaced_group_key = KEYS[1]\nlocal all_groups_key = KEYS[2]\nlocal no_namespaced_group_key = ARGV[1]\nif redis.call("exists", namespaced_group_key) == 0 then\n return -1\nend\nif redis.call("sismember", all_groups_key, no_namespaced_group_key) == 0 then\n return -2\nend\nif redis.call("hlen", namespaced_group_key) > 1 then\n return -3\nend\n-- First remove from the set (then delete the group); if the set removal\n-- fails, at least the group will still exist (and can be fixed manually)...\nif redis.call("srem", all_groups_key, no_namespaced_group_key) == 0 then\n return -4\nend\nredis.call("del", namespaced_group_key)\nreturn 1\n', 'create_group': '\n-- Extract *all* the variables (so we can easily know what they are)...\nlocal namespaced_group_key = KEYS[1]\nlocal all_groups_key = KEYS[2]\nlocal no_namespaced_group_key = ARGV[1]\nif redis.call("exists", namespaced_group_key) == 1 then\n return 0\nend\nredis.call("sadd", all_groups_key, no_namespaced_group_key)\nredis.call("hset", namespaced_group_key,\n "${group_existence_key}", "${group_existence_value}")\nreturn 1\n', 'update_capabilities': '\n-- Extract *all* the variables (so we can easily know what they are)...\nlocal group_key = KEYS[1]\nlocal member_id = ARGV[1]\nlocal caps = ARGV[2]\nif redis.call("exists", group_key) == 0 then\n return -1\nend\nif redis.call("hexists", group_key, member_id) == 0 then\n return -2\nend\nredis.call("hset", group_key, member_id, caps)\nreturn 1\n'}¶ Lua template scripts that will be used by various methods (they are turned into real scripts and loaded on call into the
start()
method).
-
STILL_ALIVE
= 'Not dead!'¶ String used to keep a key/member alive (until it next expires).
Zake¶
Zookeeper¶
-
class
tooz.drivers.zookeeper.
KazooDriver
(member_id, parsed_url, options)[source]¶ Bases:
tooz.drivers.zookeeper.BaseZooKeeperDriver
This driver uses the kazoo client against real zookeeper servers.
It is fully functional and implements all of the coordination driver API(s). It stores data into zookeeper using znodes and msgpack encoded values.
Exceptions¶
-
class
tooz.coordination.
ToozError
(message, cause=None)[source]¶ Bases:
exceptions.Exception
Exception raised when an internal error occurs, for instance in case of server internal error.
Variables: cause – the cause of the exception being raised, when not none this will itself be an exception instance, this is useful for creating a chain of exceptions for versions of python where this is not yet implemented/supported natively. -
__weakref__
¶ list of weak references to the object (if defined)
-
-
class
tooz.coordination.
ToozConnectionError
(message, cause=None)[source]¶ Bases:
tooz.coordination.ToozError
Exception raised when the client cannot manage to connect to the server.
-
class
tooz.coordination.
OperationTimedOut
(message, cause=None)[source]¶ Bases:
tooz.coordination.ToozError
Exception raised when an operation times out.
-
class
tooz.coordination.
GroupNotCreated
(group_id)[source]¶ Bases:
tooz.coordination.ToozError
Exception raised when the caller request a group which does not exist.
-
class
tooz.coordination.
GroupAlreadyExist
(group_id)[source]¶ Bases:
tooz.coordination.ToozError
Exception raised when the caller try to create a group which already exist.
-
class
tooz.coordination.
MemberAlreadyExist
(group_id, member_id)[source]¶ Bases:
tooz.coordination.ToozError
Exception raised when the caller try to join a group but a member with the same identifier belongs to that group.
-
class
tooz.coordination.
MemberNotJoined
(group_id, member_id)[source]¶ Bases:
tooz.coordination.ToozError
Exception raised when the caller try to access a member which does not belongs to the specified group.
-
class
tooz.coordination.
GroupNotEmpty
(group_id)[source]¶ Bases:
tooz.coordination.ToozError
Exception raised when the caller try to delete a group with members.
-
tooz.coordination.
raise_with_cause
(exc_cls, message, *args, **kwargs)[source]¶ Helper to raise + chain exceptions (when able) and associate a cause.
For internal usage only.
NOTE(harlowja): Since in py3.x exceptions can be chained (due to PEP 3134) we should try to raise the desired exception with the given cause.
Parameters: - exc_cls – the
ToozError
class to raise. - message – the text/str message that will be passed to the exceptions constructor as its first positional argument.
- args – any additional positional arguments to pass to the exceptions constructor.
- kwargs – any additional keyword arguments to pass to the exceptions constructor.
- exc_cls – the