kafka.producer package¶
Submodules¶
kafka.producer.base module¶
- class kafka.producer.base.Producer(client, async=False, req_acks=1, ack_timeout=1000, codec=None, batch_send=False, batch_send_every_n=20, batch_send_every_t=20)[source]¶
Bases: object
Base class to be used by producers
Parameters: - client – The Kafka client instance to use
- async – If set to true, the messages are sent asynchronously via another thread (process). We will not wait for a response to these WARNING!!! current implementation of async producer does not guarantee message delivery. Use at your own risk! Or help us improve with a PR!
- req_acks – A value indicating the acknowledgements that the server must receive before responding to the request
- ack_timeout – Value (in milliseconds) indicating a timeout for waiting for an acknowledgement
- batch_send – If True, messages are send in batches
- batch_send_every_n – If set, messages are send in batches of this size
- batch_send_every_t – If set, messages are send after this timeout
- ACK_AFTER_CLUSTER_COMMIT = -1¶
- ACK_AFTER_LOCAL_WRITE = 1¶
- ACK_NOT_REQUIRED = 0¶
- DEFAULT_ACK_TIMEOUT = 1000¶
- send_messages(topic, partition, *msg)[source]¶
Helper method to send produce requests @param: topic, name of topic for produce request – type str @param: partition, partition number for produce request – type int @param: *msg, one or more message payloads – type bytes @returns: ResponseRequest returned by server raises on error
Note that msg type must be encoded to bytes by user. Passing unicode message will not work, for example you should encode before calling send_messages via something like unicode_message.encode(‘utf-8’)
All messages produced via this method will set the message ‘key’ to Null
kafka.producer.keyed module¶
- class kafka.producer.keyed.KeyedProducer(client, partitioner=None, async=False, req_acks=1, ack_timeout=1000, codec=None, batch_send=False, batch_send_every_n=20, batch_send_every_t=20)[source]¶
Bases: kafka.producer.base.Producer
A producer which distributes messages to partitions based on the key
Parameters: client – The kafka client instance
Keyword Arguments: - partitioner – A partitioner class that will be used to get the partition to send the message to. Must be derived from Partitioner
- async – If True, the messages are sent asynchronously via another thread (process). We will not wait for a response to these
- ack_timeout – Value (in milliseconds) indicating a timeout for waiting for an acknowledgement
- batch_send – If True, messages are send in batches
- batch_send_every_n – If set, messages are send in batches of this size
- batch_send_every_t – If set, messages are send after this timeout
kafka.producer.simple module¶
- class kafka.producer.simple.SimpleProducer(client, async=False, req_acks=1, ack_timeout=1000, codec=None, batch_send=False, batch_send_every_n=20, batch_send_every_t=20, random_start=True)[source]¶
Bases: kafka.producer.base.Producer
A simple, round-robin producer. Each message goes to exactly one partition
Parameters: client – The Kafka client instance to use
Keyword Arguments: - async – If True, the messages are sent asynchronously via another thread (process). We will not wait for a response to these
- req_acks – A value indicating the acknowledgements that the server must receive before responding to the request
- ack_timeout – Value (in milliseconds) indicating a timeout for waiting for an acknowledgement
- batch_send – If True, messages are send in batches
- batch_send_every_n – If set, messages are send in batches of this size
- batch_send_every_t – If set, messages are send after this timeout
- random_start – If true, randomize the initial partition which the the first message block will be published to, otherwise if false, the first message block will always publish to partition 0 before cycling through each partition
Module contents¶
- class kafka.producer.SimpleProducer(client, async=False, req_acks=1, ack_timeout=1000, codec=None, batch_send=False, batch_send_every_n=20, batch_send_every_t=20, random_start=True)¶
Bases: kafka.producer.base.Producer
A simple, round-robin producer. Each message goes to exactly one partition
Parameters: client – The Kafka client instance to use
Keyword Arguments: - async – If True, the messages are sent asynchronously via another thread (process). We will not wait for a response to these
- req_acks – A value indicating the acknowledgements that the server must receive before responding to the request
- ack_timeout – Value (in milliseconds) indicating a timeout for waiting for an acknowledgement
- batch_send – If True, messages are send in batches
- batch_send_every_n – If set, messages are send in batches of this size
- batch_send_every_t – If set, messages are send after this timeout
- random_start – If true, randomize the initial partition which the the first message block will be published to, otherwise if false, the first message block will always publish to partition 0 before cycling through each partition
- send_messages(topic, *msg)¶
- class kafka.producer.KeyedProducer(client, partitioner=None, async=False, req_acks=1, ack_timeout=1000, codec=None, batch_send=False, batch_send_every_n=20, batch_send_every_t=20)¶
Bases: kafka.producer.base.Producer
A producer which distributes messages to partitions based on the key
Parameters: client – The kafka client instance
Keyword Arguments: - partitioner – A partitioner class that will be used to get the partition to send the message to. Must be derived from Partitioner
- async – If True, the messages are sent asynchronously via another thread (process). We will not wait for a response to these
- ack_timeout – Value (in milliseconds) indicating a timeout for waiting for an acknowledgement
- batch_send – If True, messages are send in batches
- batch_send_every_n – If set, messages are send in batches of this size
- batch_send_every_t – If set, messages are send after this timeout
- send(topic, key, msg)¶
- send_messages(topic, key, *msg)¶