How-To : Write a Kafka Producer using Twitter Stream ( Twitter HBC Client)

Twitter opensourced it’s Hosebird client (hbc) , a robust Java HTTP library for consuming Twitter’s Streaming API . In this post, I am going to present a demo of how we can use hbc to create a Kafka twitter stream producer , which tracks few terms on twitter statuses and produces a kafka stream out of it, which can be utilized later for counting the terms, or putting that data from Kafka to Storm (Kafka-Storm pipeline ) or HDFS ( as we will see in next post for using Camus API ).

You can download and run complete Sample here

Requirements :

  • Apache Kafka 0.8
  • Twitter Developer account ( for API Key, Secret etc.)
  • Apache Zookeeper ( required for Kafka)
  • Oracle JDK 1.7 (64 bit )

Build Environment :

  • Eclipse
  • Apache Maven 2/3

How to generate Twitter API Keys using Developer Account ?

  1. Go to and log in, if necessary
  2. Enter your Application Name, Description and your website address. You can leave the callback URL empty.
  3. Accept the TOS.
  4. Submit the form by clicking the Create your Twitter Application
  5. Copy the consumer key (API key) and consumer secret from the screen into your application.
  6. After creating your Twitter Application, you have to give the access to your Twitter Account to use this Application. To do this, click the Create my Access Token.
  7. Now you will have Consumer Key, Consumer Secret, Acess token, Access Token Secret to be used in streaming API calls.

Steps to run the Sample :

  1. Start Zookeeper server in Kafka using following script in your kafka installation folder –
$bin/ config/ &

and, verify if it is running on default port 2181 using –

$netstat -anlp | grep 2181

2. Start Kafka server using following script –

$bin/ config/  &

and, verify if it is running on default port 9092

$netstat -anlp | grep 9092

3. Now, when we are all set with Kafka running ready to accept messages on any dynamically created topic ( default setting ), we will create a Kafka Producer , which makes use of hbc client API to get twitter stream for tracking terms and puts on topic named as “twitter-topic” .

  • First, we need to give maven dependencies for hbc-core for latest version and some other dependencies needed for Kafka –
<artifactId>hbc-core</artifactId> <!-- or hbc-twitter4j -->
<version>2.2.0</version> <!-- or whatever the latest version is -->
  • Then, we need to set properties to configure our Kafka Producer to publish messages to topic –
    private static final String topic = "twitter-topic";
  • Properties properties = new Properties();
    		properties.put("", "localhost:9092");
    		properties.put("serializer.class", "kafka.serializer.StringEncoder");
  • Set up a StatusFilterEndpoint , which will setup track terms to be tracked on recent status messages, as in the example, twitterapi and #AAPSweep ( change these to term you want to track) –
  • BlockingQueue<String> queue = new LinkedBlockingQueue<String>(10000);
    		StatusesFilterEndpoint endpoint = new StatusesFilterEndpoint();
    		// add some track terms
  • Provide authentication parameters for OAuth ( we are getting them using commandline parameters for this program ) for using twitter that we generated earlier and create the client using endpoint and auth –
Authentication auth = new OAuth1(consumerKey, consumerSecret, token,
		// Authentication auth = new BasicAuth(username, password);
		// Create a new BasicClient. By default gzip is enabled.
		Client client = new ClientBuilder().hosts(Constants.STREAM_HOST)
				.processor(new StringDelimitedProcessor(queue)).build();
  • Last step, connect to client, fetch messages from queue and send through Kafka Producer –
// Establish a connection
		// Do whatever needs to be done with messages
		for (int msgRead = 0; msgRead < 1000; msgRead++) {
			KeyedMessage<String, String> message = null;
			try {
				message = new KeyedMessage<String, String>(topic, queue.take());
			} catch (InterruptedException e) {

To run the complete example run class as a Java Application in your favourite IDE.

Verify the Topic and Messages

  • Check if topic is there using –
    $bin/ --zookeeper localhost:2181
  • Consume messages on topic twitter-topic to verify the incoming message stream.
    $bin/ --zookeeper localhost:2181 --topic test --from-beginning

    According to chao’s theory, you might face some issues in some of the steps mentioned above, but if you have reached this far, you have done an amazing job !! 🙂

Happy Learning !!

References :-




Interesting Reads –

Multithreaded Mappers in Mapreduce

You may also like...

%d bloggers like this: