We Keep Coding

How to make kafka consumer stop retrying and crash when broker is not available?

Currently, when broker is not available, I see that java consumer will try to reconnect to broker for infinite amount of time(or at least I was not able to wait till it stops).
What I would like to have is an ability to have an exception thrown from poll() when broker is not available. Or maybe some other way, but I want my app to crash when broker is not available. It seems that it should be very easy to configure, but I'm unable to find info anywhere.
Here is an example. It'll always return 0 records from poll and will retry connection to broker forever.
package org.example;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import java.time.Duration;
import java.util.List;
import java.util.Properties;
public class Consumer {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9094");
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "test-consumer-group");
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
consumer.subscribe(List.of("game.journal"));
while (true) {
ConsumerRecords<String, String> recs = consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, String> rec : recs) {
System.out.printf("Recieved %s: %s", rec.key(), rec.value());
}
}
}
}
}
So how can I make consumer crash when broker is not available?

Read file using kafka file connector in java

I have written a very simple code in Java to read a file and send those records to Kafka topic. Everything is working as expected. But, instead of writing file , I want to use Kafka file connector. I did it in the past using REST proxy(curl) command but never tried in java. I need some help to do it.
I can see there is Kafka-connect api in Maven repository and I can add it in my pom.xml file. What should be my next step to integrate it in my java code.
My code to read file without Kafka connect :
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.util.Properties;
import java.util.Scanner;
public class SimpleProducer_ReadFile {
public static void main(String[] args) throws FileNotFoundException {
// System.out.println("Hello Kafka ");
// setting properties
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,StringSerializer.class.getName());
// create the producer
KafkaProducer<String, String> produce = new KafkaProducer<String, String>(props);
//reading file
File read = new File("C:\\Users\\\Desktop\\TestFile.txt");
Scanner scan = new Scanner(read);
while(scan.hasNextLine()){
String data = scan.nextLine();
System.out.println(data);
//create the producer record
ProducerRecord<String, String> record = new ProducerRecord<String, String>("test-topic",data);
//send data
produce.send(record);
}
//flush and close
produce.flush();
produce.close();
}
}

All you need is Kafka Connect and FileStreamSource connector that reads data from a file and sends it to Kafka.
In your case, the configuration should be
name=local-file-source
connector.class=FileStreamSource
tasks.max=1
file=/path/to/file.txt
topic=test-topic
Now the equivalent curl command would be:
curl -X POST \
-H "Content-Type: application/json" \
--data '{"name": "local-file-source", "config": {"connector.class":"FileStreamSource", "tasks.max":"1", "file":"path/to/file.txt", "topics":"test-topic" }}' http://localhost:8083/connectors
If you want to do this programmatically, simply send a POST request as above.

Write to kafka Topic based on the content on content of record using kafkastreams

I'm trying to write from a topic(parent) to another topic(child) in kafka based on the content of the records in the parent.
A sample record if i consume from the parent topic is {"date":{"string":"2017-03-20"},"time":{"string":"20:04:13:563"},"event_nr":1572470,"interface":"Transaction Manager","event_id":5001,"date_time":1490040253563,"entity":"Transaction Manager","state":0,"msg_param_1":{"string":"ISWSnk"},"msg_param_2":{"string":"Application startup"},"msg_param_3":null,"msg_param_4":null,"msg_param_5":null,"msg_param_6":null,"msg_param_7":null,"msg_param_8":null,"msg_param_9":null,"long_msg_param_1":null,"long_msg_param_2":null,"long_msg_param_3":null,"long_msg_param_4":null,"long_msg_param_5":null,"long_msg_param_6":null,"long_msg_param_7":null,"long_msg_param_8":null,"long_msg_param_9":null,"last_sent":{"long":1490040253563},"transmit_count":{"int":1},"team_id":null,"app_id":{"int":4},"logged_by_app_id":{"int":4},"entity_type":{"int":3},"binary_data":null}.
I'll like to use the value of entity to write to a topic of the same name as the value of entity(There's a fixed amount of values of entity so i can statically create that if it's difficult programmatically to dynamically create topics). I'm trying to use this
import org.apache.kafka.common.serialization.Serde;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.KeyValue;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KStreamBuilder;
import java.util.Properties;
public class entityDataLoader {
public static void main(final String[] args) throws Exception {
final Properties streamsConfiguration = new Properties();
streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, "map-function-lambda-example");
streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
streamsConfiguration.put(StreamsConfig.KEY_SERDE_CLASS_CONFIG, Serdes.ByteArray().getClass().getName());
streamsConfiguration.put(StreamsConfig.VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
// Set up serializers and deserializers, which we will use for overriding the default serdes
// specified above.
final Serde<String> stringSerde = Serdes.String();
final Serde<byte[]> byteArraySerde = Serdes.ByteArray();
// In the subsequent lines we define the processing topology of the Streams application.
final KStreamBuilder builder = new KStreamBuilder();
// Read the input Kafka topic into a KStream instance.
final KStream<byte[], String> textLines = builder.stream(byteArraySerde, stringSerde, "postilion-events");
String content = textLines.toString();
String entity = JSONExtractor.returnJSONValue(content, "entity");
System.out.println(entity);
textLines.to(entity);
final KafkaStreams streams = new KafkaStreams(builder, streamsConfiguration);
streams.cleanUp();
streams.start();
// Add shutdown hook to respond to SIGTERM and gracefully close Kafka Streams
Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
}
}
The content of content is org.apache.kafka.streams.kstream.internals.KStreamImpl#568db2f2 making it obvious that #KStream.toString() isn't the correct method to use to attempt to get the value of entity.
P.S. The JSONExtractor class is defined as
import org.json.simple.JSONObject;
import org.json.simple.parser.ParseException;
import org.json.simple.parser.JSONParser;
class JSONExtractor {
public static String returnJSONValue(String args, String value){
JSONParser parser = new JSONParser();
String app= null;
System.out.println(args);
try{
Object obj = parser.parse(args);
JSONObject JObj = (JSONObject)obj;
app= (String) JObj.get(value);
return app;
}
catch(ParseException pe){
System.out.println("No Object found");
System.out.println(pe);
}
return app;
}
}

You can use branch() to split your parent stream into "sub streams" and write each "sub stream" to one output topic (cf. http://docs.confluent.io/current/streams/developer-guide.html#stateless-transformations)
Your branch() must create a single "sub stream" for all you output topic, but because you know all you topics, this should not be a problem.
Also, for Kafka Streams, it's recommended to create all output topic before you start you application (cf. http://docs.confluent.io/current/streams/developer-guide.html#user-topics)

Having trouble connecting android with MQTT to broker

I am trying to connect to an Apollo broker, this code works perfectly when I use it alone in a normal java project, everything is exactly the the same except now its in an android project and i try to run it when i click a button from MainActivity.
I have a text box that gets updated to "1" before i try to connect MQttClient however the second .setT("2") does not get run so I think the problem is with client.connect(opts) as if i just do client.connect() the text box gets updated to "2" but since i need the username and password it from opts the rest does not run
Just started using MQTT learning as I go along. Thanks for any help.
package com.example.androidmqtt;
import org.eclipse.paho.client.mqttv3.MqttClient;
import org.eclipse.paho.client.mqttv3.MqttConnectOptions;
import org.eclipse.paho.client.mqttv3.MqttDeliveryToken;
import org.eclipse.paho.client.mqttv3.MqttException;
import org.eclipse.paho.client.mqttv3.MqttMessage;
import org.eclipse.paho.client.mqttv3.MqttTopic;
import org.eclipse.paho.client.mqttv3.internal.MemoryPersistence;
public class Service {
MqttClient client;
MemoryPersistence persistence = new MemoryPersistence();
public Service()throws Exception{}
public static void main(String[] args) throws Exception {
new Service().doDemo();
}
public void doDemo() {
try {
client = new MqttClient("tcp://10.1.10.1:1883", "testingMyMQTT", persistence);
MainActivity.setT("2");
MqttConnectOptions opts = new MqttConnectOptions();
opts.setUserName("nabi");
opts.setPassword("M4rk3".toCharArray());
opts.setKeepAliveInterval(480);
MainActivity.setT("1");//sets the txt1 in main view to 1 so i know whats going on
client.connect(opts);
MainActivity.setT("2");
MqttMessage msg = new MqttMessage("Works".getBytes());
msg.setRetained(true);
msg.setQos(1);
MainActivity.setT("its working");
MqttTopic topic = client.getTopic("Android/Test");
MqttDeliveryToken token = topic.publish(msg);
} catch (MqttException e) {
e.printStackTrace();
}
}
}

Verify that android can connect to your local network 10.1.10.1 and if so, check the logs of the Apollo broker.

Java on ServiceBus for Windows Server- Options?

What are the options available to develop Java applications using Service Bus for Windows?
Java Message Broker API - This need ACS to work with, which SB for Win doesnt support.
AMQP - This doesnt seem to work on SB for Windows, I keep getting error
org.apache.qpid.amqp_1_0.client.Sender$SenderCreationException: Peer did not create remote endpoint for link, target:
While the same code works with Azure SB. So AMQP on SB for Windows seems to be not fully working?
Correct me if I have missed something?
Update
To test AMQP on local machine, this is what I did
Installed Service bus 1.1 on my local machine
Took the sample mentioned here http://www.windowsazure.com/en-us/develop/java/how-to-guides/service-bus-amqp/
Created a new namespace on my local machine
Specified the following connection string in servicebus.properties (which is correctly referred in the code
connectionfactory.SBCF = amqps://<username>:<password>#<MachineName>:5671/StringAnalyzerNS/
queue.QUEUE = queue1
Code is updated with certificates.
At runtime I get this error
javax.jms.JMSException: Peer did not create remote endpoint for link, target: queue1
at org.apache.qpid.amqp_1_0.jms.impl.MessageProducerImpl.<init>(MessageProducerImpl.java:77)
at org.apache.qpid.amqp_1_0.jms.impl.SessionImpl.createProducer(SessionImpl.java:348)
at org.apache.qpid.amqp_1_0.jms.impl.SessionImpl.createProducer(SessionImpl.java:63)
at com.stringcompany.Analyzer.SimpleSenderReceiver.<init>(SimpleSenderReceiver.java:70)
at com.stringcompany.Analyzer.SimpleSenderReceiver.main(SimpleSenderReceiver.java:95)
Caused by: org.apache.qpid.amqp_1_0.client.Sender$SenderCreationException: Peer did not create remote endpoint for link, target: queue1
at org.apache.qpid.amqp_1_0.client.Sender.<init>(Sender.java:171)
at org.apache.qpid.amqp_1_0.client.Sender.<init>(Sender.java:104)
at org.apache.qpid.amqp_1_0.client.Sender.<init>(Sender.java:97)
at org.apache.qpid.amqp_1_0.client.Sender.<init>(Sender.java:83)
at org.apache.qpid.amqp_1_0.client.Sender.<init>(Sender.java:69)
at org.apache.qpid.amqp_1_0.client.Sender.<init>(Sender.java:63)
at org.apache.qpid.amqp_1_0.client.Session.createSender(Session.java:74)
at org.apache.qpid.amqp_1_0.client.Session.createSender(Session.java:66)
at org.apache.qpid.amqp_1_0.jms.impl.MessageProducerImpl.<init>(MessageProducerImpl.java:72)
... 4 more
javax.jms.JMSException: Session remotely closed
With the same code If I point to Azure service bus by setting the SB namespace and queue like below
connectionfactory.SBCF = amqps://<Policy name>:<Sec. Key>#<ns>.servicebus.windows.net
queue.QUEUE = testq
This works, messages are exchanged.
Here is the code if someone wants to try it
package com.stringcompany.Analyzer;
//SimpleSenderReceiver.java
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.util.Hashtable;
import java.util.Random;
import javax.jms.Connection;
import javax.jms.ConnectionFactory;
import javax.jms.Destination;
import javax.jms.ExceptionListener;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageConsumer;
import javax.jms.MessageListener;
import javax.jms.MessageProducer;
import javax.jms.Queue;
import javax.jms.Session;
import javax.jms.TextMessage;
import javax.naming.Context;
import javax.naming.InitialContext;
public class SimpleSenderReceiver implements MessageListener {
private static boolean runReceiver = true;
private Connection connection;
private Session sendSession;
private Session receiveSession;
private MessageProducer sender;
private MessageConsumer receiver;
private static Random randomGenerator = new Random();
public SimpleSenderReceiver() throws Exception {
// Configure JNDI environment
Hashtable<String, String> env = new Hashtable<String, String>();
env.put(Context.INITIAL_CONTEXT_FACTORY,
"org.apache.qpid.amqp_1_0.jms.jndi.PropertiesFileInitialContextFactory");
env.put(Context.PROVIDER_URL, "D:\\Java\\Azure\\workspace\\Analyzer\\src\\main\\resources\\servicebus.properties");
Context context = new InitialContext(env);
// Lookup ConnectionFactory and Queue
ConnectionFactory cf = (ConnectionFactory) context.lookup("SBCF");
System.out.println("cf:"+cf);
// Create Connection
connection = cf.createConnection();
System.out.println("connection :"+connection);
connection.setExceptionListener(new ExceptionListener() {
public void onException(JMSException arg0) {
System.err.println(arg0);
}
});
connection.start();
// Create sender-side Session and MessageProducer
sendSession = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
System.out.println("Session open");
Destination queue = (Destination) context.lookup("QUEUE");
System.out.println("queue:"+queue);
sender = sendSession.createProducer(queue);
Queue q=(Queue) queue;
System.out.println(sender.getDestination());
System.out.println("sender:"+sender);
if (runReceiver) {
System.out.println("Waitng for new message");
// Create receiver-side Session, MessageConsumer,and MessageListener
receiveSession = connection.createSession(false,
Session.CLIENT_ACKNOWLEDGE);
receiver = receiveSession.createConsumer(queue);
receiver.setMessageListener(this);
connection.start();
}
}
public static void main(String[] args) {
try {
if ((args.length > 0) && args[0].equalsIgnoreCase("sendonly")) {
runReceiver = false;
}
//System.setProperty("javax.net.debug","ssl");
System.setProperty("javax.net.ssl.trustStore","D:\\Java\\Azure\\workspace\\Analyzer\\src\\main\\resources\\SBKeystore.keystore");
System.setProperty("log4j.configuration","D:\\Java\\Azure\\workspace\\Analyzer\\src\\main\\resources\\log4j.properties");
SimpleSenderReceiver simpleSenderReceiver = new SimpleSenderReceiver();
System.out
.println("Press [enter] to send a message. Type 'exit' + [enter] to quit.");
BufferedReader commandLine = new java.io.BufferedReader(
new InputStreamReader(System.in));
while (true) {
String s = "Message";//commandLine.readLine();
if (s.equalsIgnoreCase("exit")) {
simpleSenderReceiver.close();
System.exit(0);
} else {
simpleSenderReceiver.sendMessage();
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
private void sendMessage() throws JMSException {
TextMessage message = sendSession.createTextMessage();
message.setText("Test AMQP message from JMS");
long randomMessageID = randomGenerator.nextLong() >>> 1;
message.setJMSMessageID("ID:" + randomMessageID);
sender.send(message);
System.out.println("Sent message with JMSMessageID = "
+ message.getJMSMessageID());
}
public void close() throws JMSException {
connection.close();
}
public void onMessage(Message message) {
try {
System.out.println("Received message with JMSMessageID = "
+ message.getJMSMessageID());
message.acknowledge();
} catch (Exception e) {
e.printStackTrace();
}
}
}

Hi we had the same problems and thankfully MS updated their documentation to show how to do this correctly. :
http://msdn.microsoft.com/en-us/library/dn574799.aspx

The simplest answer to the question is as you should URL Encode the SASPolicyKey.
connectionfactory.SBCF = amqps://[SASPolicyName]:[SASPolicyKey]#[namespace].servicebus.windows.net
Where SASPolicyKey should be URL-Encoded.

AMQP 1.0 is supported with Service Bus 1.1 for windows server. Basically there are two differences between the cloud and on-prem usage of AMQP in ServiceBus:
1. Addressing: You will need to build an AMQP connection strings (and will need DNS in case you're looking for HA)
2. Authentication: You will need to use domain joined accounts as ACS is not there on-prem. You will also need to distribute your SB certificate to your clients.

Ok, I have sorted the first issue (Java Message Broker API not supporting SAS endpoint), by writing a wrapper which will seamlessly work with existing API. You can get the library from this GitHub repository. With this, I can develop/test my Java application on local service bus environment and host it on Azure / On-Premise Service Bus farm.
https://github.com/Dhana-Krishnasamy/ServiceBusForWindows-SASWrapper

The sender and receiver Queues you will have to configure differently. Here is an example of my working configuration (servicebus.properties):
connectionfactory.SBCF = amqps://$PolicyName:$UrlEncodedKey#$Your-EventHub-NamespaceName.servicebus.windows.net
queue.EventHubSender=$YourEventHubName
queue.EventHubReceiver=$YourEventHubName/ConsumerGroups/$YourConsumerGroupName/Partitions/1
Replace appropriately your own '$' items in there.
The Shared Policy Key has to be URL encoded.
Make sure that your sender will reference the 'EventHubSender' defined in this config and the receiver will reference the 'EventHubReciever'.

Grab the Azure Java SDK from http://www.windowsazure.com/en-us/develop/java/ and then follow this guide: http://www.windowsazure.com/en-us/develop/java/how-to-guides/service-bus-queues/