We Keep Coding

Apache flink with AWS kinesis consume data-Getting java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer

Here is my code for connecting aws kinesis. When I'm trying to connect with FlinkKinesisConsumer it's throwing an error like Classnotfound exception.
import configs.AWSConfigConstants;
import com.amazonaws.services.kinesisanalytics.runtime.KinesisAnalyticsRuntime;
import java.io.IOException;
import java.util.Map;
import java.util.Properties;
import org.apache.flink.api.common.serialization.SimpleStringSchema;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.LocalStreamEnvironment;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer;
import org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants;
public class StreamingJob {
public static void main(String[] args) {
try {
final StreamExecutionEnvironment sEnv = StreamExecutionEnvironment.getExecutionEnvironment();
String region = "us-east-1";
String inputStreamName = "";
String accesskey = "";
String secretkey = "";
String initPosition = "LATEST";
String arn = "";
Properties consumerConfig = new Properties();
consumerConfig.put(AWSConfigConstants.AWS_REGION, region);
consumerConfig.put(AWSConfigConstants.AWS_ACCESS_KEY_ID, accesskey);
consumerConfig.put(AWSConfigConstants.AWS_SECRET_ACCESS_KEY, secretkey);
consumerConfig.put(AWSConfigConstants.AWS_ROLE_ARN, arn );
consumerConfig.put(AWSConfigConstants.STREAM_INITIAL_POSITION, initPosition);
System.out.println("Consume config properties:");
System.out.println(consumerConfig);
DataStream<String> kinesisInputStream = sEnv.addSource(new FlinkKinesisConsumer<>
(inputStreamName,new SimpleStringSchema(),consumerConfig));
System.out.println(kinesisInputStream);
sEnv.execute("Flink Streaming Processor");
} catch(Exception e) {
System.out.println(e);
}
}
I'm getting this below error
Output Screenshot
Thanks in advance for your help.

You need to include the appropriate version of the connector in your build. Something like this, where 2.11 is the scala version, and 1.13.2 is the Flink version.
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kinesis_2.11</artifactId>
<version>1.13.2</version>
</dependency>
For more info, see the docs.

Unable to consume Kafka messages using Apache Camel

I am unable to consume Kafka messages running in a local mode using Apache camel. I am not viewing any Kafka consumer properties when application starts. Below is my Kafka consumer code
import org.apache.camel.CamelContext;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.component.properties.PropertiesComponent;
import org.apache.camel.impl.DefaultCamelContext;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public final class MessageConsumerClient {
private static final Logger LOG = LoggerFactory.getLogger(MessageConsumerClient.class);
private MessageConsumerClient() {
}
public static void main(String[] args) throws Exception {
LOG.info("About to run Kafka-camel integration...");
CamelContext camelContext = new DefaultCamelContext();
// Add route to send messages to Kafka
camelContext.addRoutes(new RouteBuilder() {
public void configure() {
PropertiesComponent pc = getContext().getComponent("properties", PropertiesComponent.class);
pc.setLocation("classpath:application.properties");
log.info("About to start route: Kafka Server -> Log ");
from("kafka:{{consumer.topic}}?brokers={{kafka.host}}:{{kafka.port}}"
+ "&maxPollRecords={{consumer.maxPollRecords}}" + "&consumersCount={{consumer.consumersCount}}"
+ "&seekTo={{consumer.seekTo}}" + "&groupId={{consumer.group}}").routeId("FromKafka")
.log("${body}");
}
});
camelContext.start();
Thread.sleep(5 * 60 * 1000);
camelContext.stop();
}
}
I am using the following dependencies in pom.xml:
1.Camel core - 2.21.1
2.Camel Kafka - 2.21.1
Pom.xml
<dependencies>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-core</artifactId>
<version>2.21.1</version>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-kafka</artifactId>
<version>2.21.1</version>
</dependency>
</dependencies>
Any help would be much appreciated since I am not able to view any logs when executed.

Error while sending message in Kafka

I am trying to make a producer in Kafka
My Maven pom.xml is
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.10</artifactId>
<version>0.9.0.1</version>
</dependency>
</dependencies>
My Producer Code is
import java.util.Properties;
import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;
public class KafkaProducer {
public static void main(String[] args) {
Properties props = new Properties();
props.put("zk.connect", "serverIp:2181");
props.put("serializer.class", "kafka.serializer.StringEncoder");
ProducerConfig config = new ProducerConfig(props);
Producer<String, String> producer = new Producer<String, String>(config);
producer.send(new KeyedMessage<String, String>("TestTopic", "Message1"));
}
While executing this code, I am getting this error
Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Missing required property 'metadata.broker.list'
at scala.Predef$.require(Predef.scala:233)
at kafka.utils.VerifiableProperties.getString(VerifiableProperties.scala:177)
at kafka.producer.ProducerConfig.<init>(ProducerConfig.scala:66)
at kafka.producer.ProducerConfig.<init>(ProducerConfig.scala:56)
at kafkaPOC.KafkaProducer.main(KafkaProducer.java:16)
Can anyone please help me solve this exception.
Thanks in Advance.!!!

You are missing metadata.broker.list property.
props.put("metadata.broker.list", "<Kafka broker IP address>:9092");

ObjectMapper cannot be resolved to a type

I have troubles getting JSON to work. ObjectMapper cannot be resolved. Library is imported correctly.
import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import net.sf.json.JSONException;
import net.sf.json.util.*;
import com.fasterxml.jackson.*;
public class Json {
private static final String jsonFilePath = "C:\\Users\\Juergen\\Desktop\\filesForExamples\\mapExample.json";
public static void objectToJSON(HashMap<String, Mat> map) {
//Map<String, Object> mapObject = new HashMap<String, Object>();
ObjectMapper mapper = new ObjectMapper();
try {
objectMapper.writeValue(new File(jsonFilePath), map);
} catch (IOException e) {
e.printStackTrace();
}
}
}

If you are using **maven** project then add the following in the `POM.xml`
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.12.1</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.12.1</version>
</dependency>
But if you are using a **simple java** project then you need to add the following jars in your class path:
jackson-core-2.1.X,
jackson-databind-2.1.X

Add import com.fasterxml.jackson.databind.ObjectMapper; to your project.

Connecting to Solr 1.4 with SolrJ client connector 3.6

How do I connect to Solr 1.4 search server configured with Basic Auth using the SolrJ 3.6 connector?

Based on the instructions on using SolrJ :
Fetch relevant dependencies for e.g. using maven:
<dependency>
<groupId>org.apache.solr</groupId>
<artifactId>solr-solrj</artifactId>
<version>3.6.1</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
<version>4.2.2</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpmime</artifactId>
<version>4.1.2</version>
</dependency>
A sample application connecting to Solr 1.4 using SolrJ 3.6 with the latest Apache Http Components http client to negotiate Basic Auth:
import java.io.IOException;
import java.util.Iterator;
import org.apache.http.HttpException;
import org.apache.http.HttpRequest;
import org.apache.http.HttpRequestInterceptor;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.HttpClient;
import org.apache.http.impl.auth.BasicScheme;
import org.apache.http.impl.client.AbstractHttpClient;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.protocol.HttpContext;
import org.apache.solr.client.solrj.SolrQuery;
import org.apache.solr.client.solrj.SolrServer;
import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.impl.HttpSolrServer;
import org.apache.solr.client.solrj.impl.XMLResponseParser;
import org.apache.solr.client.solrj.response.QueryResponse;
import org.apache.solr.client.solrj.response.UpdateResponse;
import org.apache.solr.common.SolrDocument;
import org.apache.solr.common.SolrDocumentList;
import org.apache.solr.common.SolrInputDocument;
/**
* Sample app
*
*/
public class App
{
public static void main( String[] args ) throws SolrServerException, IOException
{
String url = "https://localhost:8080/solr/";
String httpAuthUser = "solr_admin";
String httpAuthPass= "somePassword";
// Configure latests Apache Http Components http client
HttpClient httpClient = new DefaultHttpClient();
if (httpAuthUser != null && httpAuthPass != null) {
AbstractHttpClient client = (AbstractHttpClient) httpClient;
App a = new App();
client.addRequestInterceptor(a.new PreEmptiveBasicAuthenticator(httpAuthUser, httpAuthPass));
}
// Configure XMLResponseParser as standard javabin parser does not work with 1.4
SolrServer solr = new HttpSolrServer(url, httpClient, new XMLResponseParser());
// Test adding some data
SolrInputDocument document = new SolrInputDocument();
document.addField("id", "552199");
document.addField("name", "Gouda cheese wheel");
document.addField("price", "49.99");
UpdateResponse uresponse = solr.add(document);
System.out.println("UpdateResponse"+uresponse.getStatus());
solr.commit();
// Query for the data just added
SolrQuery parameters = new SolrQuery();
parameters.set("q", "*");
QueryResponse response = solr.query(parameters);
SolrDocumentList list = response.getResults();
Iterator<SolrDocument> si = list.iterator();
System.out.println("Solr document"+list.getNumFound());
while(si.hasNext()){
System.out.println("Solr document"+si.next().toString());
}
}
protected class PreEmptiveBasicAuthenticator implements HttpRequestInterceptor {
private final UsernamePasswordCredentials credentials;
public PreEmptiveBasicAuthenticator(String user, String pass) {
credentials = new UsernamePasswordCredentials(user, pass);
}
public void process(HttpRequest request, HttpContext context)
throws HttpException, IOException {
request.addHeader(BasicScheme.authenticate(credentials,"US-ASCII",false));
}
}
}