We Keep Coding

InvalidParameterException on SNS topic using java

While trying to send message to AWS SNS topic using com.amazonaws.services.sns java module, I am stuck on following error:
shaded.com.amazonaws.services.sns.model.InvalidParameterException: Invalid parameter: Message too long (Service: AmazonSNS; Status Code: 400; Error Code: InvalidParameter; Request ID: 3b01ce49-a37d-5aba-bec2-9ab9d5446aea)
at shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1587)
at shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1257)
at shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029)
at shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741)
at shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715)
at shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697)
at shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665)
at shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647)
at shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511)
at shaded.com.amazonaws.services.sns.AmazonSNSClient.doInvoke(AmazonSNSClient.java:2270)
at shaded.com.amazonaws.services.sns.AmazonSNSClient.invoke(AmazonSNSClient.java:2246)
at shaded.com.amazonaws.services.sns.AmazonSNSClient.executePublish(AmazonSNSClient.java:1698)
at shaded.com.amazonaws.services.sns.AmazonSNSClient.publish(AmazonSNSClient.java:1675)
Following is the AmazonSNS helper Class. This class manages the client creation and publishing message to SNS topic.
import java.io.Serializable;
import com.amazonaws.services.sns.AmazonSNS;
import com.amazonaws.services.sns.AmazonSNSClientBuilder;
import com.amazonaws.services.sns.model.PublishRequest;
import com.amazonaws.services.sns.model.PublishResult;
public class AWSSNS implements Serializable {
private static final long serialVersionUID = -4175291946259141176L;
protected AmazonSNS client;
public AWSSNS(){
this.client=AmazonSNSClientBuilder.standard().withRegion("us-west-2").build();
}
public AWSSNS(AmazonSNS client) {
this.client=client;
}
public AmazonSNS getSnsClient(){
return this.client;
}
public void setSqsClient(AmazonSNS client){
this.client = client;
}
public boolean sendMessages(String topicArn, String messageBody){
PublishRequest publishRequest = new PublishRequest(topicArn, messageBody);
PublishResult publishResult = this.client.publish(publishRequest);
if(publishResult != null && publishResult.getMessageId() != null){
return true;
}
else{
return false;
}
}
}
Following is the code snippet from where the amazonSNS helper class is being called.It does nothing but create a message of String dataType and send it forward along with the topicARN.
HashMap<String, String> variable_a = new HashMap<String, String>();
Gson gson = new Gson();
for (Object_a revoke : Object_a) {
Object_a operation = someMethod1(revoke);
String serializedOperation = gson.toJson(operation);
variable_a.put(revoke.someMethod2(), serializedOperation);
String message = gson.toJson(variable_a);
LOG.info(String.format("SNS message: %s", message));
this.awsSNS.sendMessages(topicARN, message);
}
So basically the error is thrown from inside the sendMessage.

Found the solution to the problem.
AWS SNS topic has a fixed maximum size. So, on publishing messages of larger size than the maximum size would result in invalidParameterException with message "message too long".
My message was more than that size and that was the reason for the error. I shredded the message until the size came under max size.

Get all kafka messages in a queue and stop streaming in java

I need to execute a Job at night that is going to get all the messages in a kafka queue and execute a process with them. I'm able to get the messages but the kafka stream is waiting for more messages and I'm not able to continue with my process. I have the following code:
...
private ConsumerConnector consumerConnector;
private final static String TOPIC = "test";
public MessageStreamConsumer() {
Properties properties = new Properties();
properties.put("zookeeper.connect", "localhost:2181");
properties.put("group.id", "test-group");
ConsumerConfig consumerConfig = new ConsumerConfig(properties);
consumerConnector = Consumer.createJavaConsumerConnector(consumerConfig);
}
public List<String> getMessages() {
Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
topicCountMap.put(TOPIC, new Integer(1));
Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumerConnector
.createMessageStreams(topicCountMap);
KafkaStream<byte[], byte[]> stream = consumerMap.get(TOPIC).get(0);
ConsumerIterator<byte[], byte[]> it = stream.iterator();
List<String> messages = new ArrayList<>();
while (it.hasNext())
messages.add(new String(it.next().message()));
return messages;
}
The code is able to get the messages but when it process the last message it stays in the line:
while (it.hasNext())
The question is, how can i get all the messages from the kafka, stop the stream and continue with my other tasks.
I hope you can help me
Thanks

It seems that kafka stream does not support to consume from beginning.
You could create a native kafka consumer and set auto.offset.reset to earliest, then it will consume message from beginning.

Something like this may work. Basically the idea is to use a Kafka Consumer and poll until you get some record and then stop when you get an empty batch.
package kafka.examples;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Calendar;
import java.util.Collections;
import java.util.Date;
import java.util.Properties;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.atomic.AtomicBoolean;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
public class Consumer1 extends Thread
{
private final KafkaConsumer<Integer, String> consumer;
private final String topic;
private final DateFormat df;
private final String logTag;
private boolean noMoreData = false;
private boolean gotData = false;
private int messagesReceived = 0;
AtomicBoolean isRunning = new AtomicBoolean(true);
CountDownLatch shutdownLatch = new CountDownLatch(1);
public Consumer1(Properties props)
{
logTag = "Consumer1";
consumer = new KafkaConsumer<>(props);
this.topic = props.getProperty("topic");
this.df = new SimpleDateFormat("HH:mm:ss");
consumer.subscribe(Collections.singletonList(this.topic));
}
public void getMessages() {
System.out.println("Getting messages...");
while (noMoreData == false) {
//System.out.println(logTag + ": Doing work...");
ConsumerRecords<Integer, String> records = consumer.poll(1000);
Date now = Calendar.getInstance().getTime();
int recordsCount = records.count();
messagesReceived += recordsCount;
System.out.println("recordsCount: " + recordsCount);
if (recordsCount > 0) {
gotData = true;
}
if (gotData && recordsCount == 0) {
noMoreData = true;
}
for (ConsumerRecord<Integer, String> record : records) {
int kafkaKey = record.key();
String kafkaValue = record.value();
System.out.println(this.df.format(now) + " " + logTag + ":" +
" Received: {" + kafkaKey + ":" + kafkaValue + "}" +
", partition(" + record.partition() + ")" +
", offset(" + record.offset() + ")");
}
}
System.out.println("Received " + messagesReceived + " messages");
}
public void processMessages() {
System.out.println("Processing messages...");
}
public void run() {
getMessages();
processMessages();
}
}

I'm currently developing with Kafka 0.10.0.1 and found mixed information regarding the use of consumer property auto.offset.reset so I've done some experiments to figure out what actually happens.
Based on those, I now understand it this way: when you set property:
auto.offset.reset=earliest
this positions the consumer to EITHER the first available message in the partitions assigned (when no commits have been made on the paritions) OR it positions the consumer at the last committed partition offsets (notice that you always commit last read offset + 1 or else you'll be re-reading the last committed message on each restart of your consumer)
Alternatively you do not set auto.offset.reset which means default value of 'latest' will be used.
In that case you do not receive any old messages on connecting the consumer - only messages published to the topic after connecting the consumer will be received.
As a conclusion - if you want to ensure to receive all available messages for a certain topic and assigned partitions you'll have to call seekToBeginning().
It seems advised to call poll(0L) first to ensure your consumer gets partitions assigned (or implement your code in the ConsumerRebalanceListener!), then seek each of the assigned partitions to 'beginning':
kafkaConsumer.poll(0L);
kafkaConsumer.seekToBeginning(kafkaConsumer.assignment());

Play2.5 Java WebSockets

Play 2.5 Highlights states
Better control over WebSocket frames
The Play 2.5 WebSocket API gives you direct control over WebSocket frames. You can now send and receive binary, text, ping, pong and close frames. If you don’t want to worry about this level of detail, Play will still automatically convert your JSON or XML data into the right kind of frame.
However
https://www.playframework.com/documentation/2.5.x/JavaWebSockets has examples around LegacyWebSocket which is deprecated
What is the recommended API/pattern for Java WebSockets? Is using
LegacyWebSocket the only option for java websockets?
Are there any examples using new Message types ping/pong to implement a heartbeat?

The official documentation on this is disappointingly very sparse. Perhaps in Play 2.6 we'll see an update to this. However, I will provide an example below on how to configure a chat websocket in Play 2.5, just to help out those in need.
Setup
AController.java
#Inject
private Materializer materializer;
private ActorRef chatSocketRouter;
#Inject
public AController(#Named("chatSocketRouter") ActorRef chatInjectedActor) {
this.chatSocketRouter = chatInjectedActor;
}
// Make a chat websocket for a user
public WebSocket chatSocket() {
return WebSocket.Json.acceptOrResult(request -> {
String authToken = getAuthToken();
// Checking of token
if (authToken == null) {
return forbiddenResult("No [authToken] supplied.");
}
// Could we find the token in the database?
final AuthToken token = AuthToken.findByToken(authToken);
if (token == null) {
return forbiddenResult("Could not find [authToken] in DB. Login again.");
}
User user = token.getUser();
if (user == null) {
return forbiddenResult("You are not logged in to view this stream.");
}
Long userId = user.getId();
// Create a function to be run when we initialise a flow.
// A flow basically links actors together.
AbstractFunction1<ActorRef, Props> getWebSocketActor = new AbstractFunction1<ActorRef, Props>() {
#Override
public Props apply(ActorRef connectionProperties) {
// We use the ActorRef provided in the param above to make some properties.
// An ActorRef is a fancy word for thread reference.
// The WebSocketActor manages the web socket connection for one user.
// WebSocketActor.props() means "make one thread (from the WebSocketActor) and return the properties on how to reference it".
// The resulting Props basically state how to construct that thread.
Props properties = ChatSocketActor.props(connectionProperties, chatSocketRouter, userId);
// We can have many connections per user. So we need many ActorRefs (threads) per user. As you can see from the code below, we do exactly that. We have an object called
// chatSocketRouter which holds a Map of userIds -> connectionsThreads and we "tell"
// it a lightweight object (UserMessage) that is made up of this connecting user's ID and the connection.
// As stated above, Props are basically a way of describing an Actor, or dumbed-down, a thread.
// In this line, we are using the Props above to
// reference the ActorRef we've just created above
ActorRef anotherUserDevice = actorSystem.actorOf(properties);
// Create a lightweight object...
UserMessage routeThisUser = new UserMessage(userId, anotherUserDevice);
// ... to tell the thread that has our Map that we have a new connection
// from a user.
chatSocketRouter.tell(routeThisUser, ActorRef.noSender());
// We return the properties to the thread that will be managing this user's connection
return properties;
}
};
final Flow<JsonNode, JsonNode, ?> jsonNodeFlow =
ActorFlow.<JsonNode, JsonNode>actorRef(getWebSocketActor,
100,
OverflowStrategy.dropTail(),
actorSystem,
materializer).asJava();
final F.Either<Result, Flow<JsonNode, JsonNode, ?>> right = F.Either.Right(jsonNodeFlow);
return CompletableFuture.completedFuture(right);
});
}
// Return this whenever we want to reject a
// user from connecting to a websocket
private CompletionStage<F.Either<Result, Flow<JsonNode, JsonNode, ?>>> forbiddenResult(String msg) {
final Result forbidden = Results.forbidden(msg);
final F.Either<Result, Flow<JsonNode, JsonNode, ?>> left = F.Either.Left(forbidden);
return CompletableFuture.completedFuture(left);
}
ChatSocketActor.java
public class ChatSocketActor extends UntypedActor {
private final ActorRef out;
private final Long userId;
private ActorRef chatSocketRouter;
public ChatSocketActor(ActorRef out, ActorRef chatSocketRouter, Long userId) {
this.out = out;
this.userId = userId;
this.chatSocketRouter = chatSocketRouter;
}
public static Props props(ActorRef out, ActorRef chatSocketRouter, Long userId) {
return Props.create(ChatSocketActor.class, out, chatSocketRouter, userId);
}
// Add methods here handling each chat connection...
}
ChatSocketRouter.java
public class ChatSocketRouter extends UntypedActor {
public ChatSocketRouter() {}
// Stores userIds to websockets
private final HashMap<Long, List<ActorRef>> senders = new HashMap<>();
private void addSender(Long userId, ActorRef actorRef){
if (senders.containsKey(userId)) {
final List<ActorRef> actors = senders.get(userId);
actors.add(actorRef);
senders.replace(userId, actors);
} else {
List<ActorRef> l = new ArrayList<>();
l.add(actorRef);
senders.put(userId, l);
}
}
private void removeSender(ActorRef actorRef){
for (List<ActorRef> refs : senders.values()) {
refs.remove(actorRef);
}
}
#Override
public void onReceive(Object message) throws Exception {
ActorRef sender = getSender();
// Handle messages sent to this 'router' here
if (message instanceof UserMessage) {
UserMessage userMessage = (UserMessage) message;
addSender(userMessage.userId, userMessage.actorRef);
// Watch sender so we can detect when they die.
getContext().watch(sender);
} else if (message instanceof Terminated) {
// One of our watched senders has died.
removeSender(sender);
} else {
unhandled(message);
}
}
}
Example
Now whenever you want to send a client with a websocket connection a message you can do something like:
ChatSenderController.java
private ActorRef chatSocketRouter;
#Inject
public ChatSenderController(#Named("chatSocketRouter") ActorRef chatInjectedActor) {
this.chatSocketRouter = chatInjectedActor;
}
public static void sendMessage(Long sendToId) {
// E.g. send the chat router a message that says hi
chatSocketRouter.tell(new Message(sendToId, "Hi"));
}
ChatSocketRouter.java
#Override
public void onReceive(Object message) throws Exception {
// ...
if (message instanceof Message) {
Message messageToSend = (Message) message;
// Loop through the list above and send the message to
// each connection. For example...
for (ActorRef wsConnection : senders.get(messageToSend.getSendToId())) {
// Send "Hi" to each of the other client's
// connected sessions
wsConnection.tell(messageToSend.getMessage());
}
}
// ...
}
Again, I wrote the above to help out those in need. After scouring the web I could not find a reasonable and simple example. There is an open issue for this exact topic. There are also some examples online but none of them were easy to follow. Akka has some great documentation but mixing it in with Play was a tough mental task.
Please help improve this answer if you see anything that is amiss.

How to return data from a webservice in chunks in java

Hello I am using a webservice which returns a output upon completion of code execution. Is it possible that webservice may return the status in chunks like custom strings: Test Started, Test In Progress, Test Completed etc.
What I need to do to achieve this. Here is my current code where I am expecting a json string as input, supplied json is parsed and further processing is being performed.
//Class
public class WebserviceClient
{
/** calling constructor to initialize logger */
Utils c = new Utils();
Logger logger = Logger.getLogger(WebserviceClient.class.getName());
#Path("/test")
#POST
#Consumes(MediaType.APPLICATION_JSON)
//#Produces(MediaType.APPLICATION_JSON)
public String processRequest(final String inputData)
{
String executionID = "NOT_FOUND" ;
String result = "";
try
{
/** creating a pool of threads to submit a task to a callable thread */
ExecutorService ex = Executors.newFixedThreadPool(5);
Future<String> futureObject = ex.submit(new Callable<String>() {
#Override
public String call() throws Exception
{
logger.info("Parsing Received Request: "+inputData);
String rID = new JSONObject(inputData).getString("id");
logger.info("Received Id: "+rID + " From Request: "+inputData);
if(new RunTest().isTestCompleted(rID))
{
return rID;
}
else
{
return "777";
}
}
});
result = futureObject.get();
if(futureObject.get()!=null)
{
ex.shutdown();
}
else{
logger.debug("call id: "+executionID +" result is not generated yet. ");
}
logger.info("call id && Result: "+result);
}
catch(Exception e)
{
logger.error("call id: "+executionID, e);
}
return result;
}
}

You need to do a continuous polling to the server at high frequency to achieve the current status.
For little more information have a look at the :
Continuous polling of output using spring ,rest and angular js
This includes design consideration of using WebSockets etc, but there is no straight forward solution that I'm aware of.

How to respond with the result of an actor call?

We are looking at using Akka-HTTP Java API - using Routing DSL.
It's not clear how to use the Routing functionality to respond to an HttpRequest; using an Untyped Akka Actor.
For example, upon matching a Route path, how do we hand off the request to a "handler" ActorRef, which will then respond with a HttpResponse in a asynchronous way?
A similar question was posted on Akka-User mailing list, but with no followup solutions as such - https://groups.google.com/d/msg/akka-user/qHe3Ko7EVvg/KC-aKz_o5aoJ.

This can be accomplished with a combination of the onComplete directive and the ask pattern.
In the below example the RequestHandlerActor actor is used to create a HttpResponse based on the HttpRequest. This Actor is asked from within the route.
I have never used Java for routing code so my response is in Scala.
import scala.concurrent.duration._
import akka.actor.ActorSystem
import akka.http.scaladsl.model.HttpResponse
import akka.http.scaladsl.model.HttpRequest
import akka.actor.Actor
import akka.http.scaladsl.server.Directives._
import akka.actor.Props
import akka.pattern.ask
import akka.util.Timeout
import scala.util.{Success, Failure}
import akka.http.scaladsl.model.StatusCodes.InternalServerError
class RequestHandlerActor extends Actor {
override def receive = {
case httpRequest : HttpRequest =>
sender() ! HttpResponse(entity = "actor responds nicely")
}
}
implicit val actorSystem = ActorSystem()
implicit val timeout = Timeout(5 seconds)
val requestRef = actorSystem actorOf Props[RequestHandlerActor]
val route =
extractRequest { request =>
onComplete((requestRef ? request).mapTo[HttpResponse]) {
case Success(response) => complete(response)
case Failure(ex) =>
complete((InternalServerError, s"Actor not playing nice: ${ex.getMessage}"))
}
}
This route can then be used passed into the bindAndHandle method like any other Flow.

I have been looking the solution to the same problem as described by the author of the question. Finally, I came up to the following Java code for route creation:
ActorRef ref = system.actorOf(Props.create(RequestHandlerActor.class));
return get(() -> route(
pathSingleSlash(() ->
extractRequest(httpRequest -> {
Timeout timeout = new Timeout(Duration.create(5, TimeUnit.SECONDS));
CompletionStage<HttpResponse> completionStage = PatternsCS.ask(ref, httpRequest, timeout)
.thenApplyAsync(HttpResponse.class::cast);
return completeWithFuture(completionStage);
})
))
);
And RequestHandlerActor is:
public class RequestHandlerActor extends UntypedActor {
#Override
public void onReceive(Object msg) {
if (msg instanceof HttpRequest) {
HttpResponse httpResponse = HttpResponse.create()
.withEntity(ContentTypes.TEXT_HTML_UTF8,
"<html><body>Hello world!</body></html>");
getSender().tell(httpResponse, getSelf());
} else {
unhandled(msg);
}
}
}