We Keep Coding

Java: Azure Service Bus Queue Receiving messsages with sessions

I'm writing code in java (using Azure SDK for Java), I have a Service bus queue that contains sessionful messages. I want to receive those messages and process them to another place.
I make a connection to the Queue by using QueueClient, and then I use registerSessionHandler to process through the messages (code below).
The problem is that whenever a message is received, I can print all details about it including the content, but it is printed 10 times and after each time it prints an Exception.
(printing 10 times: I understand that this is because there is a 10 times retry policy before it throws the message to the Dead letter queue and goes to the next message.)
The Exception says
> USERCALLBACK-Receiver not created. Registering a MessageHandler creates a receiver.
The output with the Exception
But I'm sure that the SessionHandler does the same thing as MessageHandler but includes support for sessions, so it should create a receiver since it receives messages. I have tried to use MessageHandler but it won't even work and stops the whole program because it doesn't support sessionful messages, and the ones I receive have sessions.
My problem is understanding what the Exception wants me to do, and how can I fix the code so it won't give me any exceptions? Does anyone have suggestions on how to improve the code? or other methods that do the same thing?
QueueClient qc = new QueueClient(
new ConnectionStringBuilder(connectionString),
ReceiveMode.PEEKLOCK);
qc.registerSessionHandler(
new ISessionHandler() {
#Override
public CompletableFuture<Void> onMessageAsync(IMessageSession messageSession, IMessage message) {
System.out.printf(
"\nMessage received: " +
"\n --> MessageId = %s " +
"\n --> SessionId = %s" +
"\n --> Content Type = %s" +
"\n --> Content = \n\t\t %s",
message.getMessageId(),
messageSession.getSessionId(),
message.getContentType(),
getMessageContent(message)
);
return qc.completeAsync(message.getLockToken());
}
#Override
public CompletableFuture<Void> OnCloseSessionAsync(IMessageSession iMessageSession) {
return CompletableFuture.completedFuture(null);
}
#Override
public void notifyException(Throwable throwable, ExceptionPhase exceptionPhase) {
System.out.println("\n Exception " + exceptionPhase + "-" + throwable.getMessage());
}
},
new SessionHandlerOptions(1, true, Duration.ofMinutes(1)),
Executors.newSingleThreadExecutor()
);
(The getMessageContent(message) method is a separate method, for those interested:)
public String getMessageContent(IMessage message){
List<byte[]> content = message.getMessageBody().getBinaryData();
StringBuilder sb = new StringBuilder();
for (byte[] b : content) {
sb.append(new String(b)
);
}
return sb.toString();
}

For those who wonder, I managed to solve the problem!
It was simply done by using Azure Functions ServiceBusQueueTrigger, it will then listen to the Service bus Queue and process the messages. By setting isSessionsEnabled to true, it will accept sessionful messages as I wanted :)
So instead of writing more than 100 lines of code, the code looks like this now:
public class Function {
#FunctionName("QueueFunction")
public void run(
#ServiceBusQueueTrigger(
name = "TriggerName", //Any name you choose
queueName = "queueName", //QueueName from the portal
connection = "ConnectionString", //ConnectionString from the portal
isSessionsEnabled = true
) String message,
ExecutionContext context
) {
// Write the code you want to do with the message here
// Using the variable messsage which contains the messageContent, messageId, sessionId etc.
}
}

Creating several Google Calendars and avoiding 403 : Rate Limit Exceeded

I need to create up to 130 calendars on a Google account using Java and Google Calendar API but keep getting
"403 : Rate Limit Exceeded".
What I've tried :
-looping with
service.insert(calendar).execute();
-> result : I receive error 403 after 25 inserts ( which weirdly enough seems to be the old limit, should be 60 according to : https://support.google.com/a/answer/2905486?hl=en )
-looping with delay between each request (up to 60 seconds)
-> result : Didn't change the outcome. still 403 after 25 inserts ( In its documentation about Exponential Backoff, Google talks about mere seconds so I should think 1 whole minute is enough even if I don't increase exponentially that delay ).
-Request Batching ( following THIS Google example code )
-> result : After about 10 callbacks the response falls into the onFailure method with.. You guessed it 403 status code.
I think I am well within my ( maxed-out ) API quotas most of the time. I've seen "quotaExceeded" only once after several tests.
Batch Request sample :
batch = this.service.batch();
JsonBatchCallback<Calendar> callback = new JsonBatchCallback<Calendar>(){
#Override
public void onFailure(GoogleJsonError e, HttpHeaders responseHeaders) throws HttpResponseException{
log.debug(e);
throw new HttpResponseException(Integer.valueOf(e.get("code").toString()), e.getMessage());
}
#Override
public void onSuccess(Calendar cal, HttpHeaders responseHeaders) {
log.debug("Calendrier créé pour " + cal.getSummary());
}
};
for( String user : usernameList ) {
cal = new Calendar().setSummary(user);
this.service.calendars().insert(cal).queue(batch, callback);
}
batch.execute();

Why doesn't this thread pool execute HTTP requests simultaneously?

I wrote a few lines of code which will send 50 HTTP GET requests to a service running on my machine. The service will always sleep 1 second and return a HTTP status code 200 with an empty body. As expected the code runs for about 50 seconds.
To speed things up a little I tried to create an ExecutorService with 4 threads so I could always send 4 requests at the same time to my service. I expected the code to run for about 13 seconds.
final List<String> urls = new ArrayList<>();
for (int i = 0; i < 50; i++)
urls.add("http://localhost:5000/test/" + i);
final RestTemplate restTemplate = new RestTemplate();
final List<Callable<String>> tasks = urls
.stream()
.map(u -> (Callable<String>) () -> {
System.out.println(LocalDateTime.now() + " - " + Thread.currentThread().getName() + ": " + u);
return restTemplate.getForObject(u, String.class);
}).collect(Collectors.toList());
final ExecutorService executorService = Executors.newFixedThreadPool(4);
final long start = System.currentTimeMillis();
try {
final List<Future<String>> futures = executorService.invokeAll(tasks);
final List<String> results = futures.stream().map(f -> {
try {
return f.get();
} catch (InterruptedException | ExecutionException e) {
throw new IllegalStateException(e);
}
}).collect(Collectors.toList());
System.out.println(results);
} finally {
executorService.shutdown();
executorService.awaitTermination(10, TimeUnit.SECONDS);
}
final long elapsed = System.currentTimeMillis() - start;
System.out.println("Took " + elapsed + " ms...");
But - if you look at the seconds of the debug output - it seems like the first 4 requests are executed simultaneously but all other request are executed one after another:
2018-10-21T17:42:16.160 - pool-1-thread-3: http://localhost:5000/test/2
2018-10-21T17:42:16.160 - pool-1-thread-1: http://localhost:5000/test/0
2018-10-21T17:42:16.160 - pool-1-thread-2: http://localhost:5000/test/1
2018-10-21T17:42:16.159 - pool-1-thread-4: http://localhost:5000/test/3
2018-10-21T17:42:17.233 - pool-1-thread-3: http://localhost:5000/test/4
2018-10-21T17:42:18.232 - pool-1-thread-2: http://localhost:5000/test/5
2018-10-21T17:42:19.237 - pool-1-thread-4: http://localhost:5000/test/6
2018-10-21T17:42:20.241 - pool-1-thread-1: http://localhost:5000/test/7
...
Took 50310 ms...
So for debugging purposes I changed the HTTP request to a sleep call:
// return restTemplate.getForObject(u, String.class);
TimeUnit.SECONDS.sleep(1);
return "";
And now the code works as expected:
...
Took 13068 ms...
So my question is why does the code with the sleep call work as expected and the code with the HTTP request doesn't? And how can I get it to behave in the way I expected?

From the information, I can see this is the most probable root cause:
The requests you make are done in parallel but the HTTP server which fulfils these request handles 1 request at a time.
So when you start making requests, the executor service fires up the requests concurrently, thus you get the first 4 at same time.
But the HTTP server can respond to requests one at a time i.e. after 1 second each.
Now when 1st request is fulfilled the executor service picks another request and fires it and this goes on till last request.
4 request are blocked at HTTP server at a time, which are being served serially one after the other.
To get a Proof of Concept of this theory what you can do is use a messaging service (queue) which can receive concurrently from 4 channels an test. That should reduce the time.

PUSH Notifications for >1000 devices through GCM Server(Java)

I have a GCM-backend Java server and I'm trying to send to all users a notification msg. Is my approach right? To just split them into 1000 each time before giving the send request? Or is there a better approach?
public void sendMessage(#Named("message") String message) throws IOException {
int count = ofy().load().type(RegistrationRecord.class).count();
if(count<=1000) {
List<RegistrationRecord> records = ofy().load().type(RegistrationRecord.class).limit(count).list();
sendMsg(records,message);
}else
{
int msgsDone=0;
List<RegistrationRecord> records = ofy().load().type(RegistrationRecord.class).list();
do {
List<RegistrationRecord> regIdsParts = regIdTrim(records, msgsDone);
msgsDone+=1000;
sendMsg(regIdsParts,message);
}while(msgsDone<count);
}
}
The regIdTrim method
private List<RegistrationRecord> regIdTrim(List<RegistrationRecord> wholeList, final int start) {
List<RegistrationRecord> parts = wholeList.subList(start,(start+1000)> wholeList.size()? wholeList.size() : start+1000);
return parts;
}
The sendMsg method
private void sendMsg(List<RegistrationRecord> records,#Named("message") String message) throws IOException {
if (message == null || message.trim().length() == 0) {
log.warning("Not sending message because it is empty");
return;
}
Sender sender = new Sender(API_KEY);
Message msg = new Message.Builder().addData("message", message).build();
// crop longer messages
if (message.length() > 1000) {
message = message.substring(0, 1000) + "[...]";
}
for (RegistrationRecord record : records) {
Result result = sender.send(msg, record.getRegId(), 5);
if (result.getMessageId() != null) {
log.info("Message sent to " + record.getRegId());
String canonicalRegId = result.getCanonicalRegistrationId();
if (canonicalRegId != null) {
// if the regId changed, we have to update the datastore
log.info("Registration Id changed for " + record.getRegId() + " updating to " + canonicalRegId);
record.setRegId(canonicalRegId);
ofy().save().entity(record).now();
}
} else {
String error = result.getErrorCodeName();
if (error.equals(Constants.ERROR_NOT_REGISTERED)) {
log.warning("Registration Id " + record.getRegId() + " no longer registered with GCM, removing from datastore");
// if the device is no longer registered with Gcm, remove it from the datastore
ofy().delete().entity(record).now();
} else {
log.warning("Error when sending message : " + error);
}
}
}
}

Quoting from Google Docs:
GCM is support for up to 1,000 recipients for a single message. This capability makes it much easier to send out important messages to your entire user base. For instance, let's say you had a message that needed to be sent to 1,000,000 of your users, and your server could handle sending out about 500 messages per second. If you send each message with only a single recipient, it would take 1,000,000/500 = 2,000 seconds, or around half an hour. However, attaching 1,000 recipients to each message, the total time required to send a message out to 1,000,000 recipients becomes (1,000,000/1,000) / 500 = 2 seconds. This is not only useful, but important for timely data, such as natural disaster alerts or sports scores, where a 30 minute interval might render the information useless.
Taking advantage of this functionality is easy. If you're using the GCM helper library for Java, simply provide a List collection of registration IDs to the send or sendNoRetry method, instead of a single registration ID.

We can not send more than 1000 push notification at time.I searched a lot but not result then i did this with same approach split whole list in sub lists of 1000 items and send push notification.

How to identify URIs of active (long-running) HTTP requests?

Imagine a webapp which (sometimes) takes long time to respond to some HTTP (POST/GET/etc) request - how would You find such a request on server side?
So far I've used tomcat AccessLogValve to see the "completed" requests, but that doesn't let me to see the "in-progress" (stuck) ones :(
For example:
with netstat I'm able to identify long-lived sockets which could give me a count of currently-stuck requests (not URIs though), but HTTP keep-alives invalidate this approach
I could stackdump the app-server (kill -3 <server_pid>) multiple times and guess which threads are running long and reverse-engineer the URIs - not a smart way either
I could inject a router/proxy in front of web-app server (substitute hostnames, clone certs) which would show me the currently-running calls - not a simple approach
I could fall into just running tcpdump continously and parsing the traffic to keep list of currently-running URIs, but what to do with httpS then?
the closest I found is tomcat7's StuckThreadDetectionValve which'd periodically reports long-running calls, but it outputs the stacktrace (not URI) and doesn't provide "live" data (e.g. only polls periodically, floods the logs and lets to see the state of 1-60 seconds ago, but not "now")
Maybe I'm just missing/overlooking one of vital/core/basic tomcat features? or maybe weblogic (or any other app-server) has something robust to offer for this?
I'm kind of lost without such simple and essential feature.
Help? Please?

OK - creation of my own Valve was a proper and easy approach, sharing below. Apache did rework AccessLogValve multiple times, but all revisions follow same concept:
invoke(...) method just uses getNext().invoke(request,response) to invoke a chain of remaining valves and the actual handler/executor
log(...) method is invoked after above is complete
So we only need to:
also invoke log(...) before the getNext().invoke(request,response)
modify log(...) to distinguish "before" and "after" invocations
Easiest way would've been:
#Override
public void invoke(Request request, Response response) throws IOException, ServletException {
log(request, response, -1); // negative time indicates "before"
super.invoke(request, response);
}
But tomcat_6.0.16 code wasn't well-extendable, so I prefixed log-messages (in a hard-coded manner) with Thread.getName() and "before"/"after" indicator. Also I preferred to use reflection to access private AccessLogValve.getDate() :
package org.apache.catalina.valves;
import java.io.IOException;
import java.lang.reflect.Method;
import java.util.Date;
import javax.servlet.ServletException;
import org.apache.catalina.connector.Request;
import org.apache.catalina.connector.Response;
public class PreAccessLogValve extends AccessLogValve {
#Override
public void invoke(Request request, Response response) throws IOException, ServletException {
long timeStart = System.currentTimeMillis();
log(request, response, -timeStart); // negative time indicates "before" request
getNext().invoke(request, response);
log(request, response, System.currentTimeMillis() - timeStart); // actual (positive) - "after"
}
public void log(Request request, Response response, long time) {
if (started && getEnabled() && null != logElements && (null == condition || null == request.getRequest().getAttribute(condition))) {
StringBuffer result = new StringBuffer();
try {
Date date = (Date) methodGetDate.invoke(this);
for (int i = 0; i < logElements.length; i++) {
logElements[i].addElement(result, date, request, response, time);
}
} catch (Throwable t) { t.printStackTrace(); }
log(Thread.currentThread().getName() + (time<0?" > ":" < ") + result.toString());
}
}
private static final Method methodGetDate;
static {
Method m = null;
try {
m = AccessLogValve.class.getDeclaredMethod("getDate");
m.setAccessible(true);
} catch (Throwable t) { t.printStackTrace(); }
methodGetDate = m;
}
}
compiled above code with catalina.jar + servlet-api.jar and produced new catalina-my.jar, which was placed into tomcat/lib folder. After that - I've modified server.xml to have:
<Valve className="org.apache.catalina.valves.PreAccessLogValve"
directory="/tmp" prefix="test." suffix=".txt"
pattern="%a %t %m %U %s %b %D" resolveHosts="false" buffered="false"/>
Here's the sample output:
http-8007-exec-1 > 10.61.105.105 [18/Jan/2014:05:54:14 +0000] POST /admin/0$en_US/secure/enduser/search.do 200 - -1390024454470
http-8007-exec-5 > 10.61.105.105 [18/Jan/2014:05:54:17 +0000] GET /admin/0$en_US/secure/enduser/search.do 200 - -1390024457300
http-8007-exec-5 < 10.61.105.105 [18/Jan/2014:05:54:17 +0000] GET /admin/0$en_US/secure/enduser/search.do 200 13933 44
http-8007-exec-3 > 10.61.105.105 [18/Jan/2014:05:54:17 +0000] GET /admin/html/main.js 200 - -1390024457490
http-8007-exec-3 < 10.61.105.105 [18/Jan/2014:05:54:17 +0000] GET /admin/html/main.js 200 3750 0
http-8007-exec-5 > 10.61.105.105 [18/Jan/2014:05:54:17 +0000] GET /admin/images/layout/logo.gif 200 - -1390024457497
http-8007-exec-5 < 10.61.105.105 [18/Jan/2014:05:54:17 +0000] GET /admin/images/layout/logo.gif 200 1996 0
http-8007-exec-1 < 10.61.105.105 [18/Jan/2014:05:54:24 +0000] POST /admin/0$en_US/secure/enduser/search.do 200 13308 10209
This way all "in-progress" URIs can be easily retrieved at any moment:
[root#serv1 tomcat]# awk '{if(">"==$2){if($1 in S)print S[$1];S[$1]=$0}else delete S[$1]}END{for(i in S)print S[i]}' test
http-8007-exec-4 > 10.61.105.105 [18/Jan/2014:06:13:20 +0000] GET /admin/images/1x1blank.gif 200 - -13
http-8007-exec-2 > 10.61.105.105 [18/Jan/2014:06:13:16 +0000] POST /admin/servlet/handlersvr 200 - -13

Unfortunately, there is not a simple way to get a list of the in-flight HTTP requests that are taking a long time. As you mention, taking several thread dumps a few seconds apart will tell you which threads are performing the HTTP operations slowly (because the thread stack will be identical in each one that is waiting for the response). But, it doesn't tell you much more than that unless you can follow the code back to a static piece of code with the URL. But, you can take the thread dumps and identify the thread IDs, then take a heap dump and find those threads in the heap dump. While not straight-forward and definitely not simple, you can get the URL that is being used, how long it has been waiting, etc.