We Keep Coding

How to scale more than 1 instance and deal with scheduled task in spring?

I am having a push notifications being send to android and ios application through spring boot every day at 8am Europe/Paris.
If I run multiple instances, the notifications will send multiple times. I am thinking to send every day notifications send on the database, and check them but I am worried it still run multiple times, this is what I am doing:
#Component
public class ScheduledTasks {
private static final Logger log = LoggerFactory.getLogger(ScheduledTasks.class);
private static final SimpleDateFormat dateFormat = new SimpleDateFormat("HH:mm:ss");
#Autowired
private ExpoPushTokenRepository expoPushTokenRepository;
#Autowired
private ExpoPushNotificationService expoPushNotificationService;
#Autowired
private MessageSource messageSource;
// TODO: if instances > 1, this will run multiple times, save to database the notifications send and prevent multiple sending.
#Scheduled(cron = "${cron.promotions.notification}", zone = "Europe/Paris")
public void sendNewPromotionsNotification() {
List<ExpoPushToken> expoPushTokenList = expoPushTokenRepository.findAll();
ArrayList<NotifyRequest> notifyRequestList = new ArrayList<>();
for (ExpoPushToken expoPushToken : expoPushTokenList) {
NotifyRequest notifyRequest = new NotifyRequest(
expoPushToken.getToken(),
"This is a test title",
"This is a test subtitle",
"This is a test body"
);
notifyRequestList.add(notifyRequest);
}
expoPushNotificationService.sendPushNotificationToList(notifyRequestList);
log.info("{} Send push notification to " + expoPushTokenList.size() + " userse", dateFormat.format(new Date()));
}
}
Does anybody have an idea on how I can prevent that safely?

Quartz would be my mostly database-agnostic solution for the task at hand, but was ruled out, so we are not going to discuss it.
The solution we are going to explore instead makes the following assumptions:
Postgres >= 9.5 is used (because we are going to use SKIP LOCKED, which was introduced in Postgresl 9.5).
It is okay to run a native query.
Under this conditions, we can retrieve batches of notifications from multiple instances of the application running through the following query:
SELECT * FROM expo_push_token FOR UPDATE SKIP LOCKED LIMIT 100;
This will retrieve and lock up to 100 entries from the table expo_push_token. If two instances of the application execute this query simultaneously, the received results will be disjoint. 100 is just some sample value. We may want to fine-tune this value for our use case. The locks stay active until the current transaction ends.
After an instance has fetched a batch of notifications, it has to also delete the entries it locked from the table or otherwise mark that this entry has been processed (if we go down this route, we have to modify the query above to filter-out already processed entires) and close the current transaction to release the locks. Each instance of the application would then repeat this query until the query returns zero entries.
There is also an alternative approach: an instance first fetches a batch size of notifications to send, keeps the transaction to the database open (thus continues holding the lock on the database), sends out its notification and then deletes/updates the entries and closes the transactions.
The two solutions have different strengths/weaknesses:
the first solutions keeps the transaction short. But if the application crashes in the middle of sending out notificatiosn, the part of its batch that was not send out is lost in this run.
the second solution keeps the transaction open, for possibly a long time. If it crashes in the middle fo sending out notifications, all entries will be unlocked and its batch would be re-processed, possibly resulting in some notifications being sent out twice.
For this solution to work, we also need some kind of job that fills table expo_push_token with the data we need. This job should run beforehand, i.e. its execution should not overlap with the notification sending process.

Periods of prolonged inactivity and frequent MessageLockLostException in QueueClient

Background
We have a data transfer solution with Azure Service Bus as the message broker. We are transferring data from x datasets through x queues - with x dedicated QueueClients as senders. Some senders publish messages at the rate of one message every two seconds, while others publish one every 15 minutes.
The application on the data source side (where senders are) is working just fine, giving us the desired throughput.
On the other side, we have an application with one QueueClient receiver per queue with the following configuration:
maxConcurrentCalls = 1
autoComplete = true (if receive mode = RECEIVEANDDELETE) and false (if receive mode = PEEKLOCK) - we have some receivers where, if they shut-down unexpectedly, would want to preserve the messages in the Service Bus Queue.
maxAutoRenewDuration = 3 minutes (lock duraition on all queues = 30 seconds)
an Executor service with a single thread
The MessageHandler registered with each of these receivers does the following:
public CompletableFuture<Void> onMessageAsync(final IMessage message) {
// deserialize the message body
final CustomObject customObject = (CustomObject)SerializationUtils.deserialize((byte[])message.getMessageBody().getBinaryData().get(0));
// process processDB1() and processDB2() asynchronously
final List<CompletableFuture<Boolean>> processFutures = new ArrayList<CompletableFuture<Boolean>>();
processFutures.add(processDB1(customObject)); // processDB1() returns Boolean
processFutures.add(processDB2(customObject)); // processDB2() returns Boolean
// join both the completablefutures to get the result Booleans
List<Boolean> results = CompletableFuture.allOf(processFutures.toArray(new CompletableFuture[processFutures.size()])).thenApply(future -> processFutures.stream()
.map(CompletableFuture<Boolean>::join).collect(Collectors.toList())
if (results.contains(false)) {
// dead-letter the message if results contains false
return getQueueClient().deadLetterAsync(message.getLockToken());
} else {
// complete the message otherwise
getQueueClient().completeAsync(message.getLockToken());
}
}
We tested with the following scenarios:
Scenario 1 - receive mode = RECEIVEANDDELETE, message publish rate: 30/ minute
Expected Behavior
The messages should be received continuosuly with a constant throughput (which need not necessarily be the throughput at source, where messages are published).
Actual behavior
We observe random, long periods of inactivity from the QueueClient - ranging from minutes to hours - there is no Outgoing Messages from the Service Bus namespace (observed on the Metrics charts) and there are no consumption logs for the same time periods!
Scenario 2 - receive mode = PEEKLOCK, message publish rate: 30/ minute
Expected Behavior
The messages should be received continuosuly with a constant throughput (which need not necessarily be the throughput at source, where messages are published).
Actual behavior
We keep seeing MessageLockLostException constantly after 20-30 minutes into the run of the application.
We tried doing the following -
we reduced the prefetch count (from 20 * processing rate - as mentioned in the Best Practices guide) to a bare minimum (to even 0 in one test cycle), to reduce the no. of messages that are locked for the client
increased the maxAutoRenewDuration to 5 minutes - our processDB1() and processDB2() do not take more than a second or two for almost 90% of the cases - so, I think the lock duration of 30 seconds and maxAutoRenewDuration are not issues here.
removed the blocking CompletableFuture.get() and made the processing synchronous.
None of these tweaks helped us fix the issue. What we observed is that the COMPLETE or RENEWMESSAGELOCK are throwing the MessageLockLostException.
We need help with finding answers for the following:
why is there a long period of inactivity of the QueueClient in scenario 1?
how do we know the MessageLockLostExceptions are thrown, because the locks have indeed expired? we suspect the locks cannot expire too soon, as our processing happens in a second or two. disabling prefetch also did not solve this for us.
Versions and Service Bus details
Java - openjdk-11-jre
Azure Service Bus namespace tier: Standard
Java SDK version - 3.4.0

For Scenario 1 :
If you have the duplicate detection history enabled, there is a possibility of this behavior happening as per the below explained scenario :
I had enabled for 30 seconds. I constantly hit Service bus with duplicate messages ( im my case messages with the same messageid from the client - 30 /per minute). I would be seeing a no activity outgoing for the window. Though the messages are received at the servicebus from the sending client, I was not be able to see them in outgoing messages. You could probably check whether you re encountering the duplicate messages which are filtered - inturn resulting inactivity in outgoing.
Also Note : You can't enable/disable duplicate detection after the queue is created. You can only do so at the time of creating the queue.

The issue was not with the QueueClient object per se. It was with the processes that we were triggering from within the MessageHandler: processDB1(customObject) and processDB2(customObject). since these processes were not optimized, the message consumption dropped and the locks gor expired (in peek-lock mode), as the handler was spending more time (in relation to the rate at which messages were published to the queues) in completing these opertations.
After optimizing the processes, the consumption and completion (in peek-lock mode) were just fine.

Hold thread in spring rest request for long-polling

As I wrote in title we need in project notify or execute method of some thread by another. This implementation is part of long polling. In following text describe and show my implementation.
So requirements are that:
UserX send request from client to server (poll action) immediately when he got response from previous. In service is executed spring async method where thread immediately check cache if there are some new data in database. I know that cache is usually used for methods where for specific input is expected specific output. This is not that case, because I use cache to reduce database calls and output of my method is always different. So cache help me store notification if I should check database or not. This checking is running in while loop which end when thread find notification to read database in cache or time expired.
Assume that UserX thread (poll action) is currently in while loop and checking cache.
In that moment UserY (push action) send some data to server, data are stored in database in separated thread, and also in cache is stored userId of recipient.
So when UserX is checking cache he found id of recipient (id of recipient == his id in this case), and then break loop and fetch these data.
So in my implementation I use google guava cache which provide manually write.
private static Cache<Long, Long> cache = CacheBuilder.newBuilder()
.maximumSize(100)
.expireAfterWrite(5, TimeUnit.MINUTES)
.build();
In create method I store id of user which should read these data.
public void create(Data data) {
dataRepository.save(data);
cache.save(data.getRecipient(), null);
System.out.println("SAVED " + userId + " in " + Thread.currentThread().getName());
}
and here is method of polling data:
#Async
public CompletableFuture<List<Data>> pollData(Long previousMessageId, Long userId) throws InterruptedException {
// check db at first, if there are new data no need go to loop and waiting
List<Data> data = findRecent(dataId, userId));
data not found so jump to loop for some time
if (data.size() == 0) {
short c = 0;
while (c < 100) {
// check if some new data added or not, if yes break loop
if (cache.getIfPresent(userId) != null) {
break;
}
c++;
Thread.sleep(1000);
System.out.println("SEQUENCE: " + c + " in " + Thread.currentThread().getName());
}
// check database on the end of loop or after break from loop
data = findRecent(dataId, userId);
}
// clear data for that recipient and return result
cache.clear(userId);
return CompletableFuture.completedFuture(data);
}
After User X got response he send poll request again and whole process is repeated.
Can you tell me if is this application design for long polling in java (spring) is correct or exists some better way? Key point is that when user call poll request, this request should be holded for new data for some time and not response immediately. This solution which I show above works, but question is if it will be works also for many users (1000+). I worry about it because of pausing threads which should make slower another requests when no threads will be available in pool. Thanks in advice for your effort.

Check Web Sockets. Spring supports it from version 4 on wards. It doesn't require client to initiate a polling, instead server pushes the data to client in real time.
Check the below:
https://spring.io/guides/gs/messaging-stomp-websocket/
http://www.baeldung.com/websockets-spring
Note - web sockets open a persistent connection between client and server and thus may result in more resource usage in case of large number of users. So, if you are not looking for real time updates and is fine with some delay then polling might be a better approach. Also, not all browsers support web sockets.
Web Sockets vs Interval Polling
Longpolling vs Websockets
In what situations would AJAX long/short polling be preferred over HTML5 WebSockets?
In your current approach, if you are having a concern with large number of threads running on server for multiple users then you can trigger the polling from front-end every time instead. This way only short lived request threads will be triggered from UI looking for any update in the cache. If there is an update, another call can be made to retrieve the data. However don't hit the server every other second as you are doing otherwise you will have high CPU utilization and user request threads may also suffer. You should do some optimization on your timing.
Instead of hitting the cache after a delay of 1 sec for 100 times, you can apply an intelligent algorithm by analyzing the pattern of cache/DB update over a period of time.
By knowing the pattern, you can trigger the polling in an exponential back off manner to hit the cache when the update is most likely expected. This way you will be hitting the cache less frequently and more accurately.

GETBULK SNMP4J Request

I'm using snmp4j to try and perform SNMP functions against a remote agent. Due to a number of limitations out of our control I need to perform a GETBULK to obtain a large table in a short space of time.
My current implementation:
public Map<String, String> doGetBulk(#NotNull VariableBinding... vbs)
throws IOException {
Map<String, String> result = new HashMap<>();
Snmp snmp = null;
try {
// Create TransportMapping and Listen
TransportMapping transport = new DefaultUdpTransportMapping();
snmp = new Snmp(transport);
transport.listen();
PDU pdu = new PDU();
pdu.setType(PDU.GETBULK);
pdu.setMaxRepetitions(200);
pdu.setNonRepeaters(0);
pdu.addAll(vbs);
ResponseEvent responseEvent = snmp.send(pdu, this.target);
PDU response = responseEvent.getResponse();
// Process Agent Response
if (response != null) {
for(VariableBinding vb : response.getVariableBindings()) {
result.put("." + vb.getOid().toString(), vb.getVariable().toString());
}
} else {
LOG.error("Error: Agent Timeout... ");
}
} catch (NullPointerException ignore) {
// The variable table is null
} finally {
if (snmp != null) snmp.close();
}
return result;
}
However, this only ever returns 100 results when I know there are 5000+. I know I cant exceed the PDU size so I have so problem with the response being truncated into blocks of 100 but I cant work out how I can get a handle to cascade the request to get the next 100 entries.

It is bad practice to use MaxRepetitions > 100 due to TCP/IP packet fragmentation and the nature of UDP that does not guarantee the order of packets. So most SNMP frameworks and agents have such built-in limit.

All details are already there in the RFC document,
https://www.rfc-editor.org/rfc/rfc1905
4.2.3 tells how the agent side should handle GET BULK requests, and
While the maximum number of variable bindings in the Response-PDU is
bounded by N + (M * R), the response may be generated with a lesser
number of variable bindings (possibly zero) for either of three
reasons.
(1) If the size of the message encapsulating the Response-PDU
containing the requested number of variable bindings would be
greater than either a local constraint or the maximum message size
of the originator, then the response is generated with a lesser
number of variable bindings. This lesser number is the ordered set
of variable bindings with some of the variable bindings at the end
of the set removed, such that the size of the message encapsulating
the Response-PDU is approximately equal to but no greater than
either a local constraint or the maximum message size of the
originator. Note that the number of variable bindings removed has
no relationship to the values of N, M, or R.
(2) The response may also be generated with a lesser number of
variable
bindings if for some value of iteration i, such that i is greater
than zero and less than or equal to M, that all of the generated
variable bindings have the value field set to the `endOfMibView'.
In this case, the variable bindings may be truncated after the (N +
(i * R))-th variable binding.
(3) In the event that the processing of a request with many
repetitions
requires a significantly greater amount of processing time than a
normal request, then an agent may terminate the request with less
than the full number of repetitions, providing at least one
repetition is completed.
About how to do a series of proper GET BULK operations to query all data you want, you can refer to section 4.2.3.1 for an example.

You've set maximum repetition count to 200, that is the server may send you at most 200 rows. So on the one hand, you'll never get more than 200 rows (and least of all 5000 or more). On the other hand, the server may decide to send you less rows, it's practically the server's choice; you tell him what you're able to process.
Usually you request 10-50 rows at max. (BTW: There are many servers out there with buggy SNMP implementations and the higher you set max-repetitions, the higher is the chance you get nothing at all.)
So you have to request row set by row set. Since you probably don't want to implement that yourself I'd recommend to use the TableUtils class. Just start with getTable().

How to fetch data effectively from kestrel queue

For some reason we plan to use kestrel queue in our project. We do some demons, the main problem is how to to fetch data from queue with low CPU utilization and effectively. The way we implemented to fetch is if we failed to fetch data from queue more than 5 times, we sleep the thread 100ms to reduce the CPU utilization.
while (running) {
try {
LoginLogQueueEntry data = kestrelQueue.fetch();
if (null != data && data.isLegal()) {
entryCacheList.add(data); //add the data to the local caceh
resetStatus();
} else {
failedCount++;
//if there is no data in the kestrel and the local cache is not empty, insert the data into mysql database
if (failedCount == 1 && !entryCacheList.isEmpty()) {
resetStatus();
insertLogList(entryCacheList); // insert current data into database
entryCacheList.clear(); //empty local cache
}
if (failedCount >= 5 && entryCacheList.isEmpty()) {
//fail 5 times. Sleep current thread.
failedCount = 0;
Thread.sleep((sleepTime + MIN_SLEEP_TIME) % MAX_SLEEP_TIME);
}
}
//Insert 1000 rows once
if (entryCacheList.size() >= 1000) {
insertLogList(entryCacheList);
entryCacheList.clear();
}
} catch (Exception e) {
logger.warn(e.getMessage());
}
Is there any other good way to do so? The perfect the way i think is the queue can notice to the worker that we got data and fetch them .

See the "Blocking Fetches" section at http://robey.lag.net/2008/11/27/scarling-to-kestrel.html

Blocking reads are described here, under "Memcache commands": https://github.com/robey/kestrel/blob/master/docs/guide.md
You can add option flags to a get command by separating them with slashes, so to fetch an item from the "jobs" queue, waiting up to one second:
get jobs/t=1000
If nothing shows up on the queue in one second, you'll get the same empty response, just one second later than you're getting it now. :)
It's important to tune your response timeout when you do this. If you use a blocking read with a timeout of one second, but your client library's response timeout is 500 milliseconds, the library will disconnect from the server before the blocking read is finished. So make sure the response timeout is greater than the timeout you're using in the read request.

You need to use a blocking get. I couldn't track down the API docs, but I found an article suggesting that it's possible in kestrel.