We Keep Coding

processElement emits record even though the state is present for the key

I have a state which holds a data for 2 mins, sometime processElement still emits the record out even though there is a state present for that key.
#Override
public void processElement(EngagerEvents value, KeyedProcessFunction<String, EngagerEvents, String>.Context ctx, Collector<String> out) throws Exception {
if (anonymousIdHasBeenSeen.value() == null) {
System.out.println("time stamp emitting: " +jsonNode.get("server_timestamp"));
// key is not available in the state
anonymousIdHasBeenSeen.update(true);
System.out.println("TIMER START TIME: " +ctx.timestamp());
out.collect(value.getEventString());
ctx.timerService().registerProcessingTimeTimer(ctx.timestamp() + (stateTtl * 1000));
}
}
TIMER TRIGGER
-------------
public void onTimer(long timestamp, OnTimerContext ctx, Collector<String> out)
throws Exception {
// triggers after ttl has passed
System.out.println("Call back triggered : time : " +timestamp + " value : " +anonymousIdHasBeenSeen.value());
if (anonymousIdHasBeenSeen.value()) {
anonymousIdHasBeenSeen.clear();
}
}
My simulator code which produces data to kafka - im pumping anonymousId 111, 5 times
public static void main(String[] args) {
int n = 1; // Number of threads
for (int i = 0; i < n; i++) {
Thread thread = new Thread(new ExecutorThread());
thread.start();
}
for (int i = 0; i < 5; i++) {
ProducerRecord<String, String> record = new ProducerRecord<>("topic", key ,""{"anonymousId": "111", "device": "ios"}"");
try {
producer.send(record);
Thread.sleep(500);
} catch (SerializationException | InterruptedException e) {
// may need to do something with it
}
}
I'm keying by anonymousId. In my case there is only one anonymousId 111. call back trigger is 60 secs
DataStream<String> keyedStream = mappedEngagerEventsDataStream.keyBy(EngagerEvents::getAnonymousId)
.process(new KeyedProcessingWithCallBack(Long.parseLong(60))
.uid("engager-events-keyed-processing");
Am I doing something wrong here ? I tried debugging setting a break points, Even if the control doesnt go inside the If condition(sout insid If condition aren't printed) I see that particular event emitted out.
Am I doing something wrong here ? Why is the event emitted even though my out.collect inside If statement ? Can someone please point out what am I doing wrong here.

How to use parallel processing in most efficient and elegant way in java

I have different sources of data from which I want to request in parallel (since each of this request is an http call and may be pretty time consuming). But I'm going to use only 1 response from these requests. So I kind of prioritize them. If the first response is invalid I'm going to check the second one. If it's also invalid I want to use the third, etc.
But I want to stop processing and return the result as soon as I receive the first correct response.
To simulate the problem I created the following code, where I'm trying to use java parallel streaming. But the problem is that I receive final results only after processing all requests.
public class ParallelExecution {
private static Supplier<Optional<Integer>> testMethod(String strInt) {
return () -> {
Optional<Integer> result = Optional.empty();
try {
result = Optional.of(Integer.valueOf(strInt));
System.out.printf("converted string %s to int %d\n",
strInt,
result.orElse(null));
} catch (NumberFormatException ex) {
System.out.printf("CANNOT CONVERT %s to int\n", strInt);
}
try {
int randomValue = result.orElse(10000);
TimeUnit.MILLISECONDS.sleep(randomValue);
System.out.printf("converted string %s to int %d in %d milliseconds\n",
strInt,
result.orElse(null), randomValue);
} catch (InterruptedException e) {
e.printStackTrace();
}
return result;
};
}
public static void main(String[] args) {
Instant start = Instant.now();
System.out.println("Starting program: " + start.toString());
List<Supplier<Optional<Integer>>> listOfFunctions = new ArrayList();
for (String arg: args) {
listOfFunctions.add(testMethod(arg));
}
Integer value = listOfFunctions.parallelStream()
.map(function -> function.get())
.filter(optValue -> optValue.isPresent()).map(val-> {
System.out.println("************** VAL: " + val);
return val;
}).findFirst().orElse(null).get();
Instant end = Instant.now();
Long diff = end.toEpochMilli() - start.toEpochMilli();
System.out.println("final value:" + value + ", worked during " + diff + "ms");
}
}
So when I execute the program using the following command:
$java ParallelExecution dfafj 34 1341 4656 dfad 245df 5767
I want to get the result "34" as soon as possible (around after 34 milliseconds) but in fact, I'm waiting for more than 10 seconds.
Could you help to find the most efficient solution for this problem?

ExecutorService#invokeAny looks like a good option.
List<Callable<Optional<Integer>>> tasks = listOfFunctions
.stream()
.<Callable<Optional<Integer>>>map(f -> f::get)
.collect(Collectors.toList());
ExecutorService service = Executors.newCachedThreadPool();
Optional<Integer> value = service.invokeAny(tasks);
service.shutdown();
I converted your List<Supplier<Optional<Integer>>> into a List<Callable<Optional<Integer>>> to be able to pass it in invokeAny. You may build Callables initially. Then, I created an ExecutorService and submitted the tasks.
The result of the first successfully executed task will be returned as soon as that result is returned from a task. Other tasks will end up interrupted.
You also may want to look into CompletionService.
List<Callable<Optional<Integer>>> tasks = Arrays
.stream(args)
.<Callable<Optional<Integer>>>map(arg -> () -> testMethod(arg).get())
.collect(Collectors.toList());
final ExecutorService underlyingService = Executors.newCachedThreadPool();
final ExecutorCompletionService<Optional<Integer>> service = new ExecutorCompletionService<>(underlyingService);
tasks.forEach(service::submit);
Optional<Integer> value = service.take().get();
underlyingService.shutdownNow();

You can use a queue to put your results in:
private static void testMethod(String strInt, BlockingQueue<Integer> queue) {
// your code, but instead of returning anything:
result.ifPresent(queue::add);
}
and then call it with
for (String s : args) {
CompletableFuture.runAsync(() -> testMethod(s, queue));
}
Integer result = queue.take();
Note that this will only handle the first result, as in your sample.

I have tried it using competableFutures and anyOf method. It will return when any one of the future is completed. Now, key to stop other tasks is to provide your own executor service to the completableFuture(s) and shutting it down when required.
public static void main(String[] args) {
Instant start = Instant.now();
System.out.println("Starting program: " + start.toString());
CompletableFuture<Optional<Integer>> completableFutures[] = new CompletableFuture[args.length];
ExecutorService es = Executors.newFixedThreadPool(args.length,r -> {
Thread t = new Thread(r);
t.setDaemon(false);
return t;
});
for (int i = 0;i < args.length; i++) {
completableFutures[i] = CompletableFuture.supplyAsync(testMethod(args[i]),es);
}
CompletableFuture.anyOf(completableFutures).
thenAccept(res-> {
System.out.println("Result - " + res + ", Time Taken : " + (Instant.now().toEpochMilli()-start.toEpochMilli()));
es.shutdownNow();
});
}
PS :It will throw interrupted exceptions that you can ignore in try catch block and not print the stack trace.Also, your thread pool size ideally should be same as length of args array.

Kafka is slow to produce messages in first seconds

i m working with kafka, and i made a producer like that:
synchronized (obj) {
while (true){
long start = Instant.now().toEpochMilli();
for (int i=0; i< NUM_MSG_SEC ; i++)
{
PriceStreamingData data = PriceStreamingData.newBuilder()
.setUser(getRequest().getUser())
.setSecurity(getRequest().getSecurity())
.setTimestamp(Instant.now().toEpochMilli())
.setPrice(new Random().nextDouble()*200)
.build();
record = new ProducerRecord<>(topic, keyBuilder.build(data),
data);
producer.send(record,new Callback(){
#Override
public void onCompletion(RecordMetadata arg0, Exception arg1) {
counter.incrementAndGet();
if(arg1 != null){
arg1.printStackTrace();
}
}
});
}
long diffCiclo = Instant.now().toEpochMilli() - start;
long diff = Instant.now().toEpochMilli() - startTime;
System.out.println("Number of sent: " + counter.get() +
" Millisecond:" + (diff) + " - NumberOfSent/Diff(K): " + counter.get()/diff );
try {
if(diffCiclo >= 1000){
System.out.println("over 1 second: " + diffCiclo);
}
else {
obj.wait( 1000 - diffCiclo );
}
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
as you can see it is extremely simple, it just make a new message and send it.
If i see the logs:
NumberOfSent/Diff(K)
in the first 10 seconds it perform very bad just
30k per second
after 60 seconds i have
180k per second
why ? and how can i already start the process going already to 180k ?
my kafka producer configuration is the Follwing
Async producer ( but also with sync producer the situation dose not change)
ACKS_CONFIG = 0
BATCH_SIZE_CONFIG = 20000
COMPRESSION_TYPE_CONFIG = none
LINGER_MS_CONFIG = 0
last detail:
NUM_MSG_SEC is set to 200000 or bigger number

I found the solution by myself and I hope this post can be useful for other people too.
The problem stand in the
ProducerConfig.BATCH_SIZE_CONFIG
and
ProducerConfig.LINGER_MS_CONFIG
My parameters were 20000 and 0, in order to fix the issue I did set them them to higher values 200000 and 1000. Finally I started the JVM with the parameters:
-XX:MinMetaspaceFreeRatio=100
-XX:MaxMetaspaceFreeRatio=100
because I saw it takes longer to set the metaspace to a decent value.
Now the producer start directly at 140k and in 1 second already is to 180k.

Updating a table when clicking on an element from another table

I need to show elements on a table depending on the element (Person) clicked on another table. The problem is that, using a Service, if the user clicks on two elements of the first table very quickly, the data of the two elements is showed in the table, and I only want to show the data from the last one clicked. Hope you can help me.
Here is my code:
personTable.getSelectionModel().selectedItemProperty().addListener(
(observable, oldValue, newValue) -> {
try {
contactoTable.setPlaceholder(new Label("Cargando..."));
showPersonDetails(newValue);
} catch (SQLException ex) {
Logger.getLogger(PersonOverviewController.class.getName()).log(Level.SEVERE, null, ex);
}
});
And showPersonDatails:
contactoTable.setVisible(true);
contactoTable.getItems().clear();
firstNameLabel.setText(person.getFirstName());
lastNameLabel.setText(person.getLastName());
mailLabel.setText(person.getMail());
phoneLabel.setText(person.getPhone());
descriptionLabel.setText(person.getDescription());
service = new Service<Void>() {
#Override
protected Task<Void> createTask() {
return new Task<Void>() {
#Override
protected Void call() throws Exception {
//Background work
DBManager db = new DBManager();
String query = "SELECT * FROM eventos";
ResultSet r = db.executeSelect(query);
contactoTable.getItems().clear();
contactoData.clear();
while (r.next()) {
String q = "SELECT * FROM " + r.getString("Nombre").replace(" ", "_") + " WHERE Nombre = '" + person.getFirstName() + "' AND Apellidos = '" + person.getLastName() + "' AND Correo = '" + person.getMail() + "'";
ResultSet result = db.executeSelect(q);
while (result.next()) {
contactoData.add(new Row(r.getString("Nombre"), result.getString("Asistencia")));
}
}
final CountDownLatch latch = new CountDownLatch(1);
Platform.runLater(() -> {
try {
//FX Stuff done here
contactoTable.setPlaceholder(new Label("No invitado a ningún evento"));
contactoTable.setItems(contactoData);
} finally {
latch.countDown();
}
});
latch.await();
//Keep with the background work
return null;
}
};
}
};
service.start();

You are referencing the same data list (contactoData) from multiple threads, with apparently no synchronization on the list. If the user selects two different items in rapid succession, you launch a service for each one, each service running its task in a different thread. Consequently you have no control over the order the two different threads perform their (multiple) manipulations on contactoData. For example, it is possible (even probable) that the order for two services executing asynchronously is:
First service clears the list
Second service clears the list
First service adds elements to the list
Second service adds elements to the list
and in this case the list contains elements generated by both services, not just one of them.
So you should have your tasks operate on, and return, a new list they create. Then process that list on the FX Application Thread.
It's also not clear why you need a service here, as you only seem to ever use each service once. You may as well just use a task directly.
You also probably want to ensure that the last selection is the one displayed. Since the tasks are running asynchronously, it's possible that if two tasks were started in quick succession, the second would complete before the first. This would result in the second selection being displayed, and then the first selection replacing it. You can avoid this by doing the UI update in an onSucceeded handler, and canceling any current task when you start a new one (thus preventing the currently-executing task from invoking its onSucceeded handler).
Finally, it's really not clear to me why you are making the task wait until the UI is updated.
Here is an updated version of your code:
private Task<List<Row>> updateContactTableTask ;
// ...
private void showPersonDetails(Person person) {
contactoTable.getItems().clear();
firstNameLabel.setText(person.getFirstName());
lastNameLabel.setText(person.getLastName());
mailLabel.setText(person.getMail());
phoneLabel.setText(person.getPhone());
descriptionLabel.setText(person.getDescription());
if (updateContactTableTask != null && updateContactTableTask.isRunning()) {
updateContactTableTask.cancel();
}
updateContactTableTask = new Task<List<Row>>() {
#Override
protected List<Row> call() throws Exception {
List<Row> resultList = new ArrayList<>() ;
//Background work
DBManager db = new DBManager();
String query = "SELECT * FROM eventos";
ResultSet r = db.executeSelect(query);
// quit if we got canceled here...
if (isCancelled()) {
return resultList;
}
while (r.next() && ! isCancelled()) {
// Note: building a query like this is inherently unsafe
// You should use a PreparedStatement in your DBManager class instead
String q = "SELECT * FROM " + r.getString("Nombre").replace(" ", "_") + " WHERE Nombre = '" + person.getFirstName() + "' AND Apellidos = '" + person.getLastName() + "' AND Correo = '" + person.getMail() + "'";
ResultSet result = db.executeSelect(q);
while (result.next()) {
resultList.add(new Row(r.getString("Nombre"), result.getString("Asistencia")));
}
}
return resultList ;
}
};
updateContactTableTask.setOnSucceeded(e -> {
// not really clear you still need contactoData, but if you do:
contactoData.setAll(updateContactTableTask.getValue());
contactoTable.setPlaceholder(new Label("No invitado a ningún evento"));
contactoTable.setItems(contactoData);
});
updateContactTableTask.setOnFailed(e -> {
// handle database errors here...
});
new Thread(updateContactTableTask).start();
}
As an aside, it's not clear to me if, and if so, how, you are closing your database resources. E.g. the result sets never seem to get closed. This could cause resource leaks. However this is incidental to the question (and relies on knowing how your DBManager class is implemented), so I won't address it here.

Start Multiple thread at the same time

I have a list of different URLs (about 10) from which I need content. I have made a program with which I am getting the content of 1 URL but I am unable to do it with multiple URLs.
I've studied lots of tutorials on threads in Java but I'm unable to find an answer.
In my case, URLs are like www.example1.com, www.example2.com, www.example3.com, www.example4.com.
I want to make thread for each URL and run it at the same time.
public class HtmlParser {
public static int searchedPageCount = 0,
skippedPageCount = 0,
productCount = 0;
public static void main(String[] args) {
List<String> URLs = new LinkedList<String>();
long t1 = System.currentTimeMillis();
URLs.add("www.example.com");
int i = 0;
for (ListIterator iterator = URLs.listIterator(); i < URLs.size();) {
i++;
System.out.println("While loop");
List<String> nextLevelURLs = processURL(URLs.get(iterator
.nextIndex()));
for (String URL : nextLevelURLs) {
if (!URLs.contains(URL)) {
System.out.println(URL);
iterator.add(new String(URL));
}
}
System.out.println(URLs.size());
}
System.out.println("Total products found: " + productCount);
System.out.println("Total searched page: " + searchedPageCount);
System.out.println("Total skipped page: " + skippedPageCount);
long t2 = System.currentTimeMillis();
System.out.println("Total time taken: " + (t2 - t1) / 60000);
}
public static List<String> processURL(String URL) {
List<String> nextLevelURLs = new ArrayList<String>();
try {
searchedPageCount++;
// System.out.println("Current URL: " + URL);
Elements products = Jsoup.connect(URL).timeout(60000).get()
.select("div.product");
for (Element product : products) {
System.out.println(product.select(" a > h2").text());
System.out.println(product.select(" a > h3").text());
System.out.println(product.select(".product > a").attr("href"));
System.out
.println(product.select(".image a > img").attr("src"));
System.out.println(product.select(".price").text());
System.out.println();
productCount++;
}
// System.out.println("Total products found until now: " +
// productCount);
Elements links = Jsoup.connect(URL).timeout(60000).get()
.select("a[href]");
for (Element link : links) {
URL = link.attr("href");
if (URL.startsWith("http://www.example.com/")) {
// System.out.println("URLs added.");
nextLevelURLs.add(URL);
} else {
skippedPageCount++;
// System.out.println("URL skipped: " + URL);
}
}
} catch (Exception e) {
e.printStackTrace();
}
return nextLevelURLs;
}
}

Unfortunately, there is no way to start two threads at the same time.
Let me explain better: first of all, the sequence thread1.Start(); and thread2.Start(); is executed with thread1 first and, after that, thread2. It means only that thread thread1 is scheduled before thread 2, not actually started. The two methods take fractions of second each one, so the fact that they are in sequence cannot be seen by a human observer.
More, Java threads are scheduled, ie. assigned to be eventually executed. Even if you have a multi-core CPU, you are not sure that 1) the threads run in parallel (other system processes may interfere) and 2) the threads both start just after the Start() method is called.
but you can run multiple threads in this way..
new Thread(thread1).start();
new Thread(thread2).start();

basically create a class that implements Runnable, put the code that deals with one url in this code. In your main class for each URL, construct a class with the information that is needs (E.g. URL) and then run run
Plenty of sites that teach how to do multi-threaded java

First of all, the code you pasted looks like bad because it is orienting a simple process. You need to turn it into OO form and then extends the Thread (or Runnable) such like:
public class URLProcessor extends Thread {
private String url;
public URLProcessor(String url) {
this.url = url;
}
#Override
public void run() {
//your business logic to parse the site with "this.url" here
}
}
And then use the main entrance to load multiple ones by using:
public static void main(String[] args) {
List<String> allmyurls = null;//get multiple urls from somewhere
for (String url : allmyurls) {
URLProcessor p = new URLProcessor(url);
p.start();
}
}