I'm new to vert.x and would like to know if its possible to configure eventbus somehow to make it work consistently?
I mean need to send requests one by one using vert.x
At the moment I got this code which uses eventloop principle and waits until all handlers finished, but I don't need this done that fast, idea is to free server from lots of requests at the same time. Here eb_send() uses default EventBus.send() method. In other words I want to execute all requests with blocking, waiting for answers before requests.
List<Future> queue = new ArrayList<>();
files.forEach(fileObj -> {
Future<JsonObject> trashStatusHandler = Future.future();
queue.add(trashStatusHandler);
eb_send(segment, StorageType.getAddress(StorageType.getStorageType(fileInfo.getString("storageType"))) + ".getTrashStatus", fileInfo, reply -> {
Entity dummy = createDummySegment();
try {
if (reply.succeeded()) {
//succeded
}
} catch (Exception ex) {
log.error(ex);
}
trashStatusHandler.complete();
});
});
The basic idea is to extract this into a function, which you would invoke recursively.
public void sendFile(List<File> files, AtomicInteger c) {
eb_send(segment, StorageType.getAddress(StorageType.getStorageType(fileInfo.getString("storageType"))) + ".getTrashStatus", fileInfo, reply -> {
Entity dummy = createDummySegment();
try {
if (reply.succeeded()) {
//succeded
}
// Recursion
if (c.incrementAndGet() < files.size()) {
sendFile(files, c);
}
} catch (Exception ex) {
log.error(ex);
}
});
}
Related
I have a Server that can receive multiple request at the same time.
In my Server, I have to make some traitement and wait for response. This traitmenet is done by externe library so I don't how much should I wait.
So the Server looks like :
public class MyServer{
#Override
//method from the library
public void workonRequest(){
//---
response=[...]
}
public void listenRequest() {
new Thread(() -> {
while (true) {
try {
socket = server.accept();
ObjectInputStream input = new ObjectInputStream(socket.getInputStream());
ObjectOutputStream output = new ObjectOutputStream(socket.getOutputStream());
socket.setTcpNoDelay(true); //TODO : Not sure !
new Thread(() -> {
try {
handleRequest(input, output);
} catch (IOException e) {
throw new RuntimeException(e);
}
}).start();
} catch (IOException ex) {
System.out.println(ex.getMessage());
}
}
}).start();
}
And the handle request method is :
public void handleRequest(ObjectInputStream input, ObjectOutputStream output) throws IOException {
try {
while (true) {
//forward the request to the library
//work on it [means using the library and waiting]
// return response
}
}
}
The response object is the result that I want return to the client
How to deal with the problem of waiting for the answer?
How can I make sure that there will be no problems when more than 2 clients send requests at the same time.
Thanks in advance
How to deal with the problem of waiting for the answer ?###
Using while(true) can create issues because you are blocking the thread and opening sub thread and multi streams will make it more complex. There is easy way called reactive programming which handles this kind of multi-threaded issues easily, quarkus async solution and spring, if you still want to manage your sockets from java code you can use akka
How can I make sure that there will be no problems when more than 2 clients send requests at the same time.
That can be done by not blocking the main thread and If you manage to use reactive and/or async approach you will not have that problem.
Reference
https://quarkus.io/guides/getting-started-reactive
https://docs.spring.io/spring-framework/docs/current/reference/html/web-reactive.html
I have a Kafka consumer that performs some migrations. Pretty simple flow
#KafkaListener(topics="client-migration-blah", groupId="migration-group",
containerFactory="kafkaListnerContainerFactory")
public void consume(ConsumerRecord<String, Object> payload) {
try {
Client client = payload.value();
if( migrationClient.clientExists(client)) {
updateClient(ClientEvent.UPDATE, client);
}
else {
migrationClient.importClient(ClientEvent.CREATE, client);
}
} catch (Exception ex) {
log.error("yada yada yada");
sendToDLQ(ClientEvent.ERROR, client);
}
}
I need a breakup of the three usecases, create, update and error(DLQ). Short of building a streaming (?) solution to collect these aggregates, what would be a simple way to gather these events to extract the breakup?
I'm trying to implement a Pulsar client with multiple producers that distributes the load among the threads, but regardless the value passed on ioThreads() and on listenerThreads(), it is always overloading the first thread (> 65% cpu while the other threads are completely idle)
I have tried a few things including this "dynamic rebalancing" every hour(last method) but closing it in the middle of the process certainly is not the best approach
This is the relevant code
...
// pulsar client
pulsarClient = PulsarClient.builder() //
.operationTimeout(config.getAppPulsarTimeout(), TimeUnit.SECONDS) //
.ioThreads(config.getAppPulsarClientThreads()) //
.listenerThreads(config.getAppPulsarClientThreads()) //
.serviceUrl(config.getPulsarServiceUrl()).build();
...
private createProducers() {
String strConsumerTopic = this.config.getPulsarTopicInput();
List<Integer> protCasesList = this.config.getEventProtoCaseList();
for (Integer e : protCasesList) {
String topicName = config.getPulsarTopicOutput().concat(String.valueOf(e));
LOG.info("Creating producer for topic: {}", topicName);
Producer<byte[]> protobufProducer = pulsarClient.newProducer().topic(topicName).enableBatching(false)
.blockIfQueueFull(true).compressionType(CompressionType.NONE)
.sendTimeout(config.getPulsarSendTimeout(), TimeUnit.SECONDS)
.maxPendingMessages(config.getPulsarMaxPendingMessages()).create();
this.mapLink.put(strConsumerTopic.concat(String.valueOf(e)), protobufProducer);
}
}
public void closeProducers() {
String strConsumerTopic = this.config.getPulsarTopicInput();
List<Integer> protCasesList = this.config.getEventProtoCaseList();
for (Integer e : protCasesList) {
try {
this.mapLink.get(strConsumerTopic.concat(String.valueOf(e))).close();
LOG.info("{} producer correctly closed...",
this.mapLink.get(strConsumerTopic.concat(String.valueOf(e))).getProducerName());
} catch (PulsarClientException e1) {
LOG.error("Producer: {} not closed cause: {}",
this.mapLink.get(strConsumerTopic.concat(String.valueOf(e))).getProducerName(),
e1.getMessage());
}
}
}
public void rebalancePulsarThreads(boolean firstRun) {
ThreadMXBean threadHandler = ManagementFactory.getThreadMXBean();
ThreadInfo[] threadsInfo = threadHandler.getThreadInfo(threadHandler.getAllThreadIds());
for (ThreadInfo threadInfo : threadsInfo) {
if (threadInfo.getThreadName().contains("pulsar-client-io")) {
// enable cpu time for all threads
threadHandler.setThreadCpuTimeEnabled(true);
// get cpu time for this specific thread
long threadCPUTime = threadHandler.getThreadCpuTime(threadInfo.getThreadId());
int thresholdCPUTime = 65;
if (threadCPUTime > thresholdCPUTime) {
LOG.warn("Pulsar client thread with CPU time greater than {}% - REBALANCING now", thresholdCPUTime);
try {
closeProducers();
} catch (Exception e) {
if (!firstRun) {
// producers will not be available in the first run
// therefore, the logging only happens when it is not the first run
LOG.warn("Unable to close Pulsar client threads on rebalancing: {}", e.getMessage());
}
}
try {
createPulsarProducers();
} catch (Exception e) {
LOG.warn("Unable to create Pulsar client threads on rebalancing: {}", e.getMessage());
}
}
}
}
}
From what you describe, the most likely scenario is that all the topics you're using are served by one single broker.
If that's indeed the case, and avoiding topic load balancing across brokers, it's normal that it's using a single thread because all these producers will be sharing a single, pooled, TCP connection and each connection is assigned to 1 IO thread (listener threads are used for consumer listeners).
If you want to force more threads, you can increase the "Max TCP connection per each broker" setting, in order to use all the configured IO threads.
eg:
PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar://localhost:6650")
.ioThreads(16)
.connectionsPerBroker(16)
.create();
I want to get an idea of using ManagedExecutorService in stateless bean. Basically i am trying to send a http call in a separate thread inside my j2EE application. executorService send this request and wait for x number of seconds to receive response, if no response comes in specified seconds OR get exeception then do another try(X times) and then finally give a feedback that either https service call successfully done or failed. Here is my code
#SuppressWarnings("EjbEnvironmentInspection")
#Resource
ManagedExecutorService executorService;
public static final long RETRY_DELAY = 3000;
public static final int MAX_RETRIES = 3;
executorService.execute(() -> {
int retry = 0;
Collection<Info> responseInfo = null;
while (responseInfo == null && retry++ < MAX_RETRIES) {
try {
responseInfo = httpsService.requestAccessInfo(requestInfo);
Thread.sleep(RETRY_DELAY);
} catch (Exception e) {
log.error("Error while receiving response retry attempt {}", retry);
}
}
boolean status = filledLockAccessInfo==null ? false : true;
event.fire(regularMessage(status,GENERATION_RESULT);
});
Can someone tell me is it a right way to do this OR not.
You shouldn't need to forcibly sleep (Thread.sleep(RETRY_DELAY);). What you need is an asynchronous invocation of the service that can support timeout.
The following two methods use the completable future API's timeout and error handling to implement that.
The following use recursion to retry the given number of times:
private static Collection<Info> callService(int retryCount) {
try {
CompletableFuture<Collection<Info>> f = invoke();
return f.get(RETRY_DELAY, TimeUnit.MILLISECONDS);
}catch(TimeoutException te) {
if(retryCount > 0) {
return callService(retryCount - 1);
} else {
throw new RuntimeException("Fatally failed!!");
}
} catch(Exception ee) {
throw new RuntimeException("Unexpectedly failed", ee);
}
}
Note that the executorService object is passed in the second argument of supplyAsync
private static CompletableFuture<Collection<Info>> invoke() {
return CompletableFuture.supplyAsync(() -> {
//call
return httpsService.requestAccessInfo(requestInfo);;
}, executorService);
}
With that, you can simply call it with the number of retries:
Collection<Info> responseInfo = callService(MAX_RETRIES);
To make the above call run asynchronously, you can replace the preceding statement with:
CompletableFuture.supplyAsync(() -> callService(MAX_RETRIES))
.thenAccept(res -> System.out.println("Result: " + res));
This will make the call in the background. Later, you can check how it completed:
f.isCompletedExceptionally() //will tell whether it completed with an exception.
In my KafkaConsumer app I want to read a batch of messages with poll() and process them. But processing may fail. In this case I want to retry until I succeed but only retry if consumer still owns partitions. I don't want to constantly call poll() because I don't want to read more data.
This is a code snippet:
consumer = new KafkaConsumer<>(consumerConfig);
try {
consumer.subscribe(config.topics() /** Callback does not work as I do not call poll in between */ );
while (true) {
ConsumerRecords<byte[], Value> values = consumer.poll(10000);
while (/* I am still owner of partitions */) {
try {
process(values);
} catch (Exception e) {
log.error("I dont care, just retry while I own the partitions", e)
}
}
}
} catch (WakeupException e) {
// shutting down
} finally {
consumer.close();
}
There is a callback method that tells you when your consumers partition assignments are about to be revoked. Keep processing message unless you get an onPartitionRevoked() event.
https://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/consumer/ConsumerRebalanceListener.html#onPartitionsRevoked(java.util.Collection)
What about simply calling assignment() ?
http://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#assignment()
I came to a conclusion that it is impossible to call poll() without reading messages with current kafka consumer 10.2.x. However, it is possible to update offset after a processing failure. So I update offset as if the messages were never read
while (!stopped) {
ConsumerRecords<byte[], Value> values = consumer.poll(timeout);
try {
process(values);
} catch (Exception e) {
rewind(records);
// Ensure a delay after errors to let dependencies recover
Thread.sleep(delay);
}
}
and rewind method is
private void rewind(ConsumerRecords<byte[], Value> records) {
records.partitions().forEach(partition -> {
long offset = records.records(partition).get(0).offset();
consumer.seek(partition, offset);
});
}
It solves the initial problem