I have 3rd party code that I connect to via DataInputStream. The 3rd party code continually spits out information as it generates it. When something of interest comes across I want to pass it along to GraphQL Subscription(s)
I'm not sure how to wire the 3rd party code to the server-side GraphQL subscription code given this scenario. Any suggestions would be appreciated.
Some conceptual code is below:
public void liveStream(DataInputStream in) {
// Sit and constantly watch input stream and report when messages come in
while(true) {
SomeMessage message = readFromInputStream(in);
System.out.println("Received Message Type:" + message.getType());
// Convert SomeMessage into the appropriate class based on its type
if (message.getType() == "foo") {
Foo foo = convertMessageToFoo(message);
} else if (message.getType() == "bar") {
Bar bar = convertMessageToBar(message);
} else if (howeverManyMoreOfThese) {
// Keep converting to different objects
}
}
}
// The client code will eventually trigger this method when
// the GraphQL Subscription query is sent over
VertxDataFetcher<Publisher<SomeClassTBD>> myTestDataFetcher() {
return new VertxDataFetcher<> (env, future) -> {
try {
future.complete(myTest());
} catch(Exception e) {
future.fail(e);
}
});
}
OK, I wrapped my liveStream code in an ObservableOnSubscribe using an executorService and I'm getting back all the data. I guess I can now either pass it straight through to the front end or create separate publishers to deal with specific object types and have graphql subscriptions point to their respective publishers.
ExecutorService executor = Executors.newSingleThreadExecutor;
ObservableOnSubscribe<SomeClassTBD> handler = emitter ->
executor.submit(() -> {
try {
//liveStream code here
emitter.onComplete();
}
catch(Exception e) {
emitter.onError(e);
}
finally {
// Cleanup here
}
});
Observable<SomeClassTBD> = Observable.create(handler);
Related
I am using software.amazon.awssdk version 2.18.21 to invoke Lambda function from spring boot application. While invoking Lambda function which takes approx 2-3 minutes to finish throws Http status 504: Gateway Time-out Exception.
I have been suggested to use Asynchronous Call to invoke Lambda function and then read the response. How can i convert this existing code to asynchronous call and get response to verify if it was success or error?
//imports from
import software.amazon.awssdk.http.ApacheHttpClient;
import software.amazon.awssdk.services.lambda.LambdaClient;
import software.amazon.awssdk.services.lambda.model.InvokeRequest;
import software.amazon.awssdk.services.lambda.model.InvokeResponse;
//Calling Lambda Function
try{
LambdaClient client = LambdaClient.builder().httpClientBuilder(ApacheHttpClient.builder()
.maxConnections(100)
.socketTimeout(Duration.ofSeconds(60))
.connectionTimeout(Duration.ofSeconds(60))
).build();
InvokeRequest req = InvokeRequest.builder().functionName("abc").build();
InvokeResponse res = client.invoke(req);
String respnonse = res.payload().asUtf8String();
System.out.println(response);
}catch(Exception e){
e.printStackTrace();
}
Edited: Tried below but unable to implement CompletableFuture. Can you suggest how to implement it for Lambda function and get response
try{
SdkAsyncHttpClient client = NettyNioAsyncHttpClient.builder().readTimeout(Duration.ofSeconds(60)).connectionTimeout(Duration.ofSeconds(60)).build();
LambdaAsyncClient lambdaClient = LambdaAsyncClient .builder().httpClient(client).build();
InvokeRequest req = InvokeRequest.builder().functionName("abc").invocationType("EVENT").build();
CompletableFuture<InvokeResponse> request = lambdaClient.invoke(req);
//InvokeResponse res = client.invoke(req);
//String respnonse = res.payload().asUtf8String();
//System.out.println(response);
}catch(Exception e){
e.printStackTrace();
}
To perform Lambda operations using Asynchronous calls with the Java V2 API, use this:
LambdaAsyncClient
The call the invoke method:
https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/lambda/LambdaAsyncClient.html#invoke(software.amazon.awssdk.services.lambda.model.InvokeRequest)
The patten to use Async methods are similar no matter what the service use. For example, most calls involve using CompletableFuture. Then you code the logic you want within the whenComplete code block.
To learn about the patterns, read this part of the Java V2 DEV Guide:
Asynchronous programming
For example, this code example is from that topic.
public class DynamoDBAsyncListTables {
public static void main(String[] args) throws InterruptedException {
// Create the DynamoDbAsyncClient object
Region region = Region.US_EAST_1;
DynamoDbAsyncClient client = DynamoDbAsyncClient.builder()
.region(region)
.build();
listTables(client);
}
public static void listTables(DynamoDbAsyncClient client) {
CompletableFuture<ListTablesResponse> response = client.listTables(ListTablesRequest.builder()
.build());
// Map the response to another CompletableFuture containing just the table names
CompletableFuture<List<String>> tableNames = response.thenApply(ListTablesResponse::tableNames);
// When future is complete (either successfully or in error) handle the response
tableNames.whenComplete((tables, err) -> {
try {
if (tables != null) {
tables.forEach(System.out::println);
} else {
// Handle error
err.printStackTrace();
}
} finally {
// Lets the application shut down. Only close the client when you are completely done with it.
client.close();
}
});
tableNames.join();
}
}
I am implementing tests for my Vert.x application, but I am having issues in making Vert.x wait for the deploy of the Verticle in a graceful way.
This is my #BeforeClass Method:
#BeforeClass
public static void before(TestContext context)
{
vertx = Vertx.vertx();
DeploymentOptions options = new DeploymentOptions();
byte[] encoded;
JsonObject config;
try {
encoded = Files.readAllBytes(Paths.get("src/main/resources/config.json"));
config = new JsonObject(new String(encoded, Charset.defaultCharset()));
options.setConfig(config);
jdbc = JDBCClient.createShared(vertx, config , "PostgreSQL");
deployVerticle((result) -> loadTestData((result), jdbc), options);
while (true)
{
if (vertx.deploymentIDs().size() > 0)
break;
}
} catch
(IOException e)
{
e.printStackTrace();
}
}
Also, here is the implementation for the deployVerticle and loadTestData methods:
private static void deployVerticle(Handler<AsyncResult<Void>> next, DeploymentOptions options) {
vertx.deployVerticle(PostgreSQLClientVerticle.class.getName(), options, deployResult ->
{
if (deployResult.succeeded())
next.handle(Future.succeededFuture());
});
}
private static void loadTestData(AsyncResult<Void> previousOperation, JDBCClient jdbc)
{
if (previousOperation.succeeded())
{
jdbc.getConnection(connection -> {
if (connection.succeeded())
{
connection.result().query(deleteTestDataGeneration, queryResult ->
{
connection.result().close();
});
}
});
}
}
As you can see, right now I have a while (true) on the beforemethod to hold the process and make sure the verticle is actually deployed.
Otherwise, when the tests start running, the verticle is not yet fully deployed and I get a NullPointerException trying to reach the resources.
I've tried many different approaches like using CompositeFuture or using Future.compose method to make the "before tasks" sequential and make the program hold for completion.
I achieved in making those tasks sequential but failed on holding the process until they are completed.
One of the issues is, I think, the fact that the deployVerticle method returns the AsyncResult with succeeded == true after every step of the "deploy procedure" is done, instead of when the Verticle is totally up.
Meaning that the process gets a successful result before everything is actually up...but this is just a wild guess.
Bottom-line: I would like to find a way to wait for the Verticle to be totally deployed before proceeding to perform the tests, without having to do the while (true)loop that I currently have in there.
What you are missing is the Async async = context.async();. With that the unittest stays in the method until it is not set to complete. Then you are able to orchestrate your asychronous code to:
first deploy the verticle
then execute the loadtestGeneration
set the async to complete so that, the other unittest methods already can access to your verticle without nullpointerexception
I also made some cleanup, check it out:
BeforeClass
#BeforeClass
public static void before2(TestContext context){
Async async = context.async();
vertx = Vertx.vertx();
DeploymentOptions options = new DeploymentOptions();
byte[] encoded;
JsonObject config;
try {
encoded = Files.readAllBytes(Paths.get("src/main/resources/config.json"));
config = new JsonObject(new String(encoded, Charset.defaultCharset()));
options.setConfig(config);
jdbc = JDBCClient.createShared(vertx, config , "PostgreSQL");
deployVerticle2(options)
.compose(c -> loadTestData2(jdbc))
.setHandler(h -> {
if(h.succeeded()){
async.complete();
}else{
context.fail(h.cause());
}
});
} catch (IOException e){
context.fail(e);
}
}
DeployVerticle
private static Future<Void> deployVerticle2(DeploymentOptions options) {
Future<Void> future = Future.future();
vertx.deployVerticle(PostgreSQLClientVerticle.class.getName(), options, deployResult -> {
if (deployResult.failed()){
future.fail(deployResult.cause());
}else {
future.complete();
}
});
return future;
}
LoadTestData
private static Future<Void> loadTestData2(JDBCClient jdbc){
Future<Void> future = Future.future();
jdbc.getConnection(connection -> {
if (connection.succeeded()) {
connection.result().query(deleteTestDataGeneration, queryResult -> {
if(queryResult.failed()){
connection.result().close();
future.fail(queryResult.cause());
}else{
connection.result().close();
future.complete();
}
});
} else {
future.fail(connection.cause());
}
});
return future;
}
I have to make calls to long running methods in a Spark API implementation. These methods return CompletableFutures, so I'd like to free up the current Thread by triggering Spark to answer the client request in a callback.
As far as I can tell this is not possible with Spark, but I'd like to make sure I am not overlooking anything.
To illustrate the question, see the small code sample below.
import spark.Spark;
import java.util.concurrent.CompletableFuture;
public class HelloSpark {
public static void main(String[] args) {
Spark.get("/what_i_have_to_do", (req, res) -> {
CompletableFuture<String> f = callToSlowWorker();
return f.get();
});
Spark.get("/what_i_would_like_to_do", (req, res) -> {
CompletableFuture<String> f = callToSlowWorker();
f.whenComplete((answer, throwable) -> {
if(throwable != null){
// send error to requesting client
res.status(500);
res.body(throwable.getMessage());
} else {
// send answer to requesting client
res.body(answer);
}
// trigger spark to return the response to the client
// ...?!?
});
return null; // obviously not what I want, just to make this sample code compile
});
}
static CompletableFuture<String> callToSlowWorker(){
return CompletableFuture.supplyAsync(() -> {
try {
Thread.sleep(1000);
} catch (InterruptedException ignored) {
}
return "Hello World";
});
}
}
SparkJava is currently blocking only, thus what you've described is not possible. There is an open enhancement request to add support for a non blocking API.
Hi.
I'm coding a website parser that is aimed to be fast and thus multi-threaded.
The external libraries I'm using are: apache HTTP client, Jsoup (for HTML parsing) and GPars (for message-driven threads).
Now I'll show some concept that I'm trying to implement
static StaticDispatchActor<String> httpActor;
public static void main(String[] args) throws ClientProtocolException, IOException {
int numThreads = 25;
try{
numThreads = Integer.parseInt(args[0]);
} catch (Exception e){
System.out.println("Number of threads defaulted to "+numThreads);
}
final int numberOfThreads = numThreads;
final ExecutorService threadpool = Executors.newCachedThreadPool();
final Async async = Async.newInstance().use(threadpool);
AtomicInteger jobCount = new AtomicInteger(0);
//.....
// This is a parser itself which parses usernames out of every page.
Actor jsoupUser = new StaticDispatchActor<String>(){ // actor to parse users
HashSet<String> users = new HashSet<>(); // found users
public void onMessage(String html){ // takes html -> adds parsed users 2 set
users.addAll(Jsoup.parse(html)
.select("a[href*=/profile/]").stream() // select links
.map(e -> e.text()) // extract usernames
.filter(s -> s.length() > 0) // empty lines -> out
.collect(Collectors.toSet()));
System.out.print("Users: "+users.size()+", Jobs: "+jobCount.get()+"\r");
}
}.start();
// This actor shall extract new links to parse out of every page
Actor jsoupLinker = new StaticDispatchActor<String>(){ // link extractor
HashSet<String> usedLinks = new HashSet<>(); // already found links
public synchronized void removeBack(String url){
#Override
public void onMessage(String html) {
Set<String> links = Jsoup.parse(html).select("a[href]").stream().parallel()
.map(e -> e.attr("href").replace("#comments", "")// here also some replacements...
)
.filter(s -> (!usedLinks.contains(s) && /* other filters */ )
.collect(Collectors.toSet());
links.forEach(url -> httpActor.send(url)); // send to process new URLs
}
}.start(); // start actor
// this actor is the processor of new links and where the error comes in:
httpActor = new StaticDispatchActor<String>(){ // process responses async
public void onMessage(String url) {
try{
while(jobCount.get()>numberOfThreads); // wait for running threads to be less than wanted value; without this number of running jobs goes out of any control
async.execute(Request.Get(defaultWebSiteUrl+url), new FutureCallback<Content>(){ #Override // do request and process async
public void completed(Content c) {
jobCount.decrementAndGet();
try{
String s = c.asString();
jsoupUser.send(s);
jsoupLinker.send(s);
} catch (OutOfMemoryError e1){
System.out.println("out of my memory, "); // This is the thrown error the question is about - [1]
}
}
#Override public void failed(Exception e) {
jobCount.decrementAndGet();
try {
throw e;
} catch (ConnectException e4){ // if the request is timed out resend it
httpActor.send(url);
System.out.println("resent\r\n");
} catch (HttpResponseException e0){
} catch (Exception e1) { // for all other exceptions
e1.printStackTrace();
}
}
#Override public void cancelled() {
jobCount.decrementAndGet(); // never done actually
}
});
jobCount.incrementAndGet();
} catch (IllegalArgumentException e3){
System.out.println("some illigal shit");
}
}
};
httpActor.start();
Now the problem is, although I limited a number of running jobs, my code somehow goes out of memory (search for [1] in the code to see where).
Maybe you have any idea on how to resolve it. Or there is some showcase for similar task, because I fill very wrong about the whole application design and maybe I should change it at all?
Thank you.
So, using BiziClop's tip, I was able to figure out the mistake.
If some is interested, I, as you can see above, was sending HTML code received from server as a string to 2 different actors and than, inside these actors, parsing them. That was the cause for all out-of-memory errors, since these HTML pages are pretty big especially given how many of them are waiting to be processed in the message queue.
The solution I used is just parse the document and select needed elements and transfer their list to matching actors for further processing.
Document doc = Jsoup.parse(c.asString());
jsoupUser.send(doc.select("a[href*=/profile/]"));
jsoupLinker.send(doc.select("a[href]"));
Still, if any one has anything to say about how to improve the algorithm I'll really appreciate it.
I'm sending files to my local server that creates a file back. My problem occurs when the user perform multiple actions one after another and I need to show an error message if one of the requests don't get a feedback file in 5 min.
How can I handle all these requests? I used newSingleThreadScheduledExecutor to check if the feedback file is there every minute but I don't know how to handle multiple ones and keep the countdown to each request for the 5 min case.
My try:
ListeningExecutorService executor = MoreExecutors.listeningDecorator(Executors.newFixedThreadPool(listPrinter.size()));
for(int i=0;i<list.size();i++){
try {
final File retrievedFile = new File("/home/"+list.get(i)+".csv");
ListenableFuture<File> future = executor.submit(new Callable<File>() {
public File call() {
// Actually send the file to your local server
// and retrieve a file back
if(retrievedFile.exists())
{
new Notification("file exits").show(Page.getCurrent());
}
else{
new Notification("file no exits").show(Page.getCurrent());
}
return retrievedFile;
}
});
future.get(5, TimeUnit.MINUTES);
} catch (InterruptedException ex) {
Exceptions.printStackTrace(ex);
} catch (ExecutionException ex) {
Exceptions.printStackTrace(ex);
} catch (TimeoutException ex) {
Exceptions.printStackTrace(ex);
new Notification("Time out").show(Page.getCurrent());
}
}
But it just get executed at the beginning and that's it but when the file is added nothing happens.
Is it possible to do this with watchService? It works pretty well for me but I didn't know about the 5 min case
Take a look to the Future interface:
http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Future.html
should fit perfectly to your problem.
When you run a thread, the result could be a Future, it is the result of a asyncronous task, and you can have one Future per asyncronous task that you are launching.
Future<File> sendReceiveFile(File inputFile) {
final Future<File> future = new YourFuture<File>(...);
new Thread() {
#Override
public void run() {
File outputFile = null;
try {
outputFile = SendFileToServer(inputFile);
} catch (final Exception e) {
// do something
} finally {
future.setValue(fileOutput);
}
}
}.start();
return future;
}
And in your main:
Future<File> future = sendReceiveFile(myFile);
File outputFile = null;
try {
outputFile = future.get(1, TimeUnit.MINUTE);
} catch(TimeOutException e) {
// do something
}
You could do this manually, but using Guava ListenableFuture would be much better:
// Here we create a fixed thread pool with 10 threads and an inifinite-capacity queue
ListeningExecutorService executor = MoreExecutors.listeningDecorator(Executors.newFixedThreadPool(10));
final File fileToSend = ...; //
ListenableFuture<File> future = executor.submit(new Callable<File>() {
public File call() {
// Actually send the file to your local server
// and retrieve a file back
File retrievedFile = YourLocalServer.sendAndRetrieve(fileToSend);
return retrievedFile;
}
});
Futures.addCallback(future, new FutureCallback<File>() {
public void onSuccess(File retrievedFile) {
// Handle the successfully retrieved file when it returns
}
public void onFailure(Throwable thrown) {
// Handle the error
}
});
By sending the file asynchronously, you can send and retrieve many files at any given time. Then, when the server responds (either with a retrieved file or with an error), you can handle the response (retrieved file or exception) just when it comes back, without needing to wait for it. This means that the onSuccess() or onFailure() methods will be automatically executed when there's a response available from your local server.
I solved the problem by using a Timer that is executed every 5 minutes getting all the db transactions that happened for the last 5 minutes and didn't get any response and show my error code. It works pretty good. Thanks everyone for the help