HtmlUnit WebClient Timeout - java

In my previous questions about HtmlUnit
Skip particular Javascript execution in HTML unit
and
Fetch Page source using HtmlUnit : URL got stuck
I had mentioned that URL is getting stuck. I also found out that it is getting stuck due to one of the methods(parse) in HtmlUnit library is not coming out of execution.
I did further work on this. I wrote code to get out of the method if it takes more than specified time-out seconds to complete.
import java.io.IOException;
import java.net.MalformedURLException;
import java.util.Date;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import com.gargoylesoftware.htmlunit.BrowserVersion;
import com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException;
import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
public class HandleHtmlUnitTimeout {
public static void main(String[] args) throws FailingHttpStatusCodeException, MalformedURLException, IOException, InterruptedException, TimeoutException
{
Date start = new Date();
String url = "http://ericaweiner.com/collections/";
doWorkWithTimeout(url, 60);
}
public static void doWorkWithTimeout(final String url, long timeoutSecs) throws InterruptedException, TimeoutException {
//maintains a thread for executing the doWork method
ExecutorService executor = Executors.newFixedThreadPool(1);
//logger.info("Starting method with "+timeoutSecs+" seconds as timeout");
//set the executor thread working
final Future<?> future = executor.submit(new Runnable() {
public void run()
{
try
{
getPageSource(url);
}
catch (Exception e)
{
throw new RuntimeException(e);
}
}
});
//check the outcome of the executor thread and limit the time allowed for it to complete
try {
future.get(timeoutSecs, TimeUnit.SECONDS);
} catch (Exception e) {
//ExecutionException: deliverer threw exception
//TimeoutException: didn't complete within downloadTimeoutSecs
//InterruptedException: the executor thread was interrupted
//interrupts the worker thread if necessary
future.cancel(true);
//logger.warn("encountered problem while doing some work", e);
throw new TimeoutException();
}finally{
executor.shutdownNow();
}
}
public static void getPageSource(String productPageUrl)
{
try {
if(productPageUrl == null)
{
productPageUrl = "http://ericaweiner.com/collections/";
}
WebClient wb = new WebClient(BrowserVersion.FIREFOX_3_6);
wb.getOptions().setTimeout(120000);
wb.getOptions().setJavaScriptEnabled(true);
wb.getOptions().setThrowExceptionOnScriptError(true);
wb.getOptions().setThrowExceptionOnFailingStatusCode(false);
HtmlPage page = wb.getPage(productPageUrl);
wb.waitForBackgroundJavaScript(4000);
wb.closeAllWindows();
}
catch (FailingHttpStatusCodeException e)
{
e.printStackTrace();
}
catch (MalformedURLException e)
{
e.printStackTrace();
}
catch (IOException e)
{
e.printStackTrace();
}
}
}
This code does come out of doWorkWithTimeout(url, 60); method. But this does not terminate.
When I try to call similiar implementation with following code:
import java.util.Date;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import org.apache.log4j.Logger;
public class HandleScraperTimeOut {
private static Logger logger = Logger.getLogger(HandleScraperTimeOut .class);
public void doWork() throws InterruptedException {
logger.info(new Date()+ "Starting worker method ");
Thread.sleep(20000);
logger.info(new Date()+ "Ending worker method ");
//perform some long running task here...
}
public void doWorkWithTimeout(int timeoutSecs) {
//maintains a thread for executing the doWork method
ExecutorService executor = Executors.newFixedThreadPool(1);
logger.info("Starting method with "+timeoutSecs+" seconds as timeout");
//set the executor thread working
final Future<?> future = executor.submit(new Runnable() {
public void run()
{
try
{
doWork();
}
catch (Exception e)
{
throw new RuntimeException(e);
}
}
});
//check the outcome of the executor thread and limit the time allowed for it to complete
try {
future.get(timeoutSecs, TimeUnit.SECONDS);
} catch (Exception e) {
//ExecutionException: deliverer threw exception
//TimeoutException: didn't complete within downloadTimeoutSecs
//InterruptedException: the executor thread was interrupted
//interrupts the worker thread if necessary
future.cancel(true);
logger.warn("encountered problem while doing some work", e);
}
executor.shutdown();
}
public static void main(String a[])
{
HandleScraperTimeOut hcto = new HandleScraperTimeOut ();
hcto.doWorkWithTimeout(30);
}
}
If anybody can have a look and tell me what is the issue, it will be really helpful.
For more details about issue, you can look into Skip particular Javascript execution in HTML unit
and
Fetch Page source using HtmlUnit : URL got stuck
Update 1
Strange thing is : future.cancel(true); is returning TRUE in both cases.
How I expected it to be was :
With HtmlUnit it should return FALSE since process is still hanging.
With normal Thread.sleep(); it should return TRUE since the process
got cancelled successfully.
Update 2
It only hangs with http://ericaweiner.com/collections/ URL. If I give any other URL i.e. http://www.google.com , http://www.yahoo.com , It does not hand. In these cases it throws IntruptedException and come out of the Process.
It seems that http://ericaweiner.com/collections/ page source has certain elements which are causing problems.

Future.cancel(boolean) returns:
false if the task could not be cancelled, typically because it has already completed normally
true otherwise
Cancelled means means the thread did not finish before cancel, the canceled flag was set to true and if requested the thread was interrupted.
Interrupt the thread menans it called Thread.interrupt and nothing more. Future.cancel(boolean) does not check if the thread actually stopped.
So it is right that cancel return true on that cases.
Interrupting a thread means it should stop as soon as possible but it is not enforced. You can try to make it stop/fail closing a resource it needs or something. I usually do that with a thread reading (waiting incoming data) from a socket. I close the socket so it stops waiting.

Related

Java Completable future thread are alive after method execute

I have written a small program to check behavior of Completable Future. I have not overridden the common pool.
I did not found any shut down method and when i print active number of thread at the end, i found my thread active.
My question is when they will end, if i am using it in life application?
And do they create to many thread if i use it in public Api, who have much traffic?
My samlple code
`
package rar;
import java.util.Arrays;
import java.util.List;
import java.util.Objects;
import java.util.Set;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.stream.Collectors;
public class Rar {
public static void main(String[] args) {
Rar r = new Rar();
Set<Thread> threadSet = Thread.getAllStackTraces().keySet();
System.out.println(threadSet); r.dfo(); threadSet =
Thread.getAllStackTraces().keySet(); System.out.println("uyuuuu"+threadSet);
}
private static void doTask3() {
for(int i=0; i<5;i++) {
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.print(3);
}
}
public void dfo() {
System.out.println("In main");
ExecutorService executor = Executors.newFixedThreadPool(3);
CompletableFuture<Void> thenCompose =CompletableFuture.allOf(
CompletableFuture.runAsync(() -> doTask1()),
CompletableFuture.runAsync(() -> doTask2(2)),
CompletableFuture.runAsync(() -> doTask3()));
//executor.shutdown();
try {
Set<Thread> threadSet = Thread.getAllStackTraces().keySet();
System.out.println(threadSet);
thenCompose.get();
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
}
System.out.println("Exiting main");
Set<Thread> threadSeWt = Thread.getAllStackTraces().keySet();
System.out.println(threadSeWt);
}
private void doTask2(int num) {
for(int i=0; i<5;i++) {
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.print("4");
}
}
private int doTask1() {
for(int i=0; i<5;i++) {
try {
Thread.sleep(5001);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.print(1);
}
return 5;
}
}
`
Sample Output:
[Thread[Finalizer,8,system], Thread[Attach Listener,5,system], Thread[Signal Dispatcher,9,system], Thread[Reference Handler,10,system], Thread[main,5,main]]
In main
[Thread[Finalizer,8,system], Thread[ForkJoinPool.commonPool-worker-1,5,main], Thread[main,5,main], Thread[ForkJoinPool.commonPool-worker-2,5,main], Thread[Attach Listener,5,system], Thread[ForkJoinPool.commonPool-worker-3,5,main], Thread[Signal Dispatcher,9,system], Thread[Reference Handler,10,system]]
431413431431431Exiting main
[Thread[Finalizer,8,system], Thread[ForkJoinPool.commonPool-worker-1,5,main], Thread[main,5,main], Thread[ForkJoinPool.commonPool-worker-2,5,main], Thread[Attach Listener,5,system], Thread[ForkJoinPool.commonPool-worker-3,5,main], Thread[Signal Dispatcher,9,system], Thread[Reference Handler,10,system]]
uyuuuu[Thread[Finalizer,8,system], Thread[ForkJoinPool.commonPool-worker-1,5,main], Thread[main,5,main], Thread[ForkJoinPool.commonPool-worker-2,5,main], Thread[Attach Listener,5,system], Thread[ForkJoinPool.commonPool-worker-3,5,main], Thread[Signal Dispatcher,9,system], Thread[Reference Handler,10,system]]
The executor in your dfo() method is not used.
When you use runAsync() method, it runs on the common pool. That's why you see ForkJoinPool in your debug messages.
The size of that pool is limited by default by "number of your CPU cores - 1".
Try running 20 tasks, and you'll see that your thread count stops growing, once it reaches the maximum.
You don't need to stop the threads of the ForkJoinPool. From the documentation:
its threads are slowly reclaimed during periods of non-use

Return to main after thread has ended

I am currently messing around with facial recognition and try to capture photos form my webcam. I am adapting this tutorial to automatically name and save the taken picture. Note that this Code is called from the main() the funtion it self is implemented in another class. Up and until I stop the thread with interrupt() it works. Afterwards the picture is frozen in the GUI and the system doesn't seem to return into the class where I operate over my GUI.
To make myself clear: I want to capture a picture from my webcam and replace the former stream from my webcam with said earlier captured picture. Up and until I capture the picture and interrupt the thread the code works. Aftwards it is stuck.
I experiemented with pulling the Thread into the ActionListener of the CaptureButton but that failed because the main GUI element was not accessible which is courious given that it is defined as public.
It does however throw in the course of compilation some warnings - tho I must admit that I have no clue what they mean:
[ WARN:0] global C:\build\master_winpack-bindings-win64-vc14-static\opencv\modules\videoio\src\cap_msmf.cpp (376) `anonymous-namespace'::SourceReaderCB::OnReadSample videoio(MSMF): OnReadSample() is called with error status: -1072873821
[ WARN:0] global C:\build\master_winpack-bindings-win64-vc14-static\opencv\modules\videoio\src\cap_msmf.cpp (388) `anonymous-namespace'::SourceReaderCB::OnReadSample videoio(MSMF): async ReadSample() call is failed with error status: -1072873821
[ WARN:1] global C:\build\master_winpack-bindings-win64-vc14-static\opencv\modules\videoio\src\cap_msmf.cpp (1021) CvCapture_MSMF::grabFrame videoio(MSMF): can't grab frame. Error: -1072873821
Exception in thread "Thread-0" CvException [org.opencv.core.CvException: cv::Exception: OpenCV(4.5.2) C:\build\master_winpack-bindings-win64-vc14-static\opencv\modules\imgcodecs\src\loadsave.cpp:896: error: (-215:Assertion failed) !image.empty() in function 'cv::imencode'
]
at org.opencv.imgcodecs.Imgcodecs.imencode_1(Native Method)
at org.opencv.imgcodecs.Imgcodecs.imencode(Imgcodecs.java:510)
at GUI.FaceRecognition.startCamera(FaceRecognition.java:129)
at main$1$1.run(main.java:19)
at java.base/java.lang.Thread.run(Thread.java:834)
[ WARN:2] global C:\build\master_winpack-bindings-win64-vc14-static\opencv\modules\videoio\src\cap_msmf.cpp (438) `anonymous-namespace'::SourceReaderCB::~SourceReaderCB terminating async callback
Thread vorbei
I have written my code rather amateurish of that I am sure and I reckon I could use something like callable() but am unsure who to use that. I also do not know how to replicate my problem outside of chucking my entire project around the internet but I can however provide the code of my modifications:
for the funtion itself
public boolean startCamera(){
capture= new VideoCapture(0);
image=new Mat();
byte[] imageData;
ImageIcon icon;
String name = null;
boolean echo = false;
while(true){
capture.read(image);
final MatOfByte buf=new MatOfByte();
Imgcodecs.imencode(".png", image,buf);
imageData= buf.toArray();
icon=new ImageIcon(imageData);
Feed.setIcon(icon);
if(trigger==true){
name="Recognition";
Imgcodecs.imwrite("src/"+name+".png",image);
trigger=false; //Auslöser zurücksetzten
}
File kamera=new File("src/Recognition.png");
if(kamera.exists()==true){
//capture.release();
/*try {
Kamera=ImageIO.read(this.getClass().getResourceAsStream("/Recognition.png"));
} catch (IOException e) {
e.printStackTrace();
}*/
return echo=true;
}
}
}
for the main()
import java.awt.*;
import java.io.File;
import java.lang.String;
import GUI.*;
import org.opencv.core.Core;
public class main {
public static void main(String[] args){
FaceRecognition Fenster=new FaceRecognition();
System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
EventQueue.invokeLater(new Runnable() {
#Override
public void run() {
//FaceRecognition Fenster=new FaceRecognition();
new Thread (new Runnable() {
#Override
public void run() {
Fenster.startCamera();
}
}).start();
//File kamera=new File(("src/Recognition.png"));
}
});
if(Fenster.startCamera()==true){
Thread.currentThread().interrupt();
System.out.println("Thread vorbei");
}
}
}
Please point out where I went wrong or what goes wrong.
Edit:
I have mean while heeded #pveentjers advice and reformed my code (see below). Yet in my main() the IDE proposes to make startCamera static which will not work because startCamera can not be static (or rather elements of it can't). Is there any way to move around this?
the startCamera function
public class startCamera implements Runnable{
private CountDownLatch end;
public startCamera(CountDownLatch one){
this.end=one;
}
#Override public void run(){
try {
capture= new VideoCapture(0);
image=new Mat();
byte[] imageData;
ImageIcon icon;
String name = null;
while(true){
capture.read(image);
final MatOfByte buf=new MatOfByte();
Imgcodecs.imencode(".png", image,buf);
imageData= buf.toArray();
icon=new ImageIcon(imageData);
Feed.setIcon(icon);
if(trigger==true){
name="Recognition";
Imgcodecs.imwrite("src/"+name+".png",image);
trigger=false; //Auslöser zurücksetzten
}
File kamera=new File("src/Recognition.png");
if(kamera.exists()==true){
capture.release();
end.countDown();
/*try {
Kamera=ImageIO.read(this.getClass().getResourceAsStream("/Recognition.png"));
} catch (IOException e) {
e.printStackTrace();
}*/
}
}
}
catch (Exception exception){
}
}
the reformed main()
import java.awt.*;
import java.io.File;
import java.lang.String;
import java.util.concurrent.CountDownLatch;
import GUI.*;
import GUI.FaceRecognition.startCamera;
import org.opencv.core.Core;
public class main {
public static void main(String[] args){
FaceRecognition Fenster=new FaceRecognition();
CountDownLatch mark=new CountDownLatch(1);
startCamera startCamera=new startCamera(mark);
try {
mark.await();
} catch (InterruptedException e) {
e.printStackTrace();
}
Might be that I don't the wood for the trees but this the first time I am working with threads.

file writing programm randomly exits without errors

currently i'm trying to write to a txt-file with Java's Files-Class. This task should be done every 5 seconds and is scheduled by an ScheduledExecutorService. For some time it's doing it's job propperly but after a random time my program exits without any warnings, errors or so. I have tried to reproduce this but it seems to occur very random. I've also tried to use a PrintWriter but that lead randomly to the same behaviour.
Thanks in advance!!!
Here is my code:
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardOpenOption;
import java.time.LocalDateTime;
import java.time.LocalTime;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
public class FileWritingClass
{
public static void main(String[] args) throws IOException
{
ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);
scheduler.scheduleAtFixedRate(
new Runnable() {
long i;
String str = "";
Path path = Paths.get("TestFile.txt");
#Override
public void run() {
str = LocalTime.now().toString() + ": still alive after " + i + " times.";
i++;
try
{
System.out.println(str);
Files.write(path, (str + System.lineSeparator()).getBytes(), StandardOpenOption.APPEND);
} catch (IOException e) {
e.printStackTrace();
}
}
},
500 /* Startdelay */,
5000 /* period */,
TimeUnit.MILLISECONDS );
}
}
The Executor is swallowing Exceptions other than IOException. Catch Throwable instead of just IOException.
The Executor fails when it encounters some Exception other than IOException. The VM is still alive, but the Executor has failed.
Here's some code to try:
try
{
System.out.println(str);
Files.write(path, (str + System.lineSeparator()).getBytes(), StandardOpenOption.APPEND);
if(new Random().nextBoolean()) {
System.out.println("Here we go.");
throw new RuntimeException("Uncaught");
}
} catch (IOException e) {
e.printStackTrace();
} catch(Throwable t) {
t.printStackTrace();
System.out.println("The show must go on...");
}
Note how I give a 50/50 chance to throw an uncaught exception after the write (just for quick failure).
If you remove the second catch, it will just stop printing.
With the catch, the output becomes:
12:46:00.780250700: still alive after 0 times.
12:46:05.706355500: still alive after 1 times.
Here we go.
java.lang.RuntimeException:
Uncaught at
KeepaTokenRecorder$FileWritingClass$1.run(KeepaTokenRecorder.java:89)
at [...]
The show must go on...
12:46:15.705899200: still alive after 3 times.

Java kill thread after some time or when return data from him [duplicate]

This question already has answers here:
How do you kill a Thread in Java?
(17 answers)
How can I pass a parameter to a Java Thread?
(19 answers)
Returning value from Thread
(9 answers)
Closed 4 years ago.
How can I kill a thread if the time for him over or it return me the data?
how I start it
I need to pass a arg for him
You can use Future.get with a timeout.
If the timeout is exceeded, you'll get a TimeoutException, which you can handle any way you like.
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
public class Main
{
private static int foo(final int arg)
{
/* Uncomment to see future get interrupted
try
{
Thread.sleep(2000);
}
catch (InterruptedException e) { } */
return 1000 + arg;
}
public static void main(String[] args)
throws ExecutionException, InterruptedException
{
final ExecutorService executor = Executors.newSingleThreadExecutor();
final Future<Integer> future = executor.submit(() -> Main.foo(3));
try
{
int result = future.get(1, TimeUnit.SECONDS);
System.out.println(result);
}
catch (TimeoutException e)
{
future.cancel(true);
}
executor.shutdown();
}
}

How can I possibly avoid infinite loop in this Java server?

I am not talking about threading or anything to make this more complicated.
Most server programs I saw are like this or while(true){...} (same concept).
import java.net.ServerSocket;
import java.net.Socket;
import java.util.Scanner;
import java.io.DataOutputStream;
import java.io.IOException;
public class TCPServer {
ServerSocket welcomeSocket;
public TCPServer(int port) throws IOException {
welcomeSocket = new ServerSocket(port);
}
public void go() throws IOException {
// This is not a valid way to wait for a socket connection, You should
// not have a forever loop or while(true)
**for (; ;) {**
Socket connectionSocket = welcomeSocket.accept();
Scanner clientIn = new Scanner(connectionSocket.getInputStream());
DataOutputStream clientOut = new DataOutputStream(connectionSocket.getOutputStream());
String clientLine = clientIn.nextLine();
String modLine = clientLine.toUpperCase();
clientOut.writeBytes(modLine + "\n");
}
}
public static void main(String[] args){
try {
TCPServer server = new TCPServer(6789);
server.go();
}
catch(IOException ioe) {
ioe.printStackTrace();
}
}
}
It is not looping permanently, your code blocks on line welcomeSocket.accept() until someone connects and only after that next lines are executed then it waits for a new connection on welcomeSocket.accept(). In other words it loops as many times as it needs( per each connection ).
If you just want to allow only one client to connect, remove for (; ;) statement. But it will require to restart your server every time.
The while(!finished) option might be a better solution than the "empty" for loop. When the exit event occurs, you just set finished to true.
You can run a scheduler with any number of thread in the pool
and prevent the main thread from termination. Here I've used input stream but you can do it in different ways. Here you can use multiple threads to get connections and customize the frequency of the scheduler.
import java.io.IOException;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.ScheduledFuture;
import java.util.concurrent.TimeUnit;
import org.junit.Test;
public class MyTest {
ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(4);
#Test
public void hello() {
final Runnable helloServer = new Runnable() {
public void run() {
// handle soket connection here
System.out.println("hadling connection");
}
};
final ScheduledFuture<?> helloHandle = scheduler.scheduleAtFixedRate(helloServer, 1000, 1000, TimeUnit.MILLISECONDS);
try {
System.in.read();
} catch (IOException e) {
e.printStackTrace();
}
}
}

Categories