We Keep Coding

All my threads are waiting in my WebScraper project. Can someone tell me why?

This is my first post on Stack Overflow so please go easy on me! I made this Web Scraper as a final project in my CS course last semester. I was able to pass with it, however, it always bothered me about how slow my program ran compared to others in the class. My program took 11 hours to gather 10,000 emails, whereas my friend took 5 minutes. I couldn't figure out why! I even tried seeing what's wrong with a java profiler, and it just showed me that my threads are waiting. I don't know how to fix that and why it only affected me. I really want to learn about how to properly use threads, so I'm asking you guys.
My CPU is an i7 7700k, so there shouldn't be a problem there and I have Gigabit internet. So it's definitely the way I coded my program. Here is the main class:
import java.net.MalformedURLException;
import java.net.URL;
import java.sql.*;
import java.util.*;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class Main {
static int EMAIL_MAX_COUNT = 10_000;
static int MAX_VISITS = 5;
static final Set<String> emails = Collections.synchronizedSet(new HashSet<>(10_000));
static Set<String> linksToVisit = Collections.synchronizedSet(new HashSet<>(20_000));
static Set<String> linksFilter = Collections.synchronizedSet(new HashSet<>(20_000));
static Set<String> linksVisited = Collections.synchronizedSet(new HashSet<>(10_000));
static Map<String, Set<String>> maxLinksVisited = Collections.synchronizedMap(new HashMap<>());
public static void main(String[] args) {
ExecutorService pool = Executors.newFixedThreadPool(200);
linksToVisit.add("https://www.touro.edu/");//starts with touro.edu
while (!linksToVisit.isEmpty() && emails.size() <= EMAIL_MAX_COUNT) {
String link;
synchronized (linksToVisit) {
link = linksToVisit.stream().findFirst().get();
System.out.println(link);
linksToVisit.remove(link);
}
if (hasTooManyVisits(link)) {
link = "";
}
if (!(link.equals(""))) {
linksVisited.add(link);
pool.execute(new WebScraper(link));
}
}
pool.shutdownNow();
}
private static boolean hasTooManyVisits(String link) {
try {
URL currentURL = new URL(link);
String host = currentURL.getHost();
int startIndex = 0;
int nextIndex = host.indexOf('.');
int lastIndex = host.lastIndexOf('.');
while (nextIndex < lastIndex) {
startIndex = nextIndex + 1;
nextIndex = host.indexOf('.', startIndex);
}
synchronized (maxLinksVisited) {
if (startIndex > 0) {
Set<String> tempSet = maxLinksVisited.get(host.substring(startIndex));
if (tempSet == null) {
tempSet = new HashSet<>();
maxLinksVisited.put(host.substring(startIndex), tempSet);
}
tempSet.add(link);
maxLinksVisited.put(host.substring(startIndex), tempSet);
if (maxLinksVisited.get(host.substring(startIndex)).size() >= MAX_VISITS) {
return true;
}
} else {
Set<String> tempSet = maxLinksVisited.get(host);
if (tempSet == null) {
tempSet = new HashSet<>();
maxLinksVisited.put(host, tempSet);
}
tempSet.add(link);
maxLinksVisited.put(host, tempSet);
if (maxLinksVisited.get(host).size() >= MAX_VISITS) {
return true;
}
}
}
} catch (MalformedURLException e) {
return false;
}
return false;
}
All it really does is setup the initial part of the program and create the threads. Here is the WebScraper class:
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import java.io.IOException;
import java.util.Arrays;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public class WebScraper implements Runnable {
String currentUrl;
String[] randomFileExtensions = {"png", "jpg", "gif", "pdf", "mp3", "css", "mp4", "mov", "7z", "zip", "mkv", "avi", "jpeg"};//common files
WebScraper(String url) {
this.currentUrl = url;
run();//for some reason it's needed
}
#Override
public void run() {
try {
try { // double try block so the program doesn't stop on errors
Document doc = Jsoup.connect(currentUrl).userAgent("Mozilla/5.0 (Windows; U; WindowsNT 5.1; en-US; rv1.8.1.6) Gecko/20070725 Firefox/2.0.0.6")
.referrer("http://www.google.com").get();
Pattern emailPattern = Pattern.compile("[\\w\\d._]+#[\\w\\d]+\\.[\\w]{2,3}");
Matcher emailMatcher = emailPattern.matcher(doc.toString());
while (emailMatcher.find()) {//find and add emails
String email = emailMatcher.group();
if (Arrays.stream(randomFileExtensions).parallel().noneMatch(email::contains)) {//filter for any files that are not emails
Main.emails.add(emailMatcher.group());
}
}
synchronized (Main.linksFilter) {
Main.linksFilter.addAll(doc.select("a[href]").eachAttr("abs:href"));//find and add all links on the page
for (String randomFileExtension : randomFileExtensions) {
Main.linksFilter.removeIf(s -> s.contains(randomFileExtension));//filter links for any files
}
synchronized (Main.linksToVisit) {
Main.linksFilter.removeAll(Main.linksVisited);
Main.linksToVisit.addAll(Main.linksFilter);
Main.linksFilter.clear();
}
}
} catch (IOException e) {
e.printStackTrace();
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
One thing that bothers me is that the program will stop after touro.edu if I take out the run(); method from the constructor. I don't know why, the program should automatically call it...
In conclusion, I just want to know what I did wrong. Please help me understand, and thank you in advanced!

first, you can't call the pool.shutdownNow(); method, suggest call pool.shutdown();
they are different can read the java document.
when you call java.util.Collections#synchronizedSet(java.util.Set<T>) method, the return Set is already thread-safe, so you don't add synchronized.

Is it necessary lock the file when multiple threads try to append the content using NIO in JAVA?

At first I have created an empty file, and then I've invoked some thread to search the database and get the result content, and then append to the file. The result content is String type and may be 20M. Each thread should write into the file one at a time. I have tested many times and I find that it is not necessary to lock. Is that right? The total lines of the example is 1000. When should I need to add a write lock to operate on the file?
String currentName = "test.txt";
final String LINE_SEPARATOR = System.getProperty("line.separator");
ThreadPoolExecutor pool = new ThreadPoolExecutor(
10, 100, 10, TimeUnit.SECONDS, new LinkedBlockingDeque<Runnable>());
for (int i = 0; i < 500; i++) {
pool.execute(() -> {
try {
appendFileByFilesWrite(currentName, "abc" +
ThreadLocalRandom.current().nextInt(1000) + LINE_SEPARATOR);
} catch (IOException e) {
e.printStackTrace();
}
});
}
IntStream.range(0, 500).<Runnable>mapToObj(a -> () -> {
try {
appendFileByFilesWrite( currentName,
"def" + ThreadLocalRandom.current().nextInt(1000) +
LINE_SEPARATOR);
} catch (IOException e) {
e.printStackTrace();
}
}).forEach(pool::execute);
pool.shutdown();
Here is the method:
public static void appendFileByFilesWrite(String fileName,String fileContent) throws IOException {
Files.write(Paths.get(fileName), fileContent.getBytes(),StandardOpenOption.APPEND);
}

The answer is: always.
Your test works for you. Right now. Today. Maybe during a full moon, it won't. Maybe if you buy a new computer, or your OS vendor updates, or the JDK updates, or you're playing a britney spears song in your winamp, it won't.
The spec says that it is legitimate for the write to be smeared out over multiple steps, and the behaviour of SOO.APPEND is undefined at that point. Possibly if you write 'Hello' and 'World' simultaneously, the file may end up containing 'HelWorllod'. It probably won't. But it could.
Generally, bugs in concurrency are very hard (sometimes literally impossible) to test for. Doesn't make it any less of a bug; mostly you end up with a ton of bug reports, and you answering 'cannot reproduce' on all of them. This is not a good place to be.
Most likely if you want to observe the problem in action, you should write extremely long strings in your writer; the aim is to end up with the actual low-level disk command involving multiple separated out blocks. Even then there is no guarantee that you'll observe a problem. And yet, absence of proof is not proof of absence.

I use this class when I need to lock a file. It allows for read write locks across multiple JVMs and multiple threads.
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.OutputStreamWriter;
import java.io.RandomAccessFile;
import java.nio.channels.Channels;
import java.nio.channels.FileChannel;
import java.util.Date;
import java.util.Map;
import java.util.Objects;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantReadWriteLock;
import com.lfp.joe.core.process.CentralExecutor;
public class FileLocks {
private static final String WRITE_MODE = "rws";
private static final String READ_MODE = "r";
private static final Map<String, LockContext> JVM_LOCK_MAP = new ConcurrentHashMap<>();
private FileLocks() {
}
public static <X> X read(File file, ReadAccessor<X> accessor) throws IOException {
return access(file, false, fc -> {
try (var is = Channels.newInputStream(fc);) {
return accessor.read(fc, is);
}
});
}
public static void write(File file, WriterAccessor accessor) throws IOException {
access(file, true, fc -> {
try (var os = Channels.newOutputStream(fc);) {
accessor.write(fc, os);
}
return null;
});
}
public static <X> X access(File file, boolean write, FileChannelAccessor<X> accessor)
throws FileNotFoundException, IOException {
Objects.requireNonNull(file);
Objects.requireNonNull(accessor);
String path = file.getAbsolutePath();
var lockContext = JVM_LOCK_MAP.compute(path, (k, v) -> {
if (v == null)
v = new LockContext();
v.incrementAndGetThreadCount();
return v;
});
var jvmLock = write ? lockContext.getAndLockWrite() : lockContext.getAndLockRead();
try (var randomAccessFile = new RandomAccessFile(file, write ? WRITE_MODE : READ_MODE);
var fileChannel = randomAccessFile.getChannel();) {
var fileLock = write ? fileChannel.lock() : null;
try {
return accessor.access(fileChannel);
} finally {
if (fileLock != null && fileLock.isValid())
fileLock.close();
}
} finally {
jvmLock.unlock();
JVM_LOCK_MAP.compute(path, (k, v) -> {
if (v == null)
return null;
var threadCount = v.decrementAndGetThreadCount();
if (threadCount <= 0)
return null;
return v;
});
}
}
public static interface FileChannelAccessor<X> {
X access(FileChannel fileChannel) throws IOException;
}
public static interface ReadAccessor<X> {
X read(FileChannel fileChannel, InputStream inputStream) throws IOException;
}
public static interface WriterAccessor {
void write(FileChannel fileChannel, OutputStream outputStream) throws IOException;
}
private static class LockContext {
private final ReentrantReadWriteLock rwLock = new ReentrantReadWriteLock();
private long threadCount = 0;
public long incrementAndGetThreadCount() {
threadCount++;
return threadCount;
}
public long decrementAndGetThreadCount() {
threadCount--;
return threadCount;
}
public Lock getAndLockWrite() {
var lock = rwLock.writeLock();
lock.lock();
return lock;
}
public Lock getAndLockRead() {
var lock = rwLock.readLock();
lock.lock();
return lock;
}
}
}
You can then use it for writing like so:
File file = new File("test/lock-test.txt");
FileLocks.write(file, (fileChannel, outputStream) -> {
try (var bw = new BufferedWriter(new OutputStreamWriter(outputStream));) {
bw.append("cool beans " + new Date().getTime());
}
});
And reading:
File file = new File("test/lock-test.txt")
var lines = FileLocks.read(file, (fileChannel, inputStream) -> {
try (var br = new BufferedReader(new InputStreamReader(inputStream));) {
return br.lines().collect(Collectors.toList());
}
});

You can use fileLock or just add synchronized to the method.
while (true) {
try {
lock = fc.lock();
break;
} catch (OverlappingFileLockException e) {
Thread.sleep(1 * 1000);
}
}
appendFileByFilesWrite( fileName, fileContent) ;
or just change like this:
public synchronized static void appendFileByFilesWrite(String fileName,String fileContent) throws IOException {
Files.write(Paths.get(fileName), fileContent.getBytes(),StandardOpenOption.APPEND);
}

Implements of thread in Android doesn't work

I'm writing a basic application in Android, the application will be connected to MySql server by quest in PHP, in Android Internet connection have to make in diffrent thread, so I create class which implements Runnable interface.
package com.company.opax.loginmysql;
import android.util.Log;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLConnection;
import java.util.ArrayList;
/**
* Created by opax on 30.08.2015.
*/
public class HttpPostMethod implements Runnable{
private String fileInHost;
private ArrayList<PostParameters> postParameterses;
private StringBuffer postResult;
public HttpPostMethod(String fileInHost, ArrayList<PostParameters> postParameterses){
this.fileInHost = fileInHost;
this.postParameterses = new ArrayList<PostParameters>(postParameterses);
}
public String getResult() {
return postResult.toString();
}
#Override
public void run() {
try {
String urlParameters = generateParameters();
URLConnection conn = initializeUrlConnection();
OutputStreamWriter writer = new OutputStreamWriter(conn.getOutputStream());
writer.write(urlParameters);
writer.flush();
String line;
BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream()));
while ((line = reader.readLine()) != null) {
postResult.append(line);
}
writer.close();
reader.close();
} catch (IOException e) {
Log.e("Exception", this.getClass().getName() + " name: " + e.toString());
}
}
private URLConnection initializeUrlConnection() throws MalformedURLException {
URL url = new URL(fileInHost);
URLConnection conn;
try {
conn = url.openConnection();
conn.setDoOutput(true);
}catch(IOException e){
throw new MalformedURLException();
}
return conn;
}
private String generateParameters(){
StringBuffer finishPostQuery = new StringBuffer();
for(PostParameters p : postParameterses){
finishPostQuery.append(p.getNameParam());
finishPostQuery.append("=");
finishPostQuery.append(p.getValueParam());
finishPostQuery.append("&");
}
if(!finishPostQuery.toString().equals("login=seba&password=pass&"))
throw new AssertionError("blad generatora zapytania: " + finishPostQuery);
return finishPostQuery.toString();
}
}
and login class:
public class Login {
private User user;
private final String paramLogin = "login";
private final String paramPass = "password";
public Login(User user){
this.user = user;
}
public boolean tryLogin(){
try{
ArrayList<PostParameters> postParameterses = new ArrayList<>();
postParameterses.add(new PostParameters(paramLogin, user.getUserName()));
postParameterses.add(new PostParameters(paramPass, user.getUserPass()));
HttpPostMethod httpPostMethod = new HttpPostMethod("http://blaba.php", postParameterses);
httpPostMethod.run();
Log.i("bla", httpPostMethod.getResult());
} catch (Exception e) {
Log.i("Exception", e.toString());
}
return false;
}
}
I'm trying to connect in other thread, but I still have an error: 'android.os.NetworkOnMainThreadException'
I would be grateful for the all suggestion what I do wrong.

Instead of:
httpPostMethod.run();
do:
new Thread(httpPostMethod).start();
In case your login call failed for some reasons (timeout, wrong login), you should report that somehow to user - this is what AsyncTask class is for. It allows you to run background code in doInBackkground, and after network operation ends - in onPostExecute you can execute UI related stuff - like show errors/results.

I suggest you two things.
First use AsyncTask instead of pure java threads.
But the main advice is to use a library that make http requests.
I like to use Retrofit, it may handle all request and thread part for you, but there are others.

Fast and asynchronous way of making multiple http requests in JAVA

I have a program that should make really fast http requests. Requests should be made asynchronously so that it won't block the main thread.
So I have created a queue which is observed by 10 separate threads that make http requests. If something is inserted in the queue then the first thread that gets the data will make the requests and process the result.
The queue gets filled with thousands of items so multithreading is really neccessary to get the response as fast as possible.
Since I have alot of code I'll give a short example.
main class
package fasthttp;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.LinkedBlockingQueue;
public class FastHTTP {
public static void main(String[] args) {
ExecutorService executor = Executors.newFixedThreadPool(10);
for (int i = 0; i < 10; i++) {
LinkedBlockingQueue queue = new LinkedBlockingQueue();
queue.add("http://www.lennar.eu/ip.php");//for example
executor.execute(new HTTPworker(queue));
}
}
}
FastHTTP class
package fasthttp;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.concurrent.LinkedBlockingQueue;
public class HTTPworker implements Runnable {
private final LinkedBlockingQueue queue;
public HTTPworker(LinkedBlockingQueue queue) {
this.queue = queue;
}
private String getResponse(String url) throws IOException {
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
StringBuilder response;
try (BufferedReader in = new BufferedReader(
new InputStreamReader(con.getInputStream()))) {
String inputLine;
response = new StringBuilder();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
}
return response.toString();
}
#Override
public void run() {
while (true) {
try {
String data = (String) queue.take();
String response = getResponse(data);
//Do something with response
System.out.println(response);
} catch (InterruptedException | IOException ex) {
//Handle exception
}
}
}
}
Is there a better or faster way to make thousands of http requests response processing asynchronously? Speed and performance is what I'm after.

Answering my own question. Tried Apaches asynchronous http client but after a while I started using Ning's async client and I am happy with it.

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.*;
import java.util.stream.Collectors;
import org.apache.http.client.methods.HttpGet;
import java.util.Iterator;
import org.apache.http.impl.client.BasicResponseHandler;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClientBuilder;
public class RestService {
private final static Executor executor = Executors.newCachedThreadPool();
private final static CloseableHttpClient closeableHttpClient = HttpClientBuilder.create().build();
public static String sendSyncGet(final String url) {
return sendAsyncGet(List.of(url)).get(0);
}
public static List<String> sendAsyncGet(final List<String> urls){
List<GetRequestTask> tasks = urls.stream().map(url -> new GetRequestTask(url, executor)).collect(Collectors.toList());
List<String> responses = new ArrayList<>();
while(!tasks.isEmpty()) {
for(Iterator<GetRequestTask> it = tasks.iterator(); it.hasNext();) {
final GetRequestTask task = it.next();
if(task.isDone()) {
responses.add(task.getResponse());
it.remove();
}
}
//if(!tasks.isEmpty()) Thread.sleep(100); //avoid tight loop in "main" thread
}
return responses;
}
private static class GetRequestTask {
private final FutureTask<String> task;
public GetRequestTask(String url, Executor executor) {
GetRequestWork work = new GetRequestWork(url);
this.task = new FutureTask<>(work);
executor.execute(this.task);
}
public boolean isDone() {
return this.task.isDone();
}
public String getResponse() {
try {
return this.task.get();
} catch(Exception e) {
throw new RuntimeException(e);
}
}
}
private static class GetRequestWork implements Callable<String> {
private final String url;
public GetRequestWork(String url) {
this.url = url;
}
public String getUrl() {
return this.url;
}
public String call() throws Exception {
return closeableHttpClient.execute(new HttpGet(getUrl()), new BasicResponseHandler());
}
}
}

Cannot send data outside runnable thread

I'm having some difficulties with my app. I created a class with functions to handle HTTP POST by starting a thread, the issue is that I cannot send data outside the thread!!! The class has a variable and I want to set value to that variable when the thread is running, please help.
Here is the code:
package com.mypackage;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import org.json.JSONObject;
import org.json.JSONException;
import android.annotation.SuppressLint;
import android.app.AlertDialog;
import android.content.DialogInterface;
import android.os.Handler;
import android.util.Log;
public class HandleJSON {
private String urlString = null;
private int errorcode ;
public int getErrorcode(){return errorcode;}
public volatile boolean parsingComplete = true;
public HandleJSON(String url){
//saving the URL
this.urlString = url;
}
#SuppressLint("NewApi")
public void readAndParseJSON(String in) {
try {
parsingComplete = false;
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public void fetchJSON(){
Thread thread = new Thread(new Runnable(){
#Override
public void run() {
try {
URL url = new URL(urlString);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setReadTimeout(10000 /* milliseconds */);
conn.setConnectTimeout(15000 /* milliseconds */);
conn.setRequestMethod("GET");
conn.setDoInput(true);
conn.connect();
//receiving message from server
InputStream stream = conn.getInputStream();
String data = convertStreamToString(stream);
// JSON thing
try{
JSONObject obj = new JSONObject(data);
//THIS IS THE ISSUE, I'm setting here the errorcode which should set the superclass variable "errorcode" so I can use "getErrorCode" to retrieve the code, but it seems like the thread does not respond after thread starts;
errorcode = obj.getInt("error_code");
}
catch(JSONException e) {
Log.e("Catch error", e.toString());
}
readAndParseJSON(data);
stream.close();
} catch (Exception e) {
e.printStackTrace();
}
}
});
thread.start();
}
static String convertStreamToString(java.io.InputStream is) {
java.util.Scanner s = new java.util.Scanner(is).useDelimiter("\\A");
return s.hasNext() ? s.next() : "";
}
}

If I understand correctly, you want to get data between a non-main thread and elsewhere. In which case you may want to create another class that has a public static variable. This is such that everyone permitted by the modifier in question (private, protected, public) can access the same content. Be careful as, if poorly managed, your code may function differently or not all on systems with execution speeds other than yours.

In such case, an anonymous daemon thread is a little tricky.
You can define a concrete class extending Thread, define data structures in it and provide the interface to access your data structures. Such as here below.
class MyThread extends Thread {
private JSONObject obj;
public void run() {
// Your parsing code goes here
// Such as obj = xxxxxx;
}
public JSONObject getData() {
return obj;
}
}
Of course, you should consider the concurrent risks when manipulating inner data structures.

The issue solved, I used AsyncTask instead and passed the variables to onPostExecute(String) which done the trick.