Let me give an overview of my question here:
There are ~ 4000 servers and millions of URLs per server. My code needs to hit each URL and write the response code to HDFS filesystem along with the URL.
Added some part here too :
Check the number of requests sent to a webpage
I am using a producer-consumer model here with 400 threads. The code has caused a DOS attack to few web servers very recently, I am having a hard time figuring out where the issue here is :
Main class:
public void readURLS(final Path inputPath, final Path outputPath) {
LOG.info("Looking for files to download, queue size: {}, DOWNLOAD_THREADS: {}", queueSize, producerThreads);
final List<Path> files = HdfsUtils.listDirectory(inputPath, hadoopConf);
final BlockingQueue<String> queue = new LinkedBlockingQueue<>(queueSize);
final UrlConsumerWriter consumerWriter =
new UrlConsumerWriter(queue, outputPath, hadoopConf);
LOG.info("Starting download of {} files from: '{}'", files.size(), inputPath);
final ExecutorService writerPool = DownloadUtils.createWriterPool();
CompletableFuture<Void> producer = downloadFilesToQueue(files, queue)
.thenRun(consumerWriter::notifyProducersDone);
CompletableFuture<Void> consumer =
CompletableFuture.runAsync(consumerWriter, writerPool)// Cancel download workers if write worker fails
.whenComplete((result, err) -> {
if (err != null) {
LOG.error("Consumer Write worker failed!", err);
producer.cancel(true);
}
});
writerPool.shutdown();
producer.join();
consumer.join();
LOG.info("Url Validation Job Complete!!!");
}
private CompletableFuture<Void> downloadFilesToQueue(
final List<Path> files,
final BlockingQueue<String> downloadQueue
) {
final ExecutorService pool = DownloadUtils.createDownloadPool(producerThreads);
final List<CompletableFuture<Void>> workers = files.stream()
.map(file -> new UrlDownloadWorker(clock, file, hadoopConf, downloadQueue,
utils, (validatorImpl.emptyTable())))
.map(worker -> CompletableFuture.runAsync(worker, pool))
.collect(Collectors.toList());
pool.shutdown();
final CompletableFuture<Void> allDownloads = CompletableFuture.allOf(workers.toArray(new CompletableFuture[0]));
// When one worker fails, cancel all the other immediately
for (final CompletableFuture<Void> worker : workers) {
worker.whenComplete((v, err) -> {
if (err != null) {
LOG.error("Download worker failed!", err);
allDownloads.cancel(true);
}
});
}
return allDownloads;
}
PRODUCER CLASS:
#Override
public void run() {
LOG.info("Starting download worker for file: '{}'", file);
long numLines = 0;
try (BufferedReader reader = new BufferedReader(new InputStreamReader(
file.getFileSystem(hadoopConf).open(file), CHARSET))) {
String line;
while ((line = reader.readLine()) != null) {
// LOG.info("Thread {} Reading file: '{}'",Thread.currentThread().getName(), file);
if (Thread.interrupted()) {
throw new InterruptedException();
}
StringBuilder builder = new StringBuilder();
//write into database
final StatusCode statusCode = utils.validateURL(line);
if (statusCode != null) {
queue.put(builder.append(line)
.append(",")
.append(statusCode.name()).toString());
builder.setLength(0);
} else {
throw new UrlValidationException(
"Failed to validate url :'" + line + "'");
}
numLines++;
}
} catch (IOException e) {
throw new DownloadException(file, e);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new DownloadException("Interrupted while downloading", file, e);
}
LOG.info("Download of {} lines complete for file: '{}'", numLines, file);
}
UrlValidationUtils CLASS:
public final class UrlValidationUtils {
private static final String WEBSITENOTCHECK = "uncheck.org";
private final Map<String, StatusCode> blockedHosts = new ConcurrentHashMap<>();
private static final int MAX_REDIRECT = 4;
public StatusCode validateURL(String url) throws IOException {
return validate(url, MAX_REDIRECT);
}
private StatusCode validate(String url, int maxRedirect) throws IOException {
URL urlValue = new URL(url);
HttpURLConnection con;
if (url.contains(WEBSITENOTCHECK)) {
blockedHosts.put(urlValue.getHost(), StatusCode.SUCCESS);
}
//first check if the host is already marked as invalid
// if (blockedHosts.containsKey(urlValue.getHost())) {
// return blockedHosts.get(urlValue.getHost());
// }
StatusCode statusCode;
con = (HttpURLConnection) urlValue.openConnection();
try {
int resCode;
con.setInstanceFollowRedirects(false);
con.setConnectTimeout(3000); //set timeout to 3 seconds
con.connect();
resCode = con.getResponseCode();
LOG.info("thread name {} connection id {} url {} ", Thread.currentThread().getName(), con.toString(), url);
if (resCode == HttpURLConnection.HTTP_OK) {
statusCode = StatusCode.SUCCESS;
} else if (resCode == HttpURLConnection.HTTP_SEE_OTHER || resCode == HttpURLConnection.HTTP_MOVED_PERM
|| resCode == HttpURLConnection.HTTP_MOVED_TEMP) {
String location = con.getHeaderField("Location");
if (location.startsWith("/")) {
location = urlValue.getProtocol() + "://" + urlValue.getHost() + location;
}
statusCode = validateRedirect(location, maxRedirect - 1, con);
} else {
blockedHosts.put(urlValue.getHost(), StatusCode.FAIL);
statusCode = StatusCode.FAIL;
}
} catch (UnknownHostException e) {
blockedHosts.put(urlValue.getHost(), StatusCode.UNKOWNHOST);
statusCode = StatusCode.UNKOWNHOST;
} catch (ConnectException e) {
blockedHosts.put(urlValue.getHost(), StatusCode.CONNECTION_ISSUE);
statusCode = StatusCode.CONNECTION_ISSUE;
} catch (IOException e) {
//if an IOException is caught possible reason is SOCKETTIMEOUT
blockedHosts.put(urlValue.getHost(), StatusCode.SOCKETTIMEOUT);
statusCode = StatusCode.SOCKETTIMEOUT;
}
con.disconnect();
LOG.info("thread name {} connection id {} url {} ", Thread.currentThread().getName(), con.toString(), url);
return statusCode;
}
private StatusCode validateRedirect(String location, int redirectCount, HttpURLConnection connection)
throws IOException {
if (redirectCount >= 0) {
connection.disconnect();
return validate(location, redirectCount);
}
return StatusCode.FAIL;
}
}
In order to not overload the servers, I suggest waiting a few milliseconds before hitting a batch of urls. For example, after hitting N urls, you could wait 20 ms, then hit the next N.. and so on. The size of the batch (N) would depend on how many requests you know your server can handle in a second. Do you have a Service Level Agreement for them in terms of performance?
Related
I am implementing HTTP server on vanilla Java. And I try to use ExecutorService.
First request is successful in browser, but second one has endless loading.
Code of my start() method of server:
public void start() throws IOException {
this.server = new ServerSocket(this.port);
ExecutorService executor = Executors.newCachedThreadPool();
client = this.server.accept();
while (true) {
executor.submit(() -> {
Socket cs = client;
try (PrintWriter out = new PrintWriter(cs.getOutputStream());
BufferedReader in = new BufferedReader(new InputStreamReader(cs.getInputStream()))
) {
// write server http headers response
out.print("HTTP/1.1 200 OK \n");
out.print("Content-Type: text/plain\n");
out.print("Accept-Language: en-US, en; q=0.5\n");
// out.print("Connection: close\n");
out.print("\n");
String data;
// read client request
while ((data = in.readLine()) != null) {
if (data.length() == 0) {
out.write("EOF(End of file)");
break;
}
// write back to client its request as response body.
out.write(data + "\n");
}
out.close();
in.close();
cs.close();
} catch (IOException e) {
e.getMessage();
}
});
}
}
What I am doing wrong?
use this code i hope it's going to help you
after catch use finally to close in
BufferedReader objReader = null;
try {
String strCurrentLine;
objReader = new BufferedReader(new FileReader("D:\\DukesDiary.txt"));
while ((strCurrentLine = objReader.readLine()) != null) {
System.out.println(strCurrentLine);
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (objReader != null)
objReader.close();
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
update
PrintWriter pw = new PrintWriter(new OutputStreamWriter(s.getOutputStream(),"UTF-8"),true);
I use jdk11 HttpClient send msg which from file,and Send once per line。but about 40 times . 400 statusCode is appearing. i don't know why? thanks for your suggestions
Below is my source：
SendMsgHttp.java
//ignorance other source
public void sendData(String rawHex,String versionType){
String json_data = key_protocolType+versionType+ key_upLinkData+rawHex+key_reqTimestamp;
System.out.println("send data："+json_data);
HttpRequest request = httpRequestBuilder
.timeout(Duration.ofSeconds(10))
.header("Content-Type", "application/json")
.header("Authorization","Basic Y21kdDpDbUR0MTIzJA==")
.POST(HttpRequest.BodyPublishers.ofString(json_data))
.build();
try {
HttpResponse response= client.send(request, HttpResponse.BodyHandlers.ofString());
System.out.println(response.statusCode());
atomicNum.addAndGet(1);
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
ReadFileAndSendThread.java
public class ReadFileAndSendThread implements Runnable{
private SendMsgHttp sendMsgHttp ;
private File file;
private CountDownLatch latch;
public ReadFileAndSendThread( File file,SendMsgHttp sendMsgHttp,CountDownLatch latch){
this.file = file;
this.sendMsgHttp = sendMsgHttp;
this.latch = latch;
}
#Override
public void run() {
try {
readSingleHandle();
latch.countDown();
} catch (IOException e) {
System.err.println("go on");
e.printStackTrace();
}
}
public void readSingleHandle() throws IOException {
InputStreamReader isr=new InputStreamReader(new FileInputStream(file), "UTF-8");
BufferedReader br = new BufferedReader(isr);
StandardOpenOption.READ);
String tempString;
while ((tempString = br.readLine()) != null) {
try {
String[] strData = tempString.split("\\|");
String rawData = strData[3].trim();
String versionType = strData[2].trim();
if(null !=rawData&&!rawData.equals("")
&& null !=versionType&&!versionType.equals("")){
sendMsgHttp.sendData(rawData,versionType);
Thread.sleep(1000);
}
}catch (Exception e){
e.printStackTrace();
}
}
br.close();
}
}
main SendMain.java
public class SendMain {
public static void main(String[] args) {
ExecutorService threadPool = new ThreadPoolExecutor(1, 25,
0L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue());
String path = new File("").getAbsolutePath()+"/export";//"/root/xzg/send/exportdata";
File[] fs = new File(path).listFiles();
final CountDownLatch latch = new CountDownLatch(fs.length);
for(File file:fs){
threadPool.execute(new ReadFileAndSendThread(file,
new SendMsgHttp("http://localhost:9997/xxx")
,latch));
}
threadPool.shutdown();
try {
latch.await();
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("count nums："+SendMsgHttp.getSendCount());
}
}
and the file content like this
At last i run it. the result like below
//ok
response.statusCode:200
response.statusCode:200
response.statusCode:200
response.statusCode:200
response.statusCode:400
response.statusCode:400
response.statusCode:400
//always 400 ??? this wrong is happend
According to the result i find the first 47 data always were successful, and the rest failed.And Running many times, the results are the same
and i try sending the data which return "response.statusCode:400" and it's ok(response.statusCode:200)
what's going on? need u help . Thanks again.
I am having an issue, when I try to update some information in the background in my app I constantly get reports of ANR (Are not responding).
The code block is as follows
class getServerTime extends AsyncTask<String, Long, Long> {
private Long getInternetData() {
String sURL = "http://crystalmathlabs.com/tracker/api.php?type=time"; //just a string
// Connect to the URL using java's native library
HttpURLConnection connect = null;
try {
int responseCode = -1;
List<String> listSet = new ArrayList();
URL url = new URL(sURL);
connect = (HttpURLConnection) url.openConnection();
connect.setConnectTimeout(R.integer.timeoutLengthWithACapitalT);
connect.setReadTimeout(R.integer.timeoutLengthWithACapitalT);
connect.connect(); //this is the timeout line
responseCode = connect.getResponseCode();
if (responseCode < 400) {
BufferedReader in = new BufferedReader(new InputStreamReader(connect.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null)
listSet.add(inputLine);
in.close();
if (listSet.get(0) != null)
return Long.parseLong(listSet.get(0));
}
return -1L;
}
catch (java.io.FileNotFoundException e) {
Log.e("getServerTime", "File Not Found: " + e.getMessage());
} catch (Exception e) {
Log.e("getServerTime", "Unknown Exception " + e.getMessage());
}
finally{
if (connect != null)
connect.disconnect();
}
return -1L;
}
#Override
protected Long doInBackground(String[] params) {
Long response;
new Scores();
try {
response = getInternetData();
if (response != null) {
return response;
}
return -1L;
} catch (Exception e) {
Log.d("Error in getServerTime", "" + Log.getStackTraceString(e.getCause().getCause()));
}
return -1L;
}
}
On connect.connect I receive timeouts.
Attached below are two examples of the ANR reports, I simply can't figure this out, please help!
https://pastebin.com/F7iZ267D
https://pastebin.com/W1eSdH4F
at android.os.AsyncTask.get (AsyncTask.java:507)
at
com.yargonauts.burk.runescapemarketwatch.crystalMathLabs.a.a
(crystalMathFunctions.java:30)
You are calling AsyncTask.get(), this pretty much negates the use of an asynctask since it blocks the thread until the async task is done.
This blocking causes the ANR since the main thread is stuck waiting for the result.
You need to override the onPostExecute(Result) method and perform the follow-up work there. Check out the Usage section of the asynctask javadoc: https://developer.android.com/reference/android/os/AsyncTask.html
I can not upload large video to server android?
i try to use this method
private void UploadLargeFile(String[] args) throws Exception {
CloseableHttpAsyncClient httpclient = HttpAsyncClients.createDefault();
try {
httpclient.start();
File upload = new File(args[0]);
File download = new File(args[1]);
ZeroCopyPost httpost = null;
try {
httpost = new ZeroCopyPost(URLTOUploadFile", upload,
ContentType.create("video/mp4"));
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
ZeroCopyConsumer<File> consumer = null;
consumer = new ZeroCopyConsumer<File>(download) {
#Override
protected File process(final HttpResponse response,
final File file, final ContentType contentType)
throws Exception {
if (response.getStatusLine().getStatusCode() != HttpStatus.SC_OK) {
throw new ClientProtocolException("Upload failed: "
+ response.getStatusLine());
}
return file;
}
};
Future<File> future = httpclient.execute(httpost, consumer, null);
File result;
result = future.get();
System.out.println("Response file length: " + result.length());
System.out.println("Shutting down");
} finally {
httpclient.close();
}
System.out.println("Done");
}
in the first line of CloseableHttpAsyncClient i got NoSuchFieldFound error
can any one help me?
When i try to asyn to upload a file use ListenableFuture,but it is not work.
if it is syn, it work ok.
ListenableFuture not support to excute a post to remote? i found a sample using URL.openStream() work ok for get.
private void asynUploadFileToRemote(final String uptoken, final String key, final File file, final PutExtra extra) {
final ListenableFuture<String> future = pool.submit(new Callable<String>() {
#Override
public String call() throws Exception {
logger.info("starting to upload file");
// here is to upload a file to remote.
PutRet putRet = IoApi.put(uptoken, key, file, extra);
logger.info("end to upload file"); //this log never excute.
return putRet.getKey();
}
});
future.addListener(new Runnable() {
#Override
public void run() {
try {
//recerve nothing here
String key = future.get();
logger.info("uploadkey:" + key);
} catch (InterruptedException e) {
logger.error("Interrupted", e);
} catch (ExecutionException e) {
logger.error("Exception in task", e.getCause());
}
}
}, MoreExecutors.sameThreadExecutor());
}
IoApi.put:Just to upload a file.
public static PutRet put(String uptoken, String key, File file,
PutExtra extra) {
if (!file.exists() || !file.canRead()) {
return new PutRet(new CallRet(400, new Exception(
"File does not exist or not readable.")));
}
if (key == null) {
key = UNDEFINED_KEY;
}
MultipartEntity requestEntity = new MultipartEntity();
try {
requestEntity.addPart("token", new StringBody(uptoken));
FileBody fileBody = new FileBody(file);
requestEntity.addPart("file", fileBody);
requestEntity.addPart("key", new StringBody(key));
if (extra.checkCrc != NO_CRC32) {
if (extra.crc32 == 0) {
return new PutRet(new CallRet(400, new Exception("no crc32 specified!")));
}
requestEntity.addPart("crc32", new StringBody(extra.crc32 + ""));
}
} catch (Exception e) {
e.printStackTrace();
return new PutRet(new CallRet(400, e));
}
String url = Config.UP_HOST;
CallRet ret = new Client().callWithMultiPart(url, requestEntity);
return new PutRet(ret);
}