I have a list of 100k users. I have to loop through the list and make an API call to the server to get the result. Every time I create a new URL connections and making the APi call then closing the connection once I read the input stream, but it is taking too much time.
Is there any optimized way to do it, like using the same instance of URL connection multiple times instead of closing it? or going for another third-party library will improve the speed of execution?
I am calling the below method in my loop to get the output.
private String getOutput(String loginName) {
String responseStatus = null;
HttpURLConnection connection = null;
try {
URL url= new URL(<<https://api.junk.123.com/output>>);
connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("apikey", "authentication key");
connection.setUseCaches(false);
connection.setDoOutput(true);
//Send request
try(DataOutputStream outputStream = new DataOutputStream(connection.getOutputStream())){
JsonObject jsonParam = new JsonObject();
jsonParam.putString("loginName", "loginName");
outputStream.writeBytes(jsonParam.toString());
outputStream.flush();
}
//Get response
InputStream inputStream;
if(connection.getResponseCode() == HttpURLConnection.HTTP_OK){
inputStream = connection.getInputStream();
} else {
inputStream = connection.getErrorStream();
}
if(null == inputStream){
return String.valueOf(connection.getResponseCode());
}
StringBuilder response = new StringBuilder();
try (BufferedReader inputBuffer = new BufferedReader(new InputStreamReader(inputStream))) {
String line;
while (null != (line = inputBuffer.readLine())) {
response.append(line);
response.append("\r");
}
}
JsonObject jsonObject = new JsonObject(response.toString());
if (connection.getResponseCode() == HttpURLConnection.HTTP_OK) {
responseStatus = "success";
} else {
responseStatus = String.valueOf(connection.getResponseCode()) + jsonObject.getString("errorMessage") ;
}
} catch (MalformedURLException e) {
logger.error("Malformed URL exception occurred while calling the API", entry.getKey(), e);
} catch (IOException e) {
logger.error("Creation of connection failed while calling the API", entry.getKey(), e);
} catch (Exception e) {
logger.error("Error occurred while calling the API", entry.getKey(), e);
} finally {
if (null != connection){
connection.disconnect();
}
}
return responseStatus;
}
This Q&A explains that HTTP persistent connections are implemented behind the scenes by HttpURLConnection:
Persistent HttpURLConnection in Java
However, that may not be sufficient. If you use a single client-side thread to do the fetching you are limited by the round trip time for the requests; i.e. you can't start a second request until the result of the first one has been returned to you. You can remedy this ... up to a point ... by using multiple client-side threads.
However (#2) sending multiple requests in parallel also has its limits. Beyond a certain point you will saturate the client, the server or the network. In addition, some servers have throttling mechanisms to cap the number of requests that a client can make.
The way to get maximum throughput would be to redesign the API so that a single request can get information for multiple users.
Related
I have scanned a java web project with the Checkmarx tool, and the analysis marks an XSS vulnerability in a method where a web service is executed that responds a JSON, the vulnerability is in the line while((output = Encode.forJava(br.readLine())) != null) {, specifically in br.readLine().
Checkmarx says:
The attacker would be able to alter the returned web page by simply
providing modified data in the user input readLine, which is read by
the NetClientPost method. This input then flows through the code
straight to the output web page, without sanitization.
This can enable a Reflected Cross-Site Scripting (XSS) attack.
I tried with OWASP for Java, implementing the method Encode.forJava(), but the vulnerability continues to appear in the analysis. This is the implementation of the method:
public String NetClientPost (String urlSer, String param){
String result ="";
try {
InetAddress ip = InetAddress.getLocalHost();
String host = ip.getHostAddress();
doTrustToCertificates();
URL url = new URL(urlSer);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setDoOutput(true);
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Type", "application/json");
conn.setConnectTimeout(2000);
String input = param;
String output = "";
try(OutputStream os = conn.getOutputStream()) {
os.write(input.getBytes());
os.flush();
if (conn.getResponseCode() != 200) {
throw new RuntimeException("Failed : HTTP code : " + conn.getResponseCode());
}
try (BufferedReader br = new BufferedReader(new InputStreamReader((conn.getInputStream())))) {
while ((output = Encode.forJava(br.readLine())) != null) {//LINE OF CHECKMARX XSS
result += output;
}
}
}
conn.disconnect();
return result;
} catch (MalformedURLException e) {
return result;
} catch (IOException e) {
return result;
} catch (Exception e) {
return result;
}
}
Any have an idea of how to solve this?
Try parsing the incoming data as JSON and then serializing it back to a string before sending it on.
That way you can be sure that your method only returns JSON to the client. If for some reason, your incoming data isn't JSON, then your method would encounter an error parsing the JSON, which you can then handle appropriately.
Encode.forJava isn't a helpful method to use here: it is used to encode a string to be inserted into a Java string literal.
output = Encode.forHtmlAttribute(br.readLine() works for me
everyone. I'm coding a function that connects to a server by using Class HttpURLConnection. In the code, I establish a connection, call getOutputStream() and getInputStream() methods in order. Then I disconnect the connection. After this, I try to get data which has been obtained by getInputStream() method, but the compiler reminds NullPointerException.
Code in below:
DataOutputStream out = null;
InputStreamReader inStrReader = null;
BufferedReader reader = null;
HttpURLConnection connection = null;
try {
URL postUrl = new URL(null, url, new sun.net.www.protocol.https.Handler());
connection = (HttpURLConnection) postUrl.openConnection();
...//some setting methods
connection.connect();
out = new DataOutputStream(connection.getOutputStream());
out.writeBytes(JSONObject.toJSONString(param));
out.flush();
out.close();
inStrReader = new InputStreamReader(connection.getInputStream(), "utf-8");
reader = new BufferedReader(inStrReader);
connection.disconnect(); //<--HERE, release the connection
StringBuilder stringBuilder = new StringBuilder();
for (String line = reader.readLine(); line != null; line = reader.readLine()) { //<--null pointer
stringBuilder.append(line);
}
} catch (Exception e) {
e.printStackTrace();
return null;
} finally {
if (out != null) {
try {
out.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if (inStrReader != null) {
try {
inStrReader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if (reader != null) {
try {
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
After debug attempts, When I move the disconnection line to the last line in finally module, everything will be ok. But I'm confused, which happens when I already assgined the 'inputstream' value to 'reader'.
Thanks a lot.
Assigning isn't equal to reading, reader.readLine() start read from connection.
InputStreamReader is using the connection to read bytes, you disconnect before it read the bytes using the connection
An InputStreamReader is a bridge from byte streams to character
streams: It reads bytes and ...
Remember it is an "stream". You need to have an active connection to read from stream. Close the connection only after you have retrieved your data from stream.
You're doing everything in the wrong order. It doesn't make sense.
You're disconnecting and then expecting to be able to read from the connection. Total nonsense here. Normally you shouldn't disconnect at all, as you interfere with HTTP connection pooling. Just remove it, or, if you must have it, do it after all the closes.
You're closing in the wrong order, but you don't need to close inStrReader at all. Closing the BufferedReader does that. Just remove all the code for inStrReader.close().
You're closing out twice. Don't do that.
connect() happens implicitly. You don't need to call it yourself.
new URL(url) is sufficient. You haven't needed to provide the HTTPS Handler since about 2003.
I have a Java Steam Trade Bot that reads through pending Trade Offers from Steam and declines them based on requirements. I am using the official Web API (using the API Key from http://steamcommunity.com/dev/apikey) to communicate requests to Steam. The variable trade is from my own API Interface (which I have debugged and works for declining offers).
SteamPlug.steamRequest(method, query); is just a basic HTTP requester:
public static String steamRequest(String method, String query) {
try {
URL obj = new URL(query);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod(method);
int responseCode = con.getResponseCode();
if (responseCode != 200 && responseCode != 201) {
return "ERR" + responseCode;
}
BufferedReader in = new BufferedReader(
new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuilder response = new StringBuilder();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
return response.toString();
} catch (MalformedURLException | ProtocolException e) {
e.printStackTrace();
} catch (IOException e) {
// Ignored
}
return null;
}
This is how it declines trade offers:
SteamPlug.steamRequest(
"POST",
"http://api.steampowered.com/IEconService/DeclineTradeOffer/v0001/?key="
+ SteamPlug.API_KEY + "&tradeofferid=" + trade.getTradeOfferId()
);
What I am trying to do is accepting trades as well. I have tried this:
SteamPlug.steamRequest(
"POST",
"https://steamcommunity.com/tradeoffer/" + trade.getTradeOfferId() + "/accept?key="
+ SteamPlug.API_KEY
);
But I receive a 411 Length Required response.
I believe I can accept offers by using a Steam session authentification, but is it possible to accept a Trade Offer using only the user's Web-API key?
Have you read steam API docs? There's no trade accept function, so I assume you can't accept trades via API.
Your option is to check SteamBot's (though it is written in C#) trade functions and think about straight http requests to steam site (with proper steam authentication I believe). I myself writing a bot currently, and single thing that stops me now - accepting trades.
I am digging for quite a while and I am wondering how do I open an HttpClient connection in Java (Android) and then close the socket(s) right away without getting CLOSE_WAIT and TIME_WAIT TCP statuses while I am checking network monitoring tools.
What I am doing is (Found this solution on stackoverflow site):
String url = "http://example.com/myfile.php";
String result = null;
InputStream is = null;
StringBuilder sb = null;
try {
HttpClient httpclient = new DefaultHttpClient();
HttpPost httppost = new HttpPost(url);
HttpResponse response = httpclient.execute(httppost);
HttpEntity entity = response.getEntity();
is = entity.getContent();
} catch (Exception e) {
Log.e("log_tag", "Error in http connection" + e.toString());
}
// convert response to string
try {
BufferedReader reader = new BufferedReader(new InputStreamReader(
is, "iso-8859-1"), 8);
sb = new StringBuilder();
sb.append(reader.readLine() + "\n");
String line = "0";
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
is.close();
result = sb.toString();
} catch (Exception e) {
}
Toast.makeText(getApplicationContext(), result, Toast.LENGTH_LONG).show();
After I run this code - The PHP file is executed well, I get the response back to TOAST, BUT - when I analyze the networking environment of my mobile device with external network analyzer tool - I see that the connection(s) stay in CLOSE_WAIT or/and TIME_WAIT for about 1 minute and only then they move to CLOSED state.
The problem is:
I am calling the above function every ~2 to 5 seconds in an infinite loop, which result over time a huge amount of CLOSE_WAITs and TIME_WAITs - which affect the overall performance of my Android app, until it gets stuck and useless !
What I want to do is (And need your answer if possible):
I wish to really close the connection RIGHT AWAY after I TOAST the response message without any open sockets. No TIME_WAIT and no CLOSE_WAIT. No left overs at all - close all communication IMMEDIATELY at the split second that I run code that should do so. I don't need the connection anymore until the next iteration of the loop.
How can I accomplish that ?
I have in mind that I don't want the application to halt or have poor performance over time, since it should run in a service/stay open forever.
I would really appreciate if you could write simple code that work after I do copy-paste.
I am new to Java and Android, so I will try to figure out the code that you write, so please keep it as simple as possible. Thanks a lot !
Question asker.
try using HttpURLConnection class. refer to following link :
http://android-developers.blogspot.de/2011/09/androids-http-clients.html
in the finally clause just call disconnect. Example code ..
try {
HttpURLConnection locConn = (HttpURLConnection) locurl.openConnection();
//URL url = locConn.getURL();
locConn.setRequestProperty("Authorization", basicAuth);
locConn.setRequestMethod("GET");
locConn.setRequestProperty("Content-Type", "application/json");
locConn.setRequestProperty("X-Myauthtoken", userCredentials);
retc = locConn.getResponseCode();
reader = new BufferedReader(new InputStreamReader(locConn.getInputStream()));
String sessionK = null;
readVal = reader.readLine();
if (retc == 200) {
}
}catch (...)
{
//handle exception
}finally {
//disconnect here
locConn.disconnect();
}
HttpClient httpclient = new HttpClient();
GetMethod httpget = new GetMethod("http://www.myhost.com/");
try {
httpclient.executeMethod(httpget);
Reader reader = new InputStreamReader(httpget.getResponseBodyAsStream(), httpget.getResponseCharSet());
// consume the response entity
} finally {
httpget.releaseConnection();
}
I am trying to post XML data to a server which processes this XML and gives a response back, some times the XML data can be very large and the server takes a while to process it, in those cases i fail to get any response back instead i receive a IOException with the message "Unexpected end of file from server". I am positive the XML being sent to the server is not full of errors, is there anything i can do on my end to make sure this doesn't happen or is this a server side issue? Below is code fragment of the method i call to post the data.
Thanks.
String encodedData="some XML"
String urlString="example.com"
try {
URL url = new URL(urlString);
HttpURLConnection urlConnection;
urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.setDoOutput(true);
urlConnection.setDoInput(true);
urlConnection.setRequestMethod("POST");
urlConnection.setRequestProperty("Content-Type", contentType);
urlConnection.setRequestProperty("Content-Length", encodedData.length()+"");
OutputStream os = urlConnection.getOutputStream();
os.write(encodedData.getBytes());
BufferedReader in = new BufferedReader(new InputStreamReader(urlConnection.getInputStream(), "UTF-8"));
StringBuffer sb = new StringBuffer();
String inputLine;
while ((inputLine = in.readLine()) != null) {
sb.append(inputLine);
}
in.close();
} catch (MalformedURLException e) {
logger.debug("MalformedURLException " + e.getMessage());
logger.debug((new StringBuilder()).append("urlString=").append(urlString).toString());
throw (e);
} catch (IOException e) {
logger.debug("IOException " + e.getMessage());
logger.debug((new StringBuilder()).append("urlString=").append(urlString).toString());
throw (e);
}
return sb.toString();
Not much could be done from the client side, it's just a server side issue in which the server takes very long to process the data which results in the servers connection to timeout before it could send a response.