I'm have created an application which sends GET requests to a URL, and then downloads the full content of that page.
The client sends a GET to e.g. stackoverflow.com, and forwards the response to a parser, which has the resposibility to find all the sources from the page that needs to be downloaded with subsequent GET requests.
The method below is used to send those GET requests. It is called many times consecutively, with the URLs returned by the parser. Most of those URLs are located on the same host, and should be able to share the TCP connection.
public static void sendGetRequestToSubObject(String RecUrl)
{
URL url = new URL(recUrl.toString());
URLConnection connection = url.openConnection ();
InputStreamReader isr = new InputStreamReader(connection.getInputStream());
}
Each time this method is called, a new TCP connection is created (with a TCP 3-way handshake) and the GET is then sent on that connection. But I want to reuse the TCP connections, to improve performance.
I guess that since I create a new URL object each time the method is called, this is the way it going to work...
Maybe someone can help me do this in a better way?
Thanks!
HttpURLConnection will reuse connections if it can!
For this to work, several preconditions need to be fulfilled, mostly on the server side. Those preconditions are described in the article linked to above.
Found the problem! I was not reading the input stream properly. This caused the input stream objects to hang, and they could not be reused.
I only defined it, like this:
InputStreamReader isr = new InputStreamReader(connection.getInputStream());
but I never read from it :-)
I changed the read method as well. Instead of a buffered reader I stole this:
InputStream in = null;
String queryResult = "";
try {
URL url = new URL(archiveQuery);
HttpURLConnection urlConn = (HttpURLConnection) url.openConnection();
HttpURLConnection httpConn = (HttpURLConnection) urlConn;
httpConn.setAllowUserInteraction(false);
httpConn.connect();
in = httpConn.getInputStream();
BufferedInputStream bis = new BufferedInputStream(in);
ByteArrayBuffer baf = new ByteArrayBuffer(50);
int read = 0;
int bufSize = 512;
byte[] buffer = new byte[bufSize];
while(true){
read = bis.read(buffer);
if(read==-1){
break;
}
baf.append(buffer, 0, read);
}
queryResult = new String(baf.toByteArray());
} catch (MalformedURLException e) {
// DEBUG
Log.e("DEBUG: ", e.toString());
} catch (IOException e) {
// DEBUG
Log.e("DEBUG: ", e.toString());
}
}
From here: Reading HttpURLConnection InputStream - manual buffer or BufferedInputStream?
Related
API route in Python (Flask)
#app.route('/secret')
def secret():
if request.get_json(force=True)['key'] == 'secret key':
return jsonify(msg='Hello!')
It is working linux terminal
curl -iX GET -d '{"key":"secret key"}' localhost
Linux terminal output this
{"msg":"Hello!"}
It doesn't need to work in browser.
try{
HttpURLConnection connection = (HttpURLConnection)
new URL("http://<my local ip>/secret").openConnection();
connection.setRequestMethod("GET");
connection.setDoOutput(true);
connection.connect();
JSONObject jsonInput = new JSONObject();
jsonInput.put("key", "secret key");
OutputStream os = connection.getOutputStream();
byte[] input = jsonInput.toString().getBytes(StandardCharsets.UTF_8);
os.write(input, 0, input.length);
os.flush();
os.close();
BufferedReader br = new BufferedReader(new InputStreamReader(connection.getInputStream()));
StringBuilder response = new StringBuilder();
String responseLine = null;
while ((responseLine = br.readLine()) != null) {
response.append(responseLine.trim());
}
return response.toString();
} catch (IOException | JSONException e) {
Log.e("MainActivity", "Error: " + e.getMessage());
}
Although the GET method is set to the connection request in my codes, a POST request is being sent to the Python server.
Python Interpreter
Is it impossible to fix this?
Request Body is not recommended in HTTP GET requests. See HERE
A payload within a GET request message has no defined semantics;
sending a payload body on a GET request might cause some existing
implementations to reject the request.
When you try to write on a URL, you are implicitly POSTing on it despite you had set GET as the HTTP method. At below lines:
OutputStream os = connection.getOutputStream();
byte[] input = jsonInput.toString().getBytes(StandardCharsets.UTF_8);
os.write(input, 0, input.length);
For confirmation of my words see Writing to a URLConnection
writing to a URL is often called posting to a URL. The server
recognizes the POST request and reads the data sent from the client.
I am trying to hit the URL and get the response from my Java code.
I am using URLConnection to get this response. And writing this response in html file.
When opening this html in browser after executing the java class, I am getting only google home page and not with the results.
Whats wrong with my code, my code here,
FileWriter fWriter = null;
BufferedWriter writer = null;
URL url = new URL("https://www.google.co.in/?gfe_rd=cr&ei=aS-BVpPGDOiK8Qea4aKIAw&gws_rd=ssl#q=google+post+request+from+java");
byte[] encodedBytes = Base64.encodeBase64("root:pass".getBytes());
String encoding = new String(encodedBytes);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("User-Agent", "Mozilla/5.0");
connection.setRequestProperty("Accept-Charset", "UTF-8");
connection.setDoInput(true);
connection.setRequestProperty("Authorization", "Basic " + encoding);
connection.connect();
InputStream content = (InputStream) connection.getInputStream();
BufferedReader in = new BufferedReader(new InputStreamReader(content));
String line;
try {
fWriter = new FileWriter(new File("f:\\fileName.html"));
writer = new BufferedWriter(fWriter);
while ((line = in.readLine()) != null) {
String s = line.toString();
writer.write(s);
}
writer.close();
} catch (Exception e) {
e.printStackTrace();
}
}
Same code works couple of days back, but not now.
The reason is that this url does not return search results it self. You have to understand google's working process to understand it. Open this url in your browser and view its source. You will only see lots of javascript there.
Actually, in a short summary, google uses Ajax requests to process search queries.
To perform required task you either have to use a headless browser (the hard way) which can execute javascript/ajax OR better use google search api as directed by anand.
This method of searching is not advised is supposed to fail, you must use google search APIs for this kind of work.
Note: Google uses some redirection and uses token, so even if you will find a clever way to handle it, it is ought to fail in long run.
Edit:
This is a sample of how using Google search APIs you can get your work done in reliable way; please do refer to the source for more information.
public static void main(String[] args) throws Exception {
String google = "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=";
String search = "stackoverflow";
String charset = "UTF-8";
URL url = new URL(google + URLEncoder.encode(search, charset));
Reader reader = new InputStreamReader(url.openStream(), charset);
GoogleResults results = new Gson().fromJson(reader, GoogleResults.class);
// Show title and URL of 1st result.
System.out.println(results.getResponseData().getResults().get(0).getTitle());
System.out.println(results.getResponseData().getResults().get(0).getUrl());
}
I'm trying to invoke a webservice call and get a response. When I tried it first time it worked perfectly and printed the response. But after that one run, how many ever times I run it, i throws me
Exception in thread "main" java.lang.IllegalStateException: Already connected
at sun.net.www.protocol.http.HttpURLConnection.setRequestProperty(Unknown Source)
at SOAPClient4XG.main(SOAPClient4XG.java:72)
I have tried various solutions provided for similar problem (like connect / disconnect) but nothing seems to make it work. I understand that it tries to perform an operation on already existing connection, but not sure how to fix. I'm fairly new to all this and I need help.
Below is my code
import java.io.*;
import java.net.*;
public class SOAPClient4XG
{
private static HttpURLConnection httpConn;
public static void main(String[] args) throws Exception {
String SOAPUrl = args[0];
String xmlFile2Send = args[1];*/
String SOAPUrl = "http://10.153.219.88:8011/celg-svcs-soap/business/ApplicantEligibility";
String xmlFile2Send =
"C:\\Users\\dkrishnamoorthy\\workspace\\SOAPUI_Automation\\src\\ApplicantElligibilty.xml";
String SOAPAction = "";
if (args.length > 2)
SOAPAction = args[2];
// Create the connection where we're going to send the file.
URL url = new URL(SOAPUrl);
URLConnection connection = url.openConnection();
//URLConnection connection = new URLConnection(url);
httpConn = (HttpURLConnection) connection;
if(httpConn.getResponseCode()==500)
{
System.out.println("Error Stream for 500 : "+httpConn.getErrorStream());
}
// Open the input file. After we copy it to a byte array, we can see
// how big it is so that we can set the HTTP Cotent-Length
// property. (See complete e-mail below for more on this.)
FileInputStream fin = new FileInputStream(xmlFile2Send);
ByteArrayOutputStream bout = new ByteArrayOutputStream();
// Copy the SOAP file to the open connection.
copy(fin,bout);
fin.close();
byte[] b = bout.toByteArray();
// Set the appropriate HTTP parameters.
httpConn.setRequestProperty( "Content-Length",
String.valueOf( b.length ) );
httpConn.setRequestProperty("Content-Type","text/xml; charset=utf-8");
httpConn.setRequestProperty("SOAPAction",SOAPAction);
httpConn.setRequestMethod( "POST" );
httpConn.setDoOutput(true);
httpConn.setDoInput(true);
// httpConn.connect();
// Everything's set up; send the XML that was read in to b.
OutputStream out = httpConn.getOutputStream();
out.write( b );
out.close();
// Read the response and write it to standard out.
InputStreamReader isr =
new InputStreamReader(httpConn.getInputStream());
BufferedReader in = new BufferedReader(isr);
String inputLine;
System.out.println("Printing the Response ");
while ((inputLine = in.readLine()) != null)
System.out.println(inputLine);
in.close();
}
public static void copy(InputStream in, OutputStream out)
throws IOException {
synchronized (in) {
synchronized (out) {
byte[] buffer = new byte[256];
while (true) {
int bytesRead = in.read(buffer);
if (bytesRead == -1) break;
out.write(buffer, 0, bytesRead);
}
}
}
}
}
If you use eclipse version just restart it. I met the same issue and I sorted out by doing that .
I solved this because I had a forgotten watch for connection.getResponseCode() in my debugging interface in NetBeans. Hope it might help others making the same mistake.
If you have any watch relative to the response value of the request, such as getResponseCode(), getResponseMessage(), getInputStream() or even just connect(), you will get this error in debugging mode.
All of the previous methods implicitly call connect() and fire the request. So when you reach setDoOutput, the connection is already made.
I am trying to use a for loop to send multiple POST requests through a DataOutputStream and then close it. At the moment, only the first index of the "trades" array list is sent to the website. Any other indexes are ignored and I'm assuming they are not being sent. I wonder if I am properly flushing the stream? Thank you!!!
Examples of trades values: "101841599", "101841801"
Example of code value: 85e4c22
Snippet of my code:
private ArrayList<String> trades = new ArrayList<String>();
private String code;
String url = "http://www.dota2lounge.com/ajax/bumpTrade.php";
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("POST");
con.setRequestProperty("Accept-Language", "en-US,en;q=0.8");
con.setRequestProperty("Cookie", cookie);
con.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
for(int i=0; i<trades.size(); i++){
wr = new DataOutputStream(con.getOutputStream());
wr.writeBytes("trade=" + trades.get(i) + "&code=" + code);
wr.flush();
System.out.println("again");
}
wr.flush();
wr.close();
It turns out I had to actually get the response for it to properly close the connection before I started a new one. Appending these lines to the end of the for loop fixed the issue:
int nothing = con.getResponseCode();
String morenothing = con.getResponseMessage();
From the HttpURLConnection javadoc:
"Each HttpURLConnection instance is used to make a single request but the underlying network connection to the HTTP server may be transparently shared by other instances."
So if you want to send multiple requests, then for each request call obj.openConnection(), set the connection settings, open the OutputStream, and write the data. Your Java runtime is permitted to keep the actual connection open to save time and bandwidth.
I'm trying to upload images to External MS SQL database using android phone. I'm using Java HttpClient to send array of bytes to web server or web page. I don't know how I should approach this. The web page should be in ASP.net. I'm fairly new to ASP.Net. I did intensive research on how to read in a byte array using ASP.Net and still don't have an answer. I want my webpage or server to read in the bytes and store them into database.
Below is my Java function (it is not tested yet since I don't have a way to read bytes yet) that I want to use to send the bytes. But I have no idea how to read them in on website side. Any suggestions would be appreciated. If you guys see that I'm doing something wrong also it would be appreciated if you let me know and tell me how I should fix it. Please be specific since I'm really new to this and don't really know much about web pages. Thanks.
private void sendImagesToServer() throws Exception
{
ImageItem image;
HttpURLConnection conn = null;
ImageIterator iterator;
DataOutputStream dos;
byte[] byteArray;
iterator = new ImageIterator(imageAdapter);
String uploadUrl;
while(iterator.hasNext())
{
image = iterator.getNext();
Uri uri = Uri.withAppendedPath(MediaStore.Images.Media.EXTERNAL_CONTENT_URI, Integer.toString(image.id));
Bitmap bmp=BitmapFactory.decodeStream(getContentResolver().openInputStream(uri));
ByteArrayOutputStream stream = new ByteArrayOutputStream();
bmp.compress(Bitmap.CompressFormat.JPEG, 100, stream);
byteArray = stream.toByteArray();
uploadUrl = "http://localhost:63776/SQLScript.aspx";
// Send request
try {
// Configure connection
URL url = new URL(uploadUrl);
conn = (HttpURLConnection) url.openConnection();
conn.setDoInput(true);
conn.setDoOutput(true);
conn.setUseCaches(false);
conn.setRequestMethod("PUT");
conn.setRequestProperty("Connection", "Keep-Alive");
dos = new DataOutputStream(conn.getOutputStream());
dos.write(byteArray, 0, byteArray.length);
dos.flush();
dos.close();
// Read response
try {
if (conn.getResponseCode() == 200) {
Toast.makeText(this,"Yay, We Got 200 as Response Code",Toast.LENGTH_SHORT).show();
}
} catch (IOException ioex) {
ioex.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
} finally{
}
}
}
If you've verified the bytes are getting out of the Java fine, check this question, it may have what you need.
Read Http Request into Byte array
As far as getting it into a database, you could save files in a binary database (different MSSQL setup) or convert to strings and back again as necessary.