We Keep Coding

HTTPUrlConnection Write POST parameters as well as binary data to PHP in Java

I am trying to write a program such that I can use $_POST['id'] to specify an ID and to write the rest of the binary data into "php://input". My current Java code looks like:
HttpURLConnection connection = (HttpURLConnection) new URL("https://example.com/write?id="+id).openConnection();
connection.setRequestMethod("POST");
connection.setDoInput(true);
connection.setDoOutput(true);
connection.addRequestProperty("Content-Type", "application/bin");
DataOutputStream os = new DataOutputStream(connection.getOutputStream());
os.flush();
gameLevel.writeCompressedBinary(os);
os.close();
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String result = reader.readLine();
reader.close();
return result;
Problem is, $_POST['id'] does not exist when processed by the PHP file, but $_REQUEST['id'] does. I do not want 'id' to be visible over a GET request. What would be the best way to accomplish what I'm trying to do?
I can't use any 3rd party libraries to accomplish this.

reading remote csv file without downloading

I have requirement to read remote big csv file line by line (basically streaming). After each read I want to persist record in db. Currently I am achieving it through below code but I am not sure if it download complete file and keep it in jvm memory. I assume it is not. Can I write this code in better way using some java 8 stream features
URL url = new URL(baseurl);
HttpURLConnection urlConnection = url.openConnection();
if(connection.getResponseCode() == 200)
{
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String current;
while((current = in.readLine()) != null)
{
persist(current);
}
}

First you should use a try-with-resources statement to automatically close your streams when reading is done.
Next BufferedReader has a method BufferedReader::lines which returns a Stream<String>.
Then your code should look like this:
URL url = new URL(baseurl);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
if (connection.getResponseCode() == 200) {
try (InputStreamReader streamReader = new InputStreamReader(connection.getInputStream());
BufferedReader br = new BufferedReader(streamReader);
Stream<String> lines = br.lines()) {
lines.forEach(s -> persist(s)); //should be a method reference
}
}
Now it's up to you to decide if the code is better and your assumption is right that you don't keep the whole file in the JVM.

HTTP URL connection response

I am trying to hit the URL and get the response from my Java code.
I am using URLConnection to get this response. And writing this response in html file.
When opening this html in browser after executing the java class, I am getting only google home page and not with the results.
Whats wrong with my code, my code here,
FileWriter fWriter = null;
BufferedWriter writer = null;
URL url = new URL("https://www.google.co.in/?gfe_rd=cr&ei=aS-BVpPGDOiK8Qea4aKIAw&gws_rd=ssl#q=google+post+request+from+java");
byte[] encodedBytes = Base64.encodeBase64("root:pass".getBytes());
String encoding = new String(encodedBytes);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("User-Agent", "Mozilla/5.0");
connection.setRequestProperty("Accept-Charset", "UTF-8");
connection.setDoInput(true);
connection.setRequestProperty("Authorization", "Basic " + encoding);
connection.connect();
InputStream content = (InputStream) connection.getInputStream();
BufferedReader in = new BufferedReader(new InputStreamReader(content));
String line;
try {
fWriter = new FileWriter(new File("f:\\fileName.html"));
writer = new BufferedWriter(fWriter);
while ((line = in.readLine()) != null) {
String s = line.toString();
writer.write(s);
}
writer.close();
} catch (Exception e) {
e.printStackTrace();
}
}
Same code works couple of days back, but not now.

The reason is that this url does not return search results it self. You have to understand google's working process to understand it. Open this url in your browser and view its source. You will only see lots of javascript there.
Actually, in a short summary, google uses Ajax requests to process search queries.
To perform required task you either have to use a headless browser (the hard way) which can execute javascript/ajax OR better use google search api as directed by anand.

This method of searching is not advised is supposed to fail, you must use google search APIs for this kind of work.
Note: Google uses some redirection and uses token, so even if you will find a clever way to handle it, it is ought to fail in long run.
Edit:
This is a sample of how using Google search APIs you can get your work done in reliable way; please do refer to the source for more information.
public static void main(String[] args) throws Exception {
String google = "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=";
String search = "stackoverflow";
String charset = "UTF-8";
URL url = new URL(google + URLEncoder.encode(search, charset));
Reader reader = new InputStreamReader(url.openStream(), charset);
GoogleResults results = new Gson().fromJson(reader, GoogleResults.class);
// Show title and URL of 1st result.
System.out.println(results.getResponseData().getResults().get(0).getTitle());
System.out.println(results.getResponseData().getResults().get(0).getUrl());
}

How Do You Upload a File to a Server With PHP using Java [duplicate]

This question already has answers here:
How to send PUT, DELETE HTTP request in HttpURLConnection?
(8 answers)
Closed 9 years ago.
I have a Java applet that needs to upload a file from the user's computer to the server.
I am unable to add other libraries (such as com.apache). Is there a low level way of doing so.
Currently, I have a php file on the server which contains:
//Sets the target path for the upload.
$target_path = "spelling/";
$_FILES = $_POST;
var_dump($_FILES);
move_uploaded_file($_FILES["tmp_name"], $target_path . $_FILES["name"]);
?>
Currently my Java program is sending parameters via POST to this php file.
It sends these parameters by POST using the following code:
try {
//Creates a new URL containing the php file that writes files on the ec2.
url = new URL(WEB_ADDRESS + phpFile);
//Opens this connection and allows for the connection to output.
connection = url.openConnection();
connection.setDoOutput(true);
//Creates a new streamwriter based on the output stream of the connection.
OutputStreamWriter wr = new OutputStreamWriter(connection.getOutputStream());
//Writes the parameters to the php file on the ec2 server.
wr.write(data);
wr.flush();
//Gets the response from the server.
//Creates a buffered input reader from the input stream of the connection.
BufferedReader rd = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String line;
//Loops through and reads the response. Loops until reaches null line.
while ((line = rd.readLine()) != null) {
//Prints to console.
System.out.println(line);
}
//Closes reader and writer.
wr.close();
rd.close();
} catch (Exception e) {
e.printStackTrace();
}
This works for POST'ing data but when I try to send a file using this method, nothing happens (no response from the server nor is the file uploaded).
If anyone has any hints I would be grateful :)

Are you using java.net.URLConnection?
You may want to get some help on this page:
http://www.codejava.net/java-se/networking/upload-files-by-sending-multipart-request-programmatically
Here is the main part:
boundary = "===" + System.currentTimeMillis() + "===";
URL url = new URL(requestURL);
httpConn = (HttpURLConnection) url.openConnection();
httpConn.setDoOutput(true); // indicates POST method
httpConn.setDoInput(true);
httpConn.setRequestProperty("Content-Type",
"multipart/form-data; boundary=" + boundary);
outputStream = httpConn.getOutputStream();
writer = new PrintWriter(new OutputStreamWriter(outputStream, charset),
true);
But, you will need to have the php script be on the same server where your applet came from.

How can I send POST data through url.openStream()?

i'm looking for tutorial or quick example, how i can send POST data throw openStream.
My code is:
URL url = new URL("http://localhost:8080/test");
InputStream response = url.openStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(response, "UTF-8"));
Could you help me ?

URL url = new URL(urlSpec);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod(method);
connection.setDoOutput(true);
connection.setDoInput(true);
// important: get output stream before input stream
OutputStream out = connection.getOutputStream();
out.write(content);
out.close();
// now you can get input stream and read.
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String line = null;
while ((line = reader.readLine()) != null) {
writer.println(line);
}

Use Apache HTTP Compoennts http://hc.apache.org/httpcomponents-client-ga/
tutorial: http://hc.apache.org/httpcomponents-client-ga/tutorial/html/fundamentals.html
Look for HttpPost - there are some examples of sending dynamic data, text, files and form data.

Apache HTTP Components in particular, the Client would be the best way to go.
It absracts a lot of that nasty coding you would normally have to do by hand