We Keep Coding

HTTP URL connection response

I am trying to hit the URL and get the response from my Java code.
I am using URLConnection to get this response. And writing this response in html file.
When opening this html in browser after executing the java class, I am getting only google home page and not with the results.
Whats wrong with my code, my code here,
FileWriter fWriter = null;
BufferedWriter writer = null;
URL url = new URL("https://www.google.co.in/?gfe_rd=cr&ei=aS-BVpPGDOiK8Qea4aKIAw&gws_rd=ssl#q=google+post+request+from+java");
byte[] encodedBytes = Base64.encodeBase64("root:pass".getBytes());
String encoding = new String(encodedBytes);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("User-Agent", "Mozilla/5.0");
connection.setRequestProperty("Accept-Charset", "UTF-8");
connection.setDoInput(true);
connection.setRequestProperty("Authorization", "Basic " + encoding);
connection.connect();
InputStream content = (InputStream) connection.getInputStream();
BufferedReader in = new BufferedReader(new InputStreamReader(content));
String line;
try {
fWriter = new FileWriter(new File("f:\\fileName.html"));
writer = new BufferedWriter(fWriter);
while ((line = in.readLine()) != null) {
String s = line.toString();
writer.write(s);
}
writer.close();
} catch (Exception e) {
e.printStackTrace();
}
}
Same code works couple of days back, but not now.

The reason is that this url does not return search results it self. You have to understand google's working process to understand it. Open this url in your browser and view its source. You will only see lots of javascript there.
Actually, in a short summary, google uses Ajax requests to process search queries.
To perform required task you either have to use a headless browser (the hard way) which can execute javascript/ajax OR better use google search api as directed by anand.

This method of searching is not advised is supposed to fail, you must use google search APIs for this kind of work.
Note: Google uses some redirection and uses token, so even if you will find a clever way to handle it, it is ought to fail in long run.
Edit:
This is a sample of how using Google search APIs you can get your work done in reliable way; please do refer to the source for more information.
public static void main(String[] args) throws Exception {
String google = "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=";
String search = "stackoverflow";
String charset = "UTF-8";
URL url = new URL(google + URLEncoder.encode(search, charset));
Reader reader = new InputStreamReader(url.openStream(), charset);
GoogleResults results = new Gson().fromJson(reader, GoogleResults.class);
// Show title and URL of 1st result.
System.out.println(results.getResponseData().getResults().get(0).getTitle());
System.out.println(results.getResponseData().getResults().get(0).getUrl());
}

Getting JSON response as part of Rest call in Java

I am trying to make Rest service call in Java. I am new to web and Rest service. I have Rest service which returns JSON as response. I have the following code but I think it's incomplete because I don't know how to process output using JSON.
public static void main(String[] args) {
try {
URL url = new URL("http://example.com:7000/test/db-api/processor");
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setInstanceFollowRedirects(false);
connection.setRequestMethod("PUT");
connection.setRequestProperty("Content-Type", "application/json");
OutputStream os = connection.getOutputStream();
//how do I get json object and print it as string
os.flush();
connection.getResponseCode();
connection.disconnect();
} catch(Exception e) {
throw new RuntimeException(e);
}
}
I am new to Rest services and JSON.

Since this is a PUT request you're missing a few things here:
OutputStream os = conn.getOutputStream();
os.write(input.getBytes()); // The input you need to pass to the webservice
os.flush();
...
BufferedReader br = new BufferedReader(new InputStreamReader(
(conn.getInputStream()))); // Getting the response from the webservice
String output;
System.out.println("Output from Server .... \n");
while ((output = br.readLine()) != null) {
System.out.println(output); // Instead of this, you could append all your response to a StringBuffer and use `toString()` to get the entire JSON response as a String.
// This string json response can be parsed using any json library. Eg. GSON from Google.
}
Have a look at this to have a more clear idea on hitting webservices.

Your code is mostly correct, but there is mistake about OutputStream.
As R.J said OutputStream is needed to pass request body to the server.
If your rest service doesn't required any body you don't need to use this one.
For reading the server response you need use InputStream(R.J also show you example) like that:
try (InputStream inputStream = connection.getInputStream();
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();) {
byte[] buf = new byte[512];
int read = -1;
while ((read = inputStream.read(buf)) > 0) {
byteArrayOutputStream.write(buf, 0, read);
}
System.out.println(new String(byteArrayOutputStream.toByteArray()));
}
This way is good if you don't want to depends on third-part libraries. So I recommend you to take a look on Jersey - very nice library with huge amount of very useful feature.
Client client = JerseyClientBuilder.newBuilder().build();
Response response = client.target("http://host:port").
path("test").path("db-api").path("processor").path("packages").
request().accept(MediaType.APPLICATION_JSON_TYPE).buildGet().invoke();
System.out.println(response.readEntity(String.class));

Since your Content-Type is application/json, you could directly cast the response to a JSON object for example
JSONObject recvObj = new JSONObject(response);

JsonKey jsonkey = objectMapper.readValue(new URL("http://echo.jsontest.com/key/value/one/two"), JsonKey.class);
System.out.println("jsonkey.getOne() : "+jsonkey.getOne())

Corrupted JSON encoding?

I'm sending a JSON object of the same class from a servlet to an applet, but
all strings variables in this class are missing some characters like: 'ą', 'ę', 'ś', 'ń', 'ł'.
However, 'ó' is displayed normally (?). For example:
"Zaznacz prawid?ow? operacj? porównywania dwóch zmiennych typu"
Solution
I wish I could explain it more thoroughly, but as Henry noticed, it's IDE causing this issue. I solved it using farmer1992's class from the google ticket. It prints escaped unicode characters (\u...) - the only way my applet could encode characters correctly. Also I have to restart NetBeans IDE from time to time to force the Tomcat servlet to work correctly (I have no idea why :) ).
Servlet code (updated with solution):
//begin of the servlet code extract
public void sendToApplet(HttpServletResponse response, String path) throws IOException
{
TestServlet x = new TestServlet();
x.load(path);
String json = new Gson().toJson(x);
response.setCharacterEncoding("UTF-8");
response.setContentType("application/json;charset=UTF-8");
PrintWriter out = response.getWriter();
//out.print(json);
//out.flush();
GhettoAsciiWriter out2 = new GhettoAsciiWriter(out);
out2.write(json);
out2.flush();
}
//end of the servlet code extract
Applet code:
//begin of the applet code extract
public void retrieveFromServlet(String path) throws MalformedURLException, IOException
{
String encoder = URLEncoder.encode(path, "UTF-8");
URL urlServlet = new URL("http://localhost:8080/ProjektServlet?action=" + encoder);
URLConnection connection = urlServlet.openConnection();
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setUseCaches(false);
connection.setRequestProperty("Content-Type", "application/json;charset=UTF-8");
InputStream inputStream = connection.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(inputStream));
String json = br.readLine();
Test y = new Gson().fromJson(json, Test.class);
inputStream.close();
}
//end of the applet code extract

those chars should encode in \uxxxx form
you can see this ticket
http://code.google.com/p/google-gson/issues/detail?id=388#c4

With this line
BufferedReader br = new BufferedReader(new InputStreamReader(inputStream));
the platform character encoding will be used (which may or may not be UTF-8). Try to set the encoding explicitly with
BufferedReader br = new BufferedReader(new InputStreamReader(inputStream, "UTF-8"));

Send HTTP Post Payload with Java

I'm trying to connect to the grooveshark API, this is the http request
POST URL
http://api.grooveshark.com/ws3.php?sig=f699614eba23b4b528cb830305a9fc77
POST payload
{"method":'addUserFavoriteSong",'parameters":{"songID":30547543},"header":
{"wsKey":'key","sessionID":'df8fec35811a6b240808563d9f72fa2'}}
My question is how can I send this request via Java?

Basically, you can do it with the standard Java API. Check out URL, URLConnection, and maybe HttpURLConnection. They are in package java.net.
As to the API specific signature, try sStringToHMACMD5 found in here.
And remember to CHANGE YOUR API KEY, this is very IMPORTANT, since everyone knows it know.
String payload = "{\"method\": \"addUserFavoriteSong\", ....}";
String key = ""; // Your api key.
String sig = sStringToHMACMD5(payload, key);
URL url = new URL("http://api.grooveshark.com/ws3.php?sig=" + sig);
URLConnection connection = url.openConnection();
connection.setDoInput(true);
connection.setDoOutput(true);
connection.connect();
OutputStream os = connection.getOutputStream();
PrintWriter pw = new PrintWriter(new OutputStreamWriter(os));
pw.write(payload);
pw.close();
InputStream is = connection.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
String line = null;
StringBuffer sb = new StringBuffer();
while ((line = reader.readLine()) != null) {
sb.append(line);
}
is.close();
String response = sb.toString();

You could look into the Commons HttpClient package.
It is fairly straight forward to create POST's, specifically you could copy the code found here: http://hc.apache.org/httpclient-3.x/methods/post.html:
PostMethod post = new PostMethod( "http://api.grooveshark.com/ws3.php?sig=f699614eba23b4b528cb830305a9fc77" );
NameValuePair[] data = {
new NameValuePair( "method", "addUserFavoriteSong..." ),
...
};
post.setRequestBody(data);
InputStream in = post.getResponseBodyAsStream();
...
Cheers,

How can I send POST data through url.openStream()?

i'm looking for tutorial or quick example, how i can send POST data throw openStream.
My code is:
URL url = new URL("http://localhost:8080/test");
InputStream response = url.openStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(response, "UTF-8"));
Could you help me ?

URL url = new URL(urlSpec);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod(method);
connection.setDoOutput(true);
connection.setDoInput(true);
// important: get output stream before input stream
OutputStream out = connection.getOutputStream();
out.write(content);
out.close();
// now you can get input stream and read.
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String line = null;
while ((line = reader.readLine()) != null) {
writer.println(line);
}

Use Apache HTTP Compoennts http://hc.apache.org/httpcomponents-client-ga/
tutorial: http://hc.apache.org/httpcomponents-client-ga/tutorial/html/fundamentals.html
Look for HttpPost - there are some examples of sending dynamic data, text, files and form data.

Apache HTTP Components in particular, the Client would be the best way to go.
It absracts a lot of that nasty coding you would normally have to do by hand