My problem is that I want to use Java to implement an application which sends an HTTP GET request to some website. However, the target website needs one cookie to be set:
ShippingCountry=US
If this cookie is not set it returns bad indications. Below are my code segment and I get null from connect().
String urlString = "http://www1.macys.com/catalog/index.ognc?CategoryID=5449&viewall=true";
try{
URL url = new URL(urlString);
URLConnection connection = url.openConnection();
connection.addRequestProperty("Cookie", "ShippingCountry=US");
connection.connect();
// Create file
FileWriter fstream = new FileWriter("d:/out.txt");
BufferedWriter out = new BufferedWriter(fstream);
BufferedReader rd = new BufferedReader(new InputStreamReader(connection.getInputStream()));
StringBuffer sb = new StringBuffer();
String line;
while ((line = rd.readLine()) != null)
{
out.write(line);
}
rd.close();
//Close the output stream
out.close();
}
Can somebody help me?
Just a guess but perhaps you might need setRequestProperty instead of addRequestProperty, since there can be only one Cookie string in a request.
connection.setRequestProperty("Cookie", "ShippingCountry=US");
If you need to send multiple cookie values you tack them on with colons:
connection.setRequestProperty("Cookie", "ShippingCountry=US;OtherValue=2");
Update:
I tried doing a GET request using python and it looks like I get a 500 error. Perhaps the server is blocking the request because it doesn't look like a web browser. Maybe adding the following headers will work:
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Connection: keep-alive
In particular the Accept and User-Agent headers might be required.
Related
I am trying to make a request to my RESTful API using Android and HttpURLConnection. The data must be sent in the JSON format via POST data.
Here is my code:
JSONObject check_request = new JSONObject();
check_request.put("username", username);
JSONObject request = BuildRequest(check_request, "username_check", false);
Log.i("DEBUG", request.toString());
// DEBUG OUTPUT: {"timestamp":1526900318,"request":{"username":"blubberfucken","type":"username_check"}}
URL request_url = new URL(apiURL);
HttpURLConnection connection = (HttpURLConnection)request_url.openConnection();
connection.setRequestProperty("User-Agent", "TheGameApp");
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-type", "application/json; charset=UTF-8");
connection.setDoOutput(true);
connection.setDoInput(true);
OutputStream os = connection.getOutputStream();
os.write(request.toString().getBytes("UTF-8"));
os.flush();
InputStream in = new BufferedInputStream(connection.getInputStream());
String result = "";
BufferedReader br = new BufferedReader(new InputStreamReader(in, "UTF8"));
String str;
while ((str = br.readLine()) != null)
{
result += str;
}
Log.i("DEBUG", result);
//JSONObject result_json = new JSONObject(result);
os.close();
in.close();
connection.disconnect();
You can see the Debug output as a Comment. The Problem is that the API does not receive any POST data. I have used PHPs var_dump to dump $_POST and $_REQUEST which both are empty arrays.
What am I missing here?
As the question popped up if the API work. This cURL command works fine with the correct result (it is the same JSON data as the debugger printed):
curl -d '{"timestamp":1526900318,"request":{"username":"blubberfucken","type":"username_check"}}' -H "Content-Type: application/json" -X POST http://localhost/v1/api.php
Just for the sake of completeness: The example above is working. The solution to the problem was pa part in PHP on the server side, where I checked the content type and used strpos to search for application/json in $_SERVER['CONTENT-TYPE'] and switched the needle and haystack (thus searching for application/json; charset=UTF8 in the string application/json instead of the other way around).
I'm behind a corporate firewall, but i can paste the URL in my browser with and without my proxy settings enabled within the browser and can retrieve the data fine. I just can't within java.
Any ideas?
Code:
private static String getURLToString(String strUrl) throws IOException {
// LOG.debug("Calling URL: [" + strUrl + "]");
String content = "";
URLConnection connection = new URL(strUrl).openConnection();
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
connection.connect();
BufferedReader br = new BufferedReader(new InputStreamReader(connection.getInputStream(), Charset.forName("UTF-8")));
String inputLine;
while ((inputLine = br.readLine()) != null) {
content += inputLine;
}
br.close();
return content;
}
Error:
java.io.FileNotFoundException: Response: '403: Forbidden' for url: '<url here>'
at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:778)
at weblogic.net.http.SOAPHttpURLConnection.getInputStream(SOAPHttpURLConnection.java:37)
Note: The '' portion is for anonymizing.
As you are receiving a "403: Forbidden" error, it means that your Java code can reach the URL, but it lacks something that is required to access it.
In the browser, press F12 (developer/debug mode) and request the URL again. Check the headers and cookies that are being sent. Most likely you will need to add one of these for you to be able to receive the content you need.
Adding "User-Agent" header fixed it for me:
connection.setRequestProperty("User-Agent", "Mozilla/5.0");
I want to update WEB UI of Rails by ajax request triggered from java.
However, it didn't work although I saw 200 OK message.
This is my step I did.
Make Ajax request in Java application (using HttpURLConnection object)
- X-Request-With : XMLHttpRequest
String url = "http://127.0.0.1:3000/java/fromjava";
String charset = "UTF-8";
String query = String.format("param1=%s¶m2=%s", param1, param2);
try {
HttpURLConnection urlConnection = (HttpURLConnection) new URL(url)
.openConnection();
urlConnection.setDoOutput(true);
urlConnection.setUseCaches(true);
urlConnection.setRequestMethod("POST");
urlConnection.setRequestProperty("Accept-Encoding", "gzip,deflate,sdch");
urlConnection.setRequestProperty("Accept-Language", "ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4");
urlConnection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded; charset=UTF-8");
urlConnection.setRequestProperty("Accept", "*/*");
urlConnection.setRequestProperty("X-Requested-With", "XMLHttpRequest");
urlConnection.setRequestProperty("Referer", "http://192.168.43.79:3000/");
urlConnection.setRequestProperty("User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36");
urlConnection.connect();
OutputStreamWriter writer = null;
writer = new OutputStreamWriter(urlConnection.getOutputStream(),
charset);
writer.write(query);
writer.flush();
if (writer != null)
try {
writer.close();
} catch (IOException logOrIgnore) {
}
respond to js in controller
- Set 'rack-CORS' to resolve same origin policy
class JavaController < ApplicationController
skip_before_filter :verify_authenticity_token
def fromjava
respond_to do|format|
format.js
end
end
end
change WEB UI
[fromjava.js.erb]
$('#container').empty().append('This is [ajax] text');
[java.html.erb]
<div id="container">
This is default text
</div>
Log message in log/development.log
Started POST "/java/fromjava" for 127.0.0.1 at [current date]
Processing by JavaController#fromjava as */*
Rendered java/fromjava.js.erb (0.0ms)
Complete 200 OK in 43ms (View: 39.0ms | ActiveRecord: 0.0ms)
I saw the 200 OK message in log. However, Didn't update WEB UI. What is the problem?
Also, Are there other way to track the progress of the request & response ??
First
you need to make sure that you are passing the the .js in your request url in order to get the response based on fromjava.js.erb. your url in the java code should be
String url = "http://127.0.0.1:3000/java/fromjava.js";
Second as I can see in the fromjava.js.erb you just passing a jQuery code, OK but you need to make sure that the jQuery code is being executed once the response is received on the client side, you just send the response which is code but it hasn't been executed on the client's machine.
So, I have some Java code that fetches the contents of a HTML page as follows:
BufferedReader bf;
String response = "";
HttpURLConnection connection;
try
{
connection = (HttpURLConnection) url.openConnection();
connection.setInstanceFollowRedirects(false);
connection.setUseCaches(false);
connection.setRequestMethod("GET");
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.0; WOW64) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.16 Safari/534.24");
connection.connect();
bf = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String line;
while ((line = bf.readLine()) != null) {
response += line;
}
connection.disconnect();
}
catch (Throwable ex)
{
response = "";
}
This works perfectly fine and will return the content to me as required. I then drill down to the area of code that I want to pull, which is as follows:
10€ de réduction chez Asos be!
Java seems to be handling the € fine since it is a HTML entity. The word "réduction" is problematic though. It seems to render it as:
10€ de r�duction chez Asos be!
As you can see it is struggling to handle the "é" character.
How do I go about solving this? I've been searching the internet and playing around with the code for the past few hours but no luck whatsoever! I'm very new to Java so it's all very difficult to get my head around.
Thanks in advance.
That code is ok but you might need to detect the character encoding of the response (see here) and pass it to the class that wraps the inputStream to get a Reader (see here).
Otherwise the problem is not reading the response but in the stuff you do with that response string.
I need to download a CSV file from Google insights programatically. Since it requires authentication, I used the clientLogin to get the session id.
How do I download the file by passing the session id as a cookie?
I tried using a new URLConnection object and set the cookie in setRequestParameter method hoping it would authenticate my login then, however it doesn't seem to be working. I have a feeling I shouldn't use two separate connections, is that true?
If so then how do I pass session id as parameter when i download the file? I also tried using the same connection this didn't work either. Please help.
try {
URL url1 = new URL("https://www.google.com/accounts/ClientLogin?accountType=GOOGLE&Email=*******.com&Passwd=*****&service=trendspro&source=test-test-v1");
URL url2 = new URL("http://www.google.com/insights/search/overviewReport?cat=0-7&geo=BR&cmpt=geo&content=1&export=1");
URLConnection conn = url1.openConnection();
// fake request coming from browser
conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11");
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream(), "UTF-8"));
String f = in.readLine();
// obtaining the sid.
String sid=f.substring(4);
System.out.println(sid);
URLConnection conn2 = url2.openConnection();
conn2.setRequestProperty("Cookie", sid);
BufferedInputStream i= new BufferedInputStream(conn2.getInputStream());
FileOutputStream fos = new FileOutputStream("f:/testplans.csv");
BufferedOutputStream bout = new BufferedOutputStream(fos,1024);
byte data[] = new byte[1024];
while(i.read(data,0,1024)>=0) {
bout.write(data);
}
bout.close();
in.close();
}
Try the following: link. Check the top answer: they don't use the SID, but the Auth.
If it's working for Google Reader, it will probably work for Google Insights as well.