I have a link to a .csv which looks like this:
http://website.com/data.csv?since=2019-03-21T00:00Z&sensor=xxx
My method get this link as URL
BufferedReader in = new BufferedReader(new InputStreamReader(URL.openStream()));
The problem is that it is not properly downloading .csv. I have realized that if I add 'amp;' after & it works properly. Is there any easy way to work this out?
I have found the solution: https://www.mkyong.com/java/java-httpurlconnection-follow-redirect-example/
public URL getFinalUrl(URL url) {
try {
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setReadTimeout(5000);
conn.addRequestProperty("Accept-Language", "en-US,en;q=0.8");
conn.addRequestProperty("User-Agent", "Mozilla");
conn.addRequestProperty("Referer", "google.com");
boolean redirect = false;
int status = conn.getResponseCode();
if (status != HttpURLConnection.HTTP_OK) {
if (status == HttpURLConnection.HTTP_MOVED_TEMP
|| status == HttpURLConnection.HTTP_MOVED_PERM
|| status == HttpURLConnection.HTTP_SEE_OTHER)
redirect = true;
}
String newUrl = null;
if (redirect) {
// get redirect url from "location" header field
newUrl = conn.getHeaderField("Location");
// get the cookie if need, for login
String cookies = conn.getHeaderField("Set-Cookie");
// open the new connnection again
conn = (HttpURLConnection) new URL(newUrl).openConnection();
conn.setRequestProperty("Cookie", cookies);
conn.addRequestProperty("Accept-Language", "en-US,en;q=0.8");
conn.addRequestProperty("User-Agent", "Mozilla");
conn.addRequestProperty("Referer", "google.com");
}
System.out.println(newUrl);
return new URL(newUrl);
} catch (IOException ioe) {
System.out.println("Wrong sensor URL");
}
return null;
}
How to call subsequent requests using j_security_check, I have followed below code but it's not working .
String url = "https://myhost/jsp/j_security_check?j_username=myuser&j_password=mypassword";
HttpURLConnection connection = (HttpURLConnection) new
URL(url).openConnection();
if (connection.getResponseCode() == 200) {
String cookie = connection.getHeaderField("Set-Cookie").split(";", 2)[0];
url = "https://myhost/getusers";
connection = (HttpURLConnection) new URL(url).openConnection();
connection.setRequestProperty("Cookie", cookie);
connection.setRequestProperty("Cache-Control", "no-cache");
InputStream input = connection.getInputStream();
try (BufferedReader buffer = new BufferedReader(new InputStreamReader(connection.getInputStream()))) {
String result = buffer.lines().collect(Collectors.joining("\n"));
System.out.println(result);
} catch (Exception e) {
e.printStackTrace();
}
}
Can anyone help me on it.
I have this method to connect to my webservice (REST API)
public static void getHttpCon() throws Exception{
String tokenUrl = AppPropertiesService.getProperty( URL_TOKEN );
String username = AppPropertiesService.getProperty( USERNAME );
String password = AppPropertiesService.getProperty( PASSWORD );
String POST_PARAMS = "username="+username+"&password="+password+"&lang=fr&grant_type=password&client_id=apiclient";
URL obj = new URL(tokenUrl);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("POST");
con.setRequestProperty("Content-Type", "application/json;odata=verbose");
con.setRequestProperty("Authorization",
"Basic Base64_encoded_clientId:clientSecret");
con.setRequestProperty("Accept",
"application/x-www-form-urlencoded");
// For POST only - START
con.setDoOutput(true);
OutputStream os = con.getOutputStream();
os.write(POST_PARAMS.getBytes());
os.flush();
os.close();
// For POST only - END
int responseCode = con.getResponseCode();
System.out.println("POST Response Code :: " + responseCode);
if (responseCode == HttpURLConnection.HTTP_OK) { //success
BufferedReader in = new BufferedReader(new InputStreamReader(
con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
// print result
System.out.println(response.toString());
} else {
System.out.println("POST request not worked");
}
}
I want to know how can i POSTa CSV File .
Should I do it directly in my method above ?
Do I have to create a new method by recovering the connection token from my method above?
My API ULR is POST /api/{id}/csv
I'm trying to connect to GDAX using their REST API.
I first want to do something very simple, i.e. getting historic rates.
I tried this:
private static final String GDAX_URL = "https://api.gdax.com";
public String getCandles(final String productId, final int granularity) {
HttpsURLConnection connection = null;
String path = "/products/" + productId + "/candles";
try {
//Create connection
URL url = new URL(GDAX_URL);
connection = (HttpsURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("Content-Type", "application/json");
connection.setRequestProperty("granularity", String.valueOf(granularity));
connection.setUseCaches(false);
connection.setDoOutput(true);
//Send request
DataOutputStream wr = new DataOutputStream(connection.getOutputStream());
wr.writeBytes(path);
wr.close();
//Get Response
InputStream is = connection.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(is));
StringBuffer response = new StringBuffer();
String line;
while ((line = rd.readLine()) != null) {
response.append(line);
response.append('\r');
}
rd.close();
System.out.println(response.toString());
} catch (Exception e) {
e.printStackTrace();
return null;
} finally {
if (connection != null) {
connection.disconnect();
}
}
return null;
}
But I get a 400 code in return "Bad Request – Invalid request format".
My problem is with the passing of the path "/products//candles" and the parameters (e.g. granularity).
I don't understand what should go in the request properties and in the message itself, and in what form.
I managed to make it work like this:
URL url = new URL(GDAX_URL + path + "?granularity="+granularity);
connection = (HttpsURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("Content-Type", "application/json");
Not sure how to use the DataOutputStream, so I just removed it. At least it works.
I want to make java program to download a file from website, but i'm stuck in following situation. When I run the following code logging to page gets done successfully, but after it is done, when I send request for the next page, it shows response code 500.
public class Ss2mydb
{
private List<String> cookies;
private HttpURLConnection conn;
private final String USER_AGENT = "Mozilla/5.0";
public static void main(String[] args) throws Exception
{
String indexPage = "http://10.100.100.142/index.asp";
String validatePage = "http://10.100.100.142/validate.asp";
String ccmenuPage = "http://10.100.100.142/callcentre/ccmenu.asp";
String reportPage = "http://10.100.100.142/topmgmt/reports/PROJECTVIJAY/get_download_cafs.asp";
String reportPageDownload = "http://10.100.100.142/topmgmt/reports/PROJECTVIJAY/get_download_cafs.asp?view=N";
Ss2mydb http = new Ss2mydb();
// make sure cookies is turn on
CookieHandler.setDefault(new CookieManager());
// 1. Send a "GET" request, so that you can extract the form's data.
String page = http.GetPageContent(indexPage);
String postParams = http.getFormParams(page, "username", "password");
// 2. Construct above post's content and then send a POST request for
// authentication
http.sendPost(validatePage, postParams);
// System.exit(0);
// 3. success then go to gmail.
http.GetPageContent2(ccmenuPage);
String result = http.GetPageContent2(reportPage);
System.out.println(result);
}
private void sendPost(String url, String postParams) throws Exception {
URL obj = new URL(url);
conn = (HttpURLConnection) obj.openConnection();
// Acts like a browser
conn.setUseCaches(false);
conn.setRequestMethod("POST");
conn.setRequestProperty("Host", "10.100.100.142");
conn.setRequestProperty("User-Agent", USER_AGENT);
conn.setRequestProperty("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
for (String cookie : this.cookies) {
System.out.println(" sendPost : "+cookie.split(";", 1)[0]);
conn.addRequestProperty("Cookie",cookie.split(";", 1)[0]);
}
conn.setRequestProperty("Connection", "keep-alive");
conn.setRequestProperty("Referer", "http://10.100.100.142/index.asp");
conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
conn.setRequestProperty("Content-Length", Integer.toString(postParams.length()));
conn.setDoOutput(true);
conn.setDoInput(true);
// Send post request
DataOutputStream wr = new DataOutputStream(conn.getOutputStream());
wr.writeBytes(postParams);
wr.flush();
wr.close();
int responseCode = conn.getResponseCode();
System.out.println("\nSending 'POST' request to URL : " + url);
System.out.println("Post parameters : " + postParams);
System.out.println("Response Code : " + responseCode);
// setCookies(conn.getHeaderFields().get("Set-Cookie"));
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
// System.out.println(response.toString());
}
private String GetPageContent2(String url) throws Exception {
URL obj = new URL(url);
conn = (HttpURLConnection) obj.openConnection();
// default is GET
conn.setRequestMethod("GET");
conn.setUseCaches(false);
// act like a browser
conn.setRequestProperty("User-Agent", USER_AGENT);
conn.setRequestProperty("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
System.out.println(" L : "+cookies);
if (cookies != null) {
for (String cookie : this.cookies) {
System.out.println(" GetPageContent : "+cookie.split(";", 1)[0]);
conn.addRequestProperty("Cookie", cookie.split(";", 1)[0]);
}
}
int responseCode = conn.getResponseCode();
System.out.println("\nSending 'GET' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
// Get the response cookies
// System.out.println(" Get : "+conn.getHeaderFields().get("Set-Cookie"));
// setCookies(conn.getHeaderFields().get("Set-Cookie"));
return response.toString();
}
private String GetPageContent(String url) throws Exception {
URL obj = new URL(url);
conn = (HttpURLConnection) obj.openConnection();
// default is GET
conn.setRequestMethod("GET");
conn.setUseCaches(false);
// act like a browser
conn.setRequestProperty("User-Agent", USER_AGENT);
conn.setRequestProperty("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
System.out.println(" L : "+cookies);
if (cookies != null) {
for (String cookie : this.cookies) {
System.out.println(" GetPageContent : "+cookie.split(";", 1)[0]);
conn.addRequestProperty("Cookie", cookie.split(";", 1)[0]);
}
}
int responseCode = conn.getResponseCode();
System.out.println("\nSending 'GET' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
// Get the response cookies
System.out.println(" Get : "+conn.getHeaderFields().get("Set-Cookie"));
setCookies(conn.getHeaderFields().get("Set-Cookie"));
return response.toString();
}
public String getFormParams(String html, String username, String password)
throws UnsupportedEncodingException {
System.out.println("Extracting form's data...");
Document doc = Jsoup.parse(html);
// Google form id
Element loginform = doc.getElementById("right");
Elements inputElements = loginform.getElementsByTag("input");
List<String> paramList = new ArrayList<String>();
for (Element inputElement : inputElements) {
String key = inputElement.attr("name");
String value = inputElement.attr("value");
if (key.equals("USERname"))
value = username;
else if (key.equals("password"))
value = password;
if(!key.equals(""))
paramList.add(key + "=" + URLEncoder.encode(value, "UTF-8"));
}
// build parameters list
StringBuilder result = new StringBuilder();
for (String param : paramList) {
if (result.length() == 0) {
result.append(param);
} else {
result.append("&" + param);
}
}
return result.toString();
}
public List<String> getCookies() {
return cookies;
}
public void setCookies(List<String> cookies) {
this.cookies = cookies;
}
}
500 means "Internal server error" (see HTTP codes and their meaning).
It does mean that the request you sent was understood and was valid HTTP, but either there is an internal, unrelated issue, or the request you send is missing some parameter or is sending some incorrect value, and that omission/mistake causes the server logic to crash (either because it fails a validation of the request content or in a deeper, uncontrolled way).
The only way to get more specific info would be seeing the server logs (and that if the developer was careful checking parameters and logging info).
Your best option is to analize the traffic between the web browser and the server using some tool like wireshark, and then analize the traffic between your client and the server and try to spot the differences.