I'm trying to use the open data sets that data.LACity.org publishes using Socrata software.
They have a Java API for it, but first I tried to just build and send a URL, as
a variant on the 'Sunshine' app several people have learned from on Udacity.
Now I'm actually building a URL, and then sending it out, but then I get a FileNotFoundException, as follows:
java.io.FileNotFoundException: http://data.lacity.org/resource/yv23-pmwf.json?$select=zip_code, issue_date, address_start, address_end, street_name, street_suffix, work_description, valuation&$where=issue_date >= '2015-02-27T00:00:00' AND zip_code = 90291
Here's the pisser: That whole URL is, as a final attempt, hardwired as a complete string, not built from pieces. The URL works if I plug it into Chrome, but not from my app.
But from my app, the old URL string that the Sunshine sample app builds, plugged in from logcat from a Sunshine run, to replace the URL on the lacity URL, well, that call works, and returns the JSON data.
So I'm doing something wrong when I call the LACity URL for Socrata data from my Android app. I've tried this both as https and http, and both failed. But the same code works when I call the weathermap data from the sample app.
Here are the two URLs:
http://api.openweathermap.org/data/2.5/forecast/daily?q=94043&mode=json&units=metric&cnt=7 <<< this works, both in Chrome and from Android
https://data.lacity.org/resource/yv23-pmwf.json?$select=zip_code, issue_date, address_start, address_end, street_name, street_suffix, work_description, valuation&$where=issue_date >= '2015-02-27T00:00:00' AND zip_code = 90291
This works in Chrome but not from Android.
Any suggestions would be appreciated. I'm going to try again to make heads or tails of the Socrata Soda2 Java API (and why, in this case, it might be necessary.)
Thanks
-k-
The immediate code fragment (pardon my newness to Android/Java):
final String PERMIT_BASE_URL = "one of the url strings above";
Uri builtUri = Uri.parse(PERMIT_BASE_URL).buildUpon()
.build();
URL url = new URL(builtUri.toString());
Log.v(LOG_TAG, "Build URL: " + url.toString());
urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.setRequestMethod("GET");
urlConnection.connect();
InputStream inputStream = urlConnection.getInputStream();
StringBuffer buffer = new StringBuffer();
if (inputStream == null) {
return null;
}
reader = new BufferedReader(new InputStreamReader(inputStream));
String line;
while ((line = reader.readLine()) != null) {
//simplify debugging
buffer.append(line + "\n");
}
if (buffer.length() == 0) {
return null;
}
permitJsonStr = buffer.toString();
Log.v(LOG_TAG, "Permit JSON string: " + permitJsonStr);
} catch (IOException e) {
Log.e(LOG_TAG, "Error on Xoom", e);
// Nothing to parse.
return null;
} finally{
if (urlConnection != null) {
urlConnection.disconnect();
}
if (reader != null) {
try {
reader.close();
} catch (final IOException e) {
Log.e(LOG_TAG, "Error closing stream on Xoom", e);
}
}
Figured this out from the way this page highlighted the URLs in my question.
Spaces.
The call out of Android seems to cough because of the spaces in the URL string.
I closed them all up, but then the 'AND' caused issues.
Replaced it with &, now it works, hardwired.
I'll work on constructing it from input parameters, as intended, but I think this is OK.
As Emily Litella would say...
Related
I have scanned a java web project with the Checkmarx tool, and the analysis marks an XSS vulnerability in a method where a web service is executed that responds a JSON, the vulnerability is in the line while((output = Encode.forJava(br.readLine())) != null) {, specifically in br.readLine().
Checkmarx says:
The attacker would be able to alter the returned web page by simply
providing modified data in the user input readLine, which is read by
the NetClientPost method. This input then flows through the code
straight to the output web page, without sanitization.
This can enable a Reflected Cross-Site Scripting (XSS) attack.
I tried with OWASP for Java, implementing the method Encode.forJava(), but the vulnerability continues to appear in the analysis. This is the implementation of the method:
public String NetClientPost (String urlSer, String param){
String result ="";
try {
InetAddress ip = InetAddress.getLocalHost();
String host = ip.getHostAddress();
doTrustToCertificates();
URL url = new URL(urlSer);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setDoOutput(true);
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Type", "application/json");
conn.setConnectTimeout(2000);
String input = param;
String output = "";
try(OutputStream os = conn.getOutputStream()) {
os.write(input.getBytes());
os.flush();
if (conn.getResponseCode() != 200) {
throw new RuntimeException("Failed : HTTP code : " + conn.getResponseCode());
}
try (BufferedReader br = new BufferedReader(new InputStreamReader((conn.getInputStream())))) {
while ((output = Encode.forJava(br.readLine())) != null) {//LINE OF CHECKMARX XSS
result += output;
}
}
}
conn.disconnect();
return result;
} catch (MalformedURLException e) {
return result;
} catch (IOException e) {
return result;
} catch (Exception e) {
return result;
}
}
Any have an idea of how to solve this?
Try parsing the incoming data as JSON and then serializing it back to a string before sending it on.
That way you can be sure that your method only returns JSON to the client. If for some reason, your incoming data isn't JSON, then your method would encounter an error parsing the JSON, which you can then handle appropriately.
Encode.forJava isn't a helpful method to use here: it is used to encode a string to be inserted into a Java string literal.
output = Encode.forHtmlAttribute(br.readLine() works for me
I use the following code
private String resultGET(String addr)
{
try
{
String result = "";
HttpURLConnection conn = null;
addr = (isFull)?addr:Statics.fullURL(addr);
try
{
URL url = new URL(addr);
conn = (HttpURLConnection)url.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("User-Agent", Statics.USER_AGENT);
InputStream ips = conn.getInputStream();
int responseCode = conn.getResponseCode();
if (200 != responseCode)
{
Feedback.setError("GET error: " + responseCode + " on " + addr);
return "";
}
BufferedReader bufr = new BufferedReader(new InputStreamReader(ips));
String line;
while ((line = bufr.readLine()) != null) result += line;
bufr.close();
} finally{if (null != conn) conn.disconnect();}
return result;
} catch(Exception e)
{
Feedback.setError("get fault " + Utils.stackTrace(e));
return "";
}
}
Feedback is simply a Java class I use internally to handle all messages that I send back to the Android app front end (this is a hybrid app and the code above is part of a plugin I have written for the app).
I find that when any significant amount of data are returned the resultGETcall gets excruciatingly slow. For instance, a 43Kb Javascript file - which I later use to run JS code via Duktape takes the best part of 1 minute to download and save.
I am still quite a newbie when it comes to Java so I imagine that I am doing something wrong here which is causing the issue. I'd be most obliged to anyone who might be able to put me on the right track.
A while later...
I have now tested the issue on an Android 6 device instead of my default Android 4.4.2 device. On the Android 6 the download + file save speed comes in at a decent 5 seconds. On Android 4.4.2 it is over 40s. Are there any known issues with HTTPURLConnection on earlier versions of Android? I
String result = "";
The += operator on a String is slow. If you have a lot of lines use a StringBuilder sb = new StringBuilder(); and use its append() method to sb.append(line + " \n");
At the end you can use result = sb.toString();
I am beginner in Java and Android Studio. I have written a code by Android Studio and Wamp as server and Genymotion as simulator. all codes work fine and I can interact with mysql by use of my .php files
Then I decide to transfer codes to real server.
but I get this error:
java.io.FileNotFoundException: http://burj.1shahrvand.com/Burj/BikerLogin.php
The File is available check it here but I get Exception that file not found
The code is like this:
String uri = rp.getUri();
if(rp.getMethod().equals("GET")){
uri += "?" + rp.getEncodedParams();
}
HttpURLConnection connection;
try {
URL url = new URL(uri);
connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod(rp.getMethod());
if (rp.getMethod().equals("POST")){
connection.setDoOutput(true);
connection.setDoInput(true);
OutputStreamWriter writer = new OutputStreamWriter(connection.getOutputStream());
writer.write(rp.getEncodedParams());
writer.flush();
}
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String line;
StringBuilder sb = new StringBuilder();
while ((line = reader.readLine()) != null) {
sb.append(line);
}
return sb.toString();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
Log.i("HESAM Original", e.getMessage());
e.printStackTrace();
}
return null;
I appreciate your Help!
You will get a FileNotFoundException if you call getInputStream after the server has responded withe a 404 or 410 status code.
If you want to avoid the exception, check that the response status code is a 2xx code. If it isn't then use getErrorStream instead of getInputStream.
In my case, There is an option in my Cpanel. It is MOD Security, Just turn it off, after 15 minutes my app worked properly.
Simply you need to add the port name(:8080) with the localhost ip address in URL String, like i did:
String login_url = "http://192.168.0.136:8080/login.php";
My goal is to get the xml from an API. The API uri I use, including parameters is http://webservices.ns.nl/ns-api-treinplanner?fromStation=Roosendaal&toStation=Eindhoven. I am given a username and password, for what I think probably is basic authorization.
I tried various things like something with an Authenticator, the format http://username:password#webservices.ns.nl/ns-api-treinplanner, but at the end of a lot of SO searching I ended up with something with a setRequestProperty with the basic authorization.
I put the code into an AsyncTask which seems to work correctly so I will just put the code from inside doInBackground in here.
As the java FileNotFoundException I first got didn't give me much information, I found out how to use the getErrorStream to find out more.
InputStream in;
int resCode;
try {
URL url = new URL("http://webservices.ns.nl/ns-api-treinplanner?fromStation=Roosendaal&toStation=Eindhoven");
String userCredentials = "username:password";
String encoding = new String(android.util.Base64.encode(userCredentials.getBytes(), Base64.DEFAULT));
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.setRequestProperty("Authorization", "Basic " + encoding);
urlConnection.setRequestMethod("GET");
urlConnection.setDoOutput(true);
try {
resCode = urlConnection.getResponseCode();
if (resCode == HttpURLConnection.HTTP_OK) {
Log.i("rescode","ok");
in = urlConnection.getInputStream();
} else {
Log.i("rescode","not ok");
in = urlConnection.getErrorStream();
}
BufferedReader bufferedReader = new BufferedReader(
new InputStreamReader(in));
StringBuilder stringBuilder = new StringBuilder();
String line;
while ((line = bufferedReader.readLine()) != null) {
stringBuilder.append(line).append("\n");
}
bufferedReader.close();
return stringBuilder.toString();
}
finally{
urlConnection.disconnect();
}
}
catch(Exception e) {
Log.e("ERROR", e.getMessage(), e);
return null;
}
Then, in onPostExecute I print the response, but the response I get is
<?xml version="1.0"?>
<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope" soap:encodingStyle="http://www.w3.org/2003/05/soap-encoding">
<soap:Header></soap:Header>
<soap:Body><soap:Fault>
<faultcode>soap:Server</faultcode>
<faultstring>006:No customer found for the specified username and password</faultstring></soap:Fault>
</soap:Body></soap:Envelope>
This is of course not right, it should give a full xml of in this case a train voyage recommendation.
I tested with my browsers, and also using a HTTP request tool called Postman which returned the correct xml so all the uri's, parameters, username and password are correct.
The encoding used is wrong. The base64 encoding used randomly returns whitespaces in the middle, adding encoding = encoding.replaceAll("\\s+",""); actually fixed it.
In my Wikipedia reader app for Android, I'm downloading an article's html by using HttpURLConnection, some users report that they are unable to see articles, instead they see some css, so it seems like their carrier is somehow preprocessing the html before it's downloaded, while other wikipedia readers seem to work fine.
Example url: http://en.m.wikipedia.org/wiki/Black_Moon_(album)
My method:
public static String downloadString(String url) throws Exception
{
StringBuilder downloadedHtml = new StringBuilder();
HttpURLConnection urlConnection = null;
String line = null;
BufferedReader rd = null;
try
{
URL targetUrl = new URL(url);
urlConnection = (HttpURLConnection) targetUrl.openConnection();
if (url.toLowerCase().contains("/special"))
urlConnection.setInstanceFollowRedirects(true);
else
urlConnection.setInstanceFollowRedirects(false);
//read the result from the server
rd = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
while ((line = rd.readLine()) != null)
downloadedHtml.append(line + '\n');
}
catch (Exception e)
{
AppLog.e("An exception occurred while downloading data.\r\n: " + e);
e.printStackTrace();
}
finally
{
if (urlConnection != null)
{
AppLog.i("Disconnecting the http connection");
urlConnection.disconnect();
}
if (rd != null)
rd.close();
}
return downloadedHtml.toString();
}
I'm unable to reproduce this problem, but there must be a way to get around that? I even disabled redirects by setting setInstanceFollowRedirects to 'false' but it didn't help.
Am I missing something?
Example of what the users are reporting:
http://pastebin.com/1E3Hn2yX
carrier is somehow preprocessing the html before it's downloaded
a way to get around that?
Use HTTPS to prevent carriers from rewriting pages. (no citation)
Am I missing something?
not that I can see