Can someone helps me to get json from the web.In the end of function jsonResponse is empty. I use this method to do it:
private String getJson() {
jsonResponsce = "";
AsyncTask.execute(new Runnable() {
#Override
public void run() {
try{
URL httpbinEndpoint = new URL(webPage);
HttpsURLConnection myConnection = (HttpsURLConnection) httpbinEndpoint.openConnection();
myConnection.setRequestMethod("GET");
// Enable writing
myConnection.setDoOutput(true);
String internetData = "";
// Write the data
myConnection.getOutputStream().write(internetData.getBytes());
jsonResponsce = internetData;
} catch (IOException e) {
e.printStackTrace();
}
}
});
return jsonResponsce;
}
I set an Internet permission to the manifest. I try go get Json from the next address: https://shori-dodjo-mobile-app.firebaseio.com/.json. Full code is placed here: https://github.com/GenkoKaradimov/Shori-Dodjo-Android-App/
You are executing the request asynchronously so the method starts the execution and then completes and therefore there is no result. The result will be there in a second but by that time the method getJson has already completed. You most probably need to put the code that uses the json at the end of the run method.
In addition your code for reading from the stream seems wrong. It should probably be something like
BufferedReader br = new BufferedReader(new InputStreamReader(myConnection.getInputStream()));
jsonResponsce = br.lines().collect(Collectors.joining("\n"));
(I haven't tested this)
There are multiple issues in your code:
First, AsyncTask means it's async(hronous), so you can't return the result right away. Instead, override AsyncTask's onPostExecute and do what you need to do with the data there. Here is the sample implementation.
Second, you're using getOutputStream, which is intended for writing to the connection, i.e. sending data to the server. In your case you need to getInputStream and read from it. Easiest way is to wrap it in a BufferedReader and read until it returns -1 (marking end of stream), and then convert to string.
There are a few quirks: You should handle, or at least recognize errors by checking HTTP status code, handle encodings (the convert-bytes-to-string part), and handle cases when response is compressed, e.g. using DEFLATE or gzip. I've implemented that in a pure Java way (reference code, warning: outdated docs), but I'd seriously recommend using one of the established libraries such as Retrofit or Volley.
Json objects usually get return as HashMaps.
So you might need something like, HashMap yourMap = new HashMap<~>();
then
yourMap.get("the objects name on the other side", the var its getting saved too.);
right now it looks like all you are trying to do is save the byte data, but this byte data needs to have a type. Hope this helps.
Related
I am trying to download web page with all its resources . First i download the html, but when to be sure to keep file formatted and use this function below .
there is and issue , i found 10 in the final file and when i found that hexadecimal code of the LF or line escape . and this makes troubles to my javascript functions .
Example of the final result :
<!DOCTYPE html>10<html lang="fr">10 <head>10 <meta http-equiv="content-type" content="text/html; charset=UTF-8" />10
Can someone help me to found the real issue ?
public static String scanfile(File file) {
StringBuilder sb = new StringBuilder();
try {
BufferedReader bufferedReader = new BufferedReader(new FileReader(file));
while (true) {
String readLine = bufferedReader.readLine();
if (readLine != null) {
sb.append(readLine);
sb.append(System.lineSeparator());
Log.i(TAG,sb.toString());
} else {
bufferedReader.close();
return sb.toString();
}
}
} catch (IOException e) {
e.printStackTrace();
return null;
}
}
There are multiple problems with your code.
Charset error
BufferedReader bufferedReader = new BufferedReader(new FileReader(file));
This isn't going to work in tricky ways.
Files (and, for that matter, data given to you by webservers) comes in bytes. A stream of numbers, each number being between 0 and 255.
So, if you are a webserver and you want to send the character ö, what byte(s) do you send?
The answer is complicated. The mapping that explains how some character is rendered in byte(s)-form is called a character set encoding (shortened to 'charset').
Anytime bytes are turned into characters or vice versa, there is always a charset involved. Always.
So, you're reading a file (that'd be bytes), and turning it into a Reader (which is chars). Thus, charset is involved.
Which charset? The API of new FileReader(path) explains which one: "The system default". You do not want that.
Thus, this code is broken. You want one of two things:
Option 1 - write the data as is
When doing the job of querying the webserver for the data and relaying this information onto disk, you'd want to just store the bytes (after all, webserver gives bytes, and disks store bytes, that's easy), but the webserver also sends the encoding, in a header, and you need to save this separately. Because to read that 'sack of bytes', you need to know the charset to turn it into characters.
How would you do this? Well, up to you. You could for example decree that the data file starts with the name of a charset encoding (as sent via that header), then a 0 byte, and then the data, unmodified. I think you should go with option 2, however
Option 2
Another, better option for text-based documents (which HTML is), is this: When reading the data, convert it to characters, using the encoding as that header tells you. Then, to save it to disk, turn the chars back to bytes, using UTF-8, which is a great encoding and an industry standard. That way, when reading, you just know it's UTF-8, period.
To read a UTF-8 text file, you do:
Files.newBufferedReader(Paths.get(file));
The reason this works, is that the Files API, unlike most other APIs (and unlike FileReader, which you should never ever use), defaults to UTF_8 and not to platform-default. If you want, you can make it more readable:
Files.newBufferedReader(Paths.get(file), StandardCharsets.UTF_8);
same thing - but now in the code it is clear what's happening.
Broken exception handling
} catch (IOException e) {
e.printStackTrace();
return null;
}
This is not okay - if you catch an exception, either [A] throw something else, or [B] handle the problem. And 'log it and keep going' is definitely not 'handling' it. Your strategy of exception handling results in 1 error resulting in a thousand things going wrong with a thousand stack traces, and all of them except the first are undesired and irrelevant, hence why this is horrible code and you should never write it this way.
The easy solution is to just put throws IOException on your scanFile method. The method inherently interacts with files, it SHOULD be throwing that. Note that your psv main(String[] args) method can, and usually should, be declared to throws Exception.
It also makes your code simpler and shorter, yay!
Resource Management failure
a filereader is a resource. You MUST close it, no matter what happens. You are not doing that: If .readLine() throws an exception, then your code will jump to the catch handler and bufferedReader.close is never executed.
The solution is to use the ARM (Automatic Resource Management) construct:
try (var br = Files.newBufferedReader(Paths.get(file), StandardCharsets.UTF_8)) {
// code goes here
}
This construct ensures that close() is invoked, regardless of how the 'code goes here' block exits. Even if it 'exits' via an exception or a return statement.
The problem
Your 'read a file and print it' code is other than the above three items mostly fine. The problem is that the HTML file on disk is corrupted; the error lies in your code that reads the data from the web server and saves it to disk. You did not paste that code.
Specifically, System.lineSeparator() returns the actual string. Thus, assuming the code you pasted really is the code you are running, if you are seeing an actual '10' show up, then that means the HTML file on disk has that in there. It's not the read code.
Closing thoughts
More generally the job of 'just print a file on disk with a known encoding' can be done in far fewer lines of code:
public static String scanFile(String path) throws IOException {
return Files.readString(Paths.get(path));
}
You should just use the above code instead. It's simple, short, doesn't have any bugs, cannot leak resources, has proper exception handling, and will use UTF-8.
Actually, there is no problem in this function I was mistakenly adding 10 using another function in my code .
I'm trying to create a web GUI for a Minecraft game server I run. The data I'm trying to read is from CoreProtect, a logging plugin.
I'm mainly using PHP and then trying to write a small Java service that can convert the serialized data into a JSON string that I can then use - since I can't deserialize a Java object directly in PHP and it's only some meta data that's stored as a Java serialized object, the rest is normal non-blob columns.
I've identified that the CoreProtect plugin is using ObjectOutputStream to serialize the object and then writes it to a MySQL BLOB field, this is the code I've identified from CoreProtect that's handling this:
try {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(bos);
oos.writeObject(data);
oos.flush();
oos.close();
bos.close();
byte[] byte_data = bos.toByteArray();
preparedStmt.setInt(1, time);
preparedStmt.setObject(2, byte_data);
preparedStmt.executeUpdate();
preparedStmt.clearParameters();
} catch (Exception e) {
e.printStackTrace();
}
This is then outputting the bytes to the database. All of the rows in the database start with the same few characters (from what I've seen this should be Java's 'magic' header). However, when trying to use the below code to unserialize the data I receive an error stating that the header is corrupt 'invalid stream header: C2ACC3AD'
byte[] serializedData = ctx.bodyAsBytes();
ByteArrayInputStream bais = new ByteArrayInputStream(serializedData);
try {
ObjectInputStream ois = new ObjectInputStream(bais);
Object object = ois.readObject();
Gson gson = new Gson();
ctx.result(gson.toJson(object));
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
I'm using Javalin as the web service and am just sending the raw output from the BLOB column to a post route, I'm then reading this with the bodyAsBytes method. I'm tried testing this by writing the BLOB column to a file and then copying the contents of the file into a test POST request using Postman. I've also tried using a PHP script to read directly from the DB and then send that as a POST request and I just get the same error.
I've looked into this and everything is pointing to corrupt data. However, the odd thing is when triggering a 'restore' via the CoreProtect plugin it correctly restores what it needs to, reading all of the relevant data from the database including this column. From what I've seen in CoreProtect's JAR it's just doing the same process with the InputStream method calls.
I'm not very familiar with Java and thought this would be a fairly simple process. Am I missing something here? I don't see anything in the CoreProtect plugin that may be overriding the stream header. It's unfortunately not open source so I'm having to use a Java decompiler to try and see how it's serializing the object so that I can then try and read it, I assume it's possible the decompiler is not reading how this is serialized/unserialized.
My other thought was maybe the 'magic' header changed between Java versions? Although I couldn't seem to confirm this online. The specific header I'm receiving I've also seen in some other similar posts, although those all lead to data corruption/using a different output stream.
I appreciate any help with this, it's not an essential feature but would be nice if I can read all of the data related to the logs generated by the server/plugin.
I understand the use case is niche, but hoping the issue/resolution is a general Java problem :).
Update ctx is an instance of Javelin's Context class. Since I'm trying to send a raw POST request to Java I needed some kind of web service and Javelin looked easy/lightweight for what I needed. On the PHP side I'm just reading the column from the database and then using Guzzle to send a raw body with the result to the Javelin service that's running.
Something, apparently ctx, is treating the binary data as a String. bodyAsBytes() is converting that String to bytes using the String’s getBytes() method, which immediately corrupts the data.
The first two bytes of a serialized Java object stream are always AC ED. getAsBytes() is treating these as characters, namely U+00AC NOT SIGN and U+00ED LATIN SMALL LETTER I WITH ACUTE. When encoded in UTF-8, these two characters produce the bytes C2 AC C3 AD, which is what you’re seeing.
Solution: Do not treat your data as a String under any circumstances. Binary data should only be treated as a byte stream. (I’m well aware that it was common to store bytes in a string in C, but Java is not C, and String and byte[] are distinct types.)
If you update your question and state what the type of ctx is, we may be able to suggest an exact course of action.
The question can seem simple, but I didn't find a good answer yet. I need to send a JSon structure (build with an unspecified libretry I'm currently developing) from a Servlet to a remote page.
I'm interested in the best way to send the structure.
I mean, in my Servlet, inside the doPost() event, how should I manage the send?
I was thinking about 2 scenarios:
try (PrintWriter out = response.getWriter()) {
out.print(myJSon.toString(); // <- recursive function that overrides
// toString() and returns the entire JSon
// structure
} (...)
or
try (OutputStream os = response.getOutputStream()) {
myJSon.write(os, StandardCharsets.UTF8); // <- function that
// recursively writes chunk of my JSon structure
// in a BufferWriter created inside the root write function
// forcing UTF-8 encoding
} (...)
Or something different, if there's a better approch.
Note that the JSon structure contains an array of objects with long text fields (descriptions with more than 1000 characterd), so it can be quite memory consuming.
For why I'm not using standard JSon libreries, it's because I don't know them and I don't know if I can trust them yet. And also I don't know if I will be able to install them on the production server.
Thanks for your answers.
From your question i see multiple points to adress:
How to send your JSon
What JSon library can you use
How to use the library in production
How to send your JSon
From your code this seems to be an HTTP response rather than a POST on your Servlet so you need to know how to send a JSON string as an HTTP response's body
Do you use a framework for your web server or are you handling everything manually ? If you use a framework it usually does it for you, just pass the JSON String
If your doing it manually:
try (PrintWriter pw = response.getWriter()) {
pw.write(myJson.toString());
}
or
try (OutputStream os = response.getOutputStream()) {
os.write(myJson.toString().getBytes());
}
Both are valid, see Writer or OutputStream?
Your JSON's size shouldn't matter given what your saying, it's just text so it won't be big enough to matter.
What libraries can you use
There are a lot of JSON libraries for Java, mainly:
Jackson
GSon
json-io
Genson
Go for the one you prefer, there will be extensive documentation and resources all over google
How to use in production
If you are not sure you are able to install dependencies on the production server, you can always create an uber-jar (See #Premraj' answer)
Basically, you bundle the dependency in your Jar
Using Gson is good way to send json
Gson gson = new Gson();
String jsonData = gson.toJson(student);
PrintWriter out = response.getWriter();
try {
out.println(jsonData);
} finally {
out.close();
}
for detail json response from servlet in java
I have a Spring REST service set up which has a POST method to receive a byte array. The goal of this method is to accept a zipped JSON string, hence the byte array. The unzipped string is to be mapped to a list and then sent to a database.
#RequestMapping(value = "/receiveData.do", method = RequestMethod.POST)
public
#ResponseBody boolean receiveData(#RequestParam byte[] data,
#RequestParam String configurationName) {
String jsonString = ZipController.unZip(data);
JSONMapper jsonMapper = new JSONMapper();
List<Message> messages = null;
try {
messages = jsonMapper.parseToMessage(jsonString, configurationName);
saveData(messages);
} catch (Exception e) {
return false;
}
return true;
}
The zipping and the unzipping of the String works without flaws, as tests that I have written show. The converting to a list and then storing it in a database works perfectly too. However, when I zip a string, and send it to the address, somehow it isn't recognized as a zip anymore. Further digging revealed that the actual byte[] has completely changed in the process. Before I send it, the array prints to
[B#7186333e
However, after sending it, it suddenly prints to (for example)
[B#fee07cc
This appears to be random, but the [B# is always the same. The rest is random. I am using the REST-assured framework to test, but SOAPui gives the same problem. The code I'm using to send the call is:
expect().
body(containsString("true")).
when().
given().
parameters("data",ZipController.zip(correctJSONMessage)).
post("/receiveData.do");
The method returns a true or false as feedback whether it succeded or not.
The whole service runs on a Tomcat 6 server, is it possible that this messes with the array? How can I fix this?
This question already has answers here:
JSON URL from StackExchange API returning jibberish?
(3 answers)
Closed 9 years ago.
I am trying to grab the data from a json file through java. If I navigate to the URL using my browser, everything displays fine, but if I try to get the data using java I get get a bunch of characters that cannot be interpreted or parsed. Note that this code works with other JSON Files. Could this be a server side thing with the way the JSON file is created? I tried messing around with different character sets and that did not seem to fix the problem.
public static void main(String[] args) throws Exception {
URL url = new URL("http://www.minecraftpvp.com/api/ping.json");
URLConnection connection = url.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
boolean hasLine = true;
while (hasLine) {
String line = in.readLine();
if (line != null) {
System.out.println(line);
} else {
hasLine = false;
}
}
}
The output I get from this is just a ton of strange characters that make no sense at all. Where if I change the url to something like google.com, it works fine.
EDIT: JSON URL from StackExchange API returning jibberish? Seemed to have answered my question. I tried searching before I asked to make sure the answer wasn't here and couldn't find anything. Guess I didn't look hard enough.
Yes that URL is returning gzip encoded content by default.
You can do one of three things:
Explicitly set the Accept-Encoding: header in your request. A web service should not return gzip compression unless it is listed as an accepted encoding in the request, so this website is not being very friendly. Your browser is setting it as accepted I suspect, that is why you can see it there. Just set it to an empty value and it should as per the spec return non-encoded responses, your mileage may vary on this one.
Or use the answer in this How to handle non-UTF8 html page in Java? that shows how to decompress the response. This should be the preferred option over #1.
And/or Ask the person hosting the service to implement the recommended scheme which is to only provide compressed responses if the client says it can handle them or if it can infer it from the browser fingerprint with high confidence.
Best of luck C.
You need to inspect the Content-Encoding header. The URL in question improperly returns gzip-compressed content even when you don't ask for it, and you'll need to run it through a decoder.