When I received an HTTP request of smaller length it's fine, but when receiving long packet getting corrupted. I took a trace through wire shark and I printed packet in hex value in JAVA console. Some additional values are showing in that printing. Why?
How can I solve it?
Is there anything wrong with conversion of HTTP request to Hex.
Following code is used to convert String to Hex.
ByteArrayOutputStream baos = new ByteArrayOutputStream();
InputStream responseData = request.getInputStream();
byte[] buffer = new byte[1000];
int bytesRead = 0;
while ((bytesRead = responseData.read(buffer)) > 0) {
baos.write(buffer, 0, bytesRead);
sb=baos.toString();
str = baos.toString();
sb.append(str);
sb = new String(baos.toByteArray(),UTF8);
}
baos.close(); // connection.close();
You can't convert the read bytes to a String until all your input is read because a fraction of the input might be invalid UTF-8 encoded data.
Also don't use ByteArrayOutputStream.toString() because it uses the platform's default character set to decode bytes to characters (String) which is indeterministic. Instead use ByteArrayOutputStream.toString(String charsetName) and specify the encoding.
Also you should use ServletRequest.getCharacterEncoding() to detect encoding and revert to UTF-8 for example if it is unknown.
First read all input, and then convert it to a String:
String encoding = ServletRequest.getCharacterEncoding();
if (encoding == null)
encoding = "UTF-8";
// First read all input data
while ((bytesRead = responseData.read(buffer)) > 0) {
baos.write(buffer, 0, bytesRead);
}
// We have all input, now convert it to String:
String text = baos.toString(encoding);
Better Alternative
Since you convert the binary input to a String, you should use ServletRequest.getReader() instead of reading binary data using ServletRequest.getInputStream() and converting it to String manually.
E.g. reading all lines:
BufferedReader reader = request.getReader();
StringBuilder sb = new StringBuilder();
String line;
while ((line = reader.readLine()) != null) {
// Process line, here I just append it to a StringBuilder
sb.append(line);
// If you want to preserve newline characters, keep the next line:
sb.append('\n');
}
Related
I have been searching the web for this particular problem. Maybe i'm doing something wrong or i'm missing something here...
So i'm trying to convert a File Stream ( an Excel file ) -> mimetype ( application/octet-stream or application/vnd.ms-excel ) doesn´t matter...to a Base64 encoded string.
The reason i'm doing this is because i want to provide the File in a REST API inside a JSON object for later decoding in the browser the base64 string and download the file.
When I receivethe InputStream and save to the disk everything works fine...
Even when i use POSTMAN to get the FILE if I save the file it opens in Excel with all the right data.
THE CODE -> Used this simple example to download a file from a URL
URL url = new URL(fileURL);
HttpURLConnection httpConn = (HttpURLConnection) url.openConnection();
//etc...i get response code OK(200) get file name etc
// opens input stream from the HTTP connection
InputStream inputStream = httpConn.getInputStream();
String saveFilePath1 = "C:\\test1.xlsx";
String saveFilePath2 = "C:\\test2.xlsx";
FileOutputStream outputStream = new FileOutputStream(saveFilePath1);
int bytesRead = -1;
byte[] buffer = new byte[BUFFER_SIZE];
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
//FOR TESTING PURPOSES AT THIS POINT I HAVE SAVED THE STREAM INTO
//**test1.xlsx** SUCCESSFULLY and opens into excel and everything
//is fine.
//THE PROBLEM RESIDES HERE IN THIS NEXT PIECE OF CODE
//import org.apache.commons.codec.binary.Base64;
//I try to encode the string to Base64
String encodedBytesBase64 = Base64.encodeBase64String(buffer);
//WHEN I DO THE DECODE AND WRITE THE BYTES into test2.xlsx this file doesn´t work...
FileOutputStream fos = new FileOutputStream(saveFilePath2);
byte[] bytes = Base64.decodeBase64(encodedBytesBase64);
fos.write(bytes);
//Close streams from saved file test2
fos.close();
//Close streams from saved file test1
outputStream.close();
inputStream.close();
I even took the string to check if it is a valid Base64 String, which it is accordind to this site -> Base64 Validator
But when i try to decode the string in the same website it tells me there's a different encoding:
Is it possible this is the problem ?
I think you can ignore those warnings. Rather, the issue is here:
int bytesRead = -1;
byte[] buffer = new byte[BUFFER_SIZE];
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
:
String encodedBytesBase64 = Base64.encodeBase64String(buffer);
As you can see in the first part, you are reusing buffer to read the input stream and write to the output stream. If this loops around more than once, buffer will be overwritten with the next chunk of data from the input stream. So, when you are encoding buffer, you are only using the last chunk of the file.
The next problem is that when you are encoding, you are encoding the full buffer array, ignoring the bytesRead.
One option might be to read the inputStream and write it to a ByteArrayOutputStream, and then encode that.
int bytesRead = -1;
byte[] buffer = new byte[BUFFER_SIZE];
ByteArrayOutputStream array = new ByteArrayOutputStream();
while ((bytesRead = inputStream.read(buffer)) != -1) {
array.write(buffer, 0, bytesRead);
}
String encoded = Base64.encodeBase64String(array.toByteArray());
I have Base64 String. I am trying to decode it, then decompress it.
String textToDecode = "H4sIAAAAAAAAAAEgAN//0JTQtdGC0LDQu9C40LfQuNGA0L7QstCw0L3QvdGL0LmRCuyiIAAAAA==\n";
byte[] data = Base64.decode(textToDecode, Base64.DEFAULT);
String result = GzipUtil.decompress(data);
Code that I am using for decompression:
public static String decompress(byte[] compressed) throws IOException {
final int BUFFER_SIZE = 32;
ByteArrayInputStream is = new ByteArrayInputStream(compressed);
GZIPInputStream gis = new GZIPInputStream(is, BUFFER_SIZE);
StringBuilder string = new StringBuilder();
byte[] data = new byte[BUFFER_SIZE];
int bytesRead;
while ((bytesRead = gis.read(data)) != -1) {
string.append(new String(data, 0, bytesRead));
}
gis.close();
is.close();
return string.toString();
}
I should get this String:
Детализированный
Insteam of it, I am getting this String with question mark symbols:
Детализирован��ый
What is my mistake? And how to solve it?
One problem is that when converting from bytes to String (internally Unicode)
the encoding is not given. And for a multi-byte encoding like UTF-8 one cannot take a fixed number of bytes (like 32) and then at the end have a valid sequence.
You experienced the loss of evidently a half sequence. Hence the encoding probably is UTF-8.
final int BUFFER_SIZE = 32;
ByteArrayInputStream is = new ByteArrayInputStream(compressed);
GZIPInputStream gis = new GZIPInputStream(is, BUFFER_SIZE);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] data = new byte[BUFFER_SIZE];
int bytesRead;
while ((bytesRead = gis.read(data)) != -1) {
baos.write(data, 0, bytesRead);
}
gis.close();
return baos.toString("UTF-8"); // Or "Windows-1251" ...
The above does away with buffer boundary problems, and specifies the encoding, so the same code runs on different computers.
And mind:
new String(bytes, encoding)
string.getBytes(encoding)
It is possible that the problem is here:
string.append(new String(data, 0, bytesRead))
You are using the default character encoding to decode bytes into a Java String. If the (current) default encoding is different to the encoding used when encoding the original characters to bytes (prior to compression, etc), then you could get bytes that don't decode correctly. The decoder will then replace them with the decoder's replacement character; i.e. '\uFFFD' by default.
If this is the problem, then the solution is to find out what the correct character encoding is and use String(byte[], int, int, Charset) to create the String.
If you work only with streams you can avoid encoding problems, this few line of code should do the job well
public static String decompress(byte[] compressed) throws IOException {
try (ByteArrayOutputStream bos = new ByteArrayOutputStream()) {
try (GZIPInputStream gis = new GZIPInputStream(
new ByteArrayInputStream(compressed))) {
org.apache.commons.compress.utils.IOUtils.copy(gis, bos);
}
return bos.toString();
}
}
There is client and server components, the client is sending the data in more secure way by converting the data in blob using POST method to the server.
Can any suggest me how to convert that blob data to string object in server side(Java).i have tried some code below
Way 1):
==============================
String streamLength = request.getHeader("Content-Length");
int streamIntLength = Integer.parseInt(streamLength);
byte[] bytes = new byte[streamIntLength];
request.getInputStream().read(bytes, 0, bytes.length);
String content = DatatypeConverter.printBase64Binary(bytes);
System.out.println(content);
Output for above code is : some junk data is displaying.
dABlAG0AcABsAGEAdABlAD0AMgAzADUAUgBfAFAAcgBvAHYAaQBkAGUAcgBfA
Way 2) :
======
BufferedReader reader = new BufferedReader(new InputStreamReader(
request.getInputStream()));
StringBuilder sb = new StringBuilder();
for (String line; (line = reader.readLine()) != null;) {
String str = new String(line.getBytes());
System.out.println(str);
}
Please suggest me any one, above both ways are not worked out.
Below code works for me.
StringBuilder stringBuilder = new StringBuilder();
BufferedReader bufferedReader = null;
try {
String streamLength = request.getHeader("Content-Length");
int streamIntLength = Integer.parseInt(streamLength);
InputStream inputStream = request.getInputStream();
if (inputStream != null) {
bufferedReader = new BufferedReader(new InputStreamReader(
inputStream));
char[] charBuffer = new char[streamIntLength];
int bytesRead = -1;
while ((bytesRead = bufferedReader.read(charBuffer)) > 0) {
stringBuilder.append(charBuffer, 0, bytesRead);
}
} else {
stringBuilder.append("");
}
} catch (IOException ex) {
throw ex;
}
String body = stringBuilder.toString();
//System.out.println(body);
byte[] bytes = body.getBytes();
System.out.println(StringUtils.newStringUtf16Le(bytes));
From the first approach, it looks like the data is encoded (possibly in Base64 format). After decoding it, what is the problem you are facing ? If the data is String and then encoded to Base64, you should get the actual string after decoding it. (Assuming platform locales on client and server side are same).
If its a binary data, better you keep it inside a byte stream only. If you anyhow want it to convert to a string, then the first approach looks okay.
If this binary data represents some kind of file, you can get the related information using the HTTP headers and write it to temp location for further use.
I received some data from server and read them from java code :
is = new BufferedInputStream(connection.getInputStream());
reader = new BufferedReader(new InputStreamReader(is, "UTF-8"));
int length;
char[] buffer = new char[4096];
StringBuilder sb = new StringBuilder();
while ((length = reader.read(buffer)) != -1) {
sb.append(new String(buffer, 0, length));//buffer is already incorrect
}
byte[] byteDatas = sb.toString().getBytes();
And I print byteDatas as Hex string:
Comparing to the wireshark's result:
Some bytes are decoded as bf bd ef , I know it's \ufffd(65533) stand for invalid data.
So I think there must have decode error in my code , after debug, I found that If I use connection.getInputStream() to read data directly , there is no invalid data.
So ,the problem must happens in BufferedReader or InputStreamReader, But I have already add "UTF-8" and the data in wireshark seems not very wired. Does UTF-8 is not correctly? Server do not reply the charset.
Please help how to let BufferedReader read the correct data.
UPDATE
My default charset is "UTF-8" and have debug to prove it . After read return , I have already got the wrong data , so it's not String's fault.
String.getBytes() will use the platform's default encoding (not necessarily UTF-8) to convert the characters of the String to bytes.
Quoting from the javadoc of String.getBytes():
Encodes this String into a sequence of bytes using the platform's default charset...
You can't compare the UTF-8 encoded input data to the result which might not be the result of UTF-8 encoded. Instead explicitly specify the encoding like this:
byte[] byteDatas = sb.toString().getBytes(StandardCharsets.UTF_8);
Note:
If your input data is NOT UTF-8 encoded text and if you attempt to decode it as UTF-8, the decoder may replace invalid byte sequences. This will cause that the bytes you get by encoding the String will not be the same as the input raw bytes.
i have to send a short string as text from client to server and then after that send a binary file.
how would I send both binary file and the string using the same socket connection?
the server is a java desktop application and the client is an Android tablet. i have already set it up to send text messages between the client and server in both directions. i have not yet done the binary file sending part.
one idea is to set up two separate servers running at the same time. I think this is possible if i use two different port numbers and set up the servers on two different threads in the application. and i would have to set up two concurrent clients running on two services in the Android app.
the other idea is to somehow use an if else statement to determine which of the two types of files is being sent, either text of binary, and use the appropriate method to receive the file for the file type being sent.
example code for sending text
PrintWriter out;
BufferedReader in;
out = new PrintWriter(new BufferedWriter
(new OutputStreamWriter(Socket.getOutputStream())) true,);
in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
out.println("test out");
String message = in.readLine();
example code for sending binary file
BufferedOutputStream out;
BufferedInputStream in;
byte[] buffer = new byte[];
int length = 0;
out = new BufferedOutputStream(new FileOutputStream("test.pdf));
in = new BufferedInputStream(new FileOutputStream("replacement.pdf"));
while((length = in.read(buffer)) > 0 ){
out.write(buffer, 0, length);
}
I don't think using two threads would be necessary in your case. Simply use the socket's InputStream and OutputStream in order to send binary data after you have sent your text messages.
Server Code
OutputStream stream = socket.getOutputStream();
PrintWriter out = new PrintWriter(
new BufferedWriter(
new OutputStreamWriter(stream)
)
);
out.println("test output");
out.flush(); // ensure that the string is not buffered by the BufferedWriter
byte[] data = getBinaryDataSomehow();
stream.write(data);
Client Code
InputStream stream = socket.getInputStream();
String message = readLineFrom(stream);
int dataSize = getSizeOfBinaryDataSomehow();
int totalBytesRead = 0;
byte[] data = new byte[dataSize];
while (totalBytesRead < dataSize) {
int bytesRemaining = dataSize - totalBytesRead;
int bytesRead = stream.read(data, totalBytesRead, bytesRemaining);
if (bytesRead == -1) {
return; // socket has been closed
}
totalBytesRead += bytesRead;
}
In order to determine the correct dataSize on the client side you have to transmit the size of the binary block somehow. You could send it as a String right before out.flush() in the Server Code or make it part of your binary data. In the latter case the first four or eight bytes could hold the actual length of the binary data in bytes.
Hope this helps.
Edit
As #EJP correctly pointed out, using a BufferedReader on the client side will probably result in corrupted or missing binary data because the BufferedReader "steals" some bytes from the binary data to fill its buffer. Instead you should read the string data yourself and either look for a delimiter or have the length of the string data transmitted by some other means.
/* Reads all bytes from the specified stream until it finds a line feed character (\n).
* For simplicity's sake I'm reading one character at a time.
* It might be better to use a PushbackInputStream, read more bytes at
* once, and push the surplus bytes back into the stream...
*/
private static String readLineFrom(InputStream stream) throws IOException {
InputStreamReader reader = new InputStreamReader(stream);
StringBuffer buffer = new StringBuffer();
for (int character = reader.read(); character != -1; character = reader.read()) {
if (character == '\n')
break;
buffer.append((char)character);
}
return buffer.toString();
}
You can read about how HTTP protocol works which essentially sends 'ascii and human readable' headers (so to speak) and after that any content can be added with appropriate encoding like base64 for example. You may create sth similar yourself.
You need to first send the String, then the size of the byte array then the byte array, use String.startsWith() method to check what is being send.