We Keep Coding

HTTP manual Client not writing to disk properly JAVA

I am trying to build a manual HTTP client (using sockets) along with a cache and I cant seem to figure out why the files are not saving to disk properly. It works pretty good for HTML files, but cant seem to work for other files types that re not text based like .gif. Could anyone tell me why? I am quite new to HTTP protocol and Socket programming in general.
The loop to grab the response.
InputStream inputStream = socket.getInputStream();
PrintWriter outputStream = new PrintWriter(socket.getOutputStream());
ArrayList<Byte> dataIn = new ArrayList<Byte>();
ArrayList<String> stringData = new ArrayList<String>();
//Indices to show the location of certain lines in arrayList
int blankIndex = 8;
int lastModIndex = 0;
int byteBlankIndex = 0;
try
{
//Get last modified date
long lastMod = getLastModified(url);
Date d = new Date(lastMod);
//Construct the get request
outputStream.print("GET "+ "/" + pathName + " HTTP/1.1\r\n");
outputStream.print("If-Modified-Since: " + ft.format(d)+ "\r\n");
outputStream.print("Host: " + hostString+"\r\n");
outputStream.print("\r\n");
outputStream.flush();
//Booleans to prevent duplicates, only need first occurrences of key strings
boolean blankDetected = false;
boolean lastModDetected = false;
//Keep track of current index
int count = 0;
int byteCount = 0;
//While loop to read response
String buff = "";
byte t;
while ( (t = (byte) inputStream.read()) != -1)
{
dataIn.add(t);
//Check for key lines
char x = (char) t;
buff = buff + x;
//For the first blank line (signaling the end of the header)
if(x == '\n')
{
stringData.add(buff);
if(buff.equals("\r\n") && !blankDetected)
{
blankDetected = true;
blankIndex = count;
byteBlankIndex = byteCount + 2;
}
//For the last modified line
if(buff.contains("Last-Modified:") && !lastModDetected)
{
lastModDetected = true;
lastModIndex = count;
}
buff = "";
count++;
}
//Increment count
byteCount++;
}
}
The the code to parse through response and write file to disk.
String catalogKey = hostString+ "/" + pathName;
//Get the directory sequence to make
String directoryPath = catalogKey.substring(0, catalogKey.lastIndexOf("/") + 1);
//Make the directory sequence if possible, ignore the boolean value that results
boolean ignoreThisBooleanVal = new File(directoryPath).mkdirs();
//Setup output file, and then write the contents of dataIn (excluding header) to the file
PrintWriter output = new PrintWriter(new FileWriter(new File(catalogKey)),true);
for(int i = byteBlankIndex + 1 ; i < dataIn.size(); i++)
{
output.print(new String(new byte[]{ (byte)dataIn.get(i)}, StandardCharsets.UTF_8));
}
output.close();

byte t;
while ( (t = (byte) inputStream.read()) != -1)
The problem is here. It should read:
int t;
while ( (t = inputStream.read()) != -1)
{
byte b = (byte)t;
// use b from now on in the loop.
The issue is that a byte of 0xff in the input will be returned to the int as 0xff, but to the byte as -1, so you are unable to distinguish it from end of stream.
And you should use a FileOutputStream, not a FileWriter, and you should not accumulate potentially binary data into a String or StringBuffer or anything to do with char. As soon as you've got to the end of the header you should open a FileOutputStream and just start copying bytes. Use buffered streams to make all this more efficient.
Not much point in any of these given that HttpURLConnection already exists.

Java: ByteArray to positive number and vice versa conversion

I'm looking for a way, how to reversibly convert a byte[] of an arbitrary length to positive number (String representation in numbers).
BigInteger offers a solution:
byte[] originalBytes = ...
String string = new BigInteger(originalBytes).toString();
...
byte[] decodedBytes = new BigInteger(string).toByteArray();
However, I'm not sure how to get gracefully rid of negative values (or where to store the sign) and keep the process reversible.

Edit: just replace
String string = new BigInteger(originalBytes).toString();
with
String string = new BigInteger(1, originalBytes).toString();
The 1, signals that the passed array represents a positive number (signum = 1)
Original:
You can just prefix the array with a zero byte:
byte[] original = new byte[] { (byte) 255 };
System.out.println(new BigInteger(original).toString()); // prints "-1"
byte[] paddedCopy = new byte[original.length + 1];
for (int i = 0; i < original.length; i++) {
paddedCopy[i + 1] = original[i];
}
System.out.println(new BigInteger(paddedCopy).toString()); // prints "255"
This will essentially nullify the sign bit, making the number unsigned.

String invalid length after writing to StringBuilder and ByteArrayOutputStream from FileInputStream, issue with "null characters"

The goal is to read a file name from a file, which is a max of 100 bytes, and the actual name is the file name filled with "null-bytes".
Here is what it looks like in GNU nano
Where .PKGINFO is the valid file name, and the ^# represent "null bytes".
I tried here with StringBuilder
package falken;
import java.io.*;
public class Testing {
public Testing() {
try {
FileInputStream tarIn = new FileInputStream("/home/gala/falken_test/test.tar");
final int byteOffset = 0;
final int readBytesLength = 100;
StringBuilder stringBuilder = new StringBuilder();
for ( int bytesRead = 1, n, total = 0 ; (n = tarIn.read()) != -1 && total < readBytesLength ; bytesRead++ ) {
if (bytesRead > byteOffset) {
stringBuilder.append((char) n);
total++;
}
}
String out = stringBuilder.toString();
System.out.println(">" + out + "<");
System.out.println(out.length());
} catch (Exception e) {
/*
This is a pokemon catch not used in final code
*/
e.printStackTrace();
}
}
}
But it gives an invalid String length of 100, while the output on IntelliJ shows the correct string passed withing the >< signs.
>.PKGINFO<
100
Process finished with exit code 0
But when i paste it here on StackOverflow I get the correct string with unknown "null-characters", whose size is actually 100.
>.PKGINFO <
What regex can i use to get rid of the characters after the valid file name?
The file I am reading is ASCII encoded.
I also tried ByteArrayOutputStream, with the same result
package falken;
import java.io.*;
import java.nio.charset.StandardCharsets;
public class Testing {
public Testing() {
try {
FileInputStream tarIn = new FileInputStream("/home/gala/falken_test/test.tar");
final int byteOffset = 0;
final int readBytesLength = 100;
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
for ( int bytesRead = 1, n, total = 0 ; (n = tarIn.read()) != -1 && total < readBytesLength ; bytesRead++ ) {
if (bytesRead > byteOffset) {
byteArrayOutputStream.write(n);
total++;
}
}
String out = byteArrayOutputStream.toString();
System.out.println(">" + out + "<");
System.out.println(out.length());
} catch (Exception e) {
/*
This is a pokemon catch not used in final code
*/
e.printStackTrace();
}
}
}
What could be the issue here?

Well, it seems to be reading null characters as actual characters, spaces in fact. If it's possible, see if you can read the filename, then, cut out the null characters. In your case, you need a data.trim(); and a data2 = data.substring(0,(data.length()-1))

You need to stop appending to the string buffer once you read the first null character from the file.
You seem to want to read a tar archive, have a look at the following code which should get you started.
byte[] buffer = new byte[500]; // POSIX tar header is 500 bytes
FileInputStream is = new FileInputStream("test.tar");
int read = is.read(buffer);
// check number of bytes read; don't bother if not at least the whole
// header has been read
if (read == buffer.length) {
// search for first null byte; this is the end of the name
int offset = 0;
while (offset < 100 && buffer[offset] != 0) {
offset++;
}
// create string from byte buffer using ASCII as the encoding (other
// encodings are not supported by tar)
String name = new String(buffer, 0, offset,
StandardCharsets.US_ASCII);
System.out.println("'" + name + "'");
}
is.close();
You really shouldn't use trim() on the filename, this will break whenever you encounter a filename with leading or trailing blanks.

Checksum doesn't match when computed string contains accented characters

First : I have a string which contains an accented character .
Second : I calcul the checksum for it .
private static String checkSumInStream(String Str, String checksumAlgorithm) throws Exception
{
InputStream stream = new ByteArrayInputStream(Str.getBytes());
MessageDigest digest = MessageDigest.getInstance(checksumAlgorithm);
InputStream input = null;
StringBuffer sb = new StringBuffer();
try{
input = stream;
byte[] buffer = new byte[8192];
do {
int read = input.read(buffer);
if(read <= 0)
break;
digest.update(buffer, 0, read);
} while(true);
byte[] sum = digest.digest();
for (int i = 0; i < sum.length; i++) {
sb.append(Integer.toString((sum[i] & 0xff) + 0x100, 16).substring(1));
}
}catch(IOException io)
{
}finally{
if(input != null)
input.close();
}
return sb.toString();
}
Then i write the string in text file and i I recalcul the checksum of the file
private String checkSum(File file,String checksumAlgorithm) throws Exception
{
MessageDigest digest = MessageDigest.getInstance(checksumAlgorithm);
InputStream input = null;
input = new FileInputStream(file);
byte[] buffer = new byte[8192];
do {
int read = input.read(buffer);
if(read <= 0)
break;
digest.update(buffer, 0, read);
} while(true);
input.close();
byte[] sum = digest.digest();
StringBuffer sb = new StringBuffer();
for (int i = 0; i < sum.length; i++) {
sb.append(Integer.toString((sum[i] & 0xff) + 0x100, 16).substring(1));
}
return sb.toString();
}
--> Result : the comparison between checksum of an output steam and the file doesn't match when text contains an accented character .

How do you write the String to a file? You must be very careful to do that in the equivalent way of how you read it back from the file.
In your case:
OutputStream out = new FileOutputStream(myfile);
out.write(str.getBytes());
out.close();
Then it should work. But you need to keep in mind that str.getBytes() is not a safe method to use when you write to files, because it uses the platform default encoding for your characters. If you send such a file to some other place and use it there, you may be reading it back with the wrong encoding.
And it's possible that your platform default encoding doesn't even support accented characters! (But if you write and read files in exactly the same way, then you should get exactly the same result, so this wouldn't be the cause of your problem)
The best thing to do is to use the UTF-8 character encoding.
Where ever you used str.getBytes(), replace it with str.getBytes("UTF-8"), or str.getBytes(Charset.forName("UTF-8")) if you want to avoid having to catch UnsupportedEncodingException [even though every Java implementation is required to support the UTF-8 encoding. It's annoying...]

Read from inputstream and add to stringbuilder

private void readIncomingMessage() {
try {
StringBuilder builder = new StringBuilder();
InputStream is = socket.getInputStream();
int length = 1024;
byte[] array = new byte[length];
int n = 0;
while ((n = is.read(array, n, 100)) != -1) {
builder.append(new String(array));
if (checkIfComplete(builder.toString())) {
buildListItems(builder.toString(), null);
builder = new StringBuilder();
}
}
} catch (IOException e) {
Log.e("TCPclient", "Something went wrong while reading the socket");
}
}
Hi,
I want to read the stream per block of 100 bytes, convert those bytes into a string and than see if that strings fits certain conditions.
But when I debug I see that builder has a count of 3072.
And I see a string like (text, , , , , , , , , , text , , , , , , , , , text)
How can I just add the text to the stringbuilder?
thx :)
private void readIncomingMessage() {
try {
StringBuilder builder = new StringBuilder();
InputStream is = socket.getInputStream();
int length = 100;
byte[] array = new byte[length];
int n = 0;
while ((n = is.read(array, 0, length)) != -1) {
builder.append(new String(array, 0, n));
if (checkIfComplete(builder.toString())) {
buildListItems(builder.toString(), null);
builder = new StringBuilder();
}
}
} catch (IOException e) {
Log.e("TCPclient", "Something went wrong while reading the socket");
}
}
this solution did the trick for me.
any drawbacks with this solution?

2 problems:
you need to use the 'n' value when converting the bytes to a String. Specifically, use this String constructor String(byte[] bytes, int offset, int length)
when converting bytes to strings on arbitrary boundaries, like you are doing, you have the potential to corrupt multi-byte characters. You'd be better off putting an InputStreamReader on top if the 'is' and reading characters from that.

For more information read the documentation for read(byte[], int, int), new String(byte[]) and new String(byte[], int, int)
n will hold the number of bytes read in the last read operation - not the total number of bytes read. If you only want to read up to 100 bytes at a time, there is no need for a byte array of size 1024, 100 will do. When you create a String from a byte array, it uses the entire array (even if only half was able to be filled by reading), unless you tell it which parts of the array you want to use. Something like this should work, but there are still improvements you could make:
private void readIncomingMessage() {
try {
StringBuilder builder = new StringBuilder();
InputStream is = socket.getInputStream();
int length = 100;
byte[] array = new byte[length];
int pos = 0;
int n = 0;
while (pos != length && ((n = is.read(array, pos, length-pos)) != -1)) {
builder.append(new String(array, pos, n));
pos += n;
if (checkIfComplete(builder.toString())) {
buildListItems(builder.toString(), null);
builder = new StringBuilder();
pos = 0;
}
}
} catch (IOException e) {
Log.e("TCPclient", "Something went wrong while reading the socket");
}
}