We Keep Coding

java ByteBuffer writing into file charset issue

There's a method that encodes a file
public static void cipher(File input, int[] key, File output) throws IOException {
ByteBuffer buffer = ByteBuffer.allocate(10);
FileInputStream fin = new FileInputStream(input);
FileChannel fcin = fin.getChannel();
FileOutputStream fout = new FileOutputStream( output );
FileChannel fcout = fout.getChannel();
ByteBuffer temp;
while (true){
buffer.clear();
int r = fcin.read(buffer);
if(r == -1){
break;
}
//cb = StandardCharsets.UTF_8.decode(cipherBuffer(buffer,key));
//System.out.println(cb.toString());
temp = cipherBuffer(buffer,key);
for (int i = 0; i < 10; i++){
System.out.print((char)temp.get(i));
}
temp.flip();
fcout.write(temp);
}
}
The cipherBuffer() method changes the initial Buffer according to a given key.
Tried to use commented code, but it didn't help. For now it even doesn't write to the output
public static ByteBuffer cipherBuffer(ByteBuffer initialBuffer, int[] key){
ByteBuffer result = ByteBuffer.allocate(10);
for (int i = 0; i < 10; i++){
result.put(i, initialBuffer.get(key[i]));
//System.out.println(result.get(i));
}
return result;
}
That's how the output looks like at this stage
￐ﾰ￐￐ﾺﾷﾻ￐￐ﾰﾲﾀ￑ ￐ ﾾ￐￐ﾳ￐￐ﾑ￐ﾾﾴ￐￐ ﾵﾃ￐ﾾﾻ￑￐￐

How to read the large text files efficiently in java

Here, I am reading the 18 MB file and store it in a two dimensional array. But this program takes almost 15 minutes to run. Is there anyway to optimize the running time of the program. The file contains only binary values. Thanks in advance…
public class test
{
public static void main(String[] args) throws FileNotFoundException, IOException
{
BufferedReader br;
FileReader fr=null;
int m = 2160;
int n = 4320;
int[][] lof = new int[n][m];
String filename = "D:/New Folder/ETOPOCHAR";
try {
Scanner input = new Scanner(new File("D:/New Folder/ETOPOCHAR"));
double range_km=1.0;
double alonn=-57.07; //180 to 180
double alat=38.53;
while (input.hasNextLine()) {
for (int i = 0; i < m; i++) {
for (int j = 0; j < n; j++) {
try
{
lof[j][i] = input.nextInt();
System.out.println("value[" + j + "][" + i + "] = "+ lof[j][i]);
}
catch (java.util.NoSuchElementException e) {
// e.printStackTrace();
}
}
} //print the input matrix
}
I have also tried with byte array but i can not save it in twoD array...
public class FileToArrayOfBytes
{
public static void main( String[] args )
{
FileInputStream fileInputStream=null;
File file = new File("name of file");
byte[] bFile = new byte[(int) file.length()];
try {
//convert file into array of bytes
fileInputStream = new FileInputStream(file);
fileInputStream.read(bFile);
fileInputStream.close();
for (int i = 0; i < bFile.length; i++) {
System.out.print((char)bFile[i]);
}
System.out.println("Done");
}catch(Exception e){
e.printStackTrace();
}
}
}

You can read the file into a byte array first, then deserialize these bytes. Start with 2048 bytes buffer (as input buffer), then experiment by increasing/decreasing its size, but the experimental buffer size values should be a power of two (512, 1024, 2048, etc).
As far as I rememenber, there are good chances that the best performance can be achived with a buffer of size 2048 bytes, but it is OS dependent and should be verified.
Code sample (here you can try different values of BUFFER_SIZE variable, in my case I've read a test file of size 7.5M in less then one second):
public static void main(String... args) throws IOException {
File f = new File(args[0]);
byte[] buffer = new byte[BUFFER_SIZE];
ByteBuffer result = ByteBuffer.allocateDirect((int) f.length());
try (FileInputStream fos = new FileInputStream(f)) {
int bytesRead;
int totalBytesRead = 0;
while ((bytesRead = fos.read(buffer, 0, BUFFER_SIZE)) != -1) {
result.put(buffer, 0, bytesRead);
totalBytesRead += bytesRead;
}
// debug info
System.out.printf("Read %d bytes\n", totalBytesRead);
// Here you can do whatever you want with the result, including creation of a 2D array...
int pos = result.position();
result.rewind();
for (int i = 0; i < pos / 4; i++) {
System.out.println(result.getInt());
}
}
}
Take your time and read docs for java.io, java.nio packages as well as Scanner class, just to improve understanding.

BufferedReader directly to byte[]

is there any possibility my following BufferedReader is able to put the input directly into a byte[]?
public static Runnable reader() throws IOException {
Log.e("Communication", "reader");
din = new DataInputStream(sock.getInputStream());
brdr = new BufferedReader(new InputStreamReader(din), 300);
boolean done = false;
while (!done) {
try {
char[] buffer = new char[200];
int length = brdr.read(buffer, 0, 200);
String message = new String(buffer, 0, length);
btrar = message.getBytes("ISO-8859-1");
int i=0;
for (int counter = 0; counter < message.length(); counter++) {
i++;
System.out.println(btrar[counter] + " = " + " btrar " + i);
}
...
thats the part of the reader, pls have a look.
I want the input directly to btrar,

is there any possibility my following BufferedReader is able to put the input directly into a byte[]?
Any Reader is designed to let you read characters, not bytes. To read binary data, just use an InputStream - using BufferedInputStream to buffer it if you want.
It's not really clear what you're trying to do, but you can use something like:
BufferedInputStream input = new BufferedInputStream(sock.getInputStream());
while (!done) {
// TODO: Rename btrar to something more meaningful
int bytesRead = input.read(btrar);
// Do something with the data...
}

Java Socket synchronization behavior

I tried to solve the problem in many ways but without success and I have also looked for information in this forum but with same results, so here we go.
I am actually doing a server daemon that accepts client requests and then it (the server) transfers all the files contained in a specific folder. I'm going to post the code of the sendFileData (on the server) and the receiveFileData (on the client).
The server uses:
public static void sendFileData(File file, Socket socket) throws FileNotFoundException, IOException, SocketException {
byte[] auxiliar = new byte[8192];
byte[] mybytearray = new byte[(int) file.length()];
int longitud = mybytearray.length;
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(file));
bis.read(mybytearray, 0, longitud);
DataOutputStream os = new DataOutputStream(socket.getOutputStream());
int paquetes = longitud / 8187;
int resto = longitud % 8187;
int i = 0;
while(i<paquetes){//The length goes on the first 4 bytes and the 5th tells if there are more packets to send (8192 bytes or less).
byte[] bytes = ByteBuffer.allocate(4).putInt(8187).array();
auxiliar[0] = bytes[0];
auxiliar[1] = bytes[1];
auxiliar[2] = bytes[2];
auxiliar[3] = bytes[3];
auxiliar[4] = 1;
for(int j = 5; j < 8192; j++){
auxiliar[j] = mybytearray[i*8187+(j-5)];
}
os.write(auxiliar, 0, 8192);
i+=1;
}
if(resto > 0){
byte[] bytes = ByteBuffer.allocate(4).putInt(resto).array();
auxiliar[0] = bytes[0];
auxiliar[1] = bytes[1];
auxiliar[2] = bytes[2];
auxiliar[3] = bytes[3];
auxiliar[4] = 0;
for(int j = 5; j < resto+5; j++){
auxiliar[j] = mybytearray[i*8187+(j-5)];
}
os.write(auxiliar, 0, resto+5);
}
os.flush();
}
And in the client side:
public static void receiveFileData(String nombreFichero, Socket s) throws IOException{
File monitored = new File(nombreFichero);
if(monitored.exists() == false){
monitored.createNewFile();
}
byte[] mybytearray;
DataInputStream is = new DataInputStream(s.getInputStream());
FileOutputStream fos = new FileOutputStream(monitored);
BufferedOutputStream bos = new BufferedOutputStream(fos);
int bytesRead = 0;
int hasNext = 1;
do {
bytesRead = is.readInt();//Leo longitud
try {
Thread.sleep(1);// HERE!!!!
} catch (InterruptedException e) {
}
// System.out.println("Bytes read "+bytesRead );
if(bytesRead <= 8187 && bytesRead > 0){
// System.out.println("Bytes leídos "+bytesRead );
hasNext = is.readByte();//Leo si hay más datos por enviar
mybytearray = new byte[bytesRead];
is.read(mybytearray);
if(monitored.exists()){
synchronized(monitored){
bos.write(mybytearray, 0, mybytearray.length);
}
}
mybytearray = null;
}else{
System.out.println("Fuera de rango "+bytesRead);
}
}while(hasNext == 1);
bos.close();
mybytearray = null;
System.out.println("Fichero recibido: "+monitored.getAbsolutePath());
}
In the receiveFileData code, if I do not put a Thread.sleep(1) or a System.out.println() or whatever who takes time to execute, I am not receiving the data in the correct way on the client, because readInt() returns a very high number randomly negative or positive (which implies Heap out of memory and other exceptions).
Sure it's something about synchronization but I think the transfering schema between the two methods is correct (maybe the client is too slow and server too fast).
What is happening?? Because I do not want to put a Thread.sleep, this is not good programming here I think.
Thank you so much!

is.read(bytes) is not guaranteed to fill the supplied byte array. You need to check its return value to see how many bytes were read or (better) use readFully().
The sleep() probably just allows time for all bytes to have been returned from the socket.

CharBuffer vs. char[]

Is there any reason to prefer a CharBuffer to a char[] in the following:
CharBuffer buf = CharBuffer.allocate(DEFAULT_BUFFER_SIZE);
while( in.read(buf) >= 0 ) {
out.append( buf.flip() );
buf.clear();
}
vs.
char[] buf = new char[DEFAULT_BUFFER_SIZE];
int n;
while( (n = in.read(buf)) >= 0 ) {
out.write( buf, 0, n );
}
(where in is a Reader and out in a Writer)?

No, there's really no reason to prefer a CharBuffer in this case.
In general, though, CharBuffer (and ByteBuffer) can really simplify APIs and encourage correct processing. If you were designing a public API, it's definitely worth considering a buffer-oriented API.

I wanted to mini-benchmark this comparison.
Below is the class I have written.
The thing is I can't believe that the CharBuffer performed so badly. What have I got wrong?
EDIT: Since the 11th comment below I have edited the code and the output time, better performance all round but still a significant difference in times. I also tried out2.append((CharBuffer)buff.flip()) option mentioned in the comments but it was much slower than the write option used in the code below.
Results: (time in ms)
char[] : 3411
CharBuffer: 5653
public class CharBufferScratchBox
{
public static void main(String[] args) throws Exception
{
// Some Setup Stuff
String smallString =
"1111111111222222222233333333334444444444555555555566666666667777777777888888888899999999990000000000";
StringBuilder stringBuilder = new StringBuilder();
for (int i = 0; i < 1000; i++)
{
stringBuilder.append(smallString);
}
String string = stringBuilder.toString();
int DEFAULT_BUFFER_SIZE = 1000;
int ITTERATIONS = 10000;
// char[]
StringReader in1 = null;
StringWriter out1 = null;
Date start = new Date();
for (int i = 0; i < ITTERATIONS; i++)
{
in1 = new StringReader(string);
out1 = new StringWriter(string.length());
char[] buf = new char[DEFAULT_BUFFER_SIZE];
int n;
while ((n = in1.read(buf)) >= 0)
{
out1.write(
buf,
0,
n);
}
}
Date done = new Date();
System.out.println("char[] : " + (done.getTime() - start.getTime()));
// CharBuffer
StringReader in2 = null;
StringWriter out2 = null;
start = new Date();
CharBuffer buff = CharBuffer.allocate(DEFAULT_BUFFER_SIZE);
for (int i = 0; i < ITTERATIONS; i++)
{
in2 = new StringReader(string);
out2 = new StringWriter(string.length());
int n;
while ((n = in2.read(buff)) >= 0)
{
out2.write(
buff.array(),
0,
n);
buff.clear();
}
}
done = new Date();
System.out.println("CharBuffer: " + (done.getTime() - start.getTime()));
}
}

If this is the only thing you're doing with the buffer, then the array is probably the better choice in this instance.
CharBuffer has lots of extra chrome on it, but none of it is relevant in this case - and will only slow things down a fraction.
You can always refactor later if you need to make things more complicated.

The difference, in practice, is actually <10%, not 30% as others are reporting.
To read and write a 5MB file 24 times, my numbers taken using a Profiler. They were on average:
char[] = 4139 ms
CharBuffer = 4466 ms
ByteBuffer = 938 (direct) ms
Individual tests a couple times favored CharBuffer.
I also tried replacing the File-based IO with In-Memory IO and the performance was similar. If you are trying to transfer from one native stream to another, then you are better off using a "direct" ByteBuffer.
With less than 10% performance difference, in practice, I would favor the CharBuffer. It's syntax is clearer, there's less extraneous variables, and you can do more direct manipulation on it (i.e. anything that asks for a CharSequence).
Benchmark is below... it is slightly wrong as the BufferedReader is allocated inside the test-method rather than outside... however, the example below allows you to isolate the IO time and eliminate factors like a string or byte stream resizing its internal memory buffer, etc.
public static void main(String[] args) throws Exception {
File f = getBytes(5000000);
System.out.println(f.getAbsolutePath());
try {
System.gc();
List<Main> impls = new java.util.ArrayList<Main>();
impls.add(new CharArrayImpl());
//impls.add(new CharArrayNoBuffImpl());
impls.add(new CharBufferImpl());
//impls.add(new CharBufferNoBuffImpl());
impls.add(new ByteBufferDirectImpl());
//impls.add(new CharBufferDirectImpl());
for (int i = 0; i < 25; i++) {
for (Main impl : impls) {
test(f, impl);
}
System.out.println("-----");
if(i==0)
continue; //reset profiler
}
System.gc();
System.out.println("Finished");
return;
} finally {
f.delete();
}
}
static int BUFFER_SIZE = 1000;
static File getBytes(int size) throws IOException {
File f = File.createTempFile("input", ".txt");
FileWriter writer = new FileWriter(f);
Random r = new Random();
for (int i = 0; i < size; i++) {
writer.write(Integer.toString(5));
}
writer.close();
return f;
}
static void test(File f, Main impl) throws IOException {
InputStream in = new FileInputStream(f);
File fout = File.createTempFile("output", ".txt");
try {
OutputStream out = new FileOutputStream(fout, false);
try {
long start = System.currentTimeMillis();
impl.runTest(in, out);
long end = System.currentTimeMillis();
System.out.println(impl.getClass().getName() + " = " + (end - start) + "ms");
} finally {
out.close();
}
} finally {
fout.delete();
in.close();
}
}
public abstract void runTest(InputStream ins, OutputStream outs) throws IOException;
public static class CharArrayImpl extends Main {
char[] buff = new char[BUFFER_SIZE];
public void runTest(InputStream ins, OutputStream outs) throws IOException {
Reader in = new BufferedReader(new InputStreamReader(ins));
Writer out = new BufferedWriter(new OutputStreamWriter(outs));
int n;
while ((n = in.read(buff)) >= 0) {
out.write(buff, 0, n);
}
}
}
public static class CharBufferImpl extends Main {
CharBuffer buff = CharBuffer.allocate(BUFFER_SIZE);
public void runTest(InputStream ins, OutputStream outs) throws IOException {
Reader in = new BufferedReader(new InputStreamReader(ins));
Writer out = new BufferedWriter(new OutputStreamWriter(outs));
int n;
while ((n = in.read(buff)) >= 0) {
buff.flip();
out.append(buff);
buff.clear();
}
}
}
public static class ByteBufferDirectImpl extends Main {
ByteBuffer buff = ByteBuffer.allocateDirect(BUFFER_SIZE * 2);
public void runTest(InputStream ins, OutputStream outs) throws IOException {
ReadableByteChannel in = Channels.newChannel(ins);
WritableByteChannel out = Channels.newChannel(outs);
int n;
while ((n = in.read(buff)) >= 0) {
buff.flip();
out.write(buff);
buff.clear();
}
}
}

I think that CharBuffer and ByteBuffer (as well as any other xBuffer) were meant for reusability so you can buf.clear() them instead of going through reallocation every time
If you don't reuse them, you're not using their full potential and it will add extra overhead. However if you're planning on scaling this function this might be a good idea to keep them there

You should avoid CharBuffer in recent Java versions, there is a bug in #subsequence(). You cannot get a subsequence from the second half of the buffer since the implementation confuses capacity and remaining. I observed the bug in java 6-0-11 and 6-0-12.

The CharBuffer version is slightly less complicated (one less variable), encapsulates buffer size handling and makes use of a standard API. Generally I would prefer this.
However there is still one good reason to prefer the array version, in some cases at least. CharBuffer was only introduced in Java 1.4 so if you are deploying to an earlier version you can't use Charbuffer (unless you roll-your-own/use a backport).
P.S If you use a backport remember to remove it once you catch up to the version containing the "real" version of the backported code.