We Keep Coding

handling comp3 and ebcidic conversion in java to ASCII for large files

I am trying to convert comp3 and EBCIDIC characters in my java code but im running into out of memory exception as the amount of data handled is huge about 5 gb. my code is currently as follows:
byte[] data = Files.readAllBytes(path);
this is resulting in an out of memory exception which i can understand, but i cant use a file scanner as well since the data in the file wont be split into lines.
Can anyone point me in the correct direction on how to handle this
Note: the file may contain records of different length hence splitting it based on record length seams not possible.

As Bill said you could (should) ask for the data to be converted to display characters on the mainframe and if English speaking you can do a ascii transfer.
Also how are you deciding where comp-3 fields start ???
You do not have to read the whole file into memory, you can still read the file in blocks, This method will fill an array of bytes:
protected final int readBuffer(InputStream in, final byte[] buf)
throws IOException {
int total = 0;
int num = in.read(buf, total, buf.length);
while (num >= 0 && total + num < buf.length) {
total += num;
num = in.read(buf, total, buf.length - total);
}
return num;
}
if all the records are the same length, create an array of the record length and the above method will read one record at a time.
Finally the JRecord project has classes to read fixed length files etc. It can do comp-3 conversion. Note: I am the author of JRecord.

I'm running into out of memory exception as the amount of data handled is huge about 5 gb.
You only need to read one record at a time.
My code is currently as follows:
byte[] data = Files.readAllBytes(path);
This is resulting in an out of memory exception which i can understand
Me too.
but i cant use a file scanner as well since the data in the file wont be split into lines.
You mean you can't use the Scanner class? That's not the only way to read a record at a time.
In any case not all files have record delimiters. Some have fixed-length records, some have length words at the start of each record, and some have record type attributes at the start of each record, or in both cases at least in the fixed part of the record.
I'll have to split it based on an attribute record_id at a particular position(say at the begining of each record) that will tell me the record length
So read that attribute, decode it if necessary, and read the rest of the record according to the record length you derive from the attribute. One at a time.
I direct your attention to the methods of DataInputStream, especially readFully(). You will also need a Java COMP-3 library. There are several available. Most of the rest can be done by built-in EBCDIC character set decoders.

How to convert "java.nio.HeapByteBuffer" to String

I have a data structure java.nio.HeapByteBuffer[pos=71098 lim=71102 cap=94870], which I need to convert into Int (in Scala), the conversion might look simple but whatever which I approach , i did not get right conversion. could you please help me?
Here is my code snippet:
val v : ByteBuffer= map.get("company").get
val utf_str = new String(v, java.nio.charset.StandardCharsets.UTF_8)
println (utf_str)
the output is just "R" ??

I can't see how you can even get that to compile, String has constructors that accepts another string or possibly an array, but not a ByteBuffer or any of its parents.
To work with the nio buffer api you first write to a buffer, then do a flip before you read from the buffer, there are lots of good resources online about that. This one for example: http://tutorials.jenkov.com/java-nio/buffers.html
How to read that as a string entirely depends on how the characters are encoded inside the buffer, if they are two bytes per character (as strings are in Java/the JVM) you can convert your buffer to a character buffer by using asCharBuffer.
So, for example:
val byteBuffer = ByteBuffer.allocate(7).order(ByteOrder.BIG_ENDIAN);
byteBuffer.putChar('H').putChar('i').putChar('!')
byteBuffer.flip()
val charBuffer = byteBuffer.asCharBuffer
assert(charBuffer.toString == "Hi!")

Base64 binary data type in java

I need to attach a Base64 binary element to a SOAP message...Im doing a dry run to check if I can convert a value read from a file into Base64 binary..
Here is the below code..In the last line I try to print the type of encoded1(I assume it should be Base64 binary values) but I get the following display..."Attachment[B"...How can I confirm if its Base64 binary string?
Path path = Paths.get("c:/tomcatupload/text.csv");
byte[] attachment1 = Files.readAllBytes(path);
byte[] encoded1 = Base64.encode(attachment1);
System.out.println("Attachment"+ encoded1.getClass().getName());

Base-64 encoding is a way to convert arbitrary bytes to bytes that fit in a range of text characters in ASCII encoding. This is done without any interpretation whatsoever - raw bytes are converted to base-64 on sender's end; the receiver converts them back to a stream of bytes, and that's all there is to it.
When your code prints encoded1.getClass().getName(), all it gets is the static type of the byte array. In order to interpret the data encoded in base-64 as something meaningful to your program, you need to know the format of underlying data transported as base-64. Once the bytes are delivered to you (in your case, that's encoded1 array of bytes) you need to decide what's inside, and act accordingly.
For example, if a serialized Java object is sent to you as base-64, you need to take encoded1, make an in-memory stream from it, and read the object using the regular serialization mechanism:
ByteArrayInputStream memStream = new ByteArrayInputStream(encoded1);
ObjectInputStream objStream = new ObjectInputStream(memStream);
Object attachedObject = objStream.readObject();

The encoding by Base64.encode() was successful if and only if size of encoded1 > size of obtained attachment1.
Please refer, to understand how the encoding works.
http://en.wikipedia.org/wiki/Base64
By the way, your last statement doesn't print the array content. It prints the name of the class to which encoded1 belongs to.

How to convert byte array in String format to byte array?

I have created a byte array of a file.
FileInputStream fileInputStream=null;
File file = new File("/home/user/Desktop/myfile.pdf");
byte[] bFile = new byte[(int) file.length()];
try {
fileInputStream = new FileInputStream(file);
fileInputStream.read(bFile);
fileInputStream.close();
}catch(Exception e){
e.printStackTrace();
}
Now,I have one API, which is expecting a json input, there I have to put the above byte array in String format. And after reading the byte array in string format, I need to convert it back to byte array again.
So, help me to find;
1) How to convert byte array to String and then back to the same byte array?

The general problem of byte[] <-> String conversion is easily solved once you know the actual character set (encoding) that has been used to "serialize" a given text to a byte stream, or which is needed by the peer component to accept a given byte stream as text input - see the perfectly valid answers already given on this. I've seen a lot of problems due to lack of understanding character sets (and text encoding in general) in enterprise java projects even with experienced software developers, so I really suggest diving into this quite interesting topic. It is generally key to keep the character encoding information as some sort of "meta" information with your binary data if it represents text in some way. Hence the header in, for example, XML files, or even suffixes as parts of file names as it is sometimes seen with Apache htdocs contents etc., not to mention filesystem-specific ways to add any kind of metadata to files. Also, when communicating via, say, http, the Content-Type header fields often contain additional charset information to allow for correct interpretation of the actual Contents.
However, since in your example you read a PDF file, I'm not sure if you can actually expect pure text data anyway, regardless of any character encoding.
So in this case - depending on the rest of the application you're working on - you may want to transfer binary data within a JSON string. A common way to do so is to convert the binary data to Base64 and, once transferred, recover the binary data from the received Base64 string.
How do I convert a byte array to Base64 in Java?
is a good starting point for such a task.

String class provides an overloaded constructor for this.
String s = new String(byteArray, "UTF-8");
byteArray = s.getBytes("UTF-8");
Providing an explicit encoding charset is encouraged because different encoding schemes may have different byte representations. Read more here and here.
Also, your inputstream maynot read all the contents in one go. You have to read in a loop until there is nothing more left to be read. Read the documentation. read() returns the number of bytes read.
Reads up to b.length bytes of data from this input stream into an
array of bytes. This method blocks until some input is available

String.getBytes() and String(byte[] bytes) are methods to consider.

Convert byte array to String
String s = new String(bFile , "ISO-8859-1" );
Convert String to byte array
byte bArray[] =s.getBytes("ISO-8859-1");

create a specific number of bytes from a string

I would like to know if there is a way to create a specific number of bytes from a string.
I am trying to unit test some part of my code and it can take an array of bytes or a string.
But the data that I am getting will consist exactly 132 bytes (where each data point is two byte signed integers that uses 2's complement).
The data I am retrieving will consist of multiple data points in the above bytes where each data point is 2bytes.
I am planning to unit test my code. So I would like to create a string and convert to byte array and pass it so that I can cross check my data points again.
Also are there any tools available by which I can send binary data via a com port. I was looking at eltima software serial port.

there is this way which I am doing, but looking for more easy way...
final String MACID = new Character((char) 48).toString();
final String STX = new Character((char) 2).toString();
final String str = MACID + STX;
final byte[] utf8Bytes = str.getBytes("UTF-8");
this would surely just take 2 bytes.