We Keep Coding

How to write large raw XML file to Oracle db using Blob object?

I have a function which converts my large XML file to byte array using FileInputStream. It runs fine within my IDE but on when run independently via the executable jar , it throws Exception in thread "main" java.lang.OutOfMemoryError: Java heap space. I'm reading this large file in a byte array to store it as a Blob in the target DB. I don't have control over how the Blob is stored, I just have access to the stored procedure to insert the Blob. Is there a way to read and write chunks of data without loading the entire file in memory ?
function which converts file to byte array -
private byte[] getBytesFromFile(Path path) throws IOException {
FileInputStream fis = new FileInputStream(path.toFile());
byte[] bytes = new byte[(int) path.toFile().length()];
int read = 0;
int offset = 0;
while(offset < bytes.length && (read = fis.read(bytes, offset, bytes.length - offset)) >= 0 ){
offset += read;
}
fis.close();
return bytes;
}
And here's the code which stores the byte array to db using the stored procedure call
private void storeFileToDb(Connection connection, int fileId, String fileName, String fileType, byte[] fileBytes) throws SQLException {
//
String storedProcedure = "{call SP(?,?,?,?,?) }";
CallableStatement callableStatement = connection.prepareCall(storedProcedure);
callableStatement.setInt(1, fileId);
callableStatement.setString(2, fileName);
callableStatement.setString(3, fileType);
Blob fileBlob = connection.createBlob();
fileBlob.setBytes(1, fileBytes);
callableStatement.setBlob(4, fileBlob);
callableStatement.registerOutParameter(5, OracleTypes.NUMBER);
callableStatement.execute();
fileBlob.free(); // not entirely sure how this helps
//callableStatement.close();
}

Use either CallableStatement.setBlob(int, InputStream) or Blob.setBinaryStream(long). Both methods will let work with InputStream or OutputStream objects and avoid creating byte[] array in the memory. Example is show in Adding Large Object Type Object to Database docs.
This should work as long as JDBC driver is smart enough not to create byte[] for the entire blob somewhere internally.

It might be that the server was configured too restrictive. Now is a good time to check the memory parameters.
Blobs can be filled just providing an InputStream.
Also it is a good idea to compress XML data. Try it out: compress some test.xml to test.xml.gz, for the size gain.
Note there exists in standard java:
private byte[] getBytesFromFile(Path path) throws IOException {
return Files.readAllBytes(path);
}
So:
private void storeFileToDb(Connection connection, int fileId, String fileName,
String fileType) throws SQLException, IOException {
Path path = Paths.get(fileName); // Or parameter
try (CallableStatement callableStatement = connection.prepareCall(storedProcedure);
GZipInputStream fileIn = new GZipInputStream(Files.newBufferedInputStream(path))) {
...
callableStatement.setBlob(4, fileIn);
...
}
}
The try-with-resources ensures closing in case of a thrown exception or return or such. Also useful for the statement.
You did not close the statement, having a Blob inside. That is not advisable, as the data may hang around a while. A CallableStatement is a PreparedStatement too, where one use-case is repeatedly executing the SQL with possibly other parameter values. Or not.
And for decompressing GZipOutputStream.

Is it possible to save pdf document to byte array (aspose.pdf for java)

I need to save a pdf document, generated by aspose.pdf for java library to memory (without using temporary file)
I was looking at the documentation and didn't find the save method with the appropriate signature. (I was looking for some kind of outputstream, or at least byte array).
Is it possible? If it is, how can I manage that?
Thanks

Aspose.Pdf for Java supports saving output to both file and stream. Please check following code snippet, It will help you to accomplish the task.
byte[] input = getBytesFromFile(new File("C:/data/HelloWorld.pdf"));
ByteArrayOutputStream output = new ByteArrayOutputStream();
com.aspose.pdf.Document pdfDocument = new com.aspose.pdf.Document(new ByteArrayInputStream(input));
pdfDocument.save(output);
//If you want to read the result into a Document object again, in Java you need to get the
//data bytes and wrap into an input stream.
InputStream inputStream=new ByteArrayInputStream(output.toByteArray());
I am Tilal Ahmad, developer evangelist at Aspose.

I did similar thing.
Here is method to write data to byte:
public byte[] toBytes() {
//create byte array output stream object
ByteArrayOutputStream byteOutStream = new ByteArrayOutputStream();
//create new data output stream object
DataOutputStream outStream = new DataOutputStream(byteOutStream);
try {//write data to bytes stream
if (data != null) {
outStream.write(data);//write data
}//return array of bytes
return byteOutStream.toByteArray();
}
Then you do something like
yourFileName.toBytes;

How to load a classpath resource to an array of byte?

I know how to get the inputstream for a given classpath resource, read from the inputstream until i reach the end, but it looks like a very common problem, and i wonder if there an API that I don't know, or a library that would make things as simple as
byte[] data = ResourceUtils.getResourceAsBytes("/assets/myAsset.bin")
or
byte[] data = StreamUtils.readStreamToEnd(myInputStream)
for example!

Java 9 native implementation:
byte[] data = this.getClass().getClassLoader().getResourceAsStream("/assets/myAsset.bin").readAllBytes();

Have a look at Google guava ByteStreams.toByteArray(INPUTSTREAM), this is might be what you want.

Although i agree with Andrew Thompson, here is a native implementation that works since Java 7 and uses the NIO-API:
byte[] data = Files.readAllBytes(Paths.get(this.getClass().getClassLoader().getResource("/assets/myAsset.bin").toURI()));

Take a look at Apache IOUtils - it has a bunch of methods to work with streams

I usually use the following two approaches to convert Resource into byte[] array.
1 - approach
What you need is to first call getInputStream() on Resource object, and then pass that to convertStreamToByteArray method like below....
InputStream stream = resource.getInputStream();
long size = resource.getFile().lenght();
byte[] byteArr = convertStreamToByteArray(stream, size);
public byte[] convertStreamToByteArray(InputStream stream, long size) throws IOException {
// check to ensure that file size is not larger than Integer.MAX_VALUE.
if (size > Integer.MAX_VALUE) {
return new byte[0];
}
byte[] buffer = new byte[(int)size];
ByteArrayOutputStream os = new ByteArrayOutputStream();
int line = 0;
// read bytes from stream, and store them in buffer
while ((line = stream.read(buffer)) != -1) {
// Writes bytes from byte array (buffer) into output stream.
os.write(buffer, 0, line);
}
stream.close();
os.flush();
os.close();
return os.toByteArray();
}
2 - approach
As Konstantin V. Salikhov suggested, you could use org.apache.commons.io.IOUtils and call its IOUtils.toByteArray(stream) static method and pass to it InputStream object like this...
byte[] byteArr = IOUtils.toByteArray(stream);
Note - Just thought I'll mention this that under the hood toByteArray(...) checks to ensure that file size is not larger than Integer.MAX_VALUE, so you don't have to check for this.

Commonly Java methods will accept an InputStream. In that majority of cases, I would recommend passing the stream directly to the method of interest.
Many of those same methods will also accept an URL (e.g. obtained from getResource(String)). That can sometimes be better, since a variety of the methods will require a repositionable InputStream and there are times that the stream returned from getResourceAsStream(String) will not be repositionable.

Convert byte array into a file without writing the file to a disk

I have saved icon size images in a mysql database in bytes, now, what I need to do is retrieve those file bytes from the database and show those images in a swing application, I have a method which gets the bytes from the database and convert it back to a file but I have to write that file in to the disk
this is my method,
public void downloadFile(int FamerId) throws Exception {
String sql = "SELECT * FROM images WHERE famer_id=?";
Connection con = JDBCConnectionPool.getInstance().checkOut();
PreparedStatement ps = con.prepareStatement(sql);
ps.setInt(1, FamerId);
ResultSet resultSet = ps.executeQuery();
int count = 0;
while (resultSet.next()) {
ByteArrayInputStream bais;
ObjectInputStream inputStream;
bais = new ByteArrayInputStream(resultSet.getBytes("image"));
inputStream = new ObjectInputStream(bais);
SaveFile sf = (SaveFile) inputStream.readObject();
FileOutputStream out = new FileOutputStream("fileLocation/" + resultSet.getString("image_name"));
byte[] bytes = sf.getArray();
int c = 0;
while (c < bytes.length) {
out.write(bytes[c]);
c++;
}
out.close();
inputStream.close();
bais.close();
JDBCConnectionPool.getInstance().checkOut();
}
}
but this method doesn't give what I need, please assist me.

You can read images directly from byte streams with the ImageIO class. Assuming of course that you have previously written the image data in a compatible format. Which is hard to say given the fact that in your code you use an intermediary object input stream when reading your byte data. Here's an example of how you can create an image directly from the database without using intermediary files:
bais = new ByteArrayInputStream(resultSet.getBytes("image"));
final BufferedImage image = ImageIO.read(bais);
// pass the image to your Swing layer to be rendered.
And an example of how you would have written the data to the database, in order to be able to use this code:
final ByteArrayOutputStream baos = new ByteArrayOutputStream(64000);
ImageIO.write(image, "PNG", baos);
final byte[] data = baos.toByteArray();
// write data to database

The answer to your question is its platform dependent. From the docs
A file output stream is an output stream for writing data to a File or
to a FileDescriptor. Whether or not a file is available or may be
created depends upon the underlying platform. Some platforms, in
particular, allow a file to be opened for writing by only one
FileOutputStream (or other file-writing object) at a time. In such
situations the constructors in this class will fail if the file
involved is already open.
FileOutputStream is meant for writing streams of raw bytes such as
image data. For writing streams of characters, consider using
FileWriter.
So if you want to write to a file then file may or may not be created.
If you don't want to create the file and you are just interested in byte[] (content of the file) you can then use solution provided by #Perception or can just pass the inputStream that you have already created.

Store BLOB in Android's SQLite using SQLOpenHelper

Is there a way to store a BLOB into Android's SQLite using SQLOpenHelper?
My BLOB of type InputStream.

SQLite doesn't support streaming BLOB or CLOB data. You have four options:
Convert the InputStream to a byte[]. This will only work if you have enough memory (see below).
Use FileOutputStream or an Android API that supports streaming
Split the data into small blocks and store then in SQLite
Use a database that works on Android and supports streaming. I only know the H2 database. But be aware the Android support for H2 is very new. In that case, you need to use the JDBC API, and you probably want to enable LOBs in the database, otherwise each large BLOB is stored in a separate file.
To convert an InputStream to a byte array, you could use:
public static byte[] readBytesAndClose(InputStream in) throws IOException {
try {
int block = 4 * 1024;
ByteArrayOutputStream out = new ByteArrayOutputStream(block);
byte[] buff = new byte[block];
while (true) {
int len = in.read(buff, 0, block);
if (len < 0) {
break;
}
out.write(buff, 0, len);
}
return out.toByteArray();
} finally {
in.close();
}
}