Is there a way to store a BLOB into Android's SQLite using SQLOpenHelper?
My BLOB of type InputStream.
SQLite doesn't support streaming BLOB or CLOB data. You have four options:
Convert the InputStream to a byte[]. This will only work if you have enough memory (see below).
Use FileOutputStream or an Android API that supports streaming
Split the data into small blocks and store then in SQLite
Use a database that works on Android and supports streaming. I only know the H2 database. But be aware the Android support for H2 is very new. In that case, you need to use the JDBC API, and you probably want to enable LOBs in the database, otherwise each large BLOB is stored in a separate file.
To convert an InputStream to a byte array, you could use:
public static byte[] readBytesAndClose(InputStream in) throws IOException {
try {
int block = 4 * 1024;
ByteArrayOutputStream out = new ByteArrayOutputStream(block);
byte[] buff = new byte[block];
while (true) {
int len = in.read(buff, 0, block);
if (len < 0) {
break;
}
out.write(buff, 0, len);
}
return out.toByteArray();
} finally {
in.close();
}
}
Related
I have a function which converts my large XML file to byte array using FileInputStream. It runs fine within my IDE but on when run independently via the executable jar , it throws Exception in thread "main" java.lang.OutOfMemoryError: Java heap space. I'm reading this large file in a byte array to store it as a Blob in the target DB. I don't have control over how the Blob is stored, I just have access to the stored procedure to insert the Blob. Is there a way to read and write chunks of data without loading the entire file in memory ?
function which converts file to byte array -
private byte[] getBytesFromFile(Path path) throws IOException {
FileInputStream fis = new FileInputStream(path.toFile());
byte[] bytes = new byte[(int) path.toFile().length()];
int read = 0;
int offset = 0;
while(offset < bytes.length && (read = fis.read(bytes, offset, bytes.length - offset)) >= 0 ){
offset += read;
}
fis.close();
return bytes;
}
And here's the code which stores the byte array to db using the stored procedure call
private void storeFileToDb(Connection connection, int fileId, String fileName, String fileType, byte[] fileBytes) throws SQLException {
//
String storedProcedure = "{call SP(?,?,?,?,?) }";
CallableStatement callableStatement = connection.prepareCall(storedProcedure);
callableStatement.setInt(1, fileId);
callableStatement.setString(2, fileName);
callableStatement.setString(3, fileType);
Blob fileBlob = connection.createBlob();
fileBlob.setBytes(1, fileBytes);
callableStatement.setBlob(4, fileBlob);
callableStatement.registerOutParameter(5, OracleTypes.NUMBER);
callableStatement.execute();
fileBlob.free(); // not entirely sure how this helps
//callableStatement.close();
}
Use either CallableStatement.setBlob(int, InputStream) or Blob.setBinaryStream(long). Both methods will let work with InputStream or OutputStream objects and avoid creating byte[] array in the memory. Example is show in Adding Large Object Type Object to Database docs.
This should work as long as JDBC driver is smart enough not to create byte[] for the entire blob somewhere internally.
It might be that the server was configured too restrictive. Now is a good time to check the memory parameters.
Blobs can be filled just providing an InputStream.
Also it is a good idea to compress XML data. Try it out: compress some test.xml to test.xml.gz, for the size gain.
Note there exists in standard java:
private byte[] getBytesFromFile(Path path) throws IOException {
return Files.readAllBytes(path);
}
So:
private void storeFileToDb(Connection connection, int fileId, String fileName,
String fileType) throws SQLException, IOException {
Path path = Paths.get(fileName); // Or parameter
try (CallableStatement callableStatement = connection.prepareCall(storedProcedure);
GZipInputStream fileIn = new GZipInputStream(Files.newBufferedInputStream(path))) {
...
callableStatement.setBlob(4, fileIn);
...
}
}
The try-with-resources ensures closing in case of a thrown exception or return or such. Also useful for the statement.
You did not close the statement, having a Blob inside. That is not advisable, as the data may hang around a while. A CallableStatement is a PreparedStatement too, where one use-case is repeatedly executing the SQL with possibly other parameter values. Or not.
And for decompressing GZipOutputStream.
I want to download large files from Google Cloud Storage using the google provided Java library com.google.cloud.storage. I have working code, but I still have one question and one major concern:
My major concern is, when is the file content actually downloaded? During (references to the code below) storage.get(blobId), during blob.reader() or during reader.read(bytes)? This gets very important when it comes to how to handle an invalid checksum, what do I need to do in order to actually trigger that the file is fetched over the network again?
The simpler question is: Is there built in functionality to do md5 (or crc32c) check on the received file in the google library? Maybe I don't need to implement it on my own.
Here is my method trying to download big files from Google Cloud Storage:
private static final int MAX_NUMBER_OF_TRIES = 3;
public Path downloadFile(String storageFileName, String bucketName) throws IOException {
// In my real code, this is a field populated in the constructor.
Storage storage = Objects.requireNonNull(StorageOptions.getDefaultInstance().getService());
BlobId blobId = BlobId.of(bucketName, storageFileName);
Path outputFile = Paths.get(storageFileName.replaceAll("/", "-"));
int retryCounter = 1;
Blob blob;
boolean checksumOk;
MessageDigest messageDigest;
try {
messageDigest = MessageDigest.getInstance("MD5");
} catch (NoSuchAlgorithmException ex) {
throw new RuntimeException(ex);
}
do {
LOGGER.debug("Start download file {} from bucket {} to Content Store (try {})", storageFileName, bucketName, retryCounter);
blob = storage.get(blobId);
if (null == blob) {
throw new CloudStorageCommunicationException("Failed to download file after " + retryCounter + " tries.");
}
if (Files.exists(outputFile)) {
Files.delete(outputFile);
}
try (ReadChannel reader = blob.reader();
FileChannel channel = new FileOutputStream(outputFile.toFile(), true).getChannel()) {
ByteBuffer bytes = ByteBuffer.allocate(128 * 1024);
int bytesRead = reader.read(bytes);
while (bytesRead > 0) {
bytes.flip();
messageDigest.update(bytes.array(), 0, bytesRead);
channel.write(bytes);
bytes.clear();
bytesRead = reader.read(bytes);
}
}
String checksum = Base64.encodeBase64String(messageDigest.digest());
checksumOk = checksum.equals(blob.getMd5());
if (!checksumOk) {
Files.delete(outputFile);
messageDigest.reset();
}
} while (++retryCounter <= MAX_NUMBER_OF_TRIES && !checksumOk);
if (!checksumOk) {
throw new CloudStorageCommunicationException("Failed to download file after " + MAX_NUMBER_OF_TRIES + " tries.");
}
return outputFile;
}
The google-cloud-java storage library does not validate checksums on its own when reading data beyond normal HTTPS/TCP correctness checking. If it compared the MD5 of the received data to the known MD5, it would need to download the entire file before it could return any results from read(), which for very large files would be infeasible.
What you're doing is a good idea if you need the additional protection of comparing MD5s. If this is a one-off task, you could use the gsutil command-line tool, which does this same sort of additional check.
As the JavaDoc of ReadChannel says:
Implementations of this class may buffer data internally to reduce remote calls.
So the implementation you get from blob.reader() could cache the whole file, some bytes or nothing and just fetch byte for byte when you call read(). You will never know and you shouldn't care.
As only read() throws an IOException and the other methods you used do not, I'd say that only calling read() will actually download stuff. You can also see this in the sources of the lib.
Btw. despite the example in the JavaDocs of the library, you should check for >= 0, not > 0. 0 just means nothing was read, not that end of stream is reached. End of stream is signaled by returning -1.
For retrying after a failed checksum check, get a new reader from the blob. If something caches the downloaded data, then the reader itself. So if you get a new reader from the blob, the file will be redownloaded from remote.
Hi i want to create a blob in hibernate from an inputstream, but i don't know the length of the stream.
Hibernate.getLobCreator(sessionFactory.getCurrentSession()).createBlob(stream, length)
how can i crate a blob without knowing the length of the stream?
EDIT1
in older hibernate versions it was possible
http://viralpatel.net/blogs/tutorial-save-get-blob-object-spring-3-mvc-hibernate/
Blob blob = Hibernate.createBlob(file.getInputStream());
EDIT2
ok but it had an buggy implementation
return new SerializableBlob( new BlobImpl( stream, stream.available() ) );
stream.available isn't the real size
EDIT 3
i tried
session.doWork(new Work() {
#Override
public void execute(Connection conn) throws SQLException {
LargeObjectManager lobj = ((org.postgresql.PGConnection) conn).getLargeObjectAPI();
but conn is just a NewProxyConnection from c3p0.
Here is what i'm using now
Session currentSession = getSessionFactory().getCurrentSession();
Blob blob = Hibernate.getLobCreator(currentSession).createBlob(new byte[0]);
OutputStream setBinaryStream = blob.setBinaryStream(1);
Utils.fastChannelCopy(input, setBinaryStream);
setBinaryStream.close();
Try passing in the InputStream and a length of -1:
session.getLobHelper().createBlob(stream, -1);
This works with SQL Server 2008. It seems that Hibernate passes this value directly to the JDBC driver, so this may or may not work depending on your database driver.
Note: If I pass in the incorrect stream length (including 0), I will get a DataException from Hibernate (from the driver: "com.microsoft.sqlserver.jdbc.SQLServerException: The stream value is not the specified length. The specified length was 10, the actual length is 13."). With a length of -1, it always works without complaining, and the data is saved successfully to the database.
Workaround could be to save input stream to a file then to read the file into blob.
I understand my answer is a little different from your question. But this may help others who land here. For my requirement, i am getting the absolute file path as the input to create a blob. If you too have a similar requirement, you can use the below snippet.
Session session = sessionFactory.getCurrentSession();
File file = new File(filePath);
Blob blob = Hibernate.getLobCreator(session).createBlob(new FileInputStream(file), file.length());
file.length() will give you the length of the file.
change the save method in Dao as below
#Transactional
public void save(Document document, InputStream inputStream) throws IOException {
Session session = sessionFactory.getCurrentSession();
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
int nRead;
byte[] data = new byte[16384];
while ((nRead = inputStream.read(data, 0, data.length)) != -1) {
buffer.write(data, 0, nRead);
}
buffer.flush();
Blob blob = Hibernate.getLobCreator(session).createBlob(buffer.toByteArray());
document.setContent(blob);
session.save(document);
}
I think, it's a goog solution, if you convert the InputStream to byte array, so you can use the createBlob method, that accepts a byte array instead of an InputStream and the length as parameter.
For the converting you can use the IOUtils from the Apache Commons IO.
Hibernate.getLobCreator(sessionFactory.getCurrentSession()).createBlob(IOUtils.toByteArray(stream));
I have a huge string that I need to cache somewhere and since I cannot write to file my only option is to store this on the data base as text, more specifically, in the clob I have I'm storing a JSON file where I'm placing the compressed string under a certain key of that JSON object.
I'm compressing the strings but somewhere across the string manipulation something happens that doesn't allow me to decompress the data, so I'm wondering if I should encode the data to base 64 but that will lose compression.
What could I do to ensure I can store the compressed string in the database so I can later fetch it?
I cannot change the database, so I'm stuck with that CLOB field
These are my compression functions:
public static String compress(String text) {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
try {
OutputStream out = new DeflaterOutputStream(baos);
out.write(text.getBytes("UTF-8"));
out.close();
} catch (IOException e) {
//ooops
}
return baos.toString();
}
public static String decompress(String bytes) {
InputStream in = new InflaterInputStream(new ByteArrayInputStream(bytes.getBytes()));
ByteArrayOutputStream baos = new ByteArrayOutputStream();
try {
byte[] buffer = new byte[8192];
int len;
while ((len = in.read(buffer)) > 0)
baos.write(buffer, 0, len);
return new String(baos.toByteArray(), "UTF-8");
} catch (IOException e) {
//ooops
}
}
As you found out, you can't store binary data in a CLOB without some corruption, so encoding to text will be required.
Base 64 will, on average add 33% to the size of your binary data. So you will lose some compression, but if your compression ratio is greater than 25% (this is often easy with particular types of text strings), then compression followed by base 64 encoding may provide you with a net storage gain. Lots of CPU use though.....
You can't convert arbitrary binary data to a String without breaking it. As you've already stated, if you want to store the data in a clob, you need to base64 encode the data (or use some other valid binary to text encoding).
Have you thought of other solutions, such as using memcached or other caching system? Or do you really want to mess around with compression?
I have saved icon size images in a mysql database in bytes, now, what I need to do is retrieve those file bytes from the database and show those images in a swing application, I have a method which gets the bytes from the database and convert it back to a file but I have to write that file in to the disk
this is my method,
public void downloadFile(int FamerId) throws Exception {
String sql = "SELECT * FROM images WHERE famer_id=?";
Connection con = JDBCConnectionPool.getInstance().checkOut();
PreparedStatement ps = con.prepareStatement(sql);
ps.setInt(1, FamerId);
ResultSet resultSet = ps.executeQuery();
int count = 0;
while (resultSet.next()) {
ByteArrayInputStream bais;
ObjectInputStream inputStream;
bais = new ByteArrayInputStream(resultSet.getBytes("image"));
inputStream = new ObjectInputStream(bais);
SaveFile sf = (SaveFile) inputStream.readObject();
FileOutputStream out = new FileOutputStream("fileLocation/" + resultSet.getString("image_name"));
byte[] bytes = sf.getArray();
int c = 0;
while (c < bytes.length) {
out.write(bytes[c]);
c++;
}
out.close();
inputStream.close();
bais.close();
JDBCConnectionPool.getInstance().checkOut();
}
}
but this method doesn't give what I need, please assist me.
You can read images directly from byte streams with the ImageIO class. Assuming of course that you have previously written the image data in a compatible format. Which is hard to say given the fact that in your code you use an intermediary object input stream when reading your byte data. Here's an example of how you can create an image directly from the database without using intermediary files:
bais = new ByteArrayInputStream(resultSet.getBytes("image"));
final BufferedImage image = ImageIO.read(bais);
// pass the image to your Swing layer to be rendered.
And an example of how you would have written the data to the database, in order to be able to use this code:
final ByteArrayOutputStream baos = new ByteArrayOutputStream(64000);
ImageIO.write(image, "PNG", baos);
final byte[] data = baos.toByteArray();
// write data to database
The answer to your question is its platform dependent. From the docs
A file output stream is an output stream for writing data to a File or
to a FileDescriptor. Whether or not a file is available or may be
created depends upon the underlying platform. Some platforms, in
particular, allow a file to be opened for writing by only one
FileOutputStream (or other file-writing object) at a time. In such
situations the constructors in this class will fail if the file
involved is already open.
FileOutputStream is meant for writing streams of raw bytes such as
image data. For writing streams of characters, consider using
FileWriter.
So if you want to write to a file then file may or may not be created.
If you don't want to create the file and you are just interested in byte[] (content of the file) you can then use solution provided by #Perception or can just pass the inputStream that you have already created.