We Keep Coding

Image byte stream is different when file is read in different ways

Under my Spring 4.3/Maven 3.3 project I have an image file, a PNG file, at:
src/main/resources/images/account.png
I have a util java application file that reads in an image, and it writes it to the database field. The code is as follows:
private static String _accountFilename = "src/main/resources/images/account.png";
private byte[] getByteArrayFromFile(String filename)
{
FileInputStream fileInputStream = null;
File file = new File(filename);
byte[] bFile = new byte[(int) file.length()];
try
{
// convert file into array of bytes
fileInputStream = new FileInputStream(file);
fileInputStream.read(bFile);
fileInputStream.close();
for (int i = 0; i < bFile.length; i++)
{
System.out.print((char) bFile[i]);
}
System.out.println("Done");
}
catch (Exception e)
{
e.printStackTrace();
}
return bFile;
}
public String getImageData(byte[] imageByteArray)
{
Base64.Encoder encoder = Base64.getEncoder();
String base64 = encoder.encodeToString(imageByteArray);
base64 = "data:image/png;base64," + base64;
return base64;
}
The String that comes back from "getImageData" works great. I can put that String in the MySQL database, in a table, and the field is defined as TEXT.
I can pull that base64 encoded data back, and display the image.
Now, If I am calling this code from a Spring Service instead of an application, then the image "src/main/resources/images/account.png" is not found.
After researching on the Net for some time, there are many, many examples of getting a file from "resources" and many of these did not work for me. Since I am in Spring, I tried a few things and finally this worked:
#Value(value = "classpath:images/account.png")
private Resource defaultAccountImage;
private byte[] getByteArrayFromFile(Resource image)
{
InputStream inputStream = null;
byte[] bFile = null;
try
{
bFile = new byte[(int) image.contentLength()];
// convert file into array of bytes
inputStream = image.getInputStream();
inputStream.read(bFile);
inputStream.close();
for (int i = 0; i < bFile.length; i++)
{
System.out.print((char) bFile[i]);
}
System.out.println("Done");
}
catch (Exception e)
{
e.printStackTrace();
}
return bFile;
}
private String getImageData(byte[] imageByteArray)
{
Base64.Encoder encoder = Base64.getEncoder();
String base64 = encoder.encodeToString(imageByteArray);
base64 = "data:image/png;base64," + base64;
return base64;
}
public String getDefaultAccountImage()
{
byte[] accountImage = getByteArrayFromFile(defaultAccountImage);
String fileString = getImageData(accountImage);
return fileString;
}
When I look at the String/Image data between the first way with the standalone java app, and the second way with the #Value and inputstream, there is a definite different in the string data.
part of the string data is similar, but then it drastically changes, and they don't match. As a result the text data for the image from the second method doesn't display as an image.
So, I was hoping I could get this image text data, and it would be the same, but it is not. If I can use my web-service, which calls the business service which calls this ImageUtil code where I use the #Value to get the image resource and it saves the text string correctly, that would be great.
If you have any advice, I would very much appreciate it. Thanks!
UPDATE 1:
This is a multi-maven project:
parent-project
entity
dao
service
ws
When I run my test code within the Service layer, the suggested solution works great! The images are found and the byte string gets loaded as it should be. And then I compiled the code into a jar.
The entity.jar gets created first.
The dao.jar gets created and pulls in the entity.jar.
The service.jar gets created and pulls in the dao.jar. This layer also has the /resources/images/account.png file. But this image is now in the jar.
The ws.WAR file pulls in the service.jar file ...
so the code in the answer does not find the image in the resources.
When I run the tests from the ws layer, I get a FileNotFoundException.
So ... now I am researching on how to get an image from jar ...
Does this change how I should be getting my image byte array?

You can get the file from the Resource, and proceed like the first example which works. Seems redundant, but if you can get the file, then you can test a number of things:
Write the file to disk and check the content
Write the file to disk and compare the sizes, etc.
import java.io.File;
import java.io.FileInputStream;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.core.io.Resource;
#Value(value = "classpath:images/account.png")
private Resource defaultAccountImage;
private byte[] getByteArrayFromFile(Resource image) {
FileInputStream fileInputStream = null;
byte[] bFile = null;
try {
File file = image.getFile();
bFile = new byte[(int) file.length()];
// convert file into array of bytes
fileInputStream = new FileInputStream(file);
fileInputStream.read(bFile);
fileInputStream.close();
for (int i = 0; i < bFile.length; i++) {
System.out.print((char) bFile[i]);
}
System.out.println("Done");
} catch (Exception e) {
e.printStackTrace();
}
return bFile;
}

How to read files with an offset from Hadoop using Java

Problem: I want to read a section of a file from HDFS and return it, such as lines 101-120 from a file of 1000 lines.
I don't want to use seek because I have read that it is expensive.
I have log files which I am using PIG to process down into meaningful sets of data. I've been writing an API to return the data for consumption and display by a front end. Those processed data sets can be large enough that I don't want to read the entire file out of Hadoop in one slurp to save wire time and bandwidth. (Let's say 5 - 10MB)
Currently I am using a BufferedReader to return small summary files which is working fine
ArrayList lines = new ArrayList();
...
for (FileStatus item: items) {
// ignoring files like _SUCCESS
if(item.getPath().getName().startsWith("_")) {
continue;
}
in = fs.open(item.getPath());
BufferedReader br = new BufferedReader(new InputStreamReader(in));
String line;
line = br.readLine();
while (line != null) {
line = line.replaceAll("(\\r|\\n)", "");
lines.add(line.split("\t"));
line = br.readLine();
}
}
I've poked around the interwebs quite a bit as well as Stack but haven't found exactly what I need.
Perhaps this is completely the wrong way to go about doing it and I need a completely separate set of code and different functions to manage this. Open to any suggestions.
Thanks!
As added noted based on research from the below discussions:
How does Hadoop process records records split across block boundaries?
Hadoop FileSplit Reading

I think SEEK is a best option for reading files with huge volumes. It did not cause any problems to me as the volume of data that i was reading was in the range of 2 - 3GB. I did not encounter any issues till today but we did use file splitting to handle the large data set. below is the code which you can use for reading purpose and test the same.
public class HDFSClientTesting {
/**
* #param args
*/
public static void main(String[] args) {
// TODO Auto-generated method stub
try{
//System.loadLibrary("libhadoop.so");
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
conf.addResource(new Path("core-site.xml"));
String Filename = "/dir/00000027";
long ByteOffset = 3185041;
SequenceFile.Reader rdr = new SequenceFile.Reader(fs, new Path(Filename), conf);
Text key = new Text();
Text value = new Text();
rdr.seek(ByteOffset);
rdr.next(key,value);
//Plain text
JSONObject jso = new JSONObject(value.toString());
String content = jso.getString("body");
System.out.println("\n\n\n" + content + "\n\n\n");
File file =new File("test.gz");
file.createNewFile();
}
catch (Exception e ){
throw new RuntimeException(e);
}
finally{
}
}
}

Issue with reading Tiff image metadata with imageIO

I'm writing a program that is supposed to taking in a bunch of tiff's and put them together. I got it to work for most of the image files I read in but a large batch of them throw out an error when I try to read them in.
Here is a snippet of code I have:
int numPages = 0;
inStream = ImageIO.createImageInputStream(imageFile);
reader.setInput(inStream);
while(true){
bufferedImages.add(reader.readAll(numPages, reader.getDefaultReadParam()));
numPages++;
}
Yes I catch the out of bounds exception so we don't have to worry about that. My problem is that I get the following error:
javax.imageio.IIOException: I/O error reading image metadata!
at com.sun.media.imageioimpl.plugins.tiff.TIFFImageReader.readMetadata(TIFFImageReader.java:340)
at com.sun.media.imageioimpl.plugins.tiff.TIFFImageReader.seekToImage(TIFFImageReader.java:310)
at com.sun.media.imageioimpl.plugins.tiff.TIFFImageReader.prepareRead(TIFFImageReader.java:971)
at com.sun.media.imageioimpl.plugins.tiff.TIFFImageReader.read(TIFFImageReader.java:1153)
at javax.imageio.ImageReader.readAll(ImageReader.java:1067)
at sel.image.appender.ImageAppender.mergeImages(ImageAppender.java:59)
at sel.imagenow.processor.AetnaLTCProcessor.processBatch(AetnaLTCProcessor.java:287)
at sel.imagenow.processor.AetnaLTCProcessor.processImpl(AetnaLTCProcessor.java:81)
at sel.processor.AbstractImageNowProcessor.process(AbstractImageNowProcessor.java:49)
at sel.RunConverter.main(RunConverter.java:37)
Caused by: java.io.EOFException
at javax.imageio.stream.ImageInputStreamImpl.readShort(ImageInputStreamImpl.java:229)
at javax.imageio.stream.ImageInputStreamImpl.readUnsignedShort(ImageInputStreamImpl.java:242)
at com.sun.media.imageioimpl.plugins.tiff.TIFFIFD.initialize(TIFFIFD.java:194)
at com.sun.media.imageioimpl.plugins.tiff.TIFFImageMetadata.initializeFromStream(TIFFImageMetadata.java:110)
at com.sun.media.imageioimpl.plugins.tiff.TIFFImageReader.readMetadata(TIFFImageReader.java:336)
... 9 more
I did make sure to add in the right JAI lib and my reader is using the "TIFF" type so the reader (and writer) is correct but for some reason the metadata is wrong. Now I can open and view all these images normally in windows so they really aren't corrupted or anything. Java just doesn't want to read them in right. Since I'm just using the stream meatadata to write them out later I don't care that much about the metadata I just need it to read in the file to the list so I can append it. I did find a writer.replaceImageMetaData method on the writer but the TIFFwriter version of IOWriter doens't have code for it. I'm stuck, anyone anything? Is there maybe a way to read in parts of the metadata to see what is wrong and fix it?

For anyone that would like to know I ended up fixing my own issue. It seems the the image metadata was a bit screwed up. Since I was just doing a plain merge and since I knew each image was one page I was able to use a buffered image to read in the picture then make it a IIOImage with null metadata. I used the stream metadata (which worked) to merge the images. Here is my complete method I use to merge a list of images:
public static File mergeImages(List<File> files, String argID, String fileType, String compressionType) throws Exception{
//find the temp location of the image
String location = ConfigManager.getInstance().getTempFileDirectory();
logger_.debug("image file type [" + fileType + "]");
ImageReader reader = ImageIO.getImageReadersByFormatName(fileType).next();
ImageWriter writer = ImageIO.getImageWritersByFormatName(fileType).next();
//set up the new image name
String filePath = location + "\\" + argID +"." + fileType;
//keeps track of the images we copied from
StringBuilder builder = new StringBuilder();
List<IIOImage> bufferedImages = new ArrayList<IIOImage>();
IIOMetadata metaData = null;
for (File imageFile:files) {
//get the name for logging later
builder.append(imageFile.getCanonicalPath()).append("\n");
if (metaData == null){
reader.setInput(ImageIO.createImageInputStream(imageFile));
metaData = reader.getStreamMetadata();
}
BufferedImage image = ImageIO.read(imageFile);
bufferedImages.add(new IIOImage(image, null, null));
}
ImageWriteParam params = writer.getDefaultWriteParam();
if (compressionType != null){
params.setCompressionMode(ImageWriteParam.MODE_EXPLICIT);
params.setCompressionType(compressionType);
}
ImageOutputStream outStream = null;
try{
outStream = ImageIO.createImageOutputStream(new File(filePath));
int numPages = 0;
writer.setOutput(outStream);
for(IIOImage image:bufferedImages){
if (numPages == 0){
writer.write(metaData, image, params);
}
else{
writer.writeInsert(numPages, image, params);
}
numPages++;
}
}
finally{
if (outStream != null){
outStream.close();
}
}
//set up the file for us to use later
File mergedFile = new File(filePath);
logger_.info("Merged image into [" + filePath + "]");
logger_.debug("Merged images [\n" + builder.toString() + "] into --> " + filePath);
return mergedFile;
}
I hope this help someone else because I know there isn't much on this issue that I could find.

Update database table without uploading a file while using a MultiPart Form - JavaEE, Servlet

I have a servlet which is responsible for enabling a user to update a reports table and upload a report at the same time. I have written code that enables a user upload a document and also be able to update the table with other details e.g date submitted etc.
However not all the times will a user have to upload a document. in this case it should be possible for a user to still edit a report's details and come back later to upload the file. i.e the user can submit the form without selecting a file and it still updates the table.
This part is what is not working. If a user selects a file and makes some changes. The code works. If a user doesn't select a file and tries to submit the form, it redirects to my servlet but it is blank. no stacktrace. No error is thrown.
Below is part of the code I have in my servlet:
if(param.equals("updateschedule"))
{
String[] allowedextensions = {"pdf","xlsx","xls","doc","docx","jpeg","jpg","msg"};
final String path = request.getParameter("uploadlocation_hidden");
final Part filepart=request.getPart("uploadreport_file");
int repid = Integer.parseInt(request.getParameter("repid_hidden"));
int reptype = Integer.parseInt(request.getParameter("reporttype_select"));
String webdocpath = request.getParameter("doclocation_hidden");
String subperiod = request.getParameter("submitperiod_select");
String duedate = request.getParameter("reportduedate_textfield");
String repname = request.getParameter("reportname_textfield");
String repdesc = request.getParameter("reportdesc_textarea");
String repinstr = request.getParameter("reportinst_textarea");
int repsubmitted = Integer.parseInt(request.getParameter("repsubmitted_select"));
String datesubmitted = request.getParameter("reportsubmitdate_textfield");
final String filename = getFileName(filepart);
OutputStream out = null;
InputStream filecontent=null;
String extension = filename.substring(filename.lastIndexOf(".") + 1, filename.length());
if(Arrays.asList(allowedextensions).contains(extension))
{
try
{
out=new FileOutputStream(new File(path+File.separator+filename));
filecontent = filepart.getInputStream();
int read=0;
final byte[] bytes = new byte[1024];
while((read=filecontent.read(bytes))!=-1)
{
out.write(bytes,0,read);
}
String fulldocpath = webdocpath+"/"+filename;
boolean succ = icreditdao.updatereportschedule(repid, reptype, subperiod, repname, repsubmitted,datesubmitted, duedate,fulldocpath, repdesc, repinstr);
if(succ==true)
{
response.sendRedirect("/webapp/Pages/Secured/ReportingSchedule.jsp?msg=Report Schedule updated successfully");
}
}
catch(Exception ex)
{
throw new ServletException(ex);
}
}
I'm still teaching myself javaee. Any help will be appreciated. Also open to other alternatives. I have thought of using jquery to detect if a file has been selected then use a different set of code. e.g
if(param.equals("updatewithnofileselected"))
{//update code here}
but I think there must be a better solution. Using jdk6, servlet3.0.

try this one.
MultipartParser parser = new MultipartParser(request, 500000000, false, false, "UTF-8");
Part part;
while ((part = parser.readNextPart()) != null) {
if(part.isParam()){
if(part.isFile()){
if(part.getName().equals("updatewithnofileselected")){
//update code here.
} else if(part.getName().equals("updateschedule")) {
//updateschedule
}
}
}
}
I used this one when I am using Multipart-form and it's working fine.

Open Microsoft Word in Java

I'm trying to open MS Word 2003 document in java, search for a specified String and replace it with a new String. I use APACHE POI to do that. My code is like the following one:
public void searchAndReplace(String inputFilename, String outputFilename,
HashMap<String, String> replacements) {
File outputFile = null;
File inputFile = null;
FileInputStream fileIStream = null;
FileOutputStream fileOStream = null;
BufferedInputStream bufIStream = null;
BufferedOutputStream bufOStream = null;
POIFSFileSystem fileSystem = null;
HWPFDocument document = null;
Range docRange = null;
Paragraph paragraph = null;
CharacterRun charRun = null;
Set<String> keySet = null;
Iterator<String> keySetIterator = null;
int numParagraphs = 0;
int numCharRuns = 0;
String text = null;
String key = null;
String value = null;
try {
// Create an instance of the POIFSFileSystem class and
// attach it to the Word document using an InputStream.
inputFile = new File(inputFilename);
fileIStream = new FileInputStream(inputFile);
bufIStream = new BufferedInputStream(fileIStream);
fileSystem = new POIFSFileSystem(bufIStream);
document = new HWPFDocument(fileSystem);
docRange = document.getRange();
numParagraphs = docRange.numParagraphs();
keySet = replacements.keySet();
for (int i = 0; i < numParagraphs; i++) {
paragraph = docRange.getParagraph(i);
text = paragraph.text();
numCharRuns = paragraph.numCharacterRuns();
for (int j = 0; j < numCharRuns; j++) {
charRun = paragraph.getCharacterRun(j);
text = charRun.text();
System.out.println("Character Run text: " + text);
keySetIterator = keySet.iterator();
while (keySetIterator.hasNext()) {
key = keySetIterator.next();
if (text.contains(key)) {
value = replacements.get(key);
charRun.replaceText(key, value);
docRange = document.getRange();
paragraph = docRange.getParagraph(i);
charRun = paragraph.getCharacterRun(j);
text = charRun.text();
}
}
}
}
bufIStream.close();
bufIStream = null;
outputFile = new File(outputFilename);
fileOStream = new FileOutputStream(outputFile);
bufOStream = new BufferedOutputStream(fileOStream);
document.write(bufOStream);
} catch (Exception ex) {
System.out.println("Caught an: " + ex.getClass().getName());
System.out.println("Message: " + ex.getMessage());
System.out.println("Stacktrace follows.............");
ex.printStackTrace(System.out);
}
}
I call this function with following arguments:
HashMap<String, String> replacements = new HashMap<String, String>();
replacements.put("AAA", "BBB");
searchAndReplace("C:/Test.doc", "C:/Test1.doc", replacements);
When the Test.doc file contains a simple line like this : "AAA EEE", it works successfully, but when i use a complicated file it will read the content successfully and generate the Test1.doc file but when I try to open it, it will give me the following error:
Word unable to read this document. It may be corrupt.
Try one or more of the following:
* Open and repair the file.
* Open the file with Text Recovery converter.
(C:\Test1.doc)
Please tell me what to do, because I'm a beginner in POI and I have not found a good tutorial for it.

First of all you should be closing your document.
Besides that, what I suggest doing is resaving your original Word document as a Word XML document, then changing the extension manually from .XML to .doc . Then look at the XML of the actual document you're working with and trace the content to make sure you're not accidentally editing hexadecimal values (AAA and EEE could be hex values in other fields).
Without seeing the actual Word document it's hard to say what's going on.
There is not much documentation about POI at all, especially for Word document unfortunately.

I don't know : is its OK to answer myself, but Just to share the knowledge, I'll answer myself.
After navigating the web, the final solution i found is :
The Library called docx4j is very good for dealing with MS docx file, although its documentation is not enough till now and its forum is still in a beginning steps, but overall it help me to do what i need..
Thanks 4 all who help me..

You could try OpenOffice API, but there arent many resources out there to tell you how to use it.

You can also try this one: http://www.dancrintea.ro/doc-to-pdf/

Looks like this could be the issue.