We Keep Coding

multi-pag tif to several jpeg files (java)

Hello I was able to convert a tif file to jpeg with the following code that I got from
https://stackoverflow.com/questions/15429011/how-to-convert-tiff-to-jpeg-png-in-java#=
String inPath = "./tifTest/113873996.002.tif";
String otPath = "./tifTest/113873996.002-0.jpeg";
BufferedInputStream input = null;
BufferedOutputStream output = null;
try {
input = new BufferedInputStream(new FileInputStream(inPath), DEFAULT_BUFFER_SIZE);
output = new BufferedOutputStream(new FileOutputStream(otPath), DEFAULT_BUFFER_SIZE);
byte[] buffer = new byte[DEFAULT_BUFFER_SIZE];
int length;
while ((length = input.read(buffer)) > 0) {
output.write(buffer, 0, length);
}
} catch (FileNotFoundException ex) {
Logger.getLogger(TifToJpeg.class.getName()).log(Level.SEVERE, null, ex);
} catch (IOException ex) {
Logger.getLogger(TifToJpeg.class.getName()).log(Level.SEVERE, null, ex);
} finally {
try {
output.flush();
output.close();
input.close();
} catch (IOException e) {
e.printStackTrace();
}
}
This only works with one-page tif file, and when I use it with a multi-page tif, it only saves the first page.
How can I modified this to save a mymultipagetif.tif into:
mymultipagetif-0.jpeg
mymultipagetif-1.jpeg
mymultipagetif-2.jpeg
Thanks!

This will takes a multi-page TIFF file (in the SeekableStream), extract the pages in the pages array ("1", "3", "4" for example) and write them into a single multi-page tiff into the file outTIffFileName. Modify as desired.
private String _ExtractListOfPages (SeekableStream ss, String outTiffFileName, String[] pages){
// pageNums is a String array of 0-based page numbers.
try {
TIFFDirectory td = new TIFFDirectory(ss, 0);
if (debugOn) {
System.out.println("Directory has " + Integer.toString(td.getNumEntries()) + " entries");
System.out.println("Getting TIFFFields");
System.out.println("X resolution = " + Float.toString(td.getFieldAsFloat(TIFFImageDecoder.TIFF_X_RESOLUTION)));
System.out.println("Y resolution = " + Float.toString(td.getFieldAsFloat(TIFFImageDecoder.TIFF_Y_RESOLUTION)));
System.out.println("Resolution unit = " + Long.toString(td.getFieldAsLong(TIFFImageDecoder.TIFF_RESOLUTION_UNIT)));
}
ImageDecoder decodedImage = ImageCodec.createImageDecoder("tiff", ss, null);
int count = decodedImage.getNumPages();
if (debugOn) { System.out.println("Input image has " + count + " page(s)"); }
TIFFEncodeParam param = new TIFFEncodeParam();
TIFFField tf = td.getField(259); // Compression as specified in the input file
param.setCompression(tf.getAsInt(0)); // Set the compression of the output to be the same.
param.setLittleEndian(false); // Intel
param.setExtraFields(td.getFields());
FileOutputStream fOut = new FileOutputStream(outTiffFileName);
Vector<RenderedImage> vector = new Vector<RenderedImage>();
RenderedImage page0 = decodedImage.decodeAsRenderedImage(Integer.parseInt(pages[0]));
BufferedImage img0 = new BufferedImage(page0.getColorModel(), (WritableRaster)page0.getData(), false, null);
int pgNum;
// Adding the extra pages starts with the second one on the list.
for (int i = 1; i < pages.length; i++ ) {
pgNum = Integer.parseInt(pages[i]);
if (debugOn) { System.out.println ("Page number " + pgNum); }
RenderedImage page = decodedImage.decodeAsRenderedImage(pgNum);
if (debugOn) { System.out.println ("Page is " + Integer.toString(page.getWidth()) + " pixels wide and "+ Integer.toString(page.getHeight()) + " pixels high."); }
if (debugOn) { System.out.println("Adding page " + pages[i] + " to vector"); }
vector.add(page);
}
param.setExtraImages(vector.iterator());
ImageEncoder encoder = ImageCodec.createImageEncoder("tiff", fOut, param);
if (debugOn) { System.out.println("Encoding page " + pages[0]); }
encoder.encode(decodedImage.decodeAsRenderedImage(Integer.parseInt(pages[0])));
fOut.close();
} catch (Exception e) {
System.out.println(e.toString());
return("Not OK " + e.getMessage());
}
return ("OK");
}

Error in my mp3 splitter program?

So, I made a program that splits a .mp3 file in Java. Basically, it works fine on some files but on some, the first split file encounters an error after playing some part. The other files work completely fine though.
I think it has something to do with how a file cannot be a multiple of the size of my array and there should be some mod value left. Can anybody please identify the error in this code and correct it?
(here, splitval = no. of splits to be made, filename1= the selected file)
int splitsize=filesize/splitval;
String filecalled;
try
{
byte []b=new byte[splitsize];
FileInputStream fis = new FileInputStream(filename1);
name1=filename2.replaceAll(".mp3", "");
for(int j=1;j<=splitval;j++)
{
filecalled=name1+"_split_"+j+".mp3";
FileOutputStream fos = new FileOutputStream(filecalled);
int i=fis.read(b);
fos.write(b, 0, i);
//System.out.println("no catch");
}
JOptionPane.showMessageDialog(this, "split process successful");
}
catch(IOException e)
{
System.out.println(e.getMessage());
}
Thanks in advance!
EDIT:
I edited the code as suggested, ran it. Here:
C:\Users\dell5050\Desktop\Julien.mp3 5383930 bytes
C:\Users\dell5050\Desktop\ Julien_split_1.mp3 1345984 bytes
C:\Users\dell5050\Desktop\ Julien_split_2.mp3 1345984 bytes
C:\Users\dell5050\Desktop\ Julien_split_3.mp3 1345984 bytes
C:\Users\dell5050\Desktop\ Julien_split_4.mp3 1345978 bytes
There is change in the last few bytes which means that the filesize%splitval is solved.. but still the first file in this.. containing '_split_1' has error while playing some of the last part.
The second file containing '_split_2' starts exactly where the first ended. So the split process is correct. Then, what exactly is the extra empty in the end of the first file?
Also, I noticed that the artwork and info of the original file carries over into the first file ONLY. No other files. Does it have something to do with that? Same thing doesnt happen in some other mp3 files.
CODE:
FileInputStream fis;
FileOutputStream fos;
int splitsize = (int)(filesize / splitval) + (int)(filesize % splitval);
byte[] b = new byte[splitsize];
System.out.println(filename1 + " " + filesize + " bytes");
try
{
fis = new FileInputStream(file);
name1 = filename2.replaceAll(".mp3", "");
for (int j = 1; j <= splitval; j++)
{
String filecalled = name1 + "_split_" + j + ".mp3";
fos = new FileOutputStream(filecalled);
int i = fis.read(b);
fos.write(b, 0, i);
fos.close();
System.out.println(filecalled + " " + i + " bytes");
}
}
catch(IOException ie)
{
System.out.println(ie.getMessage());
}

I doubt you could split a mp3 file just by copying n-bytes to a file and go to the next. Mp3 has a specific format and you'll probably need a library to handle this format.
EDIT regarding the size of the part files being all equal:
You are not writing all the bytes of the file to the split files. If you sum the sizes of all split files and compare it to the size of the original file you'll find out that your missing some bytes. This is because your loop runs from 1 to splitval and always writes the exact number of bytes to each part file i.e. splitsize. So the number of bytes your are missing is filesize % splitval.
To resolve this problem simply add filesize % splitval to splitsize. This way you'll not be missing any bytes. The files from 1 to splitval - 1 will have the same size, the last file will be smaller.
Here is a corrected version of your code with some additions to merge the split files in order to perform an assertion using SHA1-checksum.
Disclaimer - The output files are not expected to be proper mp3 files
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import junit.framework.Assert;
import org.junit.Test;
public class SplitFile {
#Test
public void splitFile() throws IOException, NoSuchAlgorithmException {
String filename1 = "mp3/Innocence_-_Nero.mp3";
File file = new File(filename1);
FileInputStream fis = null;
FileOutputStream fos = null;
long filesize = file.length();
long filesizeActual = 0L;
int splitval = 5;
int splitsize = (int)(filesize / splitval) + (int)(filesize % splitval);
byte[] b = new byte[splitsize];
System.out.println(filename1 + " " + filesize + " bytes");
try {
fis = new FileInputStream(file);
String name1 = filename1.replaceAll(".mp3", "");
String mergeFile = name1 + "_merge.mp3";
for (int j = 1; j <= splitval; j++) {
String filecalled = name1 + "_split_" + j + ".mp3";
fos = new FileOutputStream(filecalled);
int i = fis.read(b);
fos.write(b, 0, i);
fos.close();
fos = null;
System.out.println(filecalled + " " + i + " bytes");
filesizeActual += i;
}
Assert.assertEquals(filesize, filesizeActual);
mergeFileParts(filename1, splitval);
check(filename1, mergeFile);
} finally {
if(fis != null) {
fis.close();
}
if(fos != null) {
fos.close();
}
}
}
private void mergeFileParts(String filename1, int splitval) throws IOException {
FileInputStream fis = null;
FileOutputStream fos = null;
try {
String name1 = filename1.replaceAll(".mp3", "");
String mergeFile = name1 + "_merge.mp3";
fos = new FileOutputStream(mergeFile);
for (int j = 1; j <= splitval; j++) {
String filecalled = name1 + "_split_" + j + ".mp3";
File partFile = new File(filecalled);
fis = new FileInputStream(partFile);
int partFilesize = (int) partFile.length();
byte[] b = new byte[partFilesize];
int i = fis.read(b, 0, partFilesize);
fos.write(b, 0, i);
fis.close();
fis = null;
}
} finally {
if(fis != null) {
fis.close();
}
if(fos != null) {
fos.close();
}
}
}
private void check(String expectedPath, String actualPath) throws IOException, NoSuchAlgorithmException {
System.out.println("check...");
FileInputStream fis = null;
try {
File expectedFile = new File(expectedPath);
long expectedSize = expectedFile.length();
File actualFile = new File(actualPath);
long actualSize = actualFile.length();
System.out.println("exp=" + expectedSize);
System.out.println("act=" + actualSize);
Assert.assertEquals(expectedSize, actualSize);
fis = new FileInputStream(expectedFile);
String expected = makeMessageDigest(fis);
fis.close();
fis = null;
fis = new FileInputStream(actualFile);
String actual = makeMessageDigest(fis);
fis.close();
fis = null;
System.out.println("exp=" + expected);
System.out.println("act=" + actual);
Assert.assertEquals(expected, actual);
} finally {
if(fis != null) {
fis.close();
}
}
}
public String makeMessageDigest(InputStream is) throws NoSuchAlgorithmException, IOException {
byte[] data = new byte[1024];
MessageDigest md = MessageDigest.getInstance("SHA1");
int bytesRead = 0;
while(-1 != (bytesRead = is.read(data, 0, 1024))) {
md.update(data, 0, bytesRead);
}
return toHexString(md.digest());
}
private String toHexString(byte[] digest) {
StringBuffer sha1HexString = new StringBuffer();
for(int i = 0; i < digest.length; i++) {
sha1HexString.append(String.format("%1$02x", Byte.valueOf(digest[i])));
}
return sha1HexString.toString();
}
}
Output (for my test file)
mp3/Innocence_-_Nero.mp3 5048528 bytes
mp3/Innocence_-_Nero_split_1.mp3 1009708 bytes
mp3/Innocence_-_Nero_split_2.mp3 1009708 bytes
mp3/Innocence_-_Nero_split_3.mp3 1009708 bytes
mp3/Innocence_-_Nero_split_4.mp3 1009708 bytes
mp3/Innocence_-_Nero_split_5.mp3 1009696 bytes
check...
exp=5048528
act=5048528
exp=e81cf2dc65ab84e3df328e52d63a55301232b917
act=e81cf2dc65ab84e3df328e52d63a55301232b917

How to write files to an SD Card from the internet?

An example would be a simple image.
I have tried so many things and it just refuses to work despite making a whole lot of sense.
What I've done so far is I'm able to grab 25 pictures and add them to
/sdcard/app name/sub/dir/filename.jpg
They all appear there according to the DDMS but they always have a filesize of 0.
I'm guessing it's probably because of my input stream?
Here's my function that handles the downloading and saving.
public void DownloadPages()
{
for (int fileC = 0; fileC < pageAmount; fileC++)
{
URL url;
String path = "/sdcard/Appname/sub/dir/";
File file = new File(path, fileC + ".jpg");
int size=0;
byte[] buffer=null;
try{
url = new URL("http://images.bluegartr.com/bucket/gallery/56ca6f9f2ef43ab7349c0e6511edb6d6.png");
InputStream in = url.openStream();
size = in.available();
buffer = new byte[size];
in.read(buffer);
in.close();
}catch(Exception e){
}
if (!new File(path).exists())
new File(path).mkdirs();
FileOutputStream out;
try{
out = new FileOutputStream(file);
out.write(buffer);
out.flush();
out.close();
}catch(Exception e){
}
}
}
It just keeps giving me 25 files in that directory but all of their file sizes are zero. I have no idea why. This is practically the same code I've used in a java program.
PS...
If you're gonna give me a solution... I've already tried code like this. It doesn't work.
try{
url = new URL(urlString);
in = new BufferedInputStream(url.openStream());
fout = new FileOutputStream(filename);
byte data[] = new byte[1024];
int count;
System.out.println("Now downloading File: " + filename.substring(0, filename.lastIndexOf(".")));
while ((count = in.read(data, 0, 1024)) != -1){
fout.write(data, 0, count);
}
}finally{
System.out.println("Download complete.");
if (in != null)
in.close();
if (fout != null)
fout.close();
}
}
Here's an image of what my directories look like
http://oi48.tinypic.com/2cpcprm.jpg

A bit change to your second option, try it as following way,
byte data[] = new byte[1024];
long total = 0;
int count;
while ( ( count = input.read(data)) != -1 )
{
total += count;
output.write( data,0,count );
}
This one is different in while statement while ((count = in.read(data, 0, 1024)) != -1)

Using Guava something like this should work:
String fileUrl = "xxx";
File file = null;
InputStream in;
FileOutputStream out;
try {
Uri url = new URI(fileUrl);
in = url.openStream();
out = new FileOutputStream(file)
ByteStreams.copy(in, out);
}
catch (IOException e) {
System.out.println(e.toString());
}
finally {
in.close();
out.flush();
out.close();
}

Android:"Unexpected end of stream" exception downloading large files

I am building an Android Application and I need to download a file from a url, which is 33 MB large.
Here the download task:
try {
int MAX_BUFFER_SIZE = 4096;
URL mUrl = new URL(params[0]);
HttpURLConnection connection = (HttpURLConnection) mUrl.openConnection();
connection.setRequestMethod("GET");
long length = connection.getContentLength(), downloaded = 0;
int read;
byte [] buffer = new byte[(((int)length) > MAX_BUFFER_SIZE) ? MAX_BUFFER_SIZE : (int)length];
String filename = getFilename(mUrl);
File file = new File (SDCARD_ROOT);
if (!file.exists() || !file.isDirectory()){
file.mkdir();
}
this.filename = filename;
file = new File (SDCARD_ROOT + this.filename);
FileOutputStream fos = new FileOutputStream (file);
//Start downloading
InputStream stream = connection.getInputStream();
while ((read=stream.read(buffer)) > -1){
fos.write(buffer, 0, read);
downloaded += read;
publishProgress((int) ((float) downloaded/length * 100));
}
fos.close();
return 1;
} catch (Exception e){
Log.e("REV-PARTS", "Revolver parts error in DownloadTask: " + e.getMessage());
return 2;
}
It works right with small files (1-15 MB), but it will return a "unexpected end of stream" exception with large files.

Setting a chunk size seemed to work for me.
connection.setChunkedStreamingMode(1048576);

For large files you need to set the connection time out manually by using the following code.
I have set the time out to 3 minutes
connection.setConnectTimeout(180000);
connection.setReadTimeout(180000);

While you catch the exception, I try the method downContinue(). I can show my code:
private void downloadApk() {
thread1 = new Thread() {
public void run() {
File oFile = null;
try {
URL url = new URL(PQGLApplication.resrootURL + "apk/PQGLMap.apk");
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
ReadableByteChannel channel =
Channels.newChannel(urlConnection.getInputStream());
oFile =
new File(Environment.getExternalStorageDirectory().getAbsolutePath()
+ "/" + "hy_ht_new/" + "test2" + ".apk");
oFile.setWritable(true);
oFile.setReadable(true);
if (oFile.exists()) {
oFile.delete();
}
FileOutputStream fos = new FileOutputStream(oFile);
fileSize = urlConnection.getContentLength();
ByteBuffer buffer = ByteBuffer.allocate(1024);
int noOfBytes = 0;
byte[] data = null;
sendApkMessage(0, 0);
while ((noOfBytes = channel.read(buffer)) > 0) {
data = new byte[noOfBytes];
System.arraycopy(buffer.array(), 0, data, 0, noOfBytes);
buffer.clear();
fos.write(data, 0, noOfBytes);
downLoadFileSize += noOfBytes;
sendApkMessage(1, downLoadFileSize);
}
fos.flush();
fos.close();
channel.close();
sendApkMessage(2, oFile.getAbsolutePath());
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
downContinue();
}
};
};
thread1.start();
}
private void downContinue() {
continueTime++;
try {
if (continueTime == 3) {
continueTime = 0;
sendApkMessage(4, 0);
Log.e("what is the continuetime", "continueTime" + continueTime);
} else {
URL url = new URL(PQGLApplication.resrootURL + "apk/PQGLMap.apk");
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
File oFile =
new File(Environment.getExternalStorageDirectory().getAbsolutePath() + "/"
+ "hy_ht_new/" + "test2" + ".apk");
RandomAccessFile oSavedFile = new RandomAccessFile(oFile, "rw");
FileOutputStream fos = new FileOutputStream(oFile);
ReadableByteChannel channel = Channels.newChannel(urlConnection.getInputStream());
// oSavedFile.seek(nPos);
ByteBuffer buffer = ByteBuffer.allocate(1024);
byte[] data = null;
int temp = 0;
sendApkMessage(3, oFile.getAbsolutePath());
while ((temp = channel.read(buffer)) > 0) {
data = new byte[temp];
System.arraycopy(buffer.array(), 0, data, 0, temp);
buffer.clear();
fos.write(data, 0, temp);
}
fos.flush();
fos.close();
oSavedFile.close();
sendApkMessage(2, oFile.getAbsolutePath());
continueTime = 0;
}
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
Log.e("what is the exception", e.toString() + continueTime);
downContinue();
}
}
This downContinue method is used to solve this problem. At least, the file is downloaded successfully!

Uploading large gzipped data files to HDFS

I have a use case where I want to upload big gzipped text data files (~ 60 GB) on HDFS.
My code below is taking about 2 hours to upload these files in chunks of 500 MB. Following is the pseudo code. I was chekcing if somebody could help me reduce this time:
i) int fileFetchBuffer = 500000000;
System.out.println("file fetch buffer is: " + fileFetchBuffer);
int offset = 0;
int bytesRead = -1;
try {
fileStream = new FileInputStream (file);
if (fileName.endsWith(".gz")) {
stream = new GZIPInputStream(fileStream);
BufferedReader reader = new BufferedReader(new InputStreamReader(stream));
String[] fileN = fileName.split("\\.");
System.out.println("fil 0 : " + fileN[0]);
System.out.println("fil 1 : " + fileN[1]);
//logger.info("First line is: " + streamBuff.readLine());
byte[] buffer = new byte[fileFetchBuffer];
FileSystem fs = FileSystem.get(conf);
int charsLeft = fileFetchBuffer;
while (true) {
charsLeft = fileFetchBuffer;
logger.info("charsLeft outside while: " + charsLeft);
FSDataOutputStream dos = null;
while (charsLeft != 0) {
bytesRead = stream.read(buffer, 0, charsLeft);
if (bytesRead < 0) {
dos.flush();
dos.close();
break;
}
offset = offset + bytesRead;
charsLeft = charsLeft - bytesRead;
logger.info("offset in record: " + offset);
logger.info("charsLeft: " + charsLeft);
logger.info("bytesRead in record: " + bytesRead);
//prettyPrintHex(buffer);
String outFileStr = Utils.getOutputFileName(
stagingDir,
fileN[0],
outFileNum);
if (dos == null) {
Path outFile = new Path(outFileStr);
if (fs.exists(outFile)) {
fs.delete(outFile, false);
}
dos = fs.create(outFile);
}
dos.write(buffer, 0, bytesRead);
}
logger.info("done writing: " + outFileNum);
dos.flush();
dos.close();
if (bytesRead < 0) {
dos.flush();
dos.close();
break;
}
outFileNum++;
} // end of if
} else {
// Assume uncompressed file
stream = fileStream;
}
} catch(FileNotFoundException e) {
logger.error("File not found" + e);
}

You should consider using the super package IO from Apache.
It has a method
IOUtils.copy( InputStream, OutputStream )
that would tremendously reduce time needed to copy your files.

I tried with buffered input stream and saw no real difference.
I suppose a file channel implementation could be even more efficient. Tell me if it's not fast enough.
package toto;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
public class Slicer {
private static final int BUFFER_SIZE = 50000;
public static void main(String[] args) {
try
{
slice( args[ 0 ], args[ 1 ], Long.parseLong( args[2]) );
}//try
catch (IOException e)
{
e.printStackTrace();
}//catch
catch( Exception ex )
{
ex.printStackTrace();
System.out.println( "Usage : toto.Slicer <big file> <chunk name radix > <chunks size>" );
}//catch
}//met
/**
* Slices a huge files in chunks.
* #param inputFileName the big file to slice.
* #param outputFileRadix the base name of slices generated by the slicer. All slices will then be numbered outputFileRadix0,outputFileRadix1,outputFileRadix2...
* #param chunkSize the size of chunks in bytes
* #return the number of slices.
*/
public static int slice( String inputFileName, String outputFileRadix, long chunkSize ) throws IOException
{
//I would had some code to pretty print the output file names
//I mean adding a couple of 0 before chunkNumber in output file name
//so that they all have same number of chars
//use java.io.File for that, estimate number of chunks, take power of 10, got number of leading 0s
//just to get some stats
long timeStart = System.currentTimeMillis();
long timeStartSlice = timeStart;
long timeEnd = 0;
//io streams and chunk counter
int chunkNumber = 0;
FileInputStream fis = null;
FileOutputStream fos = null;
try
{
//open files
fis = new FileInputStream( inputFileName );
fos = new FileOutputStream( outputFileRadix + chunkNumber );
//declare state variables
boolean finished = false;
byte[] buffer = new byte[ BUFFER_SIZE ];
int bytesRead = 0;
long bytesInChunk = 0;
while( !finished )
{
//System.out.println( "bytes to read " +(int)Math.min( BUFFER_SIZE, chunkSize - bytesInChunk ) );
bytesRead = fis.read( buffer,0, (int)Math.min( BUFFER_SIZE, chunkSize - bytesInChunk ) );
if( bytesRead == -1 )
finished = true;
else
{
fos.write( buffer, 0, bytesRead );
bytesInChunk += bytesRead;
if( bytesInChunk == chunkSize )
{
if( fos != null )
{
fos.close();
timeEnd = System.currentTimeMillis();
System.out.println( "Chunk "+chunkNumber + " has been generated in "+ (timeEnd - timeStartSlice) +" ms");
chunkNumber ++;
bytesInChunk = 0;
timeStartSlice = timeEnd;
System.out.println( "Creating slice number " + chunkNumber );
fos = new FileOutputStream( outputFileRadix + chunkNumber );
}//if
}//if
}//else
}//while
}
catch (Exception e)
{
System.out.println( "A problem occured during slicing : " );
e.printStackTrace();
}//catch
finally
{
//whatever happens close all files
System.out.println( "Closing all files.");
if( fis != null )
fis.close();
if( fos != null )
fos.close();
}//fin
timeEnd = System.currentTimeMillis();
System.out.println( "Total slicing time : " + (timeEnd - timeStart) +" ms" );
System.out.println( "Total number of slices "+ (chunkNumber +1) );
return chunkNumber+1;
}//met
}//class
Greetings,
Stéphane