I am implementing a Direct Connect client. I am using the NMDC protocol. I can connect to a hub and other connected clients. I am trying to retrieve the file list from each client, I understand that in order to do that one must download the file files.xml.bz2 from the other client. The protocol to download a file is as follows:
-> $ADCGET file <filename> <params>|
<- $ADCSND file <fileName> <params>|
<- (*** binary data is now transfered from client B to client A ***)
I am trying to create a file named files.xml.bz2 using the binary data received. Here's my code:
//filesize is provided through the $ADCSND response from other client
byte[] data = new byte[filesize];
/*
Reading binary data from socket inputstream
*/
int read = 0;
for (int i=0; read<filesize;){
int available = in2.available();
int leftspace = filesize-read;
if (available>0){
in2.read(data, read, available>leftspace? leftspace:available);
++i;
}
read += (available>leftspace? leftspace:available)+1;
}
/*
writing the bytes to an actual file
*/
ByteArrayInputStream f = new ByteArrayInputStream(data);
FileOutputStream file = new FileOutputStream("files.xml.bz2");
file.write(data);
file.close();
The file is created, however, the contents (files.xml) are not readable. Opening it in firefox gives:
XML Parsing Error: not well-formed
Viewing the contents in the terminal only reads binary data. What am i doing wrong?
EDIT
I also tried Decompressing the file using the bz2 libray from Apache Ant.
ByteArrayInputStream f = new ByteArrayInputStream(data);
BZip2CompressorInputStream bzstream = new BZip2CompressorInputStream(f);
FileOutputStream xmlFile = new FileOutputStream("files.xml");
byte[] bytes = new byte[1024];
while((bzstream.read(bytes))!=-1){
xmlFile.write(bytes);
}
xmlFile.close();
bzstream.close();
I get an error, here's the stacktrace:
java.io.IOException: Stream is not in the BZip2 format
at org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream.init(BZip2CompressorInputStream.java:240)
at org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream.<init>(BZip2CompressorInputStream.java:132)
at org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream.<init>(BZip2CompressorInputStream.java:109)
at control.Controller$1.run(Controller.java:196)
Usual, typical misuse of available(). All you need to copy a stream in Java is as follows:
while ((count = in.read(buffer)) >= 0)
{
out.write(buffer, 0, count);
}
Use this with any size buffer greater than zero, but preferably several kilobytes. You don't need a new buffer per iteration, and you don't need to know how much data is available to read without blocking, as you have to block, otherwise you're just smoking the CPU. But you do need to know how much data was actually read per iteration, and this is the first place where your code falls down.
The error java.io.IOException: Stream is not in the BZip2 format is generated by the constructor of class BZip2CompressorInputStream. I decided to scan the bytes, looking for the magic number to make sure that the file was bz2 format, it turns out that Java was right -- it wasnt in bz2 format.
Upon examining the source code of Jucy, I saw that the reason for this was a slight error in the command I sent to the other client, in essence, this error was caused a mistake in my protocol implementation. The solution was:
Replace:
$ADCGET file files.xml.bz2 0 -1 ZL1|
With:
$ADCGET file files.xml.bz2 0 -1|
ZL1 specifies compression of the files being sent (Not necessary).
Related
I need to download .png files from a FTP server in Java.
I have 3 different servers, each one contain a folder with exactly the same .png files.
On server 1 :
If I download my .png file of 4686 bytes stored on this server with FTPClient (apache.commons.net.ftp), I get a 4706 bytes .png file, and I can't open it.
If I download it with Total Commander, I get a 4686 bytes .png file, and I can open it.
On server 2 and 3 :
With FTPClient and Total Commander, I get in both case a 4686 bytes file and I can open it without problem.
My code :
FTPClient ftpClient = new FTPClient();
ftpClient.connect("...", PORT);
ftpClient.login("...", "...");
ftpClient.enterLocalPassiveMode();
FTPFile[] imageFiles = ftpClient.listFiles(distantPathForImages);
for (FTPFile imageFile : imageFiles) {
InputStream inputStream = ftpClient.retrieveFileStream(distantPathForImages + imageFile.getName());
OutputStream outputStream = new BufferedOutputStream(new FileOutputStream(new File(PATHDESTCSS + imageFile.getName())));
byte[] bytesArray = new byte[65536];
int bytesRead;
while ((bytesRead = inputStream.read(bytesArray)) != -1) {
outputStream.write(bytesArray, 0, bytesRead);
}
outputStream.close();
inputStream.close();
ftpClient.completePendingCommand();
}
Why does my file have these "extra bytes" only when I download it from the server 1, and how can I fix this?
FTPClient uses ascii mode by default.
You have to use binary mode to transfer binary files.
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
Your current code can by chance work on some servers even in the ascii mode, if the server is using Windows EOL sequence, hence no conversion takes place. And even then probably only, if the file by chance does not contain any lone #13.
One of your servers probably attempts to transmit the file as text, and your ftp client also thinks that it receives text.
Here is an excerpt from the javadoc:
If the current file type is ASCII, the returned InputStream
will convert line separators in the file to the local
representation.
If you are on windows, every line break will be replaced by 'linebreak + cr', wreaking havoc on all data structures in the png file.
The expected number of bytes for this scenario is: 4686 * (1 + 1 / 256) = 4704.3046875 , because on average, every 256-th byte in a png file should look like an ASCII line break, and will therefore result in an extra added byte. Your file ends up having 4706 bytes, which is pretty close.
Setting file type to FTP.BINARY_FILE_TYPE should fix this: https://commons.apache.org/proper/commons-net/apidocs/org/apache/commons/net/ftp/FTPClient.html#setFileType(int)
i'm trying to transfer Files with a DatagrammSocket in Java. I'm reading the files into 4096 Byte pieces. We are using ACK, so all pieces are in the right order, we tried pdf, exe, jpg and lot more stuff successfully, but iso, zip and 7z are not working. They have exactly the same size afterwards. Do you have any idea?
Reading the Parts:
byte[] b = new byte[FileTransferClient.PACKAGE_SIZE - 32];
FileInputStream read = new FileInputStream(file);
read.skip((part - 1) * (FileTransferClient.PACKAGE_SIZE - 32));
read.read(b);
content = b;
Writing the Parts:
stream = new FileOutputStream(new File(this.filePath));
stream.write(output);
...
stream.write(output);
stream.close();
(Sorry for great grammar, i'm German)
Your write() method calls are assuming that the entire buffer was filled by receive(). You must use the length provided with the DatagramPacket:
datagramSocket.receive(packet);
stream.write(packet.getData(), packet.getOffset(), packet.getLength());
If there is overhead in the packet, e.g. a sequence number, which there should be, you will need to adjust the offset and length accordingly.
NB TCP will ensure 'everything gets transferred and is not damaged'.
I'm trying to download a file using the android DownloadManager, access the file, and write it to a new location (in this example I'm downloading a database which is compiled server side and needs to be wrote to the /database/ directory).
I've been reading up and managed to download the file, and activate the BroadcastReceiver, but at this point I get stuck.
I've returned the ParcelFileDecriptor file but I'm having trouble converting it to a stream in any way. I can't decide if the ParcelFileDecriptor.AutoCloseInputStream is a red herring or not, but I'm pretty sure the ParcelFileDecriptor has relativity to a stream, but I'm really struggling to work it out. Any help would be much appreciated.
Assuming you've started the download already and set up the Broadcast Reciver, the following code will do the job...
ParcelFileDescriptor file = dMgr.openDownloadedFile(downloadId);
File dbFile = getDatabasePath(Roads.DATABASE_NAME);
InputStream fileStream = new FileInputStream(file.getFileDescriptor());
OutputStream newDatabase = new FileOutputStream(dbFile);
byte[] buffer = new byte[1024];
int length;
while((length = fileStream.read(buffer)) > 0)
{
newDatabase.write(buffer, 0, length);
}
newDatabase.flush();
fileStream.close();
newDatabase.close();
If you're looking for more information on overwriting a database with your own check this link (Also where most of the above info has come from):
http://www.reigndesign.com/blog/using-your-own-sqlite-database-in-android-applications/
Our current project requires us to send an audio file to the server and then use the audio file for further computation.
Using the Java sound api, I was able to capture the recording and save it as a wav file in my system. Then in order to pass the audio wav to the server, I am using Apache Commons HttpClient to post a request to the server. (I am using InputstreamEntity provided by apache and sending the data as a chunk).
The problem appears when i am trying to recreate/retrieve the wav file on the server. I understand that I would have to use the AudioSystem.write API to create the wav file (exactly as what was done on my system). However what I observe is that althought the file gets created , it does not play (I am using vlc media player to test it FYI). I have searched in Google for sample codes and have tried to implement it, but is unable to play it once the file gets created.
The sample code snippets indicates the approaches i have tried:
//******************************************************************
try {
InputStream is = request.getInputStream();
FileOutputStream fs = new FileOutputStream("output123.wav");
byte[] tempbuffer = new byte[4096];
int bytesRead;
while((bytesRead=is.read(tempbuffer))!=-1)
{
fs.write(tempbuffer, 0,bytesRead);
}
is.close();
fs.close();
AudioInputStream inputStream =AudioSystem.getAudioInputStream(newFile("output123.wav"));
int numofbytes = inputStream.available();
byte[] buffer = new byte[numofbytes];
inputStream.read(buffer);
int bytesWritten = AudioSystem.write(inputStream, AudioFileFormat.Type.WAVE,new File("outputtest.wav"));
System.out.println("written"+bytesWritten);
Approach 2
InputStream is = request.getInputStream();
System.out.println("inputStream obtained : "+is.toString());
ByteArrayInputStream bais = null;
byte[] audioBuffer = IOUtils.toByteArray(is);
System.out.println(" is audioBuffer empty? : length = ? "+audioBuffer.length);
try {
AudioFileFormat ai = AudioSystem.getAudioFileFormat(is);
System.out.println("ai bytelength ? "+ai.getByteLength());
System.out.println("ai frame length = "+ai.getFrameLength());
Set<Map.Entry<String,Object>> audioProperties = ai.getFormat().properties().entrySet();
System.out.println("entry set is empty ? "+audioProperties.isEmpty());
for(Map.Entry me : audioProperties){
System.out.println("key = "+me.getKey());
System.out.println("value ="+me.getValue());}
bais = new ByteArrayInputStream(audioBuffer);
AudioInputStream ais = new AudioInputStream(bais, new AudioFormat(8000,8,2,true,true), 2);
AudioSystem.write(ais, AudioFileFormat.Type.WAVE,new File("testtest.wav"));
//*************************************************************************************
The audioFormat properties all turned out to be null. Are these null values giving the problem? So while creating the wave file on the server, I tried to set the properties manually once again. But even then the wav file would not play.
I have also tried quite a few approaches already mentioned on this site, but somehow they aren't working. I am sure i am missing something, but I am unable to pinpoint the exact problem.
Would be really helpful, if you guys can point out how to go about the conversion from ServletInputStream to getting a wav.
P.S (1) I know the code is shabby, because i have been under a trial and error situation for quite some time now. But I will give more details on the approaches if needed.
2) Apologise for the clumsiness, this happens to be my first post.. )
this is not how you copy a stream (from Approach 1). you have the correct code to copy a stream just above this.:
int numofbytes = inputStream.available();
byte[] buffer = new byte[numofbytes];
inputStream.read(buffer);
If all your server wants to do is get the data and write it to a file, then you do not need to use any of the audio API: simply treat the data as a stream of bytes.
So the part of approach 1 that is before any mention of AudioInputStream should be sufficient.
Although the approach chosen might not be the perfect solution, due to time constraints, I adopted a simpler approach. Using java.util.zip i simply zipped it up and sent it over to the server and then wrote a layer wherin the file gets unzipped . then i deleted the zip files. Seems like an immature solution (bcos the original challenge was to send the audio file). now i am incurring an overhead of zipping the files, but the file transfer would hapeen relatively faster. Thanks for your help guys.
I am having a web application with an applet which will copy a file packed witht the applet to the client machine.
When I deploy it to webserver and use: InputStream in = getClass().getResourceAsStream("filename") ;
The in.available() always return a size of 8192 bytes for every file I tried, which means the file is corrupted when it is copied to the client computer.
The InputStream is of type HttpInputStream (sun.net.protocol.http.HttpUrlConnection$httpInputStream). But while I test applet in applet viewer, the files are copied fine, with the InputStream returned is of type BufferedInputStream, which has the file's byte sizes. I guess that when getResourceStream in file system the BufferedInputStream will be used and when at http protocol, HttpInputStream will be used.
How will I copy the file completely, is there a size limited for HttpInputStream?
Thanks a lot.
in.available() tells you how many bytes you can read without blocking, not the total number of bytes you can read from a stream.
Here's an example of copying an InputStream to an OutputStream from org.apache.commons.io.IOUtils:
public static long copyLarge(InputStream input, OutputStream output)
throws IOException {
byte[] buffer = new byte[DEFAULT_BUFFER_SIZE];
long count = 0;
int n = 0;
while (-1 != (n = input.read(buffer))) {
output.write(buffer, 0, n);
count += n;
}
return count;
}
The in.available() always return a size of 8192 bytes for every file I tried, which means the file is corrupted when it is copied to the client computer.
It does not mean that at all!
The in.available() method returns the number of characters that can be read without blocking. It is not the length of the stream. In general, there is no way to determine the length of an InputStream apart from reading (or skipping) all bytes in the stream.
(You may have observed that new FileInputStream("someFile").available() usually gives you the file size. But that behaviour is not guaranteed by the spec, and is certainly untrue for some kinds of file, and possibly for some kinds of file system as well. A better way to get the size of a file is new File("someFile").length(), but even that doesn't work in some cases.)
See #tdavies answer for example code for copying an entire stream's contents. There are also third party libraries that can do this kind of thing; e.g. org.apache.commons.net.io.Util.