I am hoping someone can help me (once again).
I have a very large number of smmll files (over 4000) each only a few K.
I have writen an FTP program in java which will transfer each file individually but it is taking a very long time. Also the handshaking overhead seems to make the problem worse.
What I would like to be able to do is open the FTP connection send all the files then close it again.
I know that this is possible in FTP but quite how to acheive this in java is beyond me.
I currently have the filenames in an array so parsing through them is no problem. I have tried calling the following class and passing it the filename but after several hours it was still moving about 1 file per second.
package website;
import java.io.BufferedOutputStream;
import java.io.FileInputStream;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLConnection;
public class ftpUpload {
public ftpUpload(String target, String savename, String localFilePath) {
URL url;
try {
url = new URL(target + savename + ";type=i");
URLConnection con = url.openConnection();
BufferedOutputStream out =
new BufferedOutputStream(con.getOutputStream());
FileInputStream in =
new FileInputStream(localFilePath + savename);
int i = 0;
byte[] bytesIn = new byte[1024];
while ((i = in.read(bytesIn)) >= 0) {
out.write(bytesIn, 0, i);
}
out.close();
in.close();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}
Is there a way I can open the connection with the ftp site username and password,
then send it the files
and finally close the connection?
This would seem to me easier than creating multiple threads to send files concurrently.
Any advice greatfully received.
Paul
I don't think it's possible to send multiple files in one session using URLConnection, this means you get the overhead of opening and closing the session for every file.
FTPClient from commons net does support multiple operations in one session. For example (exception handling ommitted):
FTPClient ftp = new FTPClient();
ftp.connect("ftp.example.com");
ftp.login("admin", "secret");
ftp.setFileType(FTPClient.BINARY_FILE_TYPE);
for(File file : files) {
InputStream in = new FileInputStream(file);
ftp.storeFile(file.getName(), in);
in.close();
}
ftp.disconnect();
This should help.
If you still need better performance, I don't see any other alternative than using multiple threads.
After plenty of testing I have found reliability issues with multiple ftp threads to public servers which is what I need in this case. Most (if not all) ftp servers limit the maximum number of connections and also limit the maximum number of concurrent connections from the same IP address. Two concurrent connections from the same IP seems to be the only guaranteed maximum you are allowed. The realistic option as suggested above is to zip the files and ftp a single file. You can unzip the file when it gets there using an php script (as long as the server supports unzip you will need to check if this included in the php build). Finally if like me you need to upload in excess of 10,000 files many ftp servers will not show more than 9998 files (10,000 inlcuding . and ..)
If anyone knows of an ftp server free or cheap that supports ZipArchive in the php build and will list more than 9998 files when requesting a file listing in ftp can you please let me know.....
Related
I am quite new to StackOverflow, if my question is inappropriately asked or confusing, please let me know, thank you!
I am working on an audio streaming project, in which the clients are allowed to upload their mp3 files to the server. The server will store them into a playlist and stream the songs back to all the clients.
Here is my code for the client to upload the mp3:
public static void sendPackets(){
System.out.println("Sending test file...");
try{
while (active){
//The song needs to be uploaded;
File file = new File("Sorrow.mp3")
FileInputStream fis = new FileInputStream(file);
byte[] byteStream = new byte[(int) file.length()];
//Trying to convert mp3 to byteStream
fis.close();
InetAddress destination = InetAddress.getByName("localhost");
DatagramPacket sendingAirMail = new DatagramPacket(byteStream, byteStream.length, destination, 50010); // 50010 is the listening port
serverSocket.send(sendingAirMail); // sending the entire bytestream via UDP
// ServerSocket is a DatagramSocket
break;
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
The Problem lies here:
serverSocket.send(sendingAirMail);
As it will give me this error:
java.net.SocketException: The message is larger than the maximum supported by the underlying transport: Datagram send failed
at java.base/java.net.DualStackPlainDatagramSocketImpl.socketSend(Native Method)
at java.base/java.net.DualStackPlainDatagramSocketImpl.send(DualStackPlainDatagramSocketImpl.java:136)
at java.base/java.net.DatagramSocket.send(DatagramSocket.java:695)
at client.sendPackets(client.java:116)
at client$2.run(client.java:67)
After looking up google, I learned that it is because the UDP has a limit of size in each package delivered, so I wish to know how to separate the UDP package properly in this case? I know the TCP will be better in this case, but I think I need to learn how to separate packages anyway because I need to stream back the byte arrays from the server using UDP. Any help will be appreciated!
I can post my server and other client information if needed.
The thing is that you need to cut your file into several pieces and deliver them. For UDP, you need some things to make sure the file is complete and correct. Here are some suggestions:
First, you need to cut your file, so you need to give a seq flag in the head. What's more you may need some extra infomations like the whole size of file, the timestamp and so on.
struct msg {
int seq;
int total_seq;
int size;
void *data;
};
Then, it's better to build a send buffer and recive buffer, check every time if the buffer is empty, if not, send/receive it.
After receiving some pieces, you need to rebuild them using the seq flag. When some seq gets lost, you need retransmission. So you need a retransmission design here.
In a word, you need the following things at least:
A user defined head before information
cut/rebuild file
retransmission(GBN or FEC or both)
Hope that can help you.
The files downloaded by this, are nearly the same size but differ in some lines. Every answer points to binary file type. But this won't help.
Got anybody an idea for the problem (transferring PDF)?
FTPClient ftpClient = new FTPClient();
OutputStream outputStream = null;
boolean resultOk = true;
try {
ftpClient.connect(host, port);
ftpClient.enterLocalPassiveMode();
ftpClient.setFileTransferMode(FTP.COMPRESSED_TRANSFER_MODE);
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
if (showMessages) {
System.out.println(ftpClient.getReplyString());
}
resultOk &= ftpClient.login(usr, pwd);
if (showMessages) {
System.out.println(ftpClient.getReplyString());
}
outputStream = new FileOutputStream(localResultFile);
resultOk &= ftpClient.retrieveFile(remoteSourceFile, outputStream);
outputStream.flush();
outputStream.close();
if (showMessages) {
System.out.println(ftpClient.getReplyString());
}
if (resultOk == true) {
resultOk &= ftpClient.deleteFile(remoteSourceFile);
}
resultOk &= ftpClient.logout();
if (showMessages) {
System.out.println(ftpClient.getReplyString());
}
} finally {
ftpClient.disconnect();
}
It's clear from the files you have shared, that the transfer indeed happened in text/ascii mode.
While probably not required by FTP specification, with some FTP servers (e.g. FileZilla server or ProFTPD), you cannot change transfer type before logging in. But servers like IIS, ProFTPD or vsftpd have no problem with that. On the other hand FileZilla server defaults to binary mode anyway (what is another violation of the specification), so you are probably using yet another one.
In any case, move the .setFileType call after .login. And test its return value.
And remove the .setFileTransferMode call. It does not do any harm with most servers, as hardly any server support MODE C, hence the call is ignored anyway. But if you encounter a server that does, it would break the transfer, as FTPClient actually does not support it.
While my problem was related to corruption of the upload, I resolved similar issue by moving the set of file type after ftp login (I dont use transfer mode leaving it to its default value):
resultOk &= ftpClient.login(usr, pwd);
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
I saw in some forums that setting binary file type before invoking login method, could lead to problems in transfer. Before this change, PDF file get downloaded but show corrupted fonts and elements. Now it works. Hope it may helps someone.
It seems to happen when using unescaped paths that contain spaces. E.g. C:/Documents and Settings/test
Got it solved now by using a escaped path for the spaces. Thanks for your help
Background
I'm trying to use Akka actors to download many files. (Akka 2.5.1 & Java 8)
Each actor is assigned with a different URL it should download from.
A different actor is creating the downloaders actors and it should not wait for the downloaders to finish. Once they will finish, they will create another actor to handle the downloaded file.
The problem
When I run only one actor - it is able to download the file.
As I increas the number of actors it seems like none of them is able to finish its task. They are downloading portion of the files and stopping with no particular error / exception.
Actors creation code:
ActorRef downloaderActor = context().actorOf(Props.create(DownloaderActor.class));
downloaderActor.tell("URL to download", this.getSelf());
Inside the DownloaderActor class I have a download function where it seems like the problem occurs:
public void downloadFile(String fileURL, String saveDir) {
try {
URL url = new URL(fileURL);
HttpURLConnection httpConn = (HttpURLConnection) url.openConnection();
int responseCode = httpConn.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
InputStream inputStream = httpConn.getInputStream();
String saveFilePath = saveDir + File.separator + fileName;
FileOutputStream outputStream = new FileOutputStream(saveFilePath);
int bytesRead = -1;
byte[] buffer = new byte[4096];
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
outputStream.close();
inputStream.close();
System.out.println("File downloaded: " + fileURL);
} else {
System.out.println("No file to download. Server replied HTTP code: " + responseCode + " . when accessing " + url);
}
httpConn.disconnect();
}catch (MalformedURLException murl){
murl.printStackTrace();
}catch (IOException ioe){
ioe.printStackTrace();
}
}
And to be more specific - it seems like the problem is in the "while" loop, because if I add there logging, I can see that the loop is looping and than stops after a while.
Failed attempts
I also tried to set some http connection parameters:
httpConn.setRequestProperty("Range", "bytes=0-24");
httpConn.setConnectTimeout(10_000_000);
But it didn't seems to help.
I tried to put the download function as static function in a different Util class and it also didn't helped.
Will appriciate any help here.
You are using blocking I/O here to download the file, which means you need a thread for each concurrent download.
You're on the right track in the sense that the Actor model can help you model your problem without requiring one thread per concurrent download: actors are a good way to model concurrent asynchronous processes.
To actually take advantage of this, however, you still need to write the 'implementation' of the actor to be non-blocking. There are a number of non-blocking HTTP libraries available for Scala, for example you could use akka-http's future-based client API together with the 'ask' pattern.
I found the problem origin:
I run the code in a Junit contex. Seems like the Junit at some point cut the running threads and thus terminating the activiti of the actors.
Once I started to run the program in regular run mode, the problem (seems to be) gone.
I am using the following code to download a file using Java but i want to detect when connection is lost. I ran the following code and while in the middle of downloading i disconnected my internet purposefully but no exception was thrown and it hanged. Even after turning on the connection back nothing happened. So, it hanged forever without any exceptions. Is there a way to make it throw an exception when the connection is lost? Thanks for the help!
package toplower.top;
import java.io.FileOutputStream;
import java.net.URL;
import java.nio.channels.Channels;
import java.nio.channels.ReadableByteChannel;
import java.util.*;
import javax.mail.*;
public class testing {
public static void main(String[] args) {
try{
URL website = new URL("http://b128.ve.vc/b/data/128/1735/asd.mp3");
ReadableByteChannel rbc = Channels.newChannel(website.openStream());
FileOutputStream fos = new FileOutputStream("song.mp3");
fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);
}
catch(Exception e){
System.out.println("got here");
e.printStackTrace();
}
System.out.println("Done downloading...");
}
}
apparently one is able to set a timeout on your socket, i think this article describes it : http://technfun.wordpress.com/2009/01/29/networking-in-java-non-blocking-nio-blocking-nio-and-io/
Your current thread blocks forever - and thats your root problem. So one possible solution would be to switch to asynchronous operations and/or to set your timeout
Since you are using HTTP, you would be best of using the Apache HttpClient project. org.apache.http.client.config.RequestConfig.Builder allows to set various timeout values that have no correspondence in java.net.URL and related classes. Especially setting a socket timeout would help in your case. You can then use org.apache.http.HttpEntity.writeTo to directly write the resource to a FileOutputStream.
I want to create simple download accelerator.
How it works
Server wait for incoming connection.
Client connect to server.
Then, server send file size to client and wait for download connection.
Client got file size, then create download thread and these thread are connect to server.
After server got connection from each thread, server will wait for start and end offset file from thread.
Each thread send start and end offset file to server.
After server got offsets, server will send the portion of file to thread.
Each thread will read and write to file. For example, buffer.p01, buffer.p02, buffer.p03
Client merge all file into one file order by sequence. ( Not yet implemented )
I think server side it works correctly but client side it has some problem.
The problem is if I set MAXTHREAD to 1, it works correctly. But if I set more than one, it stuck somewhere forever.
This is server side code..
http://pastebin.com/TEakGB0c
and this is client side code with multithreading
http://pastebin.com/wKhP7DxS
Thanks your.
You have a pretty big obvious problem. ServerSocket's accept method returns a new socket every time. In your server code here
initSocket = servSock.accept();
initSocket is a class member field which means you will over write the old socket and never close it. You should start a new thread to handle this socket and from what I see it looks like you just keep reusing the same socket. That won't work. Look at tutorials on how to open sockets. Sorry I can't help more but there is a lot of things going on here that just won't work. Maybe you can start focusing on part of the code and we can help more.
I agree, it could be a small issue or it could be a big one, some example code would help us aid you, If you try to connect to a server 3 times using the same port you will get an error because you can only have 1 connection per port, the problem could be super simple or very complex, if you edit your post and add your code then we can better help you.
Please close your OutputStream os
Sending u a snippet
public static boolean sendFile() {
int start = Integer.parseInt(startAndEnd[0]) - 1;
int end = Integer.parseInt(startAndEnd[1]) - 1;
int size = (end - start) + 1;
try {
os = initSocket.getOutputStream();
os.write(byteArr, start, size);
os.flush();
System.out.println("Send file to : " + initSocket);
} catch (IOException e) {
System.out.println(e.getLocalizedMessage());
disconnected();
return false;
} finally {
if (os != null) {
try {
os.close();
} catch (IOException ex) {
Logger.getLogger(FileServer.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
return true;
}