We Keep Coding

downloading files in java in several parts or segments

I'm trying to download files in java in a multi-segment way (i.e., dividing it to several parts and downloading each part in a separate thread parallelly) but when I use the code below, it seems each thread is downloading the whole file instead of just a part of it but when it finishes, file is downloaded correctly.
note that "downloadedSizeCombined" is sum of all bytes which are downloaded by all the threads and ArrayList "downloadedSize" keeps track of bytes which are downloaded by a single thread.
this method is in class Download which extends SwingWorker.
public Void doInBackground() {
ExecutorService es = Executors.newCachedThreadPool();
for (int i = 0; i < MAX_NUMBER_OF_PARTS; i++) {
int numOfThePart = i;
es.execute(new Runnable() {
#Override
public void run() {
RandomAccessFile file = null;
InputStream stream = null;
try {
while (Download.this.getStatus() == WAITINGLIST) {
Thread.sleep(1);
}
// Open connection to URL.
HttpURLConnection connection =
(HttpURLConnection) url.openConnection();
// Specify what portion of file to download.
int startByte = numOfThePart * sizeOfFile / MAX_NUMBER_OF_PARTS;
int endByte = ((numOfThePart + 1) * sizeOfFile / MAX_NUMBER_OF_PARTS) - 1;
if (numOfThePart == MAX_NUMBER_OF_PARTS)
endByte = ((numOfThePart + 1) * sizeOfFile / MAX_NUMBER_OF_PARTS);
connection.setRequestProperty("Range",
"bytes=" + ((startByte + downloadedSize.get(numOfThePart))) + "-" + endByte);
// Connect to server.
connection.connect();
// Check for valid content length.
int contentLength = connection.getContentLength();
if (contentLength < 1) {
System.out.println("1");
}
/* Set the size for this download if it
hasn't been already set. */
if (sizeOfFile == -1) {
sizeOfFile = contentLength;
}
file = new RandomAccessFile(new File(s.getCurrentDirectory(), getFileName(url)),
"rw");
file.seek(startByte + downloadedSize.get(numOfThePart));
fileLocation = new File(s.getCurrentDirectory(), getFileName(url));
stream = connection.getInputStream();
while (status == CURRENT) {
file.seek(startByte + downloadedSize.get(numOfThePart));
byte buffer[];
buffer = new byte[MAX_BUFFER_SIZE];
// Read from server into buffer.
int read = stream.read(buffer);
if (read == -1)
break;
// Write buffer to file.
file.write(buffer, 0, read);
downloadedSizeCombined += read;
downloadedSize.set(numOfThePart, downloadedSize.get(numOfThePart) + read);
publish(numOfThePart);
while (status == PAUSED) {
Thread.sleep(1);
}
}
} catch (Exception e) {
e.printStackTrace();
} finally {
// Close file.
if (file != null) {
try {
file.close();
} catch (Exception e) {
e.printStackTrace();
}
}
// Close connection to server.
if (stream != null) {
try {
stream.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
});
}
return null;
}
Thanks in advance.

Can't we use UDP connection? So if we use DatagramSocket class, it will anyways send the data in packets. Try this.
Will get back on this soon..

How can i pause download process via asynctask

I'm making an app with a download process, it will download a certain file. If I turn off the Wi-Fi when the file is downloading, the app crashes.
This is the log: recvfrom failed: ETIMEDOUT (Connection timed out)
I have a conditional, but it seems not to work. If I debug the code, it seems to enter the conditional.
else {
Thread.sleep(4000); //doesn't work, doesn't sleep
downloadresult = false;
}
I want the download process to pause when I turn off the Wi-Fi. Is there any way of doing this?
Thanks in advance.
All code
protected String doInBackground(String... f_url) {
try {
long total = 0;
URL url = new URL(f_url[0]);
HttpURLConnection conection = (HttpURLConnection) url.openConnection();
int lenghtOfFile = conection.getContentLength();
BufferedOutputStream output = new BufferedOutputStream(new FileOutputStream(file));
conection.connect();
BufferedInputStream input = new BufferedInputStream(conection.getInputStream());
byte data[] = new byte[8192];
int lastcount = 0;
while ((count = input.read(data)) != -1) {
if (isCanceled) { // this code waiting the click button :)
file.delete();
downloadresult = false;
break;
}
if (intCheck()) { // check internet and download
total += count;
downloadresult = true;
int ProgBarCount = (int) ((total * 100) / lenghtOfFile);
if (ProgBarCount > lastcount) {
lastcount = ProgBarCount;
publishProgress(Integer.toString(ProgBarCount));
}
output.write(data, 0, count);
}
else {
Thread.sleep(4000); //doesn't work, doesn't sleep
downloadresult = false;
}
}
output.flush();
output.close();
input.close();
}
catch (Exception e) {
e.printStackTrace();
exmessage = e.getMessage().toString();
downloadresult = false;
}
return null;
}
If I debug the code, it works perfectly. If the app can't dowload the file, I want the app to wait 4000ms and then try again, but if I run the app, it crashes.
How can i pause/resume the download process. Thank You

I fixed the problem :)
Thanks for all respond, I love it <3
Code:
protected String doInBackground(String... f_url) {
try {
long total = 0;
URL url = new URL(f_url[0]);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
/* if (file.exists())
{
connection.setAllowUserInteraction(true);
connection.setRequestProperty("Range", "bytes=" + lenghtOfFile + "-");
}*/
if(file.exists()){
deneme = file.length();
connection.setRequestProperty("Range", "bytes="+(file.length())+"-");
}
else{
connection.setRequestProperty("Range", "bytes=" + deneme + "-");
}
String connectionField = connection.getHeaderField("content-range");
if (connectionField != null)
{
String[] connectionRanges = connectionField.substring("bytes=".length()).split("-");
deneme = Long.valueOf(connectionRanges[0]);
}
if (connectionField == null && file.exists())
file.delete();
connection.setConnectTimeout(14000);
connection.setReadTimeout(20000);
connection.connect();
long lenghtOfFile = connection.getContentLength() + deneme;
RandomAccessFile output = new RandomAccessFile(file,"rw");
BufferedInputStream input = new BufferedInputStream(connection.getInputStream());
output.seek(deneme);
byte data[] = new byte[1024];
int lastcount = 0;
while ((count = input.read(data,0,1024)) != -1) {
if (isCanceled) { // this code waiting the click button :)
file.delete();
downloadresult = false;
break;
}
if (intCheck()) { // check internet and download
total += count;
downloadresult = true;
int ProgBarCount = (int) ((total * 100) / lenghtOfFile);
if (ProgBarCount > lastcount) {
lastcount = ProgBarCount;
publishProgress(Integer.toString(ProgBarCount));
}
output.write(data, 0, count);
}
}
// output.flush();
output.close();
input.close();
}
catch (Exception e) {
e.printStackTrace();
exmessage = e.getMessage().toString();
downloadresult = false;
}
return null;
}

Don't pause background tasks. When anything goes wrong cancel the task. Remember your app was smart enough to start a background task. It will be smart enough to restart it again later. If your going to pause a background task it should be paused only at the users bequest.

Java socketserver how to test?

I want to receive text files through a socket connection in java, I set up the server end but before I continue with the client I would like to know if the code I made works, except I have no idea how to test this.
Any help would be much appreciated..
EDIT: I know the port is open and listening for requests, what i want is to test what happens if it receives anything, will it create a file from the input and can I test this by simulation(sending a file or bytes i dont know)?
public class Server {
private static int port = 8080;
private static int maxConnections = 100000;
// Listen for incoming connections and handle them
public static void startServer() {
int i = 0;
try {
ServerSocket listener = new ServerSocket(port);
Socket server;
System.out.println("Started server on port:" + port);
while ((i++ < maxConnections) || (maxConnections == 0)) {
RunServer connection;
server = listener.accept();
RunServer conn_c = new RunServer(server);
Thread t = new Thread(conn_c);
t.start();
System.out.println("Created new thread");
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
class RunServer implements Runnable {
private Socket server;
RunServer(Socket server) {
this.server = server;
}
public void run() {
int bytesRead;
int current = 0;
BufferedOutputStream bufferedOutputStream = null;
FileOutputStream fileOutputStream = null;
DataInputStream clientData = null;
File file = null;
try {
// creating connection.
System.out.println("connected.");
// receive file
byte[] byteArray = new byte[6022386];
System.out.println("Please wait downloading file");
// reading file from socket
InputStream inputStream = server.getInputStream();
file = new File("toread.txt");
clientData = new DataInputStream(inputStream);
fileOutputStream = new FileOutputStream(file);
bufferedOutputStream = new BufferedOutputStream(fileOutputStream);
bytesRead = inputStream.read(byteArray, 0, byteArray.length);
current = bytesRead;
do {
bytesRead = inputStream.read(byteArray, current, (byteArray.length - current));
if (bytesRead >= 0)
current += bytesRead;
} while (bytesRead > -1);
bufferedOutputStream.write(byteArray, 0, current);
bufferedOutputStream.flush();
ReaderHelper.readTextFile(file);
} catch (IOException e) {
e.printStackTrace();
} catch (NumberFormatException e) {
e.printStackTrace();
} catch (SQLException e) {
e.printStackTrace();
} finally {
try {
if (fileOutputStream != null)
fileOutputStream.close();
if (bufferedOutputStream != null)
bufferedOutputStream.close();
if (clientData != null)
clientData.close();
if (server != null)
server.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}

Well, your socket will be up on port 8080 right?
You can open your browser and type: http://localhost:8080. The browser will create a connection and the line
server = listener.accept();
will "unlock". If you just wanna test if the socket is listenning it will do. Reading the stream you will see the first message of the HTTP protocol.

At first you have a possible error in a reading loop
byte[] byteArray = new byte[6022386];
//some code
do {
bytesRead = inputStream.read(byteArray, current, (byteArray.length - current));
if (bytesRead >= 0)
current += bytesRead;
} while (bytesRead > -1);
If file length is more than byteArray.length which is possible, then (byteArray.length - current) would be negative.
I suggest you to use smaller array, e.g. byte[] byteArray = new byte[8192]; and read file like this
while ((bytesRead = in.read(byteArray)) > 0 ) {
current += bytesRead;
bufferedOutputStream.write(byteArray, 0, bytesRead);
}
storing it into disk chunk by chunk. And after exiting while loop current will hold total number of read bytes.
Send file from another thread, which will connect to server
Socket clientSocket = new Socket("localhost", Server.port);
BufferedReader input = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
DataOutputStream output = new DataOutputStream(clientSocket.getOutputStream());
FileInputStream fileInput = new FileInputStream("fileName");
byte[] buffer = new byte[8192];
while(fileInput.read(buffer) != -1){
output.write(buffer, 0, length);
}
//close resources

Sending multiple files through socket. Number of files unknown [duplicate]

This question already has answers here:
Java multiple file transfer over socket
(3 answers)
Closed 7 years ago.
I'm making a communicator with ability to send files.
So far I managed to make text sending working using additional thread (listener).
I'm trying to make the same thing with files, but I don't know, how can I make a file listener - a thread, that detects incoming file, downloads it and listens for another file. Also, I don't know if I'm making my file sender properly. Could you help?
Current sender code:
try {
InputStream in = new FileInputStream(fileToSend);
OutputStream out = fileConn.getOutputStream();
Controller.copyData(in, out);
out.close();
in.close();
} catch (IOException e) {
System.out.println("Problem!");
}
And receiver code:
while (true)
{
try {
InputStream in = socket.getInputStream();
OutputStream out = new FileOutputStream("hi.txt"); //temporary
Controller.copyData(in, out);
out.close();
in.close();
} catch (IOException e) {
System.out.println("Problem!");
}
}
EDIT: I forgot to add my copyData. There it is:
public static void copyData(InputStream in, OutputStream out) throws IOException{
byte[] buf = new byte[8192];
int len = 0;
while ((len = in.read(buf)) != -1) {
out.write(buf, 0, len);
}
}

You can achive tha by just adding to your listening thread option to wait for diffrent messages/options and react accordingly. For example:
private class WaitingThread extends Thread {
volatile boolean awaitsServer = false;
DataInputStream dataInput = new DataInputStream(inputStream);
public void run() {
while (connected) {
int message = 0;
if (awaitsServer == true) {
if (dIn.available() ==0) {
view.setLog("waiting");
} else {
message = dIn.readInt();
switch (tempMessage) {
// TO DO ALL KIND OF COMMUNICATION
case 1:
int filesize = dataInput.readInt();
int bytesRead;
int currentTot = 0;
byte[] bytearray = new byte[filesize];
int len = dataInput.readInt();
FileOutputStream fos = new FileOutputStream(currentlySelectedFile);
BufferedOutputStream bos = new BufferedOutputStream(fos);
bytesRead = dataInput.read(bytearray, 0, bytearray.length);
currentTot = bytesRead;
do {
bytesRead = dataInput.read(bytearray, currentTot,
(len - currentTot));
if (bytesRead >= 0)
currentTot += bytesRead;
} while (currentTot < len);
bos.write(bytearray, 0, currentTot);
bos.close();
}
case 2: //GET TEXT
case 3: //DO SOMETHING ELSE
}}}
Btw you have example how to send files.

Downloading Large JSON File to local file using Java

I'm attempting to download a JSON from the following URL - http://api.crunchbase.com/v/1/companies.js - to a local file. I'm using Java 1.7 and the following JSON Libraries - http://www.json.org/java/ - to attempt to make it work.
Here's my code:
public static void download(String address, String localFileName) {
OutputStream out = null;
URLConnection conn = null;
InputStream in = null;
try {
URL url = new URL(address);
out = new BufferedOutputStream(
new FileOutputStream(localFileName));
conn = url.openConnection();
in = conn.getInputStream();
byte[] buffer = new byte[1024];
int numRead;
long numWritten = 0;
while ((numRead = in.read(buffer)) != -1)
{
out.write(buffer, 0, numRead);
numWritten += numRead;
System.out.println(buffer.length);
System.out.println(" " + buffer.hashCode());
}
System.out.println(localFileName + "\t" + numWritten);
} catch (Exception exception) {
exception.printStackTrace();
} finally {
try {
if (in != null) {
in.close();
}
if (out != null) {
out.close();
}
} catch (IOException ioe) {
}
}
}
When I run the code everything seems to work until midway through the loop the program seems to stop and not continue reading the JSON Object.
Does anyone know why this would stop reading? How could I fix the issue?

Try This:
public void saveUrl(String filename, String urlString) throws MalformedURLException, IOException
{
BufferedInputStream in = null;
FileOutputStream fout = null;
try
{
in = new BufferedInputStream(new URL(urlString).openStream());
fout = new FileOutputStream(filename);
byte data[] = new byte[1024];
int count;
while ((count = in.read(data, 0, 1024)) != -1)
{
fout.write(data, 0, count);
}
}
finally
{
if (in != null)
in.close();
if (fout != null)
fout.close();
}
}

Does anyone know why this would stop reading? How could I fix the issue?
I can't see anything obviously wrong with the client-side code. In the absence of any other evidence on the client side, I'd look at the server-side logs to see if there are any clues there.
IMO, the most likely explanation is one of the following:
There's a bug in the server-side code that is generating the JSON and it is crashing halfway through.
The server (or a proxy / reverse proxy) has a timeout on the time allowed for some part of the interaction, and this particular request is taking too long.