We Keep Coding

downloading files in java in several parts or segments

I'm trying to download files in java in a multi-segment way (i.e., dividing it to several parts and downloading each part in a separate thread parallelly) but when I use the code below, it seems each thread is downloading the whole file instead of just a part of it but when it finishes, file is downloaded correctly.
note that "downloadedSizeCombined" is sum of all bytes which are downloaded by all the threads and ArrayList "downloadedSize" keeps track of bytes which are downloaded by a single thread.
this method is in class Download which extends SwingWorker.
public Void doInBackground() {
ExecutorService es = Executors.newCachedThreadPool();
for (int i = 0; i < MAX_NUMBER_OF_PARTS; i++) {
int numOfThePart = i;
es.execute(new Runnable() {
#Override
public void run() {
RandomAccessFile file = null;
InputStream stream = null;
try {
while (Download.this.getStatus() == WAITINGLIST) {
Thread.sleep(1);
}
// Open connection to URL.
HttpURLConnection connection =
(HttpURLConnection) url.openConnection();
// Specify what portion of file to download.
int startByte = numOfThePart * sizeOfFile / MAX_NUMBER_OF_PARTS;
int endByte = ((numOfThePart + 1) * sizeOfFile / MAX_NUMBER_OF_PARTS) - 1;
if (numOfThePart == MAX_NUMBER_OF_PARTS)
endByte = ((numOfThePart + 1) * sizeOfFile / MAX_NUMBER_OF_PARTS);
connection.setRequestProperty("Range",
"bytes=" + ((startByte + downloadedSize.get(numOfThePart))) + "-" + endByte);
// Connect to server.
connection.connect();
// Check for valid content length.
int contentLength = connection.getContentLength();
if (contentLength < 1) {
System.out.println("1");
}
/* Set the size for this download if it
hasn't been already set. */
if (sizeOfFile == -1) {
sizeOfFile = contentLength;
}
file = new RandomAccessFile(new File(s.getCurrentDirectory(), getFileName(url)),
"rw");
file.seek(startByte + downloadedSize.get(numOfThePart));
fileLocation = new File(s.getCurrentDirectory(), getFileName(url));
stream = connection.getInputStream();
while (status == CURRENT) {
file.seek(startByte + downloadedSize.get(numOfThePart));
byte buffer[];
buffer = new byte[MAX_BUFFER_SIZE];
// Read from server into buffer.
int read = stream.read(buffer);
if (read == -1)
break;
// Write buffer to file.
file.write(buffer, 0, read);
downloadedSizeCombined += read;
downloadedSize.set(numOfThePart, downloadedSize.get(numOfThePart) + read);
publish(numOfThePart);
while (status == PAUSED) {
Thread.sleep(1);
}
}
} catch (Exception e) {
e.printStackTrace();
} finally {
// Close file.
if (file != null) {
try {
file.close();
} catch (Exception e) {
e.printStackTrace();
}
}
// Close connection to server.
if (stream != null) {
try {
stream.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
});
}
return null;
}
Thanks in advance.

Can't we use UDP connection? So if we use DatagramSocket class, it will anyways send the data in packets. Try this.
Will get back on this soon..

Cannot download file from URL in java

I'm making a program that will download files from URL. The downloading always starts, but it is not completed. For example, if file's size is 3 MB, program download only half of that so I cannot open the downloaded file. But program says that file is downloaded succesfully.
public class FileDownloader {
public static void main (String [] args) throws IOException {
InputStream fileIn;
FileOutputStream fileOut;
Scanner s = new Scanner(System.in);
System.out.println("Enter URL: ");
String urlStr = s.nextLine();
URL url = new URL(urlStr);
URLConnection urlConnect = url.openConnection();
fileIn = urlConnect.getInputStream();
System.out.println("Enter file name: ");
String fileStr = s.nextLine();
fileOut = new FileOutputStream(fileStr);
while (fileIn.read() != -1) {
fileOut.write(fileIn.read());
}
System.out.println("File is downloaded");
}
}
So how can I solve it? Should use another way to download?

You are losing every alternate bytedue to
while (fileIn.read() != -1) { //1st read
fileOut.write(fileIn.read()); //2nd read - 1st write
}
You are reading twice and writing only once.
What you need to do is
int x;
while ((x = fileIn.read()) != -1) { //1st read
fileOut.write(x); //1st write
}
Here is your complete code
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.URL;
import java.net.URLConnection;
import java.util.Scanner;
public class FileDownloader {
public static void main(String[] args) throws IOException {
InputStream fileIn;
FileOutputStream fileOut;
Scanner s = new Scanner(System.in);
System.out.println("Enter URL: ");
String urlStr = s.nextLine();
URL url = new URL(urlStr);
URLConnection urlConnect = url.openConnection();
fileIn = urlConnect.getInputStream();
System.out.println("Enter file name: ");
String fileStr = s.nextLine();
fileOut = new FileOutputStream(fileStr);
int x;
while ((x = fileIn.read()) != -1) {
fileOut.write(x);
}
System.out.println("File is downloaded");
}

You can download a large file with below code efficiently.
public static void main(String[] args) throws IOException {
InputStream in = null;
FileOutputStream out = null;
try {
System.out.println("Starting download");
long t1 = System.currentTimeMillis();
URL url = new URL(args[0]);// or you can hard code the URL
// Open the input and out files for the streams
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
in = conn.getInputStream();
out = new FileOutputStream(args[1]);//// or you can hard code the filename
// Read data into buffer and then write to the output file
byte[] buffer = new byte[8192];
int bytesRead;
while ((bytesRead = in.read(buffer)) != -1) {
out.write(buffer, 0, bytesRead);
}
long t2 = System.currentTimeMillis();
System.out.println("Time for download & save file in millis:"+(t2-t1));
} catch (Exception e) {
// Display or throw the error
System.out.println("Erorr while execting the program: "
+ e.getMessage());
} finally {
// Close the resources
if (in != null) {
in.close();
}
if (out != null) {
out.close();
}
}
}

just simply use this：
import org.apache.commons.io.FileUtils;
import java.net.URL;
String path = "F:/"
String fileName = "song"
FileUtils.copyURLToFile(myUrl, new File(path + fileName + ".mp3"));

read and store server logs in real tine

i am using following code for read log file and matched pattern store in database.
public class MIScript {
//DB
public static void db(String email, String ip, String pdate, String hostname, String im) {
// DATABASE INSERT
}
public void pop(File f, String IM) throws FileNotFoundException, IOException, InterruptedException {
int pos = 0;
RandomAccessFile file = new RandomAccessFile(f, "r");
pos = (int) file.length() - (int) Math.min(file.length() - 1, file.length());
file.seek(pos);
for (; true; Thread.currentThread().sleep(1000)) {
int l = (int) (file.length() - pos);
if (l <= 0) {
continue;
}
byte[] buf = new byte[l];
int read = file.read(buf, 0, l);
String out = new String(buf, 0, l);
// System.out.println(out);
InputStream is = new ByteArrayInputStream(out.getBytes());
BufferedReader in = new BufferedReader(new InputStreamReader(is));
String line = null;
while (((line = in.readLine()) != null)) {
if (line.contains("LOG")) {
// SOME CODE
//INSERT INTO DATABASE
MIScript.db(// parameters //);
}
}
}
}
public static void main(String[] args) {
try {
File pop = new File("d://ABC.log");
MIScript tail1 = new MIScript();
tail1.pop(pop, "TEST");
} catch (ArrayIndexOutOfBoundsException ar) {
System.out.println("Errrrr------" + ar);
System.exit(1);
} catch (Exception io) {
io.printStackTrace();
System.out.println("Errrrr2------" + io);
System.exit(1);
}
}
}
it works great on single file but i need to 4 file to read synchronously please give me the way to do this .
i tried to do this with 2 files but that's not working

You need to read each file in a separate thread, and ensure that the code to write to database is thread safe.
Edit: I put this in a comment, but actually it's part of the answer: from Java 7 you can get the filesystem to call you back when a file changes http://docs.oracle.com/javase/7/docs/api/java/nio/file/WatchService.html
That way you don't need to poll the file size like you're doing... but you do still need 1 thread per file.
Tutorial for WatchService is here: http://docs.oracle.com/javase/tutorial/essential/io/notification.html

Trouble with java snippets

public class GenericWorldLoader implements WorldLoader {
#Override
public LoginResult checkLogin(PlayerDetails pd) {
Player player = null;
int code = 2;
File f = new File("data/savedGames/" + NameUtils.formatNameForProtocol(pd.getName()) + ".dat.gz");
if(f.exists()) {
try {
InputStream is = new GZIPInputStream(new FileInputStream(f));
String name = Streams.readRS2String(is);
String pass = Streams.readRS2String(is);
if(!name.equals(NameUtils.formatName(pd.getName()))) {
code = 3;
}
if(!pass.equals(pd.getPassword())) {
code = 3;
}
} catch(IOException ex) {
code = 11;
}
}
if(code == 2) {
player = new Player(pd);
}
return new LoginResult(code, player);
}
#Override
public boolean savePlayer(Player player) {
try {
OutputStream os = new GZIPOutputStream(new FileOutputStream("data/savedGames/" + NameUtils.formatNameForProtocol(player.getName()) + ".dat.gz"));
IoBuffer buf = IoBuffer.allocate(1024);
buf.setAutoExpand(true);
player.serialize(buf);
buf.flip();
byte[] data = new byte[buf.limit()];
buf.get(data);
os.write(data);
os.flush();
os.close();
return true;
} catch(IOException ex) {
return false;
}
}
#Override
public boolean loadPlayer(Player player) {
try {
File f = new File("data/savedGames/" + NameUtils.formatNameForProtocol(player.getName()) + ".dat.gz");
InputStream is = new GZIPInputStream(new FileInputStream(f));
IoBuffer buf = IoBuffer.allocate(1024);
buf.setAutoExpand(true);
while(true) {
byte[] temp = new byte[1024];
int read = is.read(temp, 0, temp.length);
if(read == -1) {
break;
} else {
buf.put(temp, 0, read);
}
}
buf.flip();
player.deserialize(buf);
return true;
} catch(IOException ex) {
return false;
}
}
}
Yeah so... My problem is that this seems to save 'something' in really complex and hard to read way(binary) and I'd rather have it as an .txt, in easily readable format. how to convert?
EDIT: I'm not using Apache Mina, so what should I replace
IoBuffer buf = IoBuffer.allocate(1024);
buf.setAutoExpand(true);"
with?

checkLogin() obviously checks whether the specified login has matching data present and whether the password is correct.
savePlayer() method saves the player.
loadPlayer() loads it again.
The data format used is gzip (wiki) and it is written as a stream of serialized data. If you want to make it more readable, you might want to overload (or just use it, if it is good) toString() method of Player class and to write player.toString() into a new text file using e.g. BufferedWriter wrapped around a File Writer:
String playerName = NameUtils.formatNameForProtocol(player.getName());
BufferedWriter writer = new BufferedWriter(new FileWriter(playerName + ".txt"));
writer.write(player.toString());
writer.close();

Uploading Image From Java Desktop App to Server

I am using the following code on the client side to upload to the server
public class UploaderExample{
private static final String Boundary = "--7d021a37605f0";
public void upload(URL url, List<File> files) throws Exception
{
HttpURLConnection theUrlConnection = (HttpURLConnection) url.openConnection();
theUrlConnection.setDoOutput(true);
theUrlConnection.setDoInput(true);
theUrlConnection.setUseCaches(false);
theUrlConnection.setChunkedStreamingMode(1024);
theUrlConnection.setRequestProperty("Content-Type", "multipart/form-data; boundary="
+ Boundary);
DataOutputStream httpOut = new DataOutputStream(theUrlConnection.getOutputStream());
for (int i = 0; i < files.size(); i++)
{
File f = files.get(i);
String str = "--" + Boundary + "\r\n"
+ "Content-Disposition: form-data;name=\"file" + i + "\"; filename=\"" + f.getName() + "\"\r\n"
+ "Content-Type: image/png\r\n"
+ "\r\n";
httpOut.write(str.getBytes());
FileInputStream uploadFileReader = new FileInputStream(f);
int numBytesToRead = 1024;
int availableBytesToRead;
while ((availableBytesToRead = uploadFileReader.available()) > 0)
{
byte[] bufferBytesRead;
bufferBytesRead = availableBytesToRead >= numBytesToRead ? new byte[numBytesToRead]
: new byte[availableBytesToRead];
uploadFileReader.read(bufferBytesRead);
httpOut.write(bufferBytesRead);
httpOut.flush();
}
httpOut.write(("--" + Boundary + "--\r\n").getBytes());
}
httpOut.write(("--" + Boundary + "--\r\n").getBytes());
httpOut.flush();
httpOut.close();
// read & parse the response
InputStream is = theUrlConnection.getInputStream();
StringBuilder response = new StringBuilder();
byte[] respBuffer = new byte[4096];
while (is.read(respBuffer) >= 0)
{
response.append(new String(respBuffer).trim());
}
is.close();
System.out.println(response.toString());
}
public static void main(String[] args) throws Exception
{
List<File> list = new ArrayList<File>();
list.add(new File("C:\\square.png"));
list.add(new File("C:\\narrow.png"));
UploaderExample uploader = new UploaderExample();
uploader.upload(new URL("http://systemout.com/upload.php"), list);
}
}
I have tried writing the servlet that receives the image file and saves it to a folder on the server....but have failed miserably...This is part of an academic project i need to submit as part of my degree....Please Help!!!
I want help ...can someone guide me on how the servlet will be written....
I tried the following:
response.setContentType("text/html;charset=UTF-8");
PrintWriter out = response.getWriter();
try {
InputStream input = null;
OutputStream output = null;
try {
input = request.getInputStream();
output = new FileOutputStream("C:\\temp\\file.png");
byte[] buffer = new byte[10240];
for (int length = 0; (length = input.read(buffer)) > 0 ; ) {
output.write(buffer, 0, length);
}
}
catch(Exception e){
out.println(e.getMessage());
}
finally {
if (output != null) {
output.close();
}
if (input != null) {
input.close();
}
}
out.println("Success");
}
catch(Exception e){
out.println(e.getMessage());
}
finally {
out.close();
}
}
I went ahead and tried the fileupload from apache.org....and wrote the following servlet code:
protected void processRequest(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
response.setContentType("text/html;charset=UTF-8");
PrintWriter out = response.getWriter();
try {
out.println(1);
boolean isMultipart = ServletFileUpload.isMultipartContent(request);
if (isMultipart) {
// Create a factory for disk-based file items
FileItemFactory factory = new DiskFileItemFactory();
// Create a new file upload handler
ServletFileUpload upload = new ServletFileUpload(factory);
// Parse the request
List /* FileItem */ items = upload.parseRequest(request);
// Process the uploaded items
Iterator iter = items.iterator();
while (iter.hasNext()) {
FileItem item = (FileItem) iter.next();
if (item.isFormField()) {
//processFormField(item);
} else {
//processUploadedFile(item);
String fieldName = item.getFieldName();
String fileName = item.getName();
String contentType = item.getContentType();
boolean isInMemory = item.isInMemory();
long sizeInBytes = item.getSize();
//write to file
File uploadedFile = new File("C:\\temp\\image.png");
item.write(uploadedFile);
out.println("Sucess!");
}
}
} else {
out.println("Invalid Content!");
}
} catch (Exception e) {
out.println(e.getMessage());
} finally {
out.close();
}
}
However i am still confused on how to write the multipart code on the client side...the one i posted above is not working with my servlet implementation.....help please....some links where i can learn writing posting multipart form from java desktop app would be useful

So here's my recommendation: don't write this code yourself! Use http://commons.apache.org/fileupload/ instead. It will save you a lot of headaches, and you'll be up and running quite quickly. I'm pretty sure that problem is that the InputStream contains the multi-part boundaries, and is thus not a valid image.
Here's another observation: since you're not doing any transformations on the image, there's no need to read and write the image bytes using ImageIO. You're better off writing the bytes straight from the InputStream to the file.