We Keep Coding

Name the images that i get from a web site in Java

I am scraping a web site and as a last part, I get their product images to the folder. I want to name these images like (product_id + numberOfImages) I mean if product has a 2 images, there will be 2 png like (productId_1) (productId_2).
I have productId and also images there is no problem. I just want to know how to name it as I want. Here is my code.
for(Element imageElement : imageElements){
String strImageURL = imageElement.attr("src");
String strImageName =product_id + "_" + ??;
try {
URL urlImage = new URL(strImageURL);
InputStream in = urlImage.openStream();
byte[] buffer = new byte[4096];
int n = -1;
OutputStream os = new FileOutputStream( IMAGE_DESTINATION_FOLDER + "/" + strImageName );
while ( (n = in.read(buffer)) != -1 ){
os.write(buffer, 0, n);
}
//close the stream
os.close();
} catch (IOException e) {
System.out.println("sponsored product");
}
// for loop images
}

I assume you are asking what to write instead of the ?? in the code in your question. Just create a counter variable.
int counter = 0;
for(Element imageElement : imageElements){
String strImageURL = imageElement.attr("src");
String strImageName = product_id + "_" + (++counter);
try {
URL urlImage = new URL(strImageURL);
InputStream in = urlImage.openStream();
byte[] buffer = new byte[4096];
int n = -1;
OutputStream os = new FileOutputStream( IMAGE_DESTINATION_FOLDER + "/" + strImageName );
while ( (n = in.read(buffer)) != -1 ){
os.write(buffer, 0, n);
}
//close the stream
os.close();
} catch (IOException e) {
System.out.println("sponsored product");
}
// for loop images
}

How to download a remote file using Java

I'm trying to download a single file from a web server (http or https) using as few third party libraries as possible.
The method I've come up with is as follows:
private static final int BUFFER_SIZE = 8;
public static boolean download(URL url, File f) throws IOException {
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
FileOutputStream out = new FileOutputStream(f);
BufferedInputStream in = new BufferedInputStream(conn.getInputStream());
byte[] buffer;
long dld = 0, expected = conn.getContentLengthLong(); // TODO expected will be -1 if the content length is unknown
while (true) { // TODO fix endless loop if server timeout
buffer = new byte[BUFFER_SIZE];
int n = in.read(buffer);
if (n == -1) break;
else dld += n;
out.write(buffer);
}
out.close();
System.out.println(dld + "B transmitted to " + f.getAbsolutePath());
return true;
}
However, it does by no means work as intended. I tried to download https://upload.wikimedia.org/wikipedia/commons/6/6d/Rubber_Duck_Florentijn_Hofman_Hong_Kong_2013d.jpg for example, the result was horrifying:
For some reason I was able to view the picture in IrfanView but not in any other viewer, so this is a re saved version.
I tried messing with the buffer size or downloading other images but the results are more or less the same.
If I look at the file, there are entire parts of the content simply replaced with dots:
I'm really lost on this one so thanks for any help :)

The problem occurs when there aren't 8 bytes of data to read. This leaves part of the array filled with zeros, which is why you're seeing so many in your hex editor. The solution is simple: replace out.write(buffer); with out.write(buffer, 0, n);. This tells the FileOutputStream to only read the bytes between indexes 0 and n.
Fixed code:
private static final int BUFFER_SIZE = 8;
public static boolean download(URL url, File f) throws IOException {
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
FileOutputStream out = new FileOutputStream(f);
BufferedInputStream in = new BufferedInputStream(conn.getInputStream());
// We can move the buffer declaration outside the loop
byte[] buffer = new byte[BUFFER_SIZE];
long dld = 0, expected = conn.getContentLengthLong(); // TODO expected will be -1 if the content length is unknown
while (true) {
int n = in.read(buffer);
if (n == -1) break;
else dld += n;
out.write(buffer, 0, n);
}
out.close();
System.out.println(dld + "B transmitted to " + f.getAbsolutePath());
return true;
}

Try something like this to download pictures
public static byte[] download(String param) throws IOException {
InputStream in = null;
ByteArrayOutputStream out = null;
try {
URL url = new URL(param);
HttpURLConnection con = (HttpURLConnection)url.openConnection();
con.setConnectTimeout(120000);
con.setReadTimeout(120000);
con.setRequestMethod("GET");
con.connect();
in = new BufferedInputStream(con.getInputStream());
out = new ByteArrayOutputStream();
byte[] buf = new byte[1024];
int n = 0;
while (-1 != (n = in.read(buf))) {
out.write(buf, 0, n);
}
return out.toByteArray();
} finally {
try {
out.close();
} catch (Exception e1) {
}
try {
in.close();
} catch (Exception e2) {
}
}
}

Cannot download file from URL in java

I'm making a program that will download files from URL. The downloading always starts, but it is not completed. For example, if file's size is 3 MB, program download only half of that so I cannot open the downloaded file. But program says that file is downloaded succesfully.
public class FileDownloader {
public static void main (String [] args) throws IOException {
InputStream fileIn;
FileOutputStream fileOut;
Scanner s = new Scanner(System.in);
System.out.println("Enter URL: ");
String urlStr = s.nextLine();
URL url = new URL(urlStr);
URLConnection urlConnect = url.openConnection();
fileIn = urlConnect.getInputStream();
System.out.println("Enter file name: ");
String fileStr = s.nextLine();
fileOut = new FileOutputStream(fileStr);
while (fileIn.read() != -1) {
fileOut.write(fileIn.read());
}
System.out.println("File is downloaded");
}
}
So how can I solve it? Should use another way to download?

You are losing every alternate bytedue to
while (fileIn.read() != -1) { //1st read
fileOut.write(fileIn.read()); //2nd read - 1st write
}
You are reading twice and writing only once.
What you need to do is
int x;
while ((x = fileIn.read()) != -1) { //1st read
fileOut.write(x); //1st write
}
Here is your complete code
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.URL;
import java.net.URLConnection;
import java.util.Scanner;
public class FileDownloader {
public static void main(String[] args) throws IOException {
InputStream fileIn;
FileOutputStream fileOut;
Scanner s = new Scanner(System.in);
System.out.println("Enter URL: ");
String urlStr = s.nextLine();
URL url = new URL(urlStr);
URLConnection urlConnect = url.openConnection();
fileIn = urlConnect.getInputStream();
System.out.println("Enter file name: ");
String fileStr = s.nextLine();
fileOut = new FileOutputStream(fileStr);
int x;
while ((x = fileIn.read()) != -1) {
fileOut.write(x);
}
System.out.println("File is downloaded");
}

You can download a large file with below code efficiently.
public static void main(String[] args) throws IOException {
InputStream in = null;
FileOutputStream out = null;
try {
System.out.println("Starting download");
long t1 = System.currentTimeMillis();
URL url = new URL(args[0]);// or you can hard code the URL
// Open the input and out files for the streams
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
in = conn.getInputStream();
out = new FileOutputStream(args[1]);//// or you can hard code the filename
// Read data into buffer and then write to the output file
byte[] buffer = new byte[8192];
int bytesRead;
while ((bytesRead = in.read(buffer)) != -1) {
out.write(buffer, 0, bytesRead);
}
long t2 = System.currentTimeMillis();
System.out.println("Time for download & save file in millis:"+(t2-t1));
} catch (Exception e) {
// Display or throw the error
System.out.println("Erorr while execting the program: "
+ e.getMessage());
} finally {
// Close the resources
if (in != null) {
in.close();
}
if (out != null) {
out.close();
}
}
}

just simply use this：
import org.apache.commons.io.FileUtils;
import java.net.URL;
String path = "F:/"
String fileName = "song"
FileUtils.copyURLToFile(myUrl, new File(path + fileName + ".mp3"));

java IRC client file transfer

i'm stuck with a small problem that i can't solve .... i need to create an app in java that connects to irc server and have the ability to transfer a file to another client with in a specific channel. So i tried this below code and it doesnt work out,my problem is in making the socket to transfer the file. and at this given code i make the irc client to send you a file when you send a message "sendFile" to it, but it doesnt send the file named "any.txt" to the sender of the message. so what can i possibly do to transfer the file to the other client ??? .... and almost forgot ... when you try to send the command "sendFile" to the irc client, you must have a nickname "mer" without the "".
import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.OutputStreamWriter;
import java.io.InputStreamReader;
import java.net.Socket;
import java.net.ServerSocket;
import java.io.File;
import java.io.FileInputStream;
import java.net.InetAddress;
public class MainTest {
private static String nickUse;
public static void main(String args[]) throws Exception{
// The server to connect to and our details.
File fileIn = new File("any.txt");
String server = "localhost";
String nick = "testJava";
String login = "anyName";
// The channel which the bot will join.
String channel = "#here";
// Connect directly to the IRC server.
ServerSocket serverSoc = new ServerSocket(0);
Socket socket = new Socket(server, 6667);
InetAddress intetAdd = socket.getInetAddress();
BufferedWriter writer = new BufferedWriter(
new OutputStreamWriter(socket.getOutputStream( )));
BufferedReader reader = new BufferedReader(
new InputStreamReader(socket.getInputStream( )));
// Log on to the server.
writer.write("NICK " + nick + "\r\n");
writer.write("USER " + login + " 8 * : This is a channel\r\n");
writer.flush( );
// Read lines from the server until it tells us we have connected.
String line = null;
while ((line = reader.readLine( )) != null) {
if (line.indexOf("004") >= 0) {
// We are now logged in.
break;
}
else if (line.indexOf("433") >= 0) {
System.out.println("Nickname is already in use.");
return;
}
}
// Join the channel.
writer.write("JOIN " + channel + "\r\n");
writer.flush( );
// Keep reading lines from the server.
while ((line = reader.readLine( )) != null) {
if (line.startsWith("PING ")) {
// We must respond to PINGs to avoid being disconnected.
writer.write("PONG " + line.substring(5) + "\r\n");
writer.flush();
System.out.println("THis is the line recieved when server sends a ping verification "+line);
}
else {
// Print the raw line received by the bot.
System.out.println(line);
if(line.contains(":") && line.contains("!")){
int positionOfIni = line.indexOf(":");
int lastOf = line.indexOf("!");
String nickComm = line.substring(positionOfIni+1,lastOf);
if(!nickComm.equalsIgnoreCase("mer")){
nickUse = nickComm;
}
}
if(nickUse!=null && line.endsWith(nickUse) == false){
int messagePo = line.lastIndexOf(":");
System.out.printf("%s %s %s\n",nickUse,"Says:",line.substring(messagePo+1));
nickUse = null;
}
if(line.endsWith("sendFile")){
byte[] add = intetAdd.getAddress();
writer.write("PRIVMSG " + "mer" +" :\u0001"+ "DCC SEND "+fileIn.getName()+" "+ipToLong(add)+" "+serverSoc.getLocalPort()+" "+fileIn.length()+"\u0001");
writer.flush();
Socket serSoc = serverSoc.accept();
serSoc.setSoTimeout(30000);
serverSoc.close();
BufferedOutputStream output = new BufferedOutputStream(serSoc.getOutputStream());
BufferedInputStream input = new BufferedInputStream(serSoc.getInputStream());
BufferedInputStream finput = new BufferedInputStream(new FileInputStream(fileIn));
byte[] outBuffer = new byte[1024];
byte[] inBuffer = new byte[4];
int bytesRead = 0;
while ((bytesRead = finput.read(outBuffer, 0, outBuffer.length)) != -1) {
output.write(outBuffer, 0, bytesRead);
output.flush();
input.read(inBuffer, 0, inBuffer.length);
Thread.sleep(4);
}
}
}
}
}
public static long ipToLong(byte[] address) {
if (address.length != 4) {
throw new IllegalArgumentException("byte array must be of length 4");
}
long ipNum = 0;
long multiplier = 1;
for (int i = 3; i >= 0; i--) {
int byteVal = (address[i] + 256) % 256;
ipNum += byteVal*multiplier;
multiplier *= 256;
}
System.out.println(ipNum);
return ipNum;
}
}

i just conclude the answer of having not to transfer with the help of IRC server. but instead i just create it's own server, that is capable of receiving files (Encrypted).

Download file by passing URL using java code [duplicate]

This question already has answers here:
How can I download and save a file from the Internet using Java?
(23 answers)
Closed 4 years ago.
I am trying to write a code in java in which user provide a url link and the program take url link and download a web page as it is and save at particular location..same as save as... option available on webpage.
Please can anybody help me
Thanks in advance

// Sample URL : http://www.novell.com/coolsolutions/tools/downloads/ntradping.zip
import java.io.*;
import java.net.*;
public class UrlDownload {
final static int size = 1024;
public static void fileUrl(String fAddress, String localFileName, String destinationDir) {
OutputStream outStream = null;
URLConnection uCon = null;
InputStream is = null;
try {
URL url;
byte[] buf;
int byteRead, byteWritten = 0;
url = new URL(fAddress);
outStream = new BufferedOutputStream(new FileOutputStream(destinationDir + "\\" + localFileName));
uCon = url.openConnection();
is = uCon.getInputStream();
buf = new byte[size];
while ((byteRead = is.read(buf)) != -1) {
outStream.write(buf, 0, byteRead);
byteWritten += byteRead;
}
System.out.println("Downloaded Successfully.");
System.out.println("File name:\"" + localFileName + "\"\nNo ofbytes :" + byteWritten);
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
is.close();
outStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
public static void fileDownload(String fAddress, String destinationDir) {
int slashIndex = fAddress.lastIndexOf('/');
int periodIndex = fAddress.lastIndexOf('.');
String fileName = fAddress.substring(slashIndex + 1);
if (periodIndex >= 1 && slashIndex >= 0 && slashIndex < fAddress.length() - 1) {
fileUrl(fAddress, fileName, destinationDir);
} else {
System.err.println("path or file name.");
}
}
public static void main(String[] args) {
if (args.length == 2) {
for (int i = 1; i < args.length; i++) {
fileDownload(args[i], args[0]);
}
} else {
}
}
}
It is working fully.

You can use Java URL API to get an input stream on the URL then read the from it and write through output stream on a file.
see read data from url, Write to file

Have a look at the HtmlParser. It has some features that will help you extract resources from a web page.