We Keep Coding

Name the images that i get from a web site in Java

I am scraping a web site and as a last part, I get their product images to the folder. I want to name these images like (product_id + numberOfImages) I mean if product has a 2 images, there will be 2 png like (productId_1) (productId_2).
I have productId and also images there is no problem. I just want to know how to name it as I want. Here is my code.
for(Element imageElement : imageElements){
String strImageURL = imageElement.attr("src");
String strImageName =product_id + "_" + ??;
try {
URL urlImage = new URL(strImageURL);
InputStream in = urlImage.openStream();
byte[] buffer = new byte[4096];
int n = -1;
OutputStream os = new FileOutputStream( IMAGE_DESTINATION_FOLDER + "/" + strImageName );
while ( (n = in.read(buffer)) != -1 ){
os.write(buffer, 0, n);
}
//close the stream
os.close();
} catch (IOException e) {
System.out.println("sponsored product");
}
// for loop images
}

I assume you are asking what to write instead of the ?? in the code in your question. Just create a counter variable.
int counter = 0;
for(Element imageElement : imageElements){
String strImageURL = imageElement.attr("src");
String strImageName = product_id + "_" + (++counter);
try {
URL urlImage = new URL(strImageURL);
InputStream in = urlImage.openStream();
byte[] buffer = new byte[4096];
int n = -1;
OutputStream os = new FileOutputStream( IMAGE_DESTINATION_FOLDER + "/" + strImageName );
while ( (n = in.read(buffer)) != -1 ){
os.write(buffer, 0, n);
}
//close the stream
os.close();
} catch (IOException e) {
System.out.println("sponsored product");
}
// for loop images
}

How to download a remote file using Java

I'm trying to download a single file from a web server (http or https) using as few third party libraries as possible.
The method I've come up with is as follows:
private static final int BUFFER_SIZE = 8;
public static boolean download(URL url, File f) throws IOException {
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
FileOutputStream out = new FileOutputStream(f);
BufferedInputStream in = new BufferedInputStream(conn.getInputStream());
byte[] buffer;
long dld = 0, expected = conn.getContentLengthLong(); // TODO expected will be -1 if the content length is unknown
while (true) { // TODO fix endless loop if server timeout
buffer = new byte[BUFFER_SIZE];
int n = in.read(buffer);
if (n == -1) break;
else dld += n;
out.write(buffer);
}
out.close();
System.out.println(dld + "B transmitted to " + f.getAbsolutePath());
return true;
}
However, it does by no means work as intended. I tried to download https://upload.wikimedia.org/wikipedia/commons/6/6d/Rubber_Duck_Florentijn_Hofman_Hong_Kong_2013d.jpg for example, the result was horrifying:
For some reason I was able to view the picture in IrfanView but not in any other viewer, so this is a re saved version.
I tried messing with the buffer size or downloading other images but the results are more or less the same.
If I look at the file, there are entire parts of the content simply replaced with dots:
I'm really lost on this one so thanks for any help :)

The problem occurs when there aren't 8 bytes of data to read. This leaves part of the array filled with zeros, which is why you're seeing so many in your hex editor. The solution is simple: replace out.write(buffer); with out.write(buffer, 0, n);. This tells the FileOutputStream to only read the bytes between indexes 0 and n.
Fixed code:
private static final int BUFFER_SIZE = 8;
public static boolean download(URL url, File f) throws IOException {
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
FileOutputStream out = new FileOutputStream(f);
BufferedInputStream in = new BufferedInputStream(conn.getInputStream());
// We can move the buffer declaration outside the loop
byte[] buffer = new byte[BUFFER_SIZE];
long dld = 0, expected = conn.getContentLengthLong(); // TODO expected will be -1 if the content length is unknown
while (true) {
int n = in.read(buffer);
if (n == -1) break;
else dld += n;
out.write(buffer, 0, n);
}
out.close();
System.out.println(dld + "B transmitted to " + f.getAbsolutePath());
return true;
}

Try something like this to download pictures
public static byte[] download(String param) throws IOException {
InputStream in = null;
ByteArrayOutputStream out = null;
try {
URL url = new URL(param);
HttpURLConnection con = (HttpURLConnection)url.openConnection();
con.setConnectTimeout(120000);
con.setReadTimeout(120000);
con.setRequestMethod("GET");
con.connect();
in = new BufferedInputStream(con.getInputStream());
out = new ByteArrayOutputStream();
byte[] buf = new byte[1024];
int n = 0;
while (-1 != (n = in.read(buf))) {
out.write(buf, 0, n);
}
return out.toByteArray();
} finally {
try {
out.close();
} catch (Exception e1) {
}
try {
in.close();
} catch (Exception e2) {
}
}
}

image compression using s3 and lambda in java

I am trying to fetch image from s3 bucket of aws in lambda fuction and i want to compress this image and upload back to destination bucket of aws s3. I tried to fetch file from s3 and compress it.but facing problem in uploading the compressed file

Finally this worked
public class TempComp implements RequestHandler<S3Event, String> {
private static final float MAX_WIDTH = 100;
private static final float MAX_HEIGHT = 100;
private final String JPG_TYPE = (String) "jpg";
private final String JPG_MIME = (String) "image/jpeg";
private final String PNG_TYPE = (String) "png";
private final String PNG_MIME = (String) "image/png";
public String handleRequest(S3Event s3event, Context context) {
try {
S3EventNotificationRecord record = s3event.getRecords().get(0);
String srcBucket = record.getS3().getBucket().getName();
// Object key may have spaces or unicode non-ASCII characters.
String srcKey = record.getS3().getObject().getKey()
.replace('+', ' ');
srcKey = URLDecoder.decode(srcKey, "UTF-8");
String dstBucket = srcBucket + "resized";
String dstKey = "resized-" + srcKey;
// Sanity check: validate that source and destination are different
// buckets.
if (srcBucket.equals(dstBucket)) {
System.out
.println("Destination bucket must not match source bucket.");
return "";
}
// Infer the image type.
Matcher matcher = Pattern.compile(".*\\.([^\\.]*)").matcher(srcKey);
if (!matcher.matches()) {
System.out.println("Unable to infer image type for key "
+ srcKey);
return "";
}
String imageType = matcher.group(1);
if (!(JPG_TYPE.equals(imageType)) && !(PNG_TYPE.equals(imageType))) {
System.out.println("Skipping non-image " + srcKey);
return "";
}
// Download the image from S3 into a stream
AmazonS3 s3Client = new AmazonS3Client();
S3Object s3Object = s3Client.getObject(new GetObjectRequest(
srcBucket, srcKey));
InputStream objectData = s3Object.getObjectContent();
// int b=objectData.read();
// Read the source image
BufferedImage srcImage = ImageIO.read(objectData);
// BufferedImage destImage = srcImage;
Iterator<ImageWriter> writers = ImageIO.getImageWritersByFormatName("jpg");
if (!writers.hasNext())
throw new IllegalStateException("No writers found");
ImageWriter writer = (ImageWriter) writers.next();
ByteArrayOutputStream os=new ByteArrayOutputStream() ;
ImageOutputStream ios = ImageIO.createImageOutputStream(os);
writer.setOutput(ios);
ImageWriteParam param = writer.getDefaultWriteParam();
// compress to a given quality
param.setCompressionMode(ImageWriteParam.MODE_EXPLICIT);
param.setCompressionQuality(0.5f);
// appends a complete image stream containing a single image and
//associated stream and image metadata and thumbnails to the output
writer.write(null, new IIOImage(srcImage, null, null), param);
System.out.print("file is compressed");
ByteArrayInputStream is = new ByteArrayInputStream(os.toByteArray());
//InputStream is = new ByteArrayInputStream(retValue);
BufferedImage destImage = ImageIO.read(is);
ByteArrayOutputStream os1 = new ByteArrayOutputStream();
ImageIO.write(destImage, imageType, os1);
InputStream is1 = new ByteArrayInputStream(os1.toByteArray());
// Set Content-Length and Content-Type
ObjectMetadata meta = new ObjectMetadata();
meta.setContentLength(os1.size());
System.out.println(os1.size());
if (JPG_TYPE.equals(imageType)) {
meta.setContentType(JPG_MIME);
}
if (PNG_TYPE.equals(imageType)) {
meta.setContentType(PNG_MIME);
}
// Uploading to S3 destination bucket
System.out.println("Writing to: " + dstBucket + "/" + dstKey);
s3Client.putObject(dstBucket, dstKey, is1, meta);
System.out.println("Successfully resized " + srcBucket + "/"
+ srcKey + " and uploaded to " + dstBucket + "/" + dstKey);
return "Ok";
} catch (IOException e) {
throw new RuntimeException(e);
}
}
}

File only partially uploaded to server

I'm trying to upload a file to my Spring server running on Tomcat7. It's a simple POST request, the code is below:
#RequestMapping(method = RequestMethod.POST)
public void saveFile(HttpServletRequest request, #RequestParam("file_name") String fileName) {
Logger.getLogger(FileRestAction.class).info("saving file with name " + fileName);
try {
byte[] buf = readFromRequest(request);
String filePath = writeToFile(buf, fileName);
File_ file = new File_(filePath, request.getContentType());
Logger.getLogger(FileRestAction.class).info(request.getContentType() + " " + request.getContentLength());
fService.save(file);
} catch (IOException e) {
Logger.getLogger(FileRestAction.class).error("Failed to upload file. " +
"Exception is: " + e.getMessage());
}
}
private String writeToFile(byte[] buf, String fileName) throws IOException {
String fileBasePath = ConfigurationProvider.getConfig().getString(Const.FILE_SAVE_PATH);
File file = new File(fileBasePath + fileName);
FileOutputStream fos = new FileOutputStream(file);
fos.write(buf);
fos.close();
Logger.getLogger(FileRestAction.class).info("filepath: " + file.getAbsolutePath());
return file.getAbsolutePath();
}
private byte[] readFromRequest(HttpServletRequest request) throws IOException {
InputStream is = request.getInputStream();
byte[] buf = new byte[request.getContentLength()];
is.read(buf);
is.close();
return buf;
}
Now the problem is that the file on the server is only "half-done", it's as if all the bytes aren't there. For example, if I send a 256x256 .png-file with a size of 54kB, the file written on the server is also 54kB and 256x256 in size, but the actual picture cuts off near the beginning (the rest is blank). No exceptions are thrown.
After a bit of testing I've found out that the cutoff is around 15-20Kb (images below that are fully uploaded).
Any ideas as to what might cause this?
EDIT: I changed the readFromRequest method according to what GreyBeardedGeek suggested. It's now as follows:
private byte[] readFromRequest(HttpServletRequest request) throws IOException {
InputStream is = request.getInputStream();
int fileLength = (int) request.getContentLength();
byte[] buf = new byte[fileLength];
int bytesRead = 0;
while (true) {
bytesRead += is.read(buf, bytesRead, fileLength - bytesRead);
Logger.getLogger(FileRestAction.class).info("reading file: " + bytesRead + " bytes read");
if (bytesRead == fileLength) break;
}
is.close();
return buf;
}

InputStream.read is not guaranteed to read the amount of data that you ask for.
The size that you ask for is the maximum number of bytes that it will read.
That's why the read() method returns the number of bytes actually read.
See http://docs.oracle.com/javase/6/docs/api/java/io/InputStream.html#read(byte[])
So, the answer is to read multiple times, until read() returns -1

Uploading Image From Java Desktop App to Server

I am using the following code on the client side to upload to the server
public class UploaderExample{
private static final String Boundary = "--7d021a37605f0";
public void upload(URL url, List<File> files) throws Exception
{
HttpURLConnection theUrlConnection = (HttpURLConnection) url.openConnection();
theUrlConnection.setDoOutput(true);
theUrlConnection.setDoInput(true);
theUrlConnection.setUseCaches(false);
theUrlConnection.setChunkedStreamingMode(1024);
theUrlConnection.setRequestProperty("Content-Type", "multipart/form-data; boundary="
+ Boundary);
DataOutputStream httpOut = new DataOutputStream(theUrlConnection.getOutputStream());
for (int i = 0; i < files.size(); i++)
{
File f = files.get(i);
String str = "--" + Boundary + "\r\n"
+ "Content-Disposition: form-data;name=\"file" + i + "\"; filename=\"" + f.getName() + "\"\r\n"
+ "Content-Type: image/png\r\n"
+ "\r\n";
httpOut.write(str.getBytes());
FileInputStream uploadFileReader = new FileInputStream(f);
int numBytesToRead = 1024;
int availableBytesToRead;
while ((availableBytesToRead = uploadFileReader.available()) > 0)
{
byte[] bufferBytesRead;
bufferBytesRead = availableBytesToRead >= numBytesToRead ? new byte[numBytesToRead]
: new byte[availableBytesToRead];
uploadFileReader.read(bufferBytesRead);
httpOut.write(bufferBytesRead);
httpOut.flush();
}
httpOut.write(("--" + Boundary + "--\r\n").getBytes());
}
httpOut.write(("--" + Boundary + "--\r\n").getBytes());
httpOut.flush();
httpOut.close();
// read & parse the response
InputStream is = theUrlConnection.getInputStream();
StringBuilder response = new StringBuilder();
byte[] respBuffer = new byte[4096];
while (is.read(respBuffer) >= 0)
{
response.append(new String(respBuffer).trim());
}
is.close();
System.out.println(response.toString());
}
public static void main(String[] args) throws Exception
{
List<File> list = new ArrayList<File>();
list.add(new File("C:\\square.png"));
list.add(new File("C:\\narrow.png"));
UploaderExample uploader = new UploaderExample();
uploader.upload(new URL("http://systemout.com/upload.php"), list);
}
}
I have tried writing the servlet that receives the image file and saves it to a folder on the server....but have failed miserably...This is part of an academic project i need to submit as part of my degree....Please Help!!!
I want help ...can someone guide me on how the servlet will be written....
I tried the following:
response.setContentType("text/html;charset=UTF-8");
PrintWriter out = response.getWriter();
try {
InputStream input = null;
OutputStream output = null;
try {
input = request.getInputStream();
output = new FileOutputStream("C:\\temp\\file.png");
byte[] buffer = new byte[10240];
for (int length = 0; (length = input.read(buffer)) > 0 ; ) {
output.write(buffer, 0, length);
}
}
catch(Exception e){
out.println(e.getMessage());
}
finally {
if (output != null) {
output.close();
}
if (input != null) {
input.close();
}
}
out.println("Success");
}
catch(Exception e){
out.println(e.getMessage());
}
finally {
out.close();
}
}
I went ahead and tried the fileupload from apache.org....and wrote the following servlet code:
protected void processRequest(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
response.setContentType("text/html;charset=UTF-8");
PrintWriter out = response.getWriter();
try {
out.println(1);
boolean isMultipart = ServletFileUpload.isMultipartContent(request);
if (isMultipart) {
// Create a factory for disk-based file items
FileItemFactory factory = new DiskFileItemFactory();
// Create a new file upload handler
ServletFileUpload upload = new ServletFileUpload(factory);
// Parse the request
List /* FileItem */ items = upload.parseRequest(request);
// Process the uploaded items
Iterator iter = items.iterator();
while (iter.hasNext()) {
FileItem item = (FileItem) iter.next();
if (item.isFormField()) {
//processFormField(item);
} else {
//processUploadedFile(item);
String fieldName = item.getFieldName();
String fileName = item.getName();
String contentType = item.getContentType();
boolean isInMemory = item.isInMemory();
long sizeInBytes = item.getSize();
//write to file
File uploadedFile = new File("C:\\temp\\image.png");
item.write(uploadedFile);
out.println("Sucess!");
}
}
} else {
out.println("Invalid Content!");
}
} catch (Exception e) {
out.println(e.getMessage());
} finally {
out.close();
}
}
However i am still confused on how to write the multipart code on the client side...the one i posted above is not working with my servlet implementation.....help please....some links where i can learn writing posting multipart form from java desktop app would be useful

So here's my recommendation: don't write this code yourself! Use http://commons.apache.org/fileupload/ instead. It will save you a lot of headaches, and you'll be up and running quite quickly. I'm pretty sure that problem is that the InputStream contains the multi-part boundaries, and is thus not a valid image.
Here's another observation: since you're not doing any transformations on the image, there's no need to read and write the image bytes using ImageIO. You're better off writing the bytes straight from the InputStream to the file.