We Keep Coding

Return file from Google Cloud Storage through REST API without downloading locally first

We would like to have a Java REST API to return files from Google Cloud Storage as attachment. I was able to able to get it to work using the following method. The problem is that the file has to be downloaded locally to the service container (we are deploying on Google Cloud Run) and this is a problem in the case of very large files, and may generally be bad practice. Is there a way to modify this code somehow to skip the creation of a local file?
#GetMapping(path = "/file", produces = MediaType.APPLICATION_OCTET_STREAM_VALUE)
public ResponseEntity<InputStreamResource> getSpecificFile(#RequestParam String fileName,
#RequestParam String bucketName, #RequestParam String projectName) {
Storage storage = StorageOptions.newBuilder().setProjectId(projectId).build().getService();
Blob blob = storage.get(bucketName, fileName);
ReadChannel readChannel = blob.reader();
String outputFileName = tempFileDestination.concat("\\").concat(fileName);
try (FileOutputStream fileOutputStream = new FileOutputStream(outputFileName)) {
fileOutputStream.getChannel().transferFrom(readChannel, 0, Long.MAX_VALUE);
String contentType = Files.probeContentType(Paths.get(outputFileName));
FileInputStream fileInputStream = new FileInputStream(outputFileName);
return ResponseEntity.ok().contentType(MediaType.valueOf(contentType))
.header("Content-Disposition", "attachment; filename=" + fileName)
.body(new InputStreamResource(fileInputStream));
} catch (IOException e) {
e.printStackTrace();
return ResponseEntity.internalServerError().body(null);
} finally {
// delete the local file as cleanup
try {
Files.delete(Paths.get(outputFileName));
} catch (IOException e) {
e.printStackTrace();
}
}
}

Well, that did not take me long to figure out. I was able to make it work as follows:
#GetMapping(path = "/file", produces = MediaType.APPLICATION_OCTET_STREAM_VALUE)
public ResponseEntity<InputStreamResource> getSpecificFile(#RequestParam String fileName, #RequestParam String bucketName, #RequestParam String projectName) {
Storage storage = StorageOptions.newBuilder().setProjectId(projectId).build().getService();
Blob blob = storage.get(bucketName, fileName);
ReadChannel readChannel = blob.reader();
try {
String contentType = Files.probeContentType(Paths.get(fileName));
InputStream inputStream = Channels.newInputStream(readChannel);
return ResponseEntity.ok().contentType(MediaType.valueOf(contentType))
.header("Content-Disposition", "attachment; filename=" + fileName)
.body(new InputStreamResource(inputStream));
} catch (IOException e) {
e.printStackTrace();
return ResponseEntity.internalServerError().body(null);
}
}
Basically redirect the InputStream to the readChannel instead of the file.

How to write the file from MultipartHttpServletRequest in java

As my backend end team will send the zip file , how to receive the file and store into the project location in java(eclipse)
As im tried some code using MultipartHttpServletRequest and getting multipart length and zip folder also created but when unable to extract that showing invalid. then how to write into the file . Please help me
#RequestMapping(value = "/retrieveBillerByFile1", method = RequestMethod.POST)
public #ResponseBody void retrieveBillerByFile1(#RequestPart MultipartHttpServletRequest request) throws Exception
{
System.out.println("RESPONSEEEE**"+request);
Iterator<String> itrator = request.getFileNames();
System.out.println("File Name:" + request.getFileNames());
MultipartFile multiFile = request.getFile(itrator.next());
System.out.println("itrator.next()" + itrator.next());
try {
// just to show that we have actually received the file
System.out.println("File Length:" + multiFile.getBytes().length);
String name = multiFile.getOriginalFilename();
System.out.println("name" + name);
System.out.println("multiFile.getBytes()" + multiFile.getBytes());
BufferedWriter w = Files.newBufferedWriter(Paths.get("D:\\cedge_uat\\" + name ));
w.write(new String(multiFile.getBytes()));
w.flush();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
throw new Exception("Error while loading the file");
}
}
Another Way I have tried but fileItem getting as null
#RequestMapping(value = "/retrieveBillerByFile", method = RequestMethod.POST)
public #ResponseBody void retrieveBillerByFile(#RequestPart HttpServletRequest request,
HttpServletResponse response) throws Exception
{
System.out.println("RESPONSEEEE**"+request);
System.out.println("request**"+request);
// checks if the request actually contains upload file
if (!ServletFileUpload.isMultipartContent(request)) {
System.out.println("Form must has enctype=multipart/form-data.**");
// if not, we stop here
PrintWriter writer = response.getWriter();
writer.println("Error: Form must has enctype=multipart/form-data.");
writer.flush();
return;
}
// configures upload settings
DiskFileItemFactory factory = new DiskFileItemFactory();
// sets memory threshold - beyond which files are stored in disk
factory.setSizeThreshold(MEMORY_THRESHOLD);
// sets temporary location to store files
factory.setRepository(new File(System.getProperty("java.io.tmpdir")));
ServletFileUpload upload = new ServletFileUpload(factory);
// sets maximum size of upload file
upload.setFileSizeMax(MAX_FILE_SIZE);
// sets maximum size of request (include file + form data)
upload.setSizeMax(MAX_REQUEST_SIZE);
// constructs the directory path to store upload file
// this path is relative to application's directory
// String uploadPath = servletContext.getRealPath("")+ File.separator + UPLOAD_DIRECTORY;
String uploadPath = servletContext.getRealPath("/WEB-INF/xml") + File.separator + UPLOAD_DIRECTORY;
// creates the directory if it does not exist
File uploadDir = new File(uploadPath);
if (!uploadDir.exists()) {
uploadDir.mkdir();
}
System.out.println("uploadPath\n"+ uploadPath);
try {
// parses the request's content to extract file data
#SuppressWarnings("unchecked")
List<FileItem> formItems = upload.parseRequest(request);
System.out.println("formItems**"+formItems);
if (formItems != null && formItems.size() > 0) {
// iterates over form's fields
for (FileItem item : formItems) {
System.out.println("item**"+item);
// processes only fields that are not form fields
if (!item.isFormField()) {
String fileName = new File(item.getName()).getName();
System.out.println("fileName**"+fileName);
String filePath = uploadPath + File.separator + fileName;
File storeFile = new File(filePath);
// saves the file on disk
item.write(storeFile);
System.out.println("success");
request.setAttribute("message",
"Upload has been done successfully!");
}
}
}
} catch (Exception ex) {
System.out.println("exception"+ ex.getMessage());
request.setAttribute("message",
"There was an error: " + ex.getMessage());
}
}

You can add another param in the method --
#RequestParam MultipartFile file
so your method will be --
public #ResponseBody void retrieveBillerByFile1(#RequestParam MultipartFile file, #RequestPart MultipartHttpServletRequest request) throws Exception
And then manipulate the file using java file apis

Zip file cannot be unzipped after uploading via Dropwizard framework

I faced with the issue that after uploading zip file on a server it cannot be unzipped.
I have REST API based on Dropwizard framework with next endpoint example:
#Consumes(MediaType.MULTIPART_FORM_DATA)
#Produces(MediaType.APPLICATION_JSON)
#Path("/zip")
public class ImportResource {
#POST
public Response fileService(#FormDataParam("fileData") InputStream fileDataInputStream,
#FormDataParam("fileData") FormDataContentDisposition fileDataDetail) {
File newFile = new File("/Users/alexx/Documents/"+ fileDataDetail.getFileName());
try {
final OutputStream out = new FileOutputStream(newFile);
ByteStreams.copy(fileDataInputStream, out);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return Response.ok().build();
}
// save uploaded file to new location
private void writeToFile(InputStream uploadedInputStream, String uploadedFileLocation) throws IOException {
int read;
final int BUFFER_LENGTH = 1024;
final byte[] buffer = new byte[BUFFER_LENGTH];
OutputStream out = new FileOutputStream(new File(uploadedFileLocation));
while ((read = uploadedInputStream.read(buffer)) != -1) {
out.write(buffer, 0, read);
}
out.flush();
out.close();
}
Also in Application class I provided appropriate classes:
....
bootstrap.addBundle(new MultiPartBundle());
....
environment.jersey().register(MultiPartFeature.class);
After uploading zip file, it looks like it uploads, but it cannot be unzipped, next message received:
(Error 1 - Operation not permitted).
Text and image files uploads and opens correctly.
Did I skip anything? Should I add additional params or setting somewhere?
Thank you for answers!

It was my mistake.
I found the filter which works with payload before it comes to a resource.
The filter has next code:
payload = IOUtils.toString(requestContext.getEntityStream(), StandardCharsets.UTF_8);
if (payload != null) {
RequestPayloadHolder.getRequestPayload().setPayload(payload);
InputStream in = IOUtils.toInputStream(payload, StandardCharsets.UTF_8);
requestContext.setEntityStream(in);
}
So entity with multipart/form-data cannot be transferred to String and otherwise correctly.
Before this request implementation all other request payloads were JSON.
I have to be more careful.

How to download a file and get the path location locally

I have a URL i.e http://downloadplugins.verify.com/Windows/SubAngle.exe .
If I paste it on the tab and press enter then the file (SubAngle.exe) is getting downloaded and saved in the download folder. This is a manual process. But it can be done with java code.
I wrote the code for getting the absolute path with the help of the file name i.e SubAngle.exe.
Requirement:- With the help of the URL file gets downloaded,Verify the file has been downloaded and returns the absolute path of the file.
where locfile is "http://downloadplugins.verify.com/Windows/SubAngle.exe"
public String downloadAndVerifyFile(String locfile) {
File fileLocation = new File(locfile);
File fileLocation1 = new File(fileLocation.getName());
String fileLocationPath = null;
if(fileLocation.exists()){
fileLocationPath = fileLocation1.getAbsolutePath();
}
else{
throw new FileNotFoundException("File with name "+locFile+" may not exits at the location");
}
return fileLocationPath;
}

easy and general function that im using:
import org.apache.commons.io.FileUtils;
public static void downLoadFile(String fromFile, String toFile) throws MalformedURLException, IOException {
try {
FileUtils.copyURLToFile(new URL(fromFile), new File(toFile), 60000, 60000);
} catch (FileNotFoundException e) {
e.printStackTrace();
System.out.println("exception on: downLoadFile() function: " + e.getMessage());
}
}

Instead of writing this huge code, go for Apache's commons.io
Try this:
URL ipURL = new URL("inputURL");
File opFile = new File("outputFile");
FileUtils.copyURLToFile(ipURL, opFile);

Code to DownloadFile from URL
import java.net.*;
import java.io.*;
public class DownloadFile {
public static void main(String[] args) throws IOException {
InputStream in = null;
FileOutputStream out = null;
try {
// URL("http://downloadplugins.verify.com/Windows/SubAngle.exe");
System.out.println("Starting download");
long t1 = System.currentTimeMillis();
URL url = new URL(args[0]);
// Open the input and out files for the streams
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
in = conn.getInputStream();
out = new FileOutputStream("YourFile.exe");
// Read data into buffer and then write to the output file
byte[] buffer = new byte[8192];
int bytesRead;
while ((bytesRead = in.read(buffer)) != -1) {
out.write(buffer, 0, bytesRead);
}
long t2 = System.currentTimeMillis();
System.out.println("Time for download & save file in millis:"+(t2-t1));
} catch (Exception e) {
// Display or throw the error
System.out.println("Erorr while execting the program: "
+ e.getMessage());
} finally {
// Close the resources correctly
if (in != null) {
in.close();
}
if (out != null) {
out.close();
}
}
}
}
Configure the value of fileName properly to know where the file is getting stored.
Source: http://www.devmanuals.com/tutorials/java/corejava/files/java-read-large-file-efficiently.html
The source was modified to replace local file with http URL
Output:
java DownloadFile http://download.springsource.com/release/TOOLS/update/3.7.1.RELEASE/e4.5/springsource-tool-suite-3.7.1.RELEASE-e4.5.1-updatesite.zip
Starting download
Time for download & save file in millis:100184

Password protected zip file in java

I have created zip file using java as below snippet
import java.io.*;
import java.util.zip.*;
public class ZipCreateExample {
public static void main(String[] args) throws IOException {
System.out.print("Please enter file name to zip : ");
BufferedReader input = new BufferedReader
(new InputStreamReader(System.in));
String filesToZip = input.readLine();
File f = new File(filesToZip);
if(!f.exists()) {
System.out.println("File not found.");
System.exit(0);
}
System.out.print("Please enter zip file name : ");
String zipFileName = input.readLine();
if (!zipFileName.endsWith(".zip"))
zipFileName = zipFileName + ".zip";
byte[] buffer = new byte[18024];
try {
ZipOutputStream out = new ZipOutputStream
(new FileOutputStream(zipFileName));
out.setLevel(Deflater.DEFAULT_COMPRESSION);
FileInputStream in = new FileInputStream(filesToZip);
out.putNextEntry(new ZipEntry(filesToZip));
int len;
while ((len = in.read(buffer)) > 0) {
out.write(buffer, 0, len);
}
out.closeEntry();
in.close();
out.close();
} catch (IllegalArgumentException iae) {
iae.printStackTrace();
System.exit(0);
} catch (FileNotFoundException fnfe) {
fnfe.printStackTrace();
System.exit(0);
} catch (IOException ioe) {
ioe.printStackTrace();
System.exit(0);
}
}
}
Now I want when I click on the zip file it should prompt me to type password and then decompress the zip file.
Please any help,How should I go further?

Try the following code which is based on Zip4j:
import net.lingala.zip4j.core.ZipFile;
import net.lingala.zip4j.exception.ZipException;
import net.lingala.zip4j.model.ZipParameters;
import net.lingala.zip4j.util.Zip4jConstants;
import org.apache.commons.io.FilenameUtils;
import java.io.File;
public class Zipper
{
private String password;
private static final String EXTENSION = "zip";
public Zipper(String password)
{
this.password = password;
}
public void pack(String filePath) throws ZipException
{
ZipParameters zipParameters = new ZipParameters();
zipParameters.setCompressionMethod(Zip4jConstants.COMP_DEFLATE);
zipParameters.setCompressionLevel(Zip4jConstants.DEFLATE_LEVEL_ULTRA);
zipParameters.setEncryptFiles(true);
zipParameters.setEncryptionMethod(Zip4jConstants.ENC_METHOD_AES);
zipParameters.setAesKeyStrength(Zip4jConstants.AES_STRENGTH_256);
zipParameters.setPassword(password);
String baseFileName = FilenameUtils.getBaseName(filePath);
String destinationZipFilePath = baseFileName + "." + EXTENSION;
ZipFile zipFile = new ZipFile(destinationZipFilePath);
zipFile.addFile(new File(filePath), zipParameters);
}
public void unpack(String sourceZipFilePath, String extractedZipFilePath) throws ZipException
{
ZipFile zipFile = new ZipFile(sourceZipFilePath + "." + EXTENSION);
if (zipFile.isEncrypted())
{
zipFile.setPassword(password);
}
zipFile.extractAll(extractedZipFilePath);
}
}
FilenameUtils is from Apache Commons IO.
Example usage:
public static void main(String[] arguments) throws ZipException
{
Zipper zipper = new Zipper("password");
zipper.pack("encrypt-me.txt");
zipper.unpack("encrypt-me", "D:\\");
}

Standard Java API does not support password protected zip files. Fortunately good guys have already implemented such ability for us. Please take a look on this article that explains how to create password protected zip.
(The link was dead, latest archived version: https://web.archive.org/web/20161029174700/http://java.sys-con.com/node/1258827)

Sample code below will zip and password protect your file.
This REST service accepts bytes of the original file. It zips the byte array and password protects it. Then it sends bytes of password protected zipped file as response. The code is a sample of sending and receiving binary bytes to and from a REST service, and also of zipping a file with password protect. The bytes are zipped from stream, so no files are ever stored on the server.
Uses JAX-RS API using Jersey API in java
Client is using Jersey-client API.
Uses zip4j 1.3.2 open source library, and apache commons io.
#PUT
#Path("/bindata/protect/qparam")
#Consumes(MediaType.APPLICATION_OCTET_STREAM)
#Produces(MediaType.APPLICATION_OCTET_STREAM)
public Response zipFileUsingPassProtect(byte[] fileBytes, #QueryParam(value = "pass") String pass,
#QueryParam(value = "inputFileName") String inputFileName) {
System.out.println("====2001==== Entering zipFileUsingPassProtect");
System.out.println("fileBytes size = " + fileBytes.length);
System.out.println("password = " + pass);
System.out.println("inputFileName = " + inputFileName);
byte b[] = null;
try {
b = zipFileProtected(fileBytes, inputFileName, pass);
} catch (IOException e) {
e.printStackTrace();
return Response.status(Status.INTERNAL_SERVER_ERROR).build();
}
System.out.println(" ");
System.out.println("++++++++++++++++++++++++++++++++");
System.out.println(" ");
return Response.ok(b, MediaType.APPLICATION_OCTET_STREAM)
.header("content-disposition", "attachment; filename = " + inputFileName + ".zip").build();
}
private byte[] zipFileProtected(byte[] fileBytes, String fileName, String pass) throws IOException {
ByteArrayInputStream inputByteStream = null;
ByteArrayOutputStream outputByteStream = null;
net.lingala.zip4j.io.ZipOutputStream outputZipStream = null;
try {
//write the zip bytes to a byte array
outputByteStream = new ByteArrayOutputStream();
outputZipStream = new net.lingala.zip4j.io.ZipOutputStream(outputByteStream);
//input byte stream to read the input bytes
inputByteStream = new ByteArrayInputStream(fileBytes);
//init the zip parameters
ZipParameters zipParams = new ZipParameters();
zipParams.setCompressionMethod(Zip4jConstants.COMP_DEFLATE);
zipParams.setCompressionLevel(Zip4jConstants.DEFLATE_LEVEL_NORMAL);
zipParams.setEncryptFiles(true);
zipParams.setEncryptionMethod(Zip4jConstants.ENC_METHOD_STANDARD);
zipParams.setPassword(pass);
zipParams.setSourceExternalStream(true);
zipParams.setFileNameInZip(fileName);
//create zip entry
outputZipStream.putNextEntry(new File(fileName), zipParams);
IOUtils.copy(inputByteStream, outputZipStream);
outputZipStream.closeEntry();
//finish up
outputZipStream.finish();
IOUtils.closeQuietly(inputByteStream);
IOUtils.closeQuietly(outputByteStream);
IOUtils.closeQuietly(outputZipStream);
return outputByteStream.toByteArray();
} catch (ZipException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
IOUtils.closeQuietly(inputByteStream);
IOUtils.closeQuietly(outputByteStream);
IOUtils.closeQuietly(outputZipStream);
}
return null;
}
Unit test below:
#Test
public void testPassProtectZip_with_params() {
byte[] inputBytes = null;
try {
inputBytes = FileUtils.readFileToByteArray(new File(inputFilePath));
} catch (IOException e) {
e.printStackTrace();
}
System.out.println("bytes read into array. size = " + inputBytes.length);
Client client = ClientBuilder.newClient();
WebTarget target = client.target("http://localhost:8080").path("filezip/services/zip/bindata/protect/qparam");
target = target.queryParam("pass", "mypass123");
target = target.queryParam("inputFileName", "any_name_here.pdf");
Invocation.Builder builder = target.request(MediaType.APPLICATION_OCTET_STREAM);
Response resp = builder.put(Entity.entity(inputBytes, MediaType.APPLICATION_OCTET_STREAM));
System.out.println("response = " + resp.getStatus());
Assert.assertEquals(Status.OK.getStatusCode(), resp.getStatus());
byte[] zipBytes = resp.readEntity(byte[].class);
try {
FileUtils.writeByteArrayToFile(new File(responseFilePathPasswordZipParam), zipBytes);
} catch (IOException e) {
e.printStackTrace();
}
}
Feel free to use and modify. Please let me know if you find any errors. Hope this helps.
Edit 1 - Using QueryParam but you may use HeaderParam for PUT instead to hide passwd from plain sight. Modify the test method accordingly.
Edit 2 - REST path is filezip/services/zip/bindata/protect/qparam
filezip is name of war. services is the url mapping in web.xml. zip is class level path annotation. bindata/protect/qparam is the method level path annotation.

In new version of Zip4j, class Zip4jConstants was removed. Use EncryptionMethod and AesKeyStrength class instead. Documentation : https://github.com/srikanth-lingala/zip4j
ZipParameters zipParameters = new ZipParameters();
zipParameters.setEncryptFiles(true);
zipParameters.setEncryptionMethod(EncryptionMethod.AES);
zipParameters.setAesKeyStrength(AesKeyStrength.KEY_STRENGTH_256);
List<File> filesToAdd = Arrays.asList(
new File("somefile"),
new File("someotherfile")
);
ZipFile zipFile = new ZipFile("filename.zip", "password".toCharArray());
zipFile.addFiles(filesToAdd, zipParameters);

There is no default Java API to create a password protected file. There is another example about how to do it here.

Library Zip4J seems to be the preferred answer.
In case the privacy of the password is highly recommended, one might close a security gap in class ZipFile, which carries the password in plain text, even after the ZipFile is closed. Following method destroys the password.
public static void destroyZipPassword(ZipFile zip) throws DestroyFailedException
{
try
{
Field fdPwd = ZipFile.class.getDeclaredField("password");
fdPwd.setAccessible(true);
char[] password = (char[]) fdPwd.get(zip);
Arrays.fill(password, (char) 0);
}
catch (Exception e)
{
e.printStackTrace();
throw new DestroyFailedException(e.getMessage());
}
}