We Keep Coding

Is there any efficient method in springboot to upload large files? my current api is taking long time to upload large document

I am trying to upload a file of size 5gb in SpringBoot but it takes hours to let request get inside controller from postman. once request get inside controller than it doesn't take much time to upload the document. sometimes it returns connection timeout issue. can anyone provide an efficient way to get the task done in SpringBoot?
here is my controller
#RequestMapping(
value = "/store",
method = RequestMethod.POST,
consumes = MediaType.MULTIPART_FORM_DATA_VALUE,
produces = MediaType.APPLICATION_JSON_VALUE)
public FilesDTO storeFile(
#RequestHeader("Authorization") String auth,
#RequestParam("filename") String filename,
#RequestParam("collectorId") String collectorId,
#RequestParam(name = "description", required = false) String description,
#RequestParam("FileType") DataFileType collectorFileType,
#RequestPart("file") MultipartFile file)
throws PRValidationException, IOException {
return collectorFilesService.storeFile(auth, filename, collectorId, description, collectorFileType, file);
}
here is my Implemention
#Override
public CollectorFilesDTO storeCollectorFile(String auth, String filename, String collectorId, String description, DataFileType collectorFileType, MultipartFile file) throws IOException, PrValidationException {
dataServiceApi.findCollectorById(auth,collectorId);
if (file.isEmpty()) {
throw new FileNotFoundException(environment.getProperty("pr.validation.file_not_found"));
}
if(collectorFileType == null){
throw new PrValidationException(environment.getProperty("pr.validation.collectorFileType.not.null"));
}
return storeFile(auth, filename, collectorId,description,collectorFileType ,file.getBytes());
}
#Override
public CollectorFilesDTO storeFile(String auth, String filename, String collectorId,String description,DataFileType collectorFileType, byte[] bytes) throws IOException, PrValidationException {
logger.info("UPLOAD_FOLDER=" + UPLOAD_FOLDER_PATH);
logger.info("fileName=" + filename);
String absolutePath = UPLOAD_FOLDER_PATH;
logger.info("Writing " + absolutePath + filename);
final File parentDirectory = new File(absolutePath);
if (!parentDirectory.exists()) {
if (!parentDirectory.mkdirs()) {
throw new IOException(environment.getProperty("pr.validation.file_not_created"));
}
}
File file = new File(parentDirectory, filename);
boolean fileAlreadyExists = file.exists();
try {
Files.write(Paths.get(absolutePath + filename), bytes);
CollectorFileEntity collectorFileEntity = storeDocument(auth, filename,collectorId, description,collectorFileType ,fileAlreadyExists);
logger.info("file write success and document has been saved in {}ms", System.currentTimeMillis());
return CollectorFileMapper.INSTANCE.toDto(collectorFileEntity);
} catch (IOException e) {
logger.info(e.getMessage());
throw new IOException(environment.getProperty("pr.validation.write_file_exception"));
}
}

The Apache Commons FileUpload is a popular tool for this use case, here's a guide of how to use it.

How to copy/rename a file and return it without altering the existing file?

I have a generic.exe file which doesn't contain any users detail in it.
I also have a REST API which takes the userId and returns a File to the client.
Now, what we want to implement in our project is that when someone hits the REST API, we want to take that generic.exe and rename it to manager_userId.exe and return back this "manager_userId.exe".
Points to be noted over here is that:
The generic.exe file should not be modified/deleted at all
When 2 users (userA and userB) hit that same API simultaneously , they should get their own copy of manager_userA.exe and manager_userB.exe
The code what I have written is
#RequestMapping(value = "/downloadExecutable", method = RequestMethod.POST, consumes = MediaType.APPLICATION_JSON, produces = {MediaType.APPLICATION_OCTET_STREAM})
#ResponseBody
public Response downloadExecutable(#RequestBody DownloadExecutableRequest downloadExecutableRequest,
HttpServletRequest request, HttpServletResponse response) {
File file = downloadExecutable(downloadExecutableRequest, request, response,
getUserID(request), osDetails);
return Response.ok(file, MediaType.APPLICATION_OCTET_STREAM)
.header("Content-Disposition", "attachment;filename=" + file.getName()).build();
}
public File downloadExecutable(DownloadExecutableRequest downloadExecutableRequest, HttpServletRequest request,
HttpServletResponse response, String userId, String osDetails) {
File file = null;
String path = "/home/genericCopy/generic.exe";
synchronized (this) {
BufferedWriter fileWriter = null;
try {
File source = null;
source = new File(path);
Path destination = Paths.get("/tmp/");
Files.copy(source, destination.toFile());
fileWriter = new BufferedWriter(new FileWriter(destination.getFileName().toString()+"_"+userId));
file = new File(destination.getFileName().toString());
} catch (IOException e) {
} finally {
if (fileWriter != null) {
fileWriter.close();
}
}
}
return file;
}
The code is working , but it is creating a temporary file and then renaming it and then returning it back but it will keep on creating the copies of the file for each request.
Is there any smarter way that i can achieve not to create such temporary copies of the user specific files and also handle a scenario when 2 users hit the API simultaneously ?

The name of the file which is downloaded by user has no relationship to the name of the file on disk.
You can just specify any name of the file in the header and the user will see the name.
Specifically, you would just set the filename you want the user to see to the Content-Disposition header and always load the same exe file from the disk.
Something like this:
return Response.ok(file, MediaType.APPLICATION_OCTET_STREAM)
.header("Content-Disposition", "attachment;filename=executable_" + getUserID(request) + ".exe";
You don't need to do any copying in the downloadExecutable function.

You don't need to create a copy of generic.exe file to return it with changed name. You can use correctly parametrised Content-Disposition header, so it would return same file every time, with file name provided by user.
Here you can find example:
#RestController
public class DemoController {
#GetMapping(value = "/file", produces = MediaType.APPLICATION_OCTET_STREAM_VALUE)
#ResponseBody
public ResponseEntity downloadExecutable(#RequestParam("userId") String userId) throws IOException {
byte[] file = Files.readAllBytes(Paths.get("/home/genericCopy/generic.exe"));
return ResponseEntity.ok()
.header(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=generic_" + userId + ".exe")
.contentLength(file.length)
.contentType(MediaType.APPLICATION_OCTET_STREAM)
.body(file);
}
}
and result of executing this method:

Cannot create the generated HTTP client's required return type, since no TypeConverter from ByteBuffer to class java.io.File : Micronaut

Below is the server side code for sending file to client as rest response using micronaut.
#Get(value = "/downloadFile", produces = MediaType.APPLICATION_OCTET_STREAM )
public HttpResponse<File> downloadDocument() throws IOException {
File sampleDocumentFile = new File(getClass().getClassLoader().getResource("SampleDocument.pdf").getFile());
return HttpResponse.ok(sampleDocumentFile).header("Content-Disposition", "attachment; filename=\"" + sampleDocumentFile.getName() + "\"" );
}
Below is the client for calling the above endpoint.
#Client(value = "/client")
public interface DownloadDocumentClient {
#Get(value = "/downloadDocument", processes = MediaType.APPLICATION_OCTET_STREAM)
public Flowable<File> downloadDocument();
}
Tried to retreive file as below :-
Flowable<File> fileFlowable = downloadDocumentClient.downloadDocument();
Maybe<File> fileMaybe = fileFlowable.firstElement();
return fileMaybe.blockingGet();
Getting exception as
io.micronaut.context.exceptions.ConfigurationException: Cannot create
the generated HTTP client's required return type, since no
TypeConverter from ByteBuffer to class java.io.File is registered

You cannot send file data using File instance because it contains only path and not file content. You can send file content using byte array.
Update the controller in this way:
#Get(value = "/download", produces = MediaType.APPLICATION_OCTET_STREAM)
public HttpResponse<byte[]> downloadDocument() throws IOException, URISyntaxException {
String documentName = "SampleDocument.pdf";
byte[] content = Files.readAllBytes(Paths.get(getClass().getClassLoader().getResource(documentName).toURI()));
return HttpResponse.ok(content).header("Content-Disposition", "attachment; filename=\"" + documentName + "\"");
}
Client will be then like this:
#Get(value = "/download", processes = MediaType.APPLICATION_OCTET_STREAM)
Flowable<byte[]> downloadDocument();
And finally client call:
Flowable<byte[]> fileFlowable = downloadDocumentClient.downloadDocument();
Maybe<byte[]> fileMaybe = fileFlowable.firstElement();
byte[] content = fileMaybe.blockingGet();
Update:
If you need to save received bytes (file content) into the file on a client machine (container) then you can do that for example like this:
Path targetPath = Files.write(Paths.get("target.pdf"), fileMaybe.blockingGet());
And if you really need instance of File instead of Path for further processing then simply:
File file = targetPath.toFile();

Get all path variable

I want to develop a service that return a json file.
#RequestMapping(value = "/{fileName}/**", method = RequestMethod.GET, produces = { "application/json" })
public String jsonREST(#PathVariable String fileName) {
StringBuilder jsonBuilder = new StringBuilder();
logger.info("===> File name: " + fileName);
try {
BufferedReader bf = new BufferedReader(new FileReader("fileName + ".json"));
String line = null;
while ((line = bf.readLine()) != null) {
jsonBuilder.append(line);
}
} catch (Exception e) {
System.out.println("Error Parsing: - ");
}
return jsonBuilder.toString();
}
I need to get the path for example if the json file is in subdirectory or else.
use cases:
localhost:8080/my-directory/my-sub-dir/my-json-file
localhost:8080/my-json-file
Would you have any idea how I can get the hole path for example
my-directory/my-sub-dir/my-json-file
Or if you have another solution to do the same job, I will be very happy for that
Best regards

You can get the full request url by having Spring inject the HttpServletRequest and getting it as follows:
#RequestMapping(value = "/{fileName}/**", method = RequestMethod.GET, produces = { "application/json" })
public String jsonREST(HttpServletRequest request, #PathVariable String fileName) {
String uri = request.getRequestURI();
//Do your stuff here
}

Seems like you don't need a servlet container to achieve this. If I get what you are trying to do, you want to serve the json files statically. Try tweaking this:
https://spring.io/blog/2013/12/19/serving-static-web-content-with-spring-boot

How to post multiple files (images) using Jersey Web service.?

Hi I am new to Restful web service.
My Goal is to create multiple user account's in a single request.
I am choosing Jersey API to create a web service.
This WS will create multiple user's account. Each user account was associated with an avatar (Profile picture). I am sending the user information with avatar (The avatar file was encoded into string format using Base64 encoder).
My question is, If the request has many number of users, and each user is associated with bigger size of avatar, Is Restful web service can handle these request?
Also the Request data size in the restful webservice is limited?
Please Suggest me to create a better web service in Jersey API.

Instead of passing avatars in message body you should look at Jersey Multipart support - it will allow you to stream large files to your restful service. Another plus - you'll not need Base64 encoding anymore.

I can achieve this by doing form submit instead of preparing the object in JSON format.
#Path("/upload")
public class MultipleFiles
{
private static final String FILE_UPLOAD_PATH = "/Users/arun_kumar/Pictures";
private static final String CANDIDATE_NAME = "candidateName";
private static final String SUCCESS_RESPONSE = "Successful";
private static final String FAILED_RESPONSE = "Failed";
#POST
#Consumes(MediaType.MULTIPART_FORM_DATA)
#Produces("text/plain")
#Path("/multipleFiles")
public String registerWebService(#Context HttpServletRequest request)
{
String responseStatus = SUCCESS_RESPONSE;
String candidateName = null;
//checks whether there is a file upload request or not
if (ServletFileUpload.isMultipartContent(request))
{
final FileItemFactory factory = new DiskFileItemFactory();
final ServletFileUpload fileUpload = new ServletFileUpload(factory);
try
{
/*
* parseRequest returns a list of FileItem
* but in old (pre-java5) style
*/
final List items = fileUpload.parseRequest(request);
if (items != null)
{
final Iterator iter = items.iterator();
while (iter.hasNext())
{
final FileItem item = (FileItem) iter.next();
final String itemName = item.getName();
final String fieldName = item.getFieldName();
final String fieldValue = item.getString();
if (item.isFormField())
{
candidateName = fieldValue;
System.out.println(" Name: " + fieldName + ", Value: " + fieldValue);
}
else
{
final File savedFile = new File(FILE_UPLOAD_PATH + File.separator
+ itemName);
System.out.println(" Saving the file: " + savedFile.getName());
item.write(savedFile);
}
}
}
}
catch (FileUploadException fue)
{
responseStatus = FAILED_RESPONSE;
fue.printStackTrace();
}
catch (Exception e)
{
responseStatus = FAILED_RESPONSE;
e.printStackTrace();
}
}
System.out.println("Returned Response Status: " + responseStatus);
return responseStatus;
}
}
Reference: Jersey REST Web Service to Upload Multiple Files