We Keep Coding

Is there any efficient method in springboot to upload large files? my current api is taking long time to upload large document

I am trying to upload a file of size 5gb in SpringBoot but it takes hours to let request get inside controller from postman. once request get inside controller than it doesn't take much time to upload the document. sometimes it returns connection timeout issue. can anyone provide an efficient way to get the task done in SpringBoot?
here is my controller
#RequestMapping(
value = "/store",
method = RequestMethod.POST,
consumes = MediaType.MULTIPART_FORM_DATA_VALUE,
produces = MediaType.APPLICATION_JSON_VALUE)
public FilesDTO storeFile(
#RequestHeader("Authorization") String auth,
#RequestParam("filename") String filename,
#RequestParam("collectorId") String collectorId,
#RequestParam(name = "description", required = false) String description,
#RequestParam("FileType") DataFileType collectorFileType,
#RequestPart("file") MultipartFile file)
throws PRValidationException, IOException {
return collectorFilesService.storeFile(auth, filename, collectorId, description, collectorFileType, file);
}
here is my Implemention
#Override
public CollectorFilesDTO storeCollectorFile(String auth, String filename, String collectorId, String description, DataFileType collectorFileType, MultipartFile file) throws IOException, PrValidationException {
dataServiceApi.findCollectorById(auth,collectorId);
if (file.isEmpty()) {
throw new FileNotFoundException(environment.getProperty("pr.validation.file_not_found"));
}
if(collectorFileType == null){
throw new PrValidationException(environment.getProperty("pr.validation.collectorFileType.not.null"));
}
return storeFile(auth, filename, collectorId,description,collectorFileType ,file.getBytes());
}
#Override
public CollectorFilesDTO storeFile(String auth, String filename, String collectorId,String description,DataFileType collectorFileType, byte[] bytes) throws IOException, PrValidationException {
logger.info("UPLOAD_FOLDER=" + UPLOAD_FOLDER_PATH);
logger.info("fileName=" + filename);
String absolutePath = UPLOAD_FOLDER_PATH;
logger.info("Writing " + absolutePath + filename);
final File parentDirectory = new File(absolutePath);
if (!parentDirectory.exists()) {
if (!parentDirectory.mkdirs()) {
throw new IOException(environment.getProperty("pr.validation.file_not_created"));
}
}
File file = new File(parentDirectory, filename);
boolean fileAlreadyExists = file.exists();
try {
Files.write(Paths.get(absolutePath + filename), bytes);
CollectorFileEntity collectorFileEntity = storeDocument(auth, filename,collectorId, description,collectorFileType ,fileAlreadyExists);
logger.info("file write success and document has been saved in {}ms", System.currentTimeMillis());
return CollectorFileMapper.INSTANCE.toDto(collectorFileEntity);
} catch (IOException e) {
logger.info(e.getMessage());
throw new IOException(environment.getProperty("pr.validation.write_file_exception"));
}
}

The Apache Commons FileUpload is a popular tool for this use case, here's a guide of how to use it.

Create regex which will match to all requests in Spring [duplicate]

Is there a way to get the complete path value after the requestMapping #PathVariable values have been parsed?
That is:
/{id}/{restOfTheUrl} should be able to parse /1/dir1/dir2/file.html into id=1 and restOfTheUrl=/dir1/dir2/file.html
Any ideas would be appreciated.

Non-matched part of the URL is exposed as a request attribute named HandlerMapping.PATH_WITHIN_HANDLER_MAPPING_ATTRIBUTE:
#RequestMapping("/{id}/**")
public void foo(#PathVariable("id") int id, HttpServletRequest request) {
String restOfTheUrl = new AntPathMatcher().extractPathWithinPattern(request.getAttribute(HandlerMapping.BEST_MATCHING_PATTERN_ATTRIBUTE).toString(),request.getRequestURI());
...
}

Just found that issue corresponding to my problem. Using HandlerMapping constants I was able to wrote a small utility for that purpose:
/**
* Extract path from a controller mapping. /controllerUrl/** => return matched **
* #param request incoming request.
* #return extracted path
*/
public static String extractPathFromPattern(final HttpServletRequest request){
String path = (String) request.getAttribute(
HandlerMapping.PATH_WITHIN_HANDLER_MAPPING_ATTRIBUTE);
String bestMatchPattern = (String ) request.getAttribute(HandlerMapping.BEST_MATCHING_PATTERN_ATTRIBUTE);
AntPathMatcher apm = new AntPathMatcher();
String finalPath = apm.extractPathWithinPattern(bestMatchPattern, path);
return finalPath;
}

This has been here quite a while but posting this. Might be useful for someone.
#RequestMapping( "/{id}/**" )
public void foo( #PathVariable String id, HttpServletRequest request ) {
String urlTail = new AntPathMatcher()
.extractPathWithinPattern( "/{id}/**", request.getRequestURI() );
}

Building upon Fabien Kruba's already excellent answer, I thought it would be nice if the ** portion of the URL could be given as a parameter to the controller method via an annotation, in a way which was similar to #RequestParam and #PathVariable, rather than always using a utility method which explicitly required the HttpServletRequest. So here's an example of how that might be implemented. Hopefully someone finds it useful.
Create the annotation, along with the argument resolver:
#Target(ElementType.PARAMETER)
#Retention(RetentionPolicy.RUNTIME)
#Documented
public #interface WildcardParam {
class Resolver implements HandlerMethodArgumentResolver {
#Override
public boolean supportsParameter(MethodParameter methodParameter) {
return methodParameter.getParameterAnnotation(WildcardParam.class) != null;
}
#Override
public Object resolveArgument(MethodParameter methodParameter, ModelAndViewContainer modelAndViewContainer, NativeWebRequest nativeWebRequest, WebDataBinderFactory webDataBinderFactory) throws Exception {
HttpServletRequest request = nativeWebRequest.getNativeRequest(HttpServletRequest.class);
return request == null ? null : new AntPathMatcher().extractPathWithinPattern(
(String) request.getAttribute(HandlerMapping.BEST_MATCHING_PATTERN_ATTRIBUTE),
(String) request.getAttribute(HandlerMapping.PATH_WITHIN_HANDLER_MAPPING_ATTRIBUTE));
}
}
}
Register the method argument resolver:
#Configuration
public class WebMvcConfig implements WebMvcConfigurer {
#Override
public void addArgumentResolvers(List<HandlerMethodArgumentResolver> resolvers) {
resolvers.add(new WildcardParam.Resolver());
}
}
Use the annotation in your controller handler methods to have easy access to the ** portion of the URL:
#RestController
public class SomeController {
#GetMapping("/**")
public void someHandlerMethod(#WildcardParam String wildcardParam) {
// use wildcardParam here...
}
}

You need to use built-in pathMatcher:
#RequestMapping("/{id}/**")
public void test(HttpServletRequest request, #PathVariable long id) throws Exception {
ResourceUrlProvider urlProvider = (ResourceUrlProvider) request
.getAttribute(ResourceUrlProvider.class.getCanonicalName());
String restOfUrl = urlProvider.getPathMatcher().extractPathWithinPattern(
String.valueOf(request.getAttribute(HandlerMapping.BEST_MATCHING_PATTERN_ATTRIBUTE)),
String.valueOf(request.getAttribute(HandlerMapping.PATH_WITHIN_HANDLER_MAPPING_ATTRIBUTE)));

I have used the Tuckey URLRewriteFilter to handle path elements that contain '/' characters, as I don't think Spring 3 MVC supports them yet.
http://www.tuckey.org/
You put this filter in to your app, and provide an XML config file. In that file you provide rewrite rules, which you can use to translate path elements containing '/' characters into request parameters that Spring MVC can deal with properly using #RequestParam.
WEB-INF/web.xml:
<filter>
<filter-name>UrlRewriteFilter</filter-name>
<filter-class>org.tuckey.web.filters.urlrewrite.UrlRewriteFilter</filter-class>
</filter>
<!-- map to /* -->
WEB-INF/urlrewrite.xml:
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE urlrewrite
PUBLIC "-//tuckey.org//DTD UrlRewrite 3.0//EN"
"http://tuckey.org/res/dtds/urlrewrite3.0.dtd">
<urlrewrite>
<rule>
<from>^/(.*)/(.*)$</from>
<to last="true">/$1?restOfTheUrl=$2</to>
</urlrewrite>
Controller method:
#RequestMapping("/{id}")
public void handler(#PathVariable("id") int id, #RequestParam("restOfTheUrl") String pathToFile) {
...
}

Yes the restOfTheUrl is not returning only required value but we can get the value by using UriTemplate matching.
I have solved the problem, so here the working solution for the problem:
#RequestMapping("/{id}/**")
public void foo(#PathVariable("id") int id, HttpServletRequest request) {
String restOfTheUrl = (String) request.getAttribute(
HandlerMapping.PATH_WITHIN_HANDLER_MAPPING_ATTRIBUTE);
/*We can use UriTemplate to map the restOfTheUrl*/
UriTemplate template = new UriTemplate("/{id}/{value}");
boolean isTemplateMatched = template.matches(restOfTheUrl);
if(isTemplateMatched) {
Map<String, String> matchTemplate = new HashMap<String, String>();
matchTemplate = template.match(restOfTheUrl);
String value = matchTemplate.get("value");
/*variable `value` will contain the required detail.*/
}
}

Here is how I did it. You can see how I convert the requestedURI to a filesystem path (what this SO question is about). Bonus: and also how to respond with the file.
#RequestMapping(value = "/file/{userId}/**", method = RequestMethod.GET)
public void serveFile(#PathVariable("userId") long userId, HttpServletRequest request, HttpServletResponse response) {
assert request != null;
assert response != null;
// requestURL: http://192.168.1.3:8080/file/54/documents/tutorial.pdf
// requestURI: /file/54/documents/tutorial.pdf
// servletPath: /file/54/documents/tutorial.pdf
// logger.debug("requestURL: " + request.getRequestURL());
// logger.debug("requestURI: " + request.getRequestURI());
// logger.debug("servletPath: " + request.getServletPath());
String requestURI = request.getRequestURI();
String relativePath = requestURI.replaceFirst("^/file/", "");
Path path = Paths.get("/user_files").resolve(relativePath);
try {
InputStream is = new FileInputStream(path.toFile());
org.apache.commons.io.IOUtils.copy(is, response.getOutputStream());
response.flushBuffer();
} catch (IOException ex) {
logger.error("Error writing file to output stream. Path: '" + path + "', requestURI: '" + requestURI + "'");
throw new RuntimeException("IOError writing file to output stream");
}
}

private final static String MAPPING = "/foo/*";
#RequestMapping(value = MAPPING, method = RequestMethod.GET)
public #ResponseBody void foo(HttpServletRequest request, HttpServletResponse response) {
final String mapping = getMapping("foo").replace("*", "");
final String path = (String) request.getAttribute(HandlerMapping.PATH_WITHIN_HANDLER_MAPPING_ATTRIBUTE);
final String restOfPath = url.replace(mapping, "");
System.out.println(restOfPath);
}
private String getMapping(String methodName) {
Method methods[] = this.getClass().getMethods();
for (int i = 0; i < methods.length; i++) {
if (methods[i].getName() == methodName) {
String mapping[] = methods[i].getAnnotation(RequestMapping.class).value();
if (mapping.length > 0) {
return mapping[mapping.length - 1];
}
}
}
return null;
}

To improve upon #Daniel Jay Marcaida answer
#RequestMapping( "/{id}/**" )
public void foo( #PathVariable String id, HttpServletRequest request ) {
String restOfUrl = new AntPathMatcher()
.extractPathWithinPattern(
request.getAttribute(HandlerMapping.BEST_MATCHING_PATTERN_ATTRIBUTE).toString(),
request.getAttribute(HandlerMapping.PATH_WITHIN_HANDLER_MAPPING_ATTRIBUTE).toString());
}
or
#RequestMapping( "/{id}/**" )
public void foo( #PathVariable String id, HttpServletRequest request ) {
String restOfUrl = new AntPathMatcher()
.extractPathWithinPattern(
request.getAttribute(HandlerMapping.BEST_MATCHING_PATTERN_ATTRIBUTE).toString(),
request.getServletPath());
}

I have a similar problem and I resolved in this way:
#RequestMapping(value = "{siteCode}/**/{fileName}.{fileExtension}")
public HttpEntity<byte[]> getResource(#PathVariable String siteCode,
#PathVariable String fileName, #PathVariable String fileExtension,
HttpServletRequest req, HttpServletResponse response ) throws IOException {
String fullPath = req.getPathInfo();
// Calling http://localhost:8080/SiteXX/images/argentine/flag.jpg
// fullPath conentent: /SiteXX/images/argentine/flag.jpg
}
Note that req.getPathInfo() will return the complete path (with {siteCode} and {fileName}.{fileExtension}) so you will have to process conveniently.

Get all path variable

I want to develop a service that return a json file.
#RequestMapping(value = "/{fileName}/**", method = RequestMethod.GET, produces = { "application/json" })
public String jsonREST(#PathVariable String fileName) {
StringBuilder jsonBuilder = new StringBuilder();
logger.info("===> File name: " + fileName);
try {
BufferedReader bf = new BufferedReader(new FileReader("fileName + ".json"));
String line = null;
while ((line = bf.readLine()) != null) {
jsonBuilder.append(line);
}
} catch (Exception e) {
System.out.println("Error Parsing: - ");
}
return jsonBuilder.toString();
}
I need to get the path for example if the json file is in subdirectory or else.
use cases:
localhost:8080/my-directory/my-sub-dir/my-json-file
localhost:8080/my-json-file
Would you have any idea how I can get the hole path for example
my-directory/my-sub-dir/my-json-file
Or if you have another solution to do the same job, I will be very happy for that
Best regards

You can get the full request url by having Spring inject the HttpServletRequest and getting it as follows:
#RequestMapping(value = "/{fileName}/**", method = RequestMethod.GET, produces = { "application/json" })
public String jsonREST(HttpServletRequest request, #PathVariable String fileName) {
String uri = request.getRequestURI();
//Do your stuff here
}

Seems like you don't need a servlet container to achieve this. If I get what you are trying to do, you want to serve the json files statically. Try tweaking this:
https://spring.io/blog/2013/12/19/serving-static-web-content-with-spring-boot

dynamic path to resource in springmvc

In Java-Jersey, it is possible to receive a dynamic path to a resource, e.g.
localhost:8080/webservice/this/is/my/dynamic/path
#GET
#Path("{dynamicpath : .+}")
#Produces(MediaType.APPLICATION_JSON)
public String get(#PathParam("dynamicpath") String p_dynamicpath) {
return p_dynamicpath;
}
prints out: this/is/my/dynamic/path
Question: how to do this in Spring MVC?

For multiple items inside your path you can access the dynamic path values like this:
#RequestMapping(value="/**", method = RequestMethod.GET)
public String get(HttpServletRequest request) throws Exception {
String dynPath = (String) request.getAttribute(HandlerMapping.PATH_WITHIN_HANDLER_MAPPING_ATTRIBUTE);
System.out.println("Dynamic Path: " + dynPath );
return dynPath;
}
If you know beforehand hoe many levels of path variables you'll have you can code them explicit like
#RequestMapping(value="/{path1}/{path2}/**", method = RequestMethod.GET)
public String get(#PathVariable("path1") String path1,
#PathVariable("path2") String path2,
HttpServletRequest request) throws Exception {
String dynPath = (String) request.getAttribute(HandlerMapping.PATH_WITHIN_HANDLER_MAPPING_ATTRIBUTE);
System.out.println("Dynamic Path: " + dynPath );
return dynPath;
}
If you want to see the String returned in your browser, you need to declare the method #ResponseBody as well (so the String you return is the content of your response):
#RequestMapping(value="/**", method = RequestMethod.GET, produces = "text/plain")
#ResponseBody
public String get(HttpServletRequest request) throws Exception {

Is it possible to refer to the #RequestMapping value in the annotated method?

Given a method like:
#RequestMapping(value = {"/foo"}, method = RequestMethod.GET)
public String getMappingValueInMethod() {
log.debug("requested "+foo); //how can I make this refer to /foo programmatically?
return "bar";
}
The use case is for refactoring some lengthly code. I have several GET methods doing roughly the same thing and only the request mapping value is different.
I've looked at using path variables, but this is not really what I want (unless there's some clever use of it that I don't see). I could also get a value from the HttpServletRequest like in this post, but not sure whether there's a better way.

Solution 1
With HttpServletRequest.
#RequestMapping(value = "/foo", method = RequestMethod.GET)
public String fooMethod(HttpServletRequest request) {
String path = request.getAttribute(HandlerMapping.PATH_WITHIN_HANDLER_MAPPING_ATTRIBUTE).toString();
System.out.println("path foo: " + path);
return "bar";
}
Solution 2
With reflection.
#RequestMapping(value = "/foo2", method = RequestMethod.GET)
public String fooMethod2() {
try {
Method m = YourClassController.class.getMethod("fooMethod2");
String path = m.getAnnotation(RequestMapping.class).value()[0];
System.out.println("foo2 path: " + path);
} catch (NoSuchMethodException e) {
e.printStackTrace();
}
return "bar";
}
If you want get path from class (instead method) you can use:
String path = YourClassController.class.getAnnotation(RequestMapping.class).value();
Solution 3
With #PathVariable.
#RequestMapping(value = {"/{foo3}"}, method = RequestMethod.GET)
public #ResponseBody String fooMethod3(#PathVariable("foo3") String path) {
path = "/" + path; // if you need "/"
System.out.println("foo3 path: " + path);
return "bar";
}

The simplest way of doing this would be putting the array directly in the request mapping i am assuming this is what you want.
#RequestMapping(value = {"/foo","/foo1","/foo2"}, method = RequestMethod.GET)
public String getMappingValueInMethod(HttpServletRequest request) {
log.debug("requested "+request.getRequestURI());
return request.getRequestURI();
}
Then name the jsp files similar to the uri or other wise you could store the mapping between the request uri and the name of the page in the db .