We Keep Coding

spring batch file writer to write directly to amazon s3 storage without PutObjectRequest

I'm trying to upload a file to amazon s3. Instead of uploading, I want to read the data from database using spring batch and write the file directly into the s3 storage. Is there anyway we can do that ?

Spring Cloud AWS adds support for the Amazon S3 service to load and write resources with the resource loader and the s3 protocol. Once you have configured the AWS resource loader, you can write a custom Spring Batch writer like:
import java.io.OutputStream;
import java.util.List;
import org.springframework.batch.item.ItemWriter;
import org.springframework.core.io.ResourceLoader;
import org.springframework.core.io.WritableResource;
public class AwsS3ItemWriter implements ItemWriter<String> {
private ResourceLoader resourceLoader;
private WritableResource resource;
public AwsS3ItemWriter(ResourceLoader resourceLoader, String resource) {
this.resourceLoader = resourceLoader;
this.resource = (WritableResource) this.resourceLoader.getResource(resource);
}
#Override
public void write(List<? extends String> items) throws Exception {
try (OutputStream outputStream = resource.getOutputStream()) {
for (String item : items) {
outputStream.write(item.getBytes());
}
}
}
}
Then you should be able to use this writer with an S3 resource like s3://myBucket/myFile.log.
Is there anyway we can do that ?
Please note that I did not compile/test the previous code. I just wanted to give you a starting point of how to do it.
Hope this helps.

The problem is that the OutputStream will only write the last List items sent by the step...
I think you might need to write a temporary file on file system and then send the whole file in a separate tasklet
See this example :
https://github.com/TerrenceMiao/AWS/blob/master/dynamodb-java/src/main/java/org/paradise/microservice/userpreference/service/writer/CSVFileWriter.java

I had the same thing to do. Because spring has no clas to write to a stream alone I made one my self like above example:
You need to classes for this. A Resource class which implements WriteableResource and extends AbstractResource:
...
public class S3Resource extends AbstractResource implements WritableResource {
ByteArrayOutputStream resource = new ByteArrayOutputStream();
#Override
public String getDescription() {
return null;
}
#Override
public InputStream getInputStream() throws IOException {
return new ByteArrayInputStream(resource.toByteArray());
}
#Override
public OutputStream getOutputStream() throws IOException {
return resource;
}
}
And your writer which extends ItemWriter:
public class AmazonStreamWriter<T> implements ItemWriter<T>{
private WritableResource resource;
private LineAggregator<T> lineAggregator;
private String lineSeparator;
public String getLineSeparator() {
return lineSeparator;
}
public void setLineSeparator(String lineSeparator) {
this.lineSeparator = lineSeparator;
}
AmazonStreamWriter(WritableResource resource){
this.resource = resource;
}
public WritableResource getResource() {
return resource;
}
public void setResource(WritableResource resource) {
this.resource = resource;
}
public LineAggregator<T> getLineAggregator() {
return lineAggregator;
}
public void setLineAggregator(LineAggregator<T> lineAggregator) {
this.lineAggregator = lineAggregator;
}
#Override
public void write(List<? extends T> items) throws Exception {
try (OutputStream outputStream = resource.getOutputStream()) {
StringBuilder lines = new StringBuilder();
Iterator var3 = items.iterator();
while(var3.hasNext()) {
T item = (T) var3.next();
lines.append(this.lineAggregator.aggregate(item)).append(this.lineSeparator);
}
outputStream.write(lines.toString().getBytes());
}
}
}
With this setup you will write your Item-Information you recieve from your database and write it to your Customresource via an OutputStream. The filled resource then can be used in one of your Steps zu open an InputStream and upload to S3 via Client.
I did it with: amazonS3.putObject(awsBucketName, awsBucketKey , resource.getInputStream(), new ObjectMetadata());
My solution may be not the perfect aproach, but from here on you can optimize it.

Struts 2 Download File is 0 bytes

I'm having an issue downloading a file using Struts2. I've done a pile of research and found a bunch of similar questions, but none of the answers have helped me out.
Here is what I currently have
JSP
<s:url id="fileDownload" namespace="/jsp" action="download"></s:url>
Download file - <s:a href="%{fileDownload}">MyFile.pdf</s:a>
Action
private InputStream inputStream;
private String fileName;
public String execute() throws Exception {
File fileToDownload = new File("C:My Documents/MyFile.pdf");
fileName = fileToDownload.getName();
inputStream = new FileInputStream(fileToDownload);
return SUCCESS;
}
public String getFileName() {
return fileName;
}
public void setFileName(String fileName) {
this.fileName = fileName;
}
public InputStream getInputStream() {
return inputStream;
}
Struts.xml
<action name="download" class="com.my.path.to.action.class">
<result name="success" type="stream">
<param name="contentDisposition">attachment;filename=${fileName}</param>
<param name="contentType">application/pdf</param>
<param name="inputName">inputStream</param>
<param name="bufferSize">4096</param>
</result>
</action>
When I click on the link, it will download a file that's named correctly, but it has no data in it. If anyone has any ideas as to what I'm doing wrong, I would love a suggestion as I'm sure it's just something dumb.

I found my answer. You must define the content length in struts. To do this I did the following:
Struts.xml
<param name="contentLength">${contentLength}</param>
Action
private long contentLength;
public long getContentLength() {
return contentLength;
}
public void setContentLength(long contentLength) {
this.contentLength = contentLength;
}
in execute()
contentLength = fileToDownload.length();

Java & Apache-Camel: From direct-endpoint to file-endpoint

I've tried to build a route to copy files from one directory to an other directory. But instead of using:
from(file://source-directory).to(file://destination-directory)
I want to do something like this:
from(direct:start)
.to(direct:doStuff)
.to(direct:readDirectory)
.to(file://destination-folder)
I've done the following stuff:
Route
#Component
public class Route extends AbstractRouteBuilder {
#Override
public void configure() throws Exception {
from("direct:start")
.bean(lookup(ReadDirectory.class))
.split(body())
.setHeader("FILENAME", method(lookup(CreateFilename.class)))
.to("file:///path/to/my/output/directory/?fileName=${header.FILENAME}");
}
Processor
#Component
public class ReadDirectory implements CamelProcessorBean {
#Handler
public ImmutableList<File> apply(#Header("SOURCE_DIR") final String sourceDir) {
final File directory = new File(sourceDir);
final File[] files = directory.listFiles();
if (files == null) {
return ImmutableList.copyOf(Lists.<File>newArrayList());
}
return ImmutableList.copyOf(files);
}
}
I can start my route by using the following pseudo-Test (The point is I can manually start my route by producer.sendBodyAndHeader(..))
public class RouteIT extends StandardIT {
#Produce
private ProducerTemplate producer;
#Test
public void testRoute() throws Exception {
final String uri = "direct:start";
producer.sendBodyAndHeaders(uri, InOut, null, header());
}
private Map<String, Object> header() {
final Map<String, Object> header = Maps.newHashMap();
header.put("SOURCE_DIR", "/path/to/my/input/directory/");
return header;
}
}
AbstractRouteBuilderextends SpringRouteBuilder
CamelProcessorBean is only a Marker-Interface
StandardIT loads SpringContext and stuff
The problem is, that I must set the filename. I've read some stuff that camel sets the header CamelFileNameProduced (during the file endpoint). It is a generic string with timestamp and if I don't set the filename - the written files will get this generic string as the filename.
My Question is: Is there a more beautiful solution to copy files (but starting with a direct-endpoint and read the directory in the middle of the route) and keep the filename for the destination? (I don't have to set the filename when I use from("file:source").to("file:destination"), why must I do it now?)

You can set the file name when you send using the producer template, as long as the header is propagated during the routing between the routes you are all fine, which Camel does by default.
For example
#Test
public void testRoute() throws Exception {
final String uri = "direct:start";
Map headers = ...
headers.put(Exchange.FILE_NAME, "myfile.txt");
producer.sendBodyAndHeaders(uri, InOut, null, headers);
}
The file component talks more about how to control the file name
http://camel.apache.org/file2

Excel file import java spring hibernate

I am working on spring hibernate and I want to import an excel file. I want to make a check on extensions so that no one can upload a file other than excel i.e. restrict the import to files with one of these extensions: xls or xlsx. My code is here:
public class ImportCandidatesFormController extends BNUAbstractFormController {
private ImportCandidatesBL importCandidatesBL;
private ExcelReader reader;
#Override
protected ModelAndView processFormSubmission(HttpServletRequest request,
HttpServletResponse response, Object command, BindException arg3)
throws Exception {
FileUploadVO vo = (FileUploadVO) command;
MultipartFile file = vo.getFile();
System.out.println("File Uploaded: " + file.getOriginalFilename());
boolean isSuccessful = importCandidatesBL.importAndSaveCandidates(
file.getInputStream(),
SessionUtil.getCurrentUser(request.getSession()));
return new ModelAndView(new RedirectView("importCandidates.do?s=1"));
}
public ImportCandidatesBL getImportCandidatesBL() {
return importCandidatesBL;
}
public void setImportCandidatesBL(ImportCandidatesBL importCandidatesBL) {
this.importCandidatesBL = importCandidatesBL;
}
}

String lowercaseFileName = file.getOriginalFilename().toLowerCase();
if (!(lowerCaseFileName.endsWith(".xls") || lowerCaseFileName.endsWith(".xlsx"))) {
// reject file
}

Wicket image component not found on page

I get the following problem when trying to display a list of items. For each item, I have to display an image which is dynamically loaded via a Wicket WebResource. The items are loaded step by step — 50 at a time — upon user scrolling, using an Ajax scroll.
[ERROR] 2011-04-19 09:58:18,000 btpool0-1 org.apache.wicket.RequestCycle.logRuntimeException (host=, request=, site=):
org.apache.wicket.WicketRuntimeException: component documentList:scroller:batchElem:666:content:item:3:batchItemContent:linkToPreview:imageThumbnail not found on page com.webapp.document.pages.DocumentListPage[id = 1]
listener interface = [RequestListenerInterface name=IResourceListener, method=public abstract void org.apache.wicket.IResourceListener.onResourceRequested()]
org.apache.wicket.protocol.http.request.InvalidUrlException: org.apache.wicket.WicketRuntimeException: component documentList:scroller:batchElem:666:content:item:3:batchItemContent:linkToPreview:imageThumbnail
not found on page com.webapp.document.pages.DocumentListPage[id = 1] listener interface = [RequestListenerInterface name=IResourceListener, method=public abstract void org.apache.wicket.IResourceListener.onResourceRequested()]
at org.apache.wicket.protocol.http.WebRequestCycleProcessor.resolve(WebRequestCycleProcessor.java:262)
at org.apache.wicket.RequestCycle.step(RequestCycle.java:1310)
at org.apache.wicket.RequestCycle.steps(RequestCycle.java:1428)
at org.apache.wicket.RequestCycle.request(RequestCycle.java:545)
at org.apache.wicket.protocol.http.WicketFilter.doGet(WicketFilter.java:479)
at org.apache.wicket.protocol.http.WicketFilter$$EnhancerByGuice$$51619816.CGLIB$doGet$6()
at org.apache.wicket.protocol.http.WicketFilter$$EnhancerByGuice$$51619816$$FastClassByGuice$$6d42bf5d.invoke()
at com.google.inject.internal.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64)
at com.freiheit.monitoring.PerformanceMonitoringMethodInterceptor.invoke(PerformanceMonitoringMethodInterceptor.java:115)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:64)
at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:44)
at org.apache.wicket.protocol.http.WicketFilter$$EnhancerByGuice$$51619816.doGet()
at org.apache.wicket.protocol.http.WicketFilter.doFilter(WicketFilter.java:312)
at org.apache.wicket.protocol.http.WicketFilter$$EnhancerByGuice$$51619816.CGLIB$doFilter$4()
How can this problem be solved?
Here is the part of the code responsible for adding the image:
previewLink.add(createThumbnailSmall("imageThumbnail", documentModel));
in
createThumbnailSmall(final String id, final IModel<BaseDocument> documentModel) {
// thumbnailResource is an object that contains the path of the image
if (thumbnailResource != null) {
final WebResource resource = getWebResource(thumbnailResource);
final Image image = new Image(id, resource);
return image;
}
return new InvisibleContainer(id);
}
WebResource getWebResource(final DocumentResource documentResource) {
return new WebResource() {
private static final long serialVersionUID = 1L;
#Override
public IResourceStream getResourceStream() {
return new BaseStreamResource(documentResource);
}
};
}
where BaseStreamResource is the following:
public class BaseStreamResource extends AbstractResourceStream {
private InputStream _fileInputStream = null;
private DocumentResource _resource = null;
public BaseStreamResource(final DocumentResource documentResource) {
_resource = documentResource;
}
#Override
public InputStream getInputStream() throws ResourceStreamNotFoundException {
if (_fileInputStream == null) {
try {
if (_resource == null) {
throw new ResourceStreamNotFoundException("Resource was null");
}
_fileInputStream = _resource.getFileInputStream();
} catch (final ResourceNotAvailableException ex) {
throw new ResourceStreamNotFoundException(ex);
}
}
return _fileInputStream;
}
In HTML:
<a wicket:id="linkToPreview" href="#">
<img wicket:id="imageThumbnail" alt="Attachment"></img></a>

The code added hasn't really added any clues for me, but maybe I can help narrow it down a bit anyway.
The stacktrace includes a reference to com.webapp.document.pages.DocumentListPage, which is likely calling some of the code you've posted. The error indicates a bad url, so debugging into that class, adding debug prints, and looking at the values of any field containing a url might be worthwhile.
It might even help to modify the code in DocumentListPage (maybe temporarily for debugging) to catch org.apache.wicket.protocol.http.request.InvalidUrlException and adding debugging prints specifically when the exception is caught.
This isn't really an answer, but it's too big for a comment, and maybe it'll help you get closer to an answer.

The following solution solved the problem:
- extend WebResource class
- add extended class as a resource to application shared resources
Ex:
public class MyWebResource extends WebResource {
final ValueMap map = new ValueMap();
#Override
public IResourceStream getResourceStream() {
String fileName = getFileName();
File file = new File(basePath, fileName);
if (!file.exists()) {
LOG.error("File does not exist: " + file);
throw new IllegalStateException("File does not exist: " + file);
}
return new FileResourceStream(file);
}
public final void addResource() {
Application.get().getSharedResources().add(getClass().getName(), this);
}
protected String getFileName() {
return getParameters().getString("id");
}
public final String urlFor(final String fileName) {
final ResourceReference resourceReference = new ResourceReference(getClass().getName());
final String encodedValue = WicketURLEncoder.QUERY_INSTANCE.encode(fileName);
map.add("id", encodedValue);
final CharSequence result = RequestCycle.get().urlFor(resourceReference, map);
if (result == null) {
throw new IllegalStateException("The resource was not added! "
+ "In your Application class add the following line:"
+ "MyConcreteResource.INSTANCE.addResource()");
}
String absoluteUrl = RequestUtils.toAbsolutePath(result.toString());
return absoluteUrl;
}
}
In Application class, in init(), I have added MyWebResource to shared resources:
public void init() {
...
new MyWebResource().addResource();
...
}