I have a utility class I created for my Spring controller to invoke to generate a CSV from a collection of beans using the SuperCSV library ( http://supercsv.sourceforge.net/ )
The utility class is pretty basic:
public static void export2CSV(HttpServletResponse response,
String[] header, String filePrefix, List<? extends Object> dataObjs) {
try{
response.setContentType("text/csv;charset=utf-8");
response.setHeader("Content-Disposition","attachment; filename="+filePrefix+"_Data.csv");
OutputStream fout= response.getOutputStream();
OutputStream bos = new BufferedOutputStream(fout);
OutputStreamWriter outputwriter = new OutputStreamWriter(bos);
ICsvBeanWriter writer = new CsvBeanWriter(outputwriter, CsvPreference.EXCEL_PREFERENCE);
// the actual writing
writer.writeHeader(header);
for(Object anObj : dataObjs){
writer.write(anObj, header);
}
}catch (Exception e){
e.printStackTrace();
}
};
The catch is, I'm getting different behaviors out of this operation and I don't know why. When I invoke it from one controller (we'll call it 'A'), I get the expected output of data.
When I invoke it from the other controller ('B'), I get a tiny blurb of unrecognizable binary data that cannot be opened by OO Calc. Opening it in Notepad++ yields an unreadable line of gibberish that I can only assume is an attempt by the reader to show me a binary stream.
Controller 'A' invocation (the one that works)
#RequestMapping(value="/getFullReqData.html", method = RequestMethod.GET)
public void getFullData(HttpSession session, HttpServletRequest request, HttpServletResponse response) throws IOException{
logger.info("INFO: ******************************Received request for full Req data dump");
String projName= (String)session.getAttribute("currentProject");
int projectID = ProjectService.getProjectID(projName);
List<Requirement> allRecords = reqService.getFullDataSet(projectID);
final String[] header = new String[] {
"ColumnA",
"ColumnB",
"ColumnC",
"ColumnD",
"ColumnE"
};
CSVExporter.export2CSV(response, header, projName+"_reqs_", allRecords);
};
...and here's the Controller 'B' invocation (the one that fails):
#RequestMapping(value="/getFullTCData.html", method = RequestMethod.GET)
public void getFullData(HttpSession session, HttpServletRequest request, HttpServletResponse response) throws IOException{
logger.info("INFO: Received request for full TCD data dump");
String projName= (String)session.getAttribute("currentProject");
int projectID = ProjectService.getProjectID(projName);
List<TestCase> allRecords = testService.getFullTestCaseList(projectID);
final String[] header = new String[] {
"ColumnW",
"ColumnX",
"ColumnY",
"ColumnZ"
};
CSVExporter.export2CSV(response, header, projName+"_tcs_", allRecords);
}
Observations:
Which controller I invoke first is irrelevant. 'A' always works and 'B' always produces gibberish
Both calls to this function have a list of header columns that are a subset of the total set of operations defined in the bean being passed in to CSVWriter
The simple Exception printStackTrace is working to detect when a bean's reflection field doesn't match the definition (i.e., can't find get() to get the value programmatically) suggesting that all column/variable matchups are succeeding.
In the debugger, I've verified the writer.write(Object, header) call is being hit the expected number of times based on the number of objects being passed and that these objects have the expected data
Any suggestions or insights would be greatly appreciated. I'm really stumped how to better isolate the issue...
You aren't closing the writer. Also, CsvBeanWriter will wrap the writer in a BufferedWriter, so you can probably simplify your outputwriter as well.
public static void export2CSV(HttpServletResponse response,
String[] header, String filePrefix, List<? extends Object> dataObjs) {
ICsvBeanWriter writer;
try{
response.setContentType("text/csv;charset=utf-8");
response.setHeader("Content-Disposition",
"attachment; filename="+filePrefix+"_Data.csv");
OutputStreamWriter outputwriter =
new OutputStreamWriter(response.getOutputStream());
writer = new CsvBeanWriter(outputwriter, CsvPreference.EXCEL_PREFERENCE);
// the actual writing
writer.writeHeader(header);
for(Object anObj : dataObjs){
writer.write(anObj, header);
}
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
writer.close(); // closes writer and underlying stream
} catch (Exception e){}
}
};
Super CSV 2.0.0-beta-1 is out now! As well as adding numerous other features (including Maven support and a new Dozer extension), CSV writers now expose a flush() method as well.
Related
I have a Quarkus based REST API project in which one endpoint is supposed to serve exported data as .csv files. Since i do not want to create temporary files, i was writing to a ByteArrayInputStream to be used in an octet stream response for my webservice.
However, although this works fine for latin character content we also have content that may be in Chinese. The downloaded .csv file does not view the characters properly or rather does not write them properly (they only show up as question marks, even in plain text view e.g. with notepad).
We already checked the source of the problem not being how the data is stored, for example the encoding in the database is correct and it works fine when we export it as .json (here we can set charset utf-8).
As far as i understand a charset or encoding cannot be set for an octet stream.
So how can we export/stream this content as a file download without creating an actual file?
Some code examples below on how we do it currently. We use the apache common library component CSVPrinter to create the CSV format in text in a custom CSV streamer class:
#ApplicationScoped
public class JobRunDataCsvStreamer implements DataFormatStreamer<JobData> {
#Override
public ByteArrayInputStream streamDataToFormat(List<JobData> dataList) {
try {
ByteArrayOutputStream out = getCsvOutputStreamFor(dataList);
return new ByteArrayInputStream(out.toByteArray());
} catch (IOException e) {
throw new RuntimeException("Failed to convert job data: " + e.getMessage());
}
}
private ByteArrayOutputStream getCsvOutputStreamFor(List<JobData> dataList) throws IOException {
ByteArrayOutputStream out = new ByteArrayOutputStream();
CSVPrinter csvPrinter = new CSVPrinter(new PrintWriter(out), getHeaderFormat());
for (JobData jobData : dataList) {
csvPrinter.printRecord(extractStringRowData(jobData));
}
csvPrinter.flush();
csvPrinter.close();
return out;
}
private CSVFormat getHeaderFormat() {
return CSVFormat.EXCEL
.builder()
.setDelimiter(";")
.setHeader("ID", "Source term", "Target term")
.build();
}
private List<String> extractStringRowData(JobData jobData) {
return Arrays.asList(
String.valueOf(jobData.getId()),
jobData.getSourceTerm(),
jobData.getTargetTerm()
);
}
}
Here is the quarkus API endpoint for the download:
#Path("/jobs/data")
public class JobDataResource {
#Inject JobDataRepository jobDataRepository;
#Inject JobDataCsvStreamer jobDataCsvStreamer;
...
#GET
#Path("/export/csv")
#Produces(MediaType.APPLICATION_OCTET_STREAM)
public Response getAllAsCsvExport() {
List<JobData> jobData = jobDataRepository.getAll();
ByteArrayInputStream stream = jobDataCsvStreamer.streamDataToFormat(jobData);
return Response.ok(stream, MediaType.APPLICATION_OCTET_STREAM)
.header("content-disposition", "attachment; filename = job-data.csv")
.build();
}
}
Screenshot of result in the downloaded file for chinese characters in the second column:
We tried setting headers etc. for encoding, but none of it worked. Is there a way to stream content which requires specific encoding as a file in Java web services? We tried using PrintWriter which works, but requies creating a local file on the server.
Edit: We tried using PrintWriter(out, false, StandardCharsets.UTF_8) for the PrintWriter to write to a byte array out stream for the response, which yields a different result but still with broken view in both Excel and plain text:
Screenshot:
Code for endpoint:
#GET
#Path("/export/csv")
#Produces(MediaType.APPLICATION_OCTET_STREAM)
public Response getAllAsCsvExport() {
List<JobData> jobData = jobRunDataRepository.getAll();
ByteArrayOutputStream out = new ByteArrayOutputStream();
try{
PrintWriter pw = new PrintWriter(out, false, StandardCharsets.UTF_8);
pw.println(String.format("%s, %s, %s", "ID", "Source", "Target"));
for (JobData item : jobData) {
pw.println(String.format("%s, %s, %s",
String.valueOf(item.getId()),
String.valueOf(item.getSourceTerm()),
String.valueOf(item.getTargetTerm()))
);
}
pw.flush();
pw.close();
} catch (Exception e) {
throw new RuntimeException("Failed to convert job data: " + e.getMessage());
}
return Response.ok(out).build();
}
I would like to write a Spring controller method which returns an image from storage. Below is my current version, but it has two problems:
The #GetMapping annotation requires the 'produces' parameter which is a string array of media types. The program does not work if that parameter is not present; it just displays the image data as text. The problem is that if I want to support an additional media type then I have to recompile the program. Is there a way to set up the 'produces' media type from inside the viewImg method?
The code below will display any image type except svg, which will display only the message "The image cannot be displayed because it contains errors". The web browser (Firefox) identifies it as media type "webp". However, if I remove all media types from the 'produces' string array except the "image/svg+xml" entry, the image is displayed.
Please advise how to write a controller method that is more general (so that it works with any media type) and does not have issues with svg media type.
Here is my test code:
#GetMapping(value = "/pic/{id}",
produces = {
"image/bmp",
"image/gif",
"image/jpeg",
"image/png",
"image/svg+xml",
"image/tiff",
"image/webp"
}
)
public #ResponseBody
byte[] viewImg(#PathVariable Long id) {
byte[] data = new byte[0];
String inputFile = "/path/to/image.svg";
try {
InputStream inputStream = new FileInputStream(inputFile);
long fileSize = new File(inputFile).length();
data = new byte[(int) fileSize];
inputStream.read(data);
} catch (IOException e) {
e.printStackTrace();
}
return data;
}
I recommend FileSystemResource for handling file contents. You can avoid .contentType(..) started line if you don't want to send Content-Type value.
#GetMapping("/pic/{id}")
public ResponseEntity<Resource> viewImg(#PathVariable Long id) throws IOException {
String inputFile = "/path/to/image.svg";
Path path = new File(inputFile).toPath();
FileSystemResource resource = new FileSystemResource(path);
return ResponseEntity.ok()
.contentType(MediaType.parseMediaType(Files.probeContentType(path)))
.body(resource);
}
I'm working on a spring boot project with thymeleaf and I need to create a file and put some lines on it then send it for the user to download it.
#PostMapping("/filegenerator")
public String createFile(#ModelAttribute("page") Page page, Model model) {
List<String> lines = new ArrayList<String>();
//Some code ..........
lines.forEach(l->{
System.out.println(l);
});
//Here I need to create a file with the List of lines
//And also put some code to download it
return "filegenerator";
}
So if you want to return a file, you probably want to stream it to limit the amount of memory used (or at least that was probably the reasoning of Spring Framework creators). In your case I understand that the file is fairly small and doesn't really have to be persisted anywhere. It's just a one time download based on the uploaded form, right?
so this approach worked on my PC:
#PostMapping("/filegenerator")
public void createFile(HttpServletResponse response) throws IOException {
List<String> lines = Arrays.asList("line1", "line2");
InputStream is = new ByteArrayInputStream(lines.stream().collect(Collectors.joining("\n")).getBytes());
IOUtils.copy(is, response.getOutputStream());
response.setContentType("application/sql");
response.setHeader("Content-Disposition", "attachment; filename=\"myquery.sql\"");
response.flushBuffer();
}
note the content-disposition header. It explicitly states that you don't want to display the file in the browser, but instead you want it to be downloaded as a file and myquery.sql is the name of that file that will be downloaded.
#Kamil janowski
This is how it looks like now
#PostMapping("/filegenerator")
public String createFile(#ModelAttribute("page") Page page, Model model,HttpServletResponse response) throws IOException{
List<String> lines = new ArrayList<String>();
if(page.getTables().get(0).getName()!="") {
List<String> output = so.createTable(page.getTables());
output.forEach(line -> lines.add(line));
}
if(page.getInserttables().get(0).getName()!="") {
List<String> output = so.insert(page.getInserttables());
output.forEach(line -> lines.add(line));
}
if(page.getUpdatetables().get(0).getName()!="") {
List<String> output = so.update(page.getUpdatetables());
output.forEach(line -> lines.add(line));
}
lines.forEach(l->{
System.out.println(l);
});
InputStream is = new ByteArrayInputStream(lines.stream().collect(Collectors.joining("\n")).getBytes());
IOUtils.copy(is, response.getOutputStream());
response.setContentType("application/sql");
response.setHeader("Content-Disposition", "attachment; filename=\"myquery.sql\"");
response.flushBuffer();
model.addAttribute("page", getPage());
return "filegenerator";
}
I noticed that the following code will use the same OutputStream object (same hash code on debug) if a curl request is ctrl+c, then re-run. Even with different parameters, it uses the same OutputStream. This causes very odd output obviously.
The original output stream starts to throw NullPointerException when writing to the OutputStream because the underlying HttpOutputStream no longer exists (broken pipe). This is why I think it is odd that a subsequent request would reuse the same OutputStream object.
Closing the output stream in a finally block fixes the issue. Some examples I've seen around the web don't explicitly close the output stream. Is the reuse of the OutputStream expected? Does anyone have any ideas on why I'd be seeing this behavior?
#RequestMapping(value= URI_ROOT, method= RequestMethod.POST, produces = {"text/event-stream"})
public StreamingResponseBody methodName(
... params
) {
return new StreamingResponseBody() {
#Override
public void writeTo(OutputStream outputStream) {
try {
... code ...
} catch (Exception e) {
LOGGER.info("Migration thread interrupted.",e);
} finally {
IoUtil.closeSilently(outputStream); // This fixes it.
}
}
};
}
I've got a Java web service in JAX-WS that returns an OutputStream from another method. I can't seem to figure out how to stream the OutputStream into the returned DataHandler any other way than to create a temporary file, write to it, then open it back up again as an InputStream. Here's an example:
#MTOM
#WebService
class Example {
#WebMethod
public #XmlMimeType("application/octet-stream") DataHandler service() {
// Create a temporary file to write to
File fTemp = File.createTempFile("my", "tmp");
OutputStream out = new FileOutputStream(fTemp);
// Method takes an output stream and writes to it
writeToOut(out);
out.close();
// Create a data source and data handler based on that temporary file
DataSource ds = new FileDataSource(fTemp);
DataHandler dh = new DataHandler(ds);
return dh;
}
}
The main issue is that the writeToOut() method can return data that are far larger than the computer's memory. That's why the method is using MTOM in the first place - to stream the data. I can't seem to wrap my head around how to stream the data directly from the OutputStream that I need to provide to the returned DataHandler (and ultimately the client, who receives the StreamingDataHandler).
I've tried playing around with PipedInputStream and PipedOutputStream, but those don't seem to be quite what I need, because the DataHandler would need to be returned after the PipedOutputStream is written to.
Any ideas?
I figured out the answer, along the lines that Christian was talking about (creating a new thread to execute writeToOut()):
#MTOM
#WebService
class Example {
#WebMethod
public #XmlMimeType("application/octet-stream") DataHandler service() {
// Create piped output stream, wrap it in a final array so that the
// OutputStream doesn't need to be finalized before sending to new Thread.
PipedOutputStream out = new PipedOutputStream();
InputStream in = new PipedInputStream(out);
final Object[] args = { out };
// Create a new thread which writes to out.
new Thread(
new Runnable(){
public void run() {
writeToOut(args);
((OutputStream)args[0]).close();
}
}
).start();
// Return the InputStream to the client.
DataSource ds = new ByteArrayDataSource(in, "application/octet-stream");
DataHandler dh = new DataHandler(ds);
return dh;
}
}
It is a tad more complex due to final variables, but as far as I can tell this is correct. When the thread is started, it blocks when it first tries to call out.write(); at the same time, the input stream is returned to the client, who unblocks the write by reading the data. (The problem with my previous implementations of this solution was that I wasn't properly closing the stream, and thus running into errors.)
Sorry, I only did this for C# and not java, but I think your method should launch a thread to run "writeToOut(out);" in parralel. You need to create a special stream and pass it to the new thread which gives that stream to writeToOut. After starting the thread you return that stream-object to your caller.
If you only have a method that writes to a stream and returns afterwards and another method that consumes a stream and returns afterwards, there is no other way.
Of coure the tricky part is to get hold of such a -multithreading safe- stream: It shall block each side if an internal buffer is too full.
Don't know if a Java-pipe-stream works for that.
Wrapper pattern ? :-).
Custom javax.activation.DataSource implementation (only 4 methods) to be able to do this ?
return new DataHandler(new DataSource() {
// implement getOutputStream to return the stream used inside writeToOut()
...
});
I don't have the IDE available to test this so i'm only doing a suggestion. I would also need the writeToOut general layout :-).
In my application I use InputStreamDataSource implementation that take InputStream as constructor argument instead of File in FileDataSource. It works so far.
public class InputStreamDataSource implements DataSource {
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
private final String name;
public InputStreamDataSource(InputStream inputStream, String name) {
this.name = name;
try {
int nRead;
byte[] data = new byte[16384];
while ((nRead = inputStream.read(data, 0, data.length)) != -1) {
buffer.write(data, 0, nRead);
}
buffer.flush();
inputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
#Override
public String getContentType() {
return new MimetypesFileTypeMap().getContentType(name);
}
#Override
public InputStream getInputStream() throws IOException {
return new ByteArrayInputStream(buffer.toByteArray());
}
#Override
public String getName() {
return name;
}
#Override
public OutputStream getOutputStream() throws IOException {
throw new IOException("Read-only data");
}
}