I have created simple rest api to serve files from hdfs (Files are large and I don't want to copy them locally).
I would like to log information that file download completed successfully i.e. whole stream was read, but I do not know how. I can only log information that file download was started.
I will appreciate any help.
#Autowired
private FileDownloadService fds;
#RequestMapping(value = GET_FILE_PATH, method = RequestMethod.GET)
#Produces(MediaType.APPLICATION_OCTET_STREAM_VALUE)
public ResponseEntity getFileStream(#RequestParam("name") String name) {
LOG.info("Processing for filename: " + name);
try {
Path p = fds.getFilePath(name);
org.apache.hadoop.fs.FSDataInputStream is = fds.getFileStream(p);
return ResponseEntity.ok().header("Content-Disposition", "attachment; filename=\"" + p.getName() + "\"'").contentLength(fds.getFileLength(p))
.contentType(MediaType.APPLICATION_OCTET_STREAM).body(new InputStreamResource(is));
} catch (Exception e) {
LOG.error(e.getLocalizedMessage(), e);
return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body("Internal Server Error");
} finally {
LOG.info("File: " + name + " download started");
}
}
You can try to create a wrapper over InputStream and trigger some flag on stream closing (close()).
For instance you can take ProxyInputStream as a basis:
ProxyInputStreamis = new ProxyInputStream(fds.getFileStream(p)) {
#Override
public void close() throws IOException {
super.close();
// some trigger
}
};
Related
on sftp i have several files with following xyz names:
40_20200313_0cd6963f-bf5b-4eb0-b310-255a23ed778e_p.dat
123_20200313_0cd6963f-bf5b-4eb0-b310-255a23ed778e_p.dat
etc.
I want camel to download all files at once as currently it is downloading file one by one.
Following is camel route and query:
private static String regex() {
return "(22|23|24|25|26|28|29|32|35|40|41|46|52|70|85|88|123)_(?:.*)_p.dat";
}
private static String sftpComponent() {
return "sftp://transit.ergogroup.no/Eyeshare/From_Eyeshare_Test"
+ "?username=Eyeshare_test"
+ "&password=epw3ePOugG" // Stored on wildfly server
+ "&download=true" //Shall be read chunk by chunk to avoid heap space issues. Earlier download=true was used: Harpreet
+ "&useList=true"
+ "&stepwise=false"
+ "&disconnect=true"
+ "&passiveMode=true"
+ "&reconnectDelay=10000"
// + "&bridgeErrorHandler=true"
+ "&delay=300000"
//+ "&fileName=" + sftpFileName
// + "&include=kiki\\.txt"
// + "&include=40_*_p\\.dat"sss
+ "&include="+regex()
+ "&preMove=$simple{file:onlyname}.$simple{date:now:yyyy-MM-dd'T'hh-mm-ss}.processing"
+ "&move=$simple{file:onlyname.noext}.$simple{date:now:yyyy-MM-dd'T'hh-mm-ss}.success"
+ "&moveFailed=$simple{file:onlyname.noext}.$simple{date:now:yyyy-MM-dd'T'hh-mm-ss}.failed";
// + "&idempotentRepository=#infinispan"
// + "&readLockRemoveOnCommit=true";
}
from(sftpComponent()).log("CHU").to(archiveReceivedFile())
Code appears fine but output is not. Anyone kindly suggest
Here some example of aggregator:
from("file:///somePath/consume/?maxMessagesPerPoll=2&delay=5000")
.aggregate(constant(true), new ZipAggregationStrategy()).completion(exchange -> exchange.getProperty("CamelBatchComplete", Boolean.class))
.to("file:///somePath/produce/")
Here maxMessagesPerPoll defining how many files will be archived. But if number of them in folder is lower then maxMessagesPerPoll value it will wait for missing files for complete archive. Here example of ZipAggregationStrategy:
private static class ZipAggregationStrategy implements AggregationStrategy {
private ZipOutputStream zipOutputStream;
private ByteArrayOutputStream out;
#Override
public Exchange aggregate(final Exchange oldExchange, final Exchange newExchange) {
try {
if (oldExchange == null) {
out = new ByteArrayOutputStream();
zipOutputStream = new ZipOutputStream(out);
}
createEntry(newExchange);
return newExchange;
} catch (Exception e) {
throw new RuntimeException(e);
}
}
private void createEntry(final Exchange exchange) throws Exception {
final ZipEntry zipEntry = new ZipEntry(exchange.getIn().getHeader(Exchange.FILE_NAME, String.class));
zipOutputStream.putNextEntry(zipEntry);
byte[] bytes = new byte[1024];
int length;
try (InputStream body = exchange.getIn().getBody(InputStream.class)) {
while ((length = body.read(bytes)) >= 0) {
zipOutputStream.write(bytes, 0, length);
}
}
}
#Override
public void onCompletion(final Exchange exchange) {
try {
zipOutputStream.close();
exchange.getIn().setBody(new ByteArrayInputStream(out.toByteArray()));
exchange.getIn().setHeader(Exchange.FILE_NAME, "someArchive.zip");
}catch (Exception e){
throw new RuntimeException(e);
} finally {
try {
out.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
It's in-memory example. You can improve it for example with using temporary file. And you can always create your own completion predicate based on your logic.
UPD: i think link for documentation is temporary unavailable
I want to get one line as header and then rest of data append in the file.but i am facing issue that it is saving the header repeatedly when i have called the function.
Expected output should be like
Id : Title : Group ID
1 : ab : 2
2 : fd : 3
3 : fwsj : 3
public void writeOutputToFile(int id, String title, int groupId) throws IOException {
OutputStream os = new FileOutputStream(new File("output_report.txt"), true);
os.write("\n Id Title Group ID \n ".getBytes());
os.write((id + " " +title + " " + groupId + "\n").getBytes());
os.close();
}
well, inside your method you write the headers to the file, so obviously whenever you call it they'll get written..
You can separate it to two methods- one that writes the headers (and called only once) and another that writes the data (and called once per row).
Alternatively, use some sort of loop inside your method to write each of the lines to the file, after writing the headers once.
The Problem
It is repeatedly putting in the header, because when you call the method, you are always going to insert the header. Instead, you may want to code a util that inputs headers for a file you are creating, and then a separate method for inserting the data.
The Solution
Solution 1)
The helper util method would look something like this:
// String... allows for multiple string parameters to be entered for all of your headers.
public void prepFile(File f, String... headers) {
StringBuffer buffer = new StringBuffer();
for (String header : headers) {
buffer.append(header + "\t");
}
OutputStream os = new FileOutputStream(f, true);
os.write(buffer.toString().getBytes());
os.close();
}
After the file is prepped, you can then use your writeOutputToFile method for all the data.
Edit
Solution 2)
If you were going to make a stand alone class for this, I would recommend you set it up like so:
import java.io.*;
public class OutputFile {
private File file;
private String[] headers;
private boolean existed;
public OutputFile(File f, String... headers) {
this.file = f;
this.headers = headers;
init();
}
private void init() {
existed = file.exists();
// If the file didn't exist, then you want to create it.
if (!existed) {
try {
file.createNewFile();
// Afterwards, you can then write your headers to it.
if (headers != null) {
writeData(headers);
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
public void writeData(int id, String title, int groupId) {
writeData("" + id, title, "" + groupId);
}
public void writeData(String... strings) {
StringBuffer buffer = new StringBuffer();
for (String s : strings) {
buffer.append(s + "\t");
}
buffer.append("\n");
writeData(buffer.toString());
}
public void writeData(String data) {
OutputStream os = null;
try {
os = new FileOutputStream(file, true);
os.write(data.getBytes());
os.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
if (os != null) {
try {
os.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
}
I am trying to upload a file to my server using an endpoint exposed through spring. However when I try to test the api through postman, I get Current request is not a multipart request error. I went through this question MultipartException: Current request is not a multipart request but still couldn't fix this. Please Help. Thanks in advance.
Here is my controller:
#RestController
#RequestMapping
public class UploadController {
#Autowired
StorageService storageService;
List<String> files = new ArrayList<String>();
#PostMapping("/post")
public ResponseEntity<String> handleFileUpload(#RequestParam("file") MultipartFile file) {
String message = "";
try {
storageService.store(file);
files.add(file.getOriginalFilename());
message = "You successfully uploaded " + file.getOriginalFilename() + "!";
return ResponseEntity.status(HttpStatus.OK).body(message);
} catch (Exception e) {
message = "FAIL to upload " + file.getOriginalFilename() + "!";
return ResponseEntity.status(HttpStatus.EXPECTATION_FAILED).body(message);
}
}
#GetMapping("/getallfiles")
public ResponseEntity<List<String>> getListFiles(Model model) {
List<String> fileNames = files
.stream().map(fileName -> MvcUriComponentsBuilder
.fromMethodName(UploadController.class, "getFile", fileName).build().toString())
.collect(Collectors.toList());
return ResponseEntity.ok().body(fileNames);
}
#GetMapping("/files/{filename:.+}")
#ResponseBody
public ResponseEntity<Resource> getFile(#PathVariable String filename) {
Resource file = storageService.loadFile(filename);
return ResponseEntity.ok()
.header(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=\"" + file.getFilename() + "\"")
.body(file);
}
}
My Service:
#Service
public class StorageService {
Logger log = LoggerFactory.getLogger(this.getClass().getName());
private final Path rootLocation = Paths.get("upload-dir");
public void store(MultipartFile file) {
try {
Files.copy(file.getInputStream(), this.rootLocation.resolve(file.getOriginalFilename()));
} catch (Exception e) {
throw new RuntimeException("FAIL!");
}
}
public Resource loadFile(String filename) {
try {
Path file = rootLocation.resolve(filename);
Resource resource = new UrlResource(file.toUri());
if (resource.exists() || resource.isReadable()) {
return resource;
} else {
throw new RuntimeException("FAIL!");
}
} catch (MalformedURLException e) {
throw new RuntimeException("FAIL!");
}
}
public void deleteAll() {
FileSystemUtils.deleteRecursively(rootLocation.toFile());
}
public void init() {
try {
Files.createDirectory(rootLocation);
} catch (IOException e) {
throw new RuntimeException("Could not initialize storage!");
}
}
}
As you can see below I am sending file as form Data and no headers are being set
see below in the image, and add key value as file
Your Controller expects a request parameter "file":
#RequestParam("file") MultipartFile file
You have to set the key "file" in postman, where the value is your file (last screenshot).
Try adding in your request header Content-Type: multipart/form-data (as far as I see in postman it is missing)
I am currently working on a project about calculations.I have done the main part of my project,Also integrated SVN Commit function to my code (using .ini file to read the specific address etc. )
I can easily Commit the files, what I am trying is I want to implement the real-time log to my console. Is there any way to implement the log to the console ? Not the general log but the commit log which should be real time.
I am using eclipse for mac, I've heard about SVNKit but I am really poor about SVN.
Thanks in advance for any information
--- EDIT ---
This is the code for reading the svn commands from .ini file
public static String iniSVNOkut(String deger, String getObje, String fetchObje){
Ini uzantilariAlIni = null;
try
{
String uzantiAyarlari = "Uzantilar.ini";
try
{
uzantilariAlIni = new Ini(new FileReader(uzantiAyarlari));
}
catch (InvalidFileFormatException e)
{
System.err.print("Hata InvalidFileFormat : " + e.getMessage() + "\n" );
e.printStackTrace();
}
catch(FileNotFoundException e)
{
System.err.print("Hata FileNotFoundException : " + e.getMessage() + "\n" );
e.printStackTrace();
}
catch (IOException e)
{
System.err.print("Hata IOException : " + e.getMessage() + "\n" );
e.printStackTrace();
}
return deger = uzantilariAlIni.get(getObje).fetch(fetchObje);
}
finally
{
}
}
This is what .ini includes
[svnAdresi]
svnAdresiniAl = svn co http://svn.svnkit.com/repos/svnkit/trunk/ /Users/sample/Documents/workspace/SatirHesaplaGUI/svnTestMAC
This is how I call it
String svnAdresi;
svnAdresi = IniFonksiyon.iniSVNOkut(svnAdresi, "svnAdresi", "svnAdresiniAl");
Runtime cmdCalistir = Runtime.getRuntime();
try
{
Process islem = cmdCalistir.exec(svnAdresi);
}
catch (Exception e)
{
e.printStackTrace();
}
If I understand your question correctly, you want to read the Subversion commit log into your console application.
The easiest way is to use SVNKit.
Here's how I did it.
private static List<SVNLogEntry> logEntryList;
/*
* Gets the Subversion log records for the directory
*/
LogHandler handler = new LogHandler();
String[] paths = { directory };
try {
repository.log(paths, latestRevision, 1L, false, true, handler);
} catch (SVNException svne) {
if (svne.getMessage().contains("not found")) {
logEntryList = new ArrayList<SVNLogEntry>();
} else {
CobolSupportLog.logError(
"Error while fetching the repository history: "
+ svne.getMessage(), svne);
return false;
}
}
logEntryList = handler.getLogEntries();
directory - string pointing to a particular directory or module
latestRevision - largest revision number from Subversion. Placing the latestRevision second in the log method invocation returns the log records in most recent order.
If you want the log records in sequential order, from 1 to latestRevision, then the 1L would be placed second, and the latestRevision would be placed third.
repository - Subversion repository that you've already authenticated.
Here's LogHandler.
public class LogHandler implements ISVNLogEntryHandler {
protected static final int REVISION_LIMIT = 5;
protected List<SVNLogEntry> logEntryList;
public LogHandler() {
logEntryList = new ArrayList<SVNLogEntry>();
}
public void handleLogEntry(SVNLogEntry logEntry) throws SVNException {
logEntryList.add(logEntry);
}
public List<SVNLogEntry> getLogEntries() {
if (logEntryList.size() <= REVISION_LIMIT) {
return logEntryList;
} else {
return logEntryList.subList(0, REVISION_LIMIT);
}
}
}
I want to change modification timestamp of a binary file. What is the best way for doing this?
Would opening and closing the file be a good option? (I require a solution where the modification of the timestamp will be changed on every platform and JVM).
The File class has a setLastModified method. That is what ANT does.
My 2 cents, based on #Joe.M answer
public static void touch(File file) throws IOException{
long timestamp = System.currentTimeMillis();
touch(file, timestamp);
}
public static void touch(File file, long timestamp) throws IOException{
if (!file.exists()) {
new FileOutputStream(file).close();
}
file.setLastModified(timestamp);
}
Since File is a bad abstraction, it is better to use Files and Path:
public static void touch(final Path path) throws IOException {
Objects.requireNonNull(path, "path is null");
if (Files.exists(path)) {
Files.setLastModifiedTime(path, FileTime.from(Instant.now()));
} else {
Files.createFile(path);
}
}
Here's a simple snippet:
void touch(File file, long timestamp)
{
try
{
if (!file.exists())
new FileOutputStream(file).close();
file.setLastModified(timestamp);
}
catch (IOException e)
{
}
}
I know Apache Ant has a Task which does just that.
See the source code of Touch (which can show you how they do it)
They use FILE_UTILS.setFileLastModified(file, modTime);, which uses ResourceUtils.setLastModified(new FileResource(file), time);, which uses a org.apache.tools.ant.types.resources.Touchable, implemented by org.apache.tools.ant.types.resources.FileResource...
Basically, it is a call to File.setLastModified(modTime).
This question only mentions updating the timestamp, but I thought I'd put this in here anyways. I was looking for touch like in Unix which will also create a file if it doesn't exist.
For anyone using Apache Commons, there's FileUtils.touch(File file) that does just that.
Here's the source from (inlined openInputStream(File f)):
public static void touch(final File file) throws IOException {
if (file.exists()) {
if (file.isDirectory()) {
throw new IOException("File '" + file + "' exists but is a directory");
}
if (file.canWrite() == false) {
throw new IOException("File '" + file + "' cannot be written to");
}
} else {
final File parent = file.getParentFile();
if (parent != null) {
if (!parent.mkdirs() && !parent.isDirectory()) {
throw new IOException("Directory '" + parent + "' could not be created");
}
}
final OutputStream out = new FileOutputStream(file);
IOUtils.closeQuietly(out);
}
final boolean success = file.setLastModified(System.currentTimeMillis());
if (!success) {
throw new IOException("Unable to set the last modification time for " + file);
}
}
If you are already using Guava:
com.google.common.io.Files.touch(file)