I want to download a few photos by URL using Webflux and AsynchronousFileChannel, and all files are created but empty.
Here is my code:
public void downloadFilesFromUrl() throws IOException {
List<Photo> notDownloadedFiles = //get photos with name and URL;
for (Photo photo : notDownloadedFiles) {
Path path = Paths.get(pathToFiles + File.separator + photo.getPhotoName());
WebClient client = WebClient.builder().baseUrl(photo.getLoadSource()).build();
Flux<DataBuffer> dataBufferFlux = client
.get().accept(MediaType.APPLICATION_OCTET_STREAM)
.retrieve()
.bodyToFlux(DataBuffer.class);
saveFileOnComputer(path, dataBufferFlux);
}
}
private void saveFileOnComputer(Path path, Flux<DataBuffer> dataBufferFlux) throws IOException {
AsynchronousFileChannel asynchronousFileChannel = AsynchronousFileChannel.open(path, CREATE, WRITE);
DataBufferUtils.write(dataBufferFlux, asynchronousFileChannel)
.doOnNext(DataBufferUtils.releaseConsumer())
.doAfterTerminate(() -> {
try {
asynchronousFileChannel.close();
} catch (IOException ignored) { }
}).then();
}
If I try to use
DataBufferUtils.write(dataBufferFlux, path, StandardOpenOption.CREATE).block();
instead of calling saveFileOnServer(..) method, everything is fine. But I want to use exactly AsynchronousFileChannel.
Okay, I think I fixed it.
private void saveFileOnServer(Path path, Flux<DataBuffer> dataBufferFlux) throws IOException {
AsynchronousFileChannel asynchronousFileChannel = AsynchronousFileChannel.open(path, CREATE, WRITE);
DataBufferUtils.write(dataBufferFlux, asynchronousFileChannel).subscribe();
}
The official documentation says "Note that the writing process does not start until the returned Flux is subscribed to".
Related
im trying to send files to FTPS server
connection method: FTPS, ACTIVE, EXPLICIT
setFileType(FTP.BINARY_FILE_TYPE);
setFileTransferMode(FTP.BLOCK_TRANSFER_MODE);
Checking the reply string right after connect i got:
234 AUTH command ok. Expecting TLS Negotiation.
from here
234 Specifies that the server accepts the authentication mechanism specified by the client, and the exchange of security data is complete. A higher level nonstandard code created by Microsoft.
while trying to send file with storeFile or storeUniqeFile i get false
checking the reply string right after store file i got: 501 Server cannot accept argument.
what is weird i was able creating a directory to this client without any issues
with makeDirectory("test1");
i was trying both this links : link1 , link2
FOR EXAMPLE when i was trying to use ftp.enterLocalPassiveMode(); before ftp.storeFile(destinationfile, in);
i got time out error .
Does anyone have any idea how to solve it ?
public static void main(String[] args) throws Exception {
FTPSProvider ftps = new FTPSProvider();
String json = "connection details";
DeliveryDetailsFTPS details = gson.fromJson(json, DeliveryDetailsFTPS .class);
File file = File.createTempFile("test", ".txt");
FileUtils.write(file, " some test", true);
try (FileInputStream stream = new FileInputStream(file)) {
ftps.sendInternal(ftps.getClient(details), details, stream, file.getName());
}
}
protected void sendInternal(FTPClient client, DeliveryDetailsFTPS details, InputStream stream, String filename) throws Exception {
try {
// release the enc
DeliveryDetailsFTPS ftpDetails = (DeliveryDetailsFTPS) details;
setClient(client, ftpDetails);
boolean isSaved = false;
try (BufferedInputStream bis = new BufferedInputStream(stream)) {
isSaved = client.storeFile(filename, bis);
}
client.makeDirectory("test1");
client.logout();
if (!isSaved) {
throw new IOException("Unable to upload file to FTP");
}
} catch (Exception ex) {
LOG.debug("Unable to send to FTP", ex);
throw ex;
} finally {
client.disconnect();
}
}
#Override
protected FTPClient getClient(DeliveryDetails details) {
return new FTPSClient(isImplicitSSL((DeliveryDetailsFTPS ) details));
}
protected void setClient(FTPClient client, DeliveryDetailsFTPS details) throws Exception {
DeliveryDetailsFTPS ftpDetails = (DeliveryDetailsFTPS ) details;
client.setConnectTimeout(100000);
client.setDefaultTimeout(10000 * 60 * 2);
client.setControlKeepAliveReplyTimeout(300);
client.setControlKeepAliveTimeout(300);
client.setDataTimeout(15000);
client.connect(ftpDetails.host, ftpDetails.port);
client.setBufferSize(1024 * 1024);
client.login(ftpDetails.username, ftpDetails.getSensitiveData());
client.setControlEncoding("UTF-8");
int code = client.getReplyCode();
if (code == 530) {
throw new IOException(client.getReplyString());
}
// Set binary file transfer
client.setFileType(FTP.BINARY_FILE_TYPE);
client.setFileTransferMode(FTP.BLOCK_TRANSFER_MODE);
if (ftpDetails.ftpMode == FtpMode.PASSIVE) {
client.enterLocalPassiveMode();
}
client.changeWorkingDirectory(ftpDetails.path);
}
I have tried this solution as well didn't solve the problem:
they only way i was able send file is with FileZilla and it is using FTPES .
But i need my Java code to do it . can anyone give me a clue
I have tried almost any possible solution offered on different websites could not make it work with Apache FTPS CLIENT ,
had to use a different class which worked like a charm here is a snippet:
com.jscape.inet.ftps Link
private Ftps sendWithFtpsJSCAPE(ConnDetails details, InputStream stream, String filename) throws FtpException, IOException {
Ftps ftp;
FtpConnectionDetails ftpDetails = FtpConnectionDetails details;
ftp = new Ftps(ftpDetails.getHost(), ftpDetails.getUsername(), ftpDetails.getPassword());
if (ftpDetails.getSecurityMode().equals(FtpConnectionDetails.SecurityMode.EXPLICIT)) {
ftp.setConnectionType(Ftps.AUTH_TLS);
} else {
ftp.setConnectionType(Ftps.IMPLICIT_SSL);
}
ftp.setPort(ftpDetails.getPort());
if (!ftpDetails.getFtpMode().equals(FtpMode.ACTIVE)) {
ftp.setPassive(true);
}
ftp.setTimeout(FTPS_JSCAPE_TIME_OUT);
ftp.connect();
ftp.setBinary();
ftp.setDir(ftpDetails.getPath());
ftp.upload(stream, filename);
return ftp;
}
I am using OkHttp 3.1.2.
I've created file upload similar to the original recipe which is found here: https://github.com/square/okhttp/blob/master/samples/guide/src/main/java/okhttp3/recipes/PostMultipart.java
I can't find example how to abort an upload of large file upon user request. I mean not how to get the user request but how to tell the OkHttp to stop sending data.
So far the only solution that I can imagine is to use custom RequestBody, add an abort() method and override the writeTo() method like this:
public void abort() {
aborted = true;
}
#Override
public void writeTo(BufferedSink sink) throws IOException {
Source source = null;
try {
source = Okio.source(mFile);
long transferred = 0;
long read;
while (!aborted && (read = source.read(sink.buffer(), SEGMENT_SIZE)) != -1) {
transferred += read;
sink.flush();
mListener.transferredSoFar(transferred);
}
} finally {
Util.closeQuietly(source);
}
}
Is there any other way?
It turns out it is quite easy:
Just hold reference to the Call object and cancel it when needed like this:
private Call mCall;
private void executeRequest (Request request) {
mCall = mOkHttpClient.newCall(request);
try {
Response response = mCall.execute();
...
} catch (IOException e) {
if (!mCall.isCanceled()) {
mLogger.error("Error uploading file: {}", e);
uploadFailed(); // notify whoever is needed
}
}
}
public void abortUpload() {
if (mCall != null) {
mCall.cancel();
}
}
Please note that when you cancel the Call while uploading an IOException will be thrown so you have to check in the catch if it is cancelled (as shown above) otherwise you will have false positive for error.
I think the same approach can be used for aborting download of large files.
I am working in a old project.The project is in Spring MVC .In the project I have to generate a pdf file from a jsp page and store in a location and download that file. For that I am using wkhtmltopdf tool to convert the one specific jsp page into pdf format. Using wkhtmltopdf sometime works fine, it generate the pdf in specific location, but sometime it require more time. Also when I am trying to download the file from specific location , sometime it download a 0KB size file or sometime the downloaded file can't be open (with some size) but sometime download perfectly. If I check the file at define location, it exist and open normally.
Here is my code in controller class.
#RequestMapping(value="/dwn.htm",method=RequestMethod.GET)
public void dwAppFm(HttpSession session,HttpServletRequest request,HttpServletResponse response,#RequestParam String id) throws IOException,InterruptedException
{
final int BUFFER_SIZES=4096;
ServletContext context=request.getServletContext();
String savePath="/tmp/";//PDF file Generate Path
String fileName="PDFFileName"; //Pdf file name
FileInputStream inputStream=null;
BufferedInputStream bufferedInputStream=null;
OutputStream outputStream=null;
printApp(id,fileName);
Thread.sleep(1000);
printApp(id,fileName);
File download=new File(savePath+fileName+".pdf");
while(!download.canRead())
{
Thread.sleep(1000);
printApp(id,fileName);
download=new File(savePath+fileName+".pdf");
}
if(download.canRead()){//if the file can read
try{
Thread.sleep(1000);
inputStream=new FileInputStream(download);
bufferedInputStream=new BufferedInputStream(inputStream);
String mimeType = context.getMimeType(savePath+fileName+".pdf");
if (mimeType == null) {
mimeType = "application/octet-stream";
}
System.out.println("MIME type: " + mimeType);
response.setContentType(mimeType);
response.setContentLength((int)download.length());
String headerKey="Content-Disposition";
String headerValue=String.format("attachment;filename=\"%s\"", download.getName());
response.setHeader(headerKey, headerValue);
outputStream=response.getOutputStream();
byte[] buffer=new byte[BUFFER_SIZES];
int bytesRead=-1;
while ((bytesRead = bufferedInputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
}catch(Exception e)
{
e.printStackTrace();
}
finally
{
try{
if(inputStream!=null)inputStream.close();
if(bufferedInputStream!=null)bufferedInputStream.close();
if(outputStream!=null)outputStream.close();
}
catch(Exception e)
{
e.printStackTrace();
}
}
}
}
public void printApp(String id,String fileName)
{
try{
String urlPath="http://localhost:8080/proj";
urlPath+="/genApp.htm?id="+id;//generate url to execute wkhtmltopdf
String wxpath="/home/exm/wkhtmltopdf";//the path where wkhtmltopdf located
String save="/tmp/"+fileName+".pdf";//File save Pathname
Process process=null;
process=Runtime.getRuntime().exec(wxpath+" "+urlPath+" "+save);
}catch(Exception e)
{}
}
#RequestMapping(value="/genApp.htm",method=RequestMethod.GET)
public String getApplicationPDF(HttpServletRequest request,HttpSession session,#RequestParam String id)
{
UDets uDets=uService.getAllById(Long.parseLong(id));//Methods to get details
request.setAttribute("uDets",uDets );
return "makeApp";//Name of the jsp page
}
In my code I have use Thread.sleep(1000) and printApp(id,fileName) method three times , since sometime wkhtmltopdf fail to generate pdf in certain time and then probability of downloading 0KB file is more. I haven't share the jsp page since the jsp page contain simple jsp page code of lots of line (the size of the generated pdf file is two page).
So the problem is what should I change in my code so that the pdf file generated and download without a failure also in heavy load in server.
If there is any best procedure or idea please share.
I don't like to use itext, since the jsp page contain complex design. Any advise is also appreciable and also thanks in advance.
I would say that your code is flawed not just a little but big time. You are checking if a file can be read, if not you start again a proces writing to the same file (at least twice). At some time you will endup with multiple processes trying to write to the same file, resulting in strange behavior.
I would refactor the printApp method to return the Process it created. Then call waitFor on that process. If it returns 0 and doesn't get interrupted it completed successfully and you should be able to download the file.
#RequestMapping(value="/dwn.htm",method=RequestMethod.GET)
public void dwAppFm(HttpSession session,HttpServletRequest request,HttpServletResponse response,#RequestParam String id) throws IOException,InterruptedException
{
String savePath="/tmp/";//PDF file Generate Path
String fileName="PDFFileName.pdf"; //Pdf file name
File download = new File(savePath, fileName);
try {
Process process = printApp(id, download.getPath());
int status = process.waitFor();
if (status == 0) {
response.setContentType("application/pdf");
response.setContentLength((int)download.length());
String headerKey="Content-Disposition";
String headerValue=String.format("attachment;filename=\"%s\"", download.getName());
StreamUtils.copy(new FileInputStream(download), response.getOutputStream())
} else {
// do something if it fails.
}
} catch (IOException ioe) {
// Do something to handle exception
} catch (InterruptedException ie) {
// Do something to handle exception
}
}
}
public Process printApp(String id, String pdf) throws IOException {
String urlPath="http://localhost:8080/proj";
urlPath+="/genApp.htm?id="+id;//generate url to execute wkhtmltopdf
String wxpath="/home/exm/wkhtmltopdf";//the path where wkhtmltopdf located
String command = wxpath+" "+urlPath+" "+pdf;
return Runtime.getRuntime().exec(command);
}
Something like the code above should to the trick.
I've developed a HTTP communication object for the purpose pof downloading files via a GET request.
This works just fine when downloading a text file. However, downloading a compressed file such as zip, gz or tar.gz appears to download the file but the file is not valid.
In the case zip I get a meesage saying it tried to move the pointer before the beginnning of the file.
In the case of .tar.gz the message is Data error in file.tar. File is broken.
In all cases the download links I use do allow a complete and correct download from the URL. Yet, the Java code based download brings the file down but it is not valid.
The code is as follows:
public class HTTPCommunicationGet {
private URIBuilder sendData;
private URI target;
private HttpGet getConnection;
public HTTPCommunicationGet(String url, TreeMap<String, String> components) {
super(url, components);
}
public HTTPCommunicationGet(String url, String queryString) {
super(url, queryString);
}
protected void defineSendData() throws URISyntaxException, IOException {
this.sendData = new URIBuilder(new URI(this.getUrl()));
if (this.getComponents() != null && this.getComponents().size() > 0) {
for (Map.Entry<String, String> component : this.getComponents().entrySet()) {
this.sendData.setParameter(component.getKey(), component.getValue());
}
}
}
protected void retrieveRemoteData() throws IOException, MalformedURLException, URISyntaxException, DataMapHTTPGetException {
this.target = this.sendData.build();
this.getConnection = new HttpGet(target);
HttpResponse response = client.execute(this.getConnection);
if (response.getStatusLine().toString().toUpperCase().contains("200 OK")) {
this.setResponse(response.getStatusLine().toString(), "Data Retrieved");
BufferedReader rd = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
String line = "";
while ((line = rd.readLine()) != null) {
this.remoteData.append(line);
}
} else {
String message = String.format("%s: Provider connection exception; response returned was not 200 OK", this.target.toASCIIString());
this.setResponse(response.getStatusLine().toString(), message);
DataMapHTTPGetException ex = new DataMapHTTPGetException(target.toString(), message);
throw ex;
}
}
public void downloadFiles(String localFile) throws DataMapConnectionException, FileNotFoundException, IOException, URISyntaxException {
// check that we have remoteData set
this.defineSendData();
this.retrieveRemoteData(); // everything is bubbled up to the controller class that is calling this.
File localMetaFile = new File(localFile);
switch (this.archiveMetaFile(localMetaFile)) {
case -1:
IOException ex = new IOException(String.format("The file %s could not be moved", localFile));
throw ex;
//break;
case 0:
infoLog.info(String.format("%s: this file did not already exist", localFile));
break;
case 1:
infoLog.info(String.format("%s: this file was found and successfully archived to the processed directory", localFile));
break;
}
BufferedWriter fileWriter = new BufferedWriter(new FileWriter(localFile));
fileWriter.write(this.remoteData.toString());
fileWriter.close();
}
}
As you can see this is called via downloadFiles after the object has been initialised. I've cut out the code that is not needed for this example such as the archiveMetaFile method.
Any pointers on why this is not working for compressed files is much appreciated.
Cheers
Nathan
The problem is likely that you are using a BufferedReader instead of an InputStream. Readers are used for text data and impose a character encoding whereas InputStreams can handle raw binary data.
Try switching to a BufferedInputStream instead. The use of any Reader class will corrupt binary data.
I've serving a files from Android assets via Netty server (images, html).
Text files such a html is saved as .mp3 to disable compression (I need an InputStream!)
My pipeline is looking like this:
pipeline.addLast("decoder", new HttpRequestDecoder());
pipeline.addLast("aggregator", new HttpChunkAggregator(65536));
pipeline.addLast("encoder", new HttpResponseEncoder());
pipeline.addLast("chunkedWriter", new ChunkedWriteHandler());
pipeline.addLast("handler", new AssetsServerHandler(context));
My handler is:
public class AssetsServerHandler extends SimpleChannelUpstreamHandler {
public void messageReceived(ChannelHandlerContext ctx, MessageEvent e) {
// some checks
final FileInputStream is;
final AssetFileDescriptor afd;
try {
afd = assetManager.openFd(path);
is = afd.createInputStream();
} catch(IOException exc) {
sendError(ctx, NOT_FOUND);
return;
}
final long fileLength = afd.getLength();
HttpResponse response = new DefaultHttpResponse(HTTP_1_1, OK);
setContentLength(response, fileLength);
final Channel ch = e.getChannel();
final ChannelFuture future;
ch.write(response);
future = ch.write(new ChunkedStream(is));
future.addListener(new ChannelFutureListener() {
#Override
public void operationComplete(ChannelFuture future) throws Exception {
future.getChannel().close();
}
});
if (!isKeepAlive(request)) {
future.addListener(ChannelFutureListener.CLOSE);
}
}
// other stuff
}
With that handler i've got my resposes truncated by at least one byte. If I change ChunkedStream to ChunkedNioFile (and so use a is.getChannel() instead of is as a constructor to it) - everything works perfectly.
Please, help me understand what is wrong with ChunkedStream.
Your code looks right to me. Does the returned FileInputStream of AssetFileDescriptor contain "all the bytes" ? You could check this with a unit test. If there is no bug in it then its a bug in netty. I make heavy use of ChunkInputStream and never had such a problem yet, but maybe it really depends on the nature of the InputStream.
Would be nice if you could write a test case and open a issue at netty's github.