I'm developing code for a Java Servlet to handle form image POST , and upload that image to Google Cloud Storage. I can upload images doing an ajax call POST, with XMLHttpRequest, and FormData. What i'm facing is that most times images stored are damaged, and can not be open.
Like this one(http://storage.googleapis.com/realtimevouchers/Porto_ribeira.jpg), but image is ok on my filesystem.
Could this be an error dealing with streams inside servlet code?
Here's the servlet code:
protected void doPost(HttpServletRequest request,
HttpServletResponse response) throws ServletException, IOException {
response.setContentType("application/json");
response.setCharacterEncoding("UTF-8");
String ajaxUpdateResult = "";
ServletFileUpload upload = new ServletFileUpload();
String filePath = "";
try {
FileItemIterator iter = upload.getItemIterator(request);
while (iter.hasNext()) {
FileItemStream item = iter.next();
String fileName = item.getName();
System.out.println("Nome ficheiro->" + fileName);
String mimeType = item.getContentType();
System.out.println("mimeType->" + mimeType);
InputStream content = item.openStream();
// Process the input stream
ByteArrayOutputStream baos = new ByteArrayOutputStream();
int len;
byte[] buffer = new byte[8192];
while ((len = content.read(buffer, 0, buffer.length)) != -1) {
baos.write(buffer, 0, len);
}
int maxFileSize = 10 * (1024 * 1024); // 10 megs max
if (baos.size() > maxFileSize) {
throw new RuntimeException("File is > than " + maxFileSize);
}
FileService fileService = FileServiceFactory.getFileService();
java.util.Date date = new java.util.Date();
java.sql.Timestamp timeStamp = new Timestamp(date.getTime());
System.out.println("DEBUG:upload timeStamp->" + timeStamp);
GSFileOptionsBuilder optionsBuilder = new GSFileOptionsBuilder()
.setBucket(BUCKETNAME).setKey(fileName)
.setMimeType(mimeType).setAcl("public_read")
.addUserMetadata("uploadTimeStamp", timeStamp.toString());
AppEngineFile writableFile = fileService
.createNewGSFile(optionsBuilder.build());
// Open a channel to write to it
boolean lock = true;
FileWriteChannel writeChannel = fileService.openWriteChannel(
writableFile, lock);
// Different standard Java ways of writing to the channel
// are possible.
// This time we write to the channel directly
writeChannel.write(ByteBuffer.wrap(buffer));
// Now finalize
writeChannel.closeFinally();
filePath = writableFile.getFullPath();
// Do whatever with the content InputStream.
// System.out.println(Streams.asString(content));//binary
ajaxUpdateResult += "File " + fileName
+ " is successfully uploaded\n\r";
}
} catch (FileUploadException e) {
throw new ServletException("Parsing file upload failed.", e);
}
org.json.JSONObject json = new JSONObject();
try {
json.put("operationStatus", "SUCCESS");
json.put("operationExecuted", "POST_TO_GCS");
json.put("filePath", filePath);
} catch (JSONException e) {
e.printStackTrace();
}
System.out.println("DEBUG:" + json.toString());
log.info("LOGGER" + json.toString());
response.getWriter().println(json.toString());
response.getWriter().print(ajaxUpdateResult);
}
Related
I'm developing a Java library for basic operations on SharePoint using Graph API.
I make a call on this entry point using SOAP UI:
https://graph.microsoft.com/v1.0/drives/{drive-id}/items/{item-id}/content
And I obtain a raw response:
%PDF-1.6
%âãÏÓ
1751 0 obj
<</Filter/FlateDecode/First 98/Length 322/N 11/Type/ObjStm>>stream
hޜԽJ1†á[ÉL’ó“–m,md±ÁElTü)¼{3“wXYDØ©¾3!ç<)&I^kˆ!ymÁ¤gë¥ÍE ...
endstream
endobj
startxref
2993893
%%EOF
It look like i'm retrieving an input stream.
In the HttpRequest class I try to build a response object that returns the InputStream. My property fileInputStream is an InputStream:
SharePointDownloadResponseModel returnValue = new SharePointDownloadResponseModel();
InputStream inputStream = new ByteArrayInputStream(response.toString().getBytes(StandardCharsets.UTF_8));
returnValue.setFileInputStream(inputStream);
return returnValue;
Now in my manager class I try to save the input stream in the hard drive. I handle 2 cases. First case, I have a fileName a folder to store the file. My request object :
if(request.getDownloadFolder() != null && request.getFileName() !=null) {
InputStream initialStream = returnValue.getFileInputStream();
FileOutputStream fos = new FileOutputStream(request.getDownloadFolder() + "/" + request.getFileName());
BufferedOutputStream bos = new BufferedOutputStream(fos );
// Read bytes from URL to the local file
byte[] buffer = new byte[4096];
int bytesRead = 0;
System.out.println("Downloading " + request.getFileName());
while ((bytesRead = initialStream.read(buffer)) != -1) {
bos.write(buffer, 0, bytesRead);
}
bos.flush();
// Close destination stream
bos.close();
// Close URL stream
initialStream.close();
}
The document is created where it should be created but the file is damaged and can't be opened. I wonder what is the issue at this stage.
I finally solved my issue. Here is a basic method that shows my implementation :
public class DownloadFile {
public static void main(String[] args) throws IOException {
String url = "https://graph.microsoft.com/v1.0/drives/{driveId}/items/{itemId}/content";
SharePointCredentialRequest sharePointCredentialRequest = new SharePointCredentialRequest(Constants.TENANT_CLIENT_ID,
Constants.TENANT_CLIENT_SECRET, Constants.TENANT_AUTHORITY);
String token = Utils.getToken(sharePointCredentialRequest);
CloseableHttpClient client = HttpClients.createDefault();
HttpGet httpGet = new HttpGet(url);
httpGet.setHeader("Authorization", "Bearer " + token);
try (CloseableHttpResponse response = client.execute(httpGet)) {
HttpEntity entity = response.getEntity();
if (entity != null) {
System.out.println(response.getAllHeaders().length);
System.out.println(entity.getContentEncoding());
System.out.println(entity.getContentLength());
System.out.println(entity.getContentType().getElements().toString());
try {
// do something useful with the stream
InputStream inputStream = IOUtils.toBufferedInputStream(response.getEntity().getContent());
File targetFile = new File("C:\\myFolder\\kant.pdf");
FileUtils.copyInputStreamToFile(inputStream, targetFile);
} catch (IOException | UnsupportedOperationException e) {
e.printStackTrace();
}
}
}
}
}
I need to generate a bunch of xml files with data from my java based web system, that represent a whole export of another system based in XML. Such system will accept this import later.
My approach is to create all files in memory, then save each one as entries it to a zip also in memory, which later is served to the client.
The data flow is working perfectly, but somehow the output is a blank file. I think I got wrong the outputstream structure
This is the part that I might be getting wrong:
...
//ZIP creation in server memory
ByteArrayOutputStream datosEnMemoria = new ByteArrayOutputStream();
ZipOutputStream zipped_out = new ZipOutputStream(datosEnMemoria)
...
//close and zip entry
xmlData.append(Tangonet.terminarCargaRegistros());
byte[] xmlBinData = xmlData.toString().getBytes();
zipped_out.write(xmlBinData, 0, xmlBinData.length);
zipped_out.closeEntry();
}
byte[] zipped_out_size = zipped_out.toString().getBytes();
response.setContentType("application/octet-stream");
response.setHeader("Content-Disposition", "attachment;filename=cierresZ_a_tangonet" + java.time.LocalDate.now() + ".zip");
response.setHeader("Content-length", "" + zipped_out_size.length);
response.setHeader("Content-Type", "application/zip");
response.setHeader("Content-Type", "application/octet-stream");
response.setHeader("Content-Transfer-Encoding", " binary");
//closing zip and send it to client
zipped_out.flush();
zipped_out.close();
// out.flush();
// out.close();
This is the full code:
#RequestMapping(value = "/cierreZ/exportar", method = RequestMethod.GET)
public void cierreZExportar(#ModelAttribute InformesFinancierosForm informesFinancierosForm, HttpServletRequest request, HttpServletResponse response) throws IOException {
HttpSession session = request.getSession(true);
String fechaInicio = null;
String fechaFin = null;
if (session.getAttribute("mesActual") != null) {
informesFinancierosForm.setFechaInicio("01-" + informesFinancierosForm.getMes());
informesFinancierosForm.setFechaFin(new SimpleDateFormat("dd-MM-yyyy").format(DateUtil.getUltimoDiaDelMes(DateUtil.traduceDateDate((String) session.getAttribute("fechaIni")))));
fechaInicio = informesFinancierosForm.getFechaInicio();
fechaFin = informesFinancierosForm.getFechaFin();
} else {
fechaInicio = (String) session.getAttribute("fechaIni");
fechaFin = (String) session.getAttribute("fechaFin");
}
if (informeService.isRangoFechaValido(informesFinancierosForm.getSalasSeleccionadas(), fechaInicio)) {
if (!(fechaInicio.compareTo("") == 0) || (fechaFin.compareTo("") == 0)
|| informesFinancierosForm.getSalasSeleccionadas().length == 0) {
// ServletOutputStream out = response.getOutputStream();
List<InformeCierreZItemForm> listadoInfCierreZ = cierreZService.getCierres(informesFinancierosForm.getSalasSeleccionadas(), fechaInicio, fechaFin);
//ZIP creation in server memory
ByteArrayOutputStream datosEnMemoria = new ByteArrayOutputStream();
ZipOutputStream zipped_out = new ZipOutputStream(datosEnMemoria);
//filling zip with static xml files
for (int i = 0; i < Tangonet.documentos_estaticos_tangonet.length; i++) {
ZipEntry xmlFile = new ZipEntry(Tangonet.documentos_estaticos_tangonet[i][0] + ".xml");
zipped_out.putNextEntry(xmlFile);
StringBuilder xmlData = new StringBuilder();
xmlData.append(Tangonet.documentos_estaticos_tangonet[i][1]);
byte[] xmlBinData = xmlData.toString().getBytes();
zipped_out.write(xmlBinData, 0, xmlBinData.length);
zipped_out.closeEntry();
}
//filling zip with dynamic xml files
for (int i = 0; i < Tangonet.documentos_dinamicos_tangonet.length; i++) {
//dynamic xml creation
ZipEntry xmlFile = new ZipEntry(Tangonet.documentos_dinamicos_tangonet[i][0] + ".xml");
zipped_out.putNextEntry(xmlFile);
//xml schema
StringBuilder xmlData = new StringBuilder();
xmlData.append(Tangonet.documentos_dinamicos_tangonet[i][1]);
//xml data rows
for (InformeCierreZItemForm informeCierreZActual : listadoInfCierreZ) {
Sala salaActual = informeCierreZActual.getSala();
CierrezList CierresZ = cierreZService.getCierresZ(salaActual, fechaInicio, fechaFin);
//fiscal data in rows
Tangonet datosFiscalesCierrezActual = tangonetDatos.getDatosFiscales(salaActual);
for (Cierrez cierreActual : CierresZ) {
if (Tangonet.documentos_dinamicos_tangonet[i][0].equals("Comp_de_Facturación_para_Cobranza_Centralizada___GVA12")) {
xmlData.append(datosFiscalesCierrezActual.crearRegistroGVA12(cierreActual));
} else {
xmlData.append(datosFiscalesCierrezActual.crearRegistroGVA42(cierreActual));
}
}
}
//close and zip entry
xmlData.append(Tangonet.terminarCargaRegistros());
byte[] xmlBinData = xmlData.toString().getBytes();
zipped_out.write(xmlBinData, 0, xmlBinData.length);
zipped_out.closeEntry();
}
byte[] zipped_out_size = zipped_out.toString().getBytes();
response.setContentType("application/octet-stream");
response.setHeader("Content-Disposition", "attachment;filename=cierresZ_a_tangonet" + java.time.LocalDate.now() + ".zip");
response.setHeader("Content-length", "" + zipped_out_size.length);
response.setHeader("Content-Type", "application/zip");
response.setHeader("Content-Type", "application/octet-stream");
response.setHeader("Content-Transfer-Encoding", " binary");
//closing zip and send it to client
zipped_out.flush();
zipped_out.close();
// out.flush();
// out.close();
}
}
}
Zip file can be big, so don't generate it in memory. Write it straight to client.
Also:
Don't set Content-Type three times. It can only have one value.
Don't specify Content-Transfer-Encoding. It's an email header, not an HTTP header.
Since you'll be streaming, don't specify Content-length.
// headers must be set before streaming
response.setContentType("application/zip");
response.setHeader("Content-Disposition", "attachment;filename=cierresZ_a_tangonet" + java.time.LocalDate.now() + ".zip");
// stream straight to client
ZipOutputStream zipped_out = new ZipOutputStream(response.getOutputStream());
// Add zip entries and data here:
// Loop:
// zipped_out.putNextEntry(...)
// Generate XML, writing it straight to zipped_out
// Remember to flush any streams/writers wrapped around zipped_out
// Do not close zipped_out or wrappers of it
// If that cannot be prevented, use a CloseShieldOutputStream (from Commons IO)
// No need to call zipped_out.closeEntry()
// make sure to finish the zip stream
zipped_out.finish();
private String ftpUrl = "ftp://%s:%s#%s/%s;";
private static String host = "www.krishnas.com.np";
private static String user = "tryagn#krishnas.com.np";
private static String pass = "technology_krixnas";
private static String filePath = "112.jpg";
private static String savePath =
Environment.getExternalStorageDirectory() + "/" +
Environment.DIRECTORY_DCIM +"/abe.jpg";
My doInBackground() in asynctask is:
protected String doInBackground(String... f_url) {
int count;
try {
ftpUrl = String.format(ftpUrl, user, pass, host, filePath);
//System.out.println("URL: " + ftpUrl)
Log.d("-----------", ""+ftpUrl);
// try {
//URL url = new URL(ftpUrl);
//URLConnection conn = url.openConnection();
// InputStream inputStream = conn.getInputStream();
URL url = new URL(f_url[0]);
URLConnection conection = url.openConnection();
conection.connect();
// this will be useful so that you can show a tipical 0-100% progress bar
int lenghtOfFile = conection.getContentLength();
// download the file
InputStream input = new BufferedInputStream(url.openStream(), 8192);
// Output stream
OutputStream output = new FileOutputStream(savePath);
byte data[] = new byte[1024];
long total = 0;
while ((count = input.read(data)) != -1) {
total += count;
// publishing the progress....
// After this onProgressUpdate will be called
i=(int)((total*100)/lenghtOfFile);
publishProgress(""+(int)((total*100)/lenghtOfFile));
// i = (int)((total*100)/lenghtOfFile);
// writing data to file
output.write(data, 0, count);
}
// flushing output
output.flush();
// c
// losing streams
output.close();
input.close();
} catch (Exception e) {
Log.e("Error: ", e.getMessage());
}
return null;
}
I want to download a file/image from my FTP server, it shows image in link but does not download in internal storage. I am learning ftp server downloader not the http request!! Don't know where it goes wrong??
You are using external storage to store file Environment.getExternalStorageDirectory()
So if you want to store it in internal storage,
You can use getFilesDir() or getCacheDir():
File file = new File(context.getFilesDir(), filename);
For more Information refer this link:
I have written the following code to download files from the file server. While downloading, I do not get any runtime error but it always says file corrupt.
I am retrieving the file details from the database:
Reading the contents in byte form to the output stream.
This code works fine for all txt files however for all other types, the size of file increases and when i try to open, it says file is corrupted
public void download(Long requestId, HttpServletResponse response) throws Exception {
byte[] out;
ServletOutputStream outputStream;
String fileServerPath = parametersService.getParameters(AppConstants.FILE_SERVER_PATH);
FileVO fileVO = dao.getFileDetails(requestId);
String filePath = fileServerPath + fileVO.getFilePath() + fileVO.getFileName();
try {
if (FileUtil.isFileExist(filePath)) {
File apFile = new File(filePath);
InputStream in = new BufferedInputStream(new FileInputStream(apFile));
String mimeType= URLConnection.guessContentTypeFromName(apFile.getName());
if(mimeType==null){
//if mimetype is not detectable, will take default
mimeType = "application/octet-stream";
}
/* Start: Set the MIME type, file name and meta detail in response */
response.setContentType(mimeType);
response.addHeader("Content-Disposition", String.format("attachment; filename=\"%s\"", apFile.getName()));
response.addHeader("Content-Length", String.valueOf(apFile.length()));
/* End: Set the MIME type, file name and meta detail in response */
/* START: Read byte by byte and write to servlet output stream */
outputStream = response.getOutputStream();
int read;
final byte[] data = new byte[1 * 1024];
int i=0;
while ((read = in.read(data)) != -1) {
outputStream.write(data, 0, read);
i++;
slf4jLogger.debug("Writing data "+ i );
}
outputStream.flush();
in.close();
outputStream.close();
}
} catch (IOException ex) {
//Exception
}
}
My front end is in Angular 1.5. I am using file-saver lib to achieve.
Following is the code
function download() {
var url = apiHelper.getRootUrl() + '/request/' + '/download/' + vm.request.requestId;
$http.post(url, {responseType: "arraybuffer"}).success(
function (response, status, headers) {
var type = headers()['content-type'];
var disposition = headers()['content-disposition'];
if (disposition) {
var match = disposition.match(/.*filename=\"?([^;\"]+)\"?.*/);
if (match[1])
var defaultFileName = match[1];
}
defaultFileName = defaultFileName.replace(/[<>:"\/\\|?*]+/g, '_');
var blob = new Blob([response], {type: type});
saveAs(blob, defaultFileName);
});
}
There are ways to download an entire webpage, using HTMLEditorKit. However, I need to download an entire webpage which needs scrolling in order to load its entire content. This technology is achieved most commonly through JavaScript bundled with Ajax.
Q.: Is there a way to trick the destined webpage, using only Java code, in order to download its full content?
Q.2: If this is not possible only with Java, then is it possible in combination with JavaScript?
Simple notice, what I wrote:
public class PageDownload {
public static void main(String[] args) throws Exception {
String webUrl = "...";
URL url = new URL(webUrl);
URLConnection connection = url.openConnection();
InputStream is = connection.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
HTMLEditorKit htmlKit = new HTMLEditorKit();
HTMLDocument htmlDoc = (HTMLDocument) htmlKit.createDefaultDocument();
HTMLEditorKit.Parser parser = new ParserDelegator();
HTMLEditorKit.ParserCallback callback = htmlDoc.getReader(0);
parser.parse(br, callback, true);
for (HTMLDocument.Iterator iterator = htmlDoc.getIterator(HTML.Tag.IMG);
iterator.isValid(); iterator.next()) {
AttributeSet attributes = iterator.getAttributes();
String imgSrc = (String) attributes.getAttribute(HTML.Attribute.SRC);
if (imgSrc != null && (imgSrc.endsWith(".jpg") || (imgSrc.endsWith(".jpeg"))
|| (imgSrc.endsWith(".png")) || (imgSrc.endsWith(".ico"))
|| (imgSrc.endsWith(".bmp")))) {
try {
downloadImage(webUrl, imgSrc);
} catch (IOException ex) {
System.out.println(ex.getMessage());
}
}
}
}
private static void downloadImage(String url, String imgSrc) throws IOException {
BufferedImage image = null;
try {
if (!(imgSrc.startsWith("http"))) {
url = url + imgSrc;
} else {
url = imgSrc;
}
imgSrc = imgSrc.substring(imgSrc.lastIndexOf("/") + 1);
String imageFormat = null;
imageFormat = imgSrc.substring(imgSrc.lastIndexOf(".") + 1);
String imgPath = null;
imgPath = "..." + imgSrc + "";
URL imageUrl = new URL(url);
image = ImageIO.read(imageUrl);
if (image != null) {
File file = new File(imgPath);
ImageIO.write(image, imageFormat, file);
}
} catch (Exception ex) {
ex.printStackTrace();
}
}
}
Use HtmlUnit library to get all text and images/css files.
HTMLUnit [link] htmlunit.sourceforge.net
1) To download text content use code on below link s
all Text content [link] How to get a HTML page using HtmlUnit
Specific tag such as span [link] how to get text between a specific span with HtmlUnit
2) To get images/files use below [link] How can I tell HtmlUnit's WebClient to download images and css?
Yes you can trick a a webpage to download on your locals by Java code. You can not Download HTMl Static content by Java Script. JavaScript is not providing you to create a files as Java Provides.
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.URL;
public class HttpDownloadUtility {
private static final int BUFFER_SIZE = 4096;
/**
* Downloads a file from a URL
* #param fileURL HTTP URL of the file to be downloaded
* #param saveDir path of the directory to save the file
* #throws IOException
*/
public static void downloadFile(String fileURL, String saveDir)
throws IOException {
URL url = new URL(fileURL);
HttpURLConnection httpConn = (HttpURLConnection) url.openConnection();
int responseCode = httpConn.getResponseCode();
// always check HTTP response code first
if (responseCode == HttpURLConnection.HTTP_OK) {
String fileName = "";
String disposition = httpConn.getHeaderField("Content-Disposition");
String contentType = httpConn.getContentType();
int contentLength = httpConn.getContentLength();
if (disposition != null) {
// extracts file name from header field
int index = disposition.indexOf("filename=");
if (index > 0) {
fileName = disposition.substring(index + 10,
disposition.length() - 1);
}
} else {
// extracts file name from URL
fileName = fileURL.substring(fileURL.lastIndexOf("/") + 1,
fileURL.length());
}
System.out.println("Content-Type = " + contentType);
System.out.println("Content-Disposition = " + disposition);
System.out.println("Content-Length = " + contentLength);
System.out.println("fileName = " + fileName);
// opens input stream from the HTTP connection
InputStream inputStream = httpConn.getInputStream();
String saveFilePath = saveDir + File.separator + fileName;
// opens an output stream to save into file
FileOutputStream outputStream = new FileOutputStream(saveFilePath);
int bytesRead = -1;
byte[] buffer = new byte[BUFFER_SIZE];
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
outputStream.close();
inputStream.close();
System.out.println("File downloaded");
} else {
System.out.println("No file to download. Server replied HTTP code: " + responseCode);
}
httpConn.disconnect();
}
}
You can achieve this with Selenium Webdriver java classes...
https://code.google.com/p/selenium/wiki/GettingStarted
Generally, webdriver is used for testing, but it is able to emulate a user scrolling down the page, until the page stops changing, and then you can use java code to save the content to a file.
You can do it using IDM's grabber.
This should help:
https://www.internetdownloadmanager.com/support/idm-grabber/grabber_wizard.html