We Keep Coding

Downloaded files with whitespaces in path are damaged

I have following problem. I have to download pdf files from a server and some of them have whitespaces in their names. So every file will be downloaded, but those, which have whitespaces can not be opened.
If I access this files on the server via chrome, they open well (also with the whitespace in the url).
And what I am wondering about is, that java says the files will be downloaded. But when I try to open them in Acrobat Reader, it shows me an error message, that the files are damaged. Here is the sample of my code:
public static void downloadFile(String fileURL, String saveDir) throws IOException {
Authenticator.setDefault(new Authenticator() {
#Override
protected PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication("*****", "*********".toCharArray());
}
});
final int BUFFER_SIZE = 4096;
URL url = new URL(fileURL);
HttpURLConnection httpConn = (HttpURLConnection) url.openConnection();
String credentials = "ptt" + ":" + "ptt123";
String encoding = Base64.getEncoder().encodeToString(credentials.getBytes(StandardCharsets.UTF_8));
httpConn.setRequestProperty("Authorization", String.format("Basic %s", encoding));
int responseCode = 0;
responseCode = httpConn.getResponseCode();
// always check HTTP response code first
if (responseCode == HttpURLConnection.HTTP_OK) {
String fileName = "";
String disposition = httpConn.getHeaderField("Content-Disposition");
String contentType = httpConn.getContentType();
int contentLength = httpConn.getContentLength();
if (disposition != null) {
// extracts file name from header field
int index = disposition.indexOf("filename=");
if (index > 0) {
fileName = disposition.substring(index + 10,
disposition.length() - 1);
}
} else {
// extracts file name from URL
fileName = fileURL.substring(fileURL.lastIndexOf("/") + 1,
fileURL.length());
}
// opens input stream from the HTTP connection
InputStream inputStream = httpConn.getInputStream();
String saveFilePath = saveDir + File.separator + fileName;
// opens an output stream to save into file
FileOutputStream outputStream = new FileOutputStream(saveFilePath);
int bytesRead = -1;
byte[] buffer = new byte[BUFFER_SIZE];
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
outputStream.close();
inputStream.close();
System.out.println("File downloaded");
} else {
System.out.println("No file to download. Server replied HTTP code: " + responseCode);
}
httpConn.disconnect();
}
I also tried to replace the whitespace through "%20" in the fileUrl.
So what can be the problem? As I wrote above, the files without any whitespace can be opened after the the download without any problems.
I use Java 1.7.
Cheers,
Andrej

if fileName contains space then replace it to some other charecter. it may work, if not please let me know.
if(fileName.trim().contains(" "))
fileName.replace(" ","_");

URL url = new URL(URLEncoder.encode(fileUrl, "UTF-8"));

Download the entire webpage

There are ways to download an entire webpage, using HTMLEditorKit. However, I need to download an entire webpage which needs scrolling in order to load its entire content. This technology is achieved most commonly through JavaScript bundled with Ajax.
Q.: Is there a way to trick the destined webpage, using only Java code, in order to download its full content?
Q.2: If this is not possible only with Java, then is it possible in combination with JavaScript?
Simple notice, what I wrote:
public class PageDownload {
public static void main(String[] args) throws Exception {
String webUrl = "...";
URL url = new URL(webUrl);
URLConnection connection = url.openConnection();
InputStream is = connection.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
HTMLEditorKit htmlKit = new HTMLEditorKit();
HTMLDocument htmlDoc = (HTMLDocument) htmlKit.createDefaultDocument();
HTMLEditorKit.Parser parser = new ParserDelegator();
HTMLEditorKit.ParserCallback callback = htmlDoc.getReader(0);
parser.parse(br, callback, true);
for (HTMLDocument.Iterator iterator = htmlDoc.getIterator(HTML.Tag.IMG);
iterator.isValid(); iterator.next()) {
AttributeSet attributes = iterator.getAttributes();
String imgSrc = (String) attributes.getAttribute(HTML.Attribute.SRC);
if (imgSrc != null && (imgSrc.endsWith(".jpg") || (imgSrc.endsWith(".jpeg"))
|| (imgSrc.endsWith(".png")) || (imgSrc.endsWith(".ico"))
|| (imgSrc.endsWith(".bmp")))) {
try {
downloadImage(webUrl, imgSrc);
} catch (IOException ex) {
System.out.println(ex.getMessage());
}
}
}
}
private static void downloadImage(String url, String imgSrc) throws IOException {
BufferedImage image = null;
try {
if (!(imgSrc.startsWith("http"))) {
url = url + imgSrc;
} else {
url = imgSrc;
}
imgSrc = imgSrc.substring(imgSrc.lastIndexOf("/") + 1);
String imageFormat = null;
imageFormat = imgSrc.substring(imgSrc.lastIndexOf(".") + 1);
String imgPath = null;
imgPath = "..." + imgSrc + "";
URL imageUrl = new URL(url);
image = ImageIO.read(imageUrl);
if (image != null) {
File file = new File(imgPath);
ImageIO.write(image, imageFormat, file);
}
} catch (Exception ex) {
ex.printStackTrace();
}
}
}

Use HtmlUnit library to get all text and images/css files.
HTMLUnit [link] htmlunit.sourceforge.net
1) To download text content use code on below link s
all Text content [link] How to get a HTML page using HtmlUnit
Specific tag such as span [link] how to get text between a specific span with HtmlUnit
2) To get images/files use below [link] How can I tell HtmlUnit's WebClient to download images and css?

Yes you can trick a a webpage to download on your locals by Java code. You can not Download HTMl Static content by Java Script. JavaScript is not providing you to create a files as Java Provides.
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.URL;
public class HttpDownloadUtility {
private static final int BUFFER_SIZE = 4096;
/**
* Downloads a file from a URL
* #param fileURL HTTP URL of the file to be downloaded
* #param saveDir path of the directory to save the file
* #throws IOException
*/
public static void downloadFile(String fileURL, String saveDir)
throws IOException {
URL url = new URL(fileURL);
HttpURLConnection httpConn = (HttpURLConnection) url.openConnection();
int responseCode = httpConn.getResponseCode();
// always check HTTP response code first
if (responseCode == HttpURLConnection.HTTP_OK) {
String fileName = "";
String disposition = httpConn.getHeaderField("Content-Disposition");
String contentType = httpConn.getContentType();
int contentLength = httpConn.getContentLength();
if (disposition != null) {
// extracts file name from header field
int index = disposition.indexOf("filename=");
if (index > 0) {
fileName = disposition.substring(index + 10,
disposition.length() - 1);
}
} else {
// extracts file name from URL
fileName = fileURL.substring(fileURL.lastIndexOf("/") + 1,
fileURL.length());
}
System.out.println("Content-Type = " + contentType);
System.out.println("Content-Disposition = " + disposition);
System.out.println("Content-Length = " + contentLength);
System.out.println("fileName = " + fileName);
// opens input stream from the HTTP connection
InputStream inputStream = httpConn.getInputStream();
String saveFilePath = saveDir + File.separator + fileName;
// opens an output stream to save into file
FileOutputStream outputStream = new FileOutputStream(saveFilePath);
int bytesRead = -1;
byte[] buffer = new byte[BUFFER_SIZE];
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
outputStream.close();
inputStream.close();
System.out.println("File downloaded");
} else {
System.out.println("No file to download. Server replied HTTP code: " + responseCode);
}
httpConn.disconnect();
}
}

You can achieve this with Selenium Webdriver java classes...
https://code.google.com/p/selenium/wiki/GettingStarted
Generally, webdriver is used for testing, but it is able to emulate a user scrolling down the page, until the page stops changing, and then you can use java code to save the content to a file.

You can do it using IDM's grabber.
This should help:
https://www.internetdownloadmanager.com/support/idm-grabber/grabber_wizard.html

No Response on Google App Engine Java Blobstore Upload from Android

I'm attempting to upload an image to the blobstore from Android but when my server servlet is called upon completion, the Blobstore fails to hand off any BlobKeys. And when I check the actual blobstore, no image has been uploaded. There is no error code, no warnings, no errors, nothing. Now I know the server stuff works because the upload works in iOS. And the Android code works just fine on the local development server but it's not working in production. I'm hoping that maybe someone can spot an error in my code.
Thanks in advance.
HttpURLConnection connection = null;
try {
Log.d(LOG_TAG, "Starting multipart POST request to: " +fullUrlValue);
URL fullUrl = new URL(fullUrlValue);
//make server call
connection = getClient().getConnection(fullUrl, RequestMethod.POST);
connection.setConnectTimeout(60000); //60 seconds to complete an upload
connection.setUseCaches(false);
connection.setDoInput(true);
connection.setDoOutput(true);
//see if we are streaming the upload
if(requestOptions == null || requestOptions.isChunckedStreamingMode())
{
connection.setChunkedStreamingMode(2048);
}
String boundry = "z6fQbdm2TTgLwPQj9u1HjAM25z9AJuGSx7WG9dnD";
connection.setRequestProperty("Connection", "Keep-Alive");
connection.setRequestProperty("Cache-Control", "no-cache");
connection.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundry);
//attach post objects
DataOutputStream outputStream = new DataOutputStream(connection.getOutputStream());
//add form fields
if(additionalParameters != null && additionalParameters.keySet().size() > 0)
{
Iterator<String> it = additionalParameters.keySet().iterator();
while(it.hasNext())
{
String fieldName = it.next();
if(fieldName != null)
{
String value = additionalParameters.get(fieldName);
if(value != null)
{
//add form field to upload form
outputStream.writeBytes("--" + boundry + "\r\n" + "Content-Disposition: form-data; name=\"" + fieldName + "\"\r\nContent-Type: text/plain; charset=UTF-8\r\n\r\n"+value+"\r\n");
}
}
}
}
//attach file
if(postObjectBytes != null)
{
//build the request body
outputStream.writeBytes("--" + boundry + "\r\n" + "Content-Disposition: form-data; name=\"" + formElementName + "\";filename=\"" + fileName + "\"\r\n\r\n");
//object upload content size
long totalUploadSize = postObjectBytes.length;
//we have to manually process the stream in order to correctly update the progress listener
byte[] buffer = new byte[2048];
long totalBytes = 0;
int bytesRead = 0;
InputStream inputStream = new ByteArrayInputStream(postObjectBytes);
while ((bytesRead = inputStream.read(buffer)) > -1) {
totalBytes += bytesRead;
outputStream.write(buffer, 0, bytesRead);
if(progressListener != null){
progressListener.transferred(totalBytes, totalUploadSize);
}
}
outputStream.writeBytes("\r\n--" + boundry + "--\r\n");
}
outputStream.flush();
outputStream.close();

Well, I found the answer thanks to AppEngine BlobStore upload failing with a request that works in the Development Environment .
Apparently I was missing a space between the semi-colon and the start of "filename". That's just insane.
formElementName + "\";filename

Android, Download a .txt file and save internally

like the title states i am simply trying to download a test.txt file, the following url and save it internally, ideally within drawable.
i have been trying to modify this to work but will little success i keep getting "unable to download null" errors
int count;
try {
URL url = new URL("https://www.darkliteempire.gaming.multiplay.co.uk/testdownload.txt");
URLConnection conexion = url.openConnection();
conexion.connect();
int lenghtOfFile = conexion.getContentLength();
InputStream is = url.openStream();
File testDirectory = new File(Environment.getExternalStorageDirectory() + "/Download");
if (!testDirectory.exists()) {
testDirectory.mkdir();
}
FileOutputStream fos = new FileOutputStream(testDirectory + "/test.txt");
byte data[] = new byte[1024];
long total = 0;
int progress = 0;
while ((count = is.read(data)) != -1) {
total += count;
int progress_temp = (int) total * 100 / lenghtOfFile;
fos.write(data, 0, count);
}
is.close();
fos.close();
} catch (Exception e) {
Log.e("ERROR DOWNLOADING", "Unable to download" + e.getMessage());
}
There must be a simpler way to do this?
the file itself is tiny with perhaps 3 or 4 lines of text so i dont need anything fancy

Please Update your below code line and write valid url.
URL url = new URL("https://www.http://darkliteempire.gaming.multiplay.co.uk/testdownload.txt");
after write valid url your code line look like this.
URL url = new URL("http://www.darkliteempire.gaming.multiplay.co.uk/testdownload.txt");
it will solve your problem.

Using AQuery library you get something pretty straightforward. Plus you'll get hips of other cool functions to shorten your code.
http://code.google.com/p/android-query/wiki/AsyncAPI
String url = "https://picasaweb.google.com/data/feed/base/featured?max-results=16";
File ext = Environment.getExternalStorageDirectory();
File target = new File(ext, "aquery/myfolder/photos.xml");
aq.progress(R.id.progress).download(url, target, new AjaxCallback<File>(){
public void callback(String url, File file, AjaxStatus status) {
if(file != null){
showResult("File:" + file.length() + ":" + file, status);
}else{
showResult("Failed", status);
}
}
});

How do I download part of a file using Java?

I'm using this Java code to download a file from the Internet:
String address = "http://melody.syr.edu/pzhang/publications/AMCIS99_vonDran_Zhang.pdf";
URL url = new URL(address);
System.out.println("Opening connection to " + address + "...");
URLConnection urlC = url.openConnection();
urlC.setRequestProperty("User-Agent", "");
urlC.connect();
InputStream is = urlC.getInputStream();
FileOutputStream fos = null;
fos = new FileOutputStream("myFileName");
int oneChar, count = 0;
while ((oneChar = is.read()) != -1) {
System.out.print((char)oneChar);
fos.write(oneChar);
count++;
}
is.close();
fos.close();
System.out.println(count + " byte(s) copied");
I'd like to know if there is a way for me to download only a part of a file.
For example, for a 5MB file to download the last 2MB.

If the server supports it (and HTTP 1.1 servers should), you can use range requests:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35
Also, reading one character at a time is hugely inefficient - you should be reading in blocks, say 4, 16 or 32 KB.

Please have a look at Java: resume Download in URLConnection