We Keep Coding

Remote file transfer using java nio

I am trying to use JAVA NIO to transfer a file from host A to client B without having to download the file locally and then giving the client B a link to download the file.
I am running a spark Apache framework and using maven project.
I mapped the request http://localhost:8080/download/hello in Spark using :
get("/download/:id",RequestHandler::downloadHandler);
Inside of the function is the code that downloads the file from :
"https://download.springsource.com/release/STS/3.8.1.RELEASE/dist/e4.6/spring-tool-suite-3.8.1.RELEASE-e4.6-linux-gtk-x86_64.tar.gz"
try {
URL url = new URL("https://download.springsource.com/release/STS/3.8.1.RELEASE/dist/e4.6/spring-tool-suite-3.8.1.RELEASE-e4.6-linux-gtk-x86_64.tar.gz");
HttpURLConnection httpURLConnection = (HttpURLConnection) url.openConnection();
int respCode = +httpURLConnection.getResponseCode();
System.out.println("response code : "+respCode);
if (respCode == HttpURLConnection.HTTP_OK){
String fileName = "";
String disposition = httpURLConnection.getHeaderField("Content-Disposition");
String contentType = httpURLConnection.getContentType();
int contentLength = httpURLConnection.getContentLength();
if (disposition != null) {
// extracts file name from header field
int index = disposition.indexOf("filename=");
if (index > 0) {
fileName = disposition.substring(index + 10,
disposition.length() - 1);
}
} else {
// extracts file name from URL
fileName = url.toString().substring(url.toString().lastIndexOf("/") + 1,
url.toString().length());
}
System.out.println("Content-Type = " + contentType);
System.out.println("Content-Disposition = " + disposition);
System.out.println("Content-Length = " + contentLength);
System.out.println("fileName = " + fileName);
httpURLConnection.disconnect();
System.out.println("other stuff : ");
System.out.println(url.getHost());
ReadableByteChannel readableByteChannel = Channels.newChannel(url.openStream());
FileOutputStream fileOutputStream = new FileOutputStream(fileName);
FileChannel fileChannel = fileOutputStream.getChannel();
fileChannel.transferFrom(readableByteChannel, 0, Long.MAX_VALUE);
fileOutputStream.close();
readableByteChannel.close();
}
} catch (IOException e) {
e.printStackTrace();
}
I fetch the filename and file size using httpURLConnection and then processed to download the file. what I am trying to do is, instead of downloading the file locally using fileChannel.transferFrom(readableByteChannel, 0, Long.MAX_VALUE) transfer the file directly to the client.
I did some research and I think it is possible with using Socketchannels but I didn't understand how it is supposed to work.
I also read this article
https://examples.javacodegeeks.com/core-java/nio/java-nio-large-file-transfer-tutorial/
and tried to understand the class Reciever, but it is still not clear to me how.
I would appreciate some guidance. Thank you

Download the entire webpage

There are ways to download an entire webpage, using HTMLEditorKit. However, I need to download an entire webpage which needs scrolling in order to load its entire content. This technology is achieved most commonly through JavaScript bundled with Ajax.
Q.: Is there a way to trick the destined webpage, using only Java code, in order to download its full content?
Q.2: If this is not possible only with Java, then is it possible in combination with JavaScript?
Simple notice, what I wrote:
public class PageDownload {
public static void main(String[] args) throws Exception {
String webUrl = "...";
URL url = new URL(webUrl);
URLConnection connection = url.openConnection();
InputStream is = connection.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
HTMLEditorKit htmlKit = new HTMLEditorKit();
HTMLDocument htmlDoc = (HTMLDocument) htmlKit.createDefaultDocument();
HTMLEditorKit.Parser parser = new ParserDelegator();
HTMLEditorKit.ParserCallback callback = htmlDoc.getReader(0);
parser.parse(br, callback, true);
for (HTMLDocument.Iterator iterator = htmlDoc.getIterator(HTML.Tag.IMG);
iterator.isValid(); iterator.next()) {
AttributeSet attributes = iterator.getAttributes();
String imgSrc = (String) attributes.getAttribute(HTML.Attribute.SRC);
if (imgSrc != null && (imgSrc.endsWith(".jpg") || (imgSrc.endsWith(".jpeg"))
|| (imgSrc.endsWith(".png")) || (imgSrc.endsWith(".ico"))
|| (imgSrc.endsWith(".bmp")))) {
try {
downloadImage(webUrl, imgSrc);
} catch (IOException ex) {
System.out.println(ex.getMessage());
}
}
}
}
private static void downloadImage(String url, String imgSrc) throws IOException {
BufferedImage image = null;
try {
if (!(imgSrc.startsWith("http"))) {
url = url + imgSrc;
} else {
url = imgSrc;
}
imgSrc = imgSrc.substring(imgSrc.lastIndexOf("/") + 1);
String imageFormat = null;
imageFormat = imgSrc.substring(imgSrc.lastIndexOf(".") + 1);
String imgPath = null;
imgPath = "..." + imgSrc + "";
URL imageUrl = new URL(url);
image = ImageIO.read(imageUrl);
if (image != null) {
File file = new File(imgPath);
ImageIO.write(image, imageFormat, file);
}
} catch (Exception ex) {
ex.printStackTrace();
}
}
}

Use HtmlUnit library to get all text and images/css files.
HTMLUnit [link] htmlunit.sourceforge.net
1) To download text content use code on below link s
all Text content [link] How to get a HTML page using HtmlUnit
Specific tag such as span [link] how to get text between a specific span with HtmlUnit
2) To get images/files use below [link] How can I tell HtmlUnit's WebClient to download images and css?

Yes you can trick a a webpage to download on your locals by Java code. You can not Download HTMl Static content by Java Script. JavaScript is not providing you to create a files as Java Provides.
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.URL;
public class HttpDownloadUtility {
private static final int BUFFER_SIZE = 4096;
/**
* Downloads a file from a URL
* #param fileURL HTTP URL of the file to be downloaded
* #param saveDir path of the directory to save the file
* #throws IOException
*/
public static void downloadFile(String fileURL, String saveDir)
throws IOException {
URL url = new URL(fileURL);
HttpURLConnection httpConn = (HttpURLConnection) url.openConnection();
int responseCode = httpConn.getResponseCode();
// always check HTTP response code first
if (responseCode == HttpURLConnection.HTTP_OK) {
String fileName = "";
String disposition = httpConn.getHeaderField("Content-Disposition");
String contentType = httpConn.getContentType();
int contentLength = httpConn.getContentLength();
if (disposition != null) {
// extracts file name from header field
int index = disposition.indexOf("filename=");
if (index > 0) {
fileName = disposition.substring(index + 10,
disposition.length() - 1);
}
} else {
// extracts file name from URL
fileName = fileURL.substring(fileURL.lastIndexOf("/") + 1,
fileURL.length());
}
System.out.println("Content-Type = " + contentType);
System.out.println("Content-Disposition = " + disposition);
System.out.println("Content-Length = " + contentLength);
System.out.println("fileName = " + fileName);
// opens input stream from the HTTP connection
InputStream inputStream = httpConn.getInputStream();
String saveFilePath = saveDir + File.separator + fileName;
// opens an output stream to save into file
FileOutputStream outputStream = new FileOutputStream(saveFilePath);
int bytesRead = -1;
byte[] buffer = new byte[BUFFER_SIZE];
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
outputStream.close();
inputStream.close();
System.out.println("File downloaded");
} else {
System.out.println("No file to download. Server replied HTTP code: " + responseCode);
}
httpConn.disconnect();
}
}

You can achieve this with Selenium Webdriver java classes...
https://code.google.com/p/selenium/wiki/GettingStarted
Generally, webdriver is used for testing, but it is able to emulate a user scrolling down the page, until the page stops changing, and then you can use java code to save the content to a file.

You can do it using IDM's grabber.
This should help:
https://www.internetdownloadmanager.com/support/idm-grabber/grabber_wizard.html

Connection gets closed while uploading file(more than 2 MB) in blacberry rim app

Im trying to upload file from blackberry app to Amazon s3.
Here is my code
private synchronized void uploadFileToAmazon(CreateFileIdBean createFileIdBean) throws UnsupportedEncodingException, IOException,ConnectionNotFoundException, ConnectionClosedException, BackupCancelledException, InterruptedException, BackupInterruptedException {
String BOUNDARY = "----------V2ymHFg03ehbqgZCaKO6jy";
String Policy = "{\"expiration\": \"2020-12-01T12:00:00.000Z\","
+ "\"conditions\": ["
+ "{\"bucket\": \"" + BeanFactory.getUserCreateBackupidBean().getBucketName() + "\"},"
+ "{\"x-amz-security-token\": \"" + BeanFactory.getUserCreateBackupidBean().getAmazonToken() + "\"},"
+ "{\"success_action_status\": \"201\"},"
+ "[\"starts-with\", \"$Content-Type\", \"\"],"
+ "[\"starts-with\", \"$key\", \"" + BeanFactory.getUserCreateBackupidBean().getBackupPath() + "\"]"
+ "]"
+ "}";
String encodePolicy = Base64.encode(Policy.getBytes());
String signature = uploadSignature(Policy, BeanFactory.getUserCreateBackupidBean().getAmazonSecret());
Hashtable params = new Hashtable();
params.put("key", BeanFactory.getUserCreateBackupidBean().getBackupPath() + "/" + BeanFactory.getUserCreateFileIdBean().getFileId());
params.put("AWSAccessKeyId", BeanFactory.getUserCreateBackupidBean().getAmazonKey());
params.put("Content-Type", createFileIdBean.getFileTypeContent());
params.put("x-amz-security-token", BeanFactory.getUserCreateBackupidBean().getAmazonToken());
params.put("policy", encodePolicy);
params.put("success_action_status", "201");
params.put("Signature", signature);
send(BOUNDARY, "http://" + BeanFactory.getUserCreateBackupidBean().getBucketName() + ".s3.amazonaws.com/", params, "file", BeanFactory.getUserCreateFileIdBean().getFileId(), createFileIdBean.getFileTypeContent(), createFileIdBean.getFileByte(), createFileIdBean);
}
private synchronized String getBoundaryMessage(String boundary, Hashtable params, String fileField, String fileName, String fileType, byte[] fileBytes, CreateFileIdBean createFileIdBean) {
StringBuffer res = new StringBuffer("--").append(boundary).append("\r\n");
Enumeration keys = params.keys();
while (keys.hasMoreElements()) {
String key = (String) keys.nextElement();
String value = (String) params.get(key);
res.append("Content-Disposition: form-data; name=\"").append(key).append("\"\r\n")
.append("\r\n").append(value).append("\r\n")
.append("--").append(boundary).append("\r\n");
}
return res.toString();
}
private synchronized void send(String boundarry, String url, Hashtable params, String fileField, String fileName, String fileType, byte[] fileBytes, CreateFileIdBean createFileIdBean) throws IOException,ConnectionClosedException,ConnectionNotFoundException, BackupCancelledException, InterruptedException, BackupInterruptedException {
StringBuffer buffer = new StringBuffer();
HttpConnection hc = null;
InputStream is = null;
InputStream inputFileDataStream = null;
DataOutputStream dout = null;
String boundary = boundarry;
StringBuffer res = new StringBuffer();
int ch;
String boundaryMessage = getBoundaryMessage(boundary, params, fileField, fileName, fileType, fileBytes, createFileIdBean);
ByteArrayOutputStream bos = new ByteArrayOutputStream();
bos.write(boundaryMessage.getBytes());
res.append("Content-Disposition: form-data; name=\"").append(fileField).append("\"; filename=\"").append(fileName).append("\"\r\n")
.append("Content-Type: ").append(fileType).append("\r\n\r\n");
bos.write(res.toString().getBytes());
String end = "\r\n"+"--"+boundary+"\r\n"+"Content-Disposition: form-data; name=\""+"submit"+"\"\r\n"+"\r\n"+"Upload to Amazon S3"+"\r\n"+"--"+boundary+"--\r\n";
try {
hc = (HttpConnection) Connector.open(url+Resources.getConnectionString(), Connector.READ_WRITE,true);
hc.setRequestMethod(HttpConnection.POST);
hc.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundarry);
hc.setRequestProperty("User-Agent", Resources.getUserAgentString());
//hc.setRequestProperty("User-Agent", "Profile/MIDP-1.0 Confirguration/CLDC-1.0");
hc.setRequestProperty("Content-Language", "en-US");
hc.setRequestProperty("Connection", "Keep-Alive");
hc.setRequestProperty("Keep-Alive", "300");
hc.setRequestProperty("Expect", "100-continue");
hc.setRequestProperty("Content-Length", (bos.toByteArray().length+createFileIdBean.getFileSize()+end.getBytes().length)+"");
hc.setRequestProperty("Content-length", (bos.toByteArray().length+createFileIdBean.getFileSize()+end.getBytes().length)+"");
dout = new DataOutputStream(hc.openDataOutputStream());
dout.write(bos.toByteArray());
inputFileDataStream = readInputStream(createFileIdBean.getFilePath());
while ((ch = inputFileDataStream.read()) != -1) {
dout.write(ch);
}
dout.write(end.getBytes());
dout.flush();
//dout.close();
is = hc.openDataInputStream();
BeanFactory.getUserUploadFileBean().setResponseCode(hc.getResponseCode() + "");
BeanFactory.getUserUploadFileBean().setResponseMessage(hc.getResponseMessage());
while ((ch = is.read()) != -1) {
buffer.append((char) ch);
}
System.out.println("buffer"+buffer);
}
catch (IOException e) {
throw new BackupInterruptedException(Constants.ERROR_IN_UPLOAD);
} finally {
try {
if (is != null) {
is.close();
}
if (hc != null) {
hc.close();
}
if(inputFileDataStream !=null)
{
inputFileDataStream.close();
}
if(dout !=null)
{
dout.close();
}
} catch (IOException e2) {
System.out.println("aa"+e2.getMessage());
throw e2;
}
}
}
private synchronized String uploadSignature(String policy, String secretKey) throws UnsupportedEncodingException {
String encodePolciy = Base64.encode(policy.getBytes());
HMac m = new HMac(new SHA1Digest());
m.init(new KeyParameter(secretKey.getBytes("UTF-8")));
byte[] bytes = encodePolciy.getBytes("UTF-8");
m.update(bytes, 0, bytes.length);
byte[] mac = new byte[m.getMacSize()];
m.doFinal(mac, 0);
String signature = Base64.encode(mac);
return signature;
}
private synchronized InputStream readInputStream(String path) throws IOException {
FileConnection fc = null;
InputStream is = null;
fc = (FileConnection) Connector.open(path.toString(), Connector.READ);
if (!fc.exists()) {
Settings.ERROR_MESSAGE = "File doesn't exist!";
//throw new BackupInterruptedException(Settings.ERROR_MESSAGE);
} else {
is = fc.openInputStream();
}
if(fc !=null)
{
fc.close();
}
return is;
}
Its always getting connection closed exception when i trying to get response code(in the line BeanFactory.getUserUploadFileBean().setResponseCode(hc.getResponseCode() + "");) after uploading data. for files less than 2MB its working fine. please help me...

My experience with HttpConnection is that it always buffers the full payload before sending anything. For the BlackBerry Curve 8520, 16mb of data ends up making the device unusable. I worked around this by using a raw SocketConnection, and writing the HTTP parts directly in my app, so I could make sure there wasn't any excessive buffering before the bytes hit the socket.
One area I later realized might work, is using HTTP-Chunked mode with the built-in HttpConnection object. Since HTTP-Chunked is fundamentally a streaming mechanism, it may allow you to get out of the 'buffer it all' logic you get by default.

If you try to transfer large files, you might get the dreaded HTTP error 413. The max allowed file size was between 1 and 30 MB depending on the transport type (BIS,BES, etc) if my memory is correct.
In addition to this, there was a RAM quota available to an app. For older 83XX models, we empirically found that the max RAM for an app was about 12MB.
But we were successful transferring large files using chunked connections (using the HTTP Range header), and for the buffers we either used small arrays (<< 12 MB) or temporary files.
I'll post the classic links on the topic:
How to Download large files using the BlackBerry Mobile Data System
What is: HTTP 413 Request entity too large
You might find something about them in the BB forum or in other sites.

Android, Download a .txt file and save internally

like the title states i am simply trying to download a test.txt file, the following url and save it internally, ideally within drawable.
i have been trying to modify this to work but will little success i keep getting "unable to download null" errors
int count;
try {
URL url = new URL("https://www.darkliteempire.gaming.multiplay.co.uk/testdownload.txt");
URLConnection conexion = url.openConnection();
conexion.connect();
int lenghtOfFile = conexion.getContentLength();
InputStream is = url.openStream();
File testDirectory = new File(Environment.getExternalStorageDirectory() + "/Download");
if (!testDirectory.exists()) {
testDirectory.mkdir();
}
FileOutputStream fos = new FileOutputStream(testDirectory + "/test.txt");
byte data[] = new byte[1024];
long total = 0;
int progress = 0;
while ((count = is.read(data)) != -1) {
total += count;
int progress_temp = (int) total * 100 / lenghtOfFile;
fos.write(data, 0, count);
}
is.close();
fos.close();
} catch (Exception e) {
Log.e("ERROR DOWNLOADING", "Unable to download" + e.getMessage());
}
There must be a simpler way to do this?
the file itself is tiny with perhaps 3 or 4 lines of text so i dont need anything fancy

Please Update your below code line and write valid url.
URL url = new URL("https://www.http://darkliteempire.gaming.multiplay.co.uk/testdownload.txt");
after write valid url your code line look like this.
URL url = new URL("http://www.darkliteempire.gaming.multiplay.co.uk/testdownload.txt");
it will solve your problem.

Using AQuery library you get something pretty straightforward. Plus you'll get hips of other cool functions to shorten your code.
http://code.google.com/p/android-query/wiki/AsyncAPI
String url = "https://picasaweb.google.com/data/feed/base/featured?max-results=16";
File ext = Environment.getExternalStorageDirectory();
File target = new File(ext, "aquery/myfolder/photos.xml");
aq.progress(R.id.progress).download(url, target, new AjaxCallback<File>(){
public void callback(String url, File file, AjaxStatus status) {
if(file != null){
showResult("File:" + file.length() + ":" + file, status);
}else{
showResult("Failed", status);
}
}
});

How to send local .png to .php file on server using java application?

I have a local .png file that I want to send using POST data to a .php script that will save the data to a .png file on the server. How do I do this? Do I have to encode or something? All I have is a File and a way to POST data.
Here is how I am sending the .png:
public static byte[] imageToByte(File file) throws FileNotFoundException {
FileInputStream fis = new FileInputStream(file);
ByteArrayOutputStream bos = new ByteArrayOutputStream();
byte[] buf = new byte[1024];
try {
for (int readNum; (readNum = fis.read(buf)) != -1;) {
bos.write(buf, 0, readNum);
}
} catch (IOException ex) {
}
byte[] bytes = bos.toByteArray();
return bytes;
}
public static void sendPostData(String url, HashMap<String, String> data)
throws Exception {
URL siteUrl = new URL(url);
HttpURLConnection conn = (HttpURLConnection) siteUrl.openConnection();
conn.setRequestMethod("POST");
conn.setDoOutput(true);
conn.setDoInput(true);
DataOutputStream out = new DataOutputStream(conn.getOutputStream());
Set keys = data.keySet();
Iterator keyIter = keys.iterator();
String content = "";
for (int i = 0; keyIter.hasNext(); i++) {
Object key = keyIter.next();
if (i != 0) {
content += "&";
}
content += key + "=" + URLEncoder.encode(data.get(key), "UTF-8");
}
System.out.println(content);
out.writeBytes(content);
out.flush();
out.close();
BufferedReader in = new BufferedReader(new InputStreamReader(
conn.getInputStream()));
String line = "";
while ((line = in.readLine()) != null) {
System.out.println(line);
}
in.close();
}
The PHP script:
<?
// Config
$uploadBase = "../screenshots/";
$uploadFilename = $_GET['user'] . ".png";
$uploadPath = $uploadBase . $uploadFilename;
// Upload directory
if(!is_dir($uploadBase))
mkdir($uploadBase);
// Grab the data
$incomingData = $_POST['img'];
// Valid data?
if(!$incomingData || !isset($_POST['img']))
die("No input data");
// Write to disk
$fh = fopen($uploadPath, 'w') or die("Error opening file");
fwrite($fh, $incomingData) or die("Error writing to file");
fclose($fh) or die("Error closing file");
echo "Success";
?>

I must admit, I am surprised that you almost get the correct file. Actually, when you send a file using a browser, the form tag has an encoding defined: enctype="multipart/form-data". I don´t know how it works (It is defined in https://www.rfc-editor.org/rfc/rfc2388), but it includes encoding the file (for example, in Base64). Anyhow, you can forget about the internals if you use a http client library like the one from Apache HttpComponents

My minimalistic code works:
$body = file_get_contents('php://input');
$fh = fopen('file.txt', 'w') or die("Error opening fil
e");
fwrite($fh, $body) or die("Error writing to file");
fclose($fh)
curl --upload-file download.txt http://example.com/upload.php
However, set the method to PUT.