We Keep Coding

Creating mp4 file doesn't remove tmp files

I'm trying to write an InputStream that is an mp4 that I get from calling an external SOAP service, when I do so, it always generates this tmp files for my chosen temporary directory(java.io.tmpdir) that aren't removable and stay after the writing is done.
Writing images that I also get from the SOAP service works normal without the permanent tmp on the directory. I'm using java 1.8 SpringBoot
tmp files
This is what I'm doing:
File targetFile = new File("D:/archive/video.mp4");
targetFile.getParentFile().mkdirs();
targetFile.setWritable(true);
InputStream inputStream = filesToWrite.getInputStream();
OutputStream outputStream = new FileOutputStream(targetFile);
try {
int byteRead;
while ((byteRead = inputStream.read()) != -1) {
outputStream.write(byteRead);
}
} catch (IOException e) {
logger.fatal("Error# SaveFilesThread for guid: " + guid, e);
}finally {
try {
inputStream.close();
outputStream.flush();
outputStream.close();
}catch (Exception e){
e.printStackTrace();
}
also tried:
byte data[] = IOUtils.toByteArray(inputStream);
Path file = Paths.get("video.mp4");
Files.write(file, data);
And from apache commons IO:
FileUtils.copyInputStreamToFile(initialStream, targetFile);

When your code starts, the damage is already done. Your code is not the source of the temporary files (It's.. a ton of work for something that could be done so much simpler, though, see below), it's the framework that ends up handing you that filesToWrite variable.
It is somewhat likely that you can hook in at an earlier point and get the raw inputstream representing the socket or HTTP connection, and start saving the files straight from there. Alternatively, Perhaps filesToWrite has a way to get at the files themselves, in which case you can just move them into place instead of copying them over.
But, your code to do this is a mess, it has bad exception handling, and leaks memory, and is way too much code for a simple job, and is possibly 2000x to 10000x slower than needed depending on your harddisk (I'm not exaggerating, calling single-byte read() on unbuffered streams is thousands of times slower!)
// add `throws IOException` to your method signature.
// it saves files, it's supposed to throw IOException,
// 'doing I/O' is in the very definition of your method!
try (InputStream in = filesToWrite.getInputStream();
OutputStream out = new FileOutputStream(targetFile)) {
in.transferTo(out);
}
That's it. That solves all the problems - no leaks, no speed loss, tiny amount of code, fixes the deplorable error handling (which, here, is 'log something to the log, then print something to standard out, then potentially leak a bunch of resources, then don't tell the calling code anything went wrong and return exactly as if the copy operation succeeded).

How to download monthly Treasury Files

Up till early this year the US Treasury web site posted monthly US Receipts and Outlays data in txt format. It was easy to write a program to read and store the info. All I use were:
URL url = new URL("https://www.fiscal.treasury.gov/fsreports/rpt/mthTreasStmt/mts1214.txt")
URLConnection connection.openConnection();
InputStream is = connection.getInputStream();
Then I just read the InputStream into a local file.
Now when I try same code, for May, I get an InputStream with nothing in it.
Just clicking on "https://www.fiscal.treasury.gov/fsreports/rpt/mthTreasStmt/mts0415.xlsx" opens an excel worksheet (the download path has since changed).
Which is great if you don't mind clicking on each link separately ... saving the file somewhere ... opening it manually to enable editing ... then saving it again as a real .xlsx file (because they really hand you an .xls file.)
But when I create a URL from that link, and use it to get an InputStream, the is empty. I also tried url.openStream() directly. No different.
Can anyone see a way I can resume using a program to read the new format?
In case its of interest I now use this code to write the stream to the file bit by bit... but there are no bits, so I don't know if it works.
static void copyInputStreamToFile( InputStream in, File file ) {
try {
OutputStream out = new FileOutputStream(file);
byte[] buf = new byte[1024];
System.out.println("reading: " + in.read(buf));
//This is what tells me it is empty, i.e. the loop below is ignored.
int len;
while((len=in.read(buf))>0){
out.write(buf,0,len);
}
out.close();
in.close();
} catch (Exception e) {
e.printStackTrace();
}
}
Any help is appreciated.

Can't delete a file when it is an argument

I'll put my code first:
#Post
public Representation post(InputStream zip) throws Throwable {
createFile(zip, "C:/temp\abc.zip");
return new StringRepresentation("File uploaded");
}
public void createFile(InputStream zipStream, uploadedFileLocation) throws Exception {
try {
writeToFile(zipStream, uploadedFileLocation);
FileUtils.forceDelete(new File(uploadedFileLocation));
} catch (Exception e) {
throw e;
}
}
private void writeToFile(InputStream uploadedInputStream, String uploadedFileLocation) {
try {
OutputStream out = new FileOutputStream(new File(uploadedFileLocation));
int read = 0;
byte[] bytes = new byte[1024];
out = new FileOutputStream(new File(uploadedFileLocation));
while ((read = uploadedInputStream.read(bytes)) != -1) {
out.write(bytes, 0, read);
}
out.flush();
out.close();
uploadedInputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
I am trying make a server that allows a user to upload a zip file. The server then write the zip file to disk, unzip it, then delete the zip while keeping the unzipped portion on the server. However, when I send the zip file to my server, it cannot get deleted. When using FileUtils.forceDelete(), it says that it cannot delete the file. It is ideal for the zip to be deleted after it is unzipped.
EDIT: I can only delete the file after post(InputStream zip) returns. If I call delete from within the post method, it won't delete because post hasn't returned. Is there a way around this?

Hmm. It appears you're trying to delete a directory?
FileUtils.forceDelete(new File(uploadedFileLocation));
But uploadedFileLocation is "C:\temp" from your post method. I'm not sure if this would cause the issue, or if this is intended behavior, but that code does not delete the zip file, and is attempting to delete the actual C:\temp directory.
Also, potentially worth noting: in your writeToFile method, you're initializing OutputStream out twice. I'm not 100% on this, but it could be that the first initialization is holding the file pointer open until the entire object is deleted from the stack (i.e., when post returns). Take out the second initialization and see if any changes occur?

I guess your problem might be caused by you using "C:/temp" instead of "C:/temp/fileName", so you end up trying to delete a folder, that you might not have permissions to.

You don't need to use file location, why not use the actual file?
Pass the file as an argument and do file.delete(); when you are done.

If the problem is not that you are trying to delete a directory, then it is most likely that the file you are trying to delete is still open. On Windows, that means that will stop you from deleting the file.
By the looks of it, your writeToFile method opens the file twice ... but only closes it once. I suspect that that means that it will leak a file descriptor, and the OS will think that the application is still using the file.
It turns out I should have called file.delete(); instead of using FileUtils.
I don't think so. Sure, you won't get an exception due to the file not being deleted. But instead File.delete() will return false and your application will leave the file in the file system. In the long term, that could be worse, unless you've got something else in place to clean out the orphaned files.

Using HTTPConnection to one site to download binary files makes some errors

To make my life easier at work, I'm making a Java program to download some modules off the server (Sometimes they get deleted off of my local machine and it takes 15 minutes to build them all).
Following is my code for downloading the files:
Note that all files are less than a megabyte big.
URL url = new URL("http://www.url.com/ModuleName.swf");
URLConnection connection = url.openConnection();
InputStream input = connection.getInputStream();
byte[] buffer = new byte[4096];
int n = -1;
OutputStream output = new FileOutputStream(new File("dlFile.swf"));
while ((n = input.read(buffer)) != -1)
{
output.write(buffer, 0, n);
output.flush();
}
output.close();
If I use a hex editor to compare the file downloaded via Java andvia Firefox, it's almost the same at first. But later on there's so many errors.
Now, the strange thing is this: If I use Firefox to download the file and upload that file to dropbox, the file will be downloaded correctly with my application.
Any idea what could possibly cause this?

Hosting an executable within Android application

I am working on an Android application that depends on an ELF binary:
our Java code interacts with this binary to get things done. This
runtime needs to be started and terminated on Application startup and
application exit / on demand.
Questions:
I am assuming that we will be able to execute this binary using the
Runtime.exec() API. Is there any constraints as to where I
need to be putting my library in the folder structure? How would the system runtime locate this executable? Is there some sort of class path setting?
Since the application has dependencies on this Runtime, I was
thinking of wrapping it around a service so that it can be started or
stopped as required. What is the best way to handle such executables
in Android project?
What are other alternatives, assuming that I do not have source code for this executable?
Please advice.
Thanks.

1) No, there should be no constrains, besides those that access system files and thus require root. The best place would be straight to /data/data/[your_package_name] to avoid polluting elsewhere.
2) A very thorough discussion about compiling against native libraries can be found here: http://www.aton.com/android-native-libraries-for-java-applications/ . Another option is a cross-compiler for arm (here is the one used to compile the kernel, it's free: http://www.codesourcery.com/sgpp/lite/arm ). If you plan to maintain a service that executes your cammand, be warned that services can be stopped and restarted by android at any moment.
3) Now, if you don't have the source code, I hope that your file is at least compiled as an arm executable. If not, I don't see how you could even run it.
You will execute the file by running the following commands in your java class:
String myExec = "/data/data/APPNAME/FILENAME";
Process process = Runtime.getRuntime().exec(myExec);
DataOutputStream os = new DataOutputStream(process.getOutputStream());
DataInputStream osRes = new DataInputStream(process.getInputStream());
I know nothing about your executable, so you may or may not need to actually get the inputStream and outputStream.
I am assuming that running adb to push the binary file is out of the question, so
I was looking for a neat way to package it. I found a great post about including an executable in your app. Check it out here:
http://gimite.net/en/index.php?Run%20native%20executable%20in%20Android%20App
The important part is this one (emphasis mine):
From Android Java app, using assets folder
Include the binary in the assets folder.
Use getAssets().open(FILENAME) to get an InputStream.
Write it to /data/data/APPNAME (e.g. /data/data/net.gimite.nativeexe), where your application has access to write files and make it executable.
Run /system/bin/chmod 744 /data/data/APPNAME/FILENAME using the code above.
Run your executable using the code above.
The post uses the assets folder, insted of the raw folder that android suggests for static files:
Tip: If you want to save a static file in your application at compile time, save the file in your project res/raw/ directory. You can open it with openRawResource(), passing the R.raw. resource ID. This method returns an InputStream that you can use to read the file (but you cannot write to the original file).
To access the data folder, you can follow the instructions here:
http://developer.android.com/guide/topics/data/data-storage.html#filesInternal
Also, there's the File#setExecutable(boolean); method that should works instead of the shell command.
So, putting everything together, I would try:
InputStream ins = context.getResources().openRawResource (R.raw.FILENAME)
byte[] buffer = new byte[ins.available()];
ins.read(buffer);
ins.close();
FileOutputStream fos = context.openFileOutput(FILENAME, Context.MODE_PRIVATE);
fos.write(buffer);
fos.close();
File file = context.getFileStreamPath (FILENAME);
file.setExecutable(true);
Of course, all this should be done only once after installation. You can have a quick check inside onCreate() or whatever that checks for the presence of the file and executes all this commands if the file is not there.
Let me know if it works. Good luck!

Here is a complete guide for how to package and run the executable. I based it on what I found here and other links, as well as my own trial and error.
1.) In your SDK project, put the executable file in your /assets folder
2.) Programmatically get the String of that files directory (/data/data/your_app_name/files) like this
String appFileDirectory = getFilesDir().getPath();
String executableFilePath = appFileDirectory + "/executable_file";
3.) In your app's project Java code: copy the executable file from /assets folder into your app's "files" subfolder (usually /data/data/your_app_name/files) with a function like this:
private void copyAssets(String filename) {
AssetManager assetManager = getAssets();
InputStream in = null;
OutputStream out = null;
Log.d(TAG, "Attempting to copy this file: " + filename); // + " to: " + assetCopyDestination);
try {
in = assetManager.open(filename);
Log.d(TAG, "outDir: " + appFileDirectory);
File outFile = new File(appFileDirectory, filename);
out = new FileOutputStream(outFile);
copyFile(in, out);
in.close();
in = null;
out.flush();
out.close();
out = null;
} catch(IOException e) {
Log.e(TAG, "Failed to copy asset file: " + filename, e);
}
Log.d(TAG, "Copy success: " + filename);
}
4.) Change the file permissions on executable_file to actually make it executable. Do it with Java calls:
File execFile = new File(executableFilePath);
execFile.setExecutable(true);
5.) Execute the file like this:
Process process = Runtime.getRuntime().exec(executableFilePath);
Note that any files referred to here (such as input and output files) must have their full path Strings constructed. This is because this is a separate spawned process and it has no concept of what the "pwd" is.
If you want to read the command's stdout you can do this, but so far it's only working for me for system commands (like "ls"), not the executable file:
BufferedReader reader = new BufferedReader(
new InputStreamReader(process.getInputStream()));
int read;
char[] buffer = new char[4096];
StringBuffer output = new StringBuffer();
while ((read = reader.read(buffer)) > 0) {
output.append(buffer, 0, read);
}
reader.close();
process.waitFor();
Log.d(TAG, "output: " + output.toString());

For executing binary file starting from Android 10 it's only possible from read-only folder. It means that you should pack binary with your app. Android doc
Put android:extractNativeLibs="true" into AndroidManifest;
Put your binary to src/main/resources/lib/* directory, where * – stands for architecture of CPU, for instance armeabi-v7a;
Use code like this for executing:
private fun exec(command: String, params: String): String {
try {
val process = ProcessBuilder()
.directory(File(filesDir.parentFile!!, "lib"))
.command(command, params)
.redirectErrorStream(true)
.start()
val reader = BufferedReader(
InputStreamReader(process.inputStream)
)
val text = reader.readText()
reader.close()
process.waitFor()
return text
} catch (e: Exception) {
return e.message ?: "IOException"
}
}
Here is discussion with answer from android team on reddit.

I've done something like this using the NDK. My strategy was to recompile the program using the NDK and write some wrapper JNI code that called into the program's main function.
I'm not sure what the lifecycle of NDK code is like. Even services that are intended to be long-running can be started and stopped by the system when convenient. You would probably have to shutdown your NDK thread and restart it when necessary.