We Keep Coding

Transferring files from AWS lambda tmp folder to sftp server

I have created a AWS lambda function that takes some files from an S3 bucket, zips them and transfers the zipped file to a sftp server. When I look in the server, I see that the tmp folder has been carries over with the files and a tmp folder gets created inside the zip file. When I open the zip file, there is a tmp folder and inside that folder are the files that I had zipped. I have scoured the internet and AWS trying to figure out how to change the directory in AWS Lambda when I am retrieving the files to be zipped, but have not had any luck. I don't want to have a tmp folder in my zip file. When I unzip the zip file, I just want to see the files that I had selected to be zipped without any folders. Does anyone know how to do this? I am programming in Java.
My code is below.
private DownloadFile(){
File localFile = new File(fileName);
//pull data and audit files from s3 bucket
s3Client.getObject(new GetObjectRequest("pie-dd-demo/daniel20", fileName), localFile);
zipOS = new ZipOutputStream(fos);
//send files to be zipped
writeToZipFile(fileName, zipOS);
}
public static void writeToZipFile(String path, ZipOutputStream zipStream)
throws FileNotFoundException, IOException {
File aFile = new File(path);
FileInputStream fis = new FileInputStream(aFile);
ZipEntry zipEntry = new ZipEntry(path);
try {
zipStream.putNextEntry(zipEntry);
byte[] bytes = new byte[1024];
int length;
while ((length = fis.read(bytes)) >= 0) {
zipStream.write(bytes, 0, length);
System.out.println(path + "write to zipfile complete");
}
} catch (FileNotFoundException exception) {
// Output expected FileNotFoundExceptions.
} catch (Exception exception) {
// Output unexpected Exceptions.
}
zipStream.closeEntry();
fis.close();
}

I think the problem is that you are creating a zip entry using new ZipEntry(path) and that means that the resulting zip file will contain the full path as the name of the zip entry.
You can retrieve the actual filename from a full path/file in Java as follows:
File f = new File("/tmp/folder/cat.png");
String fname = f.getName();
You can then use fname to create the zip entry by calling new ZipEntry(fname).

Create directory in Java but don't throw error if it already exists [duplicate]

The condition is if the directory exists it has to create files in that specific directory without creating a new directory.
The below code only creates a file with the new directory but not for the existing directory . For example the directory name would be like "GETDIRECTION":
String PATH = "/remote/dir/server/";
String fileName = PATH.append(id).concat(getTimeStamp()).append(".txt");
String directoryName = PATH.append(this.getClassName());
File file = new File(String.valueOf(fileName));
File directory = new File(String.valueOf(directoryName));
if (!directory.exists()) {
directory.mkdir();
if (!file.exists() && !checkEnoughDiskSpace()) {
file.getParentFile().mkdir();
file.createNewFile();
}
}
FileWriter fw = new FileWriter(file.getAbsoluteFile());
BufferedWriter bw = new BufferedWriter(fw);
bw.write(value);
bw.close();

Java 8+ version:
Files.createDirectories(Paths.get("/Your/Path/Here"));
The Files.createDirectories() creates a new directory and parent directories that do not exist. This method does not throw an exception if the directory already exists.

This code checks for the existence of the directory first and creates it if not, and creates the file afterwards. Please note that I couldn't verify some of your method calls as I don't have your complete code, so I'm assuming the calls to things like getTimeStamp() and getClassName() will work. You should also do something with the possible IOException that can be thrown when using any of the java.io.* classes - either your function that writes the files should throw this exception (and it be handled elsewhere), or you should do it in the method directly. Also, I assumed that id is of type String - I don't know as your code doesn't explicitly define it. If it is something else like an int, you should probably cast it to a String before using it in the fileName as I have done here.
Also, I replaced your append calls with concat or + as I saw appropriate.
public void writeFile(String value){
String PATH = "/remote/dir/server/";
String directoryName = PATH.concat(this.getClassName());
String fileName = id + getTimeStamp() + ".txt";
File directory = new File(directoryName);
if (! directory.exists()){
directory.mkdir();
// If you require it to make the entire directory path including parents,
// use directory.mkdirs(); here instead.
}
File file = new File(directoryName + "/" + fileName);
try{
FileWriter fw = new FileWriter(file.getAbsoluteFile());
BufferedWriter bw = new BufferedWriter(fw);
bw.write(value);
bw.close();
}
catch (IOException e){
e.printStackTrace();
System.exit(-1);
}
}
You should probably not use bare path names like this if you want to run the code on Microsoft Windows - I'm not sure what it will do with the / in the filenames. For full portability, you should probably use something like File.separator to construct your paths.
Edit: According to a comment by JosefScript below, it's not necessary to test for directory existence. The directory.mkdir() call will return true if it created a directory, and false if it didn't, including the case when the directory already existed.

Trying to make this as short and simple as possible. Creates directory if it doesn't exist, and then returns the desired file:
/** Creates parent directories if necessary. Then returns file */
private static File fileWithDirectoryAssurance(String directory, String filename) {
File dir = new File(directory);
if (!dir.exists()) dir.mkdirs();
return new File(directory + "/" + filename);
}

I would suggest the following for Java8+.
/**
* Creates a File if the file does not exist, or returns a
* reference to the File if it already exists.
*/
public File createOrRetrieve(final String target) throws IOException {
final File answer;
Path path = Paths.get(target);
Path parent = path.getParent();
if(parent != null && Files.notExists(parent)) {
Files.createDirectories(path);
}
if(Files.notExists(path)) {
LOG.info("Target file \"" + target + "\" will be created.");
answer = Files.createFile(path).toFile();
} else {
LOG.info("Target file \"" + target + "\" will be retrieved.");
answer = path.toFile();
}
return answer;
}
Edit: Updated to fix bug as indicated by #Cataclysm and #Marcono1234. Thx guys:)

code:
// Create Directory if not exist then Copy a file.
public static void copyFile_Directory(String origin, String destDir, String destination) throws IOException {
Path FROM = Paths.get(origin);
Path TO = Paths.get(destination);
File directory = new File(String.valueOf(destDir));
if (!directory.exists()) {
directory.mkdir();
}
//overwrite the destination file if it exists, and copy
// the file attributes, including the rwx permissions
CopyOption[] options = new CopyOption[]{
StandardCopyOption.REPLACE_EXISTING,
StandardCopyOption.COPY_ATTRIBUTES
};
Files.copy(FROM, TO, options);
}

Simple Solution using using java.nio.Path
public static Path createFileWithDir(String directory, String filename) {
File dir = new File(directory);
if (!dir.exists()) dir.mkdirs();
return Paths.get(directory + File.separatorChar + filename);
}

If you create a web based application, the better solution is to check the directory exists or not then create the file if not exist. If exists, recreate again.
private File createFile(String path, String fileName) throws IOException {
ClassLoader classLoader = getClass().getClassLoader();
File file = new File(classLoader.getResource(".").getFile() + path + fileName);
// Lets create the directory
try {
file.getParentFile().mkdir();
} catch (Exception err){
System.out.println("ERROR (Directory Create)" + err.getMessage());
}
// Lets create the file if we have credential
try {
file.createNewFile();
} catch (Exception err){
System.out.println("ERROR (File Create)" + err.getMessage());
}
return file;
}

A simple solution using Java 8
public void init(String multipartLocation) throws IOException {
File storageDirectory = new File(multipartLocation);
if (!storageDirectory.exists()) {
if (!storageDirectory.mkdir()) {
throw new IOException("Error creating directory.");
}
}
}

If you're using Java 8 or above, then Files.createDirectories() method works the best.

How to upload file to Dropbox?

I want to upload a file to drop-box using a web-application. But the problem is that Java is asking the complete file path. What do I have to do? This is my code:
File inputFile = new File("D://New Text Document.txt");
System.out.println("inputFile.getAbsoluteFile(): " + inputFile);
FileInputStream inputStream = new FileInputStream("D://New Text Document.txt");
try {
DbxEntry.File uploadedFile = client.uploadFile("/magnum-opus.txt",
DbxWriteMode.add(), inputFile.length(), inputStream);
System.out.println("Uploaded: " + uploadedFile.toString());
} finally {
inputStream.close();
}
In the first line it is asking for the file path; how is this possible?

you can use relative path
File inputFile = new File("working-draft.txt");
FileInputStream inputStream = new FileInputStream(inputFile);
try {
DbxEntry.File uploadedFile = client.uploadFile("/magnum-opus.txt",
DbxWriteMode.add(), inputFile.length(), inputStream);
System.out.println("Uploaded: " + uploadedFile.toString());
} finally {
inputStream.close();
}
more details here

The Dropbox Java Core SDK tutorial has the following code as a sample for uploading a file:
File inputFile = new File("working-draft.txt");
FileInputStream inputStream = new FileInputStream(inputFile);
try {
DbxEntry.File uploadedFile = client.uploadFile("/magnum-opus.txt",
DbxWriteMode.add(), inputFile.length(), inputStream);
System.out.println("Uploaded: " + uploadedFile.toString());
} finally {
inputStream.close();
}
In the sample above, in the first line, "working-draft.txt" is the local path to a local file. This needs to point to an existing local file. Your code has "D://New Text Document.txt" so you should first make sure there is actually a file there.
In the fourth line, "/magnum-opus.txt", the first parameter passed to uploadFile, is the desired remote path, that is, where in the Dropbox folder you want to upload the file to. The API requires remote paths be referenced relatively like this. The uploadFile method is documented here.

java.util.zip has problems with periods in filenames / directorynames?

I want to unzip an iPhone app .ipa file.
This is actually zip file that extracts normally.
But the actual app file in it is a folder with the ending .app ( as all mac applications are actually folders with the ending .app).
Now the period seems to be a problem for java.util.zip.
public static void main(String[] args) throws IOException {
ZipFile zipFile = new ZipFile("file.zip");
String path = "";
Enumeration files = zipFile.entries();
while (files.hasMoreElements()) {
ZipEntry entry = (ZipEntry) files.nextElement();
if (entry.isDirectory()) {
File file = new File(path + entry.getName());
file.mkdir();
System.out.println("Create dir " + entry.getName());
} else {
File f = new File(entry.getName());
FileOutputStream fos = new FileOutputStream(f); //EXception occurs here
InputStream is = zipFile.getInputStream(entry);
byte[] buffer = new byte[1024];
int bytesRead = 0;
while ((bytesRead = is.read(buffer)) != -1) {
fos.write(buffer, 0, bytesRead);
}
fos.close();
System.out.println("Create File " + entry.getName());
}
}
}
This is my output:
Exception in thread "main" java.io.FileNotFoundException: Payload/SMA Jobs.app/06-magnifying-glass.png (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
at Main.main(Main.java:27)
enter code here
Anyone knows how to handle those periods?

First of all, you should use mkdirs(), not mkdir().
second, zip files don't always include all the directory entries (or have them in the right order). the best practice is to make the directories in both branches of the code, so add:
} else {
File f = new File(entry.getName());
f.getParent().mkdirs();
(you should add some checking to make sure getParent() is not null, etc).

I don't think the period is the problem. Look at the absolute path of the file you are trying to output and make sure it is pointing to the correct place.

if (entry.isDirectory()) {
File file = new File(path + entry.getName());
....
} else {
File f = new File(entry.getName());
....
While creating directory, file path passed is path + entry.getName()
but while creating file, file path passed is entry.getName()
After changing file path to path + entry.getName(), code works for period file names and normal file names. :)

Rename a file using Java

Can we rename a file say test.txt to test1.txt ?
If test1.txt exists will it rename ?
How do I rename it to the already existing test1.txt file so the new contents of test.txt are added to it for later use?

Copied from http://exampledepot.8waytrips.com/egs/java.io/RenameFile.html
// File (or directory) with old name
File file = new File("oldname");
// File (or directory) with new name
File file2 = new File("newname");
if (file2.exists())
throw new java.io.IOException("file exists");
// Rename file (or directory)
boolean success = file.renameTo(file2);
if (!success) {
// File was not successfully renamed
}
To append to the new file:
java.io.FileWriter out= new java.io.FileWriter(file2, true /*append=yes*/);

In short:
Files.move(source, source.resolveSibling("newname"));
More detail:
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;
The following is copied directly from http://docs.oracle.com/javase/7/docs/api/index.html:
Suppose we want to rename a file to "newname", keeping the file in the same directory:
Path source = Paths.get("path/here");
Files.move(source, source.resolveSibling("newname"));
Alternatively, suppose we want to move a file to new directory, keeping the same file name, and replacing any existing file of that name in the directory:
Path source = Paths.get("from/path");
Path newdir = Paths.get("to/path");
Files.move(source, newdir.resolve(source.getFileName()), StandardCopyOption.REPLACE_EXISTING);

You want to utilize the renameTo method on a File object.
First, create a File object to represent the destination. Check to see if that file exists. If it doesn't exist, create a new File object for the file to be moved. call the renameTo method on the file to be moved, and check the returned value from renameTo to see if the call was successful.
If you want to append the contents of one file to another, there are a number of writers available. Based on the extension, it sounds like it's plain text, so I would look at the FileWriter.

For Java 1.6 and lower, I believe the safest and cleanest API for this is Guava's Files.move.
Example:
File newFile = new File(oldFile.getParent(), "new-file-name.txt");
Files.move(oldFile.toPath(), newFile.toPath());
The first line makes sure that the location of the new file is the same directory, i.e. the parent directory of the old file.
EDIT:
I wrote this before I started using Java 7, which introduced a very similar approach. So if you're using Java 7+, you should see and upvote kr37's answer.

Renaming the file by moving it to a new name. (FileUtils is from Apache Commons IO lib)
String newFilePath = oldFile.getAbsolutePath().replace(oldFile.getName(), "") + newName;
File newFile = new File(newFilePath);
try {
FileUtils.moveFile(oldFile, newFile);
} catch (IOException e) {
e.printStackTrace();
}

This is an easy way to rename a file:
File oldfile =new File("test.txt");
File newfile =new File("test1.txt");
if(oldfile.renameTo(newfile)){
System.out.println("File renamed");
}else{
System.out.println("Sorry! the file can't be renamed");
}

import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import static java.nio.file.StandardCopyOption.*;
Path yourFile = Paths.get("path_to_your_file\text.txt");
Files.move(yourFile, yourFile.resolveSibling("text1.txt"));
To replace an existing file with the name "text1.txt":
Files.move(yourFile, yourFile.resolveSibling("text1.txt"),REPLACE_EXISTING);

Try This
File file=new File("Your File");
boolean renameResult = file.renameTo(new File("New Name"));
// todo: check renameResult
Note :
We should always check the renameTo return value to make sure rename file is successful because it’s platform dependent(different Operating system, different file system) and it doesn’t throw IO exception if rename fails.

Yes, you can use File.renameTo(). But remember to have the correct path while renaming it to a new file.
import java.util.Arrays;
import java.util.List;
public class FileRenameUtility {
public static void main(String[] a) {
System.out.println("FileRenameUtility");
FileRenameUtility renameUtility = new FileRenameUtility();
renameUtility.fileRename("c:/Temp");
}
private void fileRename(String folder){
File file = new File(folder);
System.out.println("Reading this "+file.toString());
if(file.isDirectory()){
File[] files = file.listFiles();
List<File> filelist = Arrays.asList(files);
filelist.forEach(f->{
if(!f.isDirectory() && f.getName().startsWith("Old")){
System.out.println(f.getAbsolutePath());
String newName = f.getAbsolutePath().replace("Old","New");
boolean isRenamed = f.renameTo(new File(newName));
if(isRenamed)
System.out.println(String.format("Renamed this file %s to %s",f.getName(),newName));
else
System.out.println(String.format("%s file is not renamed to %s",f.getName(),newName));
}
});
}
}
}

If it's just renaming the file, you can use File.renameTo().
In the case where you want to append the contents of the second file to the first, take a look at FileOutputStream with the append constructor option or The same thing for FileWriter. You'll need to read the contents of the file to append and write them out using the output stream/writer.

As far as I know, renaming a file will not append its contents to that of an existing file with the target name.
About renaming a file in Java, see the documentation for the renameTo() method in class File.

Files.move(file.toPath(), fileNew.toPath());
works, but only when you close (or autoclose) ALL used resources (InputStream, FileOutputStream etc.) I think the same situation with file.renameTo or FileUtils.moveFile.

Here is my code to rename multiple files in a folder successfully:
public static void renameAllFilesInFolder(String folderPath, String newName, String extension) {
if(newName == null || newName.equals("")) {
System.out.println("New name cannot be null or empty");
return;
}
if(extension == null || extension.equals("")) {
System.out.println("Extension cannot be null or empty");
return;
}
File dir = new File(folderPath);
int i = 1;
if (dir.isDirectory()) { // make sure it's a directory
for (final File f : dir.listFiles()) {
try {
File newfile = new File(folderPath + "\\" + newName + "_" + i + "." + extension);
if(f.renameTo(newfile)){
System.out.println("Rename succesful: " + newName + "_" + i + "." + extension);
} else {
System.out.println("Rename failed");
}
i++;
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
and run it for an example:
renameAllFilesInFolder("E:\\Downloads\\Foldername", "my_avatar", "gif");

I do not like java.io.File.renameTo(...) because sometimes it does not renames the file and you do not know why! It just returns true of false. It does not thrown an exception if it fails.
On the other hand, java.nio.file.Files.move(...) is more useful as it throws an exception when it fails.

Running code is here.
private static void renameFile(File fileName) {
FileOutputStream fileOutputStream =null;
BufferedReader br = null;
FileReader fr = null;
String newFileName = "yourNewFileName"
try {
fileOutputStream = new FileOutputStream(newFileName);
fr = new FileReader(fileName);
br = new BufferedReader(fr);
String sCurrentLine;
while ((sCurrentLine = br.readLine()) != null) {
fileOutputStream.write(("\n"+sCurrentLine).getBytes());
}
fileOutputStream.flush();
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
fileOutputStream.close();
if (br != null)
br.close();
if (fr != null)
fr.close();
} catch (IOException ex) {
ex.printStackTrace();
}
}
}