We Keep Coding

Transferring files from AWS lambda tmp folder to sftp server

I have created a AWS lambda function that takes some files from an S3 bucket, zips them and transfers the zipped file to a sftp server. When I look in the server, I see that the tmp folder has been carries over with the files and a tmp folder gets created inside the zip file. When I open the zip file, there is a tmp folder and inside that folder are the files that I had zipped. I have scoured the internet and AWS trying to figure out how to change the directory in AWS Lambda when I am retrieving the files to be zipped, but have not had any luck. I don't want to have a tmp folder in my zip file. When I unzip the zip file, I just want to see the files that I had selected to be zipped without any folders. Does anyone know how to do this? I am programming in Java.
My code is below.
private DownloadFile(){
File localFile = new File(fileName);
//pull data and audit files from s3 bucket
s3Client.getObject(new GetObjectRequest("pie-dd-demo/daniel20", fileName), localFile);
zipOS = new ZipOutputStream(fos);
//send files to be zipped
writeToZipFile(fileName, zipOS);
}
public static void writeToZipFile(String path, ZipOutputStream zipStream)
throws FileNotFoundException, IOException {
File aFile = new File(path);
FileInputStream fis = new FileInputStream(aFile);
ZipEntry zipEntry = new ZipEntry(path);
try {
zipStream.putNextEntry(zipEntry);
byte[] bytes = new byte[1024];
int length;
while ((length = fis.read(bytes)) >= 0) {
zipStream.write(bytes, 0, length);
System.out.println(path + "write to zipfile complete");
}
} catch (FileNotFoundException exception) {
// Output expected FileNotFoundExceptions.
} catch (Exception exception) {
// Output unexpected Exceptions.
}
zipStream.closeEntry();
fis.close();
}

I think the problem is that you are creating a zip entry using new ZipEntry(path) and that means that the resulting zip file will contain the full path as the name of the zip entry.
You can retrieve the actual filename from a full path/file in Java as follows:
File f = new File("/tmp/folder/cat.png");
String fname = f.getName();
You can then use fname to create the zip entry by calling new ZipEntry(fname).

Updating a file in a zip archive

I want to update a file inside a zip archive without the need to create a new archive. I am aware that it could be done by creating a new archive, coping over all other files and add the changed one in.
Here is the code to demonstrate the issue. The first section creates the initial zip archive containing one file with the content Hello world!. The second section should replace that content with Bye, bye!. The last section is unpacking the zip again so that the content can be expected:
try {
// Create the initial zip archive with the original file content
File file1 = new File(ReplaceFileInZipDemo.class.getResource("/helloworld.txt").toURI());
File zipFile = new File("output.zip");
FileOutputStream fos = new FileOutputStream(zipFile);
ZipOutputStream zipOut = new ZipOutputStream(fos);
FileInputStream fis = new FileInputStream(file1);
ZipEntry zipEntry = new ZipEntry(file1.getName());
zipOut.putNextEntry(zipEntry);
byte[] bytes = new byte[1024];
int length;
while ((length = fis.read(bytes)) >= 0) {
zipOut.write(bytes, 0, length);
}
fis.close();
zipOut.close();
fos.close();
// Replace the original file content with a different file content, updating the file.
File file2 = new File(ReplaceFileInZipDemo.class.getResource("/goodbyeworld.txt").toURI());
Path zipfile = zipFile.toPath();
FileSystem fs = FileSystems.newFileSystem(zipfile, null);
Path pathInZipfile = fs.getPath(file1.getName());
Files.copy(file2.toPath() , pathInZipfile, StandardCopyOption.REPLACE_EXISTING );
// Extract the content of the updated file
String destinationDir = System.getProperty("java.io.tmpdir");
File targetDir = new File(destinationDir);
ZipInputStream i = new ZipInputStream(new FileInputStream(zipFile));
ZipEntry entry = null;
System.out.println("Original file content:");
Files.readAllLines(file1.toPath(), StandardCharsets.UTF_8).forEach(line -> System.out.println(line));
System.out.println("Expected replaced file content");
Files.readAllLines(file2.toPath(), StandardCharsets.UTF_8).forEach(line -> System.out.println(line));
while ((entry = i.getNextEntry()) != null) {
File destFile = new File(targetDir, entry.getName());
String name = destFile.getAbsolutePath();
File f = new File(name);
try (OutputStream o = Files.newOutputStream(f.toPath())) {
IOUtils.copy(i, o);
}
System.out.println("Content of extracted file " + name);
Files.readAllLines(f.toPath(), StandardCharsets.UTF_8).forEach(line -> System.out.println(line));
}
} catch (Exception e) {
e.printStackTrace();
}
The output that I get is this:
Original file content:
Hello world!
Expected replaced file content
Bye, bye!
Content of extracted file /tmp/helloworld.txt
Hello world!
The only reason that I could imagine this is not working as expected is because the content of the replacement comes from a file with a different name than in the archive. But when adding Files.delete(pathInZipfile); to remove the original file completely it is still there.
How can I replace the content of a file in an archive with the content of another file?

Copy files from one directory to another and append new files with timestamp instead of overwriting in Java

I want to copy files from source directory to destination. If the file already exists in the destination directory, then append the new file to be copied with its timestamp so that there is no overwrite. How do I check for duplicates and append timestamp to the new file name? Please help!
public static void copyFolder(File src, File dest)
throws IOException{
//list all the directory contents
String files[] = src.list();
for (String file : files) {
//construct the src and dest file structure
File srcFile = new File(src, file);
File destFile = new File(dest, file);
//recursive copy
copyFolder(srcFile,destFile);
}
}else{
//if file, then copy it
//Use bytes stream to support all file types
InputStream in = new FileInputStream(src);
OutputStream out = new FileOutputStream(dest);
byte[] buffer = new byte[1024];
int length;
//copy the file content in bytes
while ((length = in.read(buffer)) > 0){
out.write(buffer, 0, length);
}
in.close();
out.close();
System.out.println("File copied from " + src + " to " + dest);
}
}

//construct the src and dest file structure
File srcFile = new File(src, file);
File destFile = new File(dest, file);
while (destFile.exists()) {
destFile = new File(dest, file + '-' + Instant.now());
}
In one case the destination file got named test-file.txt-2018-03-14T11:05:21.103706Z. The time given is in UTC. In any case you will end up with a name of file that doesn’t already exist (if the loop terminates, but I have a hard time seeing the scenario where it doesn’t).
You may want to append the timestamp only to plain files and reuse existing folders (directories), I don’t know your requirements here. And you may want to append the timestamp before the extension if there is one (to get test-file-2018-03-14T11:05:21.103706Z.txt instead). I trust you to make the necessary modifications.

You can check if the file exists using File.exist() method, if exists, you can open the file in the append mode
The code is something like this
File f = new File(oldName);
if(f.exists() && !f.isDirectory()) {
long currentTime=System.currentTimeMillis();
String newName=oldName+currentTime;
// do the copy
}

Zipping Files using util.zip No directory

I have the following situation:
I am able to zip my files with the following method:
public boolean generateZip(){
byte[] application = new byte[100000];
ByteArrayOutputStream baos = new ByteArrayOutputStream();
// These are the files to include in the ZIP file
String[] filenames = new String[]{"/subdirectory/index.html", "/subdirectory/webindex.html"};
// Create a buffer for reading the files
try {
// Create the ZIP file
ZipOutputStream out = new ZipOutputStream(baos);
// Compress the files
for (int i=0; i<filenames.length; i++) {
byte[] filedata = VirtualFile.fromRelativePath(filenames[i]).content();
ByteArrayInputStream in = new ByteArrayInputStream(filedata);
// Add ZIP entry to output stream.
out.putNextEntry(new ZipEntry(filenames[i]));
// Transfer bytes from the file to the ZIP file
int len;
while ((len = in.read(application)) > 0) {
out.write(application, 0, len);
}
// Complete the entry
out.closeEntry();
in.close();
}
// Complete the ZIP file
out.close();
} catch (IOException e) {
System.out.println("There was an error generating ZIP.");
e.printStackTrace();
}
downloadzip(baos.toByteArray());
}
This works perfectly and I can download the xy.zip which contains the following directory and file structure:
subdirectory/
----index.html
----webindex.html
My aim is to completely leave out the subdirectory and the zip should only contain the two files. Is there any way to achieve this?
(I am using Java on Google App Engine).
Thanks in advance

If you are sure the files contained in the filenames array are unique if you leave out the directory, change your line for constructing ZipEntrys:
String zipEntryName = new File(filenames[i]).getName();
out.putNextEntry(new ZipEntry(zipEntryName));
This uses java.io.File#getName()

You can use Apache Commons io to list all your files, then read them to an InputStream
Replace the line below
String[] filenames = new String[]{"/subdirectory/index.html", "/subdirectory/webindex.html"}
with the following
Collection<File> files = FileUtils.listFiles(new File("/subdirectory"), new String[]{"html"}, true);
for (File file : files)
{
FileInputStream fileStream = new FileInputStream(file);
byte[] filedata = IOUtils.toByteArray(fileStream);
//From here you can proceed with your zipping.
}
Let me know if you have issues.

You could use the isDirectory() method on VirtualFile

How to use java.util.zip to archive/deflate string in java for use in Google Earth?

Use Case
I need to package up our kml which is in a String into a kmz response for a network link in Google Earth. I would like to also wrap up icons and such while I'm at it.
Problem
Using the implementation below I receive errors from both WinZip and Google Earth that the archive is corrupted or that the file cannot be opened respectively. The part that deviates from other examples I'd built this from are the lines where the string is added:
ZipEntry kmlZipEntry = new ZipEntry("doc.kml");
out.putNextEntry(kmlZipEntry);
out.write(kml.getBytes("UTF-8"));
Please point me in the right direction to correctly write the string so that it is in doc.xml in the resulting kmz file. I know how to write the string to a temporary file, but I would very much like to keep the operation in memory for understandability and efficiency.
private static final int BUFFER = 2048;
private static void kmz(OutputStream os, String kml)
{
try{
BufferedInputStream origin = null;
ZipOutputStream out = new ZipOutputStream(os);
out.setMethod(ZipOutputStream.DEFLATED);
byte data[] = new byte[BUFFER];
File f = new File("./icons"); //folder containing icons and such
String files[] = f.list();
if(files != null)
{
for (String file: files) {
LOGGER.info("Adding to KMZ: "+ file);
FileInputStream fi = new FileInputStream(file);
origin = new BufferedInputStream(fi, BUFFER);
ZipEntry entry = new ZipEntry(file);
out.putNextEntry(entry);
int count;
while((count = origin.read(data, 0, BUFFER)) != -1) {
out.write(data, 0, count);
}
origin.close();
}
}
ZipEntry kmlZipEntry = new ZipEntry("doc.kml");
out.putNextEntry(kmlZipEntry);
out.write(kml.getBytes("UTF-8"));
}
catch(Exception e)
{
LOGGER.error("Problem creating kmz file", e);
}
}
Bonus points for showing me how to put the supplementary files from the icons folder into a similar folder within the archive as opposed to at the same layer as the doc.kml.
Update Even when saving the string to a temp file the errors occur. Ugh.
Use Case Note The use case is for use in a web app, but the code to get the list of files won't work there. For details see how-to-access-local-files-on-server-in-jboss-application

You forgot to call close() on ZipOutputStream. Best place to call it is the finally block of the try block where it's been created.
Update: To create a folder, just prepend its name in the entry name.
ZipEntry entry = new ZipEntry("icons/" + file);