I have created a AWS lambda function that takes some files from an S3 bucket, zips them and transfers the zipped file to a sftp server. When I look in the server, I see that the tmp folder has been carries over with the files and a tmp folder gets created inside the zip file. When I open the zip file, there is a tmp folder and inside that folder are the files that I had zipped. I have scoured the internet and AWS trying to figure out how to change the directory in AWS Lambda when I am retrieving the files to be zipped, but have not had any luck. I don't want to have a tmp folder in my zip file. When I unzip the zip file, I just want to see the files that I had selected to be zipped without any folders. Does anyone know how to do this? I am programming in Java.
My code is below.
private DownloadFile(){
File localFile = new File(fileName);
//pull data and audit files from s3 bucket
s3Client.getObject(new GetObjectRequest("pie-dd-demo/daniel20", fileName), localFile);
zipOS = new ZipOutputStream(fos);
//send files to be zipped
writeToZipFile(fileName, zipOS);
}
public static void writeToZipFile(String path, ZipOutputStream zipStream)
throws FileNotFoundException, IOException {
File aFile = new File(path);
FileInputStream fis = new FileInputStream(aFile);
ZipEntry zipEntry = new ZipEntry(path);
try {
zipStream.putNextEntry(zipEntry);
byte[] bytes = new byte[1024];
int length;
while ((length = fis.read(bytes)) >= 0) {
zipStream.write(bytes, 0, length);
System.out.println(path + "write to zipfile complete");
}
} catch (FileNotFoundException exception) {
// Output expected FileNotFoundExceptions.
} catch (Exception exception) {
// Output unexpected Exceptions.
}
zipStream.closeEntry();
fis.close();
}
I think the problem is that you are creating a zip entry using new ZipEntry(path) and that means that the resulting zip file will contain the full path as the name of the zip entry.
You can retrieve the actual filename from a full path/file in Java as follows:
File f = new File("/tmp/folder/cat.png");
String fname = f.getName();
You can then use fname to create the zip entry by calling new ZipEntry(fname).
Related
I want to copy files from source directory to destination. If the file already exists in the destination directory, then append the new file to be copied with its timestamp so that there is no overwrite. How do I check for duplicates and append timestamp to the new file name? Please help!
public static void copyFolder(File src, File dest)
throws IOException{
//list all the directory contents
String files[] = src.list();
for (String file : files) {
//construct the src and dest file structure
File srcFile = new File(src, file);
File destFile = new File(dest, file);
//recursive copy
copyFolder(srcFile,destFile);
}
}else{
//if file, then copy it
//Use bytes stream to support all file types
InputStream in = new FileInputStream(src);
OutputStream out = new FileOutputStream(dest);
byte[] buffer = new byte[1024];
int length;
//copy the file content in bytes
while ((length = in.read(buffer)) > 0){
out.write(buffer, 0, length);
}
in.close();
out.close();
System.out.println("File copied from " + src + " to " + dest);
}
}
//construct the src and dest file structure
File srcFile = new File(src, file);
File destFile = new File(dest, file);
while (destFile.exists()) {
destFile = new File(dest, file + '-' + Instant.now());
}
In one case the destination file got named test-file.txt-2018-03-14T11:05:21.103706Z. The time given is in UTC. In any case you will end up with a name of file that doesn’t already exist (if the loop terminates, but I have a hard time seeing the scenario where it doesn’t).
You may want to append the timestamp only to plain files and reuse existing folders (directories), I don’t know your requirements here. And you may want to append the timestamp before the extension if there is one (to get test-file-2018-03-14T11:05:21.103706Z.txt instead). I trust you to make the necessary modifications.
You can check if the file exists using File.exist() method, if exists, you can open the file in the append mode
The code is something like this
File f = new File(oldName);
if(f.exists() && !f.isDirectory()) {
long currentTime=System.currentTimeMillis();
String newName=oldName+currentTime;
// do the copy
}
I have the following situation:
I am able to zip my files with the following method:
public boolean generateZip(){
byte[] application = new byte[100000];
ByteArrayOutputStream baos = new ByteArrayOutputStream();
// These are the files to include in the ZIP file
String[] filenames = new String[]{"/subdirectory/index.html", "/subdirectory/webindex.html"};
// Create a buffer for reading the files
try {
// Create the ZIP file
ZipOutputStream out = new ZipOutputStream(baos);
// Compress the files
for (int i=0; i<filenames.length; i++) {
byte[] filedata = VirtualFile.fromRelativePath(filenames[i]).content();
ByteArrayInputStream in = new ByteArrayInputStream(filedata);
// Add ZIP entry to output stream.
out.putNextEntry(new ZipEntry(filenames[i]));
// Transfer bytes from the file to the ZIP file
int len;
while ((len = in.read(application)) > 0) {
out.write(application, 0, len);
}
// Complete the entry
out.closeEntry();
in.close();
}
// Complete the ZIP file
out.close();
} catch (IOException e) {
System.out.println("There was an error generating ZIP.");
e.printStackTrace();
}
downloadzip(baos.toByteArray());
}
This works perfectly and I can download the xy.zip which contains the following directory and file structure:
subdirectory/
----index.html
----webindex.html
My aim is to completely leave out the subdirectory and the zip should only contain the two files. Is there any way to achieve this?
(I am using Java on Google App Engine).
Thanks in advance
If you are sure the files contained in the filenames array are unique if you leave out the directory, change your line for constructing ZipEntrys:
String zipEntryName = new File(filenames[i]).getName();
out.putNextEntry(new ZipEntry(zipEntryName));
This uses java.io.File#getName()
You can use Apache Commons io to list all your files, then read them to an InputStream
Replace the line below
String[] filenames = new String[]{"/subdirectory/index.html", "/subdirectory/webindex.html"}
with the following
Collection<File> files = FileUtils.listFiles(new File("/subdirectory"), new String[]{"html"}, true);
for (File file : files)
{
FileInputStream fileStream = new FileInputStream(file);
byte[] filedata = IOUtils.toByteArray(fileStream);
//From here you can proceed with your zipping.
}
Let me know if you have issues.
You could use the isDirectory() method on VirtualFile
I need to unzip a set of files that are a zip archive. This isn't a set of zip files, this is one big zip file that has been broken up into multiple zip files based on a size requirement.
For example if you have a 2.5MB zip file and your mail system only supports 1MB files, you can ask Zip to create 3 files of at most 1MB.
So it creates a.zip.001, a.zip.002, a.zip.003 ... different libraries name them differently but essentially they all work the same way.
How do you unzip this in java? It doesn't look like the compression libs in std supports this.
Thanks.
Try to concatenate all the files into a single file and then extract the single file. Something like:
File dir = new File("D:/arc");
FileOutputStream fos = new FileOutputStream(new File(
"d:/arc/archieve-full.zip"));
FileInputStream fis = null;
Set<String> files = new TreeSet<String>();
for (String fname : dir.list()) {
files.add(fname);
}
for (String fname : files) {
try {
fis = new FileInputStream(new File(dir.getAbsolutePath(), fname));
byte[] b = new byte[fis.available()];
fis.read(b);
fos.write(b);
} finally {
if (fis != null) {
fis.close();
}
fos.flush();
}
}
fos.close();
ZipFile zipFile = new ZipFile("d:/arc/archieve-full.zip");
/*extract files from zip*/
Update: used a TreeSet to sort the file names, as dir.list() doesn't guarantee alphabetical order.
I need to create a File object out of a file path to an image that is contained in a jar file after creating a jar file. If tried using:
URL url = getClass().getResource("/resources/images/image.jpg");
File imageFile = new File(url.toURI());
but it doesn't work. Does anyone know of another way to do it?
To create a file on Android from a resource or raw file I do this:
try{
InputStream inputStream = getResources().openRawResource(R.raw.some_file);
File tempFile = File.createTempFile("pre", "suf");
copyFile(inputStream, new FileOutputStream(tempFile));
// Now some_file is tempFile .. do what you like
} catch (IOException e) {
throw new RuntimeException("Can't create temp file ", e);
}
private void copyFile(InputStream in, OutputStream out) throws IOException {
byte[] buffer = new byte[1024];
int read;
while((read = in.read(buffer)) != -1){
out.write(buffer, 0, read);
}
}
Don't forget to close your streams etc
This should work.
String imgName = "/resources/images/image.jpg";
InputStream in = getClass().getResourceAsStream(imgName);
ImageIcon img = new ImageIcon(ImageIO.read(in));
Usually, you can't directly get a java.io.File object, since there is no physical file for an entry within a compressed archive. Either you live with a stream (which is best most in the cases, since every good API can work with streams) or you can create a temporary file:
URL imageResource = getClass().getResource("image.gif");
File imageFile = File.createTempFile(
FilenameUtils.getBaseName(imageResource.getFile()),
FilenameUtils.getExtension(imageResource.getFile()));
IOUtils.copy(imageResource.openStream(),
FileUtils.openOutputStream(imageFile));
You cannot create a File object to a reference inside an archive. If you absolutely need a File object, you will need to extract the file to a temporary location first. On the other hand, most good API's will also take an input stream instead, which you can get for a file in an archive.
I have a java program as below for zipping a folder as a whole.
public static void zipDir(String dir2zip, ZipOutputStream zos)
{
try
{
File zipDir= new File(dir2zip);
String[] dirList = zipDir.list();
byte[] readBuffer = new byte[2156];
int bytesIn = 0;
for(int i=0; i<dirList.length; i++)
{
File f = new File(zipDir, dirList[i]);
if(f.isDirectory())
{
String filePath = f.getPath();
zipDir(filePath, zos);
continue;
}
FileInputStream fis = new FileInputStream(f);
ZipEntry anEntry = new ZipEntry(f.getPath());
zos.putNextEntry(anEntry);
while((bytesIn = fis.read(readBuffer)) != -1)
{
zos.write(readBuffer, 0, bytesIn);
}
fis.close();
}
}
catch(Exception e)
{
e.printStackTrace();
}
}
public static void main(){
String date=new java.text.SimpleDateFormat("MM-dd-yyyy").format(new java.util.Date());
ZipOutputStream zos = new ZipOutputStream(new FileOutputStream("Output/" + date + "_RB" + ".zip"));
zipDir("Output/" + date + "_RB", zos);
zos.close();
}
My query here is. The target folder(+date+_RB) to be zipped is present inside the folder named Output. After successful zipping, when I extract the zipped file, I find a folder Output inside which the (+date+_RB) required folder is present. I need not want that Output folder after the extraction of the zipped file, rather it should directly extract the required folder alone. Please advise on the same.
UPDATE:
I tried Isaac's answer. While extracting the resultant zip file, no folders are getting extracted. Only the files inside all the folders are getting extracted. I just dont need the folder "Output" alone in the resultant zip file. But what the program does is, it doesnt extracts all other folders inside the Output folder, rather it just extracts the files inside those folders. Kindly advise on how to proceed...
It happens because of this:
ZipEntry anEntry = new ZipEntry(f.getPath());
f.getPath() will return Output/ at the beginning of the string. This is due to the flow of your program and how it (mis)uses File objects.
I suggest you construct a File object called, say, tmp:
File tmp = new File(dirList[i]);
The change the construction of f:
File f = new File(zipDir, tmp.getPath());
Then, change this:
ZipEntry anEntry = new ZipEntry(f.getPath());
To this:
ZipEntry anEntry = new ZipEntry(tmp.getPath());
I didn't have time to actually test it, but in a nutshell, your problem is due to how the File object is constructed.