I'm downloading a zip file from an ftp server. The zip file contains a couple csv files. I'm trying to extract both csv files so that I can pass them into Opencsv, but I seem to be having some issues. I'm assuming there must be a better way to handle this than the way I'm doing it below. How do you return my csv files so that they are available in a list for my csv reader?
My code
ftp.retrieveFile(file, output);
InputStream inputStream = new ByteArrayInputStream(output.toByteArray());
Map<String, InputStream> inputStreams = new HashMap<>();
if (importTask.isZipfile()) {
inputStreams.put("products", importUtils.getZipData(new ZipInputStream(inputStream), importTask.getFilename()));
if(importTask.getCustomerFilename() != null) {
inputStream = new ByteArrayInputStream(output.toByteArray());
inputStreams.put("customers", importUtils.getZipData(new ZipInputStream(inputStream), importTask.getCustomerFilename()));
}
} else {
inputStreams.put("products", inputStream);
}
ftp.logout();
ftp.disconnect();
return inputStreams;
Zip
public InputStream getZipData(ZipInputStream zip, String filename) throws FileNotFoundException, IOException {
for (ZipEntry e; (e = zip.getNextEntry()) != null;) {
if (e.getName().equals(filename)) {
return zip;
}
}
throw new FileNotFoundException("zip://" + filename);
}
If you use Java 7+ you have an easier solution than that; you can just use the zip filesystem provider.
Here is some sample code; note that you need to .close() the generated InputStreams and FileSystems:
public static void getFsFromZipFile(final Path zipFile)
throws IOException
{
final URI uri = URI.create("jar:" + zipFile.toUri());
final Map<String, ?> env = Collections.singletonMap("readonly", "true");
return FileSystems.newFileSystem(uri, env);
}
public static getInputStreamFromZip(final FileSystem zipfs, final String name)
throws IOException
{
return Files.newInputStream(zipfs.getPath(name));
}
This is not how I'd recommend you do it however. What I'd recommend is this:
final Map<String, Path> getFilesFromZip(final Path zipFile, final String... names)
throws IOException
{
Path tmpfile;
final URI uri = URI.create("jar:" + zipFile.toUri());
final Map<String, ?> env = Collections.singletonMap("readonly", "true);
final Map<String, Path> ret = new HashMap<>();
try (
final FileSystem zipfs = FileSystems.newFileSystem(uri, env);
) {
for (final String name: names) {
tmpfile = Files.createTempFile("tmp", ".csv");
Files.copy(zipfs.getPath(name), tmpfile);
ret.put(name, tmpfile);
}
return ret;
}
}
Related
I'm running into a problem using the commons compress library to create a tar.gz of a directory. I have a directory structure that is as follows.
parent/
child/
file1.raw
file2.raw
file3.raw
I hope the compressed structure is like this.
child/
file1.raw
file2.raw
file3.raw
Is there any way to remove the outermost layer during compression?
I've seen such examples, but I can't work properly,and can only handle fixed name structures
public static void main(String[] args) throws IOException {
String hallFilePath = "E:/" + "packs";
compress(Paths.get(hallFilePath).toString(), hallFilePath + ".zip");
}
public static void compress(String fromPath, String toPath) throws IOException {
File fromFile = new File(fromPath);
File toFile = new File(toPath);
if (!fromFile.exists()) {
throw new ServiceException(fromPath + "不存在！");
}
try (FileOutputStream outputStream = new FileOutputStream(toFile); CheckedOutputStream checkedOutputStream = new CheckedOutputStream(outputStream, new CRC32()); ZipOutputStream zipOutputStream = new ZipOutputStream(checkedOutputStream)) {
String baseDir = "";
compress(fromFile, zipOutputStream, baseDir);
}
}
private static void compress(File file, ZipOutputStream zipOut, String baseDir) throws IOException {
if (file.isDirectory()) {
compressDirectory(file, zipOut, baseDir);
} else {
if (baseDir.equals("packs" + File.separator)) {
baseDir = File.separator;
} else if (baseDir.equals("packs" + File.separator + "examineeInfo" + File.separator)) {
baseDir = "examineeInfo" + File.separator;
}
compressFile(file, zipOut, baseDir);
}
}
private static void compressFile(File file, ZipOutputStream zipOut, String baseDir) throws IOException {
if (!file.exists()) {
return;
}
try (BufferedInputStream bis = new BufferedInputStream(new FileInputStream(file))) {
ZipEntry entry = new ZipEntry(baseDir + file.getName());
zipOut.putNextEntry(entry);
int count;
byte[] data = new byte[BUFFER];
while ((count = bis.read(data, 0, BUFFER)) != -1) {
zipOut.write(data, 0, count);
}
}
}
private static void compressDirectory(File dir, ZipOutputStream zipOut, String baseDir) throws IOException {
File[] files = dir.listFiles();
if (files != null && ArrayUtils.isNotEmpty(files)) {
for (File file : files) {
compress(file, zipOut, baseDir + dir.getName() + File.separator);
}
}
}
I have a method that has a few static classes as well as java.nio.Files,ResourceUtils, etc I am not sure how to write test cases for it as I keep getting NPE Paths.get(filePath);
Below is the method:
#Value("${file.path}")
private String filePath;
private List<MyJsonObj> readJSONFiles() throws IOException {
List<MyJsonObj> myJsonObjList = new ArrayList<>();
Gson gson = new GsonBuilder().setLongSerializationPolicy(LongSerializationPolicy.STRING).create();
Path folder = Paths.get(filePath);
try (DirectoryStream<Path> stream = Files.newDirectoryStream(folder)) {
for (Path entry : stream) {
String file = folder + "\\" + entry.getFileName().toString();
MyJsonObj jsonObj = gson.fromJson(new FileReader(ResourceUtils.getFile(file)), MyJsonObj.class);
myJsonObjList.add(jsonObj);
}
return myJsonObjList;
}catch (IOException e) {
logger.error(Arrays.asList(e.getStackTrace()).toString());
}
}
I've got some issue while reading ZipInputStream to byte array - result is -1. However I can read it as file stream, so I'm not sure if I'm missing something:
public void readContent () throws IOException {
int size = 2048;
byte[] file = new byte[size];
byte[] stream = new byte[size];
FileInputStream fileStream = new FileInputStream(this.file);
ZipInputStream zipStream = new ZipInputStream(fileStream);;
System.out.println("Reading as a file stream " + fileStream.read(file,0,size)); //output 2048
System.out.println("Reading as a zip stream " + zipStream.read(stream,0,size)); // output -1
}
Data from ZipInputStream should be read via ZipEntry. This is example how to read zip archive from given InputStream and retrieve Map with key - full zip entry path, value - zip entry data:
public class ZipUtils {
private static final Comparator<String> COMPARATOR_STRING_DESC = (str1, str2) -> str1 == null ? 1 : str2 == null ? -1 : -str1.compareTo(str2);
public static Map<String, byte[]> unzipIt(InputStream is) throws IOException {
try (ZipInputStream in = new ZipInputStream(is)) {
ZipEntry entry;
Map<String, byte[]> content = new HashMap<>();
Set<String> dirs = new TreeSet<>(COMPARATOR_STRING_DESC);
while ((entry = in.getNextEntry()) != null) {
String path = removeDirectoryMarker(replaceIncorrectFileSeparators(entry.getName()));
if (isDirectory(entry)) {
dirs.add(path);
} else {
content.put(path, IOUtils.toByteArray(in));
}
}
addOnlyEmptyDirectories(dirs, content);
return content.isEmpty() ? Collections.emptyMap() : content;
}
}
private static boolean isDirectory(ZipEntry entry) {
return entry.isDirectory() || entry.getName().endsWith(ILLEGAL_DIR_MARKER);
}
private static void addOnlyEmptyDirectories(Set<String> dirs, Map<String, byte[]> content) {
if (dirs.isEmpty()) {
return;
}
Set<String> paths = new HashSet<>(content.keySet());
for (String dir : dirs) {
boolean empty = true;
for (String path : paths) {
if (path.startsWith(dir)) {
empty = false;
break;
}
}
if (empty) {
content.put(dir, null);
}
}
}
private static final String DIR_MARKER = "/";
private static final String ILLEGAL_DIR_MARKER = "\\";
private static final java.util.regex.Pattern BACK_SLASH = Pattern.compile("\\\\");
private static String removeDirectoryMarker(String path) {
return path.endsWith(DIR_MARKER) || path.endsWith(ILLEGAL_DIR_MARKER) ? path.substring(0, path.length() - 1) : path;
}
private static String replaceIncorrectFileSeparators(String path) {
return BACK_SLASH.matcher(path).replaceAll(DIR_MARKER);
}
}
I've found the Resources.readLines() and Files.readLines() to be helpfull in simplifiying my code.
The problem is that I often read gzip-compressed txt-files or txt-files in zip archives from URL's (HTTP and FTP).
Is there a way to use Guava's methods to read from these URL's too? Or is that only possible with Java's GZIPInputStream/ZipInputStream?
You can create your own ByteSources:
For GZip:
public class GzippedByteSource extends ByteSource {
private final ByteSource source;
public GzippedByteSource(ByteSource gzippedSource) { source = gzippedSource; }
#Override public InputStream openStream() throws IOException {
return new GZIPInputStream(source.openStream());
}
}
Then use it:
Charset charset = ... ;
new GzippedByteSource(Resources.asByteSource(url)).toCharSource(charset).readLines();
Here is the implementation for the Zip. This assumes that you read only one entry.
public static class ZipEntryByteSource extends ByteSource {
private final ByteSource source;
private final String entryName;
public ZipEntryByteSource(ByteSource zipSource, String entryName) {
this.source = zipSource;
this.entryName = entryName;
}
#Override public InputStream openStream() throws IOException {
final ZipInputStream in = new ZipInputStream(source.openStream());
while (true) {
final ZipEntry entry = in.getNextEntry();
if (entry == null) {
in.close();
throw new IOException("No entry named " + entry);
} else if (entry.getName().equals(this.entryName)) {
return new InputStream() {
#Override
public int read() throws IOException {
return in.read();
}
#Override
public void close() throws IOException {
in.closeEntry();
in.close();
}
};
} else {
in.closeEntry();
}
}
}
}
And you can use it like this:
Charset charset = ... ;
String entryName = ... ; // Name of the entry inside the zip file.
new ZipEntryByteSource(Resources.asByteSource(url), entryName).toCharSource(charset).readLines();
As Olivier Grégoire said, you can create the necessary ByteSources for whatever compression scheme you need in order to use Guava's readLines function.
For zip archives though, although it's possible to do it, I don't think it's worth it. It will be easier to make your own readLines method that iterates over the zip entries and reads the lines of each entry on your own. Here's a class that demonstrates how to read and output the lines of a URL pointing at a zip archive:
public class ReadLinesOfZippedUrl {
public static List<String> readLines(String urlStr, Charset charset) {
List<String> retVal = new LinkedList<>();
try (ZipInputStream zipInputStream = new ZipInputStream(new URL(urlStr).openStream())) {
for (ZipEntry zipEntry = zipInputStream.getNextEntry(); zipEntry != null; zipEntry = zipInputStream.getNextEntry()) {
// don't close this reader or you'll close the underlying zip stream
BufferedReader reader = new BufferedReader(new InputStreamReader(zipInputStream, charset));
retVal.addAll(reader.lines().collect(Collectors.toList())); // slurp all the lines from one entry
}
} catch (IOException e) {
throw new UncheckedIOException(e);
}
return retVal;
}
public static void main(String[] args) {
String urlStr = "http://central.maven.org/maven2/com/google/guava/guava/18.0/guava-18.0-sources.jar";
Charset charset = StandardCharsets.UTF_8;
List<String> lines = readLines(urlStr, charset);
lines.forEach(System.out::println);
}
}
I am running Mac OSX Mavericks. Right now I am creating a JAR file from a folder (org, the package). When I use this code from here:
public void run() throws IOException
{
Manifest manifest = new Manifest();
manifest.getMainAttributes().put(Attributes.Name.MANIFEST_VERSION, "1.0");
JarOutputStream target = new JarOutputStream(new FileOutputStream("/Users/username/Library/Application Support/VSE/temp/output.jar"), manifest);
add(new File("/Users/username/Library/Application Support/VSE/temp/org"), target);
target.close();
}
private void add(File source, JarOutputStream target) throws IOException
{
BufferedInputStream in = null;
try
{
if (source.isDirectory())
{
String name = source.getPath().replace("\\", "/");
if (!name.isEmpty())
{
if (!name.endsWith("/"))
name += "/";
JarEntry entry = new JarEntry(name);
entry.setTime(source.lastModified());
target.putNextEntry(entry);
target.closeEntry();
}
for (File nestedFile: source.listFiles())
add(nestedFile, target);
return;
}
JarEntry entry = new JarEntry(source.getPath().replace("\\", "/"));
entry.setTime(source.lastModified());
target.putNextEntry(entry);
in = new BufferedInputStream(new FileInputStream(source));
byte[] buffer = new byte[1024];
while (true)
{
int count = in.read(buffer);
if (count == -1)
break;
target.write(buffer, 0, count);
}
target.closeEntry();
}
finally
{
if (in != null)
in.close();
}
}
When I extract the JAR file, There is a META-INF folder, but instead of having the org folder in the extracted jar, I have my Users folder copied into it (except because of it's size, its wasn't filled with all my stuff and my application crashed). I'm expecting this is because the code was written for a Windows system, and the differences with the filesystem (such as \ or /). How would I make the code include only the "org" directory, and not everything leading up to it?
Provided you use Java 7+ you may easily do this by using one of my packages in combination with the zip filesystem provider of the JDK to create it:
private static final Map<String, ?> ENV = Collections.singletonMap("create", "true");
public void run()
throws IOException
{
final Path zipPath = Paths.get("/Users/username/Library/Application Support/VSE/temp/output.jar");
final Path srcdir = Paths.get("/Users/username/Library/Application Support/VSE/temp/org");
final URI uri = URI.create("jar:" + zipPath.toUri());
Files.deleteIfExists(zipPath);
try (
final FileSystem zipfs = FileSystems.newFileSystem(uri, ENV);
) {
copyManifest(zipfs);
copyDirectory(srcdir, zipfs);
}
}
private void copyManifest(final FileSystem zipfs)
throws IOException
{
final Manifest manifest = new Manifest();
manifest.getMainAttributes().put(Attributes.Name.MANIFEST_VERSION, "1.0");
Files.createDirectory(zipfs.getPath("META-INF/");
try (
final OutputStream out = Files.newOutputStream(zipfs.getPath("META-INF/MANIFEST.MF"));
) {
manifest.write(out);
}
}
private void copyDirectory(final Path srcdir, final FileSystem zipfs)
{
final String lastName = srcdir.getFileName().toString();
final Path dstDir = zipfs.getPath(lastName);
Files.createDirectory(dstDir);
MoreFiles.copyRecursive(srcDir, dstDir, RecursionMode.FAIL_FAST);
}