I'm new to HDFS. When I try to write in HDFS file I received the following:
[WARN] org.apache.hadoop.hdfs.DFSClient - DataStreamer Exception >java.io.IOException: Unable to create new block]. The response code is 200,
but when read file content this is null. Here is my code:
public void writeFile(FileSystem fs, String destination) throws IOException {
Path workingDir = fs.getWorkingDirectory();
Path newFilePath = new Path("/" + destination);
newFilePath = Path.mergePaths(workingDir, newFilePath);
FsPermission fsPermission = new FsPermission(FsAction.ALL, FsAction.ALL, FsAction.ALL);
StringBuilder sb = new StringBuilder();
for (int i = 1; i <= 5; i++) {
sb.append("Data");
sb.append(i);
sb.append("\n");
}
byte[] byt = sb.toString().getBytes();
FSDataOutputStream fsOutStream = null;
try {
fsOutStream = fs.create(fs, newFilePath, fsPermission);
fsOutStream.write(byt);
} catch (Throwable e) {
log.error("Error while creating file in Hdfs",e);
} finally {
try {
if (fsOutStream != null) {
fsOutStream.close();
}
}catch(Throwable e) {
log.error("Error while closing FSDataOutputStream", e);
}
}
}
Any suggestions is welcome. Thank you in advance!
Related
I want to copy a file from one location to another location in Java. What is the best way to do this?
Here is what I have so far:
import java.io.File;
import java.io.FilenameFilter;
import java.util.ArrayList;
import java.util.List;
public class TestArrayList {
public static void main(String[] args) {
File f = new File(
"D:\\CBSE_Demo\\Demo_original\\fscommand\\contentplayer\\config");
List<String>temp=new ArrayList<String>();
temp.add(0, "N33");
temp.add(1, "N1417");
temp.add(2, "N331");
File[] matchingFiles = null;
for(final String temp1: temp){
matchingFiles = f.listFiles(new FilenameFilter() {
public boolean accept(File dir, String name) {
return name.startsWith(temp1);
}
});
System.out.println("size>>--"+matchingFiles.length);
}
}
}
This does not copy the file, what is the best way to do this?
You can use this (or any variant):
Files.copy(src, dst, StandardCopyOption.REPLACE_EXISTING);
Also, I'd recommend using File.separator or / instead of \\ to make it compliant across multiple OS, question/answer on this available here.
Since you're not sure how to temporarily store files, take a look at ArrayList:
List<File> files = new ArrayList();
files.add(foundFile);
To move a List of files into a single directory:
List<File> files = ...;
String path = "C:/destination/";
for(File file : files) {
Files.copy(file.toPath(),
(new File(path + file.getName())).toPath(),
StandardCopyOption.REPLACE_EXISTING);
}
Update:
see also
https://stackoverflow.com/a/67179064/1847899
Using Stream
private static void copyFileUsingStream(File source, File dest) throws IOException {
InputStream is = null;
OutputStream os = null;
try {
is = new FileInputStream(source);
os = new FileOutputStream(dest);
byte[] buffer = new byte[1024];
int length;
while ((length = is.read(buffer)) > 0) {
os.write(buffer, 0, length);
}
} finally {
is.close();
os.close();
}
}
Using Channel
private static void copyFileUsingChannel(File source, File dest) throws IOException {
FileChannel sourceChannel = null;
FileChannel destChannel = null;
try {
sourceChannel = new FileInputStream(source).getChannel();
destChannel = new FileOutputStream(dest).getChannel();
destChannel.transferFrom(sourceChannel, 0, sourceChannel.size());
}finally{
sourceChannel.close();
destChannel.close();
}
}
Using Apache Commons IO lib:
private static void copyFileUsingApacheCommonsIO(File source, File dest) throws IOException {
FileUtils.copyFile(source, dest);
}
Using Java SE 7 Files class:
private static void copyFileUsingJava7Files(File source, File dest) throws IOException {
Files.copy(source.toPath(), dest.toPath());
}
Or try Googles Guava :
https://github.com/google/guava
docs:
https://guava.dev/releases/snapshot-jre/api/docs/com/google/common/io/Files.html
Use the New Java File classes in Java >=7.
Create the below method and import the necessary libs.
public static void copyFile( File from, File to ) throws IOException {
Files.copy( from.toPath(), to.toPath() );
}
Use the created method as below within main:
File dirFrom = new File(fileFrom);
File dirTo = new File(fileTo);
try {
copyFile(dirFrom, dirTo);
} catch (IOException ex) {
Logger.getLogger(TestJava8.class.getName()).log(Level.SEVERE, null, ex);
}
NB:- fileFrom is the file that you want to copy to a new file fileTo in a different folder.
Credits - #Scott: Standard concise way to copy a file in Java?
public static void copyFile(File oldLocation, File newLocation) throws IOException {
if ( oldLocation.exists( )) {
BufferedInputStream reader = new BufferedInputStream( new FileInputStream(oldLocation) );
BufferedOutputStream writer = new BufferedOutputStream( new FileOutputStream(newLocation, false));
try {
byte[] buff = new byte[8192];
int numChars;
while ( (numChars = reader.read( buff, 0, buff.length ) ) != -1) {
writer.write( buff, 0, numChars );
}
} catch( IOException ex ) {
throw new IOException("IOException when transferring " + oldLocation.getPath() + " to " + newLocation.getPath());
} finally {
try {
if ( reader != null ){
writer.close();
reader.close();
}
} catch( IOException ex ){
Log.e(TAG, "Error closing files when transferring " + oldLocation.getPath() + " to " + newLocation.getPath() );
}
}
} else {
throw new IOException("Old location does not exist when transferring " + oldLocation.getPath() + " to " + newLocation.getPath() );
}
}
Copy a file from one location to another location means,need to copy the whole content to another location.Files.copy(Path source, Path target, CopyOption... options) throws IOException this method expects source location which is original file location and target location which is a new folder location with destination same type file(as original).
Either Target location needs to exist in our system otherwise we need to create a folder location and then in that folder location we need to create a file with the same name as original filename.Then using copy function we can easily copy a file from one location to other.
public static void main(String[] args) throws IOException {
String destFolderPath = "D:/TestFile/abc";
String fileName = "pqr.xlsx";
String sourceFilePath= "D:/TestFile/xyz.xlsx";
File f = new File(destFolderPath);
if(f.mkdir()){
System.out.println("Directory created!!!!");
}
else {
System.out.println("Directory Exists!!!!");
}
f= new File(destFolderPath,fileName);
if(f.createNewFile()) {
System.out.println("File Created!!!!");
} else {
System.out.println("File exists!!!!");
}
Files.copy(Paths.get(sourceFilePath), Paths.get(destFolderPath, fileName),REPLACE_EXISTING);
System.out.println("Copy done!!!!!!!!!!!!!!");
}
You can do it with the Java 8 Streaming API, PrintWriter and the Files API
try (PrintWriter pw = new PrintWriter(new File("destination-path"), StandardCharsets.UTF_8)) {
Files.readAllLines(Path.of("src/test/resources/source-file.something"), StandardCharsets.UTF_8)
.forEach(pw::println);
}
If you want to modify the content on-the-fly while copying, check out this link for the extended example https://overflowed.dev/blog/copy-file-and-modify-with-java-streams/
I modified one of the answers to make it a bit more efficient.
public void copy(){
InputStream in = null;
try {
in = new FileInputStream(Files);
} catch (FileNotFoundException e) {
e.printStackTrace();
}
try {
OutputStream out = new FileOutputStream();
try {
// Transfer bytes from in to out
byte[] buf = new byte[1024];
while (true) {
int len = 0;
try {
if (!((len = in.read(buf)) > 0)) break;
} catch (IOException e) {
e.printStackTrace();
}
try {
out.write(buf, 0, len);
} catch (IOException e) {
e.printStackTrace();
}
}
} finally {
try {
out.close();
} catch (IOException e) {
e.printStackTrace();
}
}
} finally {
try {
in.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
private void moveFile() {
copy();
File dir = getFilesDir();
File file = new File(dir, "my_filename");
boolean deleted = file.delete();
}
Files.exists()
Files.createDirectory()
Files.copy()
Overwriting Existing Files:
Files.move()
Files.delete()
Files.walkFileTree()
enter link description here
You can use
FileUtils.copy(sourceFile, destinationFile);
https://commons.apache.org/proper/commons-io/apidocs/org/apache/commons/io/FileUtils.html
create and download zip file by adding list of text files. with out creating the file in local server, it should be download at client side direct,
Here i added a code snippet, it was creating in local server, but i dont want that, it should create and download at client side instant. Please help me in this way..
#GetMapping("/download/rawdata")
public void downloadRawdata(#RequestParam("date") String date){
log.info("date : "+date);
List<Rawdata> rawdatas = rawdataRepoisotry.findRawdataByDate(date);
log.info("size of rawdata : "+rawdatas.size());
List<File> files = new ArrayList<File>();
int i = 1;
for(Rawdata rawdata : rawdatas){
log.info("rawdata : "+ rawdata.getRawdata());
File file = new File(i+".txt");
try (Writer writer = new BufferedWriter(new FileWriter(file))) {
String contents = rawdata.getRawdata();
writer.write(contents);
files.add(file);
} catch (IOException e) {
e.printStackTrace();
}
i++;
}
try {
zipFile(files, new File(date+".zip"));
} catch (IOException e) {
e.printStackTrace();
throw new RuntimeException("Failed while creating Zip file");
}
}
public FileOutputStream zipFile(final List<File> files, final File targetZipFile) throws IOException {
try {
FileOutputStream fos = new FileOutputStream(targetZipFile);
ZipOutputStream zos = new ZipOutputStream(fos);
byte[] buffer = new byte[128];
for(File currentFile : files){
if (!currentFile.isDirectory()) {
ZipEntry entry = new ZipEntry(currentFile.getName());
FileInputStream fis = new FileInputStream(currentFile);
zos.putNextEntry(entry);
int read = 0;
while ((read = fis.read(buffer)) != -1) {
zos.write(buffer, 0, read);
}
zos.closeEntry();
fis.close();
}
}
zos.close();
fos.close();
return fos;
} catch (FileNotFoundException e) {
System.out.println("File not found : " + e);
throw new FileNotFoundException();
}
}
Here is an example using FileSystemResource.
What has been modified is (see the numbers in the commented code ) :
1) Declare that the response will be of type application/octet-stream
2) #ResponseBody
Annotation that indicates a method return value should be bound to the
web response body
3) Declare that the method returns a FileSystemResource body
4) Return the FileSystemResource entity based on your created zip file
Note that this will still create the file on the server side first, but you may want to use File.createTempFile and File.deleteOnExit.
#GetMapping("/download/rawdata", produces = MediaType.APPLICATION_OCTET_STREAM_VALUE)//1
#ResponseBody //2
public ResponseEntity<FileSystemResource> downloadRawdata(#RequestParam("date") String date){ //3
log.info("date : "+date);
List<Rawdata> rawdatas = rawdataRepoisotry.findRawdataByDate(date);
log.info("size of rawdata : "+rawdatas.size());
List<File> files = new ArrayList<File>();
int i = 1;
for(Rawdata rawdata : rawdatas){
log.info("rawdata : "+ rawdata.getRawdata());
File file = new File(i+".txt");
try (Writer writer = new BufferedWriter(new FileWriter(file))) {
String contents = rawdata.getRawdata();
writer.write(contents);
files.add(file);
} catch (IOException e) {
e.printStackTrace();
}
i++;
}
try {
File resultFile = new File(date+".zip");
zipFile(files, resultFile);
return new ResponseEntity<>(new FileSystemResource(resultFile), HttpStatus.OK); //4
} catch (IOException e) {
e.printStackTrace();
throw new RuntimeException("Failed while creating Zip file");
}
}
I'm new here and i would like to ask on how properties can work with the java codes i mean the values inside of the properties will be use as variables. For example i have file1.txt and file2.txt inside config.properties and store it in an Arraylist then scan the folder and if the files are found copy it. My work only shows the names of the data from the properties that is stored in the arraylist but my another problem is how these data will be copied to another folder.
so far this is my code
public class MainClass {
static Properties prop = new Properties();
static InputStream input = null;
static String filename = "";
public static void main(String[] args) throws IOException {
File source = new File("D:/ojt");
File dest = new File("D:/ojt/New folder");
Properties prop = new Properties();
InputStream input = null;
try {
// getFiles(filename);
} catch (IOException ex) {
ex.printStackTrace();
} finally {
if (input != null) {
try {
input.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
private static void copyFileUsingStream(File source, File dest) throws IOException {
InputStream is = null;
OutputStream os = null;
try {
is = new FileInputStream(source);
os = new FileOutputStream(dest);
byte[] buffer = new byte[1024];
int length;
while ((length = is.read(buffer)) > 0) {
os.write(buffer, 0, length);
}
} finally {
is.close();
os.close();
}
}
private static void copy(String fromPath, String outputPath)
{
// filter = new FileTypeOrFolderFilter(fileType);
File currentFolder = new File(fromPath);
File outputFolder = new File(outputPath);
scanFolder(currentFolder, outputFolder);
}
private static void getFiles(String path) throws IOException{
//Put filenames in arraylist<string>
String filename = "bydatefilesdir.props";
input = MainClass.class.getClassLoader().getResourceAsStream(filename);
Scanner s = new Scanner(input);
File dir = new File(path);
final ArrayList<String> list = new ArrayList<String>();
while (s.hasNextLine()){
list.add(s.nextLine());
}
// ArrayList<String> filenames = new ArrayList<String>();
// for(File file : dir.listFiles()){
// filenames.add(file.getName());
// }
prop.load(input);
//Check if the files are in the arraylist
for (int i = 0; i < list.size(); i++){
String s1 = list.get(i);
System.out.println("File "+i+" : "+s1);
}
System.out.println("\n");
}
private static void copyFileUsingStream(File source, File dest) throws IOException {
InputStream is = null;
OutputStream os = null;
try {
is = new FileInputStream(source);
os = new FileOutputStream(dest);
byte[] buffer = new byte[1024];
int length;
while ((length = is.read(buffer)) > 0) {
os.write(buffer, 0, length);
}
} finally {
is.close();
os.close();
}
}
private static void copy(String source, String dest)
{
// filter = new FileTypeOrFolderFilter(fileType);
File currentFolder = new File(source);
File outputFolder = new File(dest);
scanFolder(currentFolder, outputFolder);
}
private static void scanFolder(File source, File dest)
{
System.out.println("Scanning folder [" + source.toString() + "]...\n");
File[] files = source.listFiles();
for (File file : files) {
if (file.isDirectory()) {
scanFolder(source, dest);
} else {
try {
copyFileUsingStream(source, dest);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
}
PS: sorry for the poor programming just new in java.. still learning
Edited: I've included the updated codes above..
I suppose java.util.Properties has build in methods to achieve what you are trying.
please refer this example and you will find a better solution.
public class App {
public static void main( String[] args ){
Properties prop = new Properties();
InputStream input = null;
try {
String filename = "config.properties";
input = App.class.getClassLoader().getResourceAsStream(filename);
if(input==null){
System.out.println("Sorry, unable to find " + filename);
return;
}
//load a properties file from class path, inside static method
prop.load(input);
//get the property value and print it out
System.out.println(prop.getProperty("file1.txt"));
System.out.println(prop.getProperty("file2.txt"));
} catch (IOException ex) {
ex.printStackTrace();
} finally{
if(input!=null){
try {
input.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
}
I have written code that should be saved file in the local directory, create zip of that file, send email and delete both files (original and zip), So this is my code:
Method wich send email
public void sendEmail(Properties emailProperties, InputStream inputStream, HttpServletRequest request) throws UnsupportedEncodingException {
MimeMessage mimeMessage = mailSender.createMimeMessage();
try {
MimeMessageHelper mimeMessageHelper = new MimeMessageHelper(mimeMessage, true);
try {
mimeMessageHelper.setFrom(from, personal);
} catch (UnsupportedEncodingException e) {
LOGGER.error(e.getMessage());
throw new SequelException(e.getMessage());
}
mimeMessageHelper.setTo(recipients);
mimeMessageHelper.setSubject(emailProperties.getProperty(PARAM_TITLE));
String message = emailProperties.getProperty(PARAM_EMLMSG);
mimeMessageHelper.setText(message);
InputStreamSource inputStreamSource = null;
if (inputStream != null) {
inputStreamSource = new ByteArrayResource(IOUtils.toByteArray(inputStream));
}
String compressType = COMPRESS_TYPE_ZIP;
String fileName = getAttachFilenameExtension(object, format);
Path filePath = Paths.get(StrUtils.getProperty("temp.email.files.path") + "\\" + fileName);
tempFile = saveTempFile(inputStreamSource.getInputStream(), filePath);
if (tempFile.length() > 0) {
inputStreamSource = compressFile(tempFile, filePath.toString(), compressType);
fileName = StringUtils.substring(fileName, 0, StringUtils.lastIndexOf(fileName, ".")+1) + compressType;
}
mimeMessageHelper.addAttachment(fileName, inputStreamSource);
mailSender.send(mimeMessage);
} catch (MessagingException | IOException e) {
LOGGER.error(e.getMessage());
throw new SequelException(e.getMessage());
} finally {
List<File> files = (List<File>) FileUtils.listFiles(tempFile.getParentFile(), new WildcardFileFilter(
FilenameUtils.removeExtension(tempFile.getName()) + "*"), null);
for (File file : files) {
try {
FileUtils.forceDelete(file);
} catch (IOException e) {
LOGGER.error(e.getMessage());
}
}
}
}
Save file in directory:
private File saveTempFile(InputStream inputStream, Path filePath) throws IOException {
Files.deleteIfExists(filePath);
Files.copy(inputStream, filePath);
return new File(filePath.toString());
}
Compress file:
private InputStreamSource compressFile(File file, String filePath, String compressType) throws IOException {
InputStream is = ZipFile(file, filePath);
InputStreamSource inputStreamSource = new ByteArrayResource(IOUtils.toByteArray(is));
return inputStreamSource;
}
public InputStream ZipFile(File file, String filePath) {
String zipArchiveFileName = StringUtils.substring(filePath, 0, filePath.lastIndexOf(".") + 1) + COMPRESS_TYPE_ZIP;
try (ZipArchiveOutputStream zipOutput = new ZipArchiveOutputStream(new File(zipArchiveFileName));) {
ZipArchiveEntry entry = new ZipArchiveEntry(StringUtils.overlay(file.getName(), "",
StringUtils.lastIndexOf(file.getName(), "_"), StringUtils.lastIndexOf(file.getName(), ".")));
zipOutput.putArchiveEntry(entry);
try (FileInputStream in = new FileInputStream(file);) {
byte[] b = new byte[1024];
int count = 0;
while ((count = in.read(b)) > 0) {
zipOutput.write(b, 0, count);
}
zipOutput.closeArchiveEntry();
}
InputStream is = new FileInputStream(zipArchiveFileName);
return is;
} catch (IOException e) {
LOGGER.error("An error occurred while trying to compress file to zip", e);
throw new SequelException(e.getMessage());
}
}
So the problem is when I try to delete files but zip file does not delete.
I am using Apache commons compress for zipping.
Can you help what's wrong?
For me this code is working perfectly. After compressing you may be trying to delete it without the extension(for eg .7z here).
public static void main(String[] args) {
File file = new File("C:\\Users\\kh1784\\Desktop\\Remote.7z");
file.delete();
if(!file.exists())
System.out.println("Sucessfully deleted the file");
}
Output:-
Sucessfully deleted the file
while downloading the object from amazon s3 i cant able to download it to diffrent folder rather then the file uploaded path...why it is happening like this may be it is metaData problem....please post your valuable comments Thanks in advance..below am posting the code for upload and download
public void AmazonUpload(String fileObj) throws IOException {
try {
this.key = fileObj;
try {
if (this.key == null) {
} else {
if (readFile(this.key) != null) {
// this.key="1";
this.putObjResult = this.amzObj.putObject(new PutObjectRequest(this.bucketName, this.key, readFile(this.key)));
}
}
} catch (AmazonServiceException ae) {
System.out.println(ae.getMessage());
}
} catch (AmazonServiceException ex) {
Logger.getLogger(AthinioCloudMigration.class.getName()).log(Level.SEVERE, null, ex);
}
}
public void AmazonDownload(String dirName, String xmlFilename, String amazonid) throws ParserConfigurationException, SAXException, TransformerException, IOException {
String cloudid;
cloudid = amazonid;
this.comm = new CommonResources(xmlFilename);
this.RequestFiles=new ArrayList();
try {
this.RequestFiles = this.comm.getXML(cloudid);
if (this.RequestFiles != null) {
int len = this.RequestFiles.size();
System.out.println(len);
for (int index = 0; index < len; index++) {
this.CRobj = (CommonResources) this.RequestFiles.get(index);
if (cloudid.equals(this.CRobj.getCloudID())) {
this.newFile = new File(dirName + this.CRobj.getFileName().concat(".rec"));
System.out.println(newFile);
newFile.createNewFile();
this.metaData = this.amzObj.getObject(new GetObjectRequest(this.bucketName, (dirName + this.CRobj.getFileName())), this.newFile);
System.out.println(metaData);
java.io.File tmp = new java.io.File(dirName + this.CRobj.getFileName());
System.out.println(tmp);
tmp.delete();
76,23 87%
Since in Amazon S3 there is no folder structure you receive everything as a object.
For Example: In your bucket you store file in a folder like structure but when you request fro the objects from S3 you will receve your file like folder1/folder2/demo.txt.
So try this one, Get the InputStream for the S3 for your file like amazonS3.getObject(bucket, "folder1/folder2/demo.txt").getObjectContent();. After getting your InputStream pass your File Name, Download location and InputStream to the below method. If you use Java 7 use FileSystems.getDefault().getPath(fullPathWithFileName).getFileName().toString() to get file name from your object name.
public void saveFile(String uploadFileName, String path, InputStream inputStream) throws Exception {
DataOutputStream dos = null;
OutputStream out = null;
try {
File newDirectory = new File(path);
if (!newDirectory.exists()) {
newDirectory.mkdirs();
}
File uploadedFile = new File(path, uploadFileName);
out = new FileOutputStream(uploadedFile);
byte[] fileAsBytes = new byte[inputStream.available()];
inputStream.read(fileAsBytes);
dos = new DataOutputStream(out);
dos.write(fileAsBytes);
} catch (IOException io) {
io.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
if (out != null) {
out.close();
}
if (dos != null) {
dos.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}