We Keep Coding

How to Save Dataframe as Text File using User Defined File Name in Spark Java

I am trying to save a dataframe to a specific location.
successDF.toJavaRDD().saveAsTextFile(successFilePath);
Here, successFilePath is: /hdfs/tmp/20200102/04.dat
I need to save the data with filename as 04.dat, where 20200102 and 04 are coming as arguments
But the process creates multiple files as below:
Folder: /hdfs/tmp/20200102/04.dat
Files:
._SUCCESS.crc
.part-00000.crc
_SUCCESS
part-00000
My requirement is, the output file should be created in /hdfs/tmp/20200102 and there should be only 1 file under the folder with file name as: 04.dat
N.B. I am using Spark Java
Please suggest

You can create file on HDFS without using Spark:
Using HDFS API
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
public class HDFSFileWrite {
public static void main(String[] args) {
Configuration conf = new Configuration();
try {
FileSystem fs = FileSystem.get(conf);
// Hadoop DFS Path - Input & Output file
Path inFile = new Path(args[0]);
Path outFile = new Path(args[1]);
// Verification
if (!fs.exists(inFile)) {
System.out.println("Input file not found");
throw new IOException("Input file not found");
}
if (fs.exists(outFile)) {
System.out.println("Output file already exists");
throw new IOException("Output file already exists");
}
// open and read from file
FSDataInputStream in = fs.open(inFile);
// Create file to write
FSDataOutputStream out = fs.create(outFile);
byte buffer[] = new byte[256];
try {
int bytesRead = 0;
while ((bytesRead = in.read(buffer)) > 0) {
out.write(buffer, 0, bytesRead);
}
} catch (IOException e) {
System.out.println("Error while copying file");
} finally {
in.close();
out.close();
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}

Unable to read properties file from java

I am unable to read the properties from the file . When I try to print it gives me null, When I debugged I understood it is not loading the file in function
pro.Load(). However my path is correct, still I am unable to load the file
package AdvancedJava;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.util.Properties;
public class ReadingPropertiesFile {
public static void main(String[] args) throws FileNotFoundException {
Properties pro = new Properties();
String path = "C://Users//310259741//Documents//ProjectManagment//JavaBasics//object.properties";
// BufferedReader reader = new BufferedReader(new FileReader(path));
File f = new File(path);
FileInputStream fis = null;
try {
fis = new FileInputStream(f);
pro.load(fis);
}
catch (IOException e) {
System.out.println(e.getMessage());
}
System.out.println(pro.getProperty("lastname"));
}
}
Properties file contents
firstname = John
lastname = harry
Automation = Selenium

I think the problem is in path:
String path = "C://Users//310259741//Documents//ProjectManagment//JavaBasics//object.properties";
should be like this:
String path = "C:\\Users\\310259741\\Documents\\ProjectManagment\\JavaBasics\\object.properties";
Also make sure you have a correct path to your properties file. If it is inside your project, the path should be like this:
String path = "C:\\...\\ProjectFolder\\src\\main\\resources\\object.properties";

Your example works fine for me. Without a stacktrace though, we won't be able to help you regarding the NPE you're getting.
In any way though, I couple of hints regarding your code. I would suggest using a try - with resources when operating with the FileInputStream to make sure that the resource is going to be closed once done.
You can avoid using new File(path);. Instead I would suggest using Paths from the java.nio.* package. An example of this based on your code snippet would be the following:
public static void main(String[] args) {
Properties properties = new Properties();
try (FileInputStream stream = new FileInputStream(Paths.get("E:\\test\\file.txt").toFile())) {
properties.load(stream);
} catch (IOException e) {
e.printStackTrace();
}
System.out.println(properties.getProperty("lastname"));
}
The advantage of using Paths is that they're (if not mistaken) system agnostic meaning that you won't need to worry about providing the proper path delimiters.

Actually, path should be with another separator
"C:\\Users\\310259741\\Documents\\ProjectManagment\\JavaBasics\\object.properties";
but what I should suggest you - it's to store your app properties files under your resource folder, kinda:
src/main/resources/config.properties
than you gonna be able to access this file like this:
public Properties extractFrom(String fileName) {
Properties properties = new Properties();
try (InputStream inputStream = PropertiesExtractor.class
.getClassLoader().getResourceAsStream(fileName)) {
properties.load(inputStream);
} catch (IOException ex) {
throw new RuntimeException("Cannot load properties", ex);
}
return properties;
}
extractFrom("config.properties");

Writing with an OutputStream to a DocumentFile: data seem to be written but file ends up empty

The application KDE Connect allows remotely browsing an Android device from a desktop computer through SFTP. Since Android 4.4, developers don't have write permission to SD cards directly through the filesystem anymore. So I am trying to port the SFTP module using the Storage Access Framework (DocumentFile, etc.)
I am taking the permission with an Intent.ACTION_OPEN_DOCUMENT_TREE and FLAG_GRANT_WRITE_URI_PERMISSION and passing the context to my classes.
I am able to create new empty files, rename files and delete files on the SD card inside my class so I believe I am getting the necessary permissions. However, transferring a file results in an empty file (0 bytes) being created. I can see the transfer taking a certain time and a progress bar on the desktop side, so it doesn't just abort.
Here is the relevant part of the SftpSubsystem class from the Apache SSHD library (see doc here) with my own comments to explain what's going on:
public class SftpSubsystem implements Command, Runnable, SessionAware, FileSystemAware {
// This method receives a buffer from an InputStream and processes it
// according to its type. In this situation, it would also contain
// a block of the file being transferred (4096 bytes)
protected void process(Buffer buffer) {
int type = buffer.getByte();
switch (type) {
case WRITE:
FileHandle fh = getHandleFromString(buffer.getString());
long offset = buffer.getLong();
byte[] data = buffer.getBytes();
fh.write(data, offset);
break;
// other cases
}
}
// This class is a handle to a file (duh) with
// an OutputStream to write and InputStream to read
protected static class FileHandle {
SshFile file;
OutputStream output;
long outputPos;
InputStream input;
long inputPos;
// Method called inside process()
public void write(byte[] data, long offset) throws IOException {
if (output != null && offset != outputPos) {
IoUtils.closeQuietly(output);
output = null;
}
if (output == null) {
// This is called once at the start of the transfer.
// This is what I think I need to rewrite to make
// it work with DocumentFile objects.
output = file.createOutputStream(offset);
}
output.write(data);
outputPos += data.length;
}
}
}
The original implementation of createOutputStream() that I want to rewrite because RandomAccessFile doesn't work with DocumentFile:
public class NativeSshFile implements SshFile {
private File file;
public OutputStream createOutputStream(final long offset)
throws IOException {
// permission check
if (!isWritable()) {
throw new IOException("No write permission : " + file.getName());
}
// move to the appropriate offset and create output stream
final RandomAccessFile raf = new RandomAccessFile(file, "rw");
try {
raf.setLength(offset);
raf.seek(offset);
// The IBM jre needs to have both the stream and the random access file
// objects closed to actually close the file
return new FileOutputStream(raf.getFD()) {
public void close() throws IOException {
super.close();
raf.close();
}
};
} catch (IOException e) {
raf.close();
throw e;
}
}
}
One of the ways I tried to implement it:
class SimpleSftpServer {
static class AndroidSshFile extends NativeSshFile {
// This is the DocumentFile that is stored after
// create() created the empty file
private DocumentFile docFile;
public OutputStream createOutputStream(final long offset) throws IOException {
// permission check
if (!isWritable()) {
throw new IOException("No write permission : " + docFile.getName());
}
ParcelFileDescriptor pfd = context.getContentResolver().openFileDescriptor(docFile.getUri(), "rw");
FileDescriptor fd = pfd.getFileDescriptor();
try {
android.system.Os.lseek(fd, offset, OsConstants.SEEK_SET);
} catch (ErrnoException e) {
Log.e("SimpleSftpServer", "" + e);
return null;
}
return new FileOutputstream(fd, offset);
}
}
}
I also tried a simple (the offset is ignored but it's just a test):
public OutputStream createOutputStream(final long offset) throws IOException {
// permission check
if (!isWritable()) {
throw new IOException("No write permission : " + docFile.getName());
}
return context.getContentResolver().openOutputStream(docFile.getUri());
}
I also tried with a FileChannel and to flush and sync the FileOutputStream.
Any idea why I end up with an empty file?
EDIT: here is a small example of a test I did to just write a new file from an existing file. It works, but this is not what I actually want to do (see code above) but I thought I'd provide an example to show that I understand the basics of how to write to an OutputStream.
private void createDocumentFileFromFile() {
File fileToRead = new File("/storage/0123-4567/lady.m4a");
File fileToWrite = new File("/storage/0123-4567/lady2.m4a");
File dir = fileToWrite.getParentFile();
DocumentFile docDir = DocumentFile.fromTreeUri(context, SimpleSftpServer.externalStorageUri);
try {
DocumentFile createdFile = docDir.createFile(null, fileToWrite.getName());
Uri uriToRead = Uri.fromFile(fileToRead);
InputStream in = context.getContentResolver().openInputStream(uriToRead);
OutputStream out = context.getContentResolver().openOutputStream(createdFile.getUri());
try {
int nbOfBytes = 0;
final int BLOCKSIZE = 4096;
byte[] bytesRead = new byte[BLOCKSIZE];
while (true) {
nbOfBytes = in.read(bytesRead);
if (nbOfBytes == -1) {
break;
}
out.write(bytesRead, 0, nbOfBytes);
}
} finally {
in.close();
out.close();
}
} catch (IOException e) {
}
}

"When using ACTION_OPEN_DOCUMENT_TREE, your app gains access only to the files in the directory that the user selects. You don't have access to other apps' files that reside outside this user-selected directory.
This user-controlled access allows users to choose exactly what content they're comfortable sharing with your app."
This means, you can only read/write/delete the content/meta data of already existing files or in sub directories in the selected directory, the scope that the user accept to be "comfortable" with.
Actually the user granted permmision to a list of Uri's in this folder for ea file/sub directory there is seperate uri permmision.
Now for example if I will try to create new file in the selected Uri using DocumentFile Ill success but if i will try to outputatream new data to this file I will fail because the user did not grant permision to write to this newly created file.
He only granted to write in the directory path level, means create new file here.
So same happens when you try to move/transfer file to other path that does not have permission from the user.
Path can be folder or file and for ea new path the user needs to grant new access.
move file = new path
write to just created file = new path

How to use TrueZip to compress files?

I have a file, let's say C:\source.dat. I want to compress it into a zip file C:\folder\destination.zip.
I can't find a straightforward example, and the HelloWorld provided with the Maven project doesn't really apply in my case because I'm not writing a plaintext file, so hopefully someone can enlighten me on this.
For reference, the code provided in the example:
#Override
protected int work(String[] args) throws IOException {
// By default, ZIP files use character set IBM437 to encode entry names
// whereas JAR files use UTF-8.
// This can be changed by configuring the respective archive driver,
// see Javadoc for TApplication.setup().
final Writer writer = new TFileWriter(
new TFile("archive.zip/dir/HälloWörld.txt"));
try {
writer.write("Hello world!\n");
} finally {
writer.close();
}
return 0;
}

It's as simple as this:
new TFile("source.dat").cp(new TFile("destination.zip/source.dat"));
For more information, please refer to the Javadoc for the TFile class at http://truezip.java.net/apidocs/de/schlichtherle/truezip/file/TFile.html .
You may also want to try the TrueZIP Archetype File*, which is introduced at http://truezip.java.net/kick-start/index.html . The archetype generates many sample programs which you should explore to get a feel for the API.

This is a straight forward thing to do.
Use
1.ZipOutputStream -- This Class of java This class implements an output stream filter for writing files in the ZIP file format. Includes support for both compressed and uncompressed entries.
Offical Docs
http://docs.oracle.com/javase/6/docs/api/java/util/zip/ZipOutputStream.html
2.ZipEntry -- This This class is used to represent a ZIP file entry.
http://docs.oracle.com/javase/1.5.0/docs/api/java/util/zip/ZipEntry.html
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
public class ConverToZip {
public static void main(String[] args) {
// Take a buffer
byte[] buffer = new byte[1024];
try {
// Create object of FileOutputStream
FileOutputStream fos = new FileOutputStream("C:\\folder\\destination.zip.");
// Get ZipOutstreamObject Object
ZipOutputStream zos = new ZipOutputStream(fos);
ZipEntry ze = new ZipEntry("source.dat");
zos.putNextEntry(ze);
FileInputStream in = new FileInputStream("C:\\source.dat");
int len;
while ((len = in .read(buffer)) > 0) {
zos.write(buffer, 0, len);
}
in .close();
zos.closeEntry();
//remember close it
zos.close();
System.out.println("Done");
} catch (IOException ex) {
ex.printStackTrace();
}
}
}

Running a .class file from java code

How can I run the compiled code (.class) java from the java code itself?
I'm doing a kind of providing service like compiling and running java code on server side and giving output to the end user.
Can anyone suggest an approach that will accomplish this?
import java.io.*;
public class demo {
public static void main(String args[]) throws IOException, InterruptedException {
int result;
try {
System.out.println("command output:");
Process proc = Runtime.getRuntime().exec("java -cp . demoh");
InputStream in = proc.getInputStream();
result = proc.waitFor();
BufferedInputStream buffer = new BufferedInputStream(proc.getInputStream());
BufferedReader commandOutput = new BufferedReader(new InputStreamReader(buffer));
System.out.print(commandOutput);
String line = null;
try {
while ((line = commandOutput.readLine()) != null) {
System.out.print(line);
System.out.println("command output: " + line);
}//end while
commandOutput.close();
} catch (IOException e) {
//log and/or handle it
}
} catch (IOException e) {
System.err.println("IOException raised: " + e.getMessage());
}
}
}

If you have the .class files somewhere on disk, simply spawn a new process and run the java command like you would from a command line:
Process p = Runtime.getRuntime().exec("java <java class file>");
After some testing around, the following code worked for me:
public static void main(String args[]) throws IOException, InterruptedException {
int result;
try {
System.out.println("command output:");
Process proc = Runtime.getRuntime().exec("java -cp . Test");
InputStream errin = proc.getErrorStream();
InputStream in = proc.getInputStream();
BufferedReader errorOutput = new BufferedReader(new InputStreamReader(errin));
BufferedReader output = new BufferedReader(new InputStreamReader(in));
String line1 = null;
String line2 = null;
try {
while ((line1 = errorOutput.readLine()) != null ||
(line2 = output.readLine()) != null) {
if(line1 != null) System.out.print(line1);
if(line2 != null) System.out.print(line2);
}//end while
errorOutput.close();
output.close();
} catch (IOException e) {
e.printStackTrace();
}//end catc
result = proc.waitFor();
} catch (IOException e) {
System.err.println("IOException raised: " + e.getMessage());
}
}
Note two things here:
The runtime errors given by the java process are sent to the error stream, not input stream, so you have to read both of them!
You have to read the stream as the process is running. Waiting for the process to finish before reading the streams causes a deadlock because the process output buffer is filled and is waiting for your parent process to read the data, while the parent is waiting for the child to finish!

Create .jar file and add that file to the build path

There are still a lot of building/compiling tools, i.e. Ant or Maven, you can check before you write your own.

Try
Process process = Runtime.getRuntime().exec("java " + filePath); // without .class
Scanner output = new Scanner(process.getInputStream());
while (output.hasNext) {
String token = output.next();
...
}

One of the options is to create an instance of the class using the class loader. The class loader can take your class as a byte array and then you can create an instance of it and run it. See this method in the JDK docs.

Here's a sample app that will compile the Java source file, load the class, instantiate an instance, and print out the toString() of the class HelloWorld. I believe you'll need tools.jar on the classpath. The sample code expects the source file in the src folder. The src folder is required on the classpath since the .class file will get generated there by default.
For more control of the Java Compiler, read up on the javax.tools package.
package sourcerunner;
import java.io.IOException;
import java.util.concurrent.TimeUnit;
import javax.tools.JavaCompiler;
import javax.tools.ToolProvider;
public class SourceRunner {
public static void main(String[] args) throws IOException, ClassNotFoundException, InstantiationException, IllegalAccessException, InterruptedException {
JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
compiler.run(System.in, System.out, System.err, "src/sourcerunner/HelloWorld.java");
TimeUnit.SECONDS.sleep(1L);
Class<?> cls = Class.forName("sourcerunner.HelloWorld");
Object instance = cls.newInstance();
System.out.println(instance);
}
}
And here's the HelloWorld class:
package sourcerunner;
public class HelloWorld {
#Override
public String toString() {
return "Hello Java Compiler World.";
}
}
The above code is insanely insecure. Once you understand the code, modify it to use a new ClassLoader to load and instantiate the class. Make sure the ClassLoader has minimal permissions.