We Keep Coding

How to Save Dataframe as Text File using User Defined File Name in Spark Java

I am trying to save a dataframe to a specific location.
successDF.toJavaRDD().saveAsTextFile(successFilePath);
Here, successFilePath is: /hdfs/tmp/20200102/04.dat
I need to save the data with filename as 04.dat, where 20200102 and 04 are coming as arguments
But the process creates multiple files as below:
Folder: /hdfs/tmp/20200102/04.dat
Files:
._SUCCESS.crc
.part-00000.crc
_SUCCESS
part-00000
My requirement is, the output file should be created in /hdfs/tmp/20200102 and there should be only 1 file under the folder with file name as: 04.dat
N.B. I am using Spark Java
Please suggest

You can create file on HDFS without using Spark:
Using HDFS API
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
public class HDFSFileWrite {
public static void main(String[] args) {
Configuration conf = new Configuration();
try {
FileSystem fs = FileSystem.get(conf);
// Hadoop DFS Path - Input & Output file
Path inFile = new Path(args[0]);
Path outFile = new Path(args[1]);
// Verification
if (!fs.exists(inFile)) {
System.out.println("Input file not found");
throw new IOException("Input file not found");
}
if (fs.exists(outFile)) {
System.out.println("Output file already exists");
throw new IOException("Output file already exists");
}
// open and read from file
FSDataInputStream in = fs.open(inFile);
// Create file to write
FSDataOutputStream out = fs.create(outFile);
byte buffer[] = new byte[256];
try {
int bytesRead = 0;
while ((bytesRead = in.read(buffer)) > 0) {
out.write(buffer, 0, bytesRead);
}
} catch (IOException e) {
System.out.println("Error while copying file");
} finally {
in.close();
out.close();
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}

PrinterWriter creates a file but does not write to it. Tried flush and close everywhere. Still, does not write to file

Tried to put flush and close everywhere. Still, does not write to file. Changed file location, path of the file, still does not write to the file; however it creates it.
public void filePatient(HashMap<Integer,Patient> collection, String filename) {
// crating a file
File file = new File (filename+".txt");
try (PrintWriter out = new PrintWriter(new FileWriter(file),true)) {
for(Patient i: collection.values()) {
out.write(i.getName());
//out.write(i.getHealthNumber());
}
out.flush();
out.close();
}
catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}

Your code works as-is.
You can also remove the close, as you use try-with-resources.
Here's a full working example:
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.PrintWriter;
import java.util.HashMap;
import java.util.Map;
public class Main {
public static void main(String[] args) {
Map<String, String> map = new HashMap<>();
map.put("a", "my name");
map.put("b", "your name");
filePatient(map, "c:\\temp\\test");
}
public static void filePatient(Map<String, String> collection, String filename) {
// crating a file
File file = new File(filename + ".txt");
try (PrintWriter out = new PrintWriter(new FileWriter(file), true)) {
for (String name : collection.values()) {
out.write(name);
}
out.flush();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}

Convert Blob to JPG and update blob

I'm trying to read in a blob, convert it to JPG and then write back to the blob (it is being passed in by reference, but when trying to compile in TOAD I get an error on ImageIO.write.
CREATE OR REPLACE AND RESOLVE JAVA SOURCE NAMED BANNADMIN.IMAGE_CONVERTER
AS package uk.co.ImageUtil;
import javax.imageio.ImageIO;
import java.io.File;
import java.io.IOException;
import java.awt.image.BufferedImage;
import oracle.sql.*;
import java.io.OutputStream;
public class ImageConverter {
public static void convertImage(BLOB[] blob) {
BufferedImage image = null;
OutputStream outputStream = null;
try {
image = ImageIO.read(blob[0].getBinaryStream());
outputStream = blob[0].setBinaryStream(0);
ImageIO.write(image, "JPG", outputStream);
} catch (IOException e) {
e.printStackTrace();
}
catch (SQLException e) {
e.printStackTrace();
}
catch(IllegalArgumentException e) {
e.printStackTrace();
}
finally {
try {
if (outputStream !== null) {
outputStream.flush();
outputStream.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
/
How would I convert a BufferedImage into a RenderedImage so I can write the JPG version back into the Blob?
Update: The error message is
[Error] (1: 0): IMAGE_CONVERTER:28: cannot find symbol
[Error] (1: 0): symbol : method write(java.awt.image.BufferedImage,java.lang.String,java.lang.Object)
[Error] (1: 0): location: class javax.imageio.ImageIO
[Error] (1: 0): ImageIO.write(image, "jpg", outputStream);
[Error] (1: 0): ^
[Error] (1: 0): 1 error

Turned out it was a simple mistake, ImageIO.write takes in a RenderedImage which meant I had to cast the BufferedImage to RenderedImage, and I had written !== instead of != in the finally block. See below for what compiles successfully
CREATE OR REPLACE AND RESOLVE JAVA SOURCE NAMED BANNADMIN.IMAGE_CONVERTER AS package uk.co.ImageUtil;
import javax.imageio.ImageIO;
import java.io.File;
import java.io.IOException;
import java.awt.image.BufferedImage;
import java.awt.image.RenderedImage;
import oracle.sql.*;
import java.io.OutputStream;
import java.sql.SQLException;
public class ImageConverter {
/**
* Take in a BLOB file (specified as an array parameter but we only ever use [0])
* Read in the binary stream of the BLOB
* Change the binary stream to jpg
* Write the binary stream jpg to the BLOB
* The BLOB parameter is passed in via out - so there is no need to return the BLOB, only edit it
*/
public static void convertImage(BLOB[] blob) {
BufferedImage bufferedImage = null;
OutputStream outputStream = null;
try {
bufferedImage = ImageIO.read(blob[0].getBinaryStream());
outputStream = blob[0].setBinaryStream(0);
RenderedImage renderedImage = (RenderedImage)bufferedImage;
ImageIO.write(renderedImage, "JPG", outputStream);
} catch (IOException e) {
e.printStackTrace();
}
catch (SQLException e) {
e.printStackTrace();
}
catch(IllegalArgumentException e) {
e.printStackTrace();
}
finally {
try {
if (outputStream != null) {
outputStream.flush();
outputStream.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
/

Java : Problems accessing and writing to file

I was testing out writing to files with this code:
package files;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileWriter;
import java.io.IOException;
public class FileTest1
{
public static void main(String[] args)
{
try
{
try
{
File f = new File("filetest1.txt");
FileWriter fWrite = new FileWriter(f);
BufferedWriter fileWrite = new BufferedWriter(fWrite);
fileWrite.write("This is a test!");
}
catch(FileNotFoundException e)
{
System.out.print("A FileNotFoundException occurred!");
e.printStackTrace();
}
}
catch(IOException e)
{
System.out.println("An IOException occurred!:");
e.printStackTrace();
}
}
}
Nothing happens when it is executed.
"This is a test!" is not written, nor the StackTrace or the "A/An [exception] occurred!"...
I don't know what's causing the problem. I have fileTest1.txt in the package right under the file...

A BufferedWriter does just that, it buffers the output before it is written to the destination. This can make the BufferedWriter faster to use as it doesn't have to write to a slow destination, like a disk or socket, straight away.
The contents will be written when the internal buffer is to full, you flush the Writer or close the writer
Remember, if you open it, you should close it...
For example...
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
public class TestFileWriter {
public static void main(String[] args) {
try {
BufferedWriter fileWrite = null;
try {
File f = new File("filetest1.txt");
System.out.println("Writing to " + f.getCanonicalPath());
FileWriter fWrite = new FileWriter(f);
fileWrite = new BufferedWriter(fWrite);
fileWrite.write("This is a test!");
fileWrite.flush();
} catch (FileNotFoundException e) {
System.out.print("A FileNotFoundException occurred!");
e.printStackTrace();
} finally {
try {
// Note, BufferedWriter#close will also close
// the parent Writer...
fileWrite.close();
} catch (Exception exp) {
}
}
} catch (IOException e) {
System.out.println("An IOException occurred!:");
e.printStackTrace();
}
try {
BufferedReader br = null;
try {
File f = new File("filetest1.txt");
System.out.println("Reading from " + f.getCanonicalPath());
FileReader fReader = new FileReader(f);
br = new BufferedReader(fReader);
String text = null;
while ((text = br.readLine()) != null) {
System.out.println(text);
}
} catch (FileNotFoundException e) {
System.out.print("A FileNotFoundException occurred!");
e.printStackTrace();
} finally {
try {
// Note, BufferedWriter#close will also close
// the parent Writer...
br.close();
} catch (Exception exp) {
}
}
} catch (IOException e) {
System.out.println("An IOException occurred!:");
e.printStackTrace();
}
}
}
If you are using Java 7, you may like to take a look at try-with-resources

After
fileWrite.write("This is a test!");
you have to flush() the writer. To avoid leaking of resources you should also close() the writer (which automatically flushes it).
So you need to add:
fileWrite.close();

Use BufferedWriter.flush() and BufferedWriter.close(). Additional info here http://docs.oracle.com/javase/7/docs/api/java/io/BufferedWriter.html

You must call close() or at least flush() on the writer in order for the buffer to be really written to the file.

how to open .mdb from ftp location jackcess

hi all with this code i can successfully download allpg.mdb and displaying...
now i want to save the downloaded file to c:/folder....
if i edit
dbTempFile = File.createTempFile("dbTempFile",".mdb"); to
dbTempFile = File.createTempFile("c:/dbTempFile",".mdb"); than it give : The filename, directory name, or volume label syntax is incorrect error.
i just want to save the downloaded file to any where to my local drive.
here is code:
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import org.apache.commons.net.ftp.FTPClient;
import com.healthmarketscience.jackcess.Database;
import com.healthmarketscience.jackcess.Table;
public class DownloadFile {
public static void main(String[] args) throws Exception {
FTPClient client = new FTPClient();
File dbTempFile=null;
FileOutputStream fileOutputStream = null;
try {
client.connect("ftp.mypak.com");
client.login("myid", "mypwd");
client.setFileType(FTPClient.BINARY_FILE_TYPE);
dbTempFile = File.createTempFile("dbTempFile",".mdb");
fileOutputStream = new FileOutputStream(dbTempFile);
client.retrieveFile("/HASSAN/MDMSTATS/allpg.mdb", fileOutputStream);
} catch (IOException ex) {
ex.printStackTrace();
} finally {
try {
if (fileOutputStream != null) {
fileOutputStream.close();
System.out.println("got");
Table table = Database.open(dbTempFile).getTable("items");
System.out.println(table.display());
System.out.println("got");
}
client.disconnect();
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
}

You are not giving the right file name to the Jackcess constructor. should be:
Table table = Database.open(dbTempFile).getTable("items");