We Keep Coding

how to write dataframe into csv file into minio bucket

I am working on a project. in which, i want to write spark dataframe data to a CSV file in MinIO bucket.
I have searched everywhere but didn't get any proper solution.
Please help me to achieve this.
I have tried many solutions but not worked

You can use below code:
String bucketName="my_bucket";
String filePath="files/my_file.csv";
List<Row> rowList = new ArrayList<>();
rowList = dataFrame.collectAsList();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
String[] fieldNames = rowList.get(0).schema().fieldNames();
String headers = String.join(delimiter, fieldNames) + "\n";
baos.write(headers.getBytes());
rowList.stream().forEach(row -> {
String[] arr = new String[row.length()];
for (int i = 0; i < row.length(); i++) {
arr[i] = convertAllDataTypeToString(row.getAs(i)); //write one method to convert different data type to string
}
String str = String.join(delimiter, arr) + "\n";
try {
baos.write(str.getBytes());
} catch (IOException e) {
e.printStackTrace();
}
});
ByteArrayInputStream bais = new ByteArrayInputStream(baos.toByteArray());
MinioClient minioClient = // get your minio client here
minioClient.putObject(PutObjectArgs.builder().bucket(bucketName).object(filePath).stream(bais, bais.available(), -1).build());
bais.close();
baos.close();

Convert base64 string to Image in Java

I have an image being sent to me through a JSON string. I want to convert that string into an image in my android app and then display that image.
The JSON string looks like this:
"data:image\/png;base64,iVBORw0KGgoAAAANSUhEUgAAAVI..."
Note: I truncated the string with a ...
I've got a function that (I think) converts the string into an image. Am I doing this right?
public Bitmap ConvertToImage(String image){
try{
InputStream stream = new ByteArrayInputStream(image.getBytes());
Bitmap bitmap = BitmapFactory.decodeStream(stream);
return bitmap;
}
catch (Exception e) {
return null;
}
}
Then I try to display it on my android activity like this
String image = jsonObject.getString("barcode_img");
Bitmap myBitmap = this.ConvertToImage(image);
ImageView cimg = (ImageView)findViewById(R.id.imageView1);
//Now try setting dynamic image
cimg.setImageBitmap(myBitmap);
However, when I do this, nothing shows up. I don't get any errors in the logcat. What am I doing wrong?
Thanks

I'm worried about that you need to decode only the base64 string to get the image bytes, so in your
"data:image\/png;base64,iVBORw0KGgoAAAANSUhEUgAAAVI..."
string, you must get the data after data:image\/png;base64,, so you get only the image bytes and then decode them:
String imageDataBytes = completeImageData.substring(completeImageData.indexOf(",")+1);
InputStream stream = new ByteArrayInputStream(Base64.decode(imageDataBytes.getBytes(), Base64.DEFAULT));
This is a code so you understand how it works, but if you receive a JSON object it should be done the correct way:
Converting the JSON string to a JSON object.
Extract the String under data key.
Make sure that starts with image/png so you know is a png image.
Make sure that contains base64 string, so you know that data must be decoded.
Decode the data after base64 string to get the image.

InputStream stream = new ByteArrayInputStream(image.getBytes());
should be changed to
InputStream stream = new ByteArrayInputStream(Base64.decode(image.getBytes(), Base64.DEFAULT));
Refer http://developer.android.com/reference/android/util/Base64.html for details on how to do Base64 decoding.
Disclaimer: I have not checked for syntax, but this is how you should do it.

Here is the working code that converts the Base64 encoded inputStream and writes it to the disk.
I spent a few time to make it work correctly. So I hope this helps other developers.
public boolean writeImageToDisk(FileItem item, File imageFile) {
// clear error message
errorMessage = null;
FileOutputStream out = null;
boolean ret = false;
try {
// write thumbnail to output folder
out = createOutputStream(imageFile);
// Copy input stream to output stream
byte[] headerBytes = new byte[22];
InputStream imageStream = item.getInputStream();
imageStream.read(headerBytes);
String header = new String(headerBytes);
// System.out.println(header);
byte[] b = new byte[4 * 1024];
byte[] decoded;
int read = 0;
while ((read = imageStream.read(b)) != -1) {
// System.out.println();
if (Base64.isArrayByteBase64(b)) {
decoded = Base64.decodeBase64(b);
out.write(decoded);
}
}
ret = true;
} catch (IOException e) {
StringWriter sw = new StringWriter();
e.printStackTrace(new PrintWriter(sw));
errorMessage = "error: " + sw;
} finally {
if (out != null) {
try {
out.close();
} catch (Exception e) {
StringWriter sw = new StringWriter();
e.printStackTrace(new PrintWriter(sw));
System.out.println("Cannot close outputStream after writing file to disk!" + sw.toString());
}
}
}
return ret;
}
/**
* Helper method for the creation of a file output stream.
*
* #param imageFolder
* : folder where images are to be saved.
* #param id
* : id of spcefic image file.
* #return FileOutputStream object prepared to store images.
* #throws FileNotFoundException
*/
protected FileOutputStream createOutputStream(File imageFile) throws FileNotFoundException {
imageFile.getParentFile().mkdirs();
return new FileOutputStream(imageFile);
}

DES encrypt/decrypt from a file

I am writting a program where I take a string, encrypt it and then write it in a file. Then later, I read from the file the string, decrypt it and then modify it. Here's my code for DES encryption/decryption:
/* class for crypting and decrypting a file */
class DESEncrypter
{
private Cipher encryptionCipher;
private Cipher decryptionCipher;
public DESEncrypter (SecretKey key) throws Exception
{
encryptionCipher = Cipher.getInstance("DES/ECB/PKCS5Padding");
encryptionCipher.init(Cipher.ENCRYPT_MODE, key);
decryptionCipher = Cipher.getInstance("DES/ECB/PKCS5Padding");
decryptionCipher.init(Cipher.DECRYPT_MODE, key);
}
/* write to 'out' the encryption of the information read from 'in' */
public String encrypt(String unencryptedString)
{
String encryptedString = "";
try {
byte[] unencryptedByteArray = unencryptedString.getBytes("UTF8");
byte[] encryptedBytes = this.encryptionCipher.doFinal(unencryptedByteArray);
encryptedString = new sun.misc.BASE64Encoder().encode(encryptedBytes);
} catch (Exception ex) {
Logger.getLogger(DESEncrypter.class.getName()).log(Level.SEVERE, null, ex);
}
return encryptedString;
}
private static String bytes2String(byte[] bytes)
{
StringBuffer stringBuffer = new StringBuffer();
for (int i = 0; i < bytes.length; i++)
{
stringBuffer.append((char) bytes[i]);
}
return stringBuffer.toString();
}
/* write to 'out' the information obtained by decrypting the information read from 'in' */
public String decrypt (String encryptedString) throws UnsupportedEncodingException
{
byte[] unencryptedByteArray = new byte[4096];
try {
// Encode bytes to base64 to get a string
byte[] decodedBytes = new sun.misc.BASE64Decoder().decodeBuffer(encryptedString);
// Decrypt
unencryptedByteArray = this.decryptionCipher.doFinal(decodedBytes);
} catch (Exception ex) {
Logger.getLogger(DESEncrypter.class.getName()).log(Level.SEVERE, null, ex);
}
return bytes2String(unencryptedByteArray);
}
}
And this is the function where I write a encrypted String in a file:
public void writeToFileEncrypted(String filename, String owner, String departament)
{
try
{
BufferedReader br = new BufferedReader(new FileReader(new File("files_encrypted")));
String crypt = "";
String aux;
while ((aux = br.readLine()) != null)
{
crypt += aux;
}
br.close();
String info = this.server.crypt.decrypt(crypt);
info += filename + " " + owner + " " + departament + "\n";
/* delete the old encryption */
File temp = new File("files_encrypted");
temp.delete();
String infoCrypt = this.server.crypt.encrypt(info);
File newFiles = new File("files_encrypted");
if (newFiles.createNewFile() == false)
{
log.severe("Failed to re-create the 'files_encrypted' file when trying to add a new file");
return;
}
BufferedWriter bw = new BufferedWriter(new FileWriter(newFiles));
bw.write(infoCrypt);
bw.close();
}
catch (Exception e)
{
log.warning("An exception was caught while trying to remove '" + clientName + "' from the banned list");
e.printStackTrace();
return;
}
}
While the server runs, I can make modification to that String from file(run that function many time). The problem is when I close the server and then I open it again because I get the error:
javax.crypto.BadPaddingException: Given final block not properly padded
This is how I read from file when the server opens:
BufferedReader br = new BufferedReader(new FileReader(new File("files_encrypted")));
String crypto = new String();
String aux;
while ((aux = br.readLine()) != null)
{
crypto += aux;
readBytes++;
}
br.close();
System.out.println(readBytes);
info = this.crypt.decrypt(crypto);
Why do I get that error? What I'm doing wrong? I must write the encrypted String in file some other way?
LATER EDIT:
I've changed the function that read a String from a file, decrypt it, modify it , encrypt it and then write it in file.
public void writeToFileEncrypted(String filename, String owner, String departament)
{
try
{
File f = new File("files_encrypted");
int nrRead = 0;
String info = null;
FileInputStream fis = new FileInputStream(f);
StringBuffer sb = new StringBuffer();
int ch;
while ((ch = fis.read()) != -1)
{
sb.append((char)ch);
nrRead++;
}
fis.close();
StringBuilder sba = null;
if (nrRead != 0)
{
info = this.server.crypt.decrypt(new String(sb.toString().getBytes("UTF-8"), "UTF-8"));
sba = new StringBuilder(info);
sba.append(filename + " " + owner + " " + departament + " ");
}
else
{
sba = new StringBuilder(filename + " " + owner + " " + departament + " ");
}
/* delete the old encryption */
File temp = new File("files_encrypted");
temp.delete();
//System.out.println("before: " + sba.toString());
String infoCrypt = this.server.crypt.encrypt(sba.toString());
//System.out.println("after: " + infoCrypt);
File newFiles = new File("files_encrypted");
if (newFiles.createNewFile() == false)
{
log.severe("Failed to re-create the 'files_encrypted' file when trying to add a new file");
return;
}
FileOutputStream fos = new FileOutputStream(newFiles);
fos.write(infoCrypt.getBytes("UTF-8"));
fos.flush();
fos.close();
}
catch (Exception e)
{
log.warning("An exception was caught while trying to remove '" + clientName + "' from the banned list");
e.printStackTrace();
return;
}
}
I've also modified where I read the info from file when server opens for the first time:
FileInputStream fis = new FileInputStream(f);
StringBuffer sb = new StringBuffer();
int ch;
while ((ch = fis.read()) != -1)
{
sb.append((char)ch);
readBytes++;
}
fis.close();
if (readBytes != 0)
{
System.out.println("on: " + sb.toString());
info = this.crypt.decrypt(new String(sb.toString().getBytes("UTF-8"), "UTF-8"));
System.out.println("load: " + info);
}
}
At the System.out.println with "on: " what I read from file is exactly what I've written encrypted, without any spaces or new lines. If I read with read(buffer), where buffer is byte[], it seems that adds a lot of spaces.
Although I've made all this modifications I still get the error javax.crypto.BadPaddingException: Given final block not properly padded
Has somebody any idea what's going on here?

There are a few things here.
private static String bytes2String(byte[] bytes)
Is dodgy, you are casting a byte to a char in this method so there is no character encoding specified here. To convert bytes to characters you should just use the String constructor that takes an array of bytes and an encoding. e.g.
byte[] tmp = new byte[10];
String a = new String(tmp, "UTF-8");
Be careful using BufferedReaders + .readLine() - this will strip out any newline characters from your file as you read it unless you add them back into your buffer. Although I don't think this is your problem.
But I think the best way to simplify your code is to write the encoded bytes via an OutputStream directly to the file. Unless you need to send the contents of the file over a transport that doesn't like binary data, there is no need to base64 encode. Just use Input/OutputStreams to write the encrypted bytes direct to disk.
RESPONSE TO LATER EDIT:
You are still mixing up your use of binary data (bytes) and character data (String/chars). You can't do things like:
int ch;
while ((ch = fis.read()) != -1)
{
sb.append((char)ch);
The input stream is retuning bytes, a byte is not a character and just casting it to one is going to cause problems. When using encryption the output from the encryption operation is binary data, and the input to the decryption operation is also binary data. The fact that your are encrypting text is something you deal with before the encryption occurs, and after the decryption occurs. You basic operation should go along the following lines.
Take the text you want to encrypt and convert it to bytes, specifying an encoding using the .getBytes(String charsetName) on your String.
Pass these bytes into your encryption routine
Write the resulting bytes directly to disk
To decrypt:
Read the bytes from the file
Pass the bytes to your decryption routine (as bytes! no Strings/ text involved)
Take the out put bytes and re-construct you String using new String(byte[] bytes, String charsetName) specifying the same encoding as before.
You might find the following (untested, but should work) methods useful:
public byte[] readBinaryFile(File f) throws IOException
{
byte[] contents = new byte[(int)f.length()];
BufferedInputStream bis = null;
try
{
bis = new BufferedInputStream(new FileInputStream(f));
DataInputStream dis = new DataInputStream(bis);
dis.readFully(contents);
}
finally
{
if(bis != null)
{
bis.close();
}
}
return contents;
}
public void writeBinaryFile(byte[] contents, File f) throws IOException
{
BufferedOutputStream bos = null;
try
{
bos = new BufferedOutputStream(new FileOutputStream(f));
bos.write(contents);
}
finally
{
if(bos != null)
{
bos.close();
}
}
}
So you will also need to change the interface, and internals of your encrypt and decrypt methods so they take and return byte arrays, and ditch the base64 encoding.

You have several problems. The reading and decrypting process should be symmetric with the encrypting and writing process. But
you transform your String into a byte[] using getBytes("UTF8"), which is fine, but you don't use new String(byte[], "UTF8") to do the reverse operation.
you write a whole String to a file, including potential line breaks, but you read it line by line and concatenate each line, thus losing the line breaks in the process. You must read each and every char that has been written.
Also, relying on undocumented, unsupported classes like sun.misc.Base64Encoder/Decoder shouldn't be done. Use Apache commons-codec to find a documented Base64 encoding, guaranteed to still be there when the next JDK comes out, and which can be used on every JVM, including non-Sun JVMs.

I think it is in the initialization
SecureRandom sr = new SecureRandom();
cipher.init( Cipher.DECRYPT_MODE, desKey ,sr);

Not sure this is the primary problem, but when you return the decrypted String from decrypt(), you should be using:
return new String(unencryptedByteArray, "UTF-8");

Flex File reading problem

i am having a problem in reading a file from Flex. The file contains a base64encoded string. when i read the file i get the length as 47856 and the decoded base64 byte array length as 34157.
When i read the same File from java i get the length as 48068 and 35733 respectively.
What is the problem?
private function init():void{
var file:File = File.desktopDirectory.resolvePath("Files/sample.txt");
stream = new FileStream();
stream.open(file, FileMode.READ);
var str:String = stream.readUTFBytes(stream.bytesAvailable);
stream.close();
str = str.replace(File.lineEnding, "\n");
contents.text = str;
fileName.text = file.name;
}
public function playSound(contents:String):void{
try{
var byteData: ByteArray;
byteData = new ByteArray();
byteData.writeUTFBytes(contents);
var dec:Base64Decoder = new Base64Decoder();
dec.decode(contents);
byteData = dec.toByteArray();
Alert.show("byte Array " + byteData.toString().length +" :: " +contents.length);
}
And this is my java code for reading the file...Whatever result i am expecting is achieved in the java side.
private static String readFile(String path) throws IOException {
FileInputStream stream = new FileInputStream(new File(path));
try {
FileChannel fc = stream.getChannel();
MappedByteBuffer bb = fc.map(FileChannel.MapMode.READ_ONLY, 0, fc.size());
return Charset.defaultCharset().decode(bb).toString(); }
finally { stream.close();
}
}
Java Code where i am printing the length
byte[] decodedBase64 = new byte[byteLength];
String speexData = null;
try {
speexData = readFile(userDir +"//" +xmlFileName);
} catch (IOException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
// System.out.println("sa " + sa);
try{
decodedBase64= Base64.decodeToByteArray(speexData);
System.out.println("decodednase64 length " + decodedBase64.length +" :: " +speexData.length());
}
catch(Exception e){
}

You would have to post your java code to show what you're doing there, as well.
However, without knowing more, I could take a guess and say that when you replace the line ending, you may be removing a byte each time (if it was \r\n and you're making it \n, for example).

Download attachments using Java Mail

Now that I`ve downloaded all the messages, and store them to
Message[] temp;
How do I get the list of attachments for each of those messages to
List<File> attachments;
Note: no thirdparty libs, please, just JavaMail.

Without exception handling, but here goes:
List<File> attachments = new ArrayList<File>();
for (Message message : temp) {
Multipart multipart = (Multipart) message.getContent();
for (int i = 0; i < multipart.getCount(); i++) {
BodyPart bodyPart = multipart.getBodyPart(i);
if(!Part.ATTACHMENT.equalsIgnoreCase(bodyPart.getDisposition()) &&
StringUtils.isBlank(bodyPart.getFileName())) {
continue; // dealing with attachments only
}
InputStream is = bodyPart.getInputStream();
// -- EDIT -- SECURITY ISSUE --
// do not do this in production code -- a malicious email can easily contain this filename: "../etc/passwd", or any other path: They can overwrite _ANY_ file on the system that this code has write access to!
// File f = new File("/tmp/" + bodyPart.getFileName());
FileOutputStream fos = new FileOutputStream(f);
byte[] buf = new byte[4096];
int bytesRead;
while((bytesRead = is.read(buf))!=-1) {
fos.write(buf, 0, bytesRead);
}
fos.close();
attachments.add(f);
}
}

Question is very old, but maybe it will help someone. I would like to expand David Rabinowitz`s answer.
if(!Part.ATTACHMENT.equalsIgnoreCase(bodyPart.getDisposition()))
should not return all atachments as you expect, because you can have mail where mixed part is without defined disposition.
----boundary_328630_1e15ac03-e817-4763-af99-d4b23cfdb600
Content-Type: application/octet-stream;
name="00000000009661222736_236225959_20130731-7.txt"
Content-Transfer-Encoding: base64
so in this case, you can also check for filename. Like this:
if (!Part.ATTACHMENT.equalsIgnoreCase(part.getDisposition()) && StringUtils.isBlank(part.getFileName())) {...}
EDIT
there is whole working code using condition descibed above.. Because each part can encapsulate another parts and attachment should be nested in, recursion is used to traverse through all parts
public List<InputStream> getAttachments(Message message) throws Exception {
Object content = message.getContent();
if (content instanceof String)
return null;
if (content instanceof Multipart) {
Multipart multipart = (Multipart) content;
List<InputStream> result = new ArrayList<InputStream>();
for (int i = 0; i < multipart.getCount(); i++) {
result.addAll(getAttachments(multipart.getBodyPart(i)));
}
return result;
}
return null;
}
private List<InputStream> getAttachments(BodyPart part) throws Exception {
List<InputStream> result = new ArrayList<InputStream>();
Object content = part.getContent();
if (content instanceof InputStream || content instanceof String) {
if (Part.ATTACHMENT.equalsIgnoreCase(part.getDisposition()) || StringUtils.isNotBlank(part.getFileName())) {
result.add(part.getInputStream());
return result;
} else {
return new ArrayList<InputStream>();
}
}
if (content instanceof Multipart) {
Multipart multipart = (Multipart) content;
for (int i = 0; i < multipart.getCount(); i++) {
BodyPart bodyPart = multipart.getBodyPart(i);
result.addAll(getAttachments(bodyPart));
}
}
return result;
}

Some time saver for the code where you save the attachment file :
with javax mail version 1.4 and after , you can say
// SECURITY LEAK - do not do this! Do not trust the 'getFileName' input. Imagine it is: "../etc/passwd", for example.
// bodyPart.saveFile("/tmp/" + bodyPart.getFileName());
instead of
InputStream is = bodyPart.getInputStream();
File f = new File("/tmp/" + bodyPart.getFileName());
FileOutputStream fos = new FileOutputStream(f);
byte[] buf = new byte[4096];
int bytesRead;
while((bytesRead = is.read(buf))!=-1) {
fos.write(buf, 0, bytesRead);
}
fos.close();

You can simply use Apache Commons Mail API MimeMessageParser - getAttachmentList() along Commons IO and Commons Lang.
MimeMessageParser parser = ....
parser.parse();
for(DataSource dataSource : parser.getAttachmentList()) {
if (StringUtils.isNotBlank(dataSource.getName())) {}
//use apache commons IOUtils to save attachments
IOUtils.copy(dataSource.getInputStream(), ..dataSource.getName()...)
} else {
//handle how you would want attachments without file names
//ex. mails within emails have no file name
}
}

Returns list of body parts with attachments.
#Throws(Exception::class)
fun getAttachments(message: Message): List<BodyPart>{
val content = message.content
if (content is String) return ArrayList<BodyPart>()
if (content is Multipart) {
val result: MutableList<BodyPart> = ArrayList<BodyPart>()
for (i in 0 until content.count) {
result.addAll(getAttachments(content.getBodyPart(i)))
}
return result
}
return ArrayList<BodyPart>()
}
#Throws(Exception::class)
private fun getAttachments(part: BodyPart): List<BodyPart> {
val result: MutableList<BodyPart> = ArrayList<BodyPart>()
if (Part.ATTACHMENT == part.disposition && !part.fileName.isNullOrBlank()){
result.add(part)
}
val content = part.content
if (content is Multipart) {
for (i in 0 until (content ).count) {
val bodyPart = content.getBodyPart(i)
result.addAll(getAttachments(bodyPart)!!)
}
}
return result
}