I am working on a project. in which, i want to write spark dataframe data to a CSV file in MinIO bucket.
I have searched everywhere but didn't get any proper solution.
Please help me to achieve this.
I have tried many solutions but not worked
You can use below code:
String bucketName="my_bucket";
String filePath="files/my_file.csv";
List<Row> rowList = new ArrayList<>();
rowList = dataFrame.collectAsList();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
String[] fieldNames = rowList.get(0).schema().fieldNames();
String headers = String.join(delimiter, fieldNames) + "\n";
baos.write(headers.getBytes());
rowList.stream().forEach(row -> {
String[] arr = new String[row.length()];
for (int i = 0; i < row.length(); i++) {
arr[i] = convertAllDataTypeToString(row.getAs(i)); //write one method to convert different data type to string
}
String str = String.join(delimiter, arr) + "\n";
try {
baos.write(str.getBytes());
} catch (IOException e) {
e.printStackTrace();
}
});
ByteArrayInputStream bais = new ByteArrayInputStream(baos.toByteArray());
MinioClient minioClient = // get your minio client here
minioClient.putObject(PutObjectArgs.builder().bucket(bucketName).object(filePath).stream(bais, bais.available(), -1).build());
bais.close();
baos.close();
Related
I am trying to transfer a .mp4 file using WebRTC and it's DataChannel. In order to do that I am breaking the file into chunks like below:
FileInputStream is = new FileInputStream(file);
byte[] chunk = new byte[260000];
int chunkLen = 0;
sentFileByte = new ArrayList<>();
while ((chunkLen = is.read(chunk)) != -1) {
sentFileByte.add(chunk);
}
After that, sending the chunks by index like:
byte[] b = sentFileByte.get(index);
ByteBuffer bb = ByteBuffer.wrap(b);
bb.put(b);
bb.flip();
dataChannel.send(new DataChannel.Buffer(bb, true));
On the receiver end I am receiving the chunks and adding it to an Arraylist
receivedFileByteArr.add(chunkByteArr);
After receiving all the chunks successfully I am trying to convert these in to a file like below:
String path = Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOWNLOADS).getAbsolutePath() + "/" + fileName;
File file = new File(path);
try {
FileOutputStream fileOutputStream = new FileOutputStream(file);
for (int i = 0; i < receivedFileByteArr.size(); i++) {
fileOutputStream.write(receivedFileByteArr.get(i));
}
fileOutputStream.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
After completing all these steps, file is created successfully. File size is also same. But problem is the file is not playable in any video player. I guess I am making some mistake on FileInputStream and FileOutputStream. I need help to fix this error.
I'm converting Json to avro. I have json data in JSONArray. So while converting it into byte array i'm facing the problem.
below is my code:
static byte [] fromJsonToAvro(JSONArray json, String schemastr) throws Exception {
ExcelToJson ejj = new ExcelToJson();
List<String> list = new ArrayList<String>();
if (json != null) {
int len = json.length();
for (int i=0;i<len;i++){
list.add(json.get(i).toString());
}
}
InputStream input = new ByteArrayInputStream(list.getBytes()); //json.toString().getBytes()
DataInputStream din = new DataInputStream(input);
.
.
.//rest of the logic
So how can i do it? How to convert JsonArray object to bytes(i.e., how to use getBytes() method for JsonArray objects). The above code giving an error at list.getBytes() and saying getBytes() is undifined for list.
Avro works at the record level, bound to a schema. I don't think there's such a concept as "convert this JSON fragment to bytes for an Avro field independent of any schema or record".
Assuming the array is part of a larger JSON record, if you're starting with a string of the record, you could do
public static byte[] jsonToAvro(String json, String schemaStr) throws IOException {
InputStream input = null;
DataFileWriter<GenericRecord> writer = null;
Encoder encoder = null;
ByteArrayOutputStream output = null;
try {
Schema schema = new Schema.Parser().parse(schemaStr);
DatumReader<GenericRecord> reader = new GenericDatumReader<GenericRecord>(schema);
input = new ByteArrayInputStream(json.getBytes());
output = new ByteArrayOutputStream();
DataInputStream din = new DataInputStream(input);
writer = new DataFileWriter<GenericRecord>(new GenericDatumWriter<GenericRecord>());
writer.create(schema, output);
Decoder decoder = DecoderFactory.get().jsonDecoder(schema, din);
GenericRecord datum;
while (true) {
try {
datum = reader.read(null, decoder);
} catch (EOFException eofe) {
break;
}
writer.append(datum);
}
writer.flush();
return output.toByteArray();
} finally {
try { input.close(); } catch (Exception e) { }
}
}
For an on-line json to avro converter check the following URL
http://avro4s-ui.landoop.com
It is using the library avro4s that offers a lot of conversions including json=>avro
This discussion is likely useful:
http://mail-archives.apache.org/mod_mbox/avro-user/201209.mbox/%3CCALEq1Z8s1sfaAVB7YE2rpZ=v3q1V_h7Vm39h0HsOzxJ+qfQRSg#mail.gmail.com%3E
The gist is that there is a special Json schema and you can use JsonReader/Writer to get to and from that. The Json schema you should use is defined here:
https://github.com/apache/avro/blob/trunk/share/schemas/org/apache/avro/data/Json.avsc
This question already has answers here:
How can I concatenate two arrays in Java?
(66 answers)
Closed 9 years ago.
I'm trying to list files from two different directories in an Android device in order to upload them to a server.
To do that, I'm getting the directory files under a File type and then throwing both listFiles() into correspondents File[] types.
String d1 = Environment.getExternalStorageDirectory().getPath()+"/Download/";
String d2 = Environment.getExternalStorageDirectory().getPath()+"/DCIM/Camera";
File DirDocuments = new File(d1);
File DirCamera = new File(d2);
File[] ListDocuments = DirDocuments.listFiles();
File[] ListCamera = DirCamera.listFiles();
I've tryed to merge then and uploadign using just a single "for" loop
List<String> listTemp = new ArrayList<String>();
for( int i=0; i< ListaDocuments.length; i++)
{
listTemp.add( ListaDocuments[i].getName() );
}
for( int i=0; i< ListaCamera.length; i++)
{
listTemp.add( ListaCamera[i].getName() );
}
File[] DirTotal = (String[]) listTemp.toArray(); //toArray gives Object[] and cannot be converted to File[], so I used String[]
But my app just stop and exits. What am I doing wrong?
Do anyone knows how can I merge ListDocuments and ListCamera into a single File[]?
Thanks in advance. (sorry about my typos and bad english)
You could join both arrays' contents into a List<File> and not a List<String>:
List<File> listTemp = new ArrayList<File>();
for( int i=0; i< ListaDocuments.length; i++) {
listTemp.add( ListaDocuments[i] );
}
for( int i=0; i< ListaCamera.length; i++)
{
listTemp.add( ListaCamera[i] );
}
Notice there are other ways of joining those arrays (i.e. Apache Commons Collections ArrayUtils.addAll(T[], T...) as explained in How to concatenate two arrays in Java? and numerous duplicates).
And then just use the other version of toArray():
File[] DirTotal = listTemp.toArray(new File[0]);
toArray() without arguments will always return an Object[]. The version that takes an argument will fill up the passed array if it's big enough, and return a newly allocated array in case there's not enough room. The returned array is of the same type than the passed one, so if you pass a File[], you get a File[].
The problem is that the last line is casting to String array but the variable is defined as a File array.
public static void mergeFiles(File[] files, File mergedFile) {
FileWriter fstream = null;
BufferedWriter out = null;
try {
fstream = new FileWriter(mergedFile, true);
out = new BufferedWriter(fstream);
} catch (IOException e1) {
e1.printStackTrace();
}
for (File f : files) {
System.out.println("merging: " + f.getName());
FileInputStream fis;
try {
fis = new FileInputStream(f);
BufferedReader in = new BufferedReader(new InputStreamReader(fis));
String aLine;
while ((aLine = in.readLine()) != null) {
out.write(aLine);
out.newLine();
}
in.close();
} catch (IOException e) {
e.printStackTrace();
}
}
try {
out.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
String sourceFile1Path = "/mnt/sdcard/s1/t1.txt";
String sourceFile2Path = "/mnt/sdcard/s2/t2.txt";
String mergedFilePath = "/mnt/sdcard/merge/t.txt";
File[] files = new File[2];
files[0] = new File(sourceFile1Path);
files[1] = new File(sourceFile2Path);
File mergedFile = new File(mergedFilePath);
mergeFiles(files, mergedFile);
Use this
Object[] DirTotal = listTemp.toArray();
But why you can't use this temp list?
you can use addAll() method in which is available in java.util package.
hey ive got a chunk of code here trying to read a single line in a .csv file:
rows = new WarehouseItem[];
public void readCSV(String filename) {
FileInputStream fileStrm = null;
InputStreamReader rdr;
BufferedReader bufRdr;
int lineNum;
String line;
try {
fileStrm = new FileInputStream(filename);
rdr = new InputStreamReader(fileStrm);
bufRdr = new BufferedReader(rdr);
numRows = 0;
line = bufRdr.readLine();
while (line != null) {
rows[numRows] = line;
numRows++;
line = bufRdr.readLine();
}
fileStrm.close();
}
catch (IOException e) {
if (fileStrm != null) {
try {
fileStrm.close();
} catch (IOException ex2) {}
}
System.out.println("Error in file processing: " + e.getMessage());
}
}
on the rows[numRows] = line im trying to store the line into an array of objects(i have premade an object which contains an array of strings and the number of columns)
im not entirely sure how to store the single line im trying to read in my object.
any help would be really appreciated :)
Your life would be an awful lot easier if you used a CSV library to do this. With jackson it's really simple to read CSV into an array of objects.
For example:
CsvMapper mapper = new CsvMapper();
mapper.enable(CsvParser.Feature.WRAP_AS_ARRAY);
File csvFile = new File("input.csv"); // or from String, URL etc
MappingIterator<Object[]> it = mapper.reader(Object[].class).readValues(csvFile);
See here for more info on parsing CSV in java: http://demeranville.com/how-not-to-parse-csv-using-java/
I'm sending compressed data from a java app via nodejs to a webpage. The data is compressed with the java deflater and base64 encoded. On the webpage I'm trying to inflate the data with https://github.com/dankogai/js-deflate, but it does not work (empty result). Am I missing something?
Java side:
private String compress(String s) {
DeflaterOutputStream def = null;
String compressed = null;
try {
ByteArrayOutputStream out = new ByteArrayOutputStream();
// create deflater without header
def = new DeflaterOutputStream(out, new Deflater(Deflater.DEFAULT_COMPRESSION, true));
def.write(s.getBytes());
def.close();
compressed = Base64.encodeBase64String(out.toByteArray());
System.out.println(compressed);
} catch(Exception e) {
Log.c(TAG, "could not compress data: " + e);
}
return compressed;
}
Javascript side:
var data = RawDeflate.inflate(Base64.fromBase64(compressed));
Try this:
public static String compressAndEncodeString(String str) {
DeflaterOutputStream def = null;
String compressed = null;
try {
ByteArrayOutputStream out = new ByteArrayOutputStream();
// create deflater without header
def = new DeflaterOutputStream(out, new Deflater(Deflater.BEST_COMPRESSION, true));
def.write(str.getBytes());
def.close();
compressed = Base64.encodeToString(out.toByteArray(), Base64.DEFAULT);
} catch(Exception e) {
Log.e(TAG, "could not compress data: " + e);
}
return compressed;
}
I ran into the same problem. The js-deflate project inflater appears broken. I found it would work on a short input but fail on a long input (e.g., lorem ipsum as test data).
A better option turned out to be zlib.js.
Here is how I'm using it to inflate in Javascript a JSON object that is generated, compressed, and base64 encoded on the server:
var base64toBinary = function (base64) {
var binary_string = window.atob(base64);
var len = binary_string.length;
var bytes = new Uint8Array( len );
for (var i = 0; i < len; i++) {
var ascii = binary_string.charCodeAt(i);
bytes[i] = ascii;
}
return bytes.buffer;
}
var utf8ToString = function (uintArray) {
var encodedString = String.fromCharCode.apply(null, uintArray),
decodedString = decodeURIComponent(escape(encodedString));
return decodedString;
}
var object = JSON.parse(utf8ToString(
new Zlib.RawInflate(base64toBinary(base64StringFromServer)).decompress()));
(FYI, the helper functions are derived from other stackoverflow answers).