We Keep Coding

Apache Beam : Java : write results to multiple files using a variable in TextIO.write()

Need to write results to multiple output files and names of output files are generated at run time.
Example - Read from input csv that has students names and marks in each of the course. Output file has sum of marks for all the courses against student name.
Scenario 1 : Output results in a single csv - this is simple.
Input csv -
'''
ID,Name,Physics,Chemistry,Math,English,Biology,History
1001,James,56,60,78,67,50,70
1002,Robert,64,72,62,58,78,80
1003,Nina,72,70,83,72,75,85
Output csv -
Name,Total
Robert,414
Nina,457
James,381
Code snippet-
pipeline.apply(TextIO.read().from(options.getInputFilePath()))
.apply(ParDo.of(new FilterHeaderFn(CSV_HEADER)))
.apply(ParDo.of(new ComputeTotalScoresFn()))
.apply(ParDo.of(new ConvertToStringFn()))
.apply(TextIO.write().to(options.getOutputFilePath())
.withHeader("Name,Total").withNumShards(1));
Scenario 2: Output results in multiple files such that each student has a file created with their name and it contains total marks. Note - file names are generated at run time.
example -
Robert.csv => contains 414
Nina.csv => 457
James.csv => 381
I tried setting output file path inside
Code snippet -
`ProcessContext.getPipelineOptions().as(MyOptionClass.class).setOuputPath(output_path)`
but that doesn't work. Any help is appreciated.

You can use a dynamic file destination to achieve this, or a more generic Runtime value provider

while Parsing TXT to CSV using JAVA long numbers are getting converted to exponential

I have big txt file which needs to converted to CSV format using JAVA ,
I am parsing the file but file is having some long nos in a column which is getting converted to exponential .
like :89148000006119921953 ->8.9148E
I need to parse the txt to csv in such a way so what csv opens with excel doesn't not convert to exponentials values
Please find the below code :
public static String converter(String filename) throws Exception {
FileWriter writer = null;
if (filename.toString().endsWith("TXT")) {
File file = new File("C:\\convertertool\\inputFiles\\" + filename + "");
Scanner scan = new Scanner(file);
filename = filename.replace("TXT", "CSV");
File file2 = new File("C:\\convertertool\\ParsedFiles\\" + filename + "");
file.createNewFile();
writer = new FileWriter(file2);
while (scan.hasNext()) {
String csv = scan.nextLine().replace("|", ",");
System.out.println(csv);
writer.append(csv);
writer.append("\n");
writer.flush();
}
}
return filename;
}

Double.parseDouble("891413E")//or BigDecimal.parse...
Can help you

The assumptions are:
(1) You want to open the CSV file (e.g. test.csv) in Excel by double-clicking on the file, so that the data is displayed as a sequence of digits (instead of using scientific notation).
(2) You do not want to perform any manual formatting on the Excel file, after it has been opened (so, for example you do not want to have to change the Excel cell formatting).
(3) You do not want to use the Excel manual import wizard.
The only way I know of doing this is to write an Excel formula to the CSV file, instead of writing the actual, unadulterated numeric value. You can use any Excel formula which converts the number to a string.
For example, if the numeric value is this:
89148000006119921953
then the CSV file needs to contain the following (I added field one just for this demo):
field one,"=TEXT(89148000006119921953,""#"")"
You can try this for yourself by pasting the above into a test.csv file, and seeing what happens when you double-click on it.
The result in Excel is this:
In this case the Excel formula (if created directly in Excel) would be this:
=TEXT(1234567890123450000,"#")
So in the CSV file, we have to surround the field in double quotes, and we have to escape the double quotes inside the formula by doubling them.
There are some disadvantages to doing this:
(a) The text file no longer contains the pure, unchanged data. It contains an Excel formula instead. So it is more-or-less useless outside of Excel.
(b) The value in Excel is a text formula not a number (so you can't perform arithmetic on it).
(c) The Excel column widths will not auto-resize, so if there is any data in cell C1, then cell B1's display will be truncated (but the full text will still be there).
(d) Now you almost certainly do need a proper Java CSV parsing library to make sure you generate valid CSV data.
Final thought: If the end result needs to be viewed in Excel (and only in Excel), then maybe you should generate an actual Excel file (e.g. using Apache POI) instead of generating a CSV file.

Generate text file based on excel sheet

I'm not a programmer. I'm looking for generating a script from excel file based on the following;
A B C
value1 Value2 MyVal1
Value3 Value4 MYVal2
I want to generate text file as the following;
Hello
I'm (Value1)
I'm looking (Value2) (MyVal1)
next
I'm (Value3)
I'm looking (Value4) (MyVal2)
next
and so on for each row in excel file
this photo clarify what i need enter image description here
Regards,

I imagine what you want is a way to read an excel file, and then output a text file in the format you have outlined above. To read an excel file, here is a fairly straightforward way of obtaining all the values by cells that you require: https://www.callicoder.com/java-read-excel-file-apache-poi/. And to write these out to a file, there's multiple ways but here is a simple tutorial to follow as well: https://www.baeldung.com/java-write-to-file

Call parameters from JAVA code in R code to dynamically pass csv file and unique ID

I have a R script in which I want to call parameters from Java code. The parameters are csv file name file name and unique ID which has to be used to name the two output files.
My R script is :
df1 <- read.csv("filename.csv")
vs=colnames(df1)
md=formula(paste(vs[3],"~",vs[1],"+",vs[2]))
fit <- summary(aov(md, data=df1))[[1]]
#text output
names(fit)[1:4]=c("DF","SS","MS","F")
sink("test.txt")
In this code the first line df1 <- read.csv("filename.csv") should take file name dynamically from JAVA code and the last line sink("test.txt") should take unique ID and create the output file.
The java code is :
buildCommand.add("Rscript ");
buildCommand.add(scriptName);
buildCommand.add(inputFileWithPathExtension);
buildCommand.add(uniqueIdForR);
I have seen other post but I am unsure wether it will help in my case, also similar posts talking about rJava package`, but didn't get clear idea.
Any help will be highly appreciated. thanks in advance !

Here a very simple example for reading command line arguments in your case:
args <- commandArgs(TRUE)
input <- args[1]
output <- paste0(args[2], ".txt")
cat("Reading from", input, "\n")
cat("Writing to", output, "\n")
Example:
$ Rscript foo.R foo.csv 1234567
Reading from foo.csv
Writing to 1234567.txt

CSV and Big Values

I have string value.
The value is :
12345.123456789012345
I want to write this value to csv file from java.I use OpenCSV for this.
Here is the code.
String csv = "D:\\denemehttp\\dene.csv";
CSVWriter writer = new CSVWriter(new FileWriter(csv));
String [] country = "India#China#United States#12345.1234567890123456".split("#");
writer.writeNext(country);
writer.close();
But when I open the csv the value is :
12345.123456789
But it must be:
12345.123456789012345
And note that when I open the csv file with Notepad++ it show the true value:
12345.123456789012345
So what is the problem?
Edit:I found the solution.I try Apache POI and it works.You can set the cell type with APACHE POI and excell doesn't see the value as a number and doesn't format it.It writes the value as a string completely.

You mention that you opened up the CSV in two different applications. The first one either clips it or formats it so you don't see the true value (eg. Excel). Your program is correct.
If the application you are using is Excel which you are using to open, you need to enclose the value in double quotes (") before you write it.
The csv should look like:
"India","China","United States","12345.1234567890123456"
This is because Excel does not recognize the number with this amount of precision, it needs to be treated as a string.