I am fetching the data by reading from a csv file and storing it as List<List<String<>> "data". While trying to write that data to another csv file I need to Check a condition whether id in this "rowdata" matches with id column of another csv file or not. Please help in how to check this condition.
The condition is data.id=value.id;// value is data from another csv in the form List<List<String<>> value
public void writeRecords(List<List<String>> data) throws IOException {
FileWriter csvWriter1 = new FileWriter(OUTPUT_PATH);
Cable c1=new Cable();
List<List<String>> value=c1.readRecords();
for (List<String> rowData : data) {
csvWriter1.append(String.join(",", rowData));
csvWriter1.append("\n");
}
csvWriter1.flush();
csvWriter1.close();
}
Would not suggest you to operate with data structure like that List<List<String<>> "data".However, assumed all your lists are sorted and you don't care about code complexity level - in your situation you can do smth like that to compare data from your two lists:
for (int i =0; i<value.size(); i++){
for (int j = 0; j<value.get(i).size(); j++) {
if(value.get(i).get(j)==data.get(i).get(j));
//write data to another csv
}
}
Related
I have a requirement to parse the excel file and create the list of objects from that. TO do the same we are using the There are org.apache.poi to read the excel file and we are able to get the required details, currently we are getting the cell value based on the index and set it to the object field. But we believe it is not a good way to get the values based on the index and we should find a generic way to successfully parse the excel file in case some columns are added or removed so that we don't have to do much effort on code. I came across this article which almost fulfills the requirements but used the reflection methods which we are not allowed to use. Is there any possible way to parse the excel file without using the cell index where we don't have to put much effort if the format of the excel file gets changed?
public List<DTO> jsonConverter(Workbook workbook, Sheet sheet, String filename)
throws ParseException {
List<DTO> listOfDTOs = new ArrayList<>();
Row row;
for (int index = 1; index <= sheet.getLastRowNum(); index++) {
row = sheet.getRow(index);
if (row != null) {
DTO dto = new DTO();
dto.setFieldX(
getCellValueAsStringBasedOnCellType(
workbook, row.getCell(0, MissingCellPolicy.CREATE_NULL_AS_BLANK)));
dto.setFieldY(
getCellValueAsStringBasedOnCellType(
workbook, row.getCell(1, MissingCellPolicy.CREATE_NULL_AS_BLANK)));
listOfDTOs.add(dto);
}
}
return listOfDTOs;
}
public String getCellValueAsStringBasedOnCellType(Workbook workbook, Cell cell) {
DataFormatter formatter = new DataFormatter();
if (cell != null && cell.getCellType() == CellType.FORMULA) {
FormulaEvaluator evaluator = workbook.getCreationHelper().createFormulaEvaluator();
return formatter.formatCellValue(cell, evaluator);
}
return formatter.formatCellValue(cell);
}
Sure. It's fairly simple:
The setup
Read the first row in the excel file, treat it as columns, and read every cell as a string.
Store these in an array.
Now you can turn any index into the column name, simply using the expression headers[idx].
Thus, for any given cell, you know the header name. Now you need to translate this knowledge into the right call. Given that you're in column, say, E (i == 4), the header value is header[4] which is, say, Address, then you want to take the string ("Address") and turn that into the right call. You end up needing to invoke:
String cellValue = getCellValueAsStringBasedOnCellType(workbook, row.getCell(1, MissingCellPolicy.CREATE_NULL_AS_BLANK)));
dto.setFieldAddress(cellValue);
Everything in that code snippet is the same for any value of that string, except the setFieldAddress name.
So, we need to turn the string "Address" into the act of invoking setFieldAddress.
The solution
java.util.function and hashmaps to the rescue!
This is a way to store in a variable the concept of taking a dto instance and setting the Address field:
BiConsumer<DTO, String> setAddress = (dto, value) -> dto.setFieldAddress(value);
or even simpler:
BiConsumer<DTO, String> setAddress = DTO::setFieldAddress;
These snippets do the same thing: They don't set an address; they are a recipe for how that is done, and you store the concept of setting an address on a DTO in a variable so you can run it later and as many times as you want. This is generally called a 'closure' a 'lambda'.
We can store these things in a map:
Map<String, BiConsumer<DTO, String>> dtoSetters = new HashMap<>();
dtoSetters.put("Address", DTO::setFieldAddress);
And then we can just figure it out:
int colIdx = ...;
String headerName = header[colIdx];
var setter = dtoSetters.get(headerName);
if (setter == null) throw new IllegalStateException("Unexpected column header in excel sheet: " + headerName);
String cellValue = getCellValueAsStringBasedOnCellType(workbook, row.getCell(1, MissingCellPolicy.CREATE_NULL_AS_BLANK)));
setter.apply(dto, cellValue);
Thus, make that map (once, at system boot, e.g. with static initializers), replace your dto.setFieldX code with the above, and voila.
I have a test.csv file that is formatted as:
Home,Owner,Lat,Long
5th Street,John,5.6765,-6.56464564
7th Street,Bob,7.75,-4.4534564
9th Street,Kyle,4.64,-9.566467364
10th Street,Jim,14.234,-2.5667564
I have a hashmap that reads a file that contains the same header contents such as the CSV, just a different format, with no accompanying data.
In example:
Map<Integer, String> container = new HashMap<>();
where,
Key, Value
[0][NULL]
[1][Owner]
[2][Lat]
[3][NULL]
I have also created a second hash map that:
BufferedReader reader = new BufferedReader (new FileReader("test.csv"));
CSVParser parser = new CSVParser(reader, CSVFormat.DEFAULT);
Boolean headerParsed = false;
CSVRecord headerRecord = null;
int i;
Map<String,String> value = new HashMap<>();
for (final CSVRecord record : parser) {
if (!headerParsed = false) {
headerRecord = record;
headerParsed = true;
}
for (i =0; i< record.size(); i++) {
value.put (headerRecord.get(0), record.get(0));
}
}
I want to read and compare the hashmap, if the container map has a value that is in the value map, then I put that value in to a corresponding object.
example object
public DataSet (//args) {
this.home
this.owner
this.lat
this.longitude
}
I want to create a function where the data is set inside the object when the hashmaps are compared and when a value map key is equal to a contain map key, and the value is placed is set into the object. Something really simply that is efficient at handling the setting as well.
Please note: I made the CSV header and the rows finite, in real life, the CSV could have x number of fields(Home,Owner,Lat,Long,houseType,houseColor, ect..), and a n number of values associated to those fields
First off, your approach to this problem is too unnecessarily long. From what I see, all you are trying to do is this:
Select a two columns from a CSV file, and add them to a data structure. I highlighted the word two because in a map, you have a key and a value. One column becomes the key, and the other becomes the value.
What you should do instead:
Import the names of columns you wish to add to the data structure into two strings. (You may read them from a file).
Iterate over the CSV file using the CSVParser class that you did.
Store the value corresponding to the first desired column in a string, repeat with the value corresponding to the second desired column, and push them both into a DataSet object, and push the DataSet object into a List<DataSet>.
If you prefer to stick to your way of solving the problem:
Basically, the empty file is supposed to hold just the headers (column names), and that's why you named the corresponding hash map containers. The second file is supposed to contain the values and hence you named the corresponding hash map values.
First off, where you say
if (!headerParsed = false) {
headerRecord = record;
headerParsed = true;
}
you probably mean to say
if (!headerParsed) {
headerRecord = record;
headerParsed = true;
}
and where you say
for (i =0; i< record.size(); i++) {
value.put(headerRecord.get(0), record.get(0));
}
you probably mean
for (i =0; i< record.size(); i++) {
value.put(headerRecord.get(i), record.get(i));
}
i.e. You iterate over one record and store the value corresponding to each column.
Now I haven't tried this code on my desktop, but since the for loop also iterates over Home and Longitude, I think it should create an error and you should add an extra check before calling value.put (i.e. value.put("Home", "5th Street") should create an error I suppose). Wrap it inside an if conditional and check of the headerRecord(i) even exists in the containers hash map.
for (i =0; i< record.size(); i++) {
if (container[headerRecord.get(i)] != NULL) {
value.put(headerRecord.get(i), record.get(i));
}
}
Now thing is, that the data structure itself depends on which values from the containers hash map you want to store. It could be Home and Lat, or Owner and Long. So we are stuck. How about you create a data structure like below:
struct DataSet {
string val1;
string val2;
}
Also, note that this DataSet is only for storing ONE row. For storing information from multiple rows, you need to create a Linked List of DataSet.
Lastly, the container file contains ALL the column names. Not all these columns will be stored in the Data Set (i.e. You chose to NULL Home and Long. You could have chosen to NULL Owner and Lat), hence the header file is not what you need to make this decision.
If you think about it, just iterate over the values hash map and store the first value in string val1 and the second value in val2.
List<DataSet> myList;
DataSet row;
Iterator it = values.entrySet().iterator();
while (it.hasNext()) {
Map.Entry pair = (Map.Entry)it.next();
row.val1 = pair.getKey();
row.val2 = pair.getValue();
myList.add(row);
it.remove();
}
I hope this helps.
I am trying to write a Java program that loads the data (from a tab delimited DAT file) and determines the average amount in Euros (EUR), grouped by Country and Credit Rating.
I have 2 questions,
what is the best way to load the data into data structure after spliting into array?
How do i approach about providing group by functionality in Java
Update: I have given a first try and this is how implementation looks like. Feels like there is a room for improvement.
/**
* #param rows - Each row as a bean
* This method will group objects together based on Country/City and Credit Rating
*/
static void groupObjectsTogether(List<CompanyData> rows) {
Map<String, List<CompanyData>> map = new HashMap<String, List<CompanyData>>();
for(CompanyData companyData : rows){
String key;
if(companyData.getCountry().trim().equalsIgnoreCase("") || companyData.getCountry() == null){
key = companyData.getCity()+":"+companyData.getCreditRating(); //use city+creditRating as key
}else{
key = companyData.getCountry()+":"+companyData.getCreditRating(); //use country+creditRating as key
}
if(map.get(key) == null){
map.put(key, new ArrayList<CompanyData>());
}
map.get(key).add(companyData);
}
processGroupedRowsAndPrint(map);
}
It all depends on the amount of data and performance (CPU vs memory) of the machine. It the amount of data is not significant (less than millions of records or columns) and the number of columns is fixed then you may simply put all data in arrays using
String[] row = String.split(";");
which shall split each row using ; as delimiter. Then you may achieve your grouping functionality using HashMap, i.e.:
ArrayList<String[]> rowAr = new ArrayList<String[]>();
HashMap<String,ArrayList<Integer>> map = new HashMap<String,ArrayList<Integer>>();
int index = 0;
for (String rowStr: rows) {
String[] row = rowStr.split(";");
rowAr.add(row);
String companyCode = row[0];
//please keep in mind that for simplicity of the example I avoided
//creation of new array if it does not exist in HashMap
((ArrayList<Integer>)map.get(companyCode)).add(index);
index++;
}
Sorry for any syntax or other simple errors above (I do not have any tools in hand to verify if there is not any stupid mistake).
I have to export data to excel having dynamic number of columns
using java apache workbook,
on every execution, column details will be saved in ListObject,
which will be dynamically generated and get saved in
List<Object> expColName = new ArrayList<Object>();
From the List , I have to obtain individual values and export into every column of the excel sheet,
for(int i=0; i<expColName.size(); i++){
data.put("1",new Object[] {
expColName.get(i)
});
}
The above code gives only the last column value in the excel sheet
What type is data and how do you read the values from the map?
It seems like you are putting every object into the same "key" of the Map, thats why you only get the last item from the list.
You could try to give it a test with:
for(int i=0; i<expColName.size(); i++){
data.put(i+"",new Object[] {
expColName.get(i)
});
}
My first CSV file looks like this with header included (header is included only at the top not after every entry):
NAME,SURNAME,AGE
Fred,Krueger,Unknown
.... n records
My second file might look like this:
NAME,MIDDLENAME,SURNAME,AGE
Jason,Noname,Scarry,16
.... n records with this header template
The merged file should look like this:
NAME,SURNAME,AGE,MIDDLENAME
Fred,Krueger,Unknown,
Jason,Scarry,16,Noname
....
Basically if headers don't match, all new header titles (columns) should be added after original header and their values according to that order.
Update
Above CSV were made smaller so I can illustrate what I want to achieve, in reality CSV files are generated one step before this (merge) and can be up to 100 columns
How can I do this?
I'd create a model for the 'bigger' format (a simple class with four fields and a collection for instances of this class) and implemented two parsers, one for the first, one for the second model. Create records for all rows of both csv files and implement a writer to output the csv in the correct format. IN brief:
public void convert(File output, File...input) {
List<Record> records = new ArrayList<Record>();
for (File file:input) {
if (input.isThreeColumnFormat()) {
records.addAll(ThreeColumnFormatParser.parse(file));
} else {
records.addAll(FourColumnFormatParser.parse(file));
}
}
CsvWriter.write(output, records);
}
From your comment I see, that you a lot of different csv formats with some common columns.
You could define the model for any row in the various csv files like this:
public class Record {
Object id; // some sort of unique identifier
Map<String, String> values; // all key/values of a single row
public Record(Object id) {this.id=id;}
public void put(String key, String value){
values.put(key, value);
}
public void get(String key) {
values.get(key);
}
}
For parsing any file you would first read the header and add the column headers to a global keystore (will be needed later on for outputting), then create records for all rows, like:
//...
List<Record> records = new ArrayList<Record>()
for (File file:getAllFiles()) {
List<String> keys = getColumnsHeaders(file);
KeyStore.addAll(keys); // the store is a Set
for (String line:file.getLines()) {
String[] values = line.split(DELIMITER);
Record record = new Record(file.getName()+i); // as an example for id
for (int i = 0; i < values.length; i++) {
record.put(keys.get(i), values[i]);
}
records.add(record);
}
}
// ...
Now the keystore has all used column header names and we can iterate over the collection of all records, get all values for all keys (and get null if the file for this record didn't use the key), assemble the csv lines and write everything to a new file.
Read in the header of the first file and create a list of the column names. Now read the header of the second file and add any column names that don't exist already in the list to the end of the list. Now you have your columns in the order that you want and you can write this to the new file first.
Next I would parse each file and for each row I would create a Map of column name to value. Once the row is parsed you could then iterate over the new list of column names and pull the values from the map and write them immediately to the new file. If the value is null don't print anything (just a comma, if required).
There might be more efficient solutions available, but I think this meets the requirements you set out.
Try this:
http://ondra.zizka.cz/stranky/programovani/ruzne/querying-transforming-csv-using-sql.texy
crunch input.csv output.csv "SELECT AVG(duration) AS durAvg FROM (SELECT * FROM indata ORDER BY duration LIMIT 2 OFFSET 6)"