My first CSV file looks like this with header included (header is included only at the top not after every entry):
NAME,SURNAME,AGE
Fred,Krueger,Unknown
.... n records
My second file might look like this:
NAME,MIDDLENAME,SURNAME,AGE
Jason,Noname,Scarry,16
.... n records with this header template
The merged file should look like this:
NAME,SURNAME,AGE,MIDDLENAME
Fred,Krueger,Unknown,
Jason,Scarry,16,Noname
....
Basically if headers don't match, all new header titles (columns) should be added after original header and their values according to that order.
Update
Above CSV were made smaller so I can illustrate what I want to achieve, in reality CSV files are generated one step before this (merge) and can be up to 100 columns
How can I do this?
I'd create a model for the 'bigger' format (a simple class with four fields and a collection for instances of this class) and implemented two parsers, one for the first, one for the second model. Create records for all rows of both csv files and implement a writer to output the csv in the correct format. IN brief:
public void convert(File output, File...input) {
List<Record> records = new ArrayList<Record>();
for (File file:input) {
if (input.isThreeColumnFormat()) {
records.addAll(ThreeColumnFormatParser.parse(file));
} else {
records.addAll(FourColumnFormatParser.parse(file));
}
}
CsvWriter.write(output, records);
}
From your comment I see, that you a lot of different csv formats with some common columns.
You could define the model for any row in the various csv files like this:
public class Record {
Object id; // some sort of unique identifier
Map<String, String> values; // all key/values of a single row
public Record(Object id) {this.id=id;}
public void put(String key, String value){
values.put(key, value);
}
public void get(String key) {
values.get(key);
}
}
For parsing any file you would first read the header and add the column headers to a global keystore (will be needed later on for outputting), then create records for all rows, like:
//...
List<Record> records = new ArrayList<Record>()
for (File file:getAllFiles()) {
List<String> keys = getColumnsHeaders(file);
KeyStore.addAll(keys); // the store is a Set
for (String line:file.getLines()) {
String[] values = line.split(DELIMITER);
Record record = new Record(file.getName()+i); // as an example for id
for (int i = 0; i < values.length; i++) {
record.put(keys.get(i), values[i]);
}
records.add(record);
}
}
// ...
Now the keystore has all used column header names and we can iterate over the collection of all records, get all values for all keys (and get null if the file for this record didn't use the key), assemble the csv lines and write everything to a new file.
Read in the header of the first file and create a list of the column names. Now read the header of the second file and add any column names that don't exist already in the list to the end of the list. Now you have your columns in the order that you want and you can write this to the new file first.
Next I would parse each file and for each row I would create a Map of column name to value. Once the row is parsed you could then iterate over the new list of column names and pull the values from the map and write them immediately to the new file. If the value is null don't print anything (just a comma, if required).
There might be more efficient solutions available, but I think this meets the requirements you set out.
Try this:
http://ondra.zizka.cz/stranky/programovani/ruzne/querying-transforming-csv-using-sql.texy
crunch input.csv output.csv "SELECT AVG(duration) AS durAvg FROM (SELECT * FROM indata ORDER BY duration LIMIT 2 OFFSET 6)"
Related
I am fetching the data by reading from a csv file and storing it as List<List<String<>> "data". While trying to write that data to another csv file I need to Check a condition whether id in this "rowdata" matches with id column of another csv file or not. Please help in how to check this condition.
The condition is data.id=value.id;// value is data from another csv in the form List<List<String<>> value
public void writeRecords(List<List<String>> data) throws IOException {
FileWriter csvWriter1 = new FileWriter(OUTPUT_PATH);
Cable c1=new Cable();
List<List<String>> value=c1.readRecords();
for (List<String> rowData : data) {
csvWriter1.append(String.join(",", rowData));
csvWriter1.append("\n");
}
csvWriter1.flush();
csvWriter1.close();
}
Would not suggest you to operate with data structure like that List<List<String<>> "data".However, assumed all your lists are sorted and you don't care about code complexity level - in your situation you can do smth like that to compare data from your two lists:
for (int i =0; i<value.size(); i++){
for (int j = 0; j<value.get(i).size(); j++) {
if(value.get(i).get(j)==data.get(i).get(j));
//write data to another csv
}
}
I have got some csv-files, which I want to transform into a json.
Unfortunately the structure of the csv doesn't match the desired json format. a) because csv is a flat structure and the json should be of nested structure. b) because the column headers don't match the json property names.
Illustrating minimal example CSV:
ColumnNameX,ColumnNameY,ColumnNameZ
valueX,valueY,valueZ
should be transformed into this JSON:
{
"XZObject": {
"absurdlyNotNamedLikeCsvHeading": "valueX",
"AlsoNOTColumnNameZ": "valueZ" },
"YyyyyWhy": {
"ThisResemblesColumnNameY": "valueY"
}
I would naively go and make some representing POJO-classes and put in the values by position – like so (pseudocode):
class Container {Fields:XZObject,YyyyyWhy}
class XZObject {Fields:absurdlyNotNamedLikeCsvHeading,AlsoNOTColumnNameZ}
class YyyyyWhy {Fields:ThisResemblesColumnNameY}
new XZObject(absurdlyNotNamedLikeCsvHeading=csvLineElements[0],AlsoNOTColumnNameZ=csvLineElements[2])
new YyyyyWhy(ThisResemblesColumnNameY=csvLineElement[1])
new Container(XZObject,YyyyyWhy)
…then I'd transform the Container object to JSON with gson.
The problem is, when a field in the CSV gets added to the scheme, I'd have to adjust every positional mapping after the new column.
So I wonder: Is there a simple way to map CSV-columns by header to a specific JSON property? Preferably with gson-lib.
In other words: Can I i.e. map the value in column with header "ColumnNameZ" to property "XZobject.AlsoNOTColumnNameZ" in a simple way?
I think parsing the CSV file into Objects is the good way to go.
You can read the first column first, split it and calculate the index of each column at runtime. Then it doesn't matter if you add/remove or shuffle columns
Assuming you read the first line and you have
String firstRow = "ColumnNameX,ColumnNameY,ColumnNameZ";
Parse it this way:
List<String> columnList = Arrays.asList(firstRow.split(","));
int COLUMN_NAME_X_INDEX = columnList.indexOf("ColumnNameX");
int COLUMN_NAME_Y_INDEX = columnList.indexOf("ColumnNameY");
int COLUMN_NAME_Z_INDEX = columnList.indexOf("ColumnNameZ");
Than use your newly found indexes:
XZObject xzObject = new XZObject(csvLineElements[COLUMN_NAME_X_INDEX], csvLineElements[COLUMN_NAME_Z_INDEX]);
YyyyyWhy yyyyyWhy = new YyyyyWhy(csvLineElements[COLUMN_NAME_Y_INDEX]);
Container container = new Container(XZObject,YyyyyWhy);
I have a test.csv file that is formatted as:
Home,Owner,Lat,Long
5th Street,John,5.6765,-6.56464564
7th Street,Bob,7.75,-4.4534564
9th Street,Kyle,4.64,-9.566467364
10th Street,Jim,14.234,-2.5667564
I have a hashmap that reads a file that contains the same header contents such as the CSV, just a different format, with no accompanying data.
In example:
Map<Integer, String> container = new HashMap<>();
where,
Key, Value
[0][NULL]
[1][Owner]
[2][Lat]
[3][NULL]
I have also created a second hash map that:
BufferedReader reader = new BufferedReader (new FileReader("test.csv"));
CSVParser parser = new CSVParser(reader, CSVFormat.DEFAULT);
Boolean headerParsed = false;
CSVRecord headerRecord = null;
int i;
Map<String,String> value = new HashMap<>();
for (final CSVRecord record : parser) {
if (!headerParsed = false) {
headerRecord = record;
headerParsed = true;
}
for (i =0; i< record.size(); i++) {
value.put (headerRecord.get(0), record.get(0));
}
}
I want to read and compare the hashmap, if the container map has a value that is in the value map, then I put that value in to a corresponding object.
example object
public DataSet (//args) {
this.home
this.owner
this.lat
this.longitude
}
I want to create a function where the data is set inside the object when the hashmaps are compared and when a value map key is equal to a contain map key, and the value is placed is set into the object. Something really simply that is efficient at handling the setting as well.
Please note: I made the CSV header and the rows finite, in real life, the CSV could have x number of fields(Home,Owner,Lat,Long,houseType,houseColor, ect..), and a n number of values associated to those fields
First off, your approach to this problem is too unnecessarily long. From what I see, all you are trying to do is this:
Select a two columns from a CSV file, and add them to a data structure. I highlighted the word two because in a map, you have a key and a value. One column becomes the key, and the other becomes the value.
What you should do instead:
Import the names of columns you wish to add to the data structure into two strings. (You may read them from a file).
Iterate over the CSV file using the CSVParser class that you did.
Store the value corresponding to the first desired column in a string, repeat with the value corresponding to the second desired column, and push them both into a DataSet object, and push the DataSet object into a List<DataSet>.
If you prefer to stick to your way of solving the problem:
Basically, the empty file is supposed to hold just the headers (column names), and that's why you named the corresponding hash map containers. The second file is supposed to contain the values and hence you named the corresponding hash map values.
First off, where you say
if (!headerParsed = false) {
headerRecord = record;
headerParsed = true;
}
you probably mean to say
if (!headerParsed) {
headerRecord = record;
headerParsed = true;
}
and where you say
for (i =0; i< record.size(); i++) {
value.put(headerRecord.get(0), record.get(0));
}
you probably mean
for (i =0; i< record.size(); i++) {
value.put(headerRecord.get(i), record.get(i));
}
i.e. You iterate over one record and store the value corresponding to each column.
Now I haven't tried this code on my desktop, but since the for loop also iterates over Home and Longitude, I think it should create an error and you should add an extra check before calling value.put (i.e. value.put("Home", "5th Street") should create an error I suppose). Wrap it inside an if conditional and check of the headerRecord(i) even exists in the containers hash map.
for (i =0; i< record.size(); i++) {
if (container[headerRecord.get(i)] != NULL) {
value.put(headerRecord.get(i), record.get(i));
}
}
Now thing is, that the data structure itself depends on which values from the containers hash map you want to store. It could be Home and Lat, or Owner and Long. So we are stuck. How about you create a data structure like below:
struct DataSet {
string val1;
string val2;
}
Also, note that this DataSet is only for storing ONE row. For storing information from multiple rows, you need to create a Linked List of DataSet.
Lastly, the container file contains ALL the column names. Not all these columns will be stored in the Data Set (i.e. You chose to NULL Home and Long. You could have chosen to NULL Owner and Lat), hence the header file is not what you need to make this decision.
If you think about it, just iterate over the values hash map and store the first value in string val1 and the second value in val2.
List<DataSet> myList;
DataSet row;
Iterator it = values.entrySet().iterator();
while (it.hasNext()) {
Map.Entry pair = (Map.Entry)it.next();
row.val1 = pair.getKey();
row.val2 = pair.getValue();
myList.add(row);
it.remove();
}
I hope this helps.
I want to copy data from one HBase table to another using Java APIs, but not able to find one. Is there any Java API to do the same?
Thanks.
The following is not by far the most optimized way - but from the tone of the question it seems performance is not the critical factor here.
First, you need to set up your HBaseConfiguration and your input / output tables:
Configuration config = HBaseConfiguration.create();
HTable inputTable = new HTable(config, "input_table");
HTable outputTable = new HTable(config, "output_table");
What you want is a "Scan", which allows a range scan to be performed. You need to define the query parameters, by adding columns to a Scan object.
Scan scan = new Scan(Bytes.toBytes("smith-"));
scan.addColumn(Bytes.toBytes("personal"), Bytes.toBytes("givenName"));
scan.addColumn(Bytes.toBytes("contactinfo"), Bytes.toBytes("email"));
scan.setFilter(new PageFilter(25));
Now you are ready to invoke the scan object and retrieve results:
ResultScanner scanner = inputTable.getScanner(scan);
for (Result result : scanner) {
putToOutputTable(result);
}
Now to save to the second table, you will either do Put's within the for loop, or aggregate the results into a List/Array or similar for a bulk put.
protected void putToOutputTable(Result result) {
// Retrieve the Map of families to their most recent qualifiers and values.
NavigableMap<byte[],NavigableMap<byte[],byte[]>> map = result.getNoVersionMap();
for ( // iterate through the family/values map entries for this result ) {
// Convert the result to the row key and the column values here ..
// specifically set the rowKey, colFamily, colQualifier, and colValue(s)
Put p = new Put(Bytes.toBytes(rowKey));
// To set the value you'd like to update in the row 'myLittleRow',
// specify the column family, column qualifier, and value of the table
// cell you'd like to update. The column family must already exist
// in your table schema. The qualifier can be anything.
// All must be specified as byte arrays as hbase is all about byte
// arrays. Lets pretend the table 'myLittleHBaseTable' was created
// with a family 'myLittleFamily'.
p.add(Bytes.toBytes(colFamily), Bytes.toBytes(colQualifier),
Bytes.toBytes(colValue));
}
table.put(p);
}
If instead you want a more scalable version, take a look at how to use map/reduce to read from input hdfs files / write to output hbase tables here: Hbase Map/Reduce
I have a .csv file with 177 rows and 18,000 odd cols.Given the column label, I should pick that particular column and as a default the first two label columns.
Please help me with this,
Thanks all,
Priya
So, what's the question? Parse CSV file. You can either implement this yourself or use third party code.
If you implement it yourself read line by line, split lines line.split(",") into elements and put it into data structure that should be a map of lists:
Map<String, List<String>> table = new LinkedHashMap<String, List<String>>();
Use column name as a key and column values as a list elements.
LinkedHashMap is preferable here to preserve the order of your columns.
Read first line that contains the column names and create list instances:
table.put(columnName, new LinkedList<String>());
Additionally create an array of column names:
String[] columns = new String[0];
table.keys().toArray();
Now continue iterating over your data and populate your table:
String[] data = line.split(",");
for (int i = 0; i < data.length; i++) {
table.get(columns[i]).add(data[i]);
}
TBD...
Good luck.
Have you looked into OpenCSV ?
You may go for OpenCSV