The csv file contains more than one table, it might look like this:
"Vertical Table 1"
,
"id","visits","downloads"
1, 4324, 23
2, 664, 42
3, 73, 44
4, 914, 8
"Vertical Table 2"
,
"id_of_2nd_tab","visits_of_2nd_tab","downloads_of_2nd_tab"
1, 524, 3
2, 564, 52
3, 63, 84
4, 814, 8
To read one table I use "HeaderColumnNameTranslateMappingStrategy" from opencsv
which allows me to map the csv-table entries into a List of TableDataBean objects, as seen below:
HeaderColumnNameTranslateMappingStrategy<TableDataBean> strat = new HeaderColumnNameTranslateMappingStrategy<TableDataBean>();
CSVReader reader = new CSVReader(new FileReader(path), ',');
strat.setType(TableDataBean.class);
Map<String, String> map = new HashMap<String, String>();
map.put("Number of visits", "visits");
map.put("id", "id");
map.put("Number of downloads", "downloads");
strat.setColumnMapping(map);
CsvToBean<TableDataBean> csv = new CsvToBean<TableDataBean>();
List<TableDataBean> list = csv.parse(strat, reader);
This works fine for the first table, but when it cames to the second, the values and the attributes are mapped to the same attribute of the first table. The output for
for(TableDataBean bean : list){System.out.println(bean.getVisits());}
would look like this:
4324
664
73
914
null
null
null
visits_of_2nd_tab
524
564
63
814
I don't wanna split the file into many files containing each of them one table.
So what do you suggest ? Is there any other Library that supports this format?
I've got it! I thought that the type of reader have to be of CSVReader. It actually turned out that I can feed the methode parse with any object inheriting from the Reader class.
Now I can read the entire csv-file into a String, splitt it, pack each of the new Strings into a StringReader and than pass it to the parse methode.
Related
I am using spark-sql-2.4.1v with java8.
I have scenario like below
List data = List(
("20", "score", "school", "2018-03-31", 14 , 12 , 20),
("21", "score", "school", "2018-03-31", 13 , 13 , 21),
("22", "rate", "school", "2018-03-31", 11 , 14, 22),
("21", "rate", "school", "2018-03-31", 13 , 12, 23)
)
Dataset<Row> df = = data.toDF("id", "code", "entity", "date", "column1", "column2" ,"column3")
Dataset<Row> resultDs = df
.withColumn("column_names",
array(Arrays.asList(df.columns()).stream().map(s -> new Column(s)).toArray(Column[]::new))
);
**But this is showing respective row columns values instread of column names.
so what is wrong here ? how to get "column_names" in java **
I am trying to solve below use-case:
Lets say i have 100 columns like column1....to column100 ... each column calculation would be different depend on the column name and data .... but every time i run my spark job i will get which columns i need to calculate ... but in my code i will have all columns logic i.e. each column logic might be different ... i need to ignore the logic of unspecified columns... but as the dataframe contain all columns i am selecting specified columns..so for non-selected columns my code throws exception as the column not found ...i need to fix this
I have a form that allows me to input data about a single Item. Every time someone submits an Item, I want to add it to a JSON array, which is stored in a file.
Here's my code:
for (Item obj : list) {
out.print(obj.getId());
out.println("");
out.print(obj.getProductName());
out.println("");
out.print(obj.getPrice());
out.println("");
out.print(obj.getType());
out.println("");
}
ObjectMapper mapper = new ObjectMapper();
File file=new File("D:\\extern_2\\src\\java\\JSON\\jsonlist.json");
if (!file.exists()) {
file.createNewFile();
}
PrintWriter print = new PrintWriter(new BufferedWriter(new FileWriter(file, true)));
mapper.configure(SerializationFeature.INDENT_OUTPUT, true);
mapper.enable(SerializationFeature.INDENT_OUTPUT);
mapper.writerWithDefaultPrettyPrinter().writeValue(print, list);
The problem is that every time I add a new Item, a new JSON array is created and appended to the existing file contents.
Desired output:
[ {
"id" : 56,
"productname" : "kklll",
"price" : "56",
"type" : "Hot Coffee",
"productName" : "kklll"
} , {
"id" : 89,
"productname" : "llll",
"price" : "43",
"type" : "Drinks",
"productName" : "llll"
} ]
Actual output:
[ {
"id" : 56,
"productname" : "kklll",
"price" : "56",
"type" : "Hot Coffee",
"productName" : "kklll"
} ][ {
"id" : 89,
"productname" : "llll",
"price" : "43",
"type" : "Drinks",
"productName" : "llll"
} ]
Why is it appending a new array instead of adding my new Item to the existing array?
Look at the FileWriter you are creating: new FileWriter(file, true). That second parameter tells the FileWriter to simply append information to the end of the file. If you are modifying existing JSON, you will need to overwrite the file every time. This means that the first time you create an Item, the ObjectMapper will write it out as a valid JSON string, representing an array with a single object. The second time you create an Item, it will do the same thing for your new object, creating an array with only one object (the second Item) and writing it to the file, even though that file already contains an array. At no point are you actually looking at the file to see if it contains any existing data. You are also not parsing your file into JSON, which would allow you to take an existing JSON array and add something to it.
Your process should be like this:
Read in your existing data in your file, using the ObjectMapper. Since your file contains an array of Item objects, you should end up with a List<Item> after you've read in your file
Add your new Item to the List
Convert your List<Item> to JSON and write it to your .json file. Make sure to overwrite your .json file, not just append to it.
Using play framework 2.0 and here goes my java code :
String queryString="SELECT watchDuration, date(startTime) from SEData";
Query query=JPA.em().createNativeQuery(queryString);
List<Object[]> resultHours = (List<Object[]>) query.getResultList();
Gson gson = new Gson();
String json = gson.toJson(resultHours);
renderJSON(json);
After browsing for a while, I did try to use Gson, which resulted me with the following output :
[[5.0,"Feb 5, 2014"],[6.0,"Feb 6, 2014"],[1.0,"Feb 7, 2014"],[2.0,"May 3, 2017"],[3.0,"May 4, 2017"]]
Since I'm fetching this data to plot on a c3.js graph, I need it in the following format :
json:[{"value":5, "date":"Feb 5, 2014"},{"value":6, "date":"Feb 6, 2014"},{"value":1, "date":"Feb 7, 2014"},{"value":2, "date":"May 3, 2017"},{"value":3, "date":"May 4, 2017"}]
OR
json: {
value:[5, 6, 1, 2, 3],
date: ["Feb 5, 2014", "Feb 6, 2014", "Feb 7, 2014", "May 3, 2017", "May 4, 2017"]
}
How can I achieve the above format retrieved MySQL database?
I doubt if my approach towards Gson is wrong, because the output that I got is not even a JSON I believe. Guide me towards the right approach if I'm not moving towards one.
Thanks.
The problem is gson doesn't know what the properties are called, so it makes an array of unnamed values.
While adding a new class will simplify things, a new class for every return type of a query means a lot of rather useless classes, especially if they are only used for marshalling.
Instead, you can map a name to each list of properties like so
HashMap<String, ArrayList<Object> > map = new HashMap<String, ArrayList<Object> >();
ArrayList<Object> values = new ArrayList<Object>();
ArrayList<Object> dates = new ArrayList<Object>();
for(int i=0; i < list.size(); i++){
values.add(resultHours.get(i)[0]);
dates.add(resultHours.get(i)[1]);
}
map.put("value", values);
map.put("date", dates);
This produces the desired output:
{
"date": ["Jan","Feb","Mar","April"],
"value": [1,2,3,4]
}
Rather than returning a list of Object[] create an object which is typed
public class ResultHours {
public int value;
public Date date;
}
and then update the getResultList();
List<ResultHours[]> resultHours = (List<ResultHours[]>) query.getResultList();
I've not tested this but in theory it should work!
I have following RDD in my Java Code.
(1, List(1596, 1617, 1929, 2399, 2674))
(2, List(1702, 1785, 1933, 2054, 2583, 2913))
(3, List(1982, 2002, 2048, 2341, 2666))
What I am trying to do is to create another RDD. The contents should look like this.(not necessarily in same order)
1596
1617
1929
2399
2674
1702
1785
1933
2054
2583
2913
1982
2002
2048
2341
2666
I am not sure how do transform one RDD (JavaRDD<ArrayList<String>>) with collection of Objects to single RDD (JavaRDD<String>) with all objects in it. I would highly appreciate if anyone could point me to some JAVA resource.
You can do the same in scala as follows
val data = List((1, List(1596, 1617, 1929, 2399, 2674)),
(2, List(1702, 1785, 1933, 2054, 2583, 2913)),
(3, List(1982, 2002, 2048, 2341, 2666)))
val rdd_data = sc.parallelize(data)
val rdd_flattened = rdd_data.flatMap((index, value) => value)
Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 8 years ago.
Improve this question
I have this string
[23,22,17][17,2][23][3,29][][10,43,6][7][32,17,6][][][23,49,12][14,40,15][34,41,32][4,7,19][9,27][17][31,36,45][][32][40,27,25]
obtained from json and i saved it into ArrayList like this:
ArrayList<?> listAdress=(ArrayList<?>)jobj.get("adress");
I want to take only the numbers and save the numbers in brackets into a vector like this.
v[]={23,22,18}
v[]={17,2}
I tried to get only the numbers, but i dont know how to take the numbers untill you find ]
Someone know how to?
here is the regex you'll need for your problem :
(\d*,*)*
A link for explanation of this regex
here follows the java method to get the arrays of numbers :
public static List<String []> getNumberArrays (String toBeProcessed){
List<String[]> listOfArrays = new ArrayList<String[]>();
Pattern p = Pattern.compile("(\\d*,*)*");
Matcher m = p.matcher(toBeProcessed);
while(m.find()){
String[] a ;
a =m.group(0).split(",");
// next statement for avoiding the printing of empty arrays
if(a.length>=2)
listOfArrays.add(a);
}
return listOfArrays;
}
Test code :
String x = "[23,22,17][17,2][23][3,29][][10,43,6][7][32,17,6][][][23,49,12][14,40,15][34,41,32][4,7,19][9,27][17][31,36,45][][32][40,27,25]" ;
List<String[]> listOfArrays = new ArrayList<String[]>();
listOfArrays = getNumberArrays(x);
for(String[] a :listOfArrays){
System.out.println(Arrays.toString(a));
}
Output :
[23, 22, 17]
[17, 2]
[3, 29]
[10, 43, 6]
[32, 17, 6]
[23, 49, 12]
[14, 40, 15]
[34, 41, 32]
[4, 7, 19]
[9, 27]
[31, 36, 45]
[40, 27, 25]
What about this:
public static void main(String[] args) {
String testStr = "[23,22,17][17,2][23][3,29][][10,43,6][7][32,17,6][][][23,49,12][14,40,15][34,41,32][4,7,19][9,27][17][31,36,45][][32][40,27,25]";
ArrayList<String[]> result = new ArrayList<>();
String[] resTmp = testStr.split("\\[|\\]\\["); // First split input into vectors
for (String vecDef: resTmp) // Then split each vector into a String[]
result.add(vecDef.split(","));
for (String[] s : result) { // result = ArrayList with an element for each vector
for (String ss : s) // Each element is an array of Strings each being a number
System.out.print(ss + " ");
System.out.println();
}
}
I know you asked for a Regex but I'm not sure it's the only or the best way to go for such a simple parsing.
Here a quick (and not so safe) code:
public class HelloWorld{
public static void main(String []args){
String input = "[23,22,17][17,2][23][3,29][][10,43,6][7][32,17,6][][][23,49,12][14,40,15][34,41,32][4,7,19][9,27][17][31,36,45][][32][40,27,25]";
input = input.substring(1, input.length()-1);
String[] vectors = input.split("\\]\\[");
for(String vector : vectors)
{
System.out.println(String.format("\"%s\"", vector));
}
}
}
Output:
"23,22,17"
"17,2"
"23"
"3,29"
""
"10,43,6"
"7"
"32,17,6"
""
""
"23,49,12"
"14,40,15"
"34,41,32"
"4,7,19"
"9,27"
"17"
"31,36,45"
""
"32"
"40,27,25"
The thing is: you have to make sure that the string provided as an input is always well formatted (beginning with a [, ending with a ], and made of segments beginning with [ and ending with ]). Yet it's almost the same story with regular expressions (invalid input = no outputs, or partial outputs).
Once you have your strings with numbers separated by commas, the rest of the job is easy (you can split again and then parse to Integers).
public void importarCorreos() throws Exception{
#SuppressWarnings("deprecation")
ClientRequest cr = new ClientRequest("http://di002.edv.uniovi.es/~delacal/tew/1415/practica02/servicio_correos.php");
#SuppressWarnings("deprecation")
String result = cr.get(String.class).getEntity(String.class);
CorreosService service = Factories.services.createCorreosService();
//Imprimimos todo el flujo JSON recibido en formato cadena.
System.out.println(result);
//Procesamos el texto JSON y lo pasamos a formato SIMPLE-JSON
Object obj=JSONValue.parse(result);
JSONArray correos = (JSONArray)obj;
ListIterator li = correos.listIterator();
while(li.hasNext()){
JSONObject jobj =(JSONObject) li.next();
Correo c = new Correo();
c.setFechaHora( Long.parseLong(jobj.get("fechahora").toString()));
c.setAsunto(jobj.get("asunto").toString());
c.setCuerpo(jobj.get("cuerpo").toString());
c.setCarpeta( Integer.parseInt(jobj.get("carpeta").toString()));
c.setLogin_user(usuario.getLogin());
ArrayList<?> listaDestinatarios=(ArrayList<?>)jobj.get("destinatarios");
service.saveCorreo(c);
}
}
This is my function, mainly i obtained a json with mails from this url. I create a new mail with the fields from that url. But one of field from Mails class is mail_contacts where you should save the adresses from each contact like a vector [1,2,3] this is the id from the adress.
So how can i get the numbers into [ ], and save it into the fields mail_contacts what its a array.
I can save it like this:
c.setMailAdress(Here i want an array with the numbers from each [])
#ulix
Ok, this give the exit that i want:
00:53:20,413 INFO [stdout] (default task-6) 23 22 17
00:53:20,414 INFO [stdout] (default task-6) 17 2
00:53:20,414 INFO [stdout] (default task-6) 23
00:53:20,416 INFO [stdout] (default task-6) 3 29
00:53:20,416 INFO [stdout] (default task-6)
00:53:20,417 INFO [stdout] (default task-6) 10 43 6
But i want to save each position from string into an array of int, like int v[]={23,22,17}