Related
I am using spark-sql-2.4.1v with java8.
I have scenario like below
List data = List(
("20", "score", "school", "2018-03-31", 14 , 12 , 20),
("21", "score", "school", "2018-03-31", 13 , 13 , 21),
("22", "rate", "school", "2018-03-31", 11 , 14, 22),
("21", "rate", "school", "2018-03-31", 13 , 12, 23)
)
Dataset<Row> df = = data.toDF("id", "code", "entity", "date", "column1", "column2" ,"column3")
Dataset<Row> resultDs = df
.withColumn("column_names",
array(Arrays.asList(df.columns()).stream().map(s -> new Column(s)).toArray(Column[]::new))
);
**But this is showing respective row columns values instread of column names.
so what is wrong here ? how to get "column_names" in java **
I am trying to solve below use-case:
Lets say i have 100 columns like column1....to column100 ... each column calculation would be different depend on the column name and data .... but every time i run my spark job i will get which columns i need to calculate ... but in my code i will have all columns logic i.e. each column logic might be different ... i need to ignore the logic of unspecified columns... but as the dataframe contain all columns i am selecting specified columns..so for non-selected columns my code throws exception as the column not found ...i need to fix this
Using play framework 2.0 and here goes my java code :
String queryString="SELECT watchDuration, date(startTime) from SEData";
Query query=JPA.em().createNativeQuery(queryString);
List<Object[]> resultHours = (List<Object[]>) query.getResultList();
Gson gson = new Gson();
String json = gson.toJson(resultHours);
renderJSON(json);
After browsing for a while, I did try to use Gson, which resulted me with the following output :
[[5.0,"Feb 5, 2014"],[6.0,"Feb 6, 2014"],[1.0,"Feb 7, 2014"],[2.0,"May 3, 2017"],[3.0,"May 4, 2017"]]
Since I'm fetching this data to plot on a c3.js graph, I need it in the following format :
json:[{"value":5, "date":"Feb 5, 2014"},{"value":6, "date":"Feb 6, 2014"},{"value":1, "date":"Feb 7, 2014"},{"value":2, "date":"May 3, 2017"},{"value":3, "date":"May 4, 2017"}]
OR
json: {
value:[5, 6, 1, 2, 3],
date: ["Feb 5, 2014", "Feb 6, 2014", "Feb 7, 2014", "May 3, 2017", "May 4, 2017"]
}
How can I achieve the above format retrieved MySQL database?
I doubt if my approach towards Gson is wrong, because the output that I got is not even a JSON I believe. Guide me towards the right approach if I'm not moving towards one.
Thanks.
The problem is gson doesn't know what the properties are called, so it makes an array of unnamed values.
While adding a new class will simplify things, a new class for every return type of a query means a lot of rather useless classes, especially if they are only used for marshalling.
Instead, you can map a name to each list of properties like so
HashMap<String, ArrayList<Object> > map = new HashMap<String, ArrayList<Object> >();
ArrayList<Object> values = new ArrayList<Object>();
ArrayList<Object> dates = new ArrayList<Object>();
for(int i=0; i < list.size(); i++){
values.add(resultHours.get(i)[0]);
dates.add(resultHours.get(i)[1]);
}
map.put("value", values);
map.put("date", dates);
This produces the desired output:
{
"date": ["Jan","Feb","Mar","April"],
"value": [1,2,3,4]
}
Rather than returning a list of Object[] create an object which is typed
public class ResultHours {
public int value;
public Date date;
}
and then update the getResultList();
List<ResultHours[]> resultHours = (List<ResultHours[]>) query.getResultList();
I've not tested this but in theory it should work!
Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 8 years ago.
Improve this question
I have this string
[23,22,17][17,2][23][3,29][][10,43,6][7][32,17,6][][][23,49,12][14,40,15][34,41,32][4,7,19][9,27][17][31,36,45][][32][40,27,25]
obtained from json and i saved it into ArrayList like this:
ArrayList<?> listAdress=(ArrayList<?>)jobj.get("adress");
I want to take only the numbers and save the numbers in brackets into a vector like this.
v[]={23,22,18}
v[]={17,2}
I tried to get only the numbers, but i dont know how to take the numbers untill you find ]
Someone know how to?
here is the regex you'll need for your problem :
(\d*,*)*
A link for explanation of this regex
here follows the java method to get the arrays of numbers :
public static List<String []> getNumberArrays (String toBeProcessed){
List<String[]> listOfArrays = new ArrayList<String[]>();
Pattern p = Pattern.compile("(\\d*,*)*");
Matcher m = p.matcher(toBeProcessed);
while(m.find()){
String[] a ;
a =m.group(0).split(",");
// next statement for avoiding the printing of empty arrays
if(a.length>=2)
listOfArrays.add(a);
}
return listOfArrays;
}
Test code :
String x = "[23,22,17][17,2][23][3,29][][10,43,6][7][32,17,6][][][23,49,12][14,40,15][34,41,32][4,7,19][9,27][17][31,36,45][][32][40,27,25]" ;
List<String[]> listOfArrays = new ArrayList<String[]>();
listOfArrays = getNumberArrays(x);
for(String[] a :listOfArrays){
System.out.println(Arrays.toString(a));
}
Output :
[23, 22, 17]
[17, 2]
[3, 29]
[10, 43, 6]
[32, 17, 6]
[23, 49, 12]
[14, 40, 15]
[34, 41, 32]
[4, 7, 19]
[9, 27]
[31, 36, 45]
[40, 27, 25]
What about this:
public static void main(String[] args) {
String testStr = "[23,22,17][17,2][23][3,29][][10,43,6][7][32,17,6][][][23,49,12][14,40,15][34,41,32][4,7,19][9,27][17][31,36,45][][32][40,27,25]";
ArrayList<String[]> result = new ArrayList<>();
String[] resTmp = testStr.split("\\[|\\]\\["); // First split input into vectors
for (String vecDef: resTmp) // Then split each vector into a String[]
result.add(vecDef.split(","));
for (String[] s : result) { // result = ArrayList with an element for each vector
for (String ss : s) // Each element is an array of Strings each being a number
System.out.print(ss + " ");
System.out.println();
}
}
I know you asked for a Regex but I'm not sure it's the only or the best way to go for such a simple parsing.
Here a quick (and not so safe) code:
public class HelloWorld{
public static void main(String []args){
String input = "[23,22,17][17,2][23][3,29][][10,43,6][7][32,17,6][][][23,49,12][14,40,15][34,41,32][4,7,19][9,27][17][31,36,45][][32][40,27,25]";
input = input.substring(1, input.length()-1);
String[] vectors = input.split("\\]\\[");
for(String vector : vectors)
{
System.out.println(String.format("\"%s\"", vector));
}
}
}
Output:
"23,22,17"
"17,2"
"23"
"3,29"
""
"10,43,6"
"7"
"32,17,6"
""
""
"23,49,12"
"14,40,15"
"34,41,32"
"4,7,19"
"9,27"
"17"
"31,36,45"
""
"32"
"40,27,25"
The thing is: you have to make sure that the string provided as an input is always well formatted (beginning with a [, ending with a ], and made of segments beginning with [ and ending with ]). Yet it's almost the same story with regular expressions (invalid input = no outputs, or partial outputs).
Once you have your strings with numbers separated by commas, the rest of the job is easy (you can split again and then parse to Integers).
public void importarCorreos() throws Exception{
#SuppressWarnings("deprecation")
ClientRequest cr = new ClientRequest("http://di002.edv.uniovi.es/~delacal/tew/1415/practica02/servicio_correos.php");
#SuppressWarnings("deprecation")
String result = cr.get(String.class).getEntity(String.class);
CorreosService service = Factories.services.createCorreosService();
//Imprimimos todo el flujo JSON recibido en formato cadena.
System.out.println(result);
//Procesamos el texto JSON y lo pasamos a formato SIMPLE-JSON
Object obj=JSONValue.parse(result);
JSONArray correos = (JSONArray)obj;
ListIterator li = correos.listIterator();
while(li.hasNext()){
JSONObject jobj =(JSONObject) li.next();
Correo c = new Correo();
c.setFechaHora( Long.parseLong(jobj.get("fechahora").toString()));
c.setAsunto(jobj.get("asunto").toString());
c.setCuerpo(jobj.get("cuerpo").toString());
c.setCarpeta( Integer.parseInt(jobj.get("carpeta").toString()));
c.setLogin_user(usuario.getLogin());
ArrayList<?> listaDestinatarios=(ArrayList<?>)jobj.get("destinatarios");
service.saveCorreo(c);
}
}
This is my function, mainly i obtained a json with mails from this url. I create a new mail with the fields from that url. But one of field from Mails class is mail_contacts where you should save the adresses from each contact like a vector [1,2,3] this is the id from the adress.
So how can i get the numbers into [ ], and save it into the fields mail_contacts what its a array.
I can save it like this:
c.setMailAdress(Here i want an array with the numbers from each [])
#ulix
Ok, this give the exit that i want:
00:53:20,413 INFO [stdout] (default task-6) 23 22 17
00:53:20,414 INFO [stdout] (default task-6) 17 2
00:53:20,414 INFO [stdout] (default task-6) 23
00:53:20,416 INFO [stdout] (default task-6) 3 29
00:53:20,416 INFO [stdout] (default task-6)
00:53:20,417 INFO [stdout] (default task-6) 10 43 6
But i want to save each position from string into an array of int, like int v[]={23,22,17}
The csv file contains more than one table, it might look like this:
"Vertical Table 1"
,
"id","visits","downloads"
1, 4324, 23
2, 664, 42
3, 73, 44
4, 914, 8
"Vertical Table 2"
,
"id_of_2nd_tab","visits_of_2nd_tab","downloads_of_2nd_tab"
1, 524, 3
2, 564, 52
3, 63, 84
4, 814, 8
To read one table I use "HeaderColumnNameTranslateMappingStrategy" from opencsv
which allows me to map the csv-table entries into a List of TableDataBean objects, as seen below:
HeaderColumnNameTranslateMappingStrategy<TableDataBean> strat = new HeaderColumnNameTranslateMappingStrategy<TableDataBean>();
CSVReader reader = new CSVReader(new FileReader(path), ',');
strat.setType(TableDataBean.class);
Map<String, String> map = new HashMap<String, String>();
map.put("Number of visits", "visits");
map.put("id", "id");
map.put("Number of downloads", "downloads");
strat.setColumnMapping(map);
CsvToBean<TableDataBean> csv = new CsvToBean<TableDataBean>();
List<TableDataBean> list = csv.parse(strat, reader);
This works fine for the first table, but when it cames to the second, the values and the attributes are mapped to the same attribute of the first table. The output for
for(TableDataBean bean : list){System.out.println(bean.getVisits());}
would look like this:
4324
664
73
914
null
null
null
visits_of_2nd_tab
524
564
63
814
I don't wanna split the file into many files containing each of them one table.
So what do you suggest ? Is there any other Library that supports this format?
I've got it! I thought that the type of reader have to be of CSVReader. It actually turned out that I can feed the methode parse with any object inheriting from the Reader class.
Now I can read the entire csv-file into a String, splitt it, pack each of the new Strings into a StringReader and than pass it to the parse methode.
How can I convert unix time to Hindu calendarWikipedia time and the other way round in php, Perl or Python or Java? I know I can convert to Hebrew and Jewish. But Hindu is not an option.
To be more specific, I'm talking about the Hindu lunar calendar. The following website is working and does exactly what I want: http://web.meson.org/calendars/. For example, it 'translates' 28-1-2012 (Gregorian) to 5-11-2068 (Hind. Lun.). How can I accomplish the same task? And if there are absolutely no scrips out there, how can I write it myself?
Did you check DateTime-Indic-0.1 family of modules? At least DateTime::Indic::Chandramana
seems to have a method to convert traditional date into UTC values (utc_rd_values).
UPDATE:
I suppose Calendar::Saka may be useful as well for many users (as I have known, it's the Indian national calendar), in particular, to_gregorian() and from_gregorian() methods.
For Python, use calendar2 (note: this is not the built-in calendar module).
Sample use:
>>> from calendar2 import *
>>> old_hindu_solar_from_absolute(absolute_from_gregorian(3,1,2012))
(11, 16, 5112)
>>> old_hindu_lunar_from_absolute(absolute_from_gregorian(3,1,2012))
(12, 0, 8, 5112)
Paper: Indian Calendrical Calculations
Provides Common Lisp code in the appendix.
While a Python (or other language) solution could be written according to the paper, the authors enumerate the Indian calendar rules pretty well, so it's a pretty solid paper if you're willing to consider taking the provided Common Lisp code.
Seems to be a difficult task. According to this discussion at bytes.com there is no clear way to accomplish a 100% correct conversion. But it seems that they are wrong when they assume that the Hindu calendar has only 364 days instead of 365 (or 366 in leap years).
Here you can find a good conversion table including the handling of leap years: http://hinduism.about.com/od/basics/a/monthsdayseras.htm
If it is as easy as written there you can try something like this (php code):
<?php
function convertDateToHinduDate($date) {
$beginningDayOfMonth = array(
1 => 21,
2 => 20,
3 => 22 + (dateIsLeapYear($date) ? -1 : 0), /* 21 in leap years */
4 => 21,
5 => 22,
6 => 22,
7 => 23,
8 => 23,
9 => 23,
10 => 23,
11 => 22,
12 => 22,
);
$daysOfHinduMonth = array(
1 => 30 + (dateIsLeapYear($date) ? 1 : 0), /* 31 in leap years */
2 => 31,
3 => 31,
4 => 31,
5 => 31,
6 => 31,
7 => 30,
8 => 30,
9 => 30,
10 => 30,
11 => 30,
12 => 30,
);
$day = (int) date('d', strtotime($date));
$month = (int) date('m', strtotime($date));
$year = (int) date('Y', strtotime($date));
$monthBefore = $day < $beginningDayOfMonth[$month];
$yearBefore = $month < 3 || ($month == 3 && $day < $beginningDayOfMonth[3]);
$newYear = $year + 57 + ($yearBefore ? -1 : 0);
$newMonth = $month - 2 + ($monthBefore ? -1 : 0);
if($newMonth < 1) $newMonth = 12 + $newMonth;
$newDay = $day - $beginningDayOfMonth[$month];
if($newDay < 1) $newDay = $daysOfHinduMonth[$newMonth] + $newDay;
return date("d-m-Y", mktime(11, 59, 0, $newMonth, $newDay, $newYear));
}
function dateIsLeapYear($date) {
return date('L', strtotime($date));
}
$date = date("d-m-Y", strtotime('2012-01-28'));
echo 'Date: ', $date, ' (is leap year: ', dateIsLeapYear($date) ? 'yes' : 'no', ')<br />';
echo 'Converted Hindu date: ', convertDateToHinduDate($date);
?>
Output of this code:
Date: 28-01-2012 (is leap year: yes)
Converted Hindu date: 07-11-2068
But according to the calculator of this Java applet, it should be 05-11-2068 instead of 07-11-2068. So, there are still some conversion rules missing. Maybe you can give me some more information so that i can correct the code above.
I dont know whether it is correct approach or not but
Please go to http://calendarhome.com/converter/ site and download two javascript files astro.js and calendar.js and follow onchange events of Gregorian Date and fetch Indian Civil Date parameters.