Getting Null values using WindowSpec in spark/java - java

I have a dataframe :
+----------+------------+------------+--------------------+
| acc |id_Vehicule |id_Device |dateTracking |
+----------+------------+------------+--------------------+
| 1 | 1 | 2 |2020-02-12 14:50:00 |
| 0 | 1 | 2 |2020-02-12 14:59:00 |
| 0 | 2 | 3 |2020-02-12 15:10:00 |
| 1 | 2 | 3 |2020-02-12 15:20:00 |
+----------+------------+------------+--------------------+
I want to get in output :
+----------+------------+------------+--------------------+----------------+
| acc |id_Vehicule |id_Device |dateTracking | acc_previous |
+----------+------------+------------+--------------------+----------------+
| 1 | 1 | 2 |2020-02-12 14:50:00 | null |
| 0 | 1 | 2 |2020-02-12 14:59:00 | 1 |
| 0 | 2 | 3 |2020-02-12 15:10:00 | null |
| 1 | 2 | 3 |2020-02-12 15:20:00 | 0 |
+----------+------------+------------+--------------------+----------------+
I tried the following code :
WindowSpec w =org.apache.spark.sql.expressions.Window.partitionBy("idVehicule","idDevice","dateTracking").orderBy("dateTracking");
Dataset <Row> df= df1.withColumn("acc_previous",lag("acc",1).over(w));
df.show();
I get on result ;
+----------+------------+------------+--------------------+----------------+
| acc |id_Vehicule |id_Device |dateTracking | acc_previous |
+----------+------------+------------+--------------------+----------------+
| 1 | 1 | 2 |2020-02-12 14:50:00 | null |
| 0 | 1 | 2 |2020-02-12 14:59:00 | null |
| 0 | 2 | 3 |2020-02-12 15:10:00 | null |
| 1 | 2 | 3 |2020-02-12 15:20:00 | null |
+----------+------------+------------+--------------------+----------------+
If you have any idea
I will be very grateful

I found the solution, maybe it will help another person.
The problem was because of the "dateTracking" column it should not be like a partitioning column, so I removed it.
WindowSpec w =org.apache.spark.sql.expressions.Window.partitionBy("idVehicule","idDevice").orderBy("dateTracking");
Dataset <Row> df= df1.withColumn("acc_previous",lag("acc",1).over(w));
df.show();

Related

what is the unique id of informix temp table?

I create a temp table which consist of several table with UNION ALL statement like here below. I want later map this table to the entity for repository in spring. With other words I wanna map temp table to entity in spring jpa or hibernate.
select * from name UNION ALL
select * from soft where id >3
into temp namesoft_tmp
I tried the following.
select * from namesoft_tmp
but i can't see what is the column which can point me to the conclusion that this is primary key.
What is the unique id(primary key) of table namesoft_tmp?
How can i add auto generated id to temp table?
How can i excute select statement based on unique id?**
In general, the result of a UNION ALL query does not have a primary key; there is no guarantee that there are not duplicate rows in the result set.
Imagine a table describing the table of elements — called elements.
SELECT * FROM elements WHERE atomic_number < 10
UNION ALL
SELECT * FROM elements WHERE symbol MATCHES '[A-F]*'
INTO TEMP union_all;
Here, the elements Boron (B), Carbon (C), Beryllium (Be) and Fluorine (F) are all listed twice.
However, you can use:
SELECT ROWID, * FROM union_all ORDER BY atomic_number;
to get a unique identifier, the ROWID, in the result set. Note that this unique identifier is unique at any given time, but is not guaranteed to be stable. If you delete rows and add them again, the ROWID of the replaced rows may be different from before. But the ROWID will be unique until you edit the table.
+-------+--------+--------+--------------+-----------+--------+-------+
| rowid | atomic | symbol | name | atomic | period | group |
| | number | | | weight | | |
+-------+--------+--------+--------------+-----------+--------+-------+
| 257 | 1 | H | Hydrogen | 1.0079 | 1 | 1 |
| 258 | 2 | He | Helium | 4.0026 | 1 | 18 |
| 259 | 3 | Li | Lithium | 6.9410 | 2 | 1 |
| 260 | 4 | Be | Beryllium | 9.0122 | 2 | 2 |
| 266 | 4 | Be | Beryllium | 9.0122 | 2 | 2 |
| 267 | 5 | B | Boron | 10.8110 | 2 | 13 |
| 261 | 5 | B | Boron | 10.8110 | 2 | 13 |
| 268 | 6 | C | Carbon | 12.0110 | 2 | 14 |
| 262 | 6 | C | Carbon | 12.0110 | 2 | 14 |
| 263 | 7 | N | Nitrogen | 14.0070 | 2 | 15 |
| 264 | 8 | O | Oxygen | 15.9990 | 2 | 16 |
| 265 | 9 | F | Fluorine | 18.9980 | 2 | 17 |
| 269 | 9 | F | Fluorine | 18.9980 | 2 | 17 |
| 270 | 13 | Al | Aluminium | 26.9820 | 3 | 13 |
| 271 | 17 | Cl | Chlorine | 35.4530 | 3 | 17 |
| 272 | 18 | Ar | Argon | 39.9480 | 3 | 18 |
| 273 | 20 | Ca | Calcium | 40.0780 | 4 | 2 |
| 274 | 24 | Cr | Chromium | 51.9960 | 4 | 6 |
| 275 | 26 | Fe | Iron | 55.8450 | 4 | 8 |
| 276 | 27 | Co | Cobalt | 58.9330 | 4 | 9 |
| 277 | 29 | Cu | Copper | 63.5460 | 4 | 11 |
| 278 | 33 | As | Arsenic | 74.9220 | 4 | 15 |
| 279 | 35 | Br | Bromine | 79.9040 | 4 | 17 |
| 280 | 47 | Ag | Silver | 107.8700 | 5 | 11 |
| 281 | 48 | Cd | Cadmium | 112.4100 | 5 | 12 |
| 282 | 55 | Cs | Caesium | 132.9100 | 6 | 1 |
| 283 | 56 | Ba | Barium | 137.3300 | 6 | 2 |
| 284 | 58 | Ce | Cerium | 140.1200 | 6 | L |
| 285 | 63 | Eu | Europium | 151.9600 | 6 | L |
| 286 | 66 | Dy | Dyprosium | 162.5000 | 6 | L |
| 287 | 68 | Er | Erbium | 167.2600 | 6 | L |
| 288 | 79 | Au | Gold | 196.9700 | 6 | 11 |
| 289 | 83 | Bi | Bismuth | 208.9800 | 6 | 15 |
| 290 | 85 | At | Astatine | 209.9900 | 6 | 17 |
| 291 | 87 | Fr | Francium | 223.0200 | 7 | 1 |
| 292 | 89 | Ac | Actinium | 227.0300 | 7 | A |
| 293 | 95 | Am | Americium | 243.0600 | 7 | A |
| 294 | 96 | Cm | Curium | 247.0700 | 7 | A |
| 295 | 97 | Bk | Berkelium | 247.0700 | 7 | A |
| 296 | 98 | Cf | Californium | 251.0800 | 7 | A |
| 297 | 99 | Es | Einsteinium | 252.0800 | 7 | A |
| 298 | 100 | Fm | Fermium | 257.1000 | 7 | A |
| 299 | 105 | Db | Dubnium | 270.1300 | 7 | 5 |
| 300 | 107 | Bh | Bohrium | 270.1300 | 7 | 7 |
| 301 | 110 | Ds | Darmstadtium | 281.1700 | 7 | 10 |
| 302 | 112 | Cn | Copernicium | 285.1800 | 7 | 12 |
| 303 | 114 | Fl | Flerovium | 289.1900 | 7 | 14 |
+-------+--------+--------+--------------+-----------+--------+-------+

Iterating through lists of list in java

I have two tables User and Roles with one-to-one relation as below.
User
_________________________________________
| Id | user_name | full_name | creator |
_________________________________________
| 1 | a | A | a |
| 2 | b | B | a |
| 3 | c | C | a |
| 4 | d | D | c |
| 5 | e | E | c |
| 6 | f | F | e |
| 7 | g | G | e |
| 8 | h | H | e |
| 9 | i | I | e |
|10 | j | J | i |
_________________________________________
Roles
_______________________________________
| id | user_mgmt | others | user_id |
_______________________________________
| 1 | 1 | 1 | 1 |
| 2 | 0 | 1 | 2 |
| 3 | 1 | 0 | 3 |
| 4 | 0 | 1 | 4 |
| 5 | 1 | 1 | 5 |
| 6 | 0 | 1 | 6 |
| 7 | 0 | 0 | 7 |
| 8 | 0 | 0 | 8 |
| 9 | 1 | 0 | 9 |
________________________________________
The Roles table have boolean columns, so if an User have user_mgmt role he can add many users (How many users can be added by an user is not definite). I want to fetch all users created by an user and his child users ( a is parent, c is child of a and e is child of c ..) .
Here is my code to fetch users
public void loadUsers(){
List<Users> users = new ArrayList<>();
String creator = user.getUserName();
List<Users> createdUsers = userService.getUsersByCreator(creator);
for(Users user : createdUsers) {
Roles role = createdUsers.getRoles();
if(role.isEmpMgnt()){
users.add(user);
loadUsers();
}
}
}
This gives me an stack overflow error. If i don't call loadUsers() recursively it returns only a single child result. So is there any solutions to this ? Thanks for any help in advance.
This gives you stack overflow error because a has creator a. So you have infinite loop for user a. For a you should set creator to null or skip self references in code.
Also you should pass current user into loadUsers() method and read only users that are created by it. Like
public void loadUsers(String creator)
and only process users created by that creator. Here
String creator = user.getUserName();
what is user? You should use creator. Question is how do you obtain initial creator. Probably initial creator should be user where creator is null.

How to get an encrypted token minimized?

I have generated an encrypted token from Blowfish. Eg:- 7$127O$137kI$137mK$07WK$01$26m$05zYbJmCmUw$16nF$11G$27A2Gv$19Jm8$26eJ9kUv$07$118q$05$02$24KP8j$208$16$06$100P$11
Just out of curiosity , can I get this token more minimized/simplify/striped somehow. Can anything to be done in order to get this more so simplified , may be any encoding. Please help me out with this.
Thanks
Huffman has a static cost, the Huffman table. On the other side, algorithms of the Lempel-Ziv family have very good compression and performance.
If your string is going to be english text you better use Smaz: https://github.com/antirez/smaz/tree/master
Try Huffman encoding with your string. For the string 7$127O$137kI$137mK$07WK$01$26m$05zYbJmCmUw$16nF$11G$27A2Gv$19Jm8$26eJ9kUv$07$118q$05$02$24KP8j$208$16$06$100P$11
the compression ratio is 0.536830357143.
Analysis
Memory requirements: ASCII: 896 bit , Huffman: 593 bit
Entropy: ASCII: 1.01833029706, Huffman: 1.05223637886
Average code length: ASCII: 8 bit , Huffman: 4.29464285714 bit
Compression rate: 0.536830357143
+---------+-------+-------+-----------+---------+---------+
| Seq.no. | Chars | ASCII | Frequency | Huffman | ASCII |
+---------+-------+-------+-----------+---------+---------+
| 0 | '$' | 36 | 22 | 0 | 100100 |
| 1 | '0' | 48 | 10 | 1111 | 110000 |
| 2 | '1' | 49 | 14 | 11 | 110001 |
| 3 | '2' | 50 | 8 | 1011 | 110010 |
| 4 | '3' | 51 | 2 | 110100 | 110011 |
| 5 | '4' | 52 | 1 | 101111 | 110100 |
| 6 | '5' | 53 | 2 | 111001 | 110101 |
| 7 | '6' | 54 | 5 | 100 | 110110 |
| 8 | '7' | 55 | 7 | 1000 | 110111 |
| 9 | '8' | 56 | 4 | 11011 | 111000 |
| 10 | '9' | 57 | 2 | 110011 | 111001 |
| 11 | 'A' | 65 | 1 | 1100001 | 1000001 |
| 12 | 'C' | 67 | 1 | 1100000 | 1000011 |
| 13 | 'F' | 70 | 1 | 1001100 | 1000110 |
| 14 | 'G' | 71 | 2 | 110101 | 1000111 |
| 15 | 'I' | 73 | 1 | 101110 | 1001001 |
| 16 | 'J' | 74 | 3 | 1010 | 1001010 |
| 17 | 'K' | 75 | 3 | 10010 | 1001011 |
| 18 | 'O' | 79 | 1 | 1010001 | 1001111 |
| 19 | 'P' | 80 | 2 | 110010 | 1010000 |
| 20 | 'U' | 85 | 2 | 101010 | 1010101 |
| 21 | 'W' | 87 | 1 | 1100010 | 1010111 |
| 22 | 'Y' | 89 | 1 | 1001101 | 1011001 |
| 23 | 'b' | 98 | 1 | 1010000 | 1100010 |
| 24 | 'e' | 101 | 1 | 1100011 | 1100101 |
| 25 | 'j' | 106 | 1 | 10110 | 1101010 |
| 26 | 'k' | 107 | 2 | 101001 | 1101011 |
| 27 | 'm' | 109 | 5 | 11101 | 1101101 |
| 28 | 'n' | 110 | 1 | 1001110 | 1101110 |
| 29 | 'q' | 113 | 1 | 1001111 | 1110001 |
| 30 | 'v' | 118 | 2 | 111000 | 1110110 |
| 31 | 'w' | 119 | 1 | 1010110 | 1110111 |
| 32 | 'z' | 122 | 1 | 1010111 | 1111010 |
+---------+-------+-------+-----------+---------+---------+

Generate Excel having Formula using Apache POI

In the example below how to add the formula in individual user totals as the number of rows for the user can vary.
+------+--------------+---------+--------------+-------+
| Name | Date | Billable| Non-Billable | Total |
+------+--------------+---------+--------------+-------+
| abc | 06/23/2012 | 860 | 10 | 870 |
| | User Totals: | 860 | 10 | 870 |
| xyz | 07/12/2012 | 45 | 0 | 45 |
| | User Totals: | 45 | 0 | 45 |
| ccc | 09/19/2013 | 165 | 35 | 200 |
| | 10/15/2013 | 240 | 0 | 240 |
| | User Totals: | 405 | 35 | 440 |
| | Grand Totals | 1310| 45 | 1355 |
+------+--------------+---------+--------------+-------+

How to make JSON having loop inside lopp

I have following table client_question table
+----+------------+---------+-----+------+------+
| id | is_deleted | version | cid | pqid | qtid |
+----+------------+---------+-----+------+------+
| 1 | | 0 | 1 | 1 | 1 |
| 2 | | 0 | 1 | 2 | 4 |
| 3 | | 0 | 1 | 2 | 4 |
+----+------------+---------+-----+------+------+
This is Parent_question table
+----+------------+---------+-----+------+
| id | is_deleted | version | pid | qid |
+----+------------+---------+-----+------+
| 1 | | 0 | 1 | 1 |
| 2 | | 0 | 1 | 2 |
| 3 | | 0 | 1 | 3 |
| 4 | | 0 | 1 | 4 |
| 5 | | 0 | 1 | 5 |
| 6 | | 0 | 1 | 6 |
| 7 | | 0 | 2 | 7 |
| 8 | | 0 | 2 | 1 |
| 9 | | 0 | 2 | 2 |
| 10 | | 0 | 2 | 8 |
| 11 | | 0 | 3 | 9 |
| 12 | | 0 | 3 | 1 |
| 13 | | 0 | 3 | 10 |
| 14 | | 0 | 3 | 11 |
| 15 | | 0 | 4 | 12 |
+----+------------+---------+-----+------+
And this is question_option
+----+------------+-----------+---------+
| id | is_deleted | name | version |
+----+------------+-----------+---------+
| 1 | | Excellent | 0 |
| 2 | | Good | 0 |
| 3 | | Fair | 0 |
| 4 | | Poor | 0 |
+----+------------+-----------+---------+
I want to retrieve JSON and send to front end via ajax so
I tried this way
public List<ClientQuestionOption> getSavedQuestionOptions(Long parentId,long clientId)
{
Client client = (Client) entityManagerUtil.find(Client.class, clientId);
List<ClientQuestionOption> questionsList = (List<ClientQuestionOption>)serviceClientDaoImpl.getSavedQuestionOptionsList(parentId,client);
System.out.println("The size is nnnnnnnnnn "+questionsList.size());
List optionsList =new ArrayList();
for(int i=0;i<questionsList.size();i++)
{
//optionsList.add(questionsList.get(i).getCqid().getId());
//optionsList.add(questionsList.get(i).getOid().getName());
Map map=new HashMap();
map.put("qid", questionsList.get(i).getCqid().getPqid().getQid().getId());
map.put("name", questionsList.get(i).getOid().getName());
optionsList.add(map);
}
return optionsList;
}
The JSON i got is like this
[
{
name: "Excellent",
qid: 2
},
{
name: "Poor",
qid: 2
}
],
But I want JSON like this
{
"options": [
"Poot",
"Excellent"
],
"qid": 2
},
Can anybody please tell me how to do so?
edit
Following way I am doing to make JSON
JSONObject object=new JSONObject();
List optionslist=null;
optionslist=(List<ClientQuestionOption>)serviceClientServiceImpl.getSavedQuestionOptions(parentId , Long.valueOf(clientId) );
object.accumulate("optionslist",optionslist );
return object.toString();
First, realise what the function returns (ArrayList) is not JSON at all. It seems that happens elsewhere. You'll need to replace JsonObject() with JsonArray() to change the [..] to '{..}`.
Second, to do the grouping per qid you could write a function that has List<ClientQuestionOption> as input ánd output.
You should check google GSON project, and here is what your code will look like:
public class ClientQuestionOption {
String name;
int qid;
public ClientQuestionOption() {
}
public ClientQuestionOption(String name, int qid) {
this.name = name;
this.qid = qid;
}
}
and then use it this way:
public static void main(String[] args) {
List<ClientQuestionOption> questionOptions = Arrays.asList(
new ClientQuestionOption("Excellent", 1),
new ClientQuestionOption("Excellent", 2),
new ClientQuestionOption("Poor", 3)
);
Gson gson = new GsonBuilder().setPrettyPrinting().create();
System.out.println(gson.toJson(questionOptions));
}
You only need to have Class with same names of fields (or declar Qualifier) and gson will handle conversion from/to json.
here is the output for the above code:
[
{
"name": "Excellent",
"qid": 1
},
{
"name": "Excellent",
"qid": 2
},
{
"name": "Poor",
"qid": 3
}
]

Categories