We Keep Coding

Postgres data with interval of 8 hours

I have a table with large number of rows. It has column like timestamp(in millis), value, and a siteId(foreign key). I want to fetch data from that of last three months with an interval in timestamp of 8 hours and I want to fetch data of all siteId in the three month timestamp. I have data in there for every 5 minutes of every siteId. If I fetch data of last three months, it is coming in millions. so I want to take data of every 8 hours. Sometimes, there can be a gap too so if a siteId was not there for the 8th hour, it should get its next data which can be 5 minutes past(or 10minutes past...) of that 8th hour.
Its hard to create a query for that and normal fetching and massaging the data in afterwards will take time.
I am using postgres, java and JPA. If I can do it via query or via some JPA utility to ease the CPU? I want to drop the time taken(right now 9 seconds for each query) to the least. Can you guys help me? Thanks in advance
My Table structure:
| timestamp | value | siteId |
|----------------|-------|--------|
| 1610370000000 | 22 | 123 |
| 1610370700000 | 21 | 123 |
| 1610370028000 | 22 | 123 |
| 1610369889000 | 23 | 123 |
| 1610370000000 | 22 | 124 |
| 1613534400000 | 21 | 124 |
| 1610369889000 | 22 | 124 |
| 1610370005000 | 23 | 125 |
So every site is having data for every 5 minutes. I want data of last three months with interval of at least 8 hours of every site. Hope this helps

Assuming you want same data structure like your example in question from last 3 months on 8 hours interval for each siteID.
Try this:
select
distinct on (siteId, "group" ) siteId, value, timestamp_,
ceil((extract(epoch from current_timestamp)*1000-timestamp_)/28800000) "group"
from test
where to_timestamp(timestamp_/1000) between current_timestamp - interval '3 month' and current_timestamp
order by 1,4,3
DEMO
Here I am dividing the difference of current_timestamp and timestamp_ field with 2880000(8*60*60*1000) to get the group and getting the first value of the group by using distinct on.
You can switch order by from order by 1,4,3 (return the value of min timestamp_ of range) to order by 1,4,3 desc (return the value of max timestamp_ of range) to check the correct result.
I am not sure about performance. But is should work better than java fetching.

If you need all data but you face issue with a transfer of huge data amount I would recommend to use pagination approach.
https://www.baeldung.com/jpa-pagination

Select/Filter specific rows in jTable

My Task is to develop a Software which is measuring times of different participants.
Following problem:
each participant has 2 chances to give his/her best and the software has to select the best 32/16/.. times for the next round
The times are saved in a MySql DB, selected with Hibernate and saved in a Datatable using own Tablemodel (RunTableModel extends DefaultTableModel)
for example:
Participant | Time
------------|------
Tina | 10:15
Tina | 10:20
Carl | 11:00
Carl | 10:17
Charlie | 12:01
Charlie | 11:50
My question now is how to filter this results in a jTable.
-> One option would probably be to do this by selection from the Database (Hibernate HQL) but therefore I have to reload always from DB
Is there a possability to do this with filters?
Result should look like this:
Participant | Time
------------|------
Tina | 10:15
Carl | 10:17
Charlie | 11:50

Hibernate Query - Complex

I'm dabbling in SQL at the moment and would love some help on a problem that I've created.
To practice some programming i'm making an IOU app. Below is the table I store
my IOU records (ignoring some relevant columns). This table allows users to say "Hey, you owe me X amount of money" and "I owe you X amount of money".
| Creator | Type | User_Involved| Amount |
|:-----------|------------:|:------------:|
| 1 | 0 | 2 | 3.0
| 2 | 0 | 1 | 4.0
| 3 | 1 | 1 | 5.0
Note: Type represents if the user has "lent or requested money".
0 = Lent
1 = Requested
Goal: My goal is to get a list of all Users and their total owed amount (negative or positive).
This has become quite difficult as I can't do a simple group by, as there are two columns i need to group on (Creator and User_Involved).
I managed to write an SQL query that gives the correct results, however i'm unable to translate it into a Hibernate version.
The issue mostly comes down to JPA unable to perform unions.
The Query in question:
/** assumption 1 is owed to creator **/
select sum(owed_to_you) as owed_to_you, friend_id
from
(
/** you created **/
select sum(CASE WHEN transaction_type = 0 THEN -amount ELSE amount END) as owed_to_you, friend_involved as friend_id
from iou
where
creator = 3
group by friend_involved
union all
/** someone else created **/
select sum(CASE WHEN transaction_type = 1 THEN -amount ELSE amount END) as owed_to_you, creator as friend_id
from iou
where
friend_involved = 3
group by creator) as theunion
group by friend_id
Besides loading all Ious into memory and sorting it that way, I'm completely stumped. I've done a lot of research today and learned a lot, however, i still have not made progress.
Any help would be much appreciated.

You can execute the two queries separately and combine the results in Java-code.

How to process columns of an SQLite table in Java android?

I have an SQLite table like:
+---+-------------+----------------+
|_id| lap_time_ms |formatted_elapse|
+---+-------------+----------------+
| 1 | 5600 | 00:05.6 |
| 2 | 4612 | 00:04.6 |
| 3 | 4123 | 00:04.1 |
| 4 | 15033 | 00:15.0 |
| 5 | 4523 | 00:04.5 |
| 6 | 6246 | 00:06.2 |
Where lap_time_ms is an of the type long and represents the amount of time in milliseconds for a lap while formatter_elapse is a String that represents the formatted (displayable) form of the first column (elapse).
My question is that if I want to add (say) 5 seconds (5000) to each lap_time_ms then I use a statement like:
DB.execSQL("update table_name set KEY_ELAPSE=KEY_ELAPSE+5000);
Which works fine however the problem is that formatted_elapse still retains its outdated value!
So, what is the best way to update the values in the formatted_elapse column if I have a function like:
public static String getFormattedTime(long milliseconds) {
// custom code that processes the long
return processedString;
}
It may be a long shot (metaphorically speaking of course ;) but is it possible to have SQLite link the two columns such that if I update a lap_time_ms row, the formatted_elapse will automatically update appropriately.
Thanks in advance!

In theory, it would be possible to create a trigger to update that column, but only if the formatting can be done with some built-in SQLite function (Android does not allow user-defined functions):
CREATE TRIGGER update_formatted_elapse
AFTER UPDATE OF lap_time_ms ON MyTable
FOR EACH ROW
BEGIN
UPDATE MyTable
SET formatted_elapse = strftime('%M:%f', NEW.lap_time_ms, 'unixepoch')
WHERE _id = NEW._id;
END;
However, it would be bad design to store the formatted string in the database; it would be duplicated information that is in danger of becoming inconsistent.
Drop the formatted_elapse column and just call getFormattedTime in your code whenever you need it.

look for a database design related manner

I am working for a log analyzer system,which read the log of tomcat and display them by a chart/table in web page.
(I know there are some existed log analyzer system,I am recreating the wheel. But this is my job,my boss want it.)
Our tomcat log are saved by day. For example:
2011-01-01.txt
2011-01-02.txt
......
The following is my manner for export logs to db and read them:
1 The DB structure
I have three tables:
1)log_current:save the logs generated today.
2)log_past:save the logs generated before today.
The above two tables own the SAME schema.
+-------+-----------+----------+----------+--------+-----+----------+----------+--------+---------------------+---------+----------+-------+
| Id | hostip | username | datasend | method | uri | queryStr | protocol | status | time | browser | platform | refer |
+-------+-----------+----------+----------+--------+-----+----------+----------+--------+---------------------+---------+----------+-------+
| 44359 | 127.0.0.1 | - | 0 | GET | / | | HTTP/1.1 | 404 | 2011-02-17 08:08:25 | Unknown | Unknown | - |
+-------+-----------+----------+----------+--------+-----+----------+----------+--------+---------------------+---------+----------+-------+
3)log_record:save the information of log_past,it record the days whose logs have been exported to the log_past table.
+-----+------------+
| Id | savedDate |
+-----+------------+
| 127 | 2011-02-15 |
| 128 | 2011-02-14 |
..................
+-----+------------+
The table shows log of 2011-02-15 have been exported.
2 Export(to db)
I have two schedule work.
1) day work.
at 00:05:00,check the tomcat log directory(/tomcat/logs) to find all the latest 30 days log files(of course it include logs of yesterday.
check the log_record table to see if logs of one day is exported,for example,2011-02-16 is not find in the log_record,so I will read the 2011-02-16.txt,and export them to log_past.
After export log of yesterday,I start the file monitor for today's log(2011-02-17.txt) not matter it exist or not.
2)the file monitor
Once the monitor is started,it will read the file hour by hour. Each log it read will be saved in the log_current table.
3 tomcat server restart.
Sometimes we have to restart the tomcat,so once the tomcat is started,I will delete all logs of log_current,then do the day work.
4 My problem
1) two table (log_current and log_past).
Because if I save the today's log to log_past,I can not make sure all the log file(xxxx-xx-xx.txt) are exported to db. Since I will do a check in 00:05:00 every day which make sure that logs before today must be exported.
But this make it difficult to query logs accros yestersay and today.
For example,query from 2011-02-14 00:00:00 to 2011-02-15 00:00:00,these log must be at log_past.
But how about from 2011-02-14 00:00:00 to 2011-02-17 08:00:00 ?(suppose it is 2011-02-17 09:00:00 now).
It is complex to query across tables.
Also,I always think my desing for the table and work manner(schedule work of export/read) are not perfect,so anyone can give a good suggestion?
I just need to export and read log and can do a almost real-time analysis where real-time means I have to make logs of current day visiable by chart/table and etc.

First of all, IMO you don't need 2 different tables log_current and log_past. You can insert all the rows in the same table, say logs and retrieve using
select * from logs where id = (select id from log_record where savedDate = 'YOUR_DATE')
This will give you all the logs of the particular day.
Now, once you are able to remove the current and past distinction between tables using above way, I think the problem you are asking here would be solved. :)