We Keep Coding

Postgres data with interval of 8 hours

I have a table with large number of rows. It has column like timestamp(in millis), value, and a siteId(foreign key). I want to fetch data from that of last three months with an interval in timestamp of 8 hours and I want to fetch data of all siteId in the three month timestamp. I have data in there for every 5 minutes of every siteId. If I fetch data of last three months, it is coming in millions. so I want to take data of every 8 hours. Sometimes, there can be a gap too so if a siteId was not there for the 8th hour, it should get its next data which can be 5 minutes past(or 10minutes past...) of that 8th hour.
Its hard to create a query for that and normal fetching and massaging the data in afterwards will take time.
I am using postgres, java and JPA. If I can do it via query or via some JPA utility to ease the CPU? I want to drop the time taken(right now 9 seconds for each query) to the least. Can you guys help me? Thanks in advance
My Table structure:
| timestamp | value | siteId |
|----------------|-------|--------|
| 1610370000000 | 22 | 123 |
| 1610370700000 | 21 | 123 |
| 1610370028000 | 22 | 123 |
| 1610369889000 | 23 | 123 |
| 1610370000000 | 22 | 124 |
| 1613534400000 | 21 | 124 |
| 1610369889000 | 22 | 124 |
| 1610370005000 | 23 | 125 |
So every site is having data for every 5 minutes. I want data of last three months with interval of at least 8 hours of every site. Hope this helps

Assuming you want same data structure like your example in question from last 3 months on 8 hours interval for each siteID.
Try this:
select
distinct on (siteId, "group" ) siteId, value, timestamp_,
ceil((extract(epoch from current_timestamp)*1000-timestamp_)/28800000) "group"
from test
where to_timestamp(timestamp_/1000) between current_timestamp - interval '3 month' and current_timestamp
order by 1,4,3
DEMO
Here I am dividing the difference of current_timestamp and timestamp_ field with 2880000(8*60*60*1000) to get the group and getting the first value of the group by using distinct on.
You can switch order by from order by 1,4,3 (return the value of min timestamp_ of range) to order by 1,4,3 desc (return the value of max timestamp_ of range) to check the correct result.
I am not sure about performance. But is should work better than java fetching.

If you need all data but you face issue with a transfer of huge data amount I would recommend to use pagination approach.
https://www.baeldung.com/jpa-pagination

How to set an a second Auto increment column per user?

I know this question has been asked before and most of the answers warn about doing so or suggest a solution for MyISAM but I'm using InnoDB.
I'm trying to generate an Invoice or Bill for the user for him to give out to me (Business plan requirement) !
The thing is that in my country the reference of the ID of the bill for the same person should be ordered or he will be audited for the missing bills. Like for example he gave me one bill with 0001 and the second one is 0005. He will be interrogated for the missing 4.
So I need to have a custom auto-increment per UserID.
User 1 - idUser= 1 ,idBill = 1
User 1 - idUser= 1 ,idBill = 2
User 2 - idUsr = 2 , idBill = 1
Some threads suggested using triggers while others warned about table locks. I personally not familiar with triggers so I steer away from them since they require maintenance.
I am using Java and MySQL.

An example:
CREATE TABLE main (id INT AUTO_INCREMENT PRIMARY KEY,
primary_id CHAR(3),
secondary_id INT) ENGINE = InnoDB;
CREATE TABLE auxiliary (primary_id CHAR(3),
secondary_id INT AUTO_INCREMENT,
PRIMARY KEY (primary_id, secondary_id)) ENGINE = MyISAM;
CREATE TRIGGER generate_secondary_id
BEFORE INSERT
ON main
FOR EACH ROW
BEGIN
INSERT INTO auxiliary (primary_id) VALUES (NEW.primary_id);
SET NEW.secondary_id = LAST_INSERT_ID();
END
INSERT INTO main (primary_id) VALUES
('A01'),
('A01'),
('B01'),
('C01'),
('A01'),
('B01'),
('A01'),
('B01');
SELECT * FROM main;
id | primary_id | secondary_id
-: | :--------- | -----------:
1 | A01 | 1
2 | A01 | 2
3 | B01 | 1
4 | C01 | 1
5 | A01 | 3
6 | B01 | 2
7 | A01 | 4
8 | B01 | 3
db<>fiddle here

SQL count occurrence of every record (integer) in one table with multiple columns

I am working on an algorithm, using SQL and JAVA, concerning big datasets.
In SQL I have a table with all the data and I want to use as much of SQL queries as possible before loading it into JAVA.
I generate random datasets (in Java), consisting exclusively of integers between 1 and 40001 and then insert them into a MySQL table.
The rows can be of different lengths, with a maximum of 30 items/records (this includes the ID). So normally the amount of columns is 30 (so COL1, COL2, COL3,......COL30) but this amount will also be random at some point
What I want to do is count the occurrence of every distinct item in a table/dataset and put them in a new table with their count. This however is tricky since I want to count it over the entire table, not just one column. How do I do this?
To specify:
Take this table for example (this is a very small one in comparison with my usual tables):
ID | COL1 | COL2 | COL3 | COL4 | COL5 |
---------------------------------------
1 | 8 | 35 | 42 | 12 | 27 |
2 | 22 | 42 | 35 | 8 | NULL |
3 | 18 | 22 | 8 | NULL | NULL |
4 | 42 | 12 | 27 | 35 | 8 |
5 | 18 | 27 | 12 | 22 | NULL |
What I want to extract from this table is this:
Item | Count
-------------
8 | 3
35 | 3
40 | 1
12 | 3
27 | 3
22 | 3
42 | 2
43 | 1
18 | 2
It is also the case that an item can't be in the same row more than once, if that helps.
Can anyone help me? Or can it just simply not be done in SQL? Would it be better to do this in JAVA, performance-wise?
Thanks in advance!

You can do this by unpivoting the data and then aggregating:
select col, count(*)
from (select col1 as col from t union all
select col2 from t union all
. . .
select col30 from t
) t
group by col;
If you don't have a known set of columns, then you will need to use dynamic SQL.

Number of data generated when a sql query is called or executed

When I run my java web application, I want to know when I execute a query in my application, how much data is coming and out in kilobyte? Is there a tool to do that?

you can run a select statement with vsize to check the size of the all data retrieved
example
SCOTT#research 16-APR-15> select * from test1;
VAL1 VAL2 VAL3
---------- ---------- ----------
555 2 4
3 2 4
123 2 3
42 3
SCOTT#research 16-APR-15> select sum(vsize(val1)+vsize(val2)+vsize(val3)) "bytes" from test1;
bytes
----------
20
SCOTT#research 16-APR-15> select sum(vsize(val1)+vsize(val2)+vsize(val3)) "bytes" from test1 where val3=3;
bytes
---------
7
If specifically kilo bytes then divide by 1024
SCOTT#research 16-APR-15> select sum(vsize(val1)+vsize(val2)+vsize(val3))/1024 "kilo bytes" from test1;
kilo bytes
----------
.01953125

how much data is coming and out in kilobyte?
To see the bytes expected to be read and transferred back you could simply see the explain plan for the query.
For example,
SQL> explain plan for select * from emp;
Explained.
SQL> select * from table(dbms_xplan.display);
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------
Plan hash value: 3956160932
--------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
--------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 14 | 546 | 3 (0)| 00:00:01 |
| 1 | TABLE ACCESS FULL| EMP | 14 | 546 | 3 (0)| 00:00:01 |
--------------------------------------------------------------------------
8 rows selected.
So, you can see 546 bytes.
To see the bytes for both sent and received via SQL*Net, you could SET AUTOTRACE ON.
For example,
SQL> SET AUTOTRACE ON
SQL> select empno from emp;
EMPNO
----------
7369
7499
7521
7566
7654
7698
7782
7788
7839
7844
7876
7900
7902
7934
14 rows selected.
Execution Plan
----------------------------------------------------------
Plan hash value: 179099197
---------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
---------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 14 | 56 | 1 (0)| 00:00:01 |
| 1 | INDEX FULL SCAN | PK_EMP | 14 | 56 | 1 (0)| 00:00:01 |
---------------------------------------------------------------------------
Statistics
----------------------------------------------------------
8 recursive calls
0 db block gets
4 consistent gets
0 physical reads
0 redo size
702 bytes sent via SQL*Net to client
544 bytes received via SQL*Net from client
2 SQL*Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
14 rows processed
SQL>
So, you can see 702 bytes sent via SQLNet to client and 544 bytes received via SQLNet from client.

sql insert statement to set rows proportional

I have a table with the following data (ofcourse there are many rows).
metric_id(int) | employee_id (string) | quota (double)
1 | abcd | 100
1 | wxyz | 120
What i want to do is a SQL INSERT that takes all values with a specific metric_id.
select * from mytable where metric_id=1;
Then use the results to create new entries with different metric_id that will have the quota value set at a specific proportion. (ie: metric_id=2 quota is 50% of metric_id=1)
So my end table after the insert would look like
metric_id(int) | employee_id (string) | quota (double)
1 | abcd | 100
1 | wxyz | 120
2 | abcd | 50
2 | wxyz | 60
Can this be done in one sql statement? I would like to use prepared statement to prevent inject if possible.

Try this. It should be what you need.
insert into some_table
(metric_id, employee_id, quota)
select
2, employee_id, quota * 0.5
from
some_table
where
metric_id = 1

More alternative way (no where clause needed):
INSERT INTO TableName
(metric_id,employee_id,quota)
SELECT metric_id+1,employee_id,quota/2
FROM TableName