I currently have a webservice which inserts information in a mysql database using Hibernate. Some of this information needs to be processed by another 'import' application. I would like to not have to trigger this application from the webservice. So the webservice doesn't have a dependency on the webservice and visa versa.
Is there a way to "listen" to changes (specifically: insert) in the database from the 'import' application and then start executing an action. I have looked at triggers but these seem to only work for changes in the application's Hibernate Session and not for 'external' changes.
Edit*
In short, the answer I would like to have;
Is it possible to monitor changes to a mysql database/table (coming from any source) from a java application which does not alter the database/table itself
Bounty Update*
I will award the bounty to the person who can explain to me how to monitor changes made to a MySQL table/database using a Java application. The Java application monitoring the changes is not the application applying any changes. The source of the alterations can be anything.
I think you could acheive something like this fairly easily, assuming you didn't mind a creating some extra tables & triggers on your database, and that the monitoring java application would have to poll the database rather than specifically receive triggers.
Assuming the table you're wanting to monitor is something like this:
CREATE TABLE ToMonitor ( id INTEGER PRIMARY KEY, value TEXT );
Then you create a table to track the changes, and a trigger that populates that table:
CREATE TABLE InsertedRecords( value TEXT );
CREATE TRIGGER trig AFTER INSERT ON account
FOR EACH ROW INSERT INTO InsertedRecords( value ) VALUES ( NEW.value );
This will cause the InsertedRecords table to be populated with every insert that happens in ToMonitor.
Then you just need to set up your monitoring app to periodically SELECT * from InsertedRecords, take the appropriate action and then clear out the records from InsertedRecords
EDIT: A slight alternative, if you didn't mind a bit of C/C++ coding, would be to follow the instructions here to create a custom SQL function that triggered your monitoring application into action, and then just call that SQL function from within the trigger you'd created.
You can read mysql binary log. Here you can find some information. There is a java parser and another one - but it is marked as unfinished) also you can look for similar parsers using another languages (for example, perl) and rewrite them in Java.
Also have a look at mysql-proxy.
I know it's not what you asked (thus, this is not a proper answer), but if you consider dropping the idea of "letting the DB notify the apps", you get the perfect case for using JMS for communication between apps.
Your app originating the change could publish a message to a JMS topic, which is subscribed by the second application. Once the first changes the database, it puts a message on the topic. The second then sees this new event and act accordingly. You could even publish the delta in the message, so that the second app don't need to reach the database.
I'm a bit against dealing with this by "hacking" the database to do more than just store data, as it will innevitably get into trouble in the future (as everything eventually will), and debugging it will be hard. Imagine adding a third app the the ecosystem, and you have now to replicate whatever you did for the second app, but now for the third app. If you didn't document your steps, you might get lost.
If you just use a JMS server between those two apps, you can certainly add a third app in the future, that just listens to this topic (and publish a new message, in case it has write access to the db), and the other apps don't even have to know that there's one more app out there. Nor the database.
assume we want to monitor changes in the table 'table1'
CREATE TABLE `table1` (
`id` INT(10) NOT NULL AUTO_INCREMENT,
`value` VARCHAR(50) NULL DEFAULT NULL,
PRIMARY KEY (`id`)
)
COLLATE='latin1_swedish_ci'
ENGINE=InnoDB
AUTO_INCREMENT=10;
above is the query to create 'table1' it contain a 'id' column which is auto increment
create another table to store changes. Query is given below
CREATE TABLE `changes` (
`id` INT(10) NOT NULL AUTO_INCREMENT,
`changes` VARCHAR(200) NULL DEFAULT '0',
`change_time` TIMESTAMP NULL DEFAULT NULL,
`tablename` VARCHAR(50) NULL DEFAULT NULL,
`changed_id` VARCHAR(10) NULL DEFAULT NULL,
PRIMARY KEY (`id`)
)
COLLATE='latin1_swedish_ci'
ENGINE=InnoDB
AUTO_INCREMENT=21;
now create a trigger in first table ie.. 'table1' Query is given below
delimiter |
create trigger trg_table1 AFTER INSERT ON table1
FOR EACH ROW BEGIN
DECLARE lastid INT DEFAULT 0;
SELECT max(id) INTO lastid from table1;
insert into changes values(null,'insert',now(),'table1',lastid);
end;
|
delimiter ;
Now if you try to insert anything in 'table1' its details will be automatically inserted in to changes table.
In changes table changes indicates the type of change ie.. insert,update etc
change_time indicates time at which change occur
tablename indicates table in which change occur
changed_id indicates id of the newly inserted row in the 'table1'
Now create a java program that continuously reading from 'changes' table.
A new entry in 'changes' table means something happened to database.
From each record in the 'changes' table you can understand in which table the insert operation is occured. And based on this you can perform appropriate action. After performing appropriate operation delete that row from 'changes' table.
You can create trigger (like i did above) for each table in your database...
From 'changes' table's 'tablename' column you can understand insert occurred in which table..
You could use a queuing solution like Q4M, but it might be overkill in your situation. But you could:
In the MySQL database, add a timestamp column to the table that is being inserted into. In the 'import' application either use a java.util.timer, or an external scheduler like cron. Use one of those to trigger a task that reads the insert table where the timestamp column is null. Take the appropriate action for those rows, and then set the timestamp column with a value. If there are no rows with a null time stamp, you have no new inserts. Simple, but it works.
You may want to add an index to the timestamp column for performance reasons.
I guess this answer is pretty late but as #dbf pointed out, reading binlog might be the way to go. I suggest looking into Debezium's MySQL Connector.
The Debezium MySQL connector reads the binlog and produces change
events for row-level INSERT, UPDATE, and DELETE operations and records
the change events in a Kafka topic.
Your "another" application can then read those Kafka topics.
Related
I am using Hibernate with MSSQL server writing the software that integrates with an existing database. There is an instead of insert trigger on the table that I need to insert into and it messes up ##Identity, which means on Hibernate's save I can't get the id of inserted row. I can't control the trigger (can't modify it). I saw this question, but it involves procedures, which my trigger does not have, so I thought my question is different enough. I can't post the whole trigger, but hopefully I can post enough to get the point across:
CREATE TRIGGER TrigName ON TableName
INSTEAD OF INSERT
AS
SET XACT_ABORT ON
BEGIN TRANSACTION
-- several DECLARE, SET statements
-- a couple of inserts into other tables for business logic
-- plain T-SQL statements without procedures or functions
...
-- this is the actual insert that i need to perform
-- to be honest, I don't quite understand how INSERTED table
-- was filled with all necessary columns by this point, but for now
-- I accept it as is (I am no SQL pro...)
INSERT INTO ClientTable (<columns>)
SELECT <same columns> from INSERTED
-- a couple of UPDATE queries to unrelated tables
...
COMMIT TRANSACTION;
I was wondering if there is a reliable way to get the id of the row being inserted? One solution I thought of and tried to make is to install an on insert trigger on the same table that writes the newly inserted row into a new table I added to the db. I'd use that table as a queue. After transaction commit in Hibernate I could go into that table and run a select with the info I just inserted (I still have access to it from the same method scope), and I can get the id and finally remove that row. This is a bulky solution, but best I can come up with so far.
Would really appreciate some help. I can't modify existing triggers and procedures, but I can add something to the db if it absolutely does not affect existing logic (like that new table and a on insert trigger).
To sum up: I need to find a way to get the ID of the row I just inserted with Hibernate's save call. Because of that instead of insert trigger, hibernate always returns identity=0. I need to find a way to get that ID because I need to do the insert in a few other tables during one transaction.
I think I found an answer for my question. To reply to #SeanLange's comment: I can't actually edit insert code - it's done by another application and inquiry to change that will take too long (or won't happen - it's a legacy application). What I did is insert another trigger on insert on the same table. Since I know the order of operations in the existing instead of insert trigger I can see that the last insert operation will be in the table I want so that means my on insert trigger will fire right after that. In the scope of that trigger I have access to inserted table out of which I pull out the id.
CREATE TRIGGER Client_OnInsert ON myClientTable
FOR INSERT
AS
BEGIN
DECLARE #ID int;
SET #ID = (select ClientID from inserted);
INSERT INTO ModClient (modClientId)
OUTPUT #ID
VALUES (#ID);
END
GO
Then in Hibernate (since I can't use save() anymore), I use a NativeQuery to do this insert. I set parameters and run the list() method of NativeQuery, which returns a List where the first and only argument is the id I want.
This is a bulky way, I know. If there is anything that's really bad that will stand out to people - please let me know. I would really appreciate some feedback on this. However, I wanted to post this answer as a potential answer that worked so far, but it does not mean it's very good. For this solution to work I did have to create another small table ModClient, which I will have to use as a temp id storage for this exact purpose.
I could get notifications from an Oracle database thanks to this code and omitting this line:
prop.setProperty(OracleConnection.DCN_QUERY_CHANGE_NOTIFICATION,"true");
Also I could solve my ORA-29977 problem changing select * from act_code_metadata where product_id=1159 with select column_with_Number_type from act_code_metadata where product_id=1159
Everything works as expected :D
This is the code I use to print the row's information (Java 8):
DatabaseChangeRegistration dcp.addListener((DatabaseChangeEvent dce) ->
System.out.println(
"Changed row id : " +
dce.getTableChangeDescription()[0].getRowChangeDescription()[0].getRowid().stringValue()
+ " " + dce.getTableChangeDescription()[0].getRowChangeDescription()[0].getRowOperation().toString()));
But all the information I get is the row's physical address (rowid) and the operation involved (insert, delete or update).
I need to identify the row being modified/inserted/deleted to refresh my cached data in several Swing controls in my GUI.
I've read that, despite the rowid being imutable, the same rowid can be re-assigned if the row is deleted and a new one is inserted, and the rowid can change if the row is in a partitioned table. So the best that can be done is using the rowid and the row's primary key.
My table has a autoincrementable primary key (with a sequence and a trigger) created with this code.
I have no control on what happens on the database or if somebody inserts and deletes rows several times. So I can get the wrong row when selecting it using the rowid given by the notification.
Is there a way that I can get my row's primary key via Oracle Database Change Notification so I can identify the inserted/deleted/modified row correctly?
I'm working with Oracle Database XE 11.2 Express and Java 8. The user for database connection already has the change notification privilege.
It seems that you have a lot of overhead trying to basically maintain a fresh snapshot of the data in your GUI. It may be simpler to look at client result caching and just re-running your query every X seconds; and let Oracle do the magic of seeing if the data changed. You would be limited to a JDBC driver that supports OCI. See http://docs.oracle.com/cd/E11882_01/server.112/e41573/memory.htm#PFGRF985 for details. With client result caching, the first time the SQL is executed, it will take say 500 milliseconds. Next query using the same criteria it will take 2 or 3 milliseconds. Assuming the result set is small (less than 100 rows is quite small), you can get a lot better results without all that framework you are trying to build.
Is it possible to restart the ID column of an HSQLDB after rows were inserted? Can I even set it to restart at a value lower than existing IDs in the table?
The Situation
I have a simple Java program which connects to a HSQLDB like so:
DriverManager.getConnection("jdbc:hsqldb:file:" + hsqldbPath, "", "");
This gives me an HsqlException when executing the following script (this is an excerpt, the complete script for HSQLDB 2.2.4 can be found here):
SET SCHEMA PUBLIC
CREATE MEMORY TABLE PUBLIC.MAP(
ID BIGINT GENERATED BY DEFAULT AS IDENTITY(START WITH 0) NOT NULL PRIMARY KEY,
FOO VARCHAR(16) NOT NULL)
ALTER TABLE PUBLIC.MAP ALTER COLUMN ID RESTART WITH 1
// [...]
SET SCHEMA PUBLIC
INSERT INTO MAP VALUES(1,'Foo')
INSERT INTO MAP VALUES(2,'Bar')
ALTER TABLE PUBLIC.MAP ALTER COLUMN ID RESTART WITH 42
The message is:
HsqlException: error in script file: ALTER TABLE PUBLIC.MAP ALTER COLUMN ID RESTART WITH 42
The exception goes away when I move the RESTART-command before the INSERTs. The documentation gives no hint as to why that would be necessary.
I will eventually have to make this work on version 2.2.4 but have the same problem with the current version 2.3.2.
Background
What I am trying to do here is to recreate a situation which apparently occurred in production: An unlucky interaction with the database (I don't know what exactly happened) seems to have caused newly inserted rows to collide with existing ones because they were issued the same IDs. I want to create a test replicating the scenario in order to write a proper fix.
The .script file of the database follows a predefined order for the statements. This shouldn't be altered if it is edited and only certain manual changes are allowed (see the guide for details).
You can execute the ALTER TABLE statement via JDBC at the start of your test instead of inserting it in the script.
If IDENTITY values for the PRIMARY KEY collide, you will get an exception when you insert the values.
The actual fix for a problem like this is to RESTART WITH the max value in the primary key column plus one.
I think SEQUENCES are much more flexiblee than IDENTITY. The IDENTITY generator disabled JDBC batching, by the way.
But if you use SEQUENCE identifiers, you must pay attention to the hilo optimizers as well, because identifier are generated by Hibernate using a sequence value as a base calculation starting point.
With a SEQUENCE the restart goes like this:
ALTER SEQUENCE my_seqeunce RESTART WITH 105;
Just a quick question about locking tables in a postgres database using JDBC. I have a table for which I want to add a new record to, however, To do this for the primary key, I use an increasing integer value.
I want to be able to retrieve the max value of this column in Java and store it as a variable to be used as a new primary key when adding a new row.
This gives me a small problem, as this is going to be modelled as a multi-user system, what happens when 2 locations request the same max value? This will of course create a problem when trying to add the same primary key.
I realise that I should be using an EXCLUSIVE lock on the table to prevent reading or writing while getting the key and adding a new row. However, I can't seem to find any way to deal with table locking in JDBC, just standard transactions.
psuedo code as such:
primaryKey = "SELECT MAX(id) FROM table1;";
primary key++;
//id retrieved again from 2nd source
"INSERT INTO table1 (primaryKey, value 1, value 2);"
You're absolutely right, if two locations request at around the same time, you'll run into a race condition.
The way to handle this is to create a sequence in postgres and select the nextval as the primary key.
I don't know exactly what direction you're heading and how your handle your data, but you could also set the column as a serial and not even include the column in your insert query. The column will automatically auto increment.
I'm working at refactoring a lot of test code that uses a local host mysql. As can be imagined this is not optimal so I'm replacing mysql with hsqldb (for testing purposes, production still uses mysql). So far this is going (somewhat) smoothly as the code uses only standard sql. Now, however, I've run into a snag.
The code polls a table to get its last update time and send it to all observers. The code uses the mysql specific (AFAIK) syntax show table status from someDb like tableName and then pulls the update_time column from the result set.
I need to implement the same thing in HSQLDB and I haven't been able to find anything.
I've looked at the java.sql.DatabaseMetaData class to see if there is anything there that I can use to no avail.
I've been unable to format a google question narrow enough to be useful and so I turn to stackoverflow!
So far I've haven't found anything about HSQLDB:s special tables either.
Up to version 2.1 RC4, HSQLDB does not have built-in functionality for this feature. You can define triggers in SQL on a given table which store the update time in another user-defined table. This example stores the last INSERT time. Other triggers (AFTER UPDATE or AFTER DELETE) should be defined to store the last UPDATE or DELETE time.
create table updatetime (table_name varchar(128) PRIMARY KEY, update_time timestamp );
create trigger trig after insert on sometable
update updatetime set update_time = current_timestamp
where updatetime.table_name = 'SOMETABLE'
i dont know if im right but afaik there was an Infomration Scheme in HSQL:
check the Information Scheme section in the HSQL Manual (http://hsqldb.org/doc/2.0/guide/databaseobjects-chapt.html)
There ought to be some views in this scheme holding all the metadata.
try if you can find a View named "TABLES" there, and check if you find the information you're seeking in some column of this view.
Update: found this question with a perfect answer:
How to see all the tables in an HSQLDB database?