We are currently developing a user based web application in java.
A user can have many properties like firstname, lastname, Region (eg. Asia, Europe, US, UK), country, Division, Branch, Product etc.
There are normal CRUD screens for this user with all the above fields.
The add/edit screen for user will also have a date field called effective date. The way this add/edit user is different from normal add/edit in regular CRUD is that the update made to the user will not reflect until the effective date.
Lets say today is 6 April and I add a new user with Region say Asia and effective date 10 April. And now I go and change same user and change his region from Asia to US but effective date say 15 May.
So till 15th may the system should show his region as Asia only and on 15th may his region should change to US automatically.
You can think of it as the user who is working in Asia as on April but from 15th may he is moving to work in US i.e. Region changed from Asia to US effective date 15th may. So till 15th may he should be shown as a user in Asia only.
So I can not just update his region from Asia to us in database as a simple update operation.
And this applies to lot of other fields like division , branch and product as well.
I can not have multiple records for the same user.
EDIT 1:
We should also be able to see the user history or the future updates.
Like on the 6 april, I should see that users region will change from 15 May and his division will change from A to B starting 10 may.
I should also be able to delete updates say I come to know later that the proposed transfer of user from Asia to Us effective date 15 may is not going to happen now so I should be able to delete that update.
EDIT 2:-
Given the above situations, How do I make changes in my original user table and the user change log table?
Say for a user with region asia in the system which is going to change from asia to Us in next few weeks. user will have same update for user. He changes region from asia to user and choose effecftive date as some future date.
Now How do I check if region is changed from asia to us (there can be namy mre fields like region). Shall I do it at the code level or is it possible to do it at the datbase level using triggers etc?
Please help me out with designing the system and database for this.
I will suggest you can implement this system maintaining a CHANGELOG table and a scheduler which will run at a specific time everyday.
CHANGELOG table
Attributes :
targetTable
targetPrimaryKey
targetColumn
targetValue
effectiveDate
Now, whenever a update is made on required fields, insert a corresponding entry in changelog table.
Example,
targetTable : User
targetPrimaryKey : 3 (primary row id of user record)
targetColumn : Region
targetValue : US
effectiveDate : 15 May 2012
Now, a scheduler will run every day say(at 12:00 AM) and will check for scheduled changes made to be done that day from CHANGELOG table and make those changes in respective target tables.
Note : This approach has 2 advantages :
You dont have to maintain two entries for same record.
You can target multiple fields(region,division,branch) and multiple tables as well.
You can create a mapping table as User_Region containing the User_ID, Region_ID and Change_Date. Change_Date will give when the switch in the region for the user takes place. When the change date is null that could imply that the user is currently in the very region.
Then you can have a list of regions for a User_ID along with the dates, which can be displayed according to your convenience.
CREATE TABLE User_Regions{
User_ID INT,
Region_ID INT,
Change_Date DATE,
FOREIGN KEY User_ID REFERENCES User(ID),
FOREIGN KEY Region_ID REFERENCES Region(ID)
};
Historical table would do the trick. That means, having as many history records with date column and only treating the one that is closest to current date as current. Otherwise, you have to keep update history in a separate table and have a batch update process to update changes. I would recommend to try to overcome the impediment of unavailability to have multiple records for the same user by getting to normalized structure of user entity: users table will have key columns (e.g. id) and a joined table containing the historical updates with the rest of columns.
You have two possibilities here:
You have a different table that looks like your user table and with an additional column 'validFrom' and then have a cron (quartz) job update every morning.
You add the column to your regular user table together with a valid to and change all your queries to reflect that.
As you already said you cannot have multiple records, so 2. would be a bad solution. Stick to 1.
A "pure" data model would be to include a date in the key, and have multiple editions, but there are some issues with changing future events.
I think i would try to model future changes as a list of "operations" and changing a field in the future is in effect scheduling an operation that will modify the field on that date.
The application periodically applies the operations as they become due.
In the GUI, it's trivial to show a list of pending operations, and it's easy to cancel or modify future operations.
You may set up rules to limit or issue warnings for conflicting operations, but if the list is clearly visible, that is less of a problem.
In my opinion you should use Event statement. Here are the 2 events :
CREATE EVENT myevent
ON SCHEDULE AT DATE_CHANGING
DO
UPDATE myschema.mytable SET mycol = mycol + 1;
CREATE EVENT myevent
CREATE EVENT myevent
ON SCHEDULE AT DATE_BACKUP
DO
UPDATE myschema.mytable SET mycol = mycol - 1;
I can think of two approaches:
Keep all versions in main table
| user_name (PK) | effective_date_from (PK) | ...remaining columns
In this approach every user is represented by several rows in User table but only single row is current. In order to find current data you need to add some extra querying:
SELECT *
FROM User
WHERE user_name = ?
AND effective_date >= NOW()
ORDER BY effective_date DESC
LIMIT 1
So we are fetching all users with given user_name (there are multiple versions of such user with different effective_date, so effective_date must also be part of primary key). We limit the result to most recent version (but with effective_date not in the future).
This approach has several drawbacks:
what to do with foreign keys? Logically there is only one user in the system
poor performance, complicating queries
what to do with outdated versions?
Pending versions table
Keep the original (main) User table schema without any changes and have a second table called Pending_user_changes. The latter will also have the same schema + effective_date column. If the granularity of changes is one day, write a job (in the database or in the application) that looks for any pending changes that should take effect starting from today and eplace the main table.
I find this solution much cleaner: the primary/foreign keys in main table never change, also the main table is not cluttered with old and duplicated data.
Related
I have a table named booking. The primary key is booking_id. The table has a column named booking_num. booking_num is used to identify the booking.
Another table is booking_hist. The purpose is to transfer completed records of booking from booking table into booking_hist table after a certain period, let's say 2 days. The column to identify it will be completed_dt.
For example, completed_dt is '08-SEP-19'.
What I'm trying to achieve is that after immediately 2 days after this date, it will be moved into booking_hist table.
Should there be any null<->non-null conversions of the column?
What is the logic I need to achieve this? How can i get the date to
count 2 days?
You can schedule a SQL Agent job runs daily and call a stored procedure to go through the active bookings and check the completed_dt like:
-- add your insert here, e.g. INSERT INTO bookings_hist (...)
SELECT *
FROM booking b
LEFT JOIN booking_hist h
ON b.booking_id=h.booking_id
WHERE h.booking_id IS NULL
AND completed_dt IS NOT NULL
AND completed_dt<DATEADD(DAY,-2,GETDATE());
This sounds like the kind of thing that should happen in a scheduled job.
I would add a ready_for_archive column to booking-just a boolean flag.
I would have a query that marked all bookings that happen before a specified date/time, and pass in the date of 2 days ago from java. Something like
UPDATE booking
SET ready_for_archive = 1
WHERE completed_dt <= :MY_START_DATE
Then I would add all of these records to the historical table
INSERT INTO booking_hist
SELECT * FROM booking
WHERE ready_for_archive = 1;
Then remove them from the booking table:
DELETE FROM booking
WHERE ready_for_archive = 1;
Marking the records to be archived before doing that process means there's no risk of accidentally deleting records that were one second too young to be copied.
Passing in the date after calculating it in Java makes the sql query more generic and reusable.
Create a stored procedure to move from one table to other
how to call the stored procedure in oracle with the daily scheduled jobs?
In where condition add ,
trunc(sysdate) - to_date('completed_dt', 'yyyy-mm-dd') >2
Please refer below link for other options:
How can I get the number of days between 2 dates in Oracle 11g?
I'm developing a web application for the deacons in my church. And while I have a good deal of experience coding in several languages, I have yet had to do any serious database modeling until deciding to tackle this need for my organization.
I'm very familiar with writing sql/ddl queries on existing database in (strictly mysql console, Spring MVC, Boot, Java, etc.). But, not since college have I had to consider normalization, 2nf, 3nf, 1:1, 1: many, etc... It's been a humbling experience, to say the least, trying to refresh my memory with the database theories learned years before and attempting to apply the concepts.
I created a model that seems, at least to me, to fit the needs of the users
My specific question is about locked accounts. I did read several posts about it, which only confused me more about how to approach this concept with my given data model? I really would appreciate any other suggestions and/or critiques; I definitely understand the concept and power of learning by failure.... Thanks.
Use Case :
1. Users holding office in a particular year can sign into the web
application, and view their information *(Name, Account Status,
Ordained, Team number, Calendar of their assigned duty days)*.
They can only update their personal info (name, address,
phone). Note: The account will be created for users.
2. Director, Asst. Director and System admin can log into the web
application (admin dashboard) and see a data table of all users,
w/ all relevant fields(view). This group has full read-write
privileges.
I have a locked table in the model, but not sure if that is the correct way to handle updating a user's status from active to inactive. If inactive, he cannot log into the web application. I would also use this if the user attempts to log-in more the x number of times unsuccessfully. Additionally, it would be helpful (reporting and stats) to keep previous users in the database for x number of years, of course with an inactive status.
Sorry for not using diagram (I don't use diagram tools). Here's extremely basic sample with relevant bits for audit table:
CREATE TABLE users (
user_id SERIAL,
-- ...
);
-- ...
CREATE TABLE user_updates_audit (
audit_id SERIAL,
user_id INT NOT NULL,
audit_timestamp TIMESTAMP NOT NULL default now(),
-- just free form text describing applied update (maybe old value, new value, etc)
audit_text VARCHAR(1024) NOT NULL
);
ALTER TABLE user_updates_audit ADD CONSTRAINT user_updates_audit_pk PRIMARY KEY (audit_id);
ALTER TABLE user_updates_audit ADD CONSTRAINT user_updates_audit_user_id_fk FOREIGN KEY (user_id) REFERENCES users;
Sure, you may expand from here, for example, by changing free-form audit_text to some more strict scheme, e.g. foreign key to dictionary of possible update actions (ENABLED, DISABLED, whatever) and actual values being changed. Or some other more elaborate scheme more suitable for your case.
But free-form audit is some starting point.
Main value here is that you can see all modifications history of important entities in your system, not just current state.
I am working on a MySQL database with 3 tables - workout_data, excercises and sets tables. I'm facing issues related to generating reports based on these three tables.
To add more information, a number of sets make up an excercise and a number of excercises will be a workout.
I currently have the metrics to which a report is to be generated from the data in these tables. I've to generate reports for the past 42 days including this week. The queries run for a long time by the time I get the report by joining these tables.
For example - the sets table has more than 1 million records just for the past 42 days. The id in this table is the excercise_id in excercise table. The id of excercise table is the workout_id in workout_data table.
I'm running this query and it takes more than 10 minutes to get the data. I have to prepare a report and show it to the user in the browser. But due to this long running query the webpage times out and the user is not able to see the report.
Any advice on how to achieve this?
SELECT REPORTSETS.USER_ID,REPORTSETS.WORKOUT_LOG_ID,
REPORTSETS.SET_DATE,REPORTSETS.EXCERCISE_ID,REPORTSETS.SET_NUMBER
FROM EXCERCISES
INNER JOIN REPORTSETS ON EXCERCISES.ID=REPORTSETS.EXCERCISE_ID
where user_id=(select id from users where email='testuser1#gmail.com')
and substr(set_date,1,10)='2013-10-29'
GROUP BY REPORTSETS.USER_ID,REPORTSETS.WORKOUT_LOG_ID,
REPORTSETS.SET_DATE,REPORTSETS.EXCERCISE_ID,REPORTSETS.SET_NUMBER
Two things:
First, You have the following WHERE clause item to pull out a single day's data.
AND substr(set_date,1,10)='2013-10-29'
This definitively defeats the use of an index on the date. If your set_date column has a DATETIME datatype, what you want is
AND set_date >= `2013-10-09`
AND set date < `2013-10-09` + INTERVAL 1 DAY
This will allow the use of a range scan on an index on set_date. It looks to me like you might want a compound index on (user_id, set_date). But you should muck around with EXPLAIN to figure out whether that's right.
Second, you're misusing GROUP BY. That clause is pointless unless you have some kind of summary function like SUM() or GROUP_CONCAT() in your query. Do you want ORDER BY?
Comments on your SQL that you might want to look into:
1) Do you have an index on USER_ID and SET_DATE?
2) Your datatype for SET_DATE looks wrong, is it a varchar? Storing it as a date will mean that the db can optimise your search much more efficiently. At the moment the substring method will be called countless times per query as it has to be run for every row returned by the first part of your where clause.
3) Is the group by really required? Unless I'm missing something the 'group by' part of the statement brings nothing to the table ;)
It should make a significant difference if you could store the date either as a date, or in the format you need to make the comparison. Performing a substr() call on every date must be time consuming.
Surely the suggestions with tuning the query would help to improve the query speed. But I think the main point here is what can be done with more than 1 million plus records before session timed out. What if you have like 2 or 3 million records, will some performance tuning solve the problem? I don't think so. So:
1) If you want to display on browser, use pagination and query (for example) the first 100 record.
2) If you want to generate a report (like pdf), then use asynchronous method (JMS)
I have an employee and a corresponding employee history table.
Both the tables have same structure. History table is used to track the historical changes made to the employee over a period of time.
Now, I need to add an undo function to the changes made to the employee.
e.g. Employees title is changed on 1st August. Now, This will update the employees title in Employee table and insert an corresponding history record in employee_history table.
Now, I need to undo this change. Employee edit page will have a list of changes made to employee datewise with an undo button beside it.
Clicking on undo should revert changes in Employee table to previous value. Also I think the record in history table which says title is changed, should also be removed.
Also when I revert tghe changes to employee table i.e. revert title to previous title, this will fire an insert to history table, which I dont want.
I am not sure what is the best possible way to do this.
Any suggestions will be helpful.
In case you want to implement a "persistent" undo - one that would survive an application restart/session timeout, you should consider extending your DB schema width timestamp fields and either delete the last entry or replace it with an appropriate earlier entry.
A "light" version would be using a stack to store last interactions, including the original and the new value. You could persist the stack on session invalidation of course to combine both approaches. This seems to be what you are actually doing.
You could extend this solution by creating and storing or exporting SQL migration scripts for each change, recording the change and, if possible, the opposite action. So you could even transfer the scripts between application instances and environments and would have a complete "replayability" of your DB states.
tl;dr - it looks like you have already implemented a good solution
I would suggest using a flag telling the trigger/history logic to keep off while you have your undo running and not writing history data.
Normally this would be done by serializer-class feeding from your history table and restoring employee data and later cleaning up history-entries/unlocking history again.
You could maybe use the rollback feature of the transaction.
I have a database with three tables stud_first, stud_second and stud_audit both stud_first and stud_second have the same column names which is
name,
stud-id,
age,
class
number_of-course_taken
I want stud_second to always take any data inserted in stud_first and at the same time stud_audit should keep record of the data copied i.e a log of the name of students and the time they were copied or deleted from stud_first to stud_second. The columns in stud_audit should look like this
name,
time copied
I want to do it mysql alone or combine it with java
Not a complete answer, but this may be enough to get you started in the right direction...
DELIMTER $$
CREATE TRIGGER stud_first_ar
AFTER INSERT ON stud_first
FOR EACH ROW
BEGIN
INSERT INTO stud_second
(`name`, `stud-id`, `age`,`class`,`number_of-course_taken`)
VALUES
(NEW.`name`,NEW.`stud-id`,NEW.`age`,NEW.`class`,NEW.`number_of-course_taken`);
INSERT INTO stud_audit (`name`, `time copied`)
VALUES (NEW.`name`,UTC_TIMESTAMP());
END$$
You could use NOW() in place of UTC_TIMESTAMP(), if you aren't concerned with timezone issues.
The choice of column names containing dashes and spaces is non-standard... it's allowed, but it's usually easier when you avoid doing that.
I would actually have just one audit table, rather than two separate ones. It could be copy of the table with additional columns for "action" (identifying whether the change was due to an INSERT, UPDATE or DELETE), "actor" (identifying the process or user that caused the action, and a UTC timestamp.
You may want to consider "audit" triggers for UPDATE and DELETE actions as well, where you have the special "OLD." record available.
Again, not a complete answer, but this may be enough to get you started in the right direction.