i have got listing screens in my web app that pull quite a heavy of data from oracle database.
Each time the listing screen loads it goes to DB and pull data.
What I want is ,i want to have some caching teching technique that can extract data from DB and keep that in memory and that when any next request is made I should be getting data.and just like DB I should be able to filter out data from that with any sql query,jst that it won't go to DB rather pull data from memory.so that set of extracted data will be just like a view of the table and it should consistently moniter the corresponding tables so that if any update operation is made on d table it should again fetch new set of data from DB and serve.
Is there any API in java to achieve d same?
in ADO.net they hv got something like recordset...i dnt know much about that.
so is there any way out.my app is based on J2EE and oracle as DB.we hv got jboss as d server.Any suggestion is welcome.Thanks.
Try using Ehcache, it supports JDBC caching. And avoid creating custom solutions, if you're not JDBC guru.
You could cache the results of your query in memcached.
When your application modifies the table that you're caching, delete the cached item out of your memcached instances.
I found this quick guide to be useful: http://pragprog.com/titles/memcd/using-memcached
you can store that data into an in-memory dataset.
give this library a try:
http://casperdatasets.googlecode.com
you can iterate and scroll through the results just like a resultset, issue queries on it, sort the data, and create indexes to optimize searches - and its all memory-based.
I have 2 options for this
1) Jboss cache and you check all the details at the following link
JBOSS Cache
Related
We have an application that runs with any of IBM Informix, MySQL and Oracle, and we are using Java with Hibernate to connect to the database. We will store XML, CSV and other text-based files inside the database (clob column). The entities in Java are byte[] objects.
One feature request to the application is now to "grep" content inside the data. So I need to find all files with a specific content.
On regular char/varchar fields I can use like '%xyz%', but this is not working on byte[] / blobs.
The first approach was to load each entity, cast the byte[] into a string and use the contains method in Java. If the use enters any filter parameters on other (non-clob) columns, I will apply those filters before testing the clob in order to reduce the number of blobs I have to scan.
That worked quite well for 100 files (clobs) and as long as the application and database are on the same server. But I think it will get really slow if I have 1.000.000 files inside the database and the database is not always in the same network. So I think that is not a good idea.
My next thought was creating a database procedure. But I am not quite sure if this is possible for Informix, MySQL and Oracle. And I am not sure if this is possible.
The last but not favored method is to store the content of the data not inside a clob. Maybe I can use a different datatype for that?
Does anyone has a good idea how to realize that? I need a solution for all three DBMS. The application knows on what kind of DBMS it is connected to. So it would be okay, if I have three different solutions (one for each DBMS).
I am completely open to changing what kind of datatype I use (BLOB, CLOB ...) — I can modify that as I want.
Note: the clobs will range from about 5 KiB to about 500 KiB, with a maximum of 1 MiB.
Look into Apache Lucene or other text indexing library.
https://en.wikipedia.org/wiki/Lucene
http://en.wikipedia.org/wiki/Full_text_search
If you go with a DB specific solution like Oracle Text Search you will have to implement a custom solution for each database. I know from experience that Oracle Text search takes significant time to learn and involves a lot of tweaking to get just right.
Also, if you use a DB solution you would receive different results in each DB even if the data sets were the same (each DB would have it's own methods of indexing and retrieving the data).
By going with a 3rd party solution like Lucene -- you only have to learn one solution and results will be consistent regardless of the Db.
This is my first stack-overflow post, so please ignore/forgive if I am not being specific enough.I'm sure I will learn the process gradually.
I have built a JSON to be displayed in angular data grid. This JSON comes from a complex query over materialized view.My thought to refresh the JSON as underlying data changes is as follows:
a) Register query for Oracle CQRN (Oracle Continuous Query Result Change Notification) at application startup
b) When the Underlying data changes, Oracle Database Change Listener in Java side gets invoked and Ire-query the data (with change) and push it to socket end-point. That way the JSON gets changed with latest data.
This works fine with simple query.
Issues are:
a) In my case the query is very complex and involves multiple materialized views with UNION ALL and complex JOINS. CQRN does not support materialized view registration for query result change.
b) The query I am registering at start-up, for query result change notification, is pretty static. It does not meet the requirement of various different parameterized queries behind the data-grid.
Can anyone suggest any other alternative for example cache the grid data in the middle-tire and and refresh cache with updated data whenever the underlying grid data changes. I should be notified when underlying grid data changes so I will re-query & send the updated data to socket end-point, which will refresh the grid.
I have to show the grid-data changes in real-time, so I have used Java WebSocket (JSR 356)
Technology stack:
UI: Javascaipt/AngularJS
Middle-tier: Java 1.7
Server: Jetty 9.2
Database: Oracle 11g R2
Build Platform: Maven 3.3
Suggestion for any other suitable approach also will be much appreciated.
Thanks & regards,
- Joy
While not directly answering your question we just implemented a real time data grid involving multiple data sources and CQRN. This built in is based on a table changing. Our technique was:
add on insert trigger (data feed was real time, no deletes, no updates) to the base tables
call a stored procedure to manipulate the data. You would use the logic in your materialized view. The procedure inserts data into a destination table. That has a trigger to call the CQRN.
often with realtime you need to delete old data so everything stays fast
I'm new to open source stacks and have been playing with hibernate/jpa/jdbc and memcache. I have a large data set per jdbc query and possibly will have a number these large data sets where I eventually bind to a chart.
However, I'm very focused on performance instead of hitting the database per page load to display it on my web page chart.
Are there some examples of how (memcache, redis, local or distributed) and where to cache this data (jSON or raw result data) to load in memory? Also I need to figure out how to refresh the cache unless it's a time based eviction marking algorithm (i.e. 30min expires so grab new data from data base query instead of using cache or perhaps an automated feed of data into the cache every xhrs/min/etc).?
Thanks!
This is typical problem and solution not straight forward. There are many factor which determine your design. Here is what we did sometime ago
Since our queries to extract data were a bit complex (took around a min to execute) and large dataset, we populated the memcache from a batch which used to pull data from database every 1 hour and push it to the memcached. By keeping the expiry cache larger than the batch interval, we made sure that where will always be data in cache.
There was another used case for dynamic caching, wherein on receiving the request for data, we checked first the memcached and if data not found, query the database, fetch the data, push it to memcached and return the results. But I would advise for this approach only when your database queries are simple and fast enough not to cause the poor overall response.
You can also used Hibernat's second level cache. It depends on your database schema, queries etc. to use this feature efficiently.
Hibernate has built-in support for 2nd level caching. Take a look at EhCache for example.
Also see: http://docs.jboss.org/hibernate/orm/3.3/reference/en/html/performance.html#performance-cache
I have Problem with SQl server Performance because of Heavy Calculation query,
so we decided that we put Solr as intermediate and index all data from either Hibernate or Direct from SQl server,
so can anybody suggest/help me that it is possible ?
please suggest any tutorial link for this.
You can use DataImportHandler to transfer data, which you can schedule using DataImportScheduler.
I had the similar problem where SQL Server SP took 12 hours to update relationships between objects (rows), so we ended up using Neo4j (open source graph database), which exactly matched our data model.
We needed object relationships to be reflected in Solr searches, e.g. give me all objects whose name starts with "obj" and whose parent is of type "typ".
My use case is as follows --
I have a database table with around 1000+ entries and this table is updated/edited infrequently but i expect this to change in future. Some of the columns in the table contain strings that are of considerable length.
Now I am in the process of writing a UI application that will have some mouseover events that will display texts derived from the aforementioned database table.
I have, for my use case, decided to write a backend 'server' that will host an in-memory database that will have all the data that was present in the aforementioned table. The UI app will now, on startup, cache the required data from the in-memory database present or hosted by the backend server.
Does my use case justify using an in-memory database ? If not, what are the alternatives I should consider ?
EDIT 1 --
My use case also involves running multiple searches of varying complexity on the database very frequently.
Thanks
p1ng
Seems like an excellent use-case for an in-memory database. Writing it yourself, on the other hand, is probably not the way to go.
There are plenty of existing options for just about any imaginable scenario: http://en.wikipedia.org/wiki/In-memory_database
If you're doing complex searches on text data, Lucene is quite excellent. It has special in-memory storage backends, but really, it doesn't matter for such a tiny dataset - it will always be quickly cached anyway.