Is there any way for us to query the db to suggest index creation/index deletion that would improve the performance of the db system?
We understand that a dba can manually view the trace files to create/drop indices but can i write a java program that queries the db engine to suggest the same automatically.
Or some open source tools that i can check out to perform the same automatically.
Thx.
Well there's no standard JDBC way to do this. There may be specific driver implementations for specific DBS that would allow you to EXPLAIN your query (trace the use of indexes), etc. But there's no one-size fits all answer here.
in general I would lean to saying NO.
Your index performance is dependent on the queries you would fire on them...So No !
With MySQL specifically, you can flag slow queries, as well as queries not using indexes.
Ultimately, the database will do its best (within what you've created) to optimize your query.
http://dev.mysql.com/doc/refman/5.0/en/slow-query-log.html
This data would be logged to a file, and not necessarily available via an API.
Related
I am writing a Google Dataflow Pipeline and as one of the Sources I require a MySQL resultset via a query. A couple of questions then:
What would be proper way to extract data from MySQL as a step in my pipeline, can this simply be done in-line using JDBC?
In the case that I indeed do need to implement "User-Defined Data Format" wrapping MySQL as a source, does anyone know if an implementation already exists and I do not need to reinvent the wheel? (don't get me wrong I would enjoy writing it, but I would imagine this would be quite a common scenario to use MySQL as a source)
Thanks all!
At this time, Cloud Dataflow does not provide MySQL input source.
The preferred way to implement support for this is to implement a user-defined input source that can handle MySQL queries.
An alternative way would be to execute the query in the main program and stage the results of the query to a temporary location in GCS, process the results using Dataflow, and remove the files in temporary.
Hope this helps
A JDBC connector has been just added to Apache Beam (incubating). See JdbcIO.
Could you please clarify the need for GroupByKey in the above example? Since the previous ParDo (ReadQueryResults) returns rows key'd on primary key, wouldn't the GroupByKey essentially create a group for each row of the result set? The subsequent ParDo (Regroup) would have parallelized the processing per row even without the GroupByKey, right?
I got the following problem and no idea on how to solve it.
I am using SqlJet to access and manage a small Sqlite DB.
It's amazing the simplicity of this library! But now, I have to execute a complex query which cannot be expressed using the methods provided by the library. On the website, they say
SQLJet does not support SQL queries; there is an API to work with the
database on a lower level
And I was wondering, anybody of you know what is this lower level API to execute SQL queries?
As far as I can tell from my research, there is no support for arbitrary SQL queries in SQLJet. I first thought it was a sqlite wrapper, which it isn't, so there is also no way to use JDBC.
Other than the lookup rows by name and index stuff, the "lower level" API they are talking about on their page, seems just to be direct access to the internal DB representation, as mentioned in this post.
From a quick glance at the JavaDoc and sources, it might be the SqlJetDb.getSchema() and getTable() methods. There is no proper API documentation or Tutorials, there is even a complaint about this in the issue tracker.
Maybe you can describe what you want to do. Either there is a way to do it though the limited interface SQLJet offers, or I'm afraid I can't help.
Its also good idea to contact the developers directly, as it seems there is not much information to be found elsewhere.
I'd like to save persistent objects to the file system using Hibernate without the need for a SQL database.
Is this possible?
Hibernate works on top of JDBC, so all you need is a JDBC driver and a matching Hibernate dialect.
However, JDBC is basically an abstraction of SQL, so whatever you use is going to look, walk and quack like an SQL database - you might as well use one and spare yourself a lot of headaches. Besides, any such solution is going to be comparable in size and complexity to lighweight Java DBs like Derby.
Of course if you don't insist absolutely on using Hibernate, there are many other options.
It appears that it might technically be possible if you use a JDBC plaintext driver; however I haven't seen any opensource ones which provide write access; the one I found on sourceforge is read-only.
You already have an entity model, I suppose you do not want to lose this nor the relationships contained within it. An entity model is directed to be translated to a relational database.
Hibernate and any other JPA provider (EclipseLink) translate this entity model to SQL. They use a JDBC driver to provide a connection to an SQL database. This, you need to keep as well.
The correct question to ask is: does anybody know an embedded Java SQL database, one that you can start from within Java? There are plenty of those, mentioned in this topic:
HyperSQL: stores the result in an SQL clear-text file, readily imported into any other database
H2: uses binary files, low JAR file size
Derby: uses binary files
Ashpool: stores data in an XML-structured file
I have used HyperSQL on one project for small data, and Apache Derby for a project with huge databases (2Gb and more). Apache Derby performs better on these huge databases.
I don't know exactaly your need, but maybe it's one of below:
1 - If your need is just run away from SQL, you can use a NoSQL database.
Hibernate suports it through Hibernate OGM ( http://www.hibernate.org/subprojects/ogm ).
There are some DBs like Cassandra, MongoDB, CouchDB, Hadoop... You have some suggestions Here
.
2 - Now, if you want not to use a database server (with a service process running always), you can use Apache Derby. It's a DB just like any other SQL, but no need of a server. It uses a singular file to keep data. You can easily transport all database with your program.
Take a look: http://db.apache.org/derby/
3 - If you really want some text plain file, you can do like Michael Borgwardt said. But I don't know if Hibernate would be a good idea in this case.
Both H2 and HyperSQL support embedded mode (running inside your JVM instead of in a separate server) and saving to local file(s); these are still SQL databases, but with Hibernate there's not many other options.
Well, since the question is still opened and the OP said he's opened to new approaches/suggestions, here's mine (a little late but ok).
Do you know Prevayler? It's a Java Prevalence implementation which keep all of your business objects in RAM and mantain Snapshots/Changelogs in the File System, this way it's extremely fast and reliable, since if there's any crash, it'll restore it's last state and reapply every change to it.
Also, it's really easy to setup and run in your app.
Ofcourse this is possible, You can simply use file io features of Java, following steps are required:-
Create a File Object
2.Create an object of FileInputStream (though there are ways which use other Classes)
Wrap this object in a Buffer object or simply inside a java.util.Scanner.
use specific write functions of the object created in previous step.
Note that your object must implement Serializable interface. See following link,
We are developing an application which uses JDBC API. We deal with lot of queries
for various use cases. Any idea on how to isolate these SQL queries out of the
code? Is there any pattern or configuration API would help to manage these queries?
I'm not quite sure what you're asking, but perhaps setting up some stored procedures would provide the layer of abstraction you desire...
You could set up stored procedures for each usage case, and then the queries in your code would consist simply of calling the various procedures.
You can use the MyBatis SQL Mapper for this. You segregate your SQL statements into XML files. It's very clean and very intuitive if you are familiar with SQL.
Use a config/resource file and read the query string from it? A HashTable would be another option as well.
You can try to place your queries into separate files as is and then put files' names into any configuration file.
Use http://www.mybatis.org/ to externalise the queries into XML files
Ensure that classes that access the database only do this, and not other logic
I am working with Jsbs and want to select a DBMS for my application that require a native XML database. Can you people guide me?
1) how many DBMS are supported by Java (is it true almost all DBMS are supported by java?)
2) Which one will be the best selection for XML storage and retrieval?
Thanks in advance.
Here is a list of JDBC Drivers and the DBs they work for. Probably every database out there has a JDBC driver.
As far as supporting XML it depends if you want to be able to do queries against the XML or not. Most modern DBMSs support XML to some degree. Do you have one you are already using, or that you are recommended to use ?
1) Yes, there are JDBC drivers for all the major DBMS (Oracle, MySQL, Postgres, DB2...) and also some interesting java DBMS like HSQL
2) As far as I know Oracle, DB2, PostgreSQL (and probably MySQL) all have XML column types
It is not obvious to me that you need anything more than support for Blobs or Clobs to implement simple XML storage and retrieval. You'd only need special XML support if you needed to perform queries against the data contained in the XML.
What you are talking about is an "XML enabled" RDBMS. Depending on your actual requirements, you may also want to look into native XML databases (NXDs). There is even a standard Java API (XQJ) for querying NXDs, though not all vendors support it.
Most modern databases have JDBC-drivers, which is what is needed for Java programs to connect to the database. You generally want type 4 drivers which do not depend on native code.
For starting I would recommend Apache Derby, which is written in Java and can be part of your program, which keeps it simple. http://db.apache.org/derby/. If you later find you need another database you replace the JDBC-driver, and double-check your SQL-statements.