How to find number of database round trips by an application - java

I am a java programmer and I want to know how many database calls/trips are done by my application. We use Oracle as our relational database.
With oracle, I got to know about a way to alter session statistics and generate the trace files. Below are the queries to be fired:
ALTER SESSION SET TIMED_STATISTICS = TRUE;
ALTER SESSION SET SQL_TRACE = TRUE;
After the trace files are generated, they could be read using the TKProf utility. But this approach cannot be used because:
my application uses hibernate and spring frameworks and hence the application does not have an handle to the session.
Even if we get the trace files, I need to know whether the set of queries are fired in one go (in a batch) or separately. I am not sure if TkProf output could help to understand this.
Does anyone have any better suggestions?

In TkProf, you can basically tell the number of round-trips as the number of "calls" (although there are exceptions so that less round trips are required, e.g. parse/execute/fetch of a single row select is, theoretically, possible in a single round trip, the so called "exact fetch" feature of oracle). However as a estimate, the tkprof figures are good enough.
If trace wait events, you should directly see the 'SQL*Net from/to client' wait events in the raw trace, but I think tkprof does not show it (not sure, give it a try).
Another way is to look into the session statistics:
select value
from v$mystat ms, v$statname sn
where ms.value > 0
and ms.statistic#=sn.statistic#
and sn.name IN ('SQL*Net roundtrips to/from client')
However, if you do that in your app, you will slowdown your app, and the figures you receive will include the round-trips for that select.
A wrote a few articles about round-trip optimization:
http://blog.fatalmind.com/2009/12/22/latency-security-vs-performance/
http://blog.fatalmind.com/2010/01/29/oracle-jdbc-prefetch-portability/

Firstly, use a dedicated database (or timeframe) for this test, so it doesn't get easily confused with other sessions.
Secondly, look at the view v$session to identify the session(s) for hibernate. The USERNAME, OSUSER, TERMINAL, MACHINE should make this obvious. The SID and SERIAL# columns uniquely identify the session. Actually the SID is unique at any time. The SERIAL# is only needed if you have sessions disconnecting and reconnecting.
Thirdly, use v$sessstat (filtered on the SID,SERIAL# from the v$session) and v$statname (as shown by Markus) to pull out the number of round trips. You can take a snapshot before the test, run the test, then look at the values again and determine the work done.
That said, I'm not sure it is a particularly useful measure in itself. The TKPROF will be more detailed and is much more focussed on time (which is a more useful measure).

Best would be to get a dedicated event 10046 level 12 tracefile of the running session. You will find there all information in detail. This means that you can see how many fetches the application will do per executed command and the related wait events/elapsed time. The resul can be analyzed using tool from Oracle like TKPROF or Oracle Trace Analyzer or Third party tools like [QueryAdvisor][1].
By the way you can ask your DBA to define a database trigger activating Oracle filetrace automatic after login. So capturing the file should not be the problem.
R.U.
[1]: http://www.queryadvisor.com/"TKPROF Oracle tracefile analysis with QueryAdvisor"

Related

Secure MySQL connection in Java?

I try this:
Connection con = DriverManager.getConnection("jdbc:mysql://localhost:3306/sonoo","root","password");
but it's very easy for someone to hack strings of username and password.
Opening Application with zip, winrar or any else program look like this and read code.
How can I secure my connection?
You need to decide what permissions someone who gets a copy of your JAR has. Do they have permission to run database queries or not?
If they should not: delete the database connection. They don't have permission.
If they should: then they can have the password. They have permission.
What seems to be tripping you up is that you are giving out the root password for your database, and so you want the the third option: "They should be able to do some database queries, but not others."
The JAR file is the wrong place to try to solve that problem. If you try to solve this at the JAR file level, one of two things will happen. Either your users were trustworthy all along and you wasted your time with whatever elaborate scheme you used, or some of your end-users are untrustworthy and one of them will hack you. They will hack you by stepping it through the JVM and editing your query strings right before the JVM sends them out, at the very last second, if they absolutely have to. Everything you do at this level will be security theater, like getting frisked at the airport, it doesn't make you significantly safer but there is a tiny chance that you can say "but we encrypted it!" and your clients might not dump you after the inevitable security breach.
That problem needs to be solved within the database, by creating a user account which does not have the permissions that they should not have. When you do SHOW GRANTS FOR enduser#'%' it will show you only the sorts of queries that they are allowed to do.
In many cases you want to give the user account a more fine-grained permission than just INSERT, SELECT, or UPDATE on a table. For example, you might have the logic "you can add to this table, but only if you also update the numbers in this other table." For these, you should use stored procedures, which can have their permissions set to either "definer" or "invoker": define it by a user with the appropriate permissions and then the invoker gets to have advanced permissions to do this particular query.
In some cases you have a nasty situation where you want to distribute the same application to two different clients, but they would both benefit significantly (at the expense of the other!) from being able to read each other's data. For example you might be an order processor dealing with two rival companies; either one would love to see the order history of the other one. For these cases you have a few options:
Move even SELECT statements into stored procedures. A stored procedure can call user() which can still give you the logged-in user, even though they are not the definer.
Move the database queries out of the shared JAR. Like #g-lulu says above, you can create a web API which you lock down really well, or something like that.
Duplicate the database, move the authentication parameters to a separate file which you read on startup.
Option 3 requires you to write tooling to maintain multiple databases as perfect duplicates of each other's structure, which sucks. However it has the nice benefit over (1) and (2) that a shared database inevitably leaks some information -- for example an auto_increment ID column could leak how many orders are being created globally and there might be ways to determine something like, "oh, they send all of their orders through this unusual table access, so that will also bump this ID at the same time, so I just need to check to see if both IDs are bumped and that'll reveal an order for our rival company".
You can create a webservice in PHP (or java or others). This webservice is stocked on a server and he's contain access and query to your database.
With your desktop app, just send a request (POST, GET) to your web service.
Exemple in PHP webservice :
if (isset($_POST['getMember'])){
do a query in your database
insert result into JSON
return JSON
}

How can I know how many queries are fired in my database?

I have an employee management application. I am using a MySQL database.
In my application, I have functionality like add /edit/delete /view.
Whenever I run any functionality, one query is fired in the database. Like in add employee, it will fire the insert query.
So I want to do something on my database, so that I see how many queries have been fired till date.
I don't want to do any changes on my Java code.
You can use SHOW STATUS:
SHOW GLOBAL STATUS LIKE 'Questions'
As documented under Server Status Variables:
The status variables have the following meanings.
[ deletia ]
Questions
The number of statements executed by the server. This includes only statements sent to the server by clients and not statements executed within stored programs, unlike the Queries variable. This variable does not count COM_PING, COM_STATISTICS, COM_STMT_PREPARE, COM_STMT_CLOSE, or COM_STMT_RESET commands.
Beware that:
the statistics are reset when FLUSH STATUS is issued.
the SHOW STATUS command is itself a statement and will increment the Questions counter.
these statistics are server-wide and therefore will include other databases on the same server (if any exist)—a feature request for per-database statistics has been open since January 2006; in the meantime one can obtain per-table statistics from google-mysql-tools/UserTableMonitoring.
You should execute queries as mentioned below:
To get the SELECT query count, execute Show global status like 'com_select';
To get the UPDATE query count, execute Show global status like 'com_update';
To get the DELETE query count, execute Show global status like 'com_delete';
To get the INSERT query count, execute Show global status like 'com_insert';
You can also analyze the general log or route your application via a MySQL proxy to get all queries executed on a server.
If you don't want to modify your code then you can trace this on the database with triggers. The restriction is that triggers can only fire on insert/update/delete so can't be used to count reads (selects).
Maybe it's too "enterprise" and too "production" for your question.
When you use munin (http://munin-monitoring.org/) (other monitoring-tools have simular extenstions), you can use mysql-monitoring tools which show you how many requests (splitted in Insert/Update/Loaddata/...) you are firing.
With these tools, you see the usage and the load you are producing.
Especially when data changes, and may cause more accesses/load (missing indices, more queries because of big m:n-tables, ...) you recognize it.
It's extremely handy and you can do the check during your break. No typing, no thing, just check the graphs.
I think that the most exact method, which needs no modifications to the database or application in order to operate, would be to configure your database management system to log all events.
You are left with a log file, which is a text file that can be analyzed on demand.
Here is the The General Query Log manual page that will get you started.

Is a good idea do processing of a large amount of data directly on database?

I have a database with a lot of web pages stored.
I will need to process all the data I have so I have two options: recover the data to the program or process directly in database with some functions I will create.
What I want to know is:
do some processing in the database, and not in the application is a good
idea?
when this is recommended and when not?
are there pros and cons?
is possible to extend the language to new features (external APIs/libraries)?
I tried retrieving the content to application (worked), but was to slow and dirty. My
preoccupation was that can't do in the database what can I do in Java, but I don't know if this is true.
ONLY a example: I have a table called Token. At the moment, it has 180,000 rows, but this will increase to over 10 million rows. I need to do some processing to know if a word between two token classified as `Proper Name´ is part of name or not.
I will need to process all the data. In this case, doing directly on database is better than retrieving to application?
My preoccupation was that can't do in the database what can I do in
Java, but I don't know if this is true.
No, that is not a correct assumption. There are valid circumstances for using database to process data. For example, if it involves calling a lot of disparate SQLs that can be combined in a store procedure then you should do the processing the in the stored procedure and call the stored proc from your java application. This way you avoid making several network trips to get to the database server.
I do not know what are you processing though. Are you parsing XML data stored in your database? Then perhaps you should use XQuery and a lot of the modern databases support it.
ONLY an example: I have a table called Token. At the moment, it has
180,000 rows, but this will increase to over 10 million rows. I need
to do some processing to know if a word between two token classified
as `Proper Name´ is part of name or not.
Is there some indicator in the data that tells it's a proper name? Fetching 10 million rows (highly susceptible to OutOfMemoryException) and then going through them is not a good idea. If there are certain parameters about the data that can be put in a where clause in a SQL to limit the number of data being fetched is the way to go in my opinion. Surely you will need to do explains on your SQL, check the correct indices are in place, check index cluster ratio, type of index, all that will make a difference. Now if you can't fully eliminate all "improper names" then you should try to get rid of as many as you can with SQL and then process the rest in your application. I am assuming this is a batch application, right? If it is a web application then you definitely want to create a batch application to do the staging of the data for you before web applications query it.
I hope my explanation makes sense. Please let me know if you have questions.
Directly interacting with the DB for every single thing is a tedious job and affects the performance...there are several ways to get around this...you can use indexing, caching or tools such as Hibernate which keeps all the data in the memory so that you don't need to query the DB for every operation...there are tools such as luceneIndexer which are very popular and could solve your problem of hitting the DB everytime...

JDBC: is there are away to get a size of committed data in bytes

If I have a java application that performs some inserts against a database, if there an easy way to get how much bytes was committed (i.e. sum size of all the data in all the fields), without having to calculate it manually / fetching and checking the size of the result set?
--
As lucho points out, implementing statistics-aware statement class on top of the PreparedStatement might be the way to go. Going to stick with that and see how well this is going to work.
As far as I know, nope.
You'll have to ask your database that question; perhaps it's possible to do it without querying the same thing you inserted (because that sounds a bit pointless).
Interesting problem. I like lucho's solution, but I have two quicker (hackier) options:
You can try to use InnoDB's SHOW TABLE STATUS and keep a running log of the data size. That would let you know, but on my development machine calling it on one database takes 5.3s (56 tables) so unless you only want the data for one or two tables it's probably too slow (not to mention whatever locking it may incur).
You could monitor the DB process and use the OS to tell you how much it's writing. I know Windows can tell you this, and I'm pretty sure Linux can as well. But if you host 3 databases you'll only get the total, and it will be off some due to transactions and such.
Just random ideas.

Verfying a database is as you expect it it be

I've been writing a java app on my machine and it works perfectly using the DB I set up, but when I install it on site it blows up because the DB is slightly different.
So I'm in the process of writing some code to verify that:
A: I've got the DB details correct
B: The database has all the Tables I expect and they have the right columns.
I've got A down but I've got no idea where to start with B, any suggestions?
Target DB is for the current client is Oracle, but the app can be configured to run on SQL Server as well. So a generic solution would be appreciated, but is not nessisary as I'm sure I can figure out how to do one from the other.
You'll want to query the information_schema of the database, here are some examples for Oracle, every platform I am aware of has something similar.
http://www.alberton.info/oracle_meta_info.html
You might be able to use a database migration tool like LiquiBase for this -- most of these tools have some way of checking the database. I don't have first hand experience using it so it's a guess.
I use DbUnit to test databases. It is a Java based solution, that integrates well with Junit. It is possible to use it with almost no Java. I havent used it in exactly the same situation as you described, but it should be close enough to work.
Most generic solution would be to execute queries with select clause having the expected coulmns and from clause having table names, within try catch block. You can put where clause as 1=2 so as not to fetch any data. If query executed without throwing exception then you have got the expected table and columns.
The slightly different piece might be better handled by scripting the creation of the database in the first place. A automated process gives you a better chance of making the two identical.
Another point worth making is that you minimize your risk by making your devl and prod environments identical - same database schema and vendor for both. Change the circumstances that make the two different.
Lastly, you don't say what is "slightly" different, but sometimes these are unavoidable (e.g. Oracle uses sequences, SQL Server uses identities). Maybe Hibernate can help you to switch between vendors more reliably. It abstracts details in such a way that changing databases can mean modifying a single value in a configuration file.
What you need to have is basically Unit Tests for your database. "A column must exist named FOOBAR, the type must be Integer. No foreign keys may exist etc."
This is doable with plain JUnit and JDBC (ask the table for its meta-data) as you may want to ensure that you are absolutely certain what is being done which may be harder when using e.g. dbUnit.
You can check for the presence of tables, columns, views, etc. using these tables in Oracle
USER_TABLES
USER_VIEWS
USER_PROCEDURE
(or for everything)
USER_OBJECTS WHERE OBJECT_TYPE = '??'
To keep going... USER_TAB_COLS for table columns
Regards
K
I use MigrateDB for this. It lets you build queries that do things like check for the existence of given tables, columns, rows, indexes, etc. for a given database and use those as "tests." If a test fails, it triggers an "action" (which is just another query that knows how to remedy the problem.)
MigrateDB supports multiple database platforms (you can specify the "check for table existence query" for each platform, for example), completely configurable tests (you can make your own up), comes with fairly complete Oracle tests, and can be run in "audit only" mode so that it only tells you what the differences are.
It's a nice, robust solution.
If you're using plain JDBC, you should try utilizing this method: DatabaseMetadata.getTables and other similar methods available in the metadata class.

Categories