I write plugin for JMeter to automate database load testing. The main idea of plugin is create automate database load test when provided a minimum of information (connection string, quantity of users, throughput, schemas of tables that is tested, etc). I understand that such random test doesn't match to real behavior of user but purpose is to launch test in few minutes and get statistic of DB work such as select or DML queries time of response.
What aspect of database can be tested automaticly? How can I automaticly test this aspects? For example tables, index, trigers, functions...
What statistic I can get in such way?
Are there similar software?
Do You have any idea? :)
At first I get metadata from DB for all tables and create queries(DML and all posible select for 1 table) from existing data in DB and put them to queue. Then sampler get the query from queue and execute it.
A couple of questions to help you need to get to where you are going first.
1) What are the exact metrics you are looking to test?
2) Is this for comparing different database servers / end products, or is it for a specific database to measure how well your indexes are set up?
3) Will it be creating multiple concurrent connections to test record locking, etc?
I believe that you should start with official documentation such as Building a Database Test Plan.
Once you're comfortable with JDBC Connection Configuration details, have ojdbc6.jar in JMeter classpath (usually /lib folder) and will be able to execute sample query like
select sysdate from dual;
you can start looking into i.e. Using JDBC Sampler in JMeter 2.6 guide for advanced information.
Related
I have a java application using oracle DB, running on apache tomcat. During normal day, the java app runs fine. However, the traffic was double on a day, and the app starts to encounter increase in response time and timeouts.
After that, we tried run load test using jmeter with the same amount of load experienced but never encountered any responsive/timeout issues from the testing. BTW, we checked our network monitoring tools, no issues with the infra.
Can I check what should I be looking for if I want to replicate the same issue during testing? Replicating this would help to ensure that the changes we are going to do would works.
Thanks!
The underlying question is: how to make the load test realistic enough to replicate slowness in accessing database observed in production. The thought process in our comments exchange was to review and rule-in or rule-out factors that are often not properly emulated in load tests and give over-optimistic performance result. I reviewed 4 factors:
Does the load test script correctly correlate dynamic values? Yes it does, because the records created by the load test match with the scenario. If this were not be the case, then failed transactions in the load test would be responsible for too fast response. The recommendation would be to correlate your script manually.
Does the load test script correctly emulate multiple authenticated users? Application does not require login. If this were not the case, then running load test with a single user will fail to test system overheads on maintaining multiple user sessions. The recommendation would be to parameterize recorded credentials using dataset with multiple credentials.
Does the load test script correctly emulate anonymous users authenticated through cookies? Application does not use cookie authentication. If this were not the case, then the recommendation would be to clear browser cache before recording to make sure that the stale cookie is not recorded and then make sure that cookie correlation in the script is configured properly.
Does the load test script correctly emulate data in the recorded scenario? Assuming the test scenario recorded some user entry that is used as a criteria for a database query. If you replay multiple iterations emulating the same entry, the database may not be hit for such queries due to application or database caching. If this is the case, then the recommendation is to parameterize user entries using test datasets.
If the last factor is also not the case, then there are more factors to go through. For more about correlation and parameterization load tests check this blog http://www.stresstimulus.com/blog/post/eradicating-load-testing-errors-1
I am newbie in JBoss BPM Suite. What i want to achieve, is to access my MySQL database through a business process. I have already added a datasource to my application server including the jdbc driver. What i tried to do was to connect to my db by a script task. Although i got an exception ClassNameNotFound for my driver class 'com.mysql.jdbc.Driver'. What is the right way to connect to the db? Is there a way to do this by a service task? Or a WorkItemHandler?
Thanks in advance.
It is not recommended to execute any complicated logic (like accessing the database) in a script task. I would also assume that your application server does not put database drivers on the classpath of its applications since it is against the whole idea of datasources. You just need to make use of the datasource you have already configured.
When it comes to the right way how to connect to the database inside your process, you will need to implement your own work item handler where you can get your data from the database. There are many different ways how you can achieve this. You can find inspiration from JPAWorkItemHandler which will be available in version 7.
I have finally made the connection to my database by creating a WorkItemHandler and add it as a dependency to my BPM Suite project. After a lot of search, i think this is the best way to do it if anyone wants to access his database in a business process.
What are the options to index large data from Oracle DB to elastic search cluster? Requirement is to index 300Million records one time into multiple indexes and also incremental updates having around approximate 1 Million changes every day.
I have tried JDBC plugin for elasticsearch river/feeder, both seems to be running inside or require locally running elastic search instance. Please let me know if there is any better option for running elastic search indexer as a standalone job (probably java based). Any suggestions will be very helpful.
Thanks.
We use ES as a reporting db and when new records are written to SQL we take the following action to get them into ES:
Write the primary key into a queue (we use rabbitMQ)
Rabbit picks up the primary key (when it has time) and queries the relation DB to get the info it needs and then writes the data into ES
This process works great because it handles both new data and old data. For old data just write a quick script to write 300M primary keys into rabbit and you're done!
there are many integration options - I've listed out a few to give you some ideas, the solution is really going to depend on your specific resources and requirements though.
Oracle Golden Gate will look at the Oracle DB transaction logs and feed them in real-time to ES.
ETL for example Oracle Data Integrator could run on a schedule and pull data from your DB, transform it and send to ES.
Create triggers in the Oracle DB so that data updates can be written to ES using a stored procedure. Or use the trigger to write flags to a "changes" table that some external process (e.g. a Java application) monitors and uses to extract data from the Oracle DB.
Get the application that writes to the Oracle DB to also feed ES. Ideally your application and Oracle DB should be loosely coupled - do you have an integration platform that can feed the messages to both ES and Oracle?
Is there a way that I can use JDBC to target multiple databases when I execute statements (basic inserts, updates, deletes).
For example, assume both servers [200.200.200.1] and [200.200.200.2] have a database named MyDatabase, and the databases are exactly the same. I'd like to run "INSERT INTO TestTable VALUES(1, 2)" on both databases at the same time.
Note regarding JTA/XA:
We're developing a JTA/XA architecture to target multiple databases in the same transaction, but it won't be ready for some time. I'd like to use standard JDBC batch commands and have them hit multiple servers for now if its possible. I realize that it won't be transaction safe, I just wan't the commands to hit both servers for basic testing at the moment.
You need one connection per database. Once you have those, the standard auto commit/rollback calls will work.
You could try Spring; it already has transaction managers set up.
Even if you don't use Spring, all you have to do is get XA versions of the JDBC driver JARs in your CLASSPATH. Two phase commit will not work if you don't have them.
I'd wonder if replication using the database would not be a better idea. Why should the middle tier care about database clustering?
Best quick and dirty way for development is to use multiple database connections. They won't be in the same transaction since they are in different connection. I don't think this would be much of an issue if this is just for testing.
When your JTA/XA architecture is ready, just plug it into the already working code.
I need to develop some services and expose an API to some third parties.
In those services I may need to fetch/insert/update/delete data with some complex calculations involved(not just simple CRUD). I am planning to use Spring and MyBatis.
But the real challenge is there will be multiple DB nodes with same data(some external setup will takes care of keeping them in sync). When I got a request for some data I need to randomly pick one DB node and query it and return the results. If the selected DB is unreachable or having some network issues or some unknown problem then I need to try to connect to some other DB node.
I am aware of Spring's AbstractRoutingDataSource. But where to inject the DB Connection Retry logic? Will Spring handle transactions properly if I switch the dataSource dynamically?
Or should I avoid Spring & MyBatis out-of-the-box integration and do Transaction management by myself using MyBatis?
What do you guys suggest?
I propose to you using of NoSQL database like MongoDB. It is easy clustering. You can configure for example use 10 servers and do replication of data 3 times.
Thats mean that if 2 of your 10 servers will fails - you still got data save.
NoSQL databases is different comparing to RDBS, but they can give hight performance for clustering.
Also, there is no transactions support for NoSQL - you have to do it manually in case of financial operations.
Actually you should thing in different way when developing with NoSQL.
Yes, it will work. Get AbstractRoutingDataSource and code your own one. The only thing you cannot do is to change the target database while a transaction is running.
So what you have to do is putting the db retry code in the getConnection. If during the transaction that connection becomes invalid you should let it fail.