I have a java application using oracle DB, running on apache tomcat. During normal day, the java app runs fine. However, the traffic was double on a day, and the app starts to encounter increase in response time and timeouts.
After that, we tried run load test using jmeter with the same amount of load experienced but never encountered any responsive/timeout issues from the testing. BTW, we checked our network monitoring tools, no issues with the infra.
Can I check what should I be looking for if I want to replicate the same issue during testing? Replicating this would help to ensure that the changes we are going to do would works.
Thanks!
The underlying question is: how to make the load test realistic enough to replicate slowness in accessing database observed in production. The thought process in our comments exchange was to review and rule-in or rule-out factors that are often not properly emulated in load tests and give over-optimistic performance result. I reviewed 4 factors:
Does the load test script correctly correlate dynamic values? Yes it does, because the records created by the load test match with the scenario. If this were not be the case, then failed transactions in the load test would be responsible for too fast response. The recommendation would be to correlate your script manually.
Does the load test script correctly emulate multiple authenticated users? Application does not require login. If this were not the case, then running load test with a single user will fail to test system overheads on maintaining multiple user sessions. The recommendation would be to parameterize recorded credentials using dataset with multiple credentials.
Does the load test script correctly emulate anonymous users authenticated through cookies? Application does not use cookie authentication. If this were not the case, then the recommendation would be to clear browser cache before recording to make sure that the stale cookie is not recorded and then make sure that cookie correlation in the script is configured properly.
Does the load test script correctly emulate data in the recorded scenario? Assuming the test scenario recorded some user entry that is used as a criteria for a database query. If you replay multiple iterations emulating the same entry, the database may not be hit for such queries due to application or database caching. If this is the case, then the recommendation is to parameterize user entries using test datasets.
If the last factor is also not the case, then there are more factors to go through. For more about correlation and parameterization load tests check this blog http://www.stresstimulus.com/blog/post/eradicating-load-testing-errors-1
Related
We have several Java standalone applications (in form of Jar files) running on multiple servers. These applications mainly read and stream data between systems. We are using Java 8 mainly in our development. I was put in charge recently. My main function is to manage and maintain these apps.
Currently, I check these apps manually by accessing these servers, check if the app is running, and sometimes run some database queries to see if the app started pulling data. My problem is that in many cases, some of these apps fail and shutdown due to data issue or edge cases without anyone noticing. We need some monitoring and application recovery in place.
We don't have docker infrastructure in place. We plan to implement docker in the future, but for now this is not an option.
After research, the following are options I thought of or solutions I tried:
Have the apps create a socket client which sends a heartbeat to a monitoring app (which needs to be developed). I am keeping this as my last option.
I tried to use Eclipse Vertx to wrap the apps into Verticles. Then create a web view that can show me status and other info. After several tries, the apps fail to parse the data correctly (might be due to my lack of understanding to Vertx library).
Have a third party solution that does this, but I have no idea what solutions are out there. I am open for suggestions.
My requirements are:
Proper monitoring of the apps running and their status.
In case of failure, the app should start again while notifying the admin/developer.
I am willing to develop a solution or implement a third party one. I need you guidance on this.
Thank you.
You could use spring-boot-actuator (see health). It comes with a built-in endpoint that has some health checks(depending on your spring-boot project), but you can create your own as well.
Then, doing a http request to http://{host}:{port}/{context}/actuator/health (replace with yours), you could see those health checks status and also use the response status code to monitor your application.
Have you heard of Java Service Wrappers? Not a full management functionality, however it would monitor for JVM crashes and out of memory conditions and restart your application for sure. Alerting should also be possible.
There is a small comparison table here: https://yajsw.sourceforge.io/#mozTocId284533
So some basic monitoring and management is included already. If you need more, I suggest using JMX (https://www.oracle.com/java/technologies/javase/javamanagement.html) or Prometheus (https://prometheus.io/ and https://github.com/prometheus/client_java)
I write plugin for JMeter to automate database load testing. The main idea of plugin is create automate database load test when provided a minimum of information (connection string, quantity of users, throughput, schemas of tables that is tested, etc). I understand that such random test doesn't match to real behavior of user but purpose is to launch test in few minutes and get statistic of DB work such as select or DML queries time of response.
What aspect of database can be tested automaticly? How can I automaticly test this aspects? For example tables, index, trigers, functions...
What statistic I can get in such way?
Are there similar software?
Do You have any idea? :)
At first I get metadata from DB for all tables and create queries(DML and all posible select for 1 table) from existing data in DB and put them to queue. Then sampler get the query from queue and execute it.
A couple of questions to help you need to get to where you are going first.
1) What are the exact metrics you are looking to test?
2) Is this for comparing different database servers / end products, or is it for a specific database to measure how well your indexes are set up?
3) Will it be creating multiple concurrent connections to test record locking, etc?
I believe that you should start with official documentation such as Building a Database Test Plan.
Once you're comfortable with JDBC Connection Configuration details, have ojdbc6.jar in JMeter classpath (usually /lib folder) and will be able to execute sample query like
select sysdate from dual;
you can start looking into i.e. Using JDBC Sampler in JMeter 2.6 guide for advanced information.
I read somewhere use of webservcies in apps. After a lot of research I am able to create one Webservice which will accept Json and JsonP both format as request and response accordingly. I developed the webservcies using Java, Apache Axis2, Hibernate and MySQL as database. there are few problems and I dont know how to solve ?
Insert or delete option, sometimes if at a time more than two users call that service that is insert or delete any row the queries go in sleep mode and next time someone tries to fetch that service he couldnt. Accroding to server log it says error SQL Lockout State. If I checks Processlist in MYSQL it is showing that query in Sleep, I have to kill to resume.
The performance of webservice doesnt seems to be upto mark, it takes time some more time as what i experienced it shouldn't. In simple words how to obtain better performance by the services
How to implement security feature such that if a user logins he/she can be provided an id and validation of that id so that unauthorized access can be prevented
Or just guide me what should be the most appropriate and optmized Webservice methodology that can be used using Java
Answer to this question is not specific to Android. Below are my investigations which might be useful for you.
For the point about MySQL connections going to sleep mode, you can do the following.
Debug the datasource used by Hibernate, try to increase the pool size & check for any issues in it.
Define a timeout period for connections. JBoss has several configurations related to this like blocking-timeout-millis, idle-timeout-minutes etc.
Declare a mechanism to validate periodically the connection resources in the pool for activeness. You can explore OracleStaleConnectionChecker for options.
Configure miniumn connections in the pool. This is important because when all the stale connections are discarded, empty pool needs to be pre-filled & ready with active connections.
Coming to performance of Insert/Delete operations & SQL Lockout State, please try to re-order the sequence of the queries which you are firing to DB at every request. This may not be a deadlock situation but sequencing DB queries correctly will definitely lead to less lockout time and better performance.
This answer may be of use for you. Hibernate: Deadlock found when trying to obtain lock
Web-services which you have developed may require some performance optimization to make them upto the mark. Below are first few steps you can take to bring the performance up.
Avoid nested loops. Every extra parameter in the iterated lust increase the order of the lopp exponentially.
Remove early initialization of objects. This may lead to long unwanted GC cycles.
Apart from above optimizations, there are several frameworks & tools at your service to evaluate the code quality & its performance. PMD, FindBugs, JMeter, Java profiler are few of them to name.
Shishir
You are going to have to profile your server and see where the time is spent. I really like YourKit for doing thread profile. visualvm which comes with the JDK can help also.
There are all sorts of reasons your web service can be slow:
Latency from client to server
Handling the HTTP request on the server
Handling the HTTP response on the client
Making the database call (sounds like you already have some kind of locking / blocking going on there)
You are going to have to get markers to tell you how long it took to go from A to B to C to D back to C back to B back to A kind of thing. We would be speculating heavily from here on what is exactly going on in your program, but we can give you the ideas / tools to figure it out.
If you use YourKit, connect it to your server process. Have nothing else connecting to your server (for instance your client is not sending requests). Try it with your client requesting, you should see your accepting threads receive the HTTP request and then delegate to either your processing thread or do the processing itself. You can use YourKit to see how much time is spent in different functions during that call time.
Try it with your client making the call.
Try it using a simple HTTP request tool like wget or maybe your IDE has a webservice test tool (for instance intellij does), or you can download a simple HTTP test tool.
By testing it in a simple tool that just outputs the response, you can eliminate any client processing issues. You can also achieve a similar test in Chrome or Firefox and use the developer tools to see time to fulfill request.
In my experience, the framework for handling the requests and delegating can introduce some performance issues. I ripped Grails out of a production environment because of its performance issues (before any Grails / Groovy flames come my way, we were operating at a much higher rate than typical web applications, and I am sure Grails has made some headway in the last couple years... alas, it was not for my need at that time)
BTW, I doubt you are operating a load where you will be critiquing the web service framework you chose to use. I have been happy with Spring MVC and DropWizard (Jersey JAX-RS), and Grails is easy to use too.
You should make a simple static content response in your webservice and see how quickly that returns vs a request that makes a database call.
Also, what kind of table are you using in MySQL? InnoDB? MyISAM? They have different locking schemes. That could be causing your MySQL issue.
The key to all of it, break the problem up into parts, and measure each and eliminate parts one by one till you go, everytime I do X it is slower (like everytime I make a database call its slower)
In Java the the way you will be able to find more support online via documentation/forums is to develop the web service as a REST web service using Spring MVC.
You can base yourself on this resource and take it from there:
Spring MVC REST Hello World Web Service
Using Spring you can create a RestFul webservice easily and spring does all the ground work you needed. As others had mentioned you can consume the webservice in any type of client - including Android.
A detailed guide available here:
https://spring.io/guides/gs/rest-service/
Here are my suggestions:
Make APIs only read or write database. If an API combines reading and writing, it is possible to cause deadlock;
Use a light-weight HTTP server. Powerful HTTP server is possibly consuming more.
Make use of thread. Have more threads could be helpful when you are facing a ton of users.
Make more things static. You could avoid unnecessary queries.
I think mhoglan's answer is detailed enough.
I currently have a Java application that updates a neo4j database every day.
I then have another application that queries the database using traversals by creating an embedded database with the same storage path.
How should I go about keeping the server running and directing the queries at the already running instance every time the querying java application runs? I'm unsure how to do this without creating an embedded server instance every time.
I can keep my current approach, the problem is it has to load the database every single time a user makes a request for a query and this is expensive.
Thanks!
You can run server on top of an embedded database: http://docs.neo4j.org/chunked/milestone/server-embedded.html
So you can keep your embedded app running and import the data using a timer-task and at the same time offer the server's web-ui.
Not only is it expensive, but if I understood your application concept correctly, you have a potential lock store error.
If your updating application is doing something in the database, and thus has an instance of the embedded database running, and at the same time your other application is trying to make an instance of the embedded database to perform a query, you'd run into a lock store.
I don't know if you have taken any precautions to prevent this, or if you've just been lucky so far that these actions have not occured simultaneously, but I would look into it.
I am working on a Java web app with unit tests that deploy the app in Jetty. I use HtmlUnit to hit the app and do some high level tests. I set it up so that I can use a singleton probe to modify my system configuration and add a "test" flag--This is handy because I want to be able to run some tests without having to authenticate an actual user or check user roles.
However, it seems like it could open the door for vulnerability when the app is deployed. I'm looking for suggestions about how to make this "back door" a little more bullet proof. I could use a mock object to handle this, but I think that still leaves the back door exposed.
I have user accounts specifically for testing in all of my environments. I create them using the real registration process, nothing hand-made.
This bypasses your issue, allows me to test the signin process, and if needed I create multiple users with different traits/roles which I can test against.
Because the users are under my control, they remain consistent and match the expected test results.
Use special parameter that is long enough to assume. For example GUID. It could be even hard-coded in your application. All tests will append this parameter to each URL they are using. You can check this parameter using special HttpFilter and turn the test mode on.
Throw some kind of security around the process you use to change the app over to Test mode - Basic Authentication for that page, or something. This can all be configured directly in web.xml.