After some time of running I'm getting this error when I stress test my servlet with at least 20 browser tabs simultaneously accessing the servlet:
java.sql.SQLException: [tomcat-http--10] Timeout: Pool empty. Unable to fetch a connection in 10 seconds, none available[size:200; busy:200; idle:0; lastwait:10000].
Here is the XML config for this:
<Resource name="jdbc/MyAppHrd"
auth="Container"
type="javax.sql.DataSource"
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
testWhileIdle="true"
testOnBorrow="true"
testOnReturn="false"
validationQuery="SELECT 1"
validationInterval="30000"
timeBetweenEvictionRunsMillis="30000"
maxActive="200"
minIdle="10"
maxWait="10000"
initialSize="200"
removeAbandonedTimeout="120"
removeAbandoned="true"
logAbandoned="false"
minEvictableIdleTimeMillis="30000"
jmxEnabled="true"
jdbcInterceptors="org.apache.tomcat.jdbc.pool.interceptor.ConnectionState;
org.apache.tomcat.jdbc.pool.interceptor.StatementFinalizer"
username="sa"
password="password"
driverClassName="net.sourceforge.jtds.jdbc.Driver"
url="jdbc:jtds:sqlserver://192.168.114.130/MyApp"/>
What could be the problem?
Update:
Java Code:
public class MyServlet extends HttpServlet {
private static final long serialVersionUID = 1L;
private static final Log LOGGER = LogFactory.getLog(MyServlet.class);
private void doRequest(HttpServletRequest request, HttpServletResponse response) throws IOException, ServletException {
CallableStatement stmt = null;
ResultSet rs = null;
Connection conn = null;
try {
conn = getConnection();
stmt = conn.prepareCall("{call sp_SomeSPP(?)}");
stmt.setLong(1, getId());
rs = stmt.executeQuery();
// set mime type
while (rs.next()) {
if (rs.getInt(1)==someValue()) {
doStuff();
break;
}
}
stmt = conn.prepareCall("{call sp_SomeSP(?)}");
stmt.setLong(1, getId());
rs = stmt.executeQuery();
if (rs.next()) {
// do stuff
}
RequestDispatcher rd = getServletContext().getRequestDispatcher("/SomeJSP.jsp");
rd.forward(request, response);
return;
} catch (NamingException e) {
LOGGER.error("Database connection lookup failed", e);
} catch (SQLException e) {
LOGGER.error("Query failed", e);
} catch (IllegalStateException e) {
LOGGER.error("View failed", e);
} finally {
try {
if (rs!=null && !rs.isClosed()) {
rs.close();
}
} catch (NullPointerException e) {
LOGGER.error("Result set closing failed", e);
} catch (SQLException e) {
LOGGER.error("Result set closing failed", e);
}
try {
if (stmt!=null) stmt.close();
} catch (NullPointerException e) {
LOGGER.error("Statement closing failed", e);
} catch (SQLException e) {
LOGGER.error("Statement closing failed", e);
}
try {
if (conn != null){
conn.close();
conn = null;
}
} catch (NullPointerException e) {
LOGGER.error("Database connection closing failed", e);
} catch (SQLException e) {
LOGGER.error("Database connection closing failed", e);
}
}
}
#Override
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
doRequest(request, response);
}
#Override
protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
doRequest(request, response);
}
protected static Connection getConnection() throws NamingException, SQLException {
InitialContext cxt = new InitialContext();
String jndiName = "java:/comp/env/jdbc/MyDBHrd";
ConnectionPoolDataSource dataSource = (ConnectionPoolDataSource) cxt.lookup(jndiName);
PooledConnection pooledConnection = dataSource.getPooledConnection();
Connection conn = pooledConnection.getConnection();
return conn; // Obtain connection from pool
}
I suggest you change your getConnection method to the following you might actually be removing the Pooling support by going directly to via the javax.sql.PooledConnection interface
InitialContext cxt = new InitialContext();
String jndiName = "java:/comp/env/jdbc/MyDBHrd";
DataSource dataSource = (DataSource) cxt.lookup(jndiName);
return dataSource.getConnection();
Also use something like DBUtils#closeQuietly to clean up your connections
Update: You are removing the Pooling support from the Connection. If you run the following and look at the output you will see the connection retrieved directly from the DataSource is a ProxyConnection wrapping a PooledConnection.
public static void main(String[] args) throws Exception {
Properties properties = new Properties();
properties.put("username", "sa");
properties.put("password", "password");
properties.put("driverClassName", "net.sourceforge.jtds.jdbc.Driver");
properties.put("url", "jdbc:jtds:sqlserver://192.168.114.130/MyApp");
DataSourceFactory dsFactory = new DataSourceFactory();
DataSource ds = dsFactory.createDataSource(properties);
ConnectionPoolDataSource cpds = (ConnectionPoolDataSource) ds;
PooledConnection pooledConnection = cpds.getPooledConnection();
System.out.println("Pooled Connection - [" + ds.getConnection() + "]"); // Close will return to the Pool
System.out.println("Internal Connection - [" + pooledConnection.getConnection() + "]"); // Close will just close the connection and not return to pool
}
Probably, you are holding connection for too long.
Make sure that you do not open DB connection when you start processing request and then release it when you finally committed the response.
Typical mistake is:
#Override
protected void doGet (
final HttpServletRequest request,
final HttpServletResponse response
) throws
ServletException,
IOException
{
Connection conn = myGetConnection( );
try
{
...
// some request handling
}
finally
{
conn.close( )
}
}
In this code, database connection lifetime is totally at the mercy of the client connected to your server.
Better pattern would be
#Override
protected void doGet (
final HttpServletRequest request,
final HttpServletResponse response
) throws
ServletException,
IOException
{
// some request preprocessing
MyProcessedRequest parsedInputFromRequest =
getInputFromRequest( request );
final MyModel model;
{
// Model generation
Connection conn = myGetConnection( );
try
{
model = new MyModel( conn, parsedInputFromRequest );
}
finally
{
conn.close( );
}
}
generateResponse( response, model );
}
Note, that if the bottleneck is in model generation, you still going to run out of connections, but this is now a problem for DBA, that relates to better data management/indexing on the database side.
Check your jdbc connections are closed after completion of process. It may caused by unclosed connections.
Close your connection before the following.
RequestDispatcher rd = getServletContext().getRequestDispatcher("/SomeJSP.jsp");
rd.forward(request, response);
return;
Also remove return if it is not required.
The code you currently provide looks long/complex, but fine.
However, I guess your "doStuff" method might be a candidate for leaking more connections
First, you are not closing your Statement and ResultSet objects within the body of your method.
They should be cleaned-up when you call close on the the Connection (according to the JDBC spec), but in a pooled setting, they might not actually get cleaned up.
Second, you are unwrapping the pooled connection and returning the underlying connection which will break everything.
So, modify your code to be like this:
try {
conn = getConnection();
stmt = conn.prepareCall("{call sp_SomeSPP(?)}");
stmt.setLong(1, getId());
rs = stmt.executeQuery();
// set mime type
while (rs.next()) {
if (rs.getInt(1)==someValue()) {
doStuff();
break;
}
}
// ADD THESE LINES
rs.close(); rs = null;
stmt.close(); stmt = null;
stmt = conn.prepareCall("{call sp_SomeSP(?)}");
stmt.setLong(1, getId());
rs = stmt.executeQuery();
if (rs.next()) {
// do stuff
}
}
....
protected static Connection getConnection() throws NamingException, SQLException {
InitialContext cxt = new InitialContext();
String jndiName = "java:/comp/env/jdbc/MyDBHrd";
DataSource dataSource = (DataSource) cxt.lookup(jndiName);
return dataSource.getPooledConnection();
}
And, as others have said, you definitely want to clean-up your resources before you do things like forwarding to another page. Otherwise, you keep the connection far longer than necessary. It's a critical resource: treat it like one.
Related
I was working on a java project and it was working just fine. I was able to make connections. I closed all the connections properly in finally block. Now I am not able to make connections or even open psql in my terminal. How can I make it work as before. Much much appreciated
import java.sql.Connection;
import com.mchange.v2.c3p0.*;
public class MyConnection {
public static Connection getConnection(){
ComboPooledDataSource cpds1 = new ComboPooledDataSource();
String dbDriver = "org.postgresql.Driver";
String dbName = "jdbc:postgresql://localhost/postgres";
cpds1.setJdbcUrl(dbName);
String userName = "user_1";
cpds1.setUser(userName);
String password = "mypass";
cpds1.setPassword(password);
cpds1.setMaxStatements( 180 );
try
{
cpds1.setDriverClass(dbDriver);
return cpds1.getConnection();
}
catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
return null;
}
}
}
This is where I'm calling it
public void doPost(HttpServletRequest request, HttpServletResponse response) throws IOException
{
PrintWriter writer = response.getWriter();
JSONObject jo = new JSONObject();
JSONObject jObj;
Statement stmt = null;
Connection con = null;
PreparedStatement ps;
ResultSet rs = null;
try
{
jObj = UtilityClass.getJSON(request);
String uname = ((String) jObj.get("uname"));
String pass = ((String) jObj.get("pass"));
String sql = "SELECT * FROM users WHERE username = ?";
try
{
con = MyConnection.getConnection();
System.out.println("Got Connection");
stmt = con.createStatement();
ps = con.prepareStatement(sql);
ps.setString(1, uname);
rs = ps.executeQuery();
if(rs.next())
{
if(BCrypt.checkpw(pass,rs.getString("password")))
{
HttpSession session = request.getSession();
session.setAttribute("uname", uname);
if(session.isNew())
{
System.out.println("new");
}
if(uname.equals("admin"))
{
session.setAttribute("role", "admin");
jo.put("status", "admin");
}
else
{
session.setAttribute("role", "user");
jo.put("status", "authenticate");
}
}
}
writer.print(jo);
}
catch (Exception e)
{
e.printStackTrace();
System.out.println("Not Connected");
}
finally
{
if(rs != null)
{
rs.close();
}
if(stmt != null)
{
stmt.close();
}
if(con != null)
{
con.close();
}
}
}
catch(Exception e)
{
System.out.print("JSON Exception");
}
}
Usually, DB Admins are using pooling technologies on Databases. For PostgreSQL one of the more popularly is a PGBOUNCER. We used PGBOUNCER in our large project, the result is excellent. I recommend it to you. To get more information about the pooling system you can read this link. For About Pooling
I am developing a dynamic web project on eclipse.
Below is an example of connecting MySQL using DataSource.
Is it the correct way? I mean is it normal to get connection in a Servlet?
Moreover, I find this way to get/close connection is tedious because I need to write exact same part of codes every time when I want to get/close connection. I think there should be a better way. Could someone give me some suggestions?
Thank you!
#WebServlet(name="HelloUser", urlPatterns={"/hellouser"})
public class HelloUserServlet extends HttpServlet{
#Override
protected void doGet(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
DataSource ds = MyDataSourceFactory.getMySQLDataSource();
Connection con = null;
Statement stmt = null;
ResultSet rs = null;
try {
con = ds.getConnection();
stmt = con.createStatement();
rs = stmt.executeQuery(...);
...
} catch (SQLException e) {
e.printStackTrace();
}finally{
try {
if(rs != null) rs.close();
if(stmt != null) stmt.close();
if(con != null) con.close();
} catch (SQLException e) {
e.printStackTrace();
}
}
}
}
}
Starting from Java 7 you can use try-with-resource( JDBC api is updated to implement Autocloseable) .
The try-with-resources statement is a try statement that declares one
or more resources. A resource is an object that must be closed after
the program is finished with it
E.g.
try (Connection con = ds.getConnection();
Statement stmt = con.createStatement();
ResultSet rs = stmt.executeQuery(...)) {...}
I wrote a servlet and run it on tomcat6. When there is a query sent to the specified DNS, the servlet will get information and use them to search in the database(mysql), then write some thing back. I use connection pool in the tomcat6. However, when I'm trying to do a load test on my program, I found the throughput is extremely low and many exceptions like
Response code: Non HTTP response code:
java.net.SocketTimeoutException Non HTTP response message: Read timed
out
Your result:
Team and Account info\n at java.net.SocketInputStream.socketRead0(Native Method); at
java.net.SocketInputStream.read(SocketInputStream.java:152); at
java.net.SocketInputStream.read(SocketInputStream.java:122); at
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166); at
org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90); at
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281); at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92); at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61); at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254); at
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289); at
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252); at
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191); at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300); at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127); at
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715); at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520); at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906); at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805); at
org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.executeRequest(HTTPHC4Impl.java:475); at
org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:295); at
org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:74); at
org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1105); at
org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1094); at
org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:429); at
org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:257); at
java.lang.Thread.run(Thread.java:745)
Can somebody give me some suggestions? Here is my code of servlet and this is the only Java code I use in this web project, thanks
#SuppressWarnings("serial")
public class Servlet extends HttpServlet {
#Override
public void doGet(final HttpServletRequest req, final HttpServletResponse res) throws IOException {
Connection conn = null;
DataSource ds = null;
InitialContext ctx;
try {
ctx = new InitialContext();
ds = (DataSource) ctx.lookup("java:comp/env/jdbc/mysql");
conn = ds.getConnection();
final Statement stmt = conn.createStatement();
stmt.executeQuery("SET NAMES 'utf8mb4'; ");
} catch (final SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
res.setContentType("text/plain;charset=UTF-8");
final PrintWriter pw = res.getWriter();
try {
final Statement stmt = conn.createStatement();
String teamInfo = "aa\n";
ResultSet rs = null;
final String info = req.getQueryString();
final String[] sp = info.split("&");
final String[] child0 = sp[0].split("=");
final String[] child1 = sp[1].split("=");
final String sqlString = "select * from tweets where timeandid=\"" + child1[1] + child0[1] + "\"";
rs = stmt.executeQuery(sqlString);
rs.next();
final String a = rs.getString(2);
teamInfo = teamInfo + rs.getString(3) + ":" + rs.getString(4).trim() + ":" + a + "\n";
pw.print(teamInfo);
pw.flush();
rs.close();
stmt.close();
} catch (final SQLException ex) {
System.out.println(ex);
} finally {
System.out.println("over");
}
pw.close();
try {
conn.close();
} catch (final SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
I am trying to figure out the best structure for connection pools so that I can access the connection pool from any servlet and establish a connection to my database. I have been following some tutorials in setting up and configuring the datasource and connection pool and they all have them initialized and accessed in classes that extend HttpServlet. So it looks something like this:
public class DatabaseConnector extends HttpServlet {
private static final long serialVersionUID = 1L;
private DataSource dataSource;
private Connection connection;
private Statement statement;
public void init() throws ServletException {
try {
// Get DataSource
Context initContext = new InitialContext();
Context envContext = (Context)initContext.lookup("java:/comp/env");
dataSource = (DataSource)envContext.lookup("jdbc/test");
} catch (NamingException e) {
e.printStackTrace();
}
}
/**
* #see HttpServlet#HttpServlet()
*/
public DatabaseConnector() {
super();
}
/**
* #see HttpServlet#doGet(HttpServletRequest request, HttpServletResponse response)
*/
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
ResultSet resultSet = null;
try {
// Get Connection and Statement
connection = dataSource.getConnection();
statement = connection.createStatement();
String query = "SELECT * FROM STUDENT";
resultSet = statement.executeQuery(query);
while (resultSet.next()) {
System.out.println(resultSet.getString(1) + resultSet.getString(2) + resultSet.getString(3));
}
} catch (SQLException e) {
e.printStackTrace();
}finally {
try {
if (resultSet != null) {resultSet.close();}
if (statement != null) {statement.close();}
if (connection != null) {connection.close();}
} catch (SQLException e) {
e.printStackTrace();
}
}
}
/**
* #see HttpServlet#doPost(HttpServletRequest request, HttpServletResponse response)
*/
protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
// TODO Auto-generated method stub
}
}
It looks like to me that this connection pool is only for this servlet which would only distribute connections when a get request is sent the this servlets URL. What if I want to have another servlet that needs to access the database. I was thinking of just removing the doGet and doPost methods in this servlet and leave the init so that the connection pool would be initialized at runtime, and then have a singleton reference to this servlet which could be used in other servlets. However, this doesn't seem like the right way to me. What is the best way to structure a connection pool that can be accessed from all servlets and listeners?
Thanks for your help!
Completely wrong.
The correct way to access a connection is to use a JNDI connection pool.
Servlets are HTTP listeners. They shouldn't have anything to do with databases.
A properly layered Java EE solution will restrict data sources to the service tier. It will check connections in and out, know about units of work and transactions, and interact with data access objects.
Servlets should deal with services, not data sources.
The logic to create a connection could be placed within a simple class.
public class ConnectionManager{
public static Connection getConnection(){
Connection connection = null;
try {
// Get DataSource
Context initContext = new InitialContext();
Context envContext = (Context)initContext.lookup("java:/comp/env");
dataSource = (DataSource)envContext.lookup("jdbc/test");
connection = dataSource.getConnection();
} catch (NamingException e) {
e.printStackTrace();
}
if(connection == null){
throw new RuntimeException("Cannot connect");
}
return connection;
}
}
I'm trying to create a AJAX based SQL query with Java EE and Servlets. I'm using Glassfish 3.01 and MS SQL server with Jquery on the client side.
I put everything together, and bind my ajax function to the textfield's onkeyup event. But sometimes When I put 'teststring' into the textbox only "teststrin" passed to the Servlet. So basically the last char disappears and therefore the query result is not correct.
Not to mention when the resultset contains large amount of data the query is pretty slow. Could you please check if I'm doing something wrong on the server and client side?
On the client side I have this JQuery function:
function ajaxSearch(sstring) {
if (sstring.length < 3)
{
$("#external").html("at least 3 chars please....")
}
else
{
$('#loading').ajaxStart(function() {
$(this).show()
$("#external").hide()
});
$('#loading').ajaxComplete(function() {
$(this).hide()
$("#external").show()
});
$.ajax({
type:"GET",
url: "/myApp/getStd",
dataType: "application/x-www-form-urlencoded",
data: "sstring="+escape(sstring),
async: true,
success: function(data){
$("#external").html(data);
}
})
}
}
On the server side I have this:
#WebServlet(name="getStd", urlPatterns={"/getStd"})
public class getStd extends HttpServlet {
#Override
public void doGet (HttpServletRequest req, HttpServletResponse res) throws ServletException, IOException {
Connection conn = null;
Statement stmt = null;
ResultSet rs = null;
ArrayList rows = new ArrayList();
res.setCharacterEncoding("UTF-8");
res.setContentType("text/html");
PrintWriter out = res.getWriter();
String sql=null;
String test= req.getParameter("sstring");
try{
InitialContext cxt = new InitialContext();
if (cxt == null) {
throw new Exception("Uh oh -- no context!");}
DataSource ds = (DataSource) cxt.lookup( "jdbc/Sample" );
conn = ds.getConnection();
stmt = conn.createStatement();
sql="Select * from MYDB.dbo.testdb where myField like '%"+req.getParameter("sstring")+"%';";
rs = stmt.executeQuery(sql);
while(rs.next()){
stdRecord cols = new stdRecord();
cols.setTeljel(rs.getString("Field1"));
cols.setTitle(rs.getString("Field2"));
cols.setICS(rs.getString("Field3"));
cols.setREF(rs.getString("Field4"));
rows.add(cols);
}
req.setAttribute("std", rows);
req.setAttribute("query",test );
req.getRequestDispatcher("/showRes.jsp").forward(req, res);
// close everything to free up resources
rs.close();
rs = null;
stmt.close();
stmt = null;
conn.close(); /
conn = null;
rows=null;
} catch (SQLException e) {
e.printStackTrace(out);
} catch (Exception e) {
e.printStackTrace(out);
} finally {
if (rs != null) {
try { rs.close(); } catch (SQLException e) { ; }
rs = null;
}
if (stmt != null) {
try { stmt.close(); } catch (SQLException e) { ; }
stmt = null;
}
if (conn != null) {
try { conn.close(); } catch (SQLException e) { ; }
conn = null;
}
}
}
}
Thanks in advance.
As to the lag in keyup, I think this is related to the performance issue, so let's fix that first and then review afterwards.
As to the performance, you've given very little information about your setup, but two common solutions which are often overlooked by starters are the following:
Use a connection pooled DataSource instead of DriverManager. This saves the cost of connecting the DB on every query (which can take over 200ms while a pooled connection is returned in no-time). Consult the JNDI resource config documentation of the application server in question for details (hint: admin console).
Limit the resultset size in SQL side instead of in Java side. This saves the cost of transferring irrelevant data over network. Just return the top 10 results or something instead of the entire table. Consult the SQL manual of the database in question for details (hint: SET ROWCOUNT).