I am using Redis(3.2.100) for Windows to cache my database data in Java.This is my redis init code:
private static Dictionary<Integer, JedisPool> pools = new Hashtable();
static {
JedisPoolConfig config = new JedisPoolConfig();
config.setMaxIdle(2);
config.setMaxTotal(10);
config.setTestOnBorrow(true);
config.setMaxWaitMillis(2000);
for (int i = 0; i < 16; i++) {
JedisPool item = new JedisPool(config, "127.0.0.1", 6379,10*1000);
pools.put(i, item);
}
}
This is the cache code:
public static String get(String key, Integer db) {
JedisPool poolItem = pools.get(db);
Jedis jredis = poolItem.getResource();
String result = jredis.get(key);
return result;
}
The problem is when the program run for a while,the getResource method throws:
redis.clients.jedis.exceptions.JedisException: Could not get a resource from the pool
So how to reuse the connection or close the connection.I am using this command to find out that the client has reached the max value.
D:\Program Files\Redis>redis-cli.exe info clients
# Clients
connected_clients:11
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
How to fix it?
Remember to close the redis connection,modify this function like this:
public static String get(String key, Integer db) {
JedisPool poolItem = pools.get(db);
Jedis jredis = null;
String result = null;
try {
jredis = poolItem.getResource();
result = jredis.get(key);
} catch (Exception e) {
log.error("get value error", e);
} finally {
if (jredis != null) {
jredis.close();
}
}
return result;
}
Related
I'm using Jedis to perform a lot of insertions/reads in Redis.
The Redis server is using the default configuration.
The problem appears when I start using a few threads and the exception is:
redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketException: Pipe quebrado (Write failed)
I've searched a lot about this problem but couldn't find the reason of it or it's solve. The code I'm using to perform these tests is below:
public class RedisFacade {
private static RedisFacade instancia = null;
// Initialize the Connection
final JedisPoolConfig poolConfig = buildPoolConfig();
JedisPool pool = new JedisPool(poolConfig, "localhost");
Jedis jedis;
int i = 0;
private RedisFacade() {
}
public static RedisFacade getInstancia() {
if (instancia == null) {
instancia = new RedisFacade();
}
return instancia;
}
// retorna um cliente jedis da pool
public Jedis getDB() {
if (jedis == null) {
jedis = pool.getResource();
}
return jedis;
}
//inserting
public void insert(Document d) {
String key = i + d.getString("date") + d.getString("time");
String value = d.toString();
this.getDB().set(key, value);
i++;
}
//reading
public void read(String date, String time) {
Object doc = this.getDB().get(i + date + time);
i++;
System.out.println(doc);
}
public void destroyPool() {
this.pool.destroy();
}
private JedisPoolConfig buildPoolConfig() {
final JedisPoolConfig poolConfig = new JedisPoolConfig();
poolConfig.setMaxTotal(1100);
poolConfig.setMaxIdle(16);
poolConfig.setMinIdle(16);
poolConfig.setTestOnBorrow(true);
poolConfig.setTestOnReturn(true);
poolConfig.setTestWhileIdle(true);poolConfig.setMinEvictableIdleTimeMillis(Duration.ofSeconds(60).toMillis());
poolConfig.setTimeBetweenEvictionRunsMillis(Duration.ofSeconds(30).toMillis());
poolConfig.setNumTestsPerEvictionRun(3);
poolConfig.setBlockWhenExhausted(true);
return poolConfig;
}}
Seems like it's a timeout issue.
See this thread: Configure Jedis timeout
And also this discussion: https://github.com/xetorthio/jedis/issues/185
So I would try instantiating JedisPool with a timeout parameter
(i.e. https://github.com/xetorthio/jedis/blob/master/src/main/java/redis/clients/jedis/JedisPool.java#L201, but there are many other constructors)
and setting CONFIG SET timeout 600 in redis (with a 10 minute timeout for instance).
Edit
The JedisPool timeout is in milliseconds it seems.
After trying to implement new constructors, new configurations for the pool and clients, I tried a simple way to fix the problem: close the resources I was getting from the pool. To do so, I've changed the following code:
public Jedis getDB() {
jedis = pool.getResource();
return jedis;
}
//cria um _id pra ser usado novamente quando for buscar os documentos
public void insert(Document d) {
String key = "key" + i;
String value = d.toString();
Jedis jedis = this.getDB();
jedis.set(key, value);
jedis.close();
i++;
}
//busca pelo _id
public void read() {
Jedis jedis = this.getDB();
Object doc = jedis.get("key" + i);
jedis.close();
i++;
System.out.println(doc);
}
After changing the code, the service started to work I was planning, so I'll accept this as a solution.
I have written a DAO class which allows several threads invoked by ExecutorServices to write to MySQL DB.
EDIT: I am using c3p0 to create a JDBC ConnectionPool. So every new thread will get a new JDBC Connection by calling
DataBaseManager.getInstance().getConnection()
There seems to be random concurrency issue while executing, e.g:
java.sql.SQLException: No value specified for parameter 1
at com.eanurag.dao.DataBaseManager.writeData(DataBaseManager.java:102)
I am not able to understand all the issues with the code. Should I just synchronize entire writeData() ?
public class DataBaseManager {
private final static Logger logger = Logger.getLogger(DataBaseManager.class);
private static volatile DataBaseManager dbInstance = null;
private DataBaseManager() {
cpds = new ComboPooledDataSource();
try {
cpds.setDriverClass("com.mysql.jdbc.Driver");
} catch (PropertyVetoException e) {
logger.error("Error in Initializing DB Driver class", e);
}
cpds.setJdbcUrl("jdbc:mysql://" + DB_HOST + "/" + DB_NAME);
cpds.setUser(DB_USER);
cpds.setPassword(DB_PASS);
cpds.setMinPoolSize(MINIMUM_POOL_SIZE);
cpds.setAcquireIncrement(INCREMENT_SIZE);
cpds.setMaxPoolSize(MAXIMUM_POOL_SIZE);
cpds.setMaxStatements(MAX_STATEMENTS);
}
public static DataBaseManager getInstance() {
if (dbInstance == null) {
synchronized (WorkerManager.class) {
if (dbInstance == null) {
dbInstance = new DataBaseManager();
}
}
}
return dbInstance;
}
private ComboPooledDataSource cpds;
private static final Integer MINIMUM_POOL_SIZE = 10;
private static final Integer MAXIMUM_POOL_SIZE = 1000;
private static final Integer INCREMENT_SIZE = 5;
private static final Integer MAX_STATEMENTS = 200;
private volatile Connection connection = null;
private volatile Statement statement = null;
private volatile PreparedStatement preparedStatement = null;
private static final String DB_HOST = "localhost";
private static final String DB_PORT = "3306";
private static final String DB_USER = "root";
private static final String DB_PASS = "";
private static final String DB_NAME = "crawly";
private static final String URL_TABLE = "url";
public Connection getConnection() throws SQLException {
logger.info("Creating connection to DB!");
return this.cpds.getConnection();
}
public Boolean writeData(URL url) {
StringBuffer writeDBStatement = new StringBuffer();
writeDBStatement.append("insert into");
writeDBStatement.append(" ");
writeDBStatement.append(DB_NAME);
writeDBStatement.append(".");
writeDBStatement.append(URL_TABLE);
writeDBStatement.append(" ");
writeDBStatement.append("values (?,?,default)");
Boolean dbWriteResult = false;
try {
connection = DataBaseManager.getInstance().getConnection();
preparedStatement = connection.prepareStatement(writeDBStatement.toString());
preparedStatement.setString(1, url.getURL());
preparedStatement.setString(2, String.valueOf(url.hashCode()));
dbWriteResult = (preparedStatement.executeUpdate() == 1) ? true : false;
if(dbWriteResult){
logger.info("Successfully written to DB!");
}
} catch (SQLException e) {
logger.error("Error in writing to DB", e);
} finally {
try {
preparedStatement.close();
connection.close();
} catch (SQLException e) {
e.printStackTrace();
}
}
return dbWriteResult;
}
}
The connection and preparedStatement variables must be local, not instance members.
No synchronization required.
What is happening here?
public Connection getConnection() throws SQLException {
logger.info("Creating connection to DB!");
return this.cpds.getConnection();
}
Namely, what does cpds.getConnection() do? When you call:
connection = DataBaseManager.getInstance().getConnection();
Your connection object is a member of what's supposed to be a singleton class here but every call to writeData() overwrites it it with a new getConnection() call. Is the getConnection() call thread unsafe as well?
Also, why is the connection object declared as a class member and then overwritten each time writeData() is called? In a multi-threaded environment, the code as it exists allows for the connection object to be overwritten by another getConnection() call immediately before prepareStatement() is called, since access to writeData() is not locked. Same for preparedStatement. Move those into the writeData() method.
I have been trying to invoke an operation on ThreadMXBean in a remote jvm. The code snippet I used invoke the operation is below
bean = new ObjectName("java.lang:type=Threading");
memoryInfo = RemoteConnector.getRemote().getMBeanInfo(bean);
RemoteConnector.getRemote().getObjectInstance(bean);
MBeanOperationInfo [] mBeanAttributeInfos = memoryInfo.getOperations();
for(MBeanOperationInfo mBeanAttributeInfo : mBeanAttributeInfos){
System.out.println(mBeanAttributeInfo.getName());
}
long [] allThreadIds = (long [])RemoteConnector.getRemote().getAttribute(bean,"AllThreadIds");
Object [] params = new Object[2];
int maxDepth = 100;
params[0] = allThreadIds;
params[1] = maxDepth;
String [] opSigs = {allThreadIds.getClass().getName(),"I"};
RemoteConnector.getRemote().invoke(bean,"getThreadInfo",params,opSigs);
Note that getRemote() method returns a mbeanserverconnection
I can't invoke the method getThreadInfo() on stub. I am getting this message
2016-05-05 00:17:37 ERROR ThreadDumpCreator:67 - Operation getThreadInfo exists but not with this signature: ([J, I)
Please help me resolve this issue :)
Below is the method i used to connect remote mbeanserver
public class RemoteConnector {
private static MBeanServerConnection remote = null;
private static JMXConnector connector = null;
public static void defaultConnector(){
try {
JMXServiceURL target = new JMXServiceURL
("service:jmx:rmi://localhost:11111/jndi/rmi://localhost:9999/jmxrmi");
//for passing credentials for password
Map<String, String[]> env = new HashMap<String, String[]>();
String[] credentials = {"admin", "admin"};
env.put(JMXConnector.CREDENTIALS, credentials);
connector = JMXConnectorFactory.connect(target, env);
remote = connector.getMBeanServerConnection();
}catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
public static MBeanServerConnection getRemote() {
return remote;
}
public static void setRemote(MBeanServerConnection remote) {
RemoteConnector.remote = remote;
}
public static void closeConnection() throws IOException {
if(connector != null){
connector.close();
}
}
}
Apparently I should have used int.class.getName() as the method signature requires from invoke method
In my weblogic server ehcache is deployed , I need to get ehcahe mbeans from this program through java programming, Through JMX i am not able to connect.how can i get those custom mbeans??
i tried to get mbeans through weblogic t3 protocol
public class Test
{
private String hostName = "";
private String port = "";
private String userName = "";
private String password = "";
private String connectorURL = "service:jmx:rmi:///jndi/rmi://{0}:{1}/jmxrmi";
private JMXConnector jmxc = null;
public static void main(String []args) throws Exception
{
Test t = new Test();
t.hostName = args[0];
System.out.println(args[1]);
t.port = args[1];
t.userName = args[2];
t.password = args[3];
t.jmxc = t.initConnection();
MBeanServerConnection mbsc = t.jmxc.getMBeanServerConnection();
System.out.println(mbsc);
Set<ObjectInstance> st =mbsc.queryMBeans(new ObjectName("net.*:*"), null);
System.out.println(st.toString());
Iterator<ObjectInstance> it = st.iterator();
while(it.hasNext())
{
System.out.println(it.next());
}
t.closeConnection();
}
private JMXConnector initConnection()
{
System.out.println("initiate connection");
JMXServiceURL serviceURL = null;
try
{
String jndiroot = "/jndi/";
String mserver = "weblogic.management.mbeanservers.domainruntime";
int port1 = Integer.parseInt(port);
serviceURL = new JMXServiceURL("t3", hostName, port1, jndiroot + mserver);
Hashtable h = new Hashtable();
h.put(Context.SECURITY_PRINCIPAL, userName);
h.put(Context.SECURITY_CREDENTIALS, password);
h.put(JMXConnectorFactory.PROTOCOL_PROVIDER_PACKAGES, "weblogic.management.remote");
long lngJmxClientWTO = 10000;
h.put("jmx.remote.x.request.waiting.timeout", lngJmxClientWTO );
return JMXConnectorFactory.connect(serviceURL, h);
}
catch (Exception e)
{
e.printStackTrace();
return null;
}
}
/**
* This method closes client connection with server
* #throws IOException
*/
public void closeConnection()
{
if(jmxc != null)
{
try
{
jmxc.close();
}
catch (IOException e) {
jmxc = null;
}
}
}
import net.sf.ehcache.CacheManager;
import net.sf.ehcache.Ehcache;
CacheManager manager = CacheManager.newInstance();
Ehcache cache = manager.getEhcache("Some cache name here..."); //<-- PLEASE EDIT THE CACHE NAME...
I don't know if this is what you're asking for....
Once you've obtained your cache, you can use it, pretty much like a java Map.
You can follow the Ehcache documentation to see how to programmatically get the remote cache. Essentially, you will need to create a configuration (or configuration file) which the CacheManager can access.
Today I tried to fix some potential memory leaks in my web application.
I use the following libraries.
spring-webmvc-3.2.9.RELEASE
spring-data-mongodb-1.5.0.RELEASE
mongo-java-driver-2.12.1
First I missed to close the MongoClient but changed my configuration this way.
#Configuration
public class MongoDBConfiguration implements DisposableBean {
private MongoClient mongoClient;
#Bean
public MongoTemplate mongoTemplate() {
try {
final Properties props = loadProperties();
log.debug("Initializing Mongo DB client");
mongoClient =
new MongoClient(getProperty(props, "host", "localhost"), cint(getProperty(props, "port",
"27017")));
UserCredentials credentials = null;
final String auth = getProperty(props, "auth", null);
if (auth != null && auth.equalsIgnoreCase("true")) {
final String user = getProperty(props, "user", null);
final String pass = getProperty(props, "pass", null);
if (user != null && pass != null) {
credentials = new UserCredentials(user, pass);
}
}
final MongoDbFactory mongoDbFactory =
new SimpleMongoDbFactory(mongoClient, getProperty(props, "dbname", "Feeder"), credentials);
final MappingMongoConverter mongoConverter =
new MappingMongoConverter(new DefaultDbRefResolver(mongoDbFactory),
new MongoMappingContext());
mongoConverter.setCustomConversions(customConversions(mongoConverter));
mongoConverter.afterPropertiesSet();
final MongoTemplate mongoTemplate = new MongoTemplate(mongoDbFactory, mongoConverter);
return mongoTemplate;
} catch (final IOException e) {
log.error("", e);
}
return null;
}
/**
* close Mongo client to avoid memory leaks
*/
#Override
public void destroy() {
log.debug("Shutdown Mongo DB connection");
mongoClient.close();
log.debug("Mongo DB connection shutdown completed");
}
}
When stopping or reloading the web application, there are still messages complaining about probable memory leaks.
2014-06-24 07:58:02,114 DEBUG d.p.f.s.m.MongoDBConfiguration - Shutdown Mongo DB connection
2014-06-24 07:58:02,118 DEBUG d.p.f.s.m.MongoDBConfiguration - Mongo DB connection shutdown completed
Jun 24, 2014 7:58:02 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/feeder##1.5.1] created a ThreadLocal with key of type [com.mongodb.BaseCluster$1] (value [com.mongodb.BaseCluster$1#766465]) and a value of type [java.util.Random] (value [java.util.Random#5cb9231f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Lines 3 and 4 are repeated up to 9 times, as fare as I have seen.
How can I fix this? Can it be ignored?
You need to clean thread locals. See my answer here stop/interrupt a thread after jdbc Driver has been deregister
/**
* Cleanup function which cleans all thread local variables. Using thread
* local variables is not a good practice but unfortunately some libraries
* are still using them. We need to clean them up to prevent memory leaks.
*
* #return number of Thread local variables
*/
private int immolate() {
int count = 0;
try {
final Field threadLocalsField = Thread.class
.getDeclaredField("threadLocals");
threadLocalsField.setAccessible(true);
final Field inheritableThreadLocalsField = Thread.class
.getDeclaredField("inheritableThreadLocals");
inheritableThreadLocalsField.setAccessible(true);
for (final Thread thread : Thread.getAllStackTraces().keySet()) {
count += clear(threadLocalsField.get(thread));
count += clear(inheritableThreadLocalsField.get(thread));
}
log.info("Immolated " + count + " values in ThreadLocals");
} catch (Exception e) {
log.error("ThreadLocalImmolater.immolate()", e);
}
return count;
}
/**
* Cleaner for thread local map.
*
* #param threadLocalMap
* thread local map to clean or null
* #return number of cleaned objects
* #throws Exception
* in case of error
*/
private int clear(#NotNull final Object threadLocalMap) throws Exception {
if (threadLocalMap == null) {
return 0;
}
int count = 0;
final Field tableField = threadLocalMap.getClass().getDeclaredField(
"table");
tableField.setAccessible(true);
final Object table = tableField.get(threadLocalMap);
for (int i = 0, length = Array.getLength(table); i < length; ++i) {
final Object entry = Array.get(table, i);
if (entry != null) {
final Object threadLocal = ((WeakReference<?>) entry).get();
if (threadLocal != null) {
log(i, threadLocal);
Array.set(table, i, null);
++count;
}
}
}
return count;
}