I'm using Jedis to perform a lot of insertions/reads in Redis.
The Redis server is using the default configuration.
The problem appears when I start using a few threads and the exception is:
redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketException: Pipe quebrado (Write failed)
I've searched a lot about this problem but couldn't find the reason of it or it's solve. The code I'm using to perform these tests is below:
public class RedisFacade {
private static RedisFacade instancia = null;
// Initialize the Connection
final JedisPoolConfig poolConfig = buildPoolConfig();
JedisPool pool = new JedisPool(poolConfig, "localhost");
Jedis jedis;
int i = 0;
private RedisFacade() {
}
public static RedisFacade getInstancia() {
if (instancia == null) {
instancia = new RedisFacade();
}
return instancia;
}
// retorna um cliente jedis da pool
public Jedis getDB() {
if (jedis == null) {
jedis = pool.getResource();
}
return jedis;
}
//inserting
public void insert(Document d) {
String key = i + d.getString("date") + d.getString("time");
String value = d.toString();
this.getDB().set(key, value);
i++;
}
//reading
public void read(String date, String time) {
Object doc = this.getDB().get(i + date + time);
i++;
System.out.println(doc);
}
public void destroyPool() {
this.pool.destroy();
}
private JedisPoolConfig buildPoolConfig() {
final JedisPoolConfig poolConfig = new JedisPoolConfig();
poolConfig.setMaxTotal(1100);
poolConfig.setMaxIdle(16);
poolConfig.setMinIdle(16);
poolConfig.setTestOnBorrow(true);
poolConfig.setTestOnReturn(true);
poolConfig.setTestWhileIdle(true);poolConfig.setMinEvictableIdleTimeMillis(Duration.ofSeconds(60).toMillis());
poolConfig.setTimeBetweenEvictionRunsMillis(Duration.ofSeconds(30).toMillis());
poolConfig.setNumTestsPerEvictionRun(3);
poolConfig.setBlockWhenExhausted(true);
return poolConfig;
}}
Seems like it's a timeout issue.
See this thread: Configure Jedis timeout
And also this discussion: https://github.com/xetorthio/jedis/issues/185
So I would try instantiating JedisPool with a timeout parameter
(i.e. https://github.com/xetorthio/jedis/blob/master/src/main/java/redis/clients/jedis/JedisPool.java#L201, but there are many other constructors)
and setting CONFIG SET timeout 600 in redis (with a 10 minute timeout for instance).
Edit
The JedisPool timeout is in milliseconds it seems.
After trying to implement new constructors, new configurations for the pool and clients, I tried a simple way to fix the problem: close the resources I was getting from the pool. To do so, I've changed the following code:
public Jedis getDB() {
jedis = pool.getResource();
return jedis;
}
//cria um _id pra ser usado novamente quando for buscar os documentos
public void insert(Document d) {
String key = "key" + i;
String value = d.toString();
Jedis jedis = this.getDB();
jedis.set(key, value);
jedis.close();
i++;
}
//busca pelo _id
public void read() {
Jedis jedis = this.getDB();
Object doc = jedis.get("key" + i);
jedis.close();
i++;
System.out.println(doc);
}
After changing the code, the service started to work I was planning, so I'll accept this as a solution.
Related
I am using Redis(3.2.100) for Windows to cache my database data in Java.This is my redis init code:
private static Dictionary<Integer, JedisPool> pools = new Hashtable();
static {
JedisPoolConfig config = new JedisPoolConfig();
config.setMaxIdle(2);
config.setMaxTotal(10);
config.setTestOnBorrow(true);
config.setMaxWaitMillis(2000);
for (int i = 0; i < 16; i++) {
JedisPool item = new JedisPool(config, "127.0.0.1", 6379,10*1000);
pools.put(i, item);
}
}
This is the cache code:
public static String get(String key, Integer db) {
JedisPool poolItem = pools.get(db);
Jedis jredis = poolItem.getResource();
String result = jredis.get(key);
return result;
}
The problem is when the program run for a while,the getResource method throws:
redis.clients.jedis.exceptions.JedisException: Could not get a resource from the pool
So how to reuse the connection or close the connection.I am using this command to find out that the client has reached the max value.
D:\Program Files\Redis>redis-cli.exe info clients
# Clients
connected_clients:11
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
How to fix it?
Remember to close the redis connection,modify this function like this:
public static String get(String key, Integer db) {
JedisPool poolItem = pools.get(db);
Jedis jredis = null;
String result = null;
try {
jredis = poolItem.getResource();
result = jredis.get(key);
} catch (Exception e) {
log.error("get value error", e);
} finally {
if (jredis != null) {
jredis.close();
}
}
return result;
}
I need to figure out how many writes MongoDB has performed in the last hour against reads.
Is there an easy way to find these stats, which are needed to create an alarm. If the solution is command driven or Java based, it will be really helpful.
after digging up source code of MongoDB jdbc driver . i finally able to get what is required . below is a code for it
public class MongoJmxStat {
private MongoClient mongo;
public MongoJmxStat(MongoClient mongo) {
this.mongo = mongo;
}
public CommandResult getServerStatus() {
CommandResult result = getDb("mongoDB").command("serverStatus");
if (!result.ok()) {
throw new MongoException("could not query for server status. Command Result = " + result);
}
return result;
}
public DB getDb(String databaseName) {
return MongoDbUtils.getDB(mongo, databaseName);
}
private int getOpCounter(String key) {
DBObject opCounters = (DBObject) getServerStatus().get("opcounters");
return (Integer) opCounters.get(key);
}
#ManagedMetric(metricType = MetricType.COUNTER, displayName = "Write operation count")
public int getWriteCount() {
return getOpCounter("insert") + getOpCounter("update") + getOpCounter("delete");
}
#ManagedMetric(metricType = MetricType.COUNTER, displayName = "Read operation count")
public int getReadCount() {
return getOpCounter("query") + getOpCounter("getmore");
}
}
When I use jedis like code below:
public class JedisTest extends Sync {
private static final String _SET_KEY_1 = "test1";
private static final String _SET_KEY_2 = "test2";
public void process() throws SQLException {
Set<String> appSet = getAllUserableAppkey();
final ShardedJedis jedis = RedisHelper.getJedis();
final ShardedJedisPipeline pipeline = jedis.pipelined();
for (String key : appSet) {
Set<String> result = jedis.smembers(_SET_KEY_1);
Set<String> result2 = jedis.smembers(_SET_KEY_2);
String rangName = String.format("%s::%s", "test", key);
for (int i = 0; i < 10; i++) {
pipeline.sadd(rangName, String.valueOf(i));
}
}
pipeline.sync();
}
public Set<String> getAllUserableAppkey() {
}
public static void main(String[] args) throws Exception {
DbHelper.init();
RedisHelper.init();
JedisTest jedisTest = new JedisTest();
try {
jedisTest.process();
} catch (SQLException e) {
e.printStackTrace();
}
}
}
It throw the Exception like this:
Exception in thread "main" java.lang.ClassCastException: java.lang.Long cannot be cast to java.util.List
at redis.clients.jedis.Connection.getBinaryMultiBulkReply(Connection.java:224)
at redis.clients.jedis.Connection.getMultiBulkReply(Connection.java:217)
at redis.clients.jedis.Jedis.smembers(Jedis.java:1055)
at redis.clients.jedis.ShardedJedis.smembers(ShardedJedis.java:339)
at com.snda.sync.impl.test.JedisTest.process(JedisTest.java:29)
at com.snda.sync.impl.test.JedisTest.main(JedisTest.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
I can modify the code to correct like :
public void process() throws SQLException {
Set<String> appSet = getAllUserableAppkey();
final ShardedJedis jedis = RedisHelper.getJedis();
for (String key : appSet) {
final ShardedJedisPipeline pipeline = jedis.pipelined();
Set<String> result = jedis.smembers(_SET_KEY_1);
Set<String> result2 = jedis.smembers(_SET_KEY_2);
//log.warn("result1 :{},result2:{}",result,result2);
String rangName = String.format("%s::%s", "test", key);
for (int i = 0; i < 10; i++) {
pipeline.sadd(rangName, String.valueOf(i));
}
pipeline.sync();
}
}
But I dont know why that exception throw , ,does pipline.sadd() conflict with jedis.smembers ?
Thanks for answer me!
The jedis is latest:2.7.2
You should not use Jedis instance directly while pipelined.
Pipeline uses Jedis instance' stream (not initializing new stream), and since normal operation reads response immediately and pipeline reads whole responses later, mixing up two usages gets Jedis into trouble.
P -- Pipelined / N -- Normal
Request --> P(1) P(2) N(3) N(4) P(5)
Redis Response --> P(1) P(2) N(3) N(4) P(5)
Matched request-response --> N(1 : should be 3) N(2 : should be 4) P(3 : should be 1) P(4 : should be 2) P(5)
You can see response can be easily flawed.
In my weblogic server ehcache is deployed , I need to get ehcahe mbeans from this program through java programming, Through JMX i am not able to connect.how can i get those custom mbeans??
i tried to get mbeans through weblogic t3 protocol
public class Test
{
private String hostName = "";
private String port = "";
private String userName = "";
private String password = "";
private String connectorURL = "service:jmx:rmi:///jndi/rmi://{0}:{1}/jmxrmi";
private JMXConnector jmxc = null;
public static void main(String []args) throws Exception
{
Test t = new Test();
t.hostName = args[0];
System.out.println(args[1]);
t.port = args[1];
t.userName = args[2];
t.password = args[3];
t.jmxc = t.initConnection();
MBeanServerConnection mbsc = t.jmxc.getMBeanServerConnection();
System.out.println(mbsc);
Set<ObjectInstance> st =mbsc.queryMBeans(new ObjectName("net.*:*"), null);
System.out.println(st.toString());
Iterator<ObjectInstance> it = st.iterator();
while(it.hasNext())
{
System.out.println(it.next());
}
t.closeConnection();
}
private JMXConnector initConnection()
{
System.out.println("initiate connection");
JMXServiceURL serviceURL = null;
try
{
String jndiroot = "/jndi/";
String mserver = "weblogic.management.mbeanservers.domainruntime";
int port1 = Integer.parseInt(port);
serviceURL = new JMXServiceURL("t3", hostName, port1, jndiroot + mserver);
Hashtable h = new Hashtable();
h.put(Context.SECURITY_PRINCIPAL, userName);
h.put(Context.SECURITY_CREDENTIALS, password);
h.put(JMXConnectorFactory.PROTOCOL_PROVIDER_PACKAGES, "weblogic.management.remote");
long lngJmxClientWTO = 10000;
h.put("jmx.remote.x.request.waiting.timeout", lngJmxClientWTO );
return JMXConnectorFactory.connect(serviceURL, h);
}
catch (Exception e)
{
e.printStackTrace();
return null;
}
}
/**
* This method closes client connection with server
* #throws IOException
*/
public void closeConnection()
{
if(jmxc != null)
{
try
{
jmxc.close();
}
catch (IOException e) {
jmxc = null;
}
}
}
import net.sf.ehcache.CacheManager;
import net.sf.ehcache.Ehcache;
CacheManager manager = CacheManager.newInstance();
Ehcache cache = manager.getEhcache("Some cache name here..."); //<-- PLEASE EDIT THE CACHE NAME...
I don't know if this is what you're asking for....
Once you've obtained your cache, you can use it, pretty much like a java Map.
You can follow the Ehcache documentation to see how to programmatically get the remote cache. Essentially, you will need to create a configuration (or configuration file) which the CacheManager can access.
Today I tried to fix some potential memory leaks in my web application.
I use the following libraries.
spring-webmvc-3.2.9.RELEASE
spring-data-mongodb-1.5.0.RELEASE
mongo-java-driver-2.12.1
First I missed to close the MongoClient but changed my configuration this way.
#Configuration
public class MongoDBConfiguration implements DisposableBean {
private MongoClient mongoClient;
#Bean
public MongoTemplate mongoTemplate() {
try {
final Properties props = loadProperties();
log.debug("Initializing Mongo DB client");
mongoClient =
new MongoClient(getProperty(props, "host", "localhost"), cint(getProperty(props, "port",
"27017")));
UserCredentials credentials = null;
final String auth = getProperty(props, "auth", null);
if (auth != null && auth.equalsIgnoreCase("true")) {
final String user = getProperty(props, "user", null);
final String pass = getProperty(props, "pass", null);
if (user != null && pass != null) {
credentials = new UserCredentials(user, pass);
}
}
final MongoDbFactory mongoDbFactory =
new SimpleMongoDbFactory(mongoClient, getProperty(props, "dbname", "Feeder"), credentials);
final MappingMongoConverter mongoConverter =
new MappingMongoConverter(new DefaultDbRefResolver(mongoDbFactory),
new MongoMappingContext());
mongoConverter.setCustomConversions(customConversions(mongoConverter));
mongoConverter.afterPropertiesSet();
final MongoTemplate mongoTemplate = new MongoTemplate(mongoDbFactory, mongoConverter);
return mongoTemplate;
} catch (final IOException e) {
log.error("", e);
}
return null;
}
/**
* close Mongo client to avoid memory leaks
*/
#Override
public void destroy() {
log.debug("Shutdown Mongo DB connection");
mongoClient.close();
log.debug("Mongo DB connection shutdown completed");
}
}
When stopping or reloading the web application, there are still messages complaining about probable memory leaks.
2014-06-24 07:58:02,114 DEBUG d.p.f.s.m.MongoDBConfiguration - Shutdown Mongo DB connection
2014-06-24 07:58:02,118 DEBUG d.p.f.s.m.MongoDBConfiguration - Mongo DB connection shutdown completed
Jun 24, 2014 7:58:02 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/feeder##1.5.1] created a ThreadLocal with key of type [com.mongodb.BaseCluster$1] (value [com.mongodb.BaseCluster$1#766465]) and a value of type [java.util.Random] (value [java.util.Random#5cb9231f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Lines 3 and 4 are repeated up to 9 times, as fare as I have seen.
How can I fix this? Can it be ignored?
You need to clean thread locals. See my answer here stop/interrupt a thread after jdbc Driver has been deregister
/**
* Cleanup function which cleans all thread local variables. Using thread
* local variables is not a good practice but unfortunately some libraries
* are still using them. We need to clean them up to prevent memory leaks.
*
* #return number of Thread local variables
*/
private int immolate() {
int count = 0;
try {
final Field threadLocalsField = Thread.class
.getDeclaredField("threadLocals");
threadLocalsField.setAccessible(true);
final Field inheritableThreadLocalsField = Thread.class
.getDeclaredField("inheritableThreadLocals");
inheritableThreadLocalsField.setAccessible(true);
for (final Thread thread : Thread.getAllStackTraces().keySet()) {
count += clear(threadLocalsField.get(thread));
count += clear(inheritableThreadLocalsField.get(thread));
}
log.info("Immolated " + count + " values in ThreadLocals");
} catch (Exception e) {
log.error("ThreadLocalImmolater.immolate()", e);
}
return count;
}
/**
* Cleaner for thread local map.
*
* #param threadLocalMap
* thread local map to clean or null
* #return number of cleaned objects
* #throws Exception
* in case of error
*/
private int clear(#NotNull final Object threadLocalMap) throws Exception {
if (threadLocalMap == null) {
return 0;
}
int count = 0;
final Field tableField = threadLocalMap.getClass().getDeclaredField(
"table");
tableField.setAccessible(true);
final Object table = tableField.get(threadLocalMap);
for (int i = 0, length = Array.getLength(table); i < length; ++i) {
final Object entry = Array.get(table, i);
if (entry != null) {
final Object threadLocal = ((WeakReference<?>) entry).get();
if (threadLocal != null) {
log(i, threadLocal);
Array.set(table, i, null);
++count;
}
}
}
return count;
}