We are trying to access de ElasticCache (Redis) on aws using a Java client that runs locally using Jedis lib. We were able to access the redis using redis-cli locally following the steps here.
The problem is that when we try to connect to aws Redis using Jedis lib, the NAT public address are being translated to the redis private IPs in order to calculate the slots (initializeSlotsCache). We couldn't find a way to disable this. Are there any workaround?
Here's how we connect using Jedis:
factory = new JedisConnectionFactory(new RedisClusterConfiguration(this.clusterProperties.getNodes()));
factory.setUsePool(true);
factory.setPoolConfig(this.jedisPoolConfig());
factory.afterPropertiesSet();
return factory;
We are using the mapped NAT ips for each node. But the Jedis lib is saving the private ips in the cache, so we get the following exception:
redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
Any suggestions would be great! We are running out of options. Thank you in advance.
You cannot connect to AWS hosted redis from outside the AWS environment.
We had faced a similar issue, and the work around we had was to install a local redis instance for dev and unit testing.
Related
In AWS we have SSM which help us to connect to Cluster Instances and using SSM we got the data about the container in Java.
Same I am looking for Azure too. I want to connect with Azure Cluster Instances.
Is there any service in Azure available to get it done ?
In Azure i find out we can achieve this goal using Azure VM.
https://learn.microsoft.com/en-us/rest/api/compute/virtual-machines/run-command
We're using Google Cloud Run with a java app to store data in Cloud Sql Postgres. We need to connect to external services using a static IP.
We are connecting to the database using the example code provided in the documentation, with the following connection string: jdbc:postgresql:///<DB_NAME>?unixSocketPath=</PATH/TO/UNIX/SOCKET>&cloudSqlInstance=<CLOUD_SQL_CONNECTION_NAME>& // socketFactory=com.google.cloud.sql.postgres.SocketFactory&user=<DB_USER>&password=<DB_PASS>. In the Cloud Run service we pointed the Cloud SQL connections to the database. This works well.
To add a static egress IP to the Cloud Run service, we followed the guide to Static outbound IP address.
We're running a test service that shows the external IP, and again, this shows the static external IP.
The problem is that now the connection to the database does not work anymore.
To fix this, we removed the unixSocketPath from the database connection string, resulting in jdbc:postgresql:///<DB_NAME>?cloudSqlInstance=<CLOUD_SQL_CONNECTION_NAME>& // socketFactory=com.google.cloud.sql.postgres.SocketFactory&user=<DB_USER>&password=<DB_PASS>.
Now the following problem pops up, once in a while:
Failed to create ephemeral certificate for the Cloud SQL instance.
We believe this is due to Cloud Run throttling our service to near 0 when not in use, as alluded to in this reply in the socket factory github issues. This problem did not happen when connecting over the unix socket.
Stumped, we tried to minimise code and dependencies, resulting in the following test project: https://gist.github.com/goulashify/b05593dceae717e425b097f3fe840883
What are we missing? Is there a way to connect to the outside world using a static ip, and have a reliable connection to Cloud Sql?
I am following a Redis tutorial online that shows how to connect to Redis from a Java application.
I understand that there are many JAVA clients available, and the tutorial I was following was using Jedis. My question is, can a JAVA client (like Jedis) be used without actually installing the redis server itself? The tutorial shows a simple call to:
Jedis jedis = new Jedis("localhost");
and then beginning set/get operations, but I don't believe they installed Redis. I am new to Redis, but I am picturing installing the redis server to be the equivalent of installing something like Oracle, and then using a JAVA API to actually talk to that Oracle instance.
How is the Jedis API used without an actual Redis instance present? If the Jedis client was initialized without the host parameter, would it then expect to find an actual Redis server/instance running on port 6379?
I'm new to AWS Elasticache redis, and I got below endpoint.
I'm confused in either using Jedis and Redisson, because both provides single connection and cluster connection class.
Like in Jedis, for a single connection we can use:
Jedis conn = new Jedis("endpoint_address");
And for cluster connection we use:
Set<HostAndPort> jedisClusterNodes = new HashSet<HostAndPort>();
jedisClusterNodes.add(new HostAndPort("redis_cluster_ip", 7379));
JedisCluster jc = new JedisCluster(jedisClusterNodes);
These option also occur when I want to use Redisson. I'm not try to compare these two lib, my question is: WHICH ONE is the right method of connecting to AWS Redis Elasticache cluster, when you only have one end-point and still can utilize AWS auto scaling feature?
Expected answer is: use SINGLE or CLUSTER MODE.
Thanks :)
It depends on how you have the redis cluster configured. Whether or not cluster mode is enabled.
You can find it in the console
http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/Endpoints.html
Redis (cluster mode disabled) clusters, use the Primary Endpoint for
all write operations. Use the individual Node Endpoints for read
operations (In the API/CLI these are referred to as Read Endpoints).
Redis (cluster mode enabled) clusters, use the cluster's Configuration
Endpoint for all operations. You must use a client that supports Redis
Cluster (Redis 3.2). You can still read from individual node endpoints
(In the API/CLI these are referred to as Read Endpoints).
Or with the AWS CLI
aws elasticache describe-cache-clusters \
--cache-cluster-id mycluster \
--show-cache-node-info
http://docs.aws.amazon.com/cli/latest/reference/elasticache/describe-cache-clusters.html
ConfigurationEndpoint -> (structure) Represents a Memcached cluster
endpoint which, if Automatic Discovery is enabled on the cluster, can
be used by an application to connect to any node in the cluster. The
configuration endpoint will always have .cfg in it. Example:
mem-3.9dvc4r.cfg.usw2.cache.amazonaws.com:11211
You should use Replicated configuration in Redisson for AWS Elasticache Redis or other similar hosted services. The usage is described in the documentation.
I'm trying to write a simple spark application, and when i run it locally it works with setting the master as
.master("local[2]")
But after configuring spark cluster on AWS (EMR) i can't connet to the master url:
.master("spark://<master url>:7077")
Is this the way to do it? am i missing something here?
The cluster is up and running, and when i tried adding my application as a step jar, so it will run directly in the cluster it worked. But i want to be able to run it from a remote machine.
would appreciate some help here,
Thanks
To run from a remote machine, you will need to open the appropriate ports in the Security Group assigned to your EMR master node. You will need to add at least 7077.
If by "remote" you mean one that isn't in your AWS environment, you will also need to setup a way to route traffic to it from the outside.