My elasticache is not storing all of the keys it supposed to
I have an elasticache cluster configured and I am using it in my java application to cache values.
The main reason behind this was to reduce ddb calls from the service since it was costing us a fortune.
But even after introducing cache, the read throughput didn't seem to come down as significantly as I expected.
After Putting up cache miss logs in the service, I found out that there are massive cache misses happening.
Which made me think maybe the cache is not populating in the first place, so I put up cache hit logs as well and to my surprise there are cache hit logs as well.
I am a bit confused, either no entries should be there or all entries should be there. how come some entries are populating and some are not.
Note: The TTL for the key is kept 2 days, so no chance of invalidating keys this early.
Any suggestions?
Related
Requirements:
There has a huge number of data from rabbitMq every hour, but the environment is not stable and may cause the project restart and data lack. I want to know the number of the data lack when it occure everyday. Beacuse the volume of data is huge I have to use caffine cache. The large volume of data led me to use caffine caching instead of using database record information. Now the problem is that when the project is restarted, the data in the Caffine cache is cleared and I have no way of knowing when a data miss occurs.
I used to use mongodb to record the time I received data and calcute the time period
I made a Java Web Application hosted by Tomcat. It can be found on this URL.
The problem that I am experiencing is that whenever I visit the page the first time it takes about 10s while every time after it it takes only 100-500ms. I would say the speed improvement is due to browser caching, but not really, when I refresh cache too (ctrl+shift+r) I get the same quick response.
Then after some time, about 5 minutes I visit the page again and it's slow again.
You may try some tests yourself on the URL provided by changing the search parameter value to e.g.: 1050,1051,1052,2670,4000,2300,2200.
Also the interesting fact I have spotted is that no matter how big payload (compare 1050 with 2300) the time is almost always the same approx. 9-10s. So what I assume is that something like Java Server has got to get ready and that is what is taking time.
EDIT:
I was first thinking it could be related to Java/Tomcat having to load some resources and then after some time +-3-5mins again because for some reason it unloaded it. But as I wrote above, even if I change the URL query string (that causes a different SQL query during execution), it again loads long. Can the issue be related to the DB? I am using MySQL.
EDIT2:
The reason why it's faster is most likely the server caching. I am 95% sure and that is because I made couple of experiments such as trying it on 2 computers etc. (no it's not browser caching). Then I realized that if it's fast only when it's cached, what takes so long is the actual .executeQuery line of code. It takes 10s even though the exact same request through a client such as WorkBench takes only 0.285s. Therefore I am going to try to use a PreparedStatement or experiment further.
The content is 200kB in size. You need to optimize it in the front and backend.
The main problem is in your backend. Check why it takes so long.
In the frontend you can enable gzip compression. Check the tomcat documentation on how to do it. This will reduce the size for the download.
The second time is probably faster for you due to browser caching.
Check the response code in Firebug, if the response is 304 it is cached. Response 200 means it has been loaded from the server.
ORM systems (like Hibernate) can cause significantly slow startup if you forget to turn off the initial modell-schema sync option in production environment, for example in Spring -> JPA -> Hibernate -> application-[server].yml->
spring:
jpa:
hibernate:
ddl-auto: update
update none
If your model didn't change, switch "update" to "none" in the production environment for faster startup.
For the sake of simplicity, let's say my app needs to allow thousands of users to see a real-time read-only stream of a chat room. The host can type messages, but no other users can—they just see what's being typed by the hosts, in real time. Imagine users are following a textual play-by-play of a sporting event.
Each user checks for new messages by polling once per second using a simple /get-recent-messages call to the GAE server. (Before you ask, I believe using Google's Channels API would be far too expensive.)
Considering that this app is used by many thousands of users simultaneously, which means dozens to hundreds of GAE instances are running, how can I get these /get-recent-messages calls to return the latest chat room messages with less than 1000 ms latency, while minimizing server load and GAE costs?
Some ideas I had:
Store chat messages in datastore entities.
Obviously this is way too slow and expensive, especially if using queries/indexes
Store chat messages in memcache keys. I imagine I'd use one key to store the list of keys for the last 50 messages, and then 50 more keys, one for each message.
This would cause huge bottlenecks because App Engine's memcache shards by key and thus all 500 instances would be constantly reading from the same memcache keys and thus the same memcache server.
Store chat messages in instance memory, backed by memcache. And pull from memcache (like in #2) when instance memory is stale.
This would probably result in an expensive race condition when multiple requests see stale instance memory cache and pull from memcache simultaneously.
Use a background thread to update instance memory from memcache. This could run once per second per instance using a thread started in the warmup request. It would work like #3 but with only one thread pulling instead of random requests triggering memcache reads.
I don't think this is how background threads work on App Engine. I also don't really know if this is an appropriate use of warmup requests.
Use Google's Pub/Sub service.
I have no idea how this works and it seems like it could be overkill for this use case.
Run a once-per-second cron job to pull from memcache. This would be like #4 except without relying on background threads.
I'd need this to run on every instance every second. I don't believe the cron/taskqueue API has a way to run a job or task on all active instances.
Thoughts?
You should check this video.
I would go for the memcache/datastore version and a small amount of cache (1-2 sec) so you can reduce the amount of instances you need to serve the traffic.
If you still need like 100-500 instances to serve your traffic, i would still go for memcache/datastore version. If memcache is a bottleneck for you, shard it in like 10 keys.
Another solution is to use Compute Engine and a web server that you can connect your users via sockets. You can talk to your compute instances either via HTTP and store the value in memory or using pull queues.
If you really need to communicate to all the instances, take a look at communicating between modules
Pub/sub might be a a good option for you to communicate between the instance that publishes new messages and the instances that read the new messages. From what i read in the docs, you should be able to subscribe your users directly to Pub/Sub too (pull only tho).
We are facing two general issues in our production env and would like to get recommendations.
We are using cluster of nodes running Jboss and apache web server for load balancing.
The two problems are,
All the server nodes work fine normally, however, suddenly within a minute, one of the node reach out the maximum DB connection limit (say, from 30 to 100) and start throwing errors (Unable to get manage connection).
I have seen that sometimes, we got simultaneously lot of same webservice calls from one user. For instance, more than 1000 web service calls of the same service by same user within a minute. It looks like, may be user is stuck in some kind of repetitive loop in browser (not sure).
To fix first problem, I have seen we don't have any connection leak issue. Mostly, we found that the service response time becomes very high, however the load balancer sends the equal traffic to each node and therefore possibly, this node might get exhausted. One solution I was thinking is to timeout the service call earlier which takes more than certain time but I am not sure is it good idea. Any thoughts, what recommendations/practice available to tackle such situation?
To fix second problem, I think the application should not defend or check for such large number of service calls but it should be in the higher level like firewall or web server. However, I would like to know your thoughts for this.
I hope my question make sense but if it doesn't, please feel free to ask,
I have this scenario:
3 databases that need to be kept identical
3 java app servers that handle requests and write to the databases
each java app server connects to a single database, as well as the other app servers.
I am looking for a solution where I can have a transaction that either commits or rollsback on all 3 databases, however it seems like all the solutions I find are for a single app server connecting to all 3 databases. Normally I would implement some RPC logic such that the fact that the database is behind a second app server is transparent, however I would like to ask the following:
Is there a better way of doing this instead? (I don't see as having each app server connect to all the databases as a better solution, simply due to the sheer number of increased connections)
If not, what object should I pass with the RPC call? a Connection? a ConnectionFactory?
More context:
This is intended to be a HA solution where if a single app server or DB goes down, transactions can still occur on the remaining 2 nodes, and when the failed node comes back up, it will re-sync and then come back "online". Each app server/DB pair is in a seperate datacenter, and the cross-datacenter connections should be kept to a minimum. The writes are inevitably going to be cross-datacenter, in order to keep the DB's in sync, however the reads (main usecase) don't need to be, as if an app server is "online", then it can be fairly confident that it's data is an identical copy to the other's. I haven't found a good way to do this in the DB layer, as MMR seems to be a very restrictive PITA. Also, any solution should be scalable such that if the node count increases to 4/5/6/etc. any changes are limited to configuration, and not code changes as much as possible.