We are evaluating Apache Ignite for our product. In our scenario, we may have 10000 caches, and I have a try in the yardstick benchmark framework. I find that when the cache numbers climb to 8192, the Ignite server became abnormal. The case is expected to be finished after 1 minute since I have set the duration in the configuration, but the test keep running in 10 minutes long and I have to kill the test.
If I set the cache number to 4096, the test finished in 1 minute as expected.
So the question: Does Apache Ignite support 10 thousands of cache?
One cache will use around 20M heap for its data structures (per node). Multiple that by 10000 and you have 200G right here. In practice Java will not work with that much heap.
Why do you need 10,000 caches anyway? Please consider at least using Cache Groups. The best approach will be having a few caches and routing between them.
Related
We have a small search app in local context. For back services, we are using Apache Solr 6.6.2 for data index and storage. The front-end is in PHP with Apache2 webserver.
We have a server of 48 core and 96 GB RAM where these services are installed. The expected size of documents in index in about 200 Million and each document can have maximum 20 fields. Most fields are both indexed and stored.
The expected simultaneous requests can be hundreds of thousands at a time. So what will be the best configuration of Apache Solr to handle it? We have started Solr with 20 GB RAM and stress tested but it start to degrade performance near 100 users. Where is the problem? What is the optimal way for this issue.
We have also tested Solr in SolrCloud mode but the performance does not improve too much. We were expecting that if there will be some memory problem that their will be OOM exception but did not happen anything like that. We have just changed schema according to our requirement and change memory via command-line. All other setting are default.
Following are few references that we have consulted already
https://wiki.apache.org/solr/SolrPerformanceProblems
https://blog.cloudera.com/blog/2017/06/apache-solr-memory-tuning-for-production/
We have 200 million records in each Collection and we have 200 collections. We have 5 servers and each server has 8 cores and 64 gb ram.
I would suggest you break up your servers into multiple servers.
Do the replication of the data on each server so that requests gets divided into multiple servers. The more the number of servers, the more you'll be able to responsd quicker.
Note : Just understand the replication factor : 2F+1 formula where if you have 5 servers then 3 replicas atleast should be there. I'll suggest you to go with 5 replicas only (1 replica for each server)
If you plan to handle hundreds of thousands of requests per second you will need more than one server - no matter how big it is. Even if it's just for HA / DR purposes. So I would recommend using SolrCloud and sharding the index across multiple machines and with multiple replicas just so start.
Beyond that the devil is in the details
How fast do you want queries to perform (median and 99%ile). This will help you size CPU and memory needs.
How complex are your queries?
Are you using filters? (Requires more heap memory)
How fast is your disk acccess?
Will you be adding data in real time (impacting your setting of autoCommit and soft commit
But first and foremost you need to get away from "one big box" thinking.
I am not getting the performance that I am expecting from using Apache Ignite DataGrid. I have tried a few configuration changes but at this point don't know how to investigate performance bottlenecks, and am looking for expert help.
I am using Apache Ignite to cache a byte array using a wrapper class I call ByteArray. My test code attempts to benchmark the cache performance by calling multiple puts and then multiple gets from another process. I've tried running the get process on the same node and on different nodes. I have also created a baseline performance spec using Java HashMap as my cache, and this has much better performance (10000x for put).
Right now, on the same node, I get the following:
Hashmap cache, same node: put 2600 MB/s; get 300 MB/s
Ignite same node cache: put 0.4 MB/s; get 2.0 MB/s
Ignite cache, 2 nodes: put 0.3 MB/s; get 0.7 MB/s
I ran these in replicated mode but I see similar results for partitioned mode. I run multiple iterations of the test and average the timing. My nodes have 25GB memory and my test consumes ~1GB. I have configured the VM to use 10GB max.
First of all, comparing Ignite performance with HashMap doesn't make a lot of sense. Ignite is a distributed and scalable system, while HashMap is not even thread-safe.
The fact that you used HashMap as a baseline, makes me think that your test is single threaded. If you try to query Ignite from multiple threads/clients, I'm sure you will notice much better throughput.
Also keep in mind that working with Ignite implies sending data across network. There is a chance you're limited by its speed.
I am using spark streaming in my application. Data comes in the form of streaming files every 15 minute. I have allocated 10G of RAM to spark executors. With this setting my spark application is running fine.
But by looking the spark UI, under Storage tab -> Size in Memory the usage of RAM keep on increasing over the time.
When I started streaming job, "Size in Memory" usage was in KB. Today it has been 2 weeks 2 days 22 hours since when I started the streaming job and usage has increased to 858.4 MB.
Also I have noticed on more thing, under Streaming heading:
When I started the job, Processing Time and Total Delay (from the image) was 5 second and which after 16 days, increased to 19-23 seconds while the streaming file size is almost same.
Before increasing the executor memory to 10G, spark jobs keeps on failing almost every 5 days (with default executor memory which is 1GB). With increase of executor memory to 10G, it is running continuously from more than 16 days.
I am worried about the memory issues. If "Size in Memory" values keep on increasing like this, then sooner or later I will run out of RAM and spark job will get fail again with 10G of executor memory as well. What I can do to avoid this? Do I need to do some configuration?
Just to give the context of my spark application, I have enable following properties in spark context:
SparkConf sparkConf = new SparkConf().setMaster(sparkMaster). .set("spark.streaming.receiver.writeAheadLog.enable", "true")
.set("spark.streaming.minRememberDuration", 1440);
And also, I have enable checkpointing like following:
sc.checkpoint(hadoop_directory)
I want to highlight one more thing is that I was having issue while enabling checkpointing. Regarding checkpointing issue, I have already posted a question on following link:
Spark checkpoining error when joining static dataset with DStream
I was not able to set the checkpointing the way I wanted, so did it differently (highlighted above) and it is working fine now. I am not asking checkpointing question again, however I mentioned it so that it might help you to understand if current memory issue somehow related to previous one (checkpointing).
Environment detail:
Spark 1.4.1 with two node cluster of centos 7 machines. Hadoop 2.7.1.
I am worried about the memory issues. If "Size in Memory" values keep on increasing like this, then sooner or later I will run out of RAM and spark job will get fail again with 10G of executor memory as well.
No, that's not how RAM works. Running out is perfectly normal, and when you run out, you take RAM that you are using for less important purposes and use it for more important purposes.
For example, if your system has free RAM, it can try to keep everything it wrote to disk in RAM. Who knows, somebody might try to read it from disk again and having it in RAM will save an I/O operation. Since free RAM is forever wasted (it's not like you can use 1GB less today to use 1GB more tomorrow, any RAM not used right now is potential to avoid I/O that's forever lost) you might as well use it for anything that might help. But that doesn't mean it can't evict those things from RAM when it needs RAM for some other purpose.
It is not at all unusual, on a typical system, for almost all of its RAM to be used and almost all of its RAM to also be available. This is typical behavior on most modern systems.
I often run workloads against my own web applications to try find performance issues.
Sometimes I see memory leaks etc at variable duration.
So I created a bash script to take javacores, kill -3 pid , every minute for 10 minutes and script is executed on the hour.
For a workload that runs for 120 hours, this will produce 1200 javacores.
I'm wondering,
Is this overkill? I'd like a continuous view of system (javacore every 5 minutes, for 120 hours), but don't want to impact perf
what is a reasonable frequency to automatically capture javacores against servlet based app?
Looks like we are looking at two issues:
Performance
OutOfMemoryError
Performance: for performance, determine the longest request you can tolerate and generate the javacores when its 3 to 5 times that amount. (Anything below 5 minutes to me is fine tuning and can difficult)
Let's say your longest request you want is 3 minutes, I would generate 3 javacores at evenly from 9 minutes to 15 minutes.
I usually suggest the link (collect manually) below but if you already wrote your own script use it
"MustGather: Performance, Hang or High CPU Issues on Linux"
http://www.ibm.com/support/docview.wss?rs=180&uid=swg21115785
OutOfMemoryError: see if your product is leaking, follow the steps in URL below and go to collect manually and google IBM heap analyzer (stand alone and free) and review the heap dump for potential leak suspects.
"MustGather: Native Memory Issues on Linux"
http://www.ibm.com/support/docview.wss?rs=180&uid=swg21138462
Personnaly, I prefer looking at heap dumps memory use to equal the XMX or nearly that.
Since this is an IBM JVM you could try using Health Center instead of taking javacores regularly:
http://www.ibm.com/developerworks/java/jdk/tools/healthcenter/
This has profiling and memory monitoring views so should give you the data you are looking and save you analysing the javacore files yourself.
I'm trying to speed test jetty (to compare it with using apache) for serving dynamic content.
I'm testing this using three client threads requesting again as soon as a response comes back.
These are running on a local box (OSX 10.5.8 mac book pro). Apache is pretty much straight out of the box (XAMPP distribution) and I've tested Jetty 7.0.2 and 7.1.6
Apache is giving my spikey times : response times upto 2000ms, but an average of 50ms, and if you remove the spikes (about 2%) the average is 10ms per call. (This was to a PHP hello world page)
Jetty is giving me no spikes, but response times of about 200ms.
This was calling to the localhost:8080/hello/ that is distributed with jetty, and starting jetty with java -jar start.jar.
This seems slow to me, and I'm wondering if its just me doing something wrong.
Any sugestions on how to get better numbers out of Jetty would be appreciated.
Thanks
Well, since I am successfully running a site with some traffic on Jetty, I was pretty surprised by your observation.
So I just tried your test. With the same result.
So I decompiled the Hello Servlet which comes with Jetty. And I had to laugh - it really includes following line:
Thread.sleep(200L);
You can see for yourself.
My own experience with Jetty performance: I ran multi threaded load tests on my real-world app where I had a throughput of about 1000 requests per second on my dev workstation...
Note also that your speed test is really just a latency test, which is fine so long as you know what you are measuring. But Jetty does trade off latency for throughput, so often there are servers with lower latency, but also lower throughput as well.
Realistic traffic for a webserver is not 3 very busy connections - 1 browser will open 6 connections, so that represents half a user. More realistic traffic is many hundreds or thousands of connections, each of them mostly idle.
Have a read of my blogs on this subject:
https://webtide.com/truth-in-benchmarking/
and
https://webtide.com/lies-damned-lies-and-benchmarks-2/
You should definitely check it with profiler. Here are instructions how to setup remote profiling with Jetty:
http://sujitpal.sys-con.com/node/508048/mobile
Speedup or performance tune any application or server is really hard to get done in my experience. You'll need to benchmark several times with different work models to define what your peak load is. Once you define the peak load for the configuration/environment mixture you need to tune and benchmark, you might have to run 5+ iterations of your benchmark. Check the configuration of both apache/jetty in terms of number of working threads to process the request and get them to match if possible. Here are some recommendations:
Consider the differences of the two environments (GC in jetty, consider tuning you min and max memory threshold to the same size and later proceed to execute your test)
The load should come from another box. If you don't have a second box/PC/server take your CPU/core into count and setup your the test to a specific CPU, do the same for jetty/apache.
This is given that you cant get another machine to be the stress agent.
Run several workload model
Moving to modeling the test do the following 2 stages:
One Thread for each configuration for 30 minutes.
Start with 1 thread and going up to 5 with a 10 minutes interval to increase the count,
Base on the metrics Stage 2 define a number of threads for the test. and run that number of thread concurrent for 1 hour.
Correlate the metrics (response times) from your testing app to the server hosting the application resources (use sar, top and other unix commands to track cpu and memory), some other process might be impacting you app. (memory is relevant for apache jetty will be constraint to the JVM memory configuration so it should not change the memory usage once the server is up and running)
Be aware of the Hotspot Compiler.
Methods have to be called several times (1000 times ?), before the are compiled into native code.