We Keep Coding

Bad Gateway / Native memory allocation (mmap) failed to map x bytes

So sometimes I was getting a bad gateway error from a server when running a Jenkins job, and recently 'sometimes' became 'always'. We managed to find that it was most likely cause by:
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008c900000, 113246208, 0) failed; error='Cannot allocate memory' (errno=12)
There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (mmap) failed to map 113246208 bytes for committing reserved memory.
An error report file with more information is saved as:
//hs_err_pid33.log
[ timer expired, abort... ]
Aborted (core dumped)
When I run free -m I get:
total used free shared buff/cache available
Mem: 7687 5983 209 83 1494 1248
Swap: 0 0 0
When I run less /proc/meminfo I get:
MemTotal: 7872324 kB
MemFree: 210708 kB
MemAvailable: 1281020 kB
Buffers: 346224 kB
Cached: 754116 kB
SwapCached: 0 kB
Active: 6580964 kB
Inactive: 526484 kB
Active(anon): 6012832 kB
Inactive(anon): 82220 kB
Active(file): 568132 kB
Inactive(file): 444264 kB
Unevictable: 3652 kB
Mlocked: 3652 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 284 kB
Writeback: 0 kB
AnonPages: 6010792 kB
Mapped: 102896 kB
Shmem: 85520 kB
Slab: 436552 kB
SReclaimable: 362696 kB
SUnreclaim: 73856 kB
KernelStack: 14336 kB
PageTables: 22640 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 3936160 kB
Committed_AS: 9303016 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 419816 kB
DirectMap2M: 7661568 kB
DirectMap1G: 0 kB
The job in configured that JAVA_OPTS: '-server -Xmx512m -Dlogging.config=/config/log/logback.xml' Increasing -Xmx512m to -Xmx1024m did not help. Also, there are more Jenkins build triggered one after another.
Would be thankful for tips what to try / check to fix this.
Edit: when I run top -o RES I got:
Tasks: 183 total, 1 running, 182 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.8 us, 0.5 sy, 0.0 ni, 98.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 7872324 total, 242744 free, 6122952 used, 1506628 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 1282492 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16119 root 20 0 4766892 1.029g 16536 S 0.0 13.7 3:18.00 java
16095 root 20 0 4754704 1.014g 16640 S 0.0 13.5 2:26.41 java
16022 root 20 0 3595076 590968 16580 S 0.0 7.5 2:00.92 java
15912 root 20 0 3083232 504068 16024 S 0.0 6.4 3:08.03 java
16067 root 20 0 3106944 483148 16732 S 0.0 6.1 1:44.71 java
16037 root 20 0 3039324 464488 16692 S 0.0 5.9 1:37.74 java
16082 root 20 0 3111044 464216 16656 S 0.0 5.9 1:43.01 java
16032 root 20 0 3026936 427452 16040 S 0.0 5.4 1:31.43 java
15926 root 20 0 3118228 419600 16608 S 0.0 5.3 2:45.84 java
15967 root 20 0 3016596 357356 16472 S 0.0 4.5 1:20.40 java
1113 root 20 0 2520832 121108 1376 S 0.0 1.5 605:52.95 dockerd

Thread handling in a java application

I created a service which has in total 11 scheduled jobs running. 3 of them are scheduled by a cron job (2 of them every 15min and the last one every minute). These three tasks are only for monitoring the service (Checking ehCache and RAM used by the JVM). All the other scheduled tasks are annotated with the 'fixedDelay' attribute - so a new thread should only be started if the last one is finished and x time passed, right?
With http://ask.xmodulo.com/number-of-threads-process-linux.html I found out, that I can check the number of threads per process by executing
cat /proc/PID/status
This resulted in the following
Name: jsvc
Umask: 0022
State: S (sleeping)
Tgid: 17263
Ngid: 0
Pid: 17263
PPid: 17260
TracerPid: 0
Uid: 99 99 99 99
Gid: 99 99 99 99
FDSize: 8192
Groups: 99 11332 16600 34691 50780 52730 52823 53043 54173
NStgid: 17263
NSpid: 17263
NSpgid: 17260
NSsid: 17260
VmPeak: 35247540 kB
VmSize: 35232620 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 5679220 kB
VmRSS: 5663344 kB
RssAnon: 5660016 kB
RssFile: 3328 kB
RssShmem: 0 kB
VmData: 32106616 kB
VmStk: 1012 kB
VmExe: 44 kB
VmLib: 16648 kB
VmPTE: 50908 kB
VmPMD: 128 kB
VmSwap: 0 kB
HugetlbPages: 0 kB
Threads: 19922
SigQ: 0/64039
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000004
SigIgn: 0000000000000000
SigCgt: 2000000181005ecf
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000
Seccomp: 0
Speculation_Store_Bypass: vulnerable
Cpus_allowed: 7fff
Cpus_allowed_list: 0-14
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 5986
nonvoluntary_ctxt_switches: 26
So my first question is: What is the 'Threads' number telling me? Are there 19922 threads INCLUDING the threads which are ended or are this only the currently active threads?
I also wondered why all of this threads are currently in SLEEPING state...
I made a graph (#1) which displays the current number of threads for this process and I can see that the number is not only increasing.
So why is this number so wiggly?
Should the subdirectory of a thread be deleted after the thread has finished?
And what is with threads with state "SLEEPING" - are they finished? Because I have nothing to wait for...

So, I found out that
the number "Threads" excludes every finished thread - so all the threads are running or waiting for something.
That is also the reason why this number is wiggling like that.
After checking my source code again, I found out that some ExecutorService objects are not correctly closed so I correted that and received the following graph (which looks better!)
So when somebody else have similar issues, this is what I did:
Login to the machine where the application is running
Get the correct PID from the process by running
ps -aux | grep -i 'NAME' (replace NAME with the correct name of the application)
Get the number of running/waiting threads by executing cat /proc/[PID]/status
Create the graph-data with for x in {1..100000}; do echo $(date) - $(find /proc/[PID]/task -maxdepth 1 | wc -l); sleep 1; done >> thread_counter.csv

How to put tomcat7 always online?

I have Ubuntu 14.04 cloud server with 512MB RAM on Digital Ocean and installed tomcat7 in order to accept my Java applications, also there is a wordpress site running on it with little accesses. So, I created a REST Web Service that needs to always be online because there are accesses by Android Apps. The problem is when I don't use the WS for sometime it goes down and I have to manually start tomcat again.
When I ask for tomcat' status I have the answer below:
Tomcat Servlet engine is not running, but pid file exists.
Here is a memory log of server in normal state:
total used free shared buffers cached
Mem: 490 480 9 64 6 119
-/+ buffers/cache: 354 135
Swap: 0 0 0
Top command:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8835 www-data 20 0 363904 65472 37244 S 16.6 13.0 0:31.02 php5-fpm
12625 www-data 20 0 361052 63896 35704 S 8.3 12.7 0:13.30 php5-fpm
24655 mysql 20 0 891176 56332 1576 S 1.7 11.2 72:04.31 mysqld
11509 www-data 20 0 361696 65796 37168 S 1.3 13.1 0:16.99 php5-fpm
7 root 20 0 0 0 0 S 0.3 0.0 4:31.17 rcu_sched
28 root 20 0 0 0 0 S 0.3 0.0 0:44.41 kswapd0
123 root 20 0 0 0 0 S 0.3 0.0 3:26.29 jbd2/vda1-8
744 www-data 20 0 91112 2400 540 S 0.3 0.5 0:53.93 nginx
13305 tomcat7 20 0 1126588 144516 5792 S 0.3 28.8 0:44.17 java
14557 root 20 0 24820 1508 1100 R 0.3 0.3 0:00.07 top
1 root 20 0 33504 1504 120 S 0.0 0.3 1:59.18 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.29 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:03.83 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
8 root 20 0 0 0 0 R 0.0 0.0 4:37.10 rcuos/0
9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/0
Using the jmap -heap in tomcat process i have these details:
using thread-local object allocation.
Concurrent Mark-Sweep GC
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 134217728 (128.0MB)
NewSize = 1310720 (1.25MB)
MaxNewSize = 44695552 (42.625MB)
OldSize = 5439488 (5.1875MB)
NewRatio = 2
SurvivorRatio = 8
PermSize = 21757952 (20.75MB)
MaxPermSize = 174063616 (166.0MB)
G1HeapRegionSize = 0 (0.0MB)
Heap Usage:
New Generation (Eden + 1 Survivor Space):
capacity = 2424832 (2.3125MB)
used = 280872 (0.26786041259765625MB)
free = 2143960 (2.0446395874023438MB)
11.583152977195946% used
Eden Space:
capacity = 2162688 (2.0625MB)
used = 242168 (0.23094940185546875MB)
free = 1920520 (1.8315505981445312MB)
11.197546756628787% used
From Space:
capacity = 262144 (0.25MB)
used = 38704 (0.0369110107421875MB)
free = 223440 (0.2130889892578125MB)
14.764404296875% used
To Space:
capacity = 262144 (0.25MB)
used = 0 (0.0MB)
free = 262144 (0.25MB)
0.0% used
concurrent mark-sweep generation:
capacity = 34521088 (32.921875MB)
used = 26207256 (24.993186950683594MB)
free = 8313832 (7.928688049316406MB)
75.91665708798054% used
Perm Generation:
capacity = 50319360 (47.98828125MB)
used = 43680848 (41.65730285644531MB)
free = 6638512 (6.3309783935546875MB)
86.8072407916158% used
16661 interned Strings occupying 2074936 bytes.
Does anybody know how to always put it online?

OK, if you have a 512 MB RAM server, and you have MySQL and PHP5 running, the JVM will probably have crashed with an OutOfMemory exception.
In the jmap output, the only important number is the free memory of the concurrent mark sweep generation, where you have only 7.9 MB free, which sounds very small for a web service.
Before it crashes, the JVM will also spend a lot of time trying to garbage collect, which could lead to the process becoming non-responsive, even before it crashes completely.
You could add 1GB of swap (IIRC, linux admins recommend swap = 2 x ram).
See e.g. http://www.prowebdev.us/2012/05/amazon-ec2-linux-micro-swap-space.html for AWS, will probably work on Digital Ocean, too.
The MySQL and PHP5 processes can probably swap out a lot of unused allocated memory. If that slows your applications down too much, you'll probably need some more RAM, or move the PHP and MySQL to different servers.

Java heapsize in Linux box

Can I do -Xmx1792m in Linux box
If I have like this
MemTotal: 10145678 kb
MemFree : 256128 kb
Cached : 9216534 kb
SwapTotal: 2097124 kb
SwapFree: 2045263 kb
Buffers : 243208 kb
Active: 3283536 kb
Inactive: 6224084 kb
VmallocTotal: 34359738367 kB
VmallocUsed: 303168 kB
VmallocChunk: 34359423100 kB

It is perfectly fine to run a 1,5GB Java Heap on a 10GB box, why do you think otherwise? In Linux you need to add cached+buffer to free to see what is unused. If you use the free command, it will do that for you. Make sure to leave some room for buffers, of course.

Your command is valid, but its a bad idea. You only have 256128 kb of free memory, that's 256 mb. 256 < 1792.
Try to end some processes to free up RAM. You've got plenty, its just currently all in use.

Max memory allocation on AWS EBS

I am using ElasticBeanstalk to host my Grails app.
As of now I am using m1.small instance which has 1.7 GiB of main Memory. My question is what is the MaxHeap and Max PermGen I can allocate to my instance? As of now my configuration looks like below
Initial JVM heap size: 512m
JVM command line options: -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+UseConcMarkSweepGC
Maximum JVM heap size: 512m
Maximum JVM permanent generation size: 256m
Any suggestion for selecting the optimum numbers so that I can use max memory for my Tomcat and still have enough left for the OS itself?
Rephrasing the question what is the MAX out of 1.7 GiB can I allocate to something other than the OS(tomcat in this case)

You really need to profile your application stack.
It's easy to increase the size of the heap above the amount of physical memory by adding swap which increases Virtual memory.
For performance you want to find the right fit between instance and application.
Run your application stack in a vagrant instance first and measure it. Adapt until
you get consistent performance, but m1.small instances may be slowed down by others on the same physical host.
That said, you can run top, and sort on Virtual memory.
Example: (Compare VIRT, RES, and SHR columns)
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1220 root 20 0 308m 1316 868 S 0.0 0.3 0:00.43 VBoxService
1126 root 20 0 243m 1576 1040 S 0.0 0.3 0:00.02 rsyslogd
1324 root 20 0 114m 1380 764 S 0.0 0.3 0:00.84 crond
1966 vagrant 20 0 105m 1880 1536 S 0.0 0.4 0:00.12 bash
1962 root 20 0 95788 3740 2832 S 0.0 0.7 0:00.18 sshd
1965 vagrant 20 0 95788 1748 836 S 0.0 0.3 0:00.11 sshd
1323 postfix 20 0 78868 3280 2448 S 0.0 0.7 0:00.00 qmgr
1322 postfix 20 0 78800 3232 2408 S 0.0 0.6 0:00.00 pickup
1314 root 20 0 78720 3252 2400 S 0.0 0.6 0:00.01 master
1238 root 20 0 64116 1180 512 S 0.0 0.2 0:00.80 sshd
This is on a rather small vagrant instance:
$ cat /proc/meminfo
MemTotal: 502412 kB
MemFree: 350976 kB
Buffers: 27148 kB
Cached: 45668 kB
SwapCached: 0 kB
Active: 45240 kB
Inactive: 39616 kB
Active(anon): 12164 kB
Inactive(anon): 44 kB
Active(file): 33076 kB
Inactive(file): 39572 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 1048568 kB
SwapFree: 1048568 kB
Dirty: 4 kB
Writeback: 0 kB