JMeter : java.net.NoRouteToHostException: Cannot assign requested address (Address not available) - java

I have created a simple Spring boot Application having a HelloController.
Get API: http://localhost:8080/hello
Response: Hello World
Now I have created a JMeter Script having 0.1 million concurrent users hitting the above get API.
When I run the JMeter Script, after 30k count, I start getting the error:
java.net.NoRouteToHostException: Cannot assign requested address (Address not available)
What is the reason for this? How can I resolve this issue?
I'm using UBUNTU 18.04 with 8gb RAM.
While performing the operation, only JMeter and STS was open.

You can follow Lakshmi Narayan answer to increase available ports:
Resolution:
Increased the local port range using below command:
echo 1024 65000 > /proc/sys/net/ipv4/ip_local_port_range
This allows more local ports to be available.
Enable fast recycling TIME_WAIT sockets as below:
$ sudo sysctl -w net.ipv4.tcp_tw_recycle=1
By default,
cat /proc/sys/net/ipv4/tcp_tw_recycle
Output : 0 (disabled by default)
Be cautious if enabled in production environments, since this is our
internal Environment and machine used only for Jmeter load tests, we
enabled recycle and resolved the issue.
Enable reuse of sockets as below:
$ sudo sysctl -w net.ipv4.tcp_tw_reuse=1
By default,
cat /proc/sys/net/ipv4/tcp_tw_reuse
Output : 0 (disabled by default)
Note: The tcp_tw_reuse setting is particularly useful in environments
where numerous short connections are open and left in TIME_WAIT state,
such as web servers. Reusing the sockets can be very effective in
reducing server load.
After enabling fast recycling and reuse the server could support 5K
user Load with single Unix box.

Related

Is there a simple way to deploy a Java web app in tomcat on linux on the port 80 ( not 8080) [duplicate]

It's very annoying to have this limitation on my development box, when there won't ever be any users other than me.
I'm aware of the standard workarounds, but none of them do exactly what I want:
authbind (The version in Debian testing, 1.0, only supports IPv4)
Using the iptables REDIRECT target to redirect a low port to a high port (the "nat" table is not yet implemented for ip6tables, the IPv6 version of iptables)
sudo (Running as root is what I'm trying to avoid)
SELinux (or similar). (This is just my dev box, I don't want to introduce a lot of extra complexity.)
Is there some simple sysctl variable to allow non-root processes to bind to "privileged" ports (ports less than 1024) on Linux, or am I just out of luck?
EDIT: In some cases, you can use capabilities to do this.
Okay, thanks to the people who pointed out the capabilities system and CAP_NET_BIND_SERVICE capability. If you have a recent kernel, it is indeed possible to use this to start a service as non-root but bind low ports. The short answer is that you do:
setcap 'cap_net_bind_service=+ep' /path/to/program
And then anytime program is executed thereafter it will have the CAP_NET_BIND_SERVICE capability. setcap is in the debian package libcap2-bin.
Now for the caveats:
You will need at least a 2.6.24 kernel
This won't work if your file is a script. (i.e. uses a #! line to launch an interpreter). In this case, as far I as understand, you'd have to apply the capability to the interpreter executable itself, which of course is a security nightmare, since any program using that interpreter will have the capability. I wasn't able to find any clean, easy way to work around this problem.
Linux will disable LD_LIBRARY_PATH on any program that has elevated privileges like setcap or suid. So if your program uses its own .../lib/, you might have to look into another option like port forwarding.
Resources:
capabilities(7) man page. Read this long and hard if you're going to use capabilities in a production environment. There are some really tricky details of how capabilities are inherited across exec() calls that are detailed here.
setcap man page
"Bind ports below 1024 without root on GNU/Linux": The document that first pointed me towards setcap.
Note: RHEL first added this in v6.
Update 2017:
Use authbind
Disclaimer (update per 2021): Note that authbind works via LD_PRELOAD, which is only used if your program uses libc, which is (or might) not be the case if your program is compiled with GO, or any other compiler that avoids C. If you use go, set the kernel parameter for the protected port range, see bottom of post. </EndUpdate>
Authbind is much better than CAP_NET_BIND_SERVICE or a custom kernel.
CAP_NET_BIND_SERVICE grants trust to the binary but provides no
control over per-port access.
Authbind grants trust to the
user/group and provides control over per-port access, and
supports both IPv4 and IPv6 (IPv6 support has been added as of late).
Install: apt-get install authbind
Configure access to relevant ports, e.g. 80 and 443 for all users and groups:
sudo touch /etc/authbind/byport/80
sudo touch /etc/authbind/byport/443
sudo chmod 777 /etc/authbind/byport/80
sudo chmod 777 /etc/authbind/byport/443
Execute your command via authbind
(optionally specifying --deep or other arguments, see man authbind):
authbind --deep /path/to/binary command line args
e.g.
authbind --deep java -jar SomeServer.jar
As a follow-up to Joshua's fabulous (=not recommended unless you know what you do) recommendation to hack the kernel:
I've first posted it here.
Simple. With a normal or old kernel, you don't.
As pointed out by others, iptables can forward a port.
As also pointed out by others, CAP_NET_BIND_SERVICE can also do the job.
Of course CAP_NET_BIND_SERVICE will fail if you launch your program from a script, unless you set the cap on the shell interpreter, which is pointless, you could just as well run your service as root...
e.g. for Java, you have to apply it to the JAVA JVM
sudo /sbin/setcap 'cap_net_bind_service=ep' /usr/lib/jvm/java-8-openjdk/jre/bin/java
Obviously, that then means any Java program can bind system ports.
Ditto for mono/.NET.
I'm also pretty sure xinetd isn't the best of ideas.
But since both methods are hacks, why not just lift the limit by lifting the restriction ?
Nobody said you have to run a normal kernel, so you can just run your own.
You just download the source for the latest kernel (or the same you currently have).
Afterwards, you go to:
/usr/src/linux-<version_number>/include/net/sock.h:
There you look for this line
/* Sockets 0-1023 can't be bound to unless you are superuser */
#define PROT_SOCK 1024
and change it to
#define PROT_SOCK 0
if you don't want to have an insecure ssh situation, you alter it to this:
#define PROT_SOCK 24
Generally, I'd use the lowest setting that you need, e.g. 79 for http, or 24 when using SMTP on port 25.
That's already all.
Compile the kernel, and install it.
Reboot.
Finished - that stupid limit is GONE, and that also works for scripts.
Here's how you compile a kernel:
https://help.ubuntu.com/community/Kernel/Compile
# You can get the kernel-source via package `linux-source`, no manual download required
apt-get install linux-source fakeroot
mkdir ~/src
cd ~/src
tar xjvf /usr/src/linux-source-<version>.tar.bz2
cd linux-source-<version>
# Apply the changes to PROT_SOCK define in /include/net/sock.h
# Copy the kernel config file you are currently using
cp -vi /boot/config-`uname -r` .config
# Install ncurses libary, if you want to run menuconfig
apt-get install libncurses5 libncurses5-dev
# Run menuconfig (optional)
make menuconfig
# Define the number of threads you wanna use when compiling (should be <number CPU cores> - 1), e.g. for quad-core
export CONCURRENCY_LEVEL=3
# Now compile the custom kernel
fakeroot make-kpkg --initrd --append-to-version=custom kernel-image kernel-headers
# And wait a long long time
cd ..
In a nutshell,
use iptables if you want to stay secure,
compile the kernel if you want to be sure this restriction never bothers you again.
sysctl method
Note:
As of late, updating the kernel is no longer required.
You can now set
sysctl net.ipv4.ip_unprivileged_port_start=80
Or to persist
sysctl -w net.ipv4.ip_unprivileged_port_start=80.
And if that yields an error, simply edit /etc/sysctl.conf with nano and set the parameter there for persistence across reboots.
or via procfs
echo 80 | sudo tee /proc/sys/net/ipv4/ip_unprivileged_port_start
You can do a port redirect. This is what I do for a Silverlight policy server running on a Linux box
iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 943 -j REDIRECT --to-port 1300
For some reason no one mention about lowering sysctl net.ipv4.ip_unprivileged_port_start to the value you need.
Example: We need to bind our app to 443 port.
sysctl net.ipv4.ip_unprivileged_port_start=443
Some may say, there is a potential security problem: unprivileged users now may bind to the other privileged ports (444-1024).
But you can solve this problem easily with iptables, by blocking other ports:
iptables -I INPUT -p tcp --dport 444:1024 -j DROP
iptables -I INPUT -p udp --dport 444:1024 -j DROP
Comparison with other methods. This method:
from some point is (IMO) even more secure than setting CAP_NET_BIND_SERVICE/setuid, since an application doesn't setuid at all, even partly (capabilities actually are).
For example, to catch a coredump of capability-enabled application you will need to change sysctl fs.suid_dumpable (which leads to another potential security problems)
Also, when CAP/suid is set, /proc/PID directory is owned by root, so your non-root user will not have full information/control of running process, for example, user will not be able (in common case) to determine which connections belong to application via /proc/PID/fd/ (netstat -aptn | grep PID).
has security disadvantage: while your app (or any app that uses ports 443-1024) is down for some reason, another app could take the port. But this problem could also be applied to CAP/suid (in case you set it on interpreter, e.g. java/nodejs) and iptables-redirect. Use systemd-socket method to exclude this problem. Use authbind method to only allow special user binding.
doesn't require setting CAP/suid every time you deploy new version of application.
doesn't require application support/modification, like systemd-socket method.
doesn't require kernel rebuild (if running version supports this sysctl setting)
doesn't do LD_PRELOAD like authbind/privbind method, this could potentially affect performance, security, behavior (does it? haven't tested). In the rest authbind is really flexible and secure method.
over-performs iptables REDIRECT/DNAT method, since it doesn't require address translation, connection state tracking, etc. This only noticeable on high-load systems.
Depending on the situation, I would choose between sysctl, CAP, authbind and iptables-redirect. And this is great that we have so many options.
Or patch your kernel and remove the check.
(Option of last resort, not recommended).
In net/ipv4/af_inet.c, remove the two lines that read
if (snum && snum < PROT_SOCK && !capable(CAP_NET_BIND_SERVICE))
goto out;
and the kernel won't check privileged ports anymore.
The standard way is to make them "setuid" so that they start up as root, and then they throw away that root privilege as soon as they've bound to the port but before they start accepting connections to it. You can see good examples of that in the source code for Apache and INN. I'm told that Lighttpd is another good example.
Another example is Postfix, which uses multiple daemons that communicate through pipes, and only one or two of them (which do very little except accept or emit bytes) run as root and the rest run at a lower privilege.
Modern Linux supports /sbin/sysctl -w net.ipv4.ip_unprivileged_port_start=0.
You can setup a local SSH tunnel, eg if you want port 80 to hit your app bound to 3000:
sudo ssh $USERNAME#localhost -L 80:localhost:3000 -N
This has the advantage of working with script servers, and being very simple.
I know this is an old question, but now with recent (>= 4.3) kernels there is finally a good answer to this - ambient capabilities.
The quick answer is to grab a copy of the latest (as-yet-unreleased) version of libcap from git and compile it. Copy the resulting progs/capsh binary somewhere (/usr/local/bin is a good choice). Then, as root, start your program with
/usr/local/bin/capsh --keep=1 --user='your-service-user-name' \
--inh='cap_net_bind_service' --addamb='cap_net_bind_service' \
-- -c 'your-program'
In order, we are
Declaring that when we switch users, we want to keep our current capability sets
Switching user & group to 'your-service-user-name'
Adding the cap_net_bind_service capability to the inherited & ambient sets
Forking bash -c 'your-command' (since capsh automatically starts bash with the arguments after --)
There's a lot going on under the hood here.
Firstly, we are running as root, so by default, we get a full set of capabilities. Included in this is the ability to switch uid & gid with the setuid and setgid syscalls. However, ordinarily when a program does this, it loses its set of capabilities - this is so that the old way of dropping root with setuid still works. The --keep=1 flag tells capsh to issue the prctl(PR_SET_KEEPCAPS) syscall, which disables the dropping of capabilities when changing user. The actual changing of users by capsh happens with the --user flag, which runs setuid and setgid.
The next problem we need to solve is how to set capabilities in a way that carries on after we exec our children. The capabilities system has always had an 'inherited' set of capabilities, which is " a set of capabilities preserved across an execve(2)" [capabilities(7)]. Whilst this sounds like it solves our problem (just set the cap_net_bind_service capability to inherited, right?), this actually only applies for privileged processes - and our process is not privileged anymore, because we already changed user (with the --user flag).
The new ambient capability set works around this problem - it is "a set of capabilities that are preserved across an execve(2) of a program that is not privileged." By putting cap_net_bind_service in the ambient set, when capsh exec's our server program, our program will inherit this capability and be able to bind listeners to low ports.
If you're interested to learn more, the capabilities manual page explains this in great detail. Running capsh through strace is also very informative!
File capabilities are not ideal, because they can break after a package update.
The ideal solution, IMHO, should be an ability to create a shell with inheritable CAP_NET_BIND_SERVICE set.
Here's a somewhat convoluted way to do this:
sg $DAEMONUSER "capsh --keep=1 --uid=`id -u $DAEMONUSER` \
--caps='cap_net_bind_service+pei' -- \
YOUR_COMMAND_GOES_HERE"
capsh utility can be found in libcap2-bin package in Debian/Ubuntu distributions. Here's what goes on:
sg changes effective group ID to that of the daemon user. This is necessary because capsh leaves GID unchanged and we definitely do not want it.
Sets bit 'keep capabilities on UID change'.
Changes UID to $DAEMONUSER
Drops all caps (at this moment all caps are still present because of --keep=1), except inheritable cap_net_bind_service
Executes your command ('--' is a separator)
The result is a process with specified user and group, and cap_net_bind_service privileges.
As an example, a line from ejabberd startup script:
sg $EJABBERDUSER "capsh --keep=1 --uid=`id -u $EJABBERDUSER` --caps='cap_net_bind_service+pei' -- $EJABBERD --noshell -detached"
Two other simple possibilities: Daemon and Proxy
Daemon
There is an old (unfashionable) solution to the "a daemon that binds on a low port and hands control to your daemon". It's called inetd (or xinetd).
The cons are:
your daemon needs to talk on stdin/stdout (if you don't control the daemon -- if you don't have the source -- then this is perhaps a showstopper, although some services may have an inetd-compatibility flag)
a new daemon process is forked for every connection
it's one extra link in the chain
Pros:
available on any old UNIX
once your sysadmin has set up the config, you're good to go about your development (when you re-build your daemon, might you lose setcap capabilities? And then you'll have to go back to your admin "please sir...")
daemon doesn't have to worry about that networking stuff, just has to talk on stdin/stdout
can configure to execute your daemon as a non-root user, as requested
Proxy
Another alternative: a hacked-up proxy (netcat or even something more robust) from the privileged port to some arbitrary high-numbered port where you can run your target daemon. (Netcat is obviously not a production solution, but "just my dev box", right?). This way you could continue to use a network-capable version of your server, would only need root/sudo to start proxy (at boot), wouldn't be relying on complex/potentially fragile capabilities.
My "standard workaround" uses socat as the user-space redirector:
socat tcp6-listen:80,fork tcp6:8080
Beware that this won't scale, forking is expensive but it's the way socat works.
Linux supports capabilities to support more fine-grained permissions than just "this application is run as root". One of those capabilities is CAP_NET_BIND_SERVICE which is about binding to a privileged port (<1024).
Unfortunately I don't know how to exploit that to run an application as non-root while still giving it CAP_NET_BIND_SERVICE (probably using setcap, but there's bound to be an existing solution for this).
systemd is a sysvinit replacement which has an option to launch a daemon with specific capabilities. Options Capabilities=, CapabilityBoundingSet= in systemd.exec(5) manpage.
TLDR: For "the answer" (as I see it), jump down to the >>TLDR<< part in this answer.
OK, I've figured it out (for real this time), the answer to this question, and this answer of mine is also a way of apologizing for promoting another answer (both here and on twitter) that I thought was "the best", but after trying it, discovered that I was mistaken about that. Learn from my mistake kids: don't promote something until you've actually tried it yourself!
Again, I reviewed all the answers here. I've tried some of them (and chose not to try others because I simply didn't like the solutions). I thought that the solution was to use systemd with its Capabilities= and CapabilitiesBindingSet= settings. After wrestling with this for some time, I discovered that this is not the solution because:
Capabilities are intended to restrict root processes!
As the OP wisely stated, it is always best to avoid that (for all your daemons if possible!).
You cannot use the Capabilities related options with User= and Group= in systemd unit files, because capabilities are ALWAYS reset when execev (or whatever the function is) is called. In other words, when systemd forks and drops its perms, the capabilities are reset. There is no way around this, and all that binding logic in the kernel is basic around uid=0, not capabilities. This means that it is unlikely that Capabilities will ever be the right answer to this question (at least any time soon). Incidentally, setcap, as others have mentioned, is not a solution. It didn't work for me, it doesn't work nicely with scripts, and those are reset anyways whenever the file changes.
In my meager defense, I did state (in the comment I've now deleted), that James' iptables suggestion (which the OP also mentions), was the "2nd best solution". :-P
>>TLDR<<
The solution is to combine systemd with on-the-fly iptables commands, like this (taken from DNSChain):
[Unit]
Description=dnschain
After=network.target
Wants=namecoin.service
[Service]
ExecStart=/usr/local/bin/dnschain
Environment=DNSCHAIN_SYSD_VER=0.0.1
PermissionsStartOnly=true
ExecStartPre=/sbin/sysctl -w net.ipv4.ip_forward=1
ExecStartPre=-/sbin/iptables -D INPUT -p udp --dport 5333 -j ACCEPT
ExecStartPre=-/sbin/iptables -t nat -D PREROUTING -p udp --dport 53 -j REDIRECT --to-ports 5333
ExecStartPre=/sbin/iptables -A INPUT -p udp --dport 5333 -j ACCEPT
ExecStartPre=/sbin/iptables -t nat -A PREROUTING -p udp --dport 53 -j REDIRECT --to-ports 5333
ExecStopPost=/sbin/iptables -D INPUT -p udp --dport 5333 -j ACCEPT
ExecStopPost=/sbin/iptables -t nat -D PREROUTING -p udp --dport 53 -j REDIRECT --to-ports 5333
User=dns
Group=dns
Restart=always
RestartSec=5
WorkingDirectory=/home/dns
PrivateTmp=true
NoNewPrivileges=true
ReadOnlyDirectories=/etc
# Unfortunately, capabilities are basically worthless because they're designed to restrict root daemons. Instead, we use iptables to listen on privileged ports.
# Capabilities=cap_net_bind_service+pei
# SecureBits=keep-caps
[Install]
WantedBy=multi-user.target
Here we accomplish the following:
The daemon listens on 5333, but connections are successfully accepted on 53 thanks to iptables
We can include the commands in the unit file itself, and thus we save people headaches. systemd cleans up the firewall rules for us, making sure to remove them when the daemon isn't running.
We never run as root, and we make privilege escalation impossible (at least systemd claims to), supposedly even if the daemon is compromised and sets uid=0.
iptables is still, unfortunately, quite an ugly and difficult-to-use utility. If the daemon is listening on eth0:0 instead of eth0, for example, the commands are slightly different.
With systemd, you just need to slightly modify your service to accept preactivated sockets.
You can later use systemd socket activate.
No capabilities, iptables or other tricks are needed.
This is content of relevant systemd files from this example of simple python http server
File httpd-true.service
[Unit]
Description=Httpd true
[Service]
ExecStart=/usr/local/bin/httpd-true
User=subsonic
PrivateTmp=yes
File httpd-true.socket
[Unit]
Description=HTTPD true
[Socket]
ListenStream=80
[Install]
WantedBy=default.target
Port redirect made the most sense for us, but we ran into an issue where our application would resolve a url locally that also needed to be re-routed; (that means you shindig).
This will also allow you to be redirected when accessing the url on the local machine.
iptables -A PREROUTING -t nat -p tcp --dport 80 -j REDIRECT --to-port 8080
iptables -A OUTPUT -t nat -p tcp --dport 80 -j REDIRECT --to-port 8080
At startup:
iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080
Then you can bind to the port you forward to.
There is also the 'djb way'. You can use this method to start your process as root running on any port under tcpserver, then it will hand control of the process to the user you specify immediately after the process starts.
#!/bin/sh
UID=$(id -u username)
GID=$(id -g username)
exec tcpserver -u "${UID}" -g "${GID}" -RHl0 0 port /path/to/binary &
For more info, see: http://thedjbway.b0llix.net/daemontools/uidgid.html
Use the privbind utility: it allows an unprivileged application to bind to reserved ports.
Bind port 8080 to 80 and open port 80:
sudo iptables -t nat -A OUTPUT -o lo -p tcp --dport 80 -j REDIRECT --to-port 8080
sudo iptables -A INPUT -p tcp --dport 80 -j ACCEPT
and then run program on port 8080 as a normal user.
you will then be able to access http://127.0.0.1 on port 80
I tried the iptables PREROUTING REDIRECT method. In older kernels it seems this type of rule wasn't supported for IPv6. But apparently it is now supported in ip6tables v1.4.18 and Linux kernel v3.8.
I also found that PREROUTING REDIRECT doesn't work for connections initiated within the machine. To work for conections from the local machine, add an OUTPUT rule also — see iptables port redirect not working for localhost. E.g. something like:
iptables -t nat -I OUTPUT -o lo -p tcp --dport 80 -j REDIRECT --to-port 8080
I also found that PREROUTING REDIRECT also affects forwarded packets. That is, if the machine is also forwarding packets between interfaces (e.g. if it's acting as a Wi-Fi access point connected to an Ethernet network), then the iptables rule will also catch connected clients' connections to Internet destinations, and redirect them to the machine. That's not what I wanted—I only wanted to redirect connections that were directed to the machine itself. I found I can make it only affect packets addressed to the box, by adding -m addrtype --dst-type LOCAL. E.g. something like:
iptables -A PREROUTING -t nat -p tcp --dport 80 -m addrtype --dst-type LOCAL -j REDIRECT --to-port 8080
One other possibility is to use TCP port forwarding. E.g. using socat:
socat TCP4-LISTEN:www,reuseaddr,fork TCP4:localhost:8080
However one disadvantage with that method is, the application that is listening on port 8080 then doesn't know the source address of incoming connections (e.g. for logging or other identification purposes).
Since the OP is just development/testing, less than sleek solutions may be helpful:
setcap can be used on a script's interpreter to grant capabilities to scripts. If setcaps on the global interpreter binary is not acceptable, make a local copy of the binary (any user can) and get root to setcap on this copy. Python2 (at least) works properly with a local copy of the interpreter in your script development tree. No suid is needed so the root user can control to what capabilities users have access.
If you need to track system-wide updates to the interpreter, use a shell script like the following to run your script:
#!/bin/sh
#
# Watch for updates to the Python2 interpreter
PRG=python_net_raw
PRG_ORIG=/usr/bin/python2.7
cmp $PRG_ORIG $PRG || {
echo ""
echo "***** $PRG_ORIG has been updated *****"
echo "Run the following commands to refresh $PRG:"
echo ""
echo " $ cp $PRG_ORIG $PRG"
echo " # setcap cap_net_raw+ep $PRG"
echo ""
exit
}
./$PRG $*
Answer at 2015/Sep:
ip6tables now supports IPV6 NAT: http://www.netfilter.org/projects/iptables/files/changes-iptables-1.4.17.txt
You will need kernel 3.7+
Proof:
[09:09:23] root#X:~ ip6tables -t nat -vnL
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 REDIRECT tcp eth0 * ::/0 ::/0 tcp dpt:80 redir ports 8080
0 0 REDIRECT tcp eth0 * ::/0 ::/0 tcp dpt:443 redir ports 1443
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 6148 packets, 534K bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 6148 packets, 534K bytes)
pkts bytes target prot opt in out source destination
There is a worked example of doing this with a file capable shared library linked to an unprivileged application on the libcap website. It was recently mentioned in an answer to a question about adding capabilities to shared libraries.

Your Amazon EC2 Abuse Report

I created amazon ec2 instance to run Tomcat application, but
Amazon abuse team sent mail with the following log
<<<
AWS Account: ********
Report begin time: 14-12-2017 02:02:28 UTC
Report end time: 14-12-2017 02:03:28 UTC
Protocol: TCP
Remote IP: ...
Remote port(s): 80
Total bytes sent: 294167550
Total packets sent: 291255
Total bytes received: 0
Total packets received: 0
AWS Account: ********
Report begin time: 14-12-2017 02:03:15 UTC
Report end time: 14-12-2017 02:04:15 UTC
Protocol: TCP
Remote IP: ...
Remote port(s): 80
Total bytes sent: 1050081850
Total packets sent: 1039685
Total bytes received: 0
Total packets received: 0
Is it application issue or have I missed some security configuration?
We found one unknown executable application in /tmp as below, after killed this again came up after few seconds. Seems its generating unknown traffic, so we go ahead and powered off servers now.
[root#ip-172-19-24-90 tmp]# file Lixsyn
Lixsyn: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.2.5, not stripped
[root#ip-172-19-24-90 tmp]#
raw 104832 426240 0.0.0.0:6 0.0.0.0:* 7 19719/Lixsyn
raw 104832 228096 0.0.0.0:6 0.0.0.0:* 7 19719/Lixsyn
root#ip-172-31-40-123 tmp]# ifconfig
eth0 Link encap:Ethernet HWaddr ********
inet addr:******** Bcast:172.31.47.255 Mask:255.255.240.0
inet6 addr: ********/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9001 Metric:1
RX packets:1136962 errors:0 dropped:0 overruns:0 frame:0
TX packets:2081358186 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:860855089 (820.9 MiB) TX bytes:2130697820190 (1.9 TiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:11338 errors:0 dropped:0 overruns:0 frame:0
TX packets:11338 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:21689998 (20.6 MiB) TX bytes:21689998 (20.6 MiB)
AWS sends abuse reports when ec2 instances trigger unknown traffic to servers outside your account.
This can be due to some unknown applications which got inside your ec2 instances due to publicly open ssh/rdp ports.
What you can do here is
Configure the security group of your ec2 instances to allow ssh access (port 22) with source ip as your own homr/office ip only
Do a netstat command to find what all process ids are accessing the remote ip and port numbers mentioned in thw abuse reports
Once you get the process ids, find all processes associated to it using ps -ef command and remove all the related paths
Change all user credentials and disable ssh login without password.
Answer to your second question on preventing attacks tomcat servers on ec2.
Avoid exposing tomcat manager app on default url/path
Use "name" property in server.xml to avoid exposing the.
container name and version through http headers
Better to put tomcat behind a web layer/proxy like nginx/haproxy to limit impact of bruteforce attacks
Use very strong passwords for tomcat users.
"Lixsyn" Looks like a linux backdoor virus. Has nothing to do with app Linux.BackDoor.Gates
Takashi Katsuki, a researcher at Antivirus firm Symantec has discovered a new cyber attack ongoing in the wild, targeting an open-source Web server application server Apache Tomcat with a cross platform Java based backdoor that can be used to attack other machines.
The malware, dubbed as "Java.Tomdep" differs from other server malware and is not written in the PHP scripting language. It is basically a Java based backdoor act as Java Servlet that gives Apache Tomcat platforms malicious capabilities.
Because Java is a cross platform language, the affected platforms include Linux, Mac OS X, Solaris, and most supported versions of Windows. The malware was detected less than a month ago and so far the number of infected machines appears to be low.
You may think that this type of attack only targets personal
computers, such as desktops and laptops, but unfortunately that isn’t
true. Servers can also be attacked. They are quite valuable targets,
since they are usually high-performance computers and run 24x7.
Java worm seeks out for the system having Apache Tomcat installed-running and then attempts to log-in using the password brute-force attack using combinations of user names and passwords.
After installation, the malware servlet behaves like an IRC Bot and able to receive commands from an attacker. Malware is capable of sending-downloading files from the system, create new processes, update itself, can setup SOCKS proxy, UDP flooding i.e. Can perform massive DDoS Attack.
They have mentioned that the command-and-control servers have been traced to Taiwan and Luxembourg. In order to avoid this threat, ensure that your server and AV products are fully patched and updated.
Please Add your suggestions....
For those who are new to AWS that didn't have a team to read all the details, here is the reminder : all data transfer out from your EC2 to the internet will cost you money after the first free 1 GB.
Data Transfer OUT From Amazon EC2 To Internet
First 1 GB / month $0.00 per GB
Up to 10 TB / month $0.09 per GB
Next 40 TB / month $0.085 per GB
Since EC2s are not mean to be used as web hosting service, it is timely that AWS start sending abuse email to the admin when there is massive data download OUT from EC2 to internet.
You should check whether this is the intended traffics usage of your Tomcat webapps.
If you don't anticipate the traffics e.g. there is nobody connect to it from outside the internet, check the website for possible compromised, e.g. check your Security rules, establish a connection to only trust particular IP range than to everyone in the internet.
If you anticipate the traffics, e.g. there is multiple user connect to your apps and generate lots of traffics, relook your bandwidth needs and start consider CDN(content delivery network) to relieve the required load. Because EC2 internet outgoing traffics is not cheap, since it is not intends for content intensive download.
Bare in mind that, if you don't move this traffics to a CDN, a DDoS will cause AWS to send you a massive traffic bills.
(update)
As point out by above #Abhijit Jagtap , your server might have been compromised. perhaps at the moment you open up the server to the Internet. A botnet can easily scan through possible web services version and perform the attack. It is highly probably your Web services(Tomcat, jre/jdk, web server etc ) version contain some vulnerabilities.
You should recreate the instance, perform all hardening measurement. Please don't bother to "clean up" , it is just wasting time. If you want to make some forensic later, you can create a snapshot for the compromised EC2 instance.

Profiling tomcat application with VisualVM

I am trying to run visualvm under the username tomcat6 because apparently visualvm can only find applications running under its username. So by default it is only finding applications running under my username. I have been able to connect visualvm with tomcat6 through jmx but that lacks the fine granularity of instrumented profiling.
I tried the following to run visualvm under the username tomcat6 but got the following error that I don't understand.
$ sudo -u tomcat6 jvisualvm
No protocol specified
Exception in thread "main" java.awt.AWTError: Can't connect to X11 window server using ':0' as the value of the DISPLAY variable.
at sun.awt.X11GraphicsEnvironment.initDisplay(Native Method)
at sun.awt.X11GraphicsEnvironment.access$200(X11GraphicsEnvironment.java:65)
at sun.awt.X11GraphicsEnvironment$1.run(X11GraphicsEnvironment.java:115)
...
If the computer running your application is remote -- like a server -- then you can't run GUI applications without some work. It's probably going to be easier to enable remote access to VisualVM.
You can use two techniques to attach to a remote JVM: using jstatd or using JMX. I'm not sure what you think you are losing by using JMX, but evidently jstatd doesn't give you access to profiling tools, CPU monitor, etc.).
You need to configure your JVM and Tomcat to allow for remote access. That requires 3 steps:
Enable remote JMX. Turns out, there's a guide for that.
Fix the "wandering port" used for RMI. There's a guide for that, too.
(Optional) Arrange for secure remote-access to the server. The easiest way to do that would be to use ssh -Lport:localhost:port with a series of -L arguments to forward multiple ports from your workstation to your server. Map all the ports you had to configure in steps #1 and #2. If you don't do this, you'll need to have non-firewalled access to all the aforementioned ports.
Restart your JVM and connect with JVisualVM.
Update 2022-06-01
Note that the "wandering port" problem has been fixed at the JVM level, so there is no need for application (i.e. Tomcat) support for that. Item #1 for Tomcat 8.5 and later contains updated instructions making item #2 unnecessary with a recent JVM.
Unfortunately only sampling is available in remote mode so JMX will lack instrumentation tools.
Actually your approach to running visualvm under tomcat6 user is correct. You should take a look at this question on how to run X11 applications under sudo.
The easiest way to pass DISPLAY and XAUTHORITY environment variables is to use sudo -E command to preserve current user environment.
Also if you can't see your process under tomcat6 user you should check if CATALINA_TMPDIR is pointing to /tmp. Otherwise you should pass it to visualvm
jvisualvm -J-Djava.io.tmpdir="${CATALINA_TMPDIR}"
Actually there is a lot of alternatives like yourkit or jprofiler shipped with java agents which allows remote instrumentation profiling.
The easiest is to open a remote JXM port on Tomcat in order to be able to remotely (from your desktop computer) connect to your remote Tomcat (on your server) with jvisualvm.
You need to pass the following system properties to your JVM :
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=<whatever_port_you_want>
-Dcom.sun.management.jmxremote.ssl=false
Then open jvisualvm on your local computer (JVM version needs to be the same or newer), File -> Add Remote Host -> Enter the name on the Host. It will create an entry for this host. Right lick on this Entry -> Add JMX connection -> Enter the port -> OK
jvisualvm will then be able to access remotely to your application.
You can also secure the connection if needed by using the following system properties (you need to create the files and locate them where you want :
-Dcom.sun.management.jmxremote.password.file=jmxremote.password
-Dcom.sun.management.jmxremote.access.file=jmxremote.access
These properties needs to be added to the CATALINA_OPTS environment variable. Fr exemple :
export CATALINA_OPTS = "$CATALINA_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8888 "
More info on JMX lies here : https://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html
I tried to do something similar but I was not allowed to install JVisualVM on the server. Having JVisualVM connect to the remote machine never seemed to work correctly. I suspect firewall rules were blocking part of the the network connections.
The only way I found to remotely profile the server was via an ssh tunnel.
Set the JMX port in CATALINA_OPTS on the server
CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote.port=13333 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false; export CATALINA_OPTS
On your desktop machine open an ssh connection to the server
ssh -D 61444 username#tomcat.server.address
Add a flag to JVisualVM so that it will proxy its network connection
"C:\Program Files\Java\jdk1.7.0_79\bin\jvisualvm.exe" -J-Dnetbeans.system_socks_proxy=localhost:61444 -J-Djava.net.useSystemProxies=true
Have JVisualVM connect to the jmxport and the network traffic is tunneled via ssh.
Good luck.

Apache multiple requests with jmeter

I am using Jmeter to test multiple requests to my web application.
I used NumberOfThread in Jmeter as 50.
My process is as follows:
Login page.
Login with userID and password.
Show menu page.
Click search page.
Go to search page.
Click search button.
Click the searched result link to go to update page.
Update data and click update button.
Show the updated result page.
Go back to search page.
Log out button click.
In the above process, I used looping controller for process number 5 to 10 with 5 looping.
In that situation, if I used more than 25 thread to run Jmeter test, address already in use, the socket binding exception has occur.
I would like to know how to solve that problem.
Looks like your client ran out of ephemeral port or there's some problem with your client environment.
Are you using Windows?
You can possibly do at least the following:
Windows: look into this article for solution for Windows system as host for jmeter.
Use Linux system instead as host to run you Jmeter load-scenarios.
As well you possibly will find this article useful for your testing activities (I've seen Jboss in tags).
UPDATE:
Once more from linked article above:
When an HTTP request is made, an ephemeral port is allocated for the
TCP / IP connection. The ephemeral port range is 32678 – 61000. After
the client closes the connection, the connection is placed in the
TIME-WAIT state for 60 seconds.
If JMeter (HttpClient) is sending thousands of HTTP requests per
second and creating new TCP / IP connections, the system will run out
of available ephemeral ports for allocation.
. . .
Otherwise, the following messages may appear in the JMeter JTL files:
Non HTTP response code: java.net.BindException
Non HTTP response message: Address already in use
The solution is to enable fast recycling TIME_WAIT sockets.
echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
Other options include TCP_FIN_TIMEOUT to reduce how long a connection
is placed in the TIME_WAIT state and TCP_TW_REUSE to allow the system
to reuse connections placed in the TIME_WAIT state.
On the server's side:
This enables fast recycling of TIME_WAIT sockets
/sbin/sysctl -w net.ipv4.tcp_tw_recycle=1
This allows reusing sockets in TIME_WAIT state for new connections - a safer alternative to tcp_tw_recycle
/sbin/sysctl -w net.ipv4.tcp_tw_reuse=1
The tcp_tw_reuse setting is particularly useful in environments where numerous short connections are open and left in TIME_WAIT state, such as web-servers. Reusing the sockets can be very effective in reducing server load.
Maximum number of timewait sockets held by system simultaneously
/sbin/sysctl -w net.ipv4.tcp_max_tw_buckets=30000
or the same but in another way - add the lines below to the /etc/sysctl.conf file so that the change survives reboot:
net.ipv4.tcp_max_tw_buckets = 30000
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
As well on the server's side look onto result of ulimit -n.
Default value is 1024 for the limit of max open files, which can explain appearance of BindExceptions at 1000 connections.
As well you can monitor number of connections between the server and jmeter during test-run using e.g.
netstat -an | grep SERVER_PORT | wc -l
to define limit of connections - if any.

Problem running into java.net.bindexception cannot assign requested address

i am currently testing a server with an automatic test client that simulates a large number of users. Both the server and the client are written in Java. The client opens a tcp/ip connection for every user. Both the server and client run on Ubuntu linux, client runs on 11.04 and server on 10.04.
The testing went good up till 27000 concurrently open connections, after that i decided to jump to 36000 (the servers and clients resources weren't really all that used up at 27000 so i decided to make a slightly bigger jump). When i tried running the test for 36k i got the following exception on the client side:
java.net.BindException: cannot assign requested address
As far as i know at 36k i should still have free ports since not much else is running on either machine and tcp limits the port number at 2^16 which is 65536. Now since it is linux i also set the number of open files for the user to 100k with ulimit -n 100000.
But i am still getting the same exception.
I'm wondering what else could be a possible cause for the mentioned exception, or does linux in some other way limit the number of outgoing connections ?
Thanks in advance,
Danijel
By default, Linux picks dynamically assigned ports from the range 32768..61000. The others are available for static assignment, if you bind to a specific port number. The range can be changed if you want more of the ports to be available for dynamic assignment, but just be careful that you do not include ports that are used for specific services that you need (e.g. 6000 for X11). Also you should not allow ports < 1024 to be dynamically assigned since they are privileged. To check or change the range:
$ cat /proc/sys/net/ipv4/ip_local_port_range
32768 61000
# echo "16384 65535" > /proc/sys/net/ipv4/ip_local_port_range

Categories