I created amazon ec2 instance to run Tomcat application, but
Amazon abuse team sent mail with the following log
<<<
AWS Account: ********
Report begin time: 14-12-2017 02:02:28 UTC
Report end time: 14-12-2017 02:03:28 UTC
Protocol: TCP
Remote IP: ...
Remote port(s): 80
Total bytes sent: 294167550
Total packets sent: 291255
Total bytes received: 0
Total packets received: 0
AWS Account: ********
Report begin time: 14-12-2017 02:03:15 UTC
Report end time: 14-12-2017 02:04:15 UTC
Protocol: TCP
Remote IP: ...
Remote port(s): 80
Total bytes sent: 1050081850
Total packets sent: 1039685
Total bytes received: 0
Total packets received: 0
Is it application issue or have I missed some security configuration?
We found one unknown executable application in /tmp as below, after killed this again came up after few seconds. Seems its generating unknown traffic, so we go ahead and powered off servers now.
[root#ip-172-19-24-90 tmp]# file Lixsyn
Lixsyn: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.2.5, not stripped
[root#ip-172-19-24-90 tmp]#
raw 104832 426240 0.0.0.0:6 0.0.0.0:* 7 19719/Lixsyn
raw 104832 228096 0.0.0.0:6 0.0.0.0:* 7 19719/Lixsyn
root#ip-172-31-40-123 tmp]# ifconfig
eth0 Link encap:Ethernet HWaddr ********
inet addr:******** Bcast:172.31.47.255 Mask:255.255.240.0
inet6 addr: ********/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9001 Metric:1
RX packets:1136962 errors:0 dropped:0 overruns:0 frame:0
TX packets:2081358186 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:860855089 (820.9 MiB) TX bytes:2130697820190 (1.9 TiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:11338 errors:0 dropped:0 overruns:0 frame:0
TX packets:11338 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:21689998 (20.6 MiB) TX bytes:21689998 (20.6 MiB)
AWS sends abuse reports when ec2 instances trigger unknown traffic to servers outside your account.
This can be due to some unknown applications which got inside your ec2 instances due to publicly open ssh/rdp ports.
What you can do here is
Configure the security group of your ec2 instances to allow ssh access (port 22) with source ip as your own homr/office ip only
Do a netstat command to find what all process ids are accessing the remote ip and port numbers mentioned in thw abuse reports
Once you get the process ids, find all processes associated to it using ps -ef command and remove all the related paths
Change all user credentials and disable ssh login without password.
Answer to your second question on preventing attacks tomcat servers on ec2.
Avoid exposing tomcat manager app on default url/path
Use "name" property in server.xml to avoid exposing the.
container name and version through http headers
Better to put tomcat behind a web layer/proxy like nginx/haproxy to limit impact of bruteforce attacks
Use very strong passwords for tomcat users.
"Lixsyn" Looks like a linux backdoor virus. Has nothing to do with app Linux.BackDoor.Gates
Takashi Katsuki, a researcher at Antivirus firm Symantec has discovered a new cyber attack ongoing in the wild, targeting an open-source Web server application server Apache Tomcat with a cross platform Java based backdoor that can be used to attack other machines.
The malware, dubbed as "Java.Tomdep" differs from other server malware and is not written in the PHP scripting language. It is basically a Java based backdoor act as Java Servlet that gives Apache Tomcat platforms malicious capabilities.
Because Java is a cross platform language, the affected platforms include Linux, Mac OS X, Solaris, and most supported versions of Windows. The malware was detected less than a month ago and so far the number of infected machines appears to be low.
You may think that this type of attack only targets personal
computers, such as desktops and laptops, but unfortunately that isn’t
true. Servers can also be attacked. They are quite valuable targets,
since they are usually high-performance computers and run 24x7.
Java worm seeks out for the system having Apache Tomcat installed-running and then attempts to log-in using the password brute-force attack using combinations of user names and passwords.
After installation, the malware servlet behaves like an IRC Bot and able to receive commands from an attacker. Malware is capable of sending-downloading files from the system, create new processes, update itself, can setup SOCKS proxy, UDP flooding i.e. Can perform massive DDoS Attack.
They have mentioned that the command-and-control servers have been traced to Taiwan and Luxembourg. In order to avoid this threat, ensure that your server and AV products are fully patched and updated.
Please Add your suggestions....
For those who are new to AWS that didn't have a team to read all the details, here is the reminder : all data transfer out from your EC2 to the internet will cost you money after the first free 1 GB.
Data Transfer OUT From Amazon EC2 To Internet
First 1 GB / month $0.00 per GB
Up to 10 TB / month $0.09 per GB
Next 40 TB / month $0.085 per GB
Since EC2s are not mean to be used as web hosting service, it is timely that AWS start sending abuse email to the admin when there is massive data download OUT from EC2 to internet.
You should check whether this is the intended traffics usage of your Tomcat webapps.
If you don't anticipate the traffics e.g. there is nobody connect to it from outside the internet, check the website for possible compromised, e.g. check your Security rules, establish a connection to only trust particular IP range than to everyone in the internet.
If you anticipate the traffics, e.g. there is multiple user connect to your apps and generate lots of traffics, relook your bandwidth needs and start consider CDN(content delivery network) to relieve the required load. Because EC2 internet outgoing traffics is not cheap, since it is not intends for content intensive download.
Bare in mind that, if you don't move this traffics to a CDN, a DDoS will cause AWS to send you a massive traffic bills.
(update)
As point out by above #Abhijit Jagtap , your server might have been compromised. perhaps at the moment you open up the server to the Internet. A botnet can easily scan through possible web services version and perform the attack. It is highly probably your Web services(Tomcat, jre/jdk, web server etc ) version contain some vulnerabilities.
You should recreate the instance, perform all hardening measurement. Please don't bother to "clean up" , it is just wasting time. If you want to make some forensic later, you can create a snapshot for the compromised EC2 instance.
Related
I have created a server in Java with socket and a client that connects to the server. This works between two pc on the same wifi, how can i make it work on two different wifi (two pc in two different houses)?
P.S. sorry my bad english.
You have to make your server available publicly to be able to make connection from other client (connected from some other wifi network)
One possible solution is to use ngrok which will make it possible to map your local server to publicly accessible url. This server is absolutely free to use and I have personally used it multiple times for publicly exposing my local servers.
Edit 1 -
Let say your socket server is running on port 6666, you need to run following command -
./ngrok tcp 6666
this will result in following response
Session Status online
Account ********#gmail.com (Plan: Free)
Version 2.3.35
Region United States (us)
Web Interface http://127.0.0.1:4040
Forwarding tcp://0.tcp.ngrok.io:15341 -> localhost:6666
Connections ttl opn rt1 rt5 p50 p90
0 0 0.00 0.00 0.00 0.00
after this, you will be able to connect you server using following command from other java client -
socket = new Socket("0.tcp.ngrok.io", 15341);
I've a question which may seem strange, but I'm working in a environment which has very restricted options.
Basically, I've a job which runs on a SAP Netweaver server, which is clustered.
This job runs socket server code, which allows an ancient system to communicate with it.
My question is this:
Depending on which side of the cluster the job runs on (and I can't influence this), the sock server will either run on a .127 IP or a .129 IP.
Since the connecting system needs a fixed IP to connect to, It gives me a problem.
So, can I open the socket on the .127 IP each time, regardless of which of the two IP the Job happens to be running on, or does it have to be opened on the same IP that the code is actually running on ?
Well, if you can't influence the server, you can introduce a component between the ancient system and the cluster that will redirect the request to one of the IPs in the cluster (.127 / .129 in your example).
|-> [IP .127]
[ancient system] --> [load balancing/proxy component] |
|-> [IP .129]
An actual implementation can vary, basically it boils down to hardware based solution or software based solution.
Hardware
Some network equipment, like load balancers provide this feature, so talk to your network department about this, they'll provide a couple of options.
Software
You can install solutions like "ha-proxy" that will solve this at the level of software
I have created a simple Spring boot Application having a HelloController.
Get API: http://localhost:8080/hello
Response: Hello World
Now I have created a JMeter Script having 0.1 million concurrent users hitting the above get API.
When I run the JMeter Script, after 30k count, I start getting the error:
java.net.NoRouteToHostException: Cannot assign requested address (Address not available)
What is the reason for this? How can I resolve this issue?
I'm using UBUNTU 18.04 with 8gb RAM.
While performing the operation, only JMeter and STS was open.
You can follow Lakshmi Narayan answer to increase available ports:
Resolution:
Increased the local port range using below command:
echo 1024 65000 > /proc/sys/net/ipv4/ip_local_port_range
This allows more local ports to be available.
Enable fast recycling TIME_WAIT sockets as below:
$ sudo sysctl -w net.ipv4.tcp_tw_recycle=1
By default,
cat /proc/sys/net/ipv4/tcp_tw_recycle
Output : 0 (disabled by default)
Be cautious if enabled in production environments, since this is our
internal Environment and machine used only for Jmeter load tests, we
enabled recycle and resolved the issue.
Enable reuse of sockets as below:
$ sudo sysctl -w net.ipv4.tcp_tw_reuse=1
By default,
cat /proc/sys/net/ipv4/tcp_tw_reuse
Output : 0 (disabled by default)
Note: The tcp_tw_reuse setting is particularly useful in environments
where numerous short connections are open and left in TIME_WAIT state,
such as web servers. Reusing the sockets can be very effective in
reducing server load.
After enabling fast recycling and reuse the server could support 5K
user Load with single Unix box.
I am using an application called Splunk, which has 2 layers of data processing on separate systems. I can connect to both systems via putty and directly logging into the GUI.
Archiecture of the system attached
I want to monitor the output port on layer 1, and input port on layer 2 and find out the below information,
i. Raw data sent out of layer 1
ii. Raw data received into layer 2.
The two layers communicate using tcp.
I do not want to use an existing packet monitors/packet capture like tcpdump or winshark as I want to heavily customize the monitor to display various information. I want to write my own packet monitor in java
I want to know,
If layer 2 is listening to layer 1, can my program connect to the same port and print the raw text being sent? - the protocol is TCP
Is (1) possible for other protocols like udp, http and ftp?
(EDIT: Architecture attached
Info on the systems in case it is relevant,
system 1 is VMWare(Linux Ubuntu 4.4.10) running on a Windows machine.
System 2 is mac os.
Both systems have different IP addresses. I am connecting to both systems from the windows machine where vmware is running.
The two systems connect to the same wifi.
)
There are some packet capture libraries written in Java, jpcap, jNetPcap, and Pcap4J.
Both #1 and #2 are possible by these libraries, I think.
If you would pick Pcap4J I can help you. It's my library.
i am currently testing a server with an automatic test client that simulates a large number of users. Both the server and the client are written in Java. The client opens a tcp/ip connection for every user. Both the server and client run on Ubuntu linux, client runs on 11.04 and server on 10.04.
The testing went good up till 27000 concurrently open connections, after that i decided to jump to 36000 (the servers and clients resources weren't really all that used up at 27000 so i decided to make a slightly bigger jump). When i tried running the test for 36k i got the following exception on the client side:
java.net.BindException: cannot assign requested address
As far as i know at 36k i should still have free ports since not much else is running on either machine and tcp limits the port number at 2^16 which is 65536. Now since it is linux i also set the number of open files for the user to 100k with ulimit -n 100000.
But i am still getting the same exception.
I'm wondering what else could be a possible cause for the mentioned exception, or does linux in some other way limit the number of outgoing connections ?
Thanks in advance,
Danijel
By default, Linux picks dynamically assigned ports from the range 32768..61000. The others are available for static assignment, if you bind to a specific port number. The range can be changed if you want more of the ports to be available for dynamic assignment, but just be careful that you do not include ports that are used for specific services that you need (e.g. 6000 for X11). Also you should not allow ports < 1024 to be dynamically assigned since they are privileged. To check or change the range:
$ cat /proc/sys/net/ipv4/ip_local_port_range
32768 61000
# echo "16384 65535" > /proc/sys/net/ipv4/ip_local_port_range