We Keep Coding

DefaultHttpClient call throws connection refused in the same tomcat with public ip

centos 7, tomcat 8.5.
a.war and rest.war are in the same tomcat.
a.war use following code to call rest.war:
import org.apache.http.impl.client.DefaultHttpClient;
DefaultHttpClient httpClient = new DefaultHttpClient();
HttpPost httpPost = new HttpPost(url);
httpPost.addHeader(HTTP.CONTENT_TYPE, "application/json");
StringEntity se = new StringEntity(json.toString());
se.setContentType("text/json");
se.setContentEncoding(new BasicHeader(HTTP.CONTENT_TYPE, "application/json"));
httpPost.setEntity(se);
HttpResponse response = httpClient.execute(httpPost);
however, if url of HttpPost(url) is <public ip>:80, then httpClient.execute(httpPost) will throw connection refused.
while if url of HttpPost(url) is localhost:80 or 127.0.0.1:80, then httpClient.execute(httpPost) is success.
why? and how can solve this problem?
Note: if I access a.war from browser with public ip like http://<public ip>/a in my computer, all operations are success.
my tomcat connector is:
<Connector
port="80"
protocol="HTTP/1.1"
connectionTimeout="60000"
keepAliveTimeout="15000"
maxKeepAliveRequests="-1"
maxThreads="1000"
minSpareThreads="200"
maxSpareThreads="300"
minProcessors="100"
maxProcessors="900"
acceptCount="1000"
enableLookups="false"
executor="tomcatThreadPool"
maxPostSize="-1"
compression="on"
compressionMinSize="1024"
redirectPort="8443" />
my server has no domain, only has a public ip, its /etc/hosts is:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
updated with some commands run in server:
ss -nltp
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:111 *:* users:(("rpcbind",pid=643,fd=8))
LISTEN 0 128 *:80 *:* users:(("java",pid=31986,fd=53))
LISTEN 0 128 *:22 *:* users:(("sshd",pid=961,fd=3))
LISTEN 0 1 127.0.0.1:8005 *:* users:(("java",pid=31986,fd=68))
LISTEN 0 128 :::111 :::* users:(("rpcbind",pid=643,fd=11))
LISTEN 0 128 :::22 :::* users:(("sshd",pid=961,fd=4))
LISTEN 0 80 :::3306 :::* users:(("mysqld",pid=1160,fd=19))
netstat -nltp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 643/rpcbind
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 31986/java
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 961/sshd
tcp 0 0 127.0.0.1:8005 0.0.0.0:* LISTEN 31986/java
tcp6 0 0 :::111 :::* LISTEN 643/rpcbind
tcp6 0 0 :::22 :::* LISTEN 961/sshd
tcp6 0 0 :::3306 :::* LISTEN 1160/mysqld
ifconfig
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 1396428 bytes 179342662 (171.0 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1396428 bytes 179342662 (171.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
p2p1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.25 netmask 255.255.255.0 broadcast 192.168.1.255
ether f8:bc:12:a3:4f:b7 txqueuelen 1000 (Ethernet)
RX packets 5352432 bytes 3009606926 (2.8 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2839034 bytes 559838396 (533.9 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: p2p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether f8:bc:12:a3:4f:b7 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.25/24 brd 192.168.1.255 scope global noprefixroute dynamic p2p1
valid_lft 54621sec preferred_lft 54621sec
route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default gateway 0.0.0.0 UG 100 0 0 p2p1
192.168.1.0 0.0.0.0 255.255.255.0 U 100 0 0 p2p1
ip route
default via 192.168.1.1 dev p2p1 proto dhcp metric 100
192.168.1.0/24 dev p2p1 proto kernel scope link src 192.168.1.25 metric 100
iptables -L -n -v --line-numbers
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination

You probably have configured one of these:
Firewall public IP's ports, so that nothing goes through.
Tomcat may bind a specific IP, e.g. localhost (see Connector elements in tomcat's server.xml)
Apache httpd, nginx or another reverse proxy might handle various virtual host names, and also they might handle localhost different than the public IP
Port Forwarding - if you only forward localhost:80 to localhost:8080 (tomcat's default port), you might not have anything on publicip:80 that forwards that traffic as well.
Edit after your comment:
incoming traffic seems to be fine, but outgoing you do have those problems. Adding from #stringy05's comment: Check if the IP in question is routable from your server: You're connecting to whatever IP from that server, so use another means to create an outgoing connection, e.g. curl.
Explanation for #1 & #3:
If you connect to an external http server, it will handle the request differently based on the hostname used. It might well be that the IP "hostname" is blocked, either by a high level firewall, or just handled differently than the URL by the webserver itself. In most cases you can check this by connecting to the webserver in question from any other system, e.g. your own browser.

If Tomcat is listening (bound) to your public IP-adres it should work, but maybe your public IP-adres belongs to some other device, like a SOHO router, than your problem is similar to this:
https://superuser.com/questions/208710/public-ip-address-answered-by-router-not-internal-web-server-with-port-forwardi
But without an DNS name you cannot simply add a line to /etc/hosts but you can add the public IP-adres to one of your Network Interfaces Cards (NIC) like lo (loopback), eth0, etc. as described in one of these articles:
https://www.garron.me/en/linux/add-secondary-ip-linux.html
https://www.thegeekdiary.com/centos-rhel-6-how-to-addremove-additional-ip-addresses-to-a-network-interface/
E.g. with public IP-address 1.2.3.4 you would need (which will only be effective until next reboot and worst case might interfere with your ability to connect to the server with e.g. SSH!):
sudo ip addr add 1.2.3.4/32 dev lo
It may be useful to have the output of these commands to better understand your setup, feel free to share it in your question, with consistently anonymized public IP-adres):
Either one of these (ss = socket stat, newer replacement for good old netstat):
ss -nltp
netstat -nltp
And one of these:
ifconfig
ip addr show
And last but not least either one of these:
route
ip route
I don't expect that we need to know your firewall config, but if you use it, it may be interesting to keep an eye on it while you are at it:
iptables -L -n -v --line-numbers

Try putting your public domain names into the local /etc/hosts file of your server like this:
127.0.0.1 localhost YOURPUBLIC.DOMAIN.NAME
This way your Java code does not need to try to use the external IP-adres but instead connects directly to Tomcat.
Good luck!

I think the curl timeout explains it - you have a firewall rule somewhere that is stopping the server accessing the public IP address.
If there's no reason the service can't be accessed using localhost or the local hostname then do that but if you need to call the service via a public IP then it's a matter of working out why the request gets a timeout from the server.
Some usual suspects:
The server might not actually have internet access - can you curl https://www.google.com?
There might be a forward proxy required - a sys admin will know this sort of thing
There might be IP whitelisting on some infra around your server - think AWS security groups, load balancer IP whitelists that sort of thing. To fix that you need to know the public IP of your server curl https://canihazip.com/s and get that added to the whitelist

Java InetAddress.getLocalHost().getHostAddress() returns invalid IP

And when I say invalid, I mean invalid. I get an IP address that's not associated with any of my machine's interfaces. I'm aware there are similar questions on the site, but they're more about people getting the address of an interface they didn't want (e.g., 127.0.0.1 instead of LAN IP), but this is not that. I want to know where this IP is coming from, and how do I get rid of it.
Whenever I call InetAddress.getLocalHost().getHostAddress(), I get 10.65.90.34. This is not the assigned IP from my router, my ISP, my VPN, anything. There is no mentiond of this IP in the output from ifconfig. I have no idea where this IP is coming from. It was suggested to enumerate all interfaces on a different question, and here's what I get when I do that:
public void localIPTest() throws UnknownHostException, SocketException {
String hostAddress = InetAddress.getLocalHost().getHostAddress();
System.out.println(hostAddress);
Enumeration Interfaces = NetworkInterface.getNetworkInterfaces();
while (Interfaces.hasMoreElements()) {
NetworkInterface Interface = (NetworkInterface) Interfaces.nextElement();
System.out.println(Interface.getDisplayName());
Enumeration Addresses = Interface.getInetAddresses();
while (Addresses.hasMoreElements()) {
InetAddress Address = (InetAddress) Addresses.nextElement();
System.out.println(Address.getHostAddress());
}
}
}
This outputs:
10.65.90.34
utun0
fe80:0:0:0:a3c9:7561:3c37:3943%utun0
awdl0
fe80:0:0:0:b458:bbff:feac:2ee8%awdl0
utun1
172.20.97.60
en0
fe80:0:0:0:c88:b30a:16fc:4dbe%en0
192.168.0.106
lo0
fe80:0:0:0:0:0:0:1%lo0
0:0:0:0:0:0:0:1
127.0.0.1
This issue only happens when I am connected to my company's VPN network. And yet this IP is not coming from the VPN interface (utun1) - as can be seen in the above output that is 172.xx.
I tried to see whether this IP is coming from the underyling OS (macOS in my case), but that's not the case either.
$  ifconfig | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*'
inet 127.0.0.1
inet 192.168.0.106
inet 172.20.97.60
$  curl ifconfig.me
183.82.20.54
Output of traceroute 8.8.8.8 as suggested:
$  traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets
1 192.168.0.1 (192.168.0.1) 2.317 ms 1.067 ms 0.847 ms
2 broadband.actcorp.in (106.51.140.1) 2.220 ms 2.585 ms 2.501 ms
3 broadband.actcorp.in (202.83.20.173) 3.120 ms 2.199 ms 4.124 ms
4 broadband.actcorp.in (202.83.20.181) 2.501 ms 2.534 ms 2.635 ms
5 * * *
6 72.14.194.18 (72.14.194.18) 43.590 ms 41.858 ms 42.637 ms
7 74.125.242.131 (74.125.242.131) 42.560 ms
108.170.253.121 (108.170.253.121) 42.911 ms
108.170.253.122 (108.170.253.122) 40.468 ms
8 72.14.239.59 (72.14.239.59) 42.028 ms
74.125.252.165 (74.125.252.165) 41.743 ms
209.85.248.251 (209.85.248.251) 43.130 ms
9 64.233.175.215 (64.233.175.215) 41.679 ms
72.14.236.223 (72.14.236.223) 43.124 ms
209.85.242.12 (209.85.242.12) 42.816 ms
10 216.239.35.145 (216.239.35.145) 42.076 ms
66.249.94.185 (66.249.94.185) 42.791 ms
216.239.35.167 (216.239.35.167) 42.456 ms
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * google-public-dns-a.google.com (8.8.8.8) 41.985 ms *
Also adding full output for ifconfig:
$  ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
nd6 options=201<PERFORMNUD,DAD>
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280
XHC20: flags=0<> mtu 0
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
ether a4:5e:60:e5:b2:8b
inet6 fe80::c88:b30a:16fc:4dbe%en0 prefixlen 64 secured scopeid 0x5
inet 192.168.0.104 netmask 0xffffff00 broadcast 192.168.0.255
nd6 options=201<PERFORMNUD,DAD>
media: autoselect
status: active
p2p0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 2304
ether 06:5e:60:e5:b2:8b
media: autoselect
status: inactive
awdl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1484
ether b6:58:bb:ac:2e:e8
inet6 fe80::b458:bbff:feac:2ee8%awdl0 prefixlen 64 scopeid 0x7
nd6 options=201<PERFORMNUD,DAD>
media: autoselect
status: active
en1: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
options=60<TSO4,TSO6>
ether 6a:00:00:58:9d:c0
media: autoselect <full-duplex>
status: inactive
en2: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
options=60<TSO4,TSO6>
ether 6a:00:00:58:9d:c1
media: autoselect <full-duplex>
status: inactive
bridge0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=63<RXCSUM,TXCSUM,TSO4,TSO6>
ether 6a:00:00:58:9d:c0
Configuration:
id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
ipfilter disabled flags 0x2
member: en1 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 8 priority 0 path cost 0
member: en2 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 9 priority 0 path cost 0
nd6 options=201<PERFORMNUD,DAD>
media: <unknown type>
status: inactive
utun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 2000
inet6 fe80::a3c9:7561:3c37:3943%utun0 prefixlen 64 scopeid 0xb
nd6 options=201<PERFORMNUD,DAD>
utun1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
inet 172.20.97.60 --> 172.20.97.60 netmask 0xffffffff

From the docs for InetAddress.getLocalHost():
This is achieved by retrieving the name of the host from the system, then resolving that name into an InetAddress.
So if you have a bogus hostname, then this function will return a bogus IP address.
You can run the hostname command to see what your machine reports as its hostname, then use nslookup to see what that name resolves to (or alternatively, ping $(hostname), which will do the name resolution).
If that is the problem, and you have admin access on your Mac, then you can change the name (see man hostname for instructions). However, you'll probably need to talk with your IT department to learn what the correct hostname should be.

Finally managed to fix it.
The issue, as #guest pointed out, is for some reason connecting to VPN messes up name resolution, so that my localhost is resolved to some remote IP.
The solution was to simply add an /etc/hosts entry, manually mapping my hostname to 127.0.0.1. That fixed it.

Android-WebRTC: MediaStream.removeVideoTrack is called after setting local description

We are developing an Android Application with Scala, WebRTC and the Java SDK. Our current task consists of receiving the remote track that is sent in a second SDP negotiation when the Callee enables video. Thus the Caller is accessing that video track in the onAddTrack method of the peer-connection observer. In fact, the track is contained in the Array of MediaStreams, and then we are able to attach it to the VideoRenderer via a ProxySink - pretty much like in this example: https://webrtc.googlesource.com/src/+/refs/heads/master/examples/androidapp/src/org/appspot/apprtc/CallActivity.java#133.
This is the piece of code in charge of that:
override def onAddTrack(rtpReceiver: RtpReceiver, mediaStreams: Array[MediaStream]): Unit = {
info("FLOW: onAddTrack")
rtpReceiver.SetObserver(rtpObserver)
call.getCallType match {
case BothVideo(CallRoles.Caller) | RecvPeerVideo(CallRoles.Caller) => {
if (mediaStreams.size > 0) {
val mediaStream = mediaStreams(0)
val videoTracks: Buffer[VideoTrack] = mediaStream.videoTracks.asScala
this.remoteMediaStream = Some(mediaStream)
info(s"FLOW: onAddTrack, call-type: ${call.getCallType}")
if (call.callerShouldReceiveVideo(call.getCallType) && videoTracks.size > 0) {
info(s"FLOW: onAddTrack, enabling video-track")
val videoTrack: VideoTrack = videoTracks(0)
mediaStream.addPreservedTrack(videoTrack)
videoTrack.setEnabled(true)
videoTrack.addSink(flowManager.remoteProxySink)
...
}
}
}
The problem is that we are not able to get the frames from the Callee's remote track, and this is a bit surprising since we were already able to successfully set the Caller's remote track and render the frames when the Caller initiates the call with video and has to send the remote track to Callee.
We have detected that dispose is being called on the MediaStream (https://webrtc.googlesource.com/src/+/refs/heads/master/sdk/android/api/org/webrtc/MediaStream.java#75) right after setLocalDescription is called in the SdpObserver set when creating the answer, and we think this might be cause of our issue.
Creating answer:
pcConn.setRemoteDescription(sdpObserver, sdp)
peerConn.createAnswer(sdpObserver, WebRTCCall.sdpMediaConstraints())
In SdpObserver:
override def onCreateSuccess(sdp: SessionDescription) = {
...
peerConn.setLocalDescription(this, sdp)
...
}
What can be triggering the MediaStream.dispose method inside native setLocalDescription ?
Update
We found something interesting in the sdp, video track seems to have sendonly direction.
v=0
o=- 2191449863069826960 3 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2 3
a=msid-semantic: WMS stream0
m=audio 56825 UDP/TLS/RTP/SAVPF 111 103 9 102 0 8 105 13 110 113 126
c=IN IP4 35.167.127.194
a=rtcp:9 IN IP4 0.0.0.0
a=candidate:2294452033 1 udp 2122260223 192.168.2.159 60968 typ host generation 0 network-id 3 network-cost 10
a=candidate:559267639 1 udp 2122202367 ::1 39373 typ host generation 0 network-id 2
a=candidate:1510613869 1 udp 2122129151 127.0.0.1 56438 typ host generation 0 network-id 1
a=candidate:1876313031 1 tcp 1518222591 ::1 59473 typ host tcptype passive generation 0 network-id 2
a=candidate:344579997 1 tcp 1518149375 127.0.0.1 47922 typ host tcptype passive generation 0 network-id 1
a=candidate:842163049 1 udp 1686052607 201.233.49.70 60968 typ srflx raddr 192.168.2.159 rport 60968 generation 0 network-id 3 network-cost 10
a=candidate:1600295039 1 udp 41885439 35.167.127.194 56825 typ relay raddr 201.233.49.70 rport 60968 generation 0 network-id 3 network-cost 10
a=candidate:3811514541 1 udp 41885695 34.221.141.99 60923 typ relay raddr 201.233.49.70 rport 60968 generation 0 network-id 3 network-cost 10
a=ice-ufrag:GGkb
a=ice-pwd:owqpzSht95NQb92cgzKNkQmJ
a=ice-options:trickle renomination
a=fingerprint:sha-256 45:76:5A:81:C1:CD:6B:C8:17:C6:CB:7B:20:CD:C6:DD:59:A3:DC:11:47:78:D3:E1:E1:EE:55:1F:83:A5:32:E4
a=setup:passive
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:2 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:3 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=sendrecv
a=msid:stream0 audio0
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:9 G722/8000
a=rtpmap:102 ILBC/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:113 telephone-event/16000
a=rtpmap:126 telephone-event/8000
a=ssrc:1249625177 cname:IgXNTJvPXp2HndWi
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 127
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:GGkb
a=ice-pwd:owqpzSht95NQb92cgzKNkQmJ
a=ice-options:trickle renomination
a=fingerprint:sha-256 45:76:5A:81:C1:CD:6B:C8:17:C6:CB:7B:20:CD:C6:DD:59:A3:DC:11:47:78:D3:E1:E1:EE:55:1F:83:A5:32:E4
a=setup:passive
a=mid:1
a=extmap:14 urn:ietf:params:rtp-hdrext:toffset
a=extmap:13 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:12 urn:3gpp:video-orientation
a=extmap:2 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:11 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07
a=extmap:9 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:3 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=sendonly
a=msid:stream0 video0
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:100 red/90000
a=rtpmap:101 rtx/90000
a=fmtp:101 apt=100
a=rtpmap:127 ulpfec/90000
a=ssrc-group:FID 1487952389 3127747783
a=ssrc:1487952389 cname:IgXNTJvPXp2HndWi
a=ssrc:3127747783 cname:IgXNTJvPXp2HndWi
m=application 9 DTLS/SCTP 5000
c=IN IP4 0.0.0.0
b=AS:30
a=ice-ufrag:GGkb
a=ice-pwd:owqpzSht95NQb92cgzKNkQmJ
a

Port conflict issues

I spent some time for the ports used by jvm on krt boxes. I see each jvm opens 10 ports.
Five are defined in the command line for mgmt., http, debug, jmx and ajp. Out of the other
five I can understand 1 for activemq and 2 for jdbc. There are two unknown to me
One out of that connects back to the server and another does not show what it is listening
To. The one option I read on net it to increase the range of ephemeral ports (we have 32k
Starting we can go 16k) I am not sure how we can dictate the port numbers for the five
Which are not defined today
Some commands to describe the situation.
[krtdev7#surya:/env/krtdev7/bin]$krtport KRTDataHistory-1
PORT ASSIGNMENTS:
=================
mgmt/shutdown=17091
http=17291
ajp=17491
jmx=17691
debug=17891
[krtdev7#surya:/env/krtdev7/bin] $ netstat -ap|grep 16831
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:17291 0.0.0.0:* LISTEN 16831/java
tcp 0 0 0.0.0.0:17491 0.0.0.0:* LISTEN 16831/java
tcp 0 0 0.0.0.0:36691 0.0.0.0:* LISTEN 16831/java
tcp 0 0 0.0.0.0:40596 0.0.0.0:* LISTEN 16831/java
tcp 0 0 0.0.0.0:17691 0.0.0.0:* LISTEN 16831/java
tcp 0 0 localhost:17091 0.0.0.0:* LISTEN 16831/java
tcp 0 0 0.0.0.0:17891 0.0.0.0:* LISTEN 16831/java
tcp 0 0 surya.internal.su:51631 sky.internal.s:ncube-lm ESTABLISHED 16831/java
tcp 0 0 surya.internal.su:40938 agni.internal.sun:61616 ESTABLISHED 16831/java
tcp 0 0 surya.internal.su:51630 sky.internal.s:ncube-lm ESTABLISHED 16831/java
unix 2 [ ] STREAM CONNECTED 16386441 16831/java**
Now we can see the 5 extra ports are assigned Could anybody let me know how to control these 5 extra port assignment rather how to make the jvm choose from the range of ports for these 5 extra ports?

Apache Zookeeper multi-node communication error

I have three znode: Apache Zookeeper, 3.4.8 JAVA 1.8_77
RedHat 6.7, Selinux disabled, Firewall disabled, IPV6 disabled
Hosts:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.133 v175
192.168.0.134 v176
192.168.0.125 V177
Config:
tickTime=2000
dataDir=/home/znode/datadir
clientPort=2181
initLimit=5
syncLimit=2
server.1=v175:2888:3888
server.2=v176:2888:3888
server.3=v177:2888:3888
Errors:
essage format version), 2 (n.leader), 0x0 (n.zxid), 0x100f (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LEADING (my state)
2016-04-05 16:26:00,270 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection#600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x100f (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LEADING (my state)
2016-04-05 16:26:03,099 [myid:3] - WARN [QuorumPeer[myid=3]/0.0.0.0:2181:QuorumPeer#862] - Unexpected exception
java.lang.InterruptedException: Timeout while waiting for epoch from quorum
at org.apache.zookeeper.server.quorum.Leader.getEpochToPropose(Leader.java:881)
at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:380)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:859)
2016-04-05 16:26:03,100 [myid:3] - INFO [QuorumPeer[myid=3]/0.0.0.0:2181:Leader#496] - Shutting down
2016-04-05 16:26:03,100 [myid:3] - INFO [QuorumPeer[myid=3]/0.0.0.0:2181:Leader#502] - Shutdown called
java.lang.Exception: shutdown Leader! reason: Forcing shutdown
at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:502)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:865)
2016-04-05 16:26:03,100 [myid:3] - INFO [QuorumPeer[myid=3]/0.0.0.0:2181:QuorumPeer#774] - LOOKING
2016-04-05 16:26:03,100 [myid:3] - INFO [LearnerCnxAcceptor-V177/192.168.0.125:2888:Leader$LearnerCnxAcceptor#325] - exception while shutting down acceptor: java.net.SocketException: Socket closed
2016-04-05 16:26:03,100 [myid:3] - INFO [QuorumPeer[myid=3]/0.0.0.0:2181:FastLeaderElection#818] - New election. My id = 3, proposed zxid=0x0
2016-04-05 16:26:03,102 [myid:3] - WARN [WorkerSender[myid=3]:QuorumCnxManager#400] - Cannot open channel to 2 at election address v176/192.168.0.134:3888
java.net.NoRouteToHostException: No route to host
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
at java.lang.Thread.run(Thread.java:745)
2016-04-05 16:26:03,103 [myid:3] - INFO [WorkerSender[myid=3]:QuorumPeer$QuorumServer#149] - Resolved hostname: v176 to address: v176/192.168.0.134
2016-04-05 16:26:03,102 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection#600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x100f (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
Servers cannot communicate with each other. Help !

if you netstat -tulnap the server, are ports 2888, 3888 ad 2181 open and listening or can only localhost (0.0.0.0) on each of those boxes hit 2181, 288, and 3888?
EDIT:
Looking at your netstat I see this:
tcp 0 0 192.168.0.125:2888 0.0.0.0:* LISTEN
tcp 0 0 192.168.0.125:3888
means you need to modify your etc/hots to use 0.0.0.0 with the hostname.
So if the box's hostname is zoobox1 the /etc/hosts needs to have this line in it:
127.0.0.1 localhost
0.0.0.0 zoobox1
this way the netstat -tulnap will open port 2888 and 3888 so other servers then localhost can connect.

V175 - doesn't have a listener on 2888
[root#v175 ~]# netstat -tulnap
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:2181 0.0.0.0:* LISTEN 3201/java
tcp 0 0 0.0.0.0:45423 0.0.0.0:* LISTEN 3201/java
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1236/rpcbind
tcp 0 0 192.168.0.133:3888 0.0.0.0:* LISTEN 3201/java
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1486/sshd
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 1319/cupsd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1577/master
tcp 0 0 0.0.0.0:54618 0.0.0.0:* LISTEN 1260/rpc.statd
tcp 0 0 192.168.0.133:3888 192.168.0.134:42183 ESTABLISHED 3201/java
tcp 0 0 192.168.0.133:22 10.206.171.250:50630 ESTABLISHED 4838/sshd
tcp 0 0 192.168.0.133:3888 192.168.0.125:58200 ESTABLISHED 3201/java
udp 0 0 0.0.0.0:983 0.0.0.0:* 1236/rpcbind
udp 0 0 0.0.0.0:52328 0.0.0.0:* 1260/rpc.statd
udp 0 0 0.0.0.0:111 0.0.0.0:* 1236/rpcbind
udp 0 0 127.0.0.1:1012 0.0.0.0:* 1260/rpc.statd
udp 0 0 0.0.0.0:631 0.0.0.0:* 1319/cupsd
V176
[root#v176 ~]# netstat -antup
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:2181 0.0.0.0:* LISTEN 5553/java
tcp 0 0 192.168.0.134:2888 0.0.0.0:* LISTEN 5553/java
tcp 0 0 0.0.0.0:60845 0.0.0.0:* LISTEN 1263/rpc.statd
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1239/rpcbind
tcp 0 0 192.168.0.134:3888 0.0.0.0:* LISTEN 5553/java
tcp 0 0 0.0.0.0:38485 0.0.0.0:* LISTEN 5553/java
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1487/sshd
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 1322/cupsd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1578/master
tcp 0 64 192.168.0.134:22 10.206.171.250:50927 ESTABLISHED 10784/sshd
tcp 0 0 192.168.0.134:49506 192.168.0.133:3888 ESTABLISHED 5553/java
udp 0 0 0.0.0.0:38979 0.0.0.0:* 1263/rpc.statd
udp 0 0 0.0.0.0:985 0.0.0.0:* 1239/rpcbind
udp 0 0 0.0.0.0:111 0.0.0.0:* 1239/rpcbind
udp 0 0 0.0.0.0:631 0.0.0.0:* 1322/cupsd
udp 0 0 127.0.0.1:1015 0.0.0.0:* 1263/rpc.statd
V177
[root#v177 ~]# netstat -tulnap
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:2181 0.0.0.0:* LISTEN 5547/java
tcp 0 0 192.168.0.125:2888 0.0.0.0:* LISTEN 5547/java
tcp 0 0 0.0.0.0:40904 0.0.0.0:* LISTEN 5547/java
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1245/rpcbind
tcp 0 0 192.168.0.125:3888 0.0.0.0:* LISTEN 5547/java
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1494/sshd
tcp 0 0 0.0.0.0:40694 0.0.0.0:* LISTEN 1269/rpc.statd
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 1328/cupsd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1585/master
tcp 0 0 192.168.0.125:22 10.206.171.250:50933 ESTABLISHED 10771/sshd
tcp 0 0 192.168.0.125:58200 192.168.0.133:3888 ESTABLISHED 5547/java
udp 0 0 127.0.0.1:1023 0.0.0.0:* 1269/rpc.statd
udp 0 0 0.0.0.0:992 0.0.0.0:* 1245/rpcbind
udp 0 0 0.0.0.0:59619 0.0.0.0:* 1269/rpc.statd
udp 0 0 0.0.0.0:111 0.0.0.0:* 1245/rpcbind
udp 0 0 0.0.0.0:631 0.0.0.0:* 1328/cupsd