We Keep Coding

The number of TCP Connection for AWS S3 Multipart upload is increased

all.
I'd developed AWS S3 multipart upload function.
And because of this source code, the number of TCP connections is increased slowly.
I found there are received packets data from s3 which is not handled using netstat.
tcp6 263112 0 000.0.000.000:53634 [Storage ip]:443 ESTABLISHED
tcp6 253957 0 000.0.000.000:23800 [Storage ip]:443 ESTABLISHED
Could you help me with letting me know which part is causing the problem?
private void completeMultipartUploadThread(String filePath, String uploadPath, int pageMarker) {
partETagList.clear();
while(true)
{
LOG.debug("pageMarker : " + pageMarker);
PartListing pl = s3.listParts(new ListPartsRequest(bucketName, uploadPath, uploadId).withPartNumberMarker(pageMarker*1000));
List<PartSummary> partList = pl.getParts();
for(PartSummary p : partList) {
partETagList.add(new PartETag(p.getPartNumber(),p.getETag()));
tmpEtagList.add(p.getETag());
}
if(partList.size() < 1000)
break;
else
pageMarker++;
}
this.setCalEtagList(tmpEtagList);
LOG.debug("Total Part Count : "+partETagList.size());
try {
CompleteMultipartUploadRequest completeRequest = new CompleteMultipartUploadRequest(bucketName, uploadPath, uploadId, partETagList);
s3.completeMultipartUpload(completeRequest);
} catch(Exception e) {
AbortMultipartUploadRequest amur = new AbortMultipartUploadRequest(bucketName,uploadPath,uploadId);
s3.abortMultipartUpload(amur);
e.printStackTrace();
}
}
Thanks!

JedisCluster configurations and how it maintains the pool of connections

I have recently started using JedisCluster for my application. There is little to no documentation and examples for the same. I tested a use case and the results are not what I expected
public class test {
private static JedisCluster setConnection(HashSet<HostAndPort> IP) {
JedisCluster jediscluster = new JedisCluster(IP, 30000, 3,
new GenericObjectPoolConfig() {{
setMaxTotal(500);
setMinIdle(1);
setMaxIdle(500);
setBlockWhenExhausted(true);
setMaxWaitMillis(30000);
}});
return jediscluster;
}
public static int getIdleconn(Map<String, JedisPool> nodes){
int i = 0;
for (String k : nodes.keySet()) {
i+=nodes.get(k).getNumIdle();
}
return i;
}
public static void main(String[] args) {
HashSet IP = new HashSet<HostAndPort>() {
{
add(new HostAndPort("host1", port1));
add(new HostAndPort("host2", port2));
}};
JedisCluster cluster = setConnection(IP);
System.out.println(getIdleconn(cluster.getClusterNodes()));
cluster.set("Dummy", "0");
cluster.set("Dummy1", "0");
cluster.set("Dummy3", "0");
System.out.println(getIdleconn(cluster.getClusterNodes()));
try {
Thread.sleep(60000);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println(getIdleconn(cluster.getClusterNodes()));
}
}
The output for this snippet is:
0
3
3
Questions=>
I have set the timeout to 30000 JedisCluster(IP, 30000, 3,new GenericObjectPoolConfig() . I believe this is the connection timeout which means Idle connections are closed after 30 seconds. Although this doesn't seem to be happening. After sleeping for 60 seconds, the number of idle connections is still 3. What I am doing/understanding wrong here? I want the pool to close the connection if not used for more than 30 seconds.
setMinIdle(1). Does this mean that regardless the connection timeout, the pool will always maintain one connection?
I prefer availability more than throughput for my app. What should be the value for setMaxWaitMillis if conn timeout is 30 secs?
Though rare, the app fails with redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster. This i think is connected to 1. How to prevent this?

30000 or 30 seconds here refers to (socket) timeout; the timeout for single socket (read) operation. It is not related with closing idle connections.
Closing idle connections are controlled by GenericObjectPoolConfig. So check the parameters there.
Yes (mostly).
setMaxWaitMillis is the timeout for getting a connection object from a connection object pool. It is not related to 30 secs and not really solve you anything in terms of availability.
Keep your cluster nodes available.
There has been changes in Jedis related to this. You can try a recent version (4.x, even better 4.2.x).

A function cannot be accessed when the program is running

I am working on homework about thread synchronization. The method reduceLoad() cannot be accessed when I call this method. All print function inside the reduceLoad() method are never called. If I can't run this method, the key will never be released.
It might be caused by Semaphore.accquire() and Semaphore.realse(). So I tried to delete all Semaphore methods that may cause the problem.
Protion of the program result below:
As you can see the program is held after all threads print "disembarks from ferry at port "
..... omit some result here....
Arrive at port 1 with a load of 5 vehicles
Auto 4 arrives at port 0
Ambulance 1 arrives at port 1
Ambulance 1 boards the ferry at port 1
Auto 6 boards on the ferry at port 1
Auto 8 boards on the ferry at port 1
Auto 0 disembarks from ferry at port 1
Auto 2 disembarks from ferry at port 1
Auto 1 disembarks from ferry at port 1
Ambulance 0 disembarks the ferry at port 1
Auto 7 boards on the ferry at port 1
Auto 3 disembarks from ferry at port 1
This is where the function is called
// Arrive at the next port
port = 1 - port;
// wait for ferry arrives
while (fry.getPort() != port) {
try {
sleep(1000);
} catch (Exception e) {
}
}
// disembarking
System.out.println("Auto " + id_auto + " disembarks from ferry at port " + port);
logger.check(fry.getPort() == port, "error unloading at wrong port");
fry.reduceLoad(); // Reduce load
System.out.println("Auto " + id_auto + " successfully disembarks from ferry at port " + port);
This method reduceLoad() should release a key when the ferry is empty, this key is used to notify other threads to continue adding more cars into the ferry
public synchronized void reduceLoad() {
logger.check(load > 0, "error unloading an empty Ferry!");
load = load - 1;
System.out.println("removed load, now " + load);
if (load == 0) {
unloadingDone.release();
}
}

ArrayList in serializable object not carrying over writeUnshared/readUnshared (over network)

I am developing a multiplayer game in Java built around my own client-server architecture. In short, the client requests a copy of the server's World object 30 times a second and, upon receiving it, sets its client-side copy to the response. This is all done using Java's standard net API.
The issue I am having is that I also store an ArrayList of Player objects in the world, and when I add a Player to this list, the client doesn't get the update. It still receives a copy of the world from the server, but its not up to date.
I experienced a similar problem in a past project that was caused by write/readObject and fixed it by using write/readUnshared, but even that isn't working.
Here's the important stuff from the server end of the communication:
String message;
int sum = 0;
while(active)
{
message = "";
try {
message = in.readUTF();
} catch (IOException e) {
active = false;
System.out.println("Lost connection with client " + socket.getInetAddress());
}
if(message.equals("GETWORLD"))
{
try {
sum++;
if(sum == 100)
main.world.addPlayer(999, 2, 2);
System.out.println("Client requested world (#" + sum + ")");
System.out.println(main.world.players.size());
out.writeUnshared(main.world);
out.flush();
System.out.println("Sent client world (#" + sum + ")");
} catch (IOException e) {
active = false;
System.out.println("Lost connection with client " + socket.getInetAddress());
}
}
if(message.equals("DISCONNECT"))
{
active = false;
System.out.println("Client " + socket.getInetAddress() + " requested disconnect");
}
}
And then the client end:
Object read = null;
int sum = 0;
while(active)
{
try {
Thread.sleep((long)(1000 / 30.0));
if(connected)
{
sum++;
System.out.println("Asking server for world (#" + sum + ")");
out.writeUTF("GETWORLD");
out.flush();
read = in.readUnshared();
if(read instanceof World)
{
World temp = (World)read;
System.out.println(temp.players.size());
frame.panel.updateWorld((World)read);
System.out.println("Got world from server (#" + sum + ")");
}
}
} catch (InterruptedException | ClassNotFoundException e1) {
active = false;
e1.printStackTrace();
} catch (IOException e2) {
active = false;
System.out.println("Lost connection with server # " + socket.getInetAddress());
frame.dispose();
System.exit(0);
}
}
Obviously the sum variable is for debugging.
I further tested this with some output, here's what is scaring me:
Client log:
...
Asking server for world (#99)
1
Got world from server (#99)
Asking server for world (#100)
1
Got world from server (#100)
Asking server for world (#101)
1
Got world from server (#101)
...
Server log:
...
Client requested world (#99)
1
Sent client world (#99)
Client requested world (#100)
2
Sent client world (#100)
Client requested world (#101)
2
Sent client world (#101)
...
You can see here that even though the request numbers match up, there's a clear discrepancy between the number of Player objects in the World object.
Here's the important stuff from the World and Player classes for those curious:
public class World implements Serializable
{
public ArrayList<Room> rooms;
public ArrayList<Player> players;
private QuickMaths qm;
...
public class Player implements Serializable
{
private double xPos;
private double yPos;
private Color color;
int id;
...
I apologize if this is a long yet easy problem. I'm not sure if it's a referencing issue or some other network quirk, but it's really driving me nuts. Thanks in advance.

Your problem is with writeUnshared which is a little misleading.
Read here:
"Note that the rules described above only apply to the base-level
object written with writeUnshared, and not to any transitively
referenced sub-objects in the object graph to be serialized. "
This means that the player object will not be written twice but the old reference to that object in the serialization tree will be used.
The solution to this would be to call the reset method after each write call to ensure that the old written objects will not be referenced again.
So:
out.writeUnshared(main.world);
out.flush();
out.reset();

How can i get already occupied ports?

I am developing eclipse plugin for our organization . We are opening multiple servers[minimum 10 servers] on a user machine using this plugin via eclipse . For starting servers we want port numbers which has been not already binded . For that , I am using serversocket to check this . I think it's a costly operation to open a serversocket object . Internally serversocket will check the port is already binded or not It takes minimum 50 milliseconds . Here is my code to return a free port . Is there any way to find already occupied ports without using OS Commands and opening ServerSocket ?
/**
*Tries 100 times
* #param port
* modes
* 1.increment - 1
* This mode increment the port with your start value . But it's costly operation because each time we open a socket and check the port is free .
* 2.decrement - 2
* Invert of increment.
* 3.random - 3
* Randomly choose based on your starting point
* #return
*/
public static String getDefaultPort(int port , int mode){
int retry = 100;
int random = 3;
int increment = 1;
int decrement = 2;
while(true){
//this is for preventing stack overflow error.
if(retry < 1){ //retries 100 times .
break;
}
if(mode==increment){
port++;
}else if(mode == decrement){
port--;
}else if(mode == random){
port = (int) (port+Math.floor((Math.random()*1000)));
}
if(validate(port+"")){
long end = System.currentTimeMillis();
return port+"";
}
}
return "";
}
public boolean validate(String input) {
boolean status = true;
try {
int port = Integer.parseInt(input);
ServerSocket ss = new ServerSocket(port);
ss.close();
}
catch (Exception e) {
status = false;
}
return status;
}

The quickest way would be to run native netstat command and parse the output. It's available on Windows as well and Linux platform. A typical netstat command output is as follows
Proto Local Address Foreign Address State
TCP MYHOST:8080 MYHOST.mydomain.co.in:0 LISTENING
TCP MYHOST:9090 MYHOST.mydomain.co.in:0 LISTENING
TCP MYHOST:3389 MYHOST.mydomain.co.in:0 LISTENING
TCP MYHOST:7717 MYHOST.mydomain.co.in:0 LISTENING
TCP MYHOST:51114 MYHOST.mydomain.co.in:0 LISTENING
TCP MYHOST:netbios-ssn MYHOST.mydomain.co.in:0 LISTENING
TCP MYHOST:netbios-ssn MYHOST.mydomain.co.in:0 LISTENING
TCP MYHOST:2573 hj-lyncfe.mydomain.co.in:5061 ESTABLISHED
TCP MYHOST:2591 mail.mydomain.co.in:8502 ESTABLISHED
TCP MYHOST:2593 mail.mydomain.co.in:8502 ESTABLISHED
The ports of your interest are in the column Local Address with State in LISTENING