We Keep Coding

How to keep TCP sockets open?

Our application has a ping-pong like conversation with many servers (each server has a corresponding thread where those connections are made). Code below works, but it opens a new connection for every new request and is used only once, which soon leads to reaching max connection cap set by server.
DataProvider.java
public static ZnResult sendTcpQuery(String xml, String url, int port) {
List<ZnXmlResult> results = new ArrayList<>();
String xmlString = xml != null ? new String((xml + "\n").getBytes()) : "";
int error = ZnResult.OK;
try (Socket clientSocket = new Socket(url, port)) {
clientSocket.setSoTimeout(CONNECTION_TIMEOUT);
DataOutputStream outToServer = new DataOutputStream(clientSocket.getOutputStream());
try (BufferedReader inFromServer = new BufferedReader(new InputStreamReader(clientSocket.getInputStream(), "UTF-8"))) {
outToServer.writeBytes(xmlString);
try (StringWriter responseFromServer = new StringWriter()) {
String readLine;
while ((readLine = inFromServer.readLine()) != null) {
...
}
}
outToServer.close();
clientSocket.close();
}
} catch (Exception ex) {
LOG.error("Exception {}", url + ":" + port, ex);
error = ZnResult.ERR;
}
return error == ZnResult.OK ? new ZnResult(results) : new ZnResult(error);
}
How can I transform it, so everything can be done within one connection?
I figured I would do something like this:
SocketFactory.java
public class SocketFactory {
private static HashMap<String, Socket> socketsByAddress = new HashMap<>();
private static HashMap<Socket, DataOutputStream> outputStreamsBySocket = new HashMap<>();
private static HashMap<Socket, BufferedReader> readersBySocket = new HashMap<>();
public static Socket getSocket(String address) {
String ip = Tools.getIpFromAddress(address);
int port = Tools.getPortFromAddress(address);
Socket socket = socketsByAddress.get(address);
if (socket == null) {
try {
socket = new Socket(ip, port);
socket.setSoTimeout(60000);
socketsByAddress.put(address, socket);
} catch (IOException ex) {
Logger.getLogger(SocketFactory.class.getName()).log(Level.SEVERE, null, ex);
}
}
return socket;
}
public static DataOutputStream getOutputStream(Socket socket) {
DataOutputStream outputStream = outputStreamsBySocket.get(socket);
if (outputStream == null) {
try {
outputStream = new DataOutputStream(socket.getOutputStream());
outputStreamsBySocket.put(socket, outputStream);
} catch (IOException ex) {
Logger.getLogger(SocketFactory.class.getName()).log(Level.SEVERE, null, ex);
}
}
return outputStream;
}
public static BufferedReader getReader(Socket socket) {
BufferedReader reader = readersBySocket.get(socket);
if (reader == null) {
try {
reader = new BufferedReader(new InputStreamReader(socket.getInputStream()));
readersBySocket.put(socket, reader);
} catch (IOException ex) {
Logger.getLogger(SocketFactory.class.getName()).log(Level.SEVERE, null, ex);
}
}
return reader;
}
}
DataProvider.java
public static ZnResult sendTcpQuery(String xml, String url, int port) {
List<ZnXmlResult> results = new ArrayList<>();
int error = ZnResult.OK;
try {
String xmlString = xml != null ? new String((xml + "\n").getBytes()) : "";
Socket clientSocket = SocketFactory.getSocket(url + ":" + port);
DataOutputStream outToServer = SocketFactory.getOutputStream(clientSocket);
BufferedReader inFromServer = SocketFactory.getReader(clientSocket);
outToServer.writeBytes(xmlString);
try (StringWriter responseFromServer = new StringWriter()) {
String readLine;
while ((readLine = inFromServer.readLine()) != null) {
...
}
}
} catch (Exception ex) {
LOG.error("Exception {}", url + ":" + port, ex);
error = ZnResult.ERR;
}
return error == ZnResult.OK ? new ZnResult(results) : new ZnResult(error);
}
but it just doesn't work and only the first one go through.

This loop reads until the end of the stream.
while ((readLine = inFromServer.readLine()) != null) {
A stream only ends once. i.e. you can't end the stream but later use it again.
What you need to do instead;
have a terminating line which can't occur in your data. e.g. wait for "[EOF]"
send the length of data first and read only that much data.

Try to initiate the Socket object using URL rather than IP address as you were doing in your first code and see if it works for you.

Is there a way to read a file from GitHub and print it in the console using Java?

I want to store a file in Github as a string in java and process it later in the code. How can I do that ?

We can execute curl commands in Java and use Basic authentication to access files.
URL url;String username="username",password="password",file="";
try {
url = new URL("https://www.bitbucket.com/raw-file-url");
URLConnection uc;
uc = url.openConnection();
uc.setRequestProperty("X-Requested-With", "Curl");
ArrayList<String> list=new ArrayList<String>();
String userpass = username + ":" + password;
String basicAuth = "Basic " + new String(new Base64().encode(userpass.getBytes()));//needs Base64 encoder, apache.commons.codec
uc.setRequestProperty("Authorization", basicAuth);
BufferedReader reader = new BufferedReader(new InputStreamReader(uc.getInputStream()));
String line = null;
while ((line = reader.readLine()) != null)
file=file+line+"\n";
System.out.println(file);
return file;
} catch (IOException e) {
System.out.println("Wrong username and password");
return null;
}

This is plain java way of doing this...
I used standard java.net.URL api and Base64 class to connect github and print a json like the below example :
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.Base64;
public class GitConnect {
public static void main(String... args) throws Exception {
java.net.URL url = null;
String username = "user";
String password = "gitpwd";
String file = "";
try {
url = new java.net.URL("https://raw.githubusercontent.com/lrjoshi/webpage/master/public/post/c159s.csv");
java.net.URLConnection uc;
uc = url.openConnection();
uc.setRequestProperty("X-Requested-With", "Curl");
java.util.ArrayList<String> list = new java.util.ArrayList<String>();
String userpass = username + ":" + password;
String basicAuth = "Basic " + new String(Base64.getEncoder().encode(userpass.getBytes()));//needs Base64 encoder, apache.commons.codec
uc.setRequestProperty("Authorization", basicAuth);
BufferedReader reader = new BufferedReader(new InputStreamReader(uc.getInputStream()));
String line = null;
while ((line = reader.readLine()) != null)
file = file + line + "\n";
System.out.println(file);
} catch (IOException e) {
System.out.println("Wrong username and password");
}
}
}

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.Reader;
import java.io.StringWriter;
import java.io.Writer;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.List;
import java.util.Map;
public static String getTextFromGithub(String link) {
URL Url = null;
try {
Url = new URL(link);
} catch (MalformedURLException e1) {
e1.printStackTrace();
}
HttpURLConnection Http = null;
try {
Http = (HttpURLConnection) Url.openConnection();
} catch (IOException e1) {
e1.printStackTrace();
}
Map<String, List<String>> Header = Http.getHeaderFields();
for (String header : Header.get(null)) {
if (header.contains(" 302 ") || header.contains(" 301 ")) {
link = Header.get("Location").get(0);
try {
Url = new URL(link);
} catch (MalformedURLException e) {
e.printStackTrace();
}
try {
Http = (HttpURLConnection) Url.openConnection();
} catch (IOException e) {
e.printStackTrace();
}
Header = Http.getHeaderFields();
}
}
InputStream Stream = null;
try {
Stream = Http.getInputStream();
} catch (IOException e) {
e.printStackTrace();
}
String Response = null;
try {
Response = GetStringFromStream(Stream);
} catch (IOException e) {
e.printStackTrace();
}
return Response;
}
private static String GetStringFromStream(InputStream Stream) throws IOException {
if (Stream != null) {
Writer Writer = new StringWriter();
char[] Buffer = new char[2048];
try {
Reader Reader = new BufferedReader(new InputStreamReader(Stream, "UTF-8"));
int counter;
while ((counter = Reader.read(Buffer)) != -1) {
Writer.write(Buffer, 0, counter);
}
} finally {
Stream.close();
}
return Writer.toString();
} else {
return "No Contents";
}
}

SMTP client for Gmail on java using sockets authentication issue

I found a socket SMTP client example slightly modified for it to connect to gmail using SSLsockets, but now I don't know how to authorise account, that I am sending from. (I don't use JAVAMAIL, because this is homework)
public class SMTP {
public static void main(String args[]) throws IOException,
UnknownHostException {
String msgFile = "file.txt";
String from = "from#gmail.com";
String to = "to#gmail.com";
String mailHost = "smtp.gmail.com";
SMTP mail = new SMTP(mailHost);
if (mail != null) {
if (mail.send(new FileReader(msgFile), from, to)) {
System.out.println("Mail sent.");
} else {
System.out.println("Connect to SMTP server failed!");
}
}
System.out.println("Done.");
}
static class SMTP {
private final static int SMTP_PORT = 25;
InetAddress mailHost;
InetAddress localhost;
BufferedReader in;
PrintWriter out;
public SMTP(String host) throws UnknownHostException {
mailHost = InetAddress.getByName(host);
localhost = InetAddress.getLocalHost();
System.out.println("mailhost = " + mailHost);
System.out.println("localhost= " + localhost);
System.out.println("SMTP constructor done\n");
}
public boolean send(FileReader msgFileReader, String from, String to)
throws IOException {
SSLSocket smtpPipe;
InputStream inn;
OutputStream outt;
BufferedReader msg;
msg = new BufferedReader(msgFileReader);
smtpPipe = (SSLSocket) ((SSLSocketFactory) SSLSocketFactory.getDefault()).createSocket(InetAddress.getByName("smtp.gmail.com"), 465);
if (smtpPipe == null) {
return false;
}
inn = smtpPipe.getInputStream();
outt = smtpPipe.getOutputStream();
in = new BufferedReader(new InputStreamReader(inn));
out = new PrintWriter(new OutputStreamWriter(outt), true);
if (inn == null || outt == null) {
System.out.println("Failed to open streams to socket.");
return false;
}
String initialID = in.readLine();
System.out.println(initialID);
System.out.println("HELO " + localhost.getHostName());
out.println("HELO " + localhost.getHostName());
String welcome = in.readLine();
System.out.println(welcome);
System.out.println("MAIL From:<" + from + ">");
out.println("MAIL From:<" + from + ">");
String senderOK = in.readLine();
System.out.println(senderOK);
System.out.println("RCPT TO:<" + to + ">");
out.println("RCPT TO:<" + to + ">");
String recipientOK = in.readLine();
System.out.println(recipientOK);
System.out.println("DATA");
out.println("DATA");
String line;
while ((line = msg.readLine()) != null) {
out.println(line);
}
System.out.println(".");
out.println(".");
String acceptedOK = in.readLine();
System.out.println(acceptedOK);
System.out.println("QUIT");
out.println("QUIT");
return true;
}
}
}

Rewrote the code. This works fine.
public class TotalTemp
{
private static DataOutputStream dos;
public static void main(String[] args) throws Exception
{
int delay = 1000;
String user = "xxxxx#gmail.com";
String pass = "xxxxxxxx11";
String username = Base64.encodeBase64String(user.getBytes(StandardCharsets.UTF_8));
String password = Base64.encodeBase64String(pass.getBytes(StandardCharsets.UTF_8));
SSLSocket sock = (SSLSocket)((SSLSocketFactory)SSLSocketFactory.getDefault()).createSocket("smtp.gmail.com", 465);
// Socket sock = new Socket("smtp.gmail.com", 587);
final BufferedReader br = new BufferedReader(new InputStreamReader(sock.getInputStream()));
(new Thread(new Runnable()
{
public void run()
{
try
{
String line;
while((line = br.readLine()) != null)
System.out.println("SERVER: "+line);
}
catch (IOException e)
{
e.printStackTrace();
}
}
})).start();
dos = new DataOutputStream(sock.getOutputStream());
send("EHLO smtp.gmail.com\r\n");
Thread.sleep(delay);
send("AUTH LOGIN\r\n");
Thread.sleep(delay);
send(username + "\r\n");
Thread.sleep(delay);
send(password + "\r\n");
Thread.sleep(delay);
send("MAIL FROM:<XXXXXXXX#gmail.com>\r\n");
//send("\r\n");
Thread.sleep(delay);
send("RCPT TO:<YYYYYYYY#gmail.com>\r\n");
Thread.sleep(delay);
send("DATA\r\n");
Thread.sleep(delay);
send("Subject: Email test\r\n");
Thread.sleep(delay);
send("Test 1 2 3\r\n");
Thread.sleep(delay);
send(".\r\n");
Thread.sleep(delay);
send("QUIT\r\n");
}
private static void send(String s) throws Exception
{
dos.writeBytes(s);
System.out.println("CLIENT: "+s);
}
}

First, Make sure you have turned on 'Allow Less Secure Apps' from your Gmail.
We can improve the code by ignoring the Multi-threading part by just reading the output from server. As we know from the RFC that server sends 9 lines after getting the first 'EHLO' request. So, we are just reading 9 lines with bufferedReader. Then for the next few commands, it returns only one line. So, the simplified code without multithreading will be like this :
import java.nio.charset.StandardCharsets;
import javax.net.ssl.*;
import java.io.*;
import java.util.Base64;
public class SMTP_Simplified_v2 {
// Credentials
public static String user = "xxxxxxx#gmail.com";
public static String pass = "xxxxxxxxxx";
private static DataOutputStream dataOutputStream;
public static BufferedReader br = null;
public static void main(String[] args) throws Exception {
int delay = 1000;
String username = Base64.getEncoder().encodeToString(user.getBytes(StandardCharsets.UTF_8));
String password = Base64.getEncoder().encodeToString(pass.getBytes(StandardCharsets.UTF_8));
SSLSocketFactory sslSocketFactory = (SSLSocketFactory) SSLSocketFactory.getDefault();
SSLSocket sslSocket = (SSLSocket) sslSocketFactory.createSocket("smtp.gmail.com", 465);
br = new BufferedReader(new InputStreamReader(sslSocket.getInputStream()));
dataOutputStream = new DataOutputStream(sslSocket.getOutputStream());
send("EHLO smtp.gmail.com\r\n",9);
send("AUTH LOGIN\r\n",1);
send(username+"\r\n",1);
send(password+"\r\n",1);
send("MAIL FROM:<xxxxxxxx#gmail.com>\r\n",1);
send("RCPT TO:<xxxxx#gmail.com>\r\n",1);
send("DATA\r\n",1);
send("Subject: Email test\r\n",0);
send("Email Body\r\n",0);
send(".\r\n",0);
send("QUIT\r\n",1);
}
private static void send(String s, int no_of_response_line) throws Exception
{
dataOutputStream.writeBytes(s);
System.out.println("CLIENT: "+s);
Thread.sleep(1000);
// Just reading the number of lines the server will respond.
for (int i = 0; i < no_of_response_line; i++) {
System.out.println("SERVER : " +br.readLine());
}
}
}

Why do i get the UnknownHostException if there are no proxies?

I am trying to test this code from oracle.com, but get the UnknownHostException. However I can easily go to oracle.com, and everything displays fine in the browser. There are no blocks or something on my computer.
Why does this happen?
import java.net.*;
import java.io.*;
public class URLConnectionReader {
public static void main(String[] args) throws Exception {
URL oracle = new URL("http://www.oracle.com/");
URLConnection yc = oracle.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(
yc.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null)
System.out.println(inputLine);
in.close();
}
}
EDIT
I appears I don't have any proxies . I checked like this
System.setProperty("java.net.useSystemProxies", "true");
System.out.println("detecting proxies");
List l = null;
try {
l = ProxySelector.getDefault().select(new URI("http://foo/bar"));
}
catch (URISyntaxException e) {
e.printStackTrace();
}
if (l != null) {
for (Iterator iter = l.iterator(); iter.hasNext();) {
java.net.Proxy proxy = (java.net.Proxy) iter.next();
System.out.println("proxy type: " + proxy.type());
InetSocketAddress addr = (InetSocketAddress) proxy.address();
if (addr == null) {
System.out.println("No Proxy");
} else {
System.out.println("proxy hostname: " + addr.getHostName());
System.setProperty("http.proxyHost", addr.getHostName());
System.out.println("proxy port: " + addr.getPort());
System.setProperty("http.proxyPort", Integer.toString(addr.getPort()));
}
}
}
the result is
detecting proxies
proxy type: DIRECT
No Proxy

Java, Sockets, BufferedReader and StringBuilder

Yestarday I wrote a post about Java and Sockets, and today I'm still here because I'm having an issue with BufferedReaders.
I searched some questions here in StackOverflow and I understand the problem, but I can't fix it
My "application" has got two parts: a server and a client, and the scope of the application is to execute MS-DOS commands on the machine where the server is running (the commands are sent by the client).
Now the code (I will post the total code because it's easier to understand, I will put a comment in non-working part of the code) Server:
import java.net.*;
import java.io.*;
public class TCPCmdServer {
public int port;
public ServerSocket server;
public final String version = "Beta 1.0";
TCPCmdServer(int port) {
this.port = port;
if (!createServer())
System.out.println("Cannot start the server");
else {
System.out.println("**********************************************");
System.out.println("Command executer, server version: " + version);
System.out.println("Server running on port " + port);
System.out.println("Code by luc99a alias L99");
System.out.println("**********************************************");
}
}
public boolean createServer() {
try {
server = new ServerSocket(port);
} catch (IOException e) {
e.printStackTrace();
return false;
}
return true;
}
public static void main(String[] args) {
TCPCmdServer tcp = new TCPCmdServer(5000);
while (true) {
Socket socket = null;
BufferedReader in = null;
BufferedWriter out = null;
try {
socket = tcp.server.accept();
System.out.println("A client has connected");
in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
out = new BufferedWriter(new OutputStreamWriter(socket.getOutputStream()));
out.write("Welcome on the server... type the commands you like, type END to close the connection\n");
out.flush();
} catch (IOException exc) {
exc.printStackTrace();
}
if (socket != null && in != null && out != null) {
try {
String cmd = null;
while (!(cmd = in.readLine()).equals("END")) {
System.out.println("Recieved: " + cmd);
Process p = Runtime.getRuntime().exec(cmd);
BufferedReader pRead = new BufferedReader(new InputStreamReader(p.getInputStream()));
String line;
StringBuilder builder = new StringBuilder();
while ((line = pRead.readLine()) != null) {
builder = builder.append(line + "\n");
}
out.write(builder.toString() + "\n");
//here is sent "EnD"
out.write("EnD \n");
out.flush();
System.out.println(builder.toString());
pRead.close();
}
} catch (IOException ex) {
ex.printStackTrace();
} finally {
System.out.println("Closing connection...");
try {
socket.close();
in.close();
out.close();
} catch (IOException excp) {
excp.printStackTrace();
}
}
}
}
}
}
And now the code for the client part
import java.net.*;
import java.io.*;
public class TCPCmdClient {
public Socket socket;
public int port;
public String ip;
public final String version = "Beta 1.0";
TCPCmdClient(String ip, int port) {
this.ip = ip;
this.port = port;
if (!createSocket())
System.out.println("Cannot connect to the server. IP: " + ip + " PORT: " + port);
else {
System.out.println("**********************************************");
System.out.println("Command executer, client version: " + version);
System.out.println("Connected to " + ip + ":" + port);
System.out.println("Code by luc99a alias L99");
System.out.println("**********************************************");
}
}
public boolean createSocket() {
try {
socket = new Socket(ip, port);
} catch (IOException e) {
return false;
}
return true;
}
public static void main(String[] args) {
TCPCmdClient client = new TCPCmdClient("127.0.0.1", 5000);
try {
BufferedReader sysRead = new BufferedReader(new InputStreamReader(System.in));
BufferedReader in = new BufferedReader(new InputStreamReader(client.socket.getInputStream()));
BufferedWriter out = new BufferedWriter(new OutputStreamWriter(client.socket.getOutputStream()));
String response = in.readLine();
System.out.println("Server: " + response);
boolean flag = true;
while (flag) {
System.out.println("Type a command... type END to close the connection");
String cmd = sysRead.readLine();
out.write(cmd + "\n");
out.flush();
if (cmd.equals("END")) {
client.socket.close();
sysRead.close();
in.close();
out.close();
flag = false;
} else {
//The loop doesn't finish because the reader
//listens for a new line
//so I used the string "EnD", sent by the server to
//stop the loop, anyway it doesn't seem to work
//I put a comment in the server where "EnD" is sent
String output;
while (((output = in.readLine()) != null)) {
if (output.equals("EnD")) {
break;
} else {
System.out.println(output);
}
}
System.out.println(" *************************************** ");
}
}
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
The problem is that the BufferedReader waits for a new line forever in the while loop (I wrote a comment in the code). I tryed to stop it using a "special string", but it doesn't seem to work.
I can't change the while in
String output;
while (((output = in.readLine()) != null) && output.length > 0)
{
//code here...
}
because in the output of the MS-DOS command (think on "ipconfig") are also present empty lines.
How could I correct it?
Thank you for your help!

your client Sends "EnD " (with a whitespace at the end) and you are comparing to "EnD" without a whitespace. So the two strings are not equal. try to send it without the white space:
out.write("EnD\n");

Space is missing. In TCPCmdClient.java change
if (output.equals("EnD")) {
to
if (output.equals("EnD ")) {