We Keep Coding

How to communicate with a python Twisted server from Java successfully?

These questions may sound silly, but I am new to this networking thing.
I have been trying for quite a few days now to implement a client that works with a Twisted server, but I am failing to get any response back from the server. I have read a lot of docs and watched a few tutorials and I got some of the stuff fixed and got some of the concepts better understood.
Before I step on to asking any questions, I wanna show you my code first. This is what I use to talk to the Twisted-based server:
val socketfactory: SocketFactory = SocketFactory.getDefault()
val socket = socketfactory.createSocket(host, port)
socket.keepAlive = true
socket.tcpNoDelay = true
val isSocketConnected = socket.isConnected //this checks socket's connectivity
val dOut = DataOutputStream(socket.getOutputStream())
val dIn = DataInputStream(socket.getInputStream())
val teststring = "Hi server!"
dOut.writeUTF(teststring)
Log.d("MILESTONE", "MESSAGE SENT AT THIS POINT, Socket is connected ?: $isSocketConnected")
var testreader = ""
while (true) {
testreader = dIn.readUTF()
Log.d("READING:", "RECEIVED THIS: $testreader")
}
My code seems to never get to the second "Log" line. It never gets there. I assume that's because I never get any input from the server. This is getting me confused. Because "socket.isConnected" returns true. Doesn't that mean there is an ongoing connection between the client (me) and the server ? But when I send any output the server doesn't talk back.
So my questions are:
1- Am I doing something wrong? Why do I receive no talk from the server and it blocks the code?
2- Is SocketFactory necessary ?
3- Is there any library that communicates with Twisted from Java ?
Thanks in advance !

For everyone who's struggling to communicate with a Twisted-running python server, I came with the absolutely best solution ever! After inspecting Twisted's open source code, I came to realize it has a "LineReceiver" class that only responds to a message if the line is finished. In other words, you can keep sending data forever and it will never respond until you finish the line and start a new one. Twisted will know the line has finished when a delimiter is used. (It is configured on the server-side). Most servers running Twisted will use a line delimiter : "\r\n"
That's the tricky thing! Once you send that little string, it will start responding to you. Here it is in an example :
val dOut = DataOutputStream(socket.getOutputStream()) //This is my favorite way of sending data!
val dIn = socket.getInputStream().bufferedReader(Charsets.UTF_8) //This is my favorite way of reading data !
val teststring = "hi server! \r\n" //This is the tricky part !
That's it ! all you have to do after that is read the lines from the bufferedReader, like this !
var testreader: List<String>
while (true) {
testreader = dIn.bufferedReader(Charsets.UTF_8).readLines()
for (line in testreader)
Log.e("MILESTONE", line)
}
After I started reading the input, I found out the server started actually sending me strings and communicating back with me. I hope everyone will get their codes working concerning this or any other thing!

Getting a `500 Unknown Command` when invoking `GET` in Java

Alright, so I'm given a Java FtpClient class that I am supposed to finish/modify so that the finished product will serve as a WebServer.
The following is a method that lets me interact with the server through command lines.
/*
* Send ftp command
* #param command: the full command line to send to the ftp server
* #param expected_code: the expected response code from the ftp server
* #return the response line from the ftp server after sending the command
*/
private String sendCommand(String command, int expected_response_code){
String response = "";
try {
// send command to the ftp server
controlWriter.writeBytes(command);
// get response from ftp server
response = controlReader.readLine();
if (DEBUG) {
System.out.println("Current FTP response: " + response);
}
// check validity of response
if (!response.startsWith(String.valueOf(expected_response_code)))
{
throw new IOException(
"Bad response: " + response);
}
} catch (IOException ex) {
System.out.println("IOException: " + ex);
}
return response;
}
However, when I invoke the GET command, i.e.
sendCommand("get " + __file__name__ + "\r\l", 200);),
I get the following response:
500 Unknown command.
I am almost 100% sure this issue has nothing to do with the method I've posted above, but I only posted it so you'll know what I am referring to by the sendCommand method).
Has anyone had a similar issue with this command before? If so, how did you work around it?

I've done a very similar side project to the one you're doing here, and I've encountered the same problem you've discussed here. I still haven't figured out why I wasn't able to simly invoke GET and read off the data stream, but here's my get-around.
First, you'll need to use RETR instead of GET. If you're not familiar with what RETR does, it basically lets you send a file as a packet of bytes through a temporary port you'll generate for data transmission.
To instantiate a temporary port, you will need to be im Passive Mode. So, type in:
quote pasv
Your output would look something like the following:
227 Entering Passive Mode (127,0,0,1,143,155).
A quick glance at the numbers shown between the two parentheses will probably not mean anything to you, however, two details can be derived from there.
The first 3 numbers represent your localhost which is always 127.0.0.1, an the other two are referred to p1 and p2. In this case, we have p1 = 143 and p2 = 155. These two numbers can be used to figure out which port has been assigned to us for data transfer.
Fire up your command line interface
To find out the port number, plug in the numbers in the following formula:
PORT = p1 * 256 + p2
So, our port number in this case is (143 * 256) + 155 ==> (36763).
Now that we have a transfer port open for us and ready for data transfer, you can go ahead and instantiate a new Socket with the port number derived from the formula mentioned above (please note that the numbers will be different every time you run the quote pasv, so don't assume these are constants).
The next step here is to send the file from yourself to the client. Note that you're not directing the data packet to a specific client, rather, any client that's currently connected will be receiving the packets.
To send the file, type in the following command:
quote retr
now you can use the DataInputStream class from your Socket to read all the bytes, display them, then them into an identical copy of the original file, or do whatever you're planning to do with them.
Note.. Note... Note... : the commands listed above were meant to be entered from the command line But since you want your application to handle all the job (I assume), the same commands can be passed from your Java application with a little bit of tweaking around. You will basically only need to take the word quote out of all the commands we've used them in.
I think I've typed enough tonight. I am headed to bed now. Let me know if you need further help in a comment below and I will try to respond as soon as possible. Also, let me know if anything I have said is not making sense to you.

Java Socket : Write to client not working

I currently have a problem with a Java server thingy.
It's a simple TCP server which send images. Problem is, I don't have the code of the client program... Moreover, it seems that there is no way to check the client socket for writing event nor the amount of data already sent to the client.
Do someone have any idea about what could prevent the client to get the image correctly ?
Here's my code :
byte[] response = process ( cmd );
if ( response == null )
{
controlSock.close();
dataSock.close();
stop = true;
}
else if ( dataSock != null )
{
dos.write( response );
dos.flush();
}
By the way, the server is working fine with Telnet.

If the server works fine with telnet then your server is fine.
The problem is more likely to be in the assumptions the client is making are not the same as yours. for example the client might assume you are sending the size first in big or little endian format (as an int, short or long) Perhaps it expects the name of the file/image in some format as well.
The only way to know this for sure is to read the code for the client or ask someone who knows what assumptions the client makes.

Writing a WebSocket Server

I'm trying to write a WebSocket Server in both, java and C++ but I'm stuck right now.
Using java and java.net.ServerSocket/java.net.Socket I managed to get a connection and succesfully do the handshake but the data sent by the WebSocket to the Java Server is not quite what I expected.
When sending messages from javascript like this:
var count = 0;
function loop(){
websocket.send("loop: " + count + "\n");
count++;
setTimeout(loop, 100);
}
The Java server receives this, with line feeds every now and then but not for every websocket.send() that has been invoked.
?‡½÷"˜Ñ˜Mè‡×?‡AÎ3-¡C{îN?‡ŒÍ[Uà¢4%¶íi?‡$ÍåøH¢ŠˆíÖ?‡·†ÞžÛé±î?¦ê?‡'½Ø…KÒ·õ?í?‡dÒÛ‘½´á^òí?‡+ù?YG–â)Ùº?‡›?
Ë÷àb»¡¯5?‡mÉŒQ¦ã!Wéµ?ˆ:J FV%f6
The Java server retrieves values from the socket using BufferedReader.readLine()
BufferedReader socketReader = new BufferedReader(new InputStreamReader(socket.getInputStream()));
String line =socketReader.readLine();
This works fine for the handshake and all handshake data is readable but it does not work after the handshake is done.
Is the data after the handshake somehow encrypted? How can I read it?
EDIT :
The program files:
SocketConnectTest.html
ServerTest.java
ClientSessionTest.java
ResponseGenerator.java
output
Just run ServerTest.java and then open SocketConnectTest.html.
ClientSessionTest.initClientListener() handles the messages from the client.
SOLUTION :
For the solution see pimvdbs post below and his answer at How to (de)construct data frames in WebSockets hybi 08+?

The data coming across web sockets is raw, not string encoded data.
I'd suggest not wrapping a BufferedReader around the incoming data as packets are framed with 0x00 bytes. The crazy characters you are seeing are a result of Java not understanding the encoding that the data is in.
You will need to be responsible for splitting up the data into character and control parts. Once you've split the data up into the appropriate areas, then you can decode the data as a string.

using sockets to fetch a webpage with java

I'd like to fetch a webpage, just fetching the data (not parsing or rendering anything), just catch the data returned after a http request.
I'm trying to do this using the high-level Class Socket of the JavaRuntime Library.
I wonder if this is possible since I'm not at ease figuring out the beneath layer used for this two-point communication or I don't know if the trouble is coming from my own system.
.
Here's what my code is doing:
1) setting the socket.
this.socket = new Socket( "www.example.com", 80 );
2) setting the appropriate streams used for this communication.
this.out = new PrintWriter( socket.getOutputStream(), true);
this.in = new BufferedReader( new InputStreamReader( socket.getInputStream() ) );
3) requesting the page (and this is where I'm not sure it's alright to do like this).
String query = "";
query += "GET / HTTP/1.1\r\n";
query += "Host: www.example.com\r\n";
...
query += "\r\n";
this.out.print(query);
4) reading the result (nothing in my case).
System.out.print( this.in.readLine() );
5) closing socket and streams.

If you're on a *nix system, look into CURL, which allows you to retrieve information off the internet using the command line. More lightweight than a Java socket connection.
If you want to use Java, and are just retrieving information from a webpage, check out the Java URL library (java.net.URL). Some sample Java code:
URL ur = new URL("www.google.com");
URLConnection conn = ur.openConnection();
InputStream is = conn.getInputStream();
String foo = new Scanner(is).useDelimiter("\\A").next();
System.out.println(foo);
That'll grab the specified URL, grab the data (html in this case) and spit it out to the console. Might have to tweak the delimiter abit, but this will work with most network endpoints sending data.

Your code looks pretty close. Your GET request is probably malformed in some way. Try this: open up a telnet client and connect to a web server. Paste in the GET request as you believe it should work. See if that returns anything. If it doesn't it means there is a problem with the GET request. The easiest thing to do that point would be write a program that listens on a socket (more or less the inverse of what you're doing) and point a web browser to localhost:[correct port] and see what the web browser sends you. Use that as your template for the GET request.
Alternatively you could try and piece it together from the HTTP specification.

I had to add the full URL to the GET parameter. To make it work. Although I see you can specify HOST also if you want.
Socket socket = new Socket("youtube.com",80);
PrintWriter out = new PrintWriter(new BufferedWriter(new
OutputStreamWriter(socket.getOutputStream())));
out.println("GET http://www.youtube.com/yts/img/favicon_48-vflVjB_Qk.png
HTTP/1.0");
out.println();
out.flush();

Yes, it is possible. You just need to figure out the protocol. You are close.
I would create a simple server socket that prints out what it gets in. You can then use your browser to connect to the socket using a url like: http://localhost:8080. Then use your client socket to mimic the HTTP protocol from the browser.

Not sure why you're going lower down than URLConnection - its designed to do what you want to do: http://download.oracle.com/javase/tutorial/networking/urls/readingWriting.html.
The Java Tutorial on Sockets even says: "URLs and URLConnections provide a relatively high-level mechanism for accessing resources on the Internet. Sometimes your programs require lower-level network communication, for example, when you want to write a client-server application." Since you're not going lower than HTTP, I'm not sure what the point is of using a Socket.