I rewrote a simple example of file transfer code between server and client.
And it works.
But i want to make it able to transfer multiple files in a particular directory. User will write the file names (which sits in that particular directory) and client will download them from the server. How can i do that? Any ideas? Thank you.
Client code:
import java.net.*;
import java.io.*;
public class Client {
static String hostname = "127.0.0.1";
static int port = 4588;
static int processedByte;
static byte[] theByte = new byte[1];
static Socket client = null;
static InputStream inuputSt = null;
public static void main(String[] args) throws InterruptedException {
System.out.println("connecting...");
Thread.sleep(500);
try {
client = new Socket(hostname, port);
inuputSt = client.getInputStream();
} catch (IOException ex) {
System.out.println("connection error.");
}
ByteArrayOutputStream arrayOutput = new ByteArrayOutputStream();
if (inuputSt != null) {
FileOutputStream fileOutput = null;
BufferedOutputStream bufferedOutput = null;
try {
System.out.println("downloading target file...");
Thread.sleep(800);
fileOutput = new FileOutputStream("file1_downloaded.txt");
bufferedOutput = new BufferedOutputStream(fileOutput);
processedByte = inuputSt.read(theByte, 0, theByte.length);
do {
arrayOutput.write(theByte);
processedByte = inuputSt.read(theByte);
} while (processedByte != -1);
bufferedOutput.write(arrayOutput.toByteArray());
bufferedOutput.flush();
bufferedOutput.close();
System.out.println("file downloaded");
client.close();
} catch (IOException ex) {
System.out.println("file transfer error.");
}
}
}
}
Server code:
import java.net.*;
import java.io.*;
public class Server {
static int port = 4588;
public static void main(String[] args) {
while (true) {
ServerSocket server = null;
Socket connection = null;
BufferedOutputStream bufferedOutput = null;
try {
server = new ServerSocket(port);
connection = server.accept();
bufferedOutput = new BufferedOutputStream(connection.getOutputStream());
} catch (IOException ex) {
// Do exception handling
}
if (bufferedOutput != null) {
File fileToSend = new File("files\\file1.txt");
byte[] mybytearray = new byte[(int) fileToSend.length()];
FileInputStream fileInputSt = null;
try {
fileInputSt = new FileInputStream(fileToSend);
} catch (FileNotFoundException ex) {
// exception stuff
}
BufferedInputStream bufferedInput = new BufferedInputStream(fileInputSt);
try {
bufferedInput.read(mybytearray, 0, mybytearray.length);
bufferedOutput.write(mybytearray, 0, mybytearray.length);
bufferedOutput.flush();
bufferedOutput.close();
connection.close();
//file1.txt has been downloaded
return;
} catch (IOException ex) {
// exception stuff
}
}
}
}
}
You suggest HTTP as a protocol for your clients and servers -- HTTP is a fine protocol but may be a large implementation hurdle if you want to do the whole thing yourself. The HTTP PUT verb can be used to upload a file, and the benefit of using HTTP in this fashion is that your client and server could communicate with other tools designed to use PUT requests. (PUT is less-used than other HTTP verbs, so not all HTTP tools will work; the curl(1) program does support PUT via the -T command line option. This will be a great implementation aid, should you chose HTTP.)
There are a variety of REST Frameworks that can assist you in writing HTTP software; I have heard good things about Restlet, it would be my recommended starting point.
But you don't have to pick HTTP as your protocol. I think you can learn a lot about networking programming if you implement your own protocol -- it will teach you a lot about API design and sockets programming in a way that would be difficult to learn by using pre-written HTTP protocol tools (and frustrating if you tried to implement HTTP in its entirety yourself).
Consider this conversation:
client -> server: upload file named "flubber" sized 200000 bytes
server -> client: ok
client -> server: flubber contents
server -> client: ok
client -> server: upload file named "blort" sized 10 bytes
server -> client: error, file exists
...
You might want to add new commands to provide for hashing the file on both ends to ensure the file transfer succeeded, commands for sending specific byte ranges (either to append to an existing file or re-start a failed transfer), commands to list existing file names on the server, commands to overwrite existing files, commands to delete files from the server, and so forth. The best part of writing your own protocol is you get to decide what your programs will support. The downside is that you get to test the features you write, and testing some cases may be difficult. (Say, consider that a client may send each character of a command in a different TCP packet. Implementing the buffering to store up an entire command isn't trivial, but it is already done for you with a tool such as Restlet.)
Juan's advice to use multiple TCP sessions isn't strictly necessary -- though it may be the easiest path forward for you. You'll need to add some mechanism to provide the filename to the remote peer, and that might be best done through the "control" channel (the first session running -- similar to FTP) or it might be something you send immediately before the file's contents (similar to HTTP).
I'd like to suggest avoiding multiple connections, though -- each connection requires three times the round-trip time between systems to set up and start transferring bytes. This delay can be extremely annoying, especially when you're trying to transfer hundreds of small files. You can even spend more time setting up connections than you do actually sending data.
But it's your tool -- you get to design it as you wish. Have fun.
I think you need to create a new connection for each file so in that situation you'll be able to transfer files simultaneously.
You may have to modify your server to create a new thread (or get one from a thread pool) for each client connection so it can work with many at the same time.
Then you can run the client once per file.
Cheers
Ok, can you transfer multi files making a ArrayList or List files. Getting into in the array after get out in a filesystem path. I hope help you.
Related
Suppose that I have a multi-threaded web server that only allow clients to perform GET requests for a couple of HTML files. I want to maintain a persistent connection (i.e HTTP Connection: keep-alive) while "dynamically" displaying the content for each request the client makes. Like if they first request index.html then foo.html etc. The problem right now is when I don't close the streams and socket, the program will hang until it happens.
Simply put, the multi-threaded web server consist of a thread pool (Java's ExecutorService) with a ServerSocket that listens to a specific port (e.g 9000) and selects a thread from the threadpool to handle the opening of a client socket to the server. It is basically the same setup as showed in http://tutorials.jenkov.com/java-multithreaded-servers/thread-pooled-server.html.
My modified setup looks like this:
WorkerRunnable.java:
public void run() {
try {
InputStream input = this.clientSocket.getInputStream();
OutputStream output = this.clientSocket.getOutputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(input));
List<String> headers = readInputStream(input)
Request request = new Request(headers);
Response response = new Response(request);
// response.raw() returns correctly formatted HTTP
output.write(response.raw().getBytes(StandardCharsets.UTF_8));
// close the socket if the client specifies Connection: close
if (!request.keepAlive()) {
output.close();
input.close();
} else {
this.clientSocket.setKeepAlive(true);
}
} catch (IOException e) {
e.printStackTrace();
}
private List<String> readInputStream(InputStream input) throws IOException {
BufferedReader reader = new BufferedReader(new InputStreamReader(input));
String line;
List<String> headers = new ArrayList<>();
while ((line = reader.readLine()) != null && !line.isEmpty()) {
headers.add(line);
}
return headers;
}
My problem is that the HTML only will be displayed when the input/output stream (and thus also the socket) are closed. As far as I understand, the Socket.InputStream will basically hang until it receives an EOF token - which it receives when the stream closes. But if I want to maintain a persistent connection, it doesn't really make sense to close the streams and client socket. So I was wondering how to maintain a persistent connection while also displaying the content of multiple GET requests from clients (assuming this is the correct approach)? If not, please let me know if I've approached this task wrongly.
I have tried to flush the output stream as suggested here, but the problem still persists.
I am creating a program where an android device requests a file from a Web Server(running python).The server can receive over sockets with no problem the path of the requested file but i dont know how i can make my android device to wait for a responce.
Here is the android code(as a client requesting a file from web server):
try {
Socket socket = null;
socket = new Socket("192.168.1.9", 4000);
DataInputStream input = new DataInputStream(socket.getInputStream());
DataOutputStream output = new DataOutputStream(socket.getOutputStream());
String str = getURL();
output.writeBytes(str);
output.close();
input.close();
socket.close();
{
}
} catch (IOException e) {
}
Log.d("communicationService", "URL transferred with success");
And the python script running on Web Server(It can receive thefile path but i have problem sending the file)
import socket
import sys
HOST, PORT = '192.168.1.9', 4000
serverSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serverSocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
serverSocket.bind((HOST,PORT))
serverSocket.listen(10)
print 'Server is on and listening to %s ... ' % PORT
while True:
clientSocket, clientAddress = serverSocket.accept()
print 'A client was connected.....'
incomingURL = clientSocket.recv(1024)
print incomingURL
clientSocket.close()
Any advice and tip would be really helpful...
I imagine you should be able to get away with SimpleHTTPServer
If you need to get fancier with a full blown webservice, WSGI is very popular.
On the client side Requests library is by far the easiest way that I've found to make http requests in python. (just had to plug that one because it's that good)
Well i managed to transfer the files in the end(For those that are interested in apps of this kind).What i did was to create another socket and sent a stream back to client.
file = open("path_of_file", "rb")
s = socket.socket()
s = connect((addr,port))
l = file.read(1024)
while (l):
s.send(l)
l.f.read(1024)
file.close()
s.close()
I am new to netty and I followed this example to write a static file server using netty. But whenever the server serves a large js file. It runs into ClosedChannelException.
The following is my code where I write chunkedFile as http response.
When a large js file is being served I get closedChannelException and the raf file is also closed.
Could you help me figure out what I have done wrong here? Also, is there a simple tutorial where I get understand the basic flow of control in netty?
// Write the content.
ChannelFuture writeFuture = null;
try
{
long fileLength = raf.length();
HttpResponse response = new DefaultHttpResponse(
HttpVersion.HTTP_1_1, HttpResponseStatus.OK);
response.setHeader(HttpHeaders.Names.CONTENT_LENGTH, fileLength);
Channel c = ctx.getChannel();
// Write the initial line and the header.
c.write(response);
writeFuture = c.write(new ChunkedFile(raf, 0, fileLength, 8192));
}
finally
{
raf.close();
if (writeFuture != null)
writeFuture.addListener(ChannelFutureListener.CLOSE);
}
}
<
Calling raf.close() in the finally block is wrong as it may not have it written yet. In fact netty will take care to close it after the write is complete.
I'm trying to write a Java HTTP Proxy Tunnelling program, and I need an experts advice about the best and fastest stream to use for the communication.
I've implemented the basic functionality and everything works fine. The only matter is communication speed or performance. My HTTP proxy system consists of a server program, running on a remote server and a client program running on the local machine. So far, the program looks like this:
Listener.java :
/**
* Listens and accepts connection requests from the browser
*/
ServerSocket listener = null;
try {
listener = new ServerSocket(port, 128);
} catch (IOException ex) {
ex.printStackTrace(System.err);
}
ExecutorService executor = Executors.newCachedThreadPool();
Socket connection;
while (!shutdown) {
try {
connection = listener.accept();
executor.execute(new ProxyTunnel(connection));
} catch (IOException ex) {
ex.printStackTrace(System.err);
}
}
ProxyTunnel.java :
try {
byte[] buffer = new byte[8192]; // 8-KB buffer
InputStream browserInput = browser.getInputStream();
OutputStream browserOutput = browser.getOutputStream();
// Reading browser request ...
StringBuilder request = new StringBuilder(2048);
do {
int read = browserInput.read(buffer);
logger.log(read + " bytes read from browser.");
if (read > 0) {
request.append(new String(buffer, 0, read));
}
} while (browserInput.available() > 0 && read > 0);
// Connecting to proxy server ...
Socket server = new Socket(SERVER_IP, SERVER_PORT);
server.setSoTimeout(5000); // Setting 5 sec read timeout
OutputStream serverOutput = server.getOutputStream();
InputStream serverInput = server.getInputStream();
// Sending request to server ...
serverOutput.write(request.toString().getBytes());
serverOutput.flush();
// Waiting for server response ...
StringBuilder response = new StringBuilder(16384);
do {
try {
read = serverInput.read(buffer);
} catch (SocketTimeoutException ex) {
break; // Timeout!
}
if (read > 0) {
// Send response to browser.");
response.append(new String(buffer, 0, read));
browserOutput.write(buffer, 0, read);
browserOutput.flush();
}
} while (read > 0);
// Closing connections ...
server.close();
} catch (IOException ex) {
ex.printStackTrace(System.err);
} finally {
try {
browser.close();
} catch (IOException ex) {
ex.printStackTrace(System.err);
}
}
The server program uses a similar fashion and sends the HTTP request to the destination server (e.g. www.stackoverflow.com) and forwards the response to the client program, where the client program forwards the response to the local browser.
How can I improve the performance of these TCP/HTTP communications?
Does using buffered streams such as BufferedInputSream and BufferedOutputStream improve communication performance?
Will I gain any performance improvements if I use java.nio Channels and Buffers, instead of java.net Sockets and java.io Stream?
Don't do it yourself
Advice 0: there are plenty of proxy servers out there, much more scalable, stable and mature. Do you really need to write your own?
Don't use StringBuilder/String to buffer request
byte[] buffer = new byte[8192]; // 8-KB buffer
//...
browserInput.read(buffer);
//...
request.append(new String(buffer, 0, read));
//...
serverOutput.write(request.toString().getBytes());
This is flawed for several reasons:
you are assuming your HTTP calls are text (ASCII) only, binary data will be malformed after transforming to String and back to byte[], see: String, byte[] and compression
even if the protocol is text-based, you are using system's default encoding. I bet this is not what you want
finally, the most important part: do not buffer the whole request. Read chunk of data from incoming request and forward it immediately to target server in one iteration. There is absolutely no need for the extra memory overhead and latency. Immediately after receiving few bytes dispatch them and forget about them.
Don't use Executors.newCachedThreadPool()
This pool can grow inifinitely, creating thousands of threads during peak. Essentially you create one thread per connection (except that the pool reuses free threads, but creates new if none available). Consider Executors.newFixedThreadPool(100) - 100-200 threads should be enough in most cases. Above that you'll most likely burn your CPU barely in context switching, without doing much work. Don't be afraid of latency, scale out.
Use non-blocking netty stack
Which brings us to the final advice. Drop blocking sockets altogether. They are handy, but don't scale well due to thread-per-connection requirement. Too much memory is spent to hold stack, too much CPU is wasted for context switching. netty is great and it builds powerful abstraction over NIO.
Check out the examples, they include HTTP client/server code. There is a bit of a learning curve, but you can expect performance growth by several order of magnitued.
I have a java server which is using TCP and sockets to connect to an Android application (the client) and sends strings (currently taken in from a scanner object) which are then displayed as notifications by the client.
heres the Server code without all the imports.
public class Server {
// define our Main method
public static void main(String[] args) throws IOException {
// set up our Server Socket, set to null for the moment.
ServerSocket serverSocket = null;
boolean isConnected = false;
// Lets try and instantiate our server and define a port number .
try {
serverSocket = new ServerSocket(6789);
isConnected = true;
System.out.println("*** I am the Server ***\n");
// make sure to always catch any exceptions that may occur.
} catch (IOException e) {
// always print error to "System.err"
System.err.println("Could not listen on port: 6789.");
System.exit(1);
}
// We always want to check for connection from Clients, so lets define
// a for ever loop.
for (;;) {
// define a local client socket
Socket clientSocket = null;
// lets see if there are any connections and if so, accept it.
try {
clientSocket = serverSocket.accept();
// don't forget to catch your exceptions
} catch (IOException e) {
System.err.println("Accept failed.");
System.exit(1);
}
// Prepare the input & output streams via the client socket.
// fancy way of saying we want to be able to read and write data to
// the client socket that is connected.
BufferedReader inFromClient = new BufferedReader(new InputStreamReader(
clientSocket.getInputStream()));
PrintWriter outToClient = new PrintWriter(clientSocket.getOutputStream(),
true);
while (isConnected) {
// read a sentence from client
String hiFromClient = inFromClient.readLine();
// Set up the logging system and timestamp and log message.
Calendar currentDate = Calendar.getInstance();
SimpleDateFormat formatter=
new SimpleDateFormat("yyyy/MMM/dd HH:mm:ss");
String dateNow = formatter.format(currentDate.getTime());
try{
// Create file
File fstream = new File("log.txt");
FileWriter out = new FileWriter(fstream);
out.write(hiFromClient + " " + dateNow);
//Close the output stream
out.close();
} catch (Exception e){//Catch exception if any
System.err.println("Error: " + e.getMessage());
}
// Print the client sentence to the screen
System.out.println("The Client said: "+hiFromClient);
// Reply to the client
Scanner scanner = new Scanner(System.in);
String sentence = scanner.nextLine();
outToClient.println(sentence );
System.out.println("The Server said: " + sentence);
}
// always remember to close all connections.
inFromClient.close(); // the reader
outToClient.close(); // the writer
clientSocket.close(); // and the client socket
}
}}
Growl uses port 23053 for notification forwarding. What i am hoping to do is to listen in on 23053 and send anything in from that as a string to the client connected at 6789. Sadly Growl binds the port number so a new Socket connection cant be made.
Any one have any ideas on how i could get notifications from the port number growl uses or even just use growl as the server for the client itself (the client's code is very similar to the servers by the way just using Socket instead of ServerSocket)
Any help on this would be greatly appreciated, its wrecking my brain
All the best,
Patrick.
There is a round-about way you could do it. If you are desperate, read on:
Growl can forward any notifications it receives to another machine running Growl (configured on the Network tab). Growl uses the GNTP protocol (a TCP-based protocol: http://www.growlforwindows.com/gfw/help/gntp.aspx) to forward the messages. The trick is that that 'other machine running Growl' doesnt have to really be another machine OR running Growl per se, it just needs to appear to Growl that it is. Growl (on the Mac, which is what I assume you are using) will automatically detect any other machines on the network running Growl (using Bonjour and looking for the _gntp._tcp service name), so if your server advertises itself as supporting the GNTP protocol, Growl should show it in the list of available destinations. (Growl for Windows also lets you manually add a hostname/port to forward to, but I dont believe the Mac version currently allows that).
So then you could configure Growl to forward notifications to your server using its already-built-in capabilities. You would have to implement code in your server to receive the GNTP packets (the format is very similar to HTTP headers) and parse them. Then you could forward the notifications using your current server code.
Still with me? I did say it was round-about, but it is not only technically possible, but I have built a Growl-impersonating daemon before so that I could have notifications forwarded from Growl to my custom code. Not suggesting it as the best idea, but just an alternative since you asked.