I have a situation where I need to show the progress of multiple FTP clients which are accessing the same file on a location X in FTP server. I need to show this progress from the server itself, not from the client. (The operation is initiated from the server like the server communicates the client the FTP site user name/password, file location and then the client does a health check of itself and starts a FTP get operation).
I know from the client there is a deterministic way of calculating the number of bytes downloaded with the actual file size so that it would be easy to show that on a percentage or time scale. But I was wondering waht would I do if I need the same capability from the FTP server itself.
Related
I have a Requirement to send Data from Server (Tomcat : Java Process, Odata API's) to Client (React Based)
Data can range from few KB's to Hundred's of MB (Say 700 MB) which is retrieved from DB : RedShift , Processed and Sent to Client.
There can be multiple clients accessing at the same time as well to keep more stress on the system.
We added Pagination so that data for that page alone is loaded, but we have a functionality to export complete data set in CSV format.
Processing of all the data is consuming lot of memory and application's heap gets exhausted sometime, Increasing heap is not the solution expected, I want to know from Application side anything can be done to Optimize system resources.
Kindly suggest what could be the best way to transfer data also whould like to see if there are any other kind of API(Streaming) which can help me here
Can you change the integration between client and your system?
Something like: the client sends the request to export a CSV with a callback url in payload.
You put this request in a queue (rabbitmq). The queue consumer process the request, generate the CSV and put it in a temporary area (S3 or behind a NGINX). Then your consumer notifies the client in the callback url with then new url for the client to download de full CSV.
This way, the system that process the incoming requests don't use too much heap. You only need to scale the queue consumer, but it's more easy because the concurrency is your configuration of how many consumers are consuming the messages, not the incoming requests from the clients.
I have been working on a desktop app using AppJS (HTML5, JQUERY). In my application, I ask user to input a folder path of his/her local machine (where application is running) and store that info in my database on the server.
I have to push a file to that folder (on the user's machine) whenever the file gets updated on the server. how can I achieve this? how I can keep client and the server in sync so that server can push that file to local machine.
I have written the scheduler app for the server which will push the file down to client. But, I am not getting how to push that file to specific client. what parameters I need to push the file to client like client's machine MAC Address or something else.
how dropbox does sync?
Please help!!!
If you are able to run nodejs on the server then you can use socket.io to communicate between client and server. Think of it like a chatroom like msn or skype. As users open the appjs application it can connect to the server.
When you have a file to push send a message like "updatedFile" and then list the url. Clients can then download the file and save to the local disk.
This way all connected clients can download updates and no need to poll all of the time - just have websocket open.
When a client connects it could send a message changesSince timestamp and receive a list of all files changed on server since that time which it can then download and save to local disk one by one.
You can try using tools like rsync which can directly help you achieve this.
http://en.wikipedia.org/wiki/Rsync
In my webapp users can download the files among themselves. If a user A has shared a file F , then user B after connecting to A can download the file F from A. Till now each user makes a simple HTTP connection like :xxx.xxx.xxx.xxx/FileList with another user. The file resides on the local hard disk of each user. So that a user can download a file there were two options in my mind.
As the user shares a file,copy that file into the web-app directory of the server,so that the download link becomes as simple as Click to download.
Run a separate FTP server on each node.
I don't know which one of these is a better option but the first one seems very simple to me. What are the ways each client can share the files,without having to copy the stuff somewhere in the webapp directory. How in this case I can use a P2P protocol ?
NOTE : I am using Tomcat 7.
Real P2P is impossible without opening a listening socket on the client machine (that imposes you have to install something on client machine).
If you don't want to STORE the files on the server, I would rather recommend a "connection server", which serves as a gateway between the two users. User A will upload, user B will download at the same time, all you need is to make the bytebuffer in memory. The downloaded bytes can be dropped.
You can write a small client-side program in any language for updating the available files, and receiving the upload request from the server side (also execute the upload)
I would recommend using TCP sockets for upload to the server side, this way you have direct control over the uploaded bytes (streams).
There are some interesting technical issues here (blocking streams, metadata (filename, length, createdate, ...), data consistency, error handling, etc.) that should be taken into consideration. Nice task.
I don't recommend FTP because you cannot control the authentication and authorisation (who can see the files).
I have several PC's on each of them I set small swing application that get data with JSON request to one web server. Can I receive the data from web server without to send request to the web server, with other words can the Web server send the data without the Java application to ask for this?
If you have enough server resources
you can consider usage of websockets.
Every PC can open a socket to the server.
When you open the socket you need to send to the server, the pc's unique ID.
Then you need to store this ID in some database or file that will contain all online pc's and sockets .
Then the Server will be aware which pc's are online and which socket to use to communicate with this pc. After this you can send whatever information you need to this PC depending on your application.
This can be implemented in several ways. One common way would be to open a connection and do blocking read in the client application. On receiving something it will look like push from the server. Then you process the push and do another blocking read.
Another option would be doing regular checks if there is something for you on the web server. You set the retry interval frequent enough so it will look like real time push from your app point of view.
If you use HTTP i think the smartest way is to drop the realtime requirement and use a thread that polls the server every 5 seconds. Keeping a HTTP Connection open all time is expensive as it blocks a request processor thread and limits the amount of clients you can have.
You might also consider moving to something like a registration mechanism if you really need near-realtime updates which is often not the case. You would have to open a Server on the clients and have the server push the updates after clients registered their Address with the server.
My question is about reading a remote XML file by using Java.
My files are stored in one device that runs Windows CE. I should access to few of these devices several times per day.
Which solution is more efficient considering network constraints, stablishment of a TCP session and data loss: to open and read the file remotelly or get a copy locally to the server and process it then?
Thank you very much.
It seems u want the files to be read from client by server, whereas it most cases its the other way round. In this case you should have some push functionality from the client to server and this can be over HTTP.
Or you can have a Http connection listener running in the client which accepts request from server and sends back the XML file to the server. Essentially its like a server thread running in the client.
Not sure if u running JAVA on Windows CE. Look for solutions in Windows CE HTTP listener.
See if it helps