We have a Java web application which collects the inputs from the user and executes the legacy simulation tools in a remote machine by sharing the inputs in the JSON files.
The simulation tools doesn't have the option to run via HTTP request, therefore we have opted SSH. But there are some bottlenecks as below per the current design,
In order to avoid the manual steps of SSH authentication by keys/password, we have implemented Kerberos ticket based authentication which breaks often when windows updates, user changes the password and some other scenarios
Input and results are in JSON file formats in specified directories and each workflows will process it based on workflows the worflow id's matches with the directory name
Monitoring is the very tedious task as per current design. Because few of the workflows run for hours-days. So we need to update the application with the current status as Running-UpToDate-Compeleted/Error. Workflow will be running in commandline mode, it will update the logs/progress from remote system to server machine via SSH and we write this content into files. once user clicks on Refresh button, Application checks the progress file in specific directory and update the status. But it involves additional steps to get the current status and also it is slow because it involves file operation and copying from one machine to another.
Is there anyway we could optimize our design?
Also I found the other options like TCP/IP and RPC. To communicate with Remote systems and trigger the workflow. Our application is Spring boot/React web based, is it fine to mix http and TCP/IP in single application?
We are using Waffle SSO with Spring security, how to implement TCP/IP without compromising security.
I'm working on a Project which aims to develop an app for a bloomberg anywhere subscriber. We have a Framework available, which we usually use to write our apps. These run ususally on a cloud as webapplications.
As apps using bloombergs open API can only run on the same machine as the subscriber is logged in biometrically, due to data acess restrictions and prohibition to distribute the data further, we plan to run our framework locally on the clients machine as a desktop application, in order to use the tools we are familiar with.
the back-end, which would use the bloomberg open api to get the data from the local interface provided by the local bloomberg-software, runs on a tomcat-server. the frontend is a javascript client, accessing the back end via API-calls locally.
My Questions are:
As bloomberg data-access is very restrictive, i'm wondering whether the bloomberg-interface accepts calls which come from a server software (like tomcat), even when running on the same machine an not used to distribute the data further?
Would set-up: framework running on a second machine (linux, its easyer to run there), calls Bloomberg Port on first machine via private local network, data not leaving second machine, be possible 1. technically and 2. legally?
Does anybody know that? Any advice highly appreciated, many thanks in advance!
I would like to download somefiles which is in remote shared folder from the local system using java code. Can you please guide us, how i can proceed this. Whether i need use http or ftp. But i want to execute the code only in the local machine. It should not like client server program. Please post some tips over here.
Thanks in advance.
You cannot solve this without knowing the platform of your machine (where the app runs) and remote machine. But once you know you have many possibilities.
Agent full solution
Install agent on remote machine. The agent should expose API that allows access to the files. For example you can use FTP server as an agent. Then access this server and download your files. Alternatively you can implement your own application or servlet running in one of available servlet containers and provide access to files over HTTP.
Agent-less solution
Use the built-in functionality of target machine. For example use SSH to connect to Unix machine and SMB for windows. Both support Telnet
You can use VFS package from jakarta. There are several implementations of SSH. I recommend JSH.
I want to place a server program written in Java on the cloud. It would accept TCP socket connections from clients (clients are android phones using 3G), do some computations, save stuff to a MySQL database (also on EC2), and send stuff back to the clients over the TCP connections. It may even be necessary to create several instances of the server (i.e. a process group).
Is this easy to do? I think I can make a AMI, but I'm not sure how to upload Java files, compile and run them, and create a MySQL database etc
Any help would be much appreciated.
Take a look at using Amazon Elastic Beanstalk. Beanstalk is Amazon's PaaS offering and it will alleviate a lot of the system administration burden. Here's a quick description from their docs:
AWS Elastic Beanstalk is an even
easier way for you to quickly deploy
and manage applications in the AWS
cloud. You simply upload your
application, and Elastic Beanstalk
automatically handles the deployment
details of capacity provisioning, load
balancing, auto-scaling, and
application health monitoring.
Also, if you're interested in using MySQL then you should look at Amazon RDS. Again, this will alleviate the system administration burden for your database tier. Here's a quick description from their docs:
Amazon Relational Database Service
(Amazon RDS) is a web service that
makes it easy to set up, operate, and
scale a relational database in the
cloud. It provides cost-efficient and
resizable capacity while managing
time-consuming database administration
tasks, freeing you up to focus on your
applications and business.
Is this easy to do? I think I can make
a AMI, ...
I think the answer depends on how comfortable you are with system administration in general. Creating a AMI to run in EC2 is really pretty much the same as creating a physical server or a VM image. You'll need to install an operating system, and then install tools, libraries and programs you need (like mysql, the jdk, ssh, etc).
You can save yourself a little work by using one of Amazon's pre-built AMI's http://aws.amazon.com/amis/. But, ultimately, you'll be responsible for all system administration of the server. If you've never built a server from the ground up, you have a pretty big learning curve ahead of you. It's not insurmountable, but just be warned that the devil is in the details; there's a ton of stuff you'll need to learn ;-)
... but I'm not sure how to upload Java
files, compile and run them, ...
Once the server is setup and running in EC2, compiling them and running java files is just the same as compiling and running on your local. Normally, you probably want to compile and package your java app into a jar or war and then transfer that up to your EC2 server. If you install linux os on your EC2 server, you can use scp or a FTP client to transfer your files over sftp to move the files from your local up to the server. Once the latest files are up on your server, you can ssh to the server and start your app.
... and create a MySQL database etc ...
Installing mysql is going to be specific to the OS you choose to install on your server. For example, you can install mysql easily on Ubuntu with a command like:
sudo aptitude install mysql
Again, there will be more system-admin-type stuff to learn here specific to mysql databases.
So, it's definitely doable. An experienced sys admin could build a AMI instance pretty easily/quickly. If this is your first experience with system administration, I'd suggest finding an old Desktop you have lying around and try installing Ubuntu and all the required libraries and tools you need (mysql, jdk, ssh, etc..). Get your java program working on the old desktop and then it should be pretty easy to create an AMI from that. Then you can run your custom AMI on EC2 and will be set up.
If you don't have a spare desktop lying around, you can use one of the Virtual Machine products like VMWare Player or Sun's VirtualBox and build a server instance on one of those.
If you want to avoid the hassle of managing the entire install of the Operating system, you might want to look at services like slicehost and/or linode instead of EC2. They give you ssh access to a pre-installed server. And it's as easy as clicking a button to install programs like mysql, etc.
Hope this answer is relevant and helpful, good luck.
- Dave
If you can use Tomcat as your server - you might want to try Amazon Web Services Elastic Beanstalk It will greatly simplify your task by providing an easy instance of Tomcat to deploy.
EDIT: AWS has a full section on how to develop using Java here: http://aws.amazon.com/java/
How comfortable are you with remote administering a server and solving problems most people have never heard of? That's what you're talking about.
You'll create your EC2 instance, log into it and configure it like you would any other server you're working with. You can download JDKs to it, dbs etc. You might consider using a tool like Chef to help you. You'll use ftp and scp to copy files to the server.
You're probably going to want your Java server on one box and have it talk to a separate db server since you say you may want multiple servers.
Once the server is working the way you like it you can create an image of it to use to launch multiple instances, then configure a load balancer to point at your servers.
If you can create a MySql db on your local box you can create it in the cloud. If not?....
Running Applications needs app. s/w to run them.
Apps such as Cyber Duck helps to upload files via SFTP.
I am successful in doing the same problem addressed.. Trust me, YOU CAN DO IT. All u need is interfaces for server services such as MySQL (use WorkBench to connect using key pair), terminal access (using PUTTY/SSH via MAC) and well you are good to go
I think to get started is to use a a whole unit packed into one archive.
Think of it as you have your Java files compiled, a embedded http server say tomcat. Now all of this packed in a jar ready to be deployed on Amazon's EC2. Use this link for more on embedded servers and relevant code.
As far as database is concerned, you can use Amazon's RDS. You can configure mysql on RDS and obtain a link to it for connection. Amazon RDS would help you to get started with database.
Now that you have everything ready to start and work with application. Now is the time to deploy on Amazon AMI
Perform Following steps to deploy you app on Amazon EC2:
Create a server instance on Amazon EC2 refer EC2 getting started.
Receive Private key file i.e. .pem(receive from Step 1), this file would help you to login to your server instance and perform SFTP.
Use Putty or similar SSH client to login to your EC2 server instance created on Step-1 using .pem file and server's public DNS, refer Accessing Instance for SSH login.
Transfer the archived files packed into jar to server using SFTP client, WinnSCP being one such client, refer SFTP for more on it.
Run application. Note in case of jar file you can simply fire java -jar TomcatApp.jar.
If everything is configured properly you would be to access the app using EC2 instance's public DNS or public IP, which would be of the form:
http://<public_dns_address>:<port number>/servlet
Hope it helps you to get started and provide you an overall view.
I am developing a java based application; its pertinent requirements are listed below
Large datasets exist on several machines on network. my program needs to (remotely) execute a java program to process these data sets and fetch the results
A user on a windows desktop will need to process datasets (several gigs) on machine A. My program can reside on the user's machine. He will execute my program from his machine and initiate the dataset processing on remote machine(s)
Instead of getting the dataset over the network from the remote machine to his machine, he will execute the program on the remote machine and fetch results
The user may have open access to the other machines but ftp is the requirement
Data should not be brought through network to the user's machine.
Users have windows OS
My question(s)
How can I perform this kind of remote process execution ? Any ideas?
I am looking at hadoop; I am working on Windows XP. I was unable to get hadoop working for a single node cluster; I am unable to find good documentation. I therefore haven't quite tested hadoop. Any comments on if I am on the right track?
Any links any of you has found useful for installation of hadoop and trouble shooting?
Thanks in advance for any responses. Do please let me know if I should provide any more/specific details.
-jv
Java has a RMI API that you could use, assuming that you can have a JAVA VM running on your remote machines. That's the lightest weight solution. The next lightest weight would be straight socket communication. After that you're getting into EJB servers or Web Servers, which is probably overkill.
Have a look at how to write web services with Java 6. That allows you to publish a method as a web service with an annotation. A web service client is small and does not require additional software. I found the Idea IntelliJ IDE easy to use, and generated a pure Java 6 client.
Then it essentially boils down to making a "normal" method call, and processing the result.
Keep it simple. Grid software is most likely not what you want.