I have this project that involves both a client and a server. I developped both parts of the application in java and I want to test it in a hadoop cluster, since the server side is a simulation of a cloud, so by using hadoop I want to give my application a real sense of cloud environment. I started by creatin a multi-node Hadoop Cluster but I don"t know what should be the next step.
I would appreciate any enlightening.
The proper way to accomplish this would be to use a restful interface to send the commands.
For instance, on the computer that is the JobTracker, you could host a tomcat rest server. (Make sure that the hadoop dameons are running in the background). You could use a Spring/Hibernate based servlet in order to process the requests to the server. Finally, in the servlet, you could either include the hadoop-jars and call hadoop through the hadoop API, or you can call hadoop through the console (./hadoop runjar blah).
In order to upload files to the Server, you can use an sftp interface, or possibly directly upload files to the hdfs.
If you would like, I can share some code with you, because I have done a similar project.
Also, you can look into Apache Oozie. They host a restful job flow api for hadoop.
Related
I want to use Apache Solr, or Elastic Search, or some form of Apache Lucene on my website. Do I have to have a completely separate server for it to run on?
My website runs on AWS on a Tomcat instance. I know that I can simply spin up an elastic search instance on AWS and use that. My goal is to do it without having to spend that money, and just use it on the server I already have. Is this possible?
The question is usually what problem you want to solve, and then deciding whether Elastic or Solr can be the solution to that problem.
But yes, you can run both elasticsearch and Solr on your existing server. You'd install either by following the installation guides in their manuals, and they'll both run as server daemons separate from your existing Tomcat installation.
Older versions of Solr was distributed as a .war file that you could run in Tomcat, but this is not (officially) supported any longer, and is not recommended (at it would require a bit of manual hacking).
You'd then write code in your existing webapp to query Solr or Elasticsearch for the information you want to retrieve.
My requirement is to serve a web service request using Apache Spark.
I developed a XML RPC server which will be triggered by a servlet code and it will in-turn run a spark-submit application. XML RPC will then wait for spark-submit to complete and send the result back to servlet.
My questions are:
Can I write a XML RPC server inside my spark job, and serve all request directly from my servlet with using standalone XML RPC server?
What is the correct method to trigger Apache spark for each web service request?
Integrating spark with servlets can be challenging due to dependency issues, but leaving that aside, you might be better suited to having a long running driver program with a shared spark context since creating a spark context can take time (and you can't share cached data between different spark contexts). The Ooyala Spark job server can be useful for this, IBM also has a project intended for doing something similar called the Spark Kernel. Hope one of these projects
I've read some articles on the Internet that this is not possible. To communicate own SQL database that is located on other server from GWT application. Jetty doesn't allow it.
I found a way how to perform it but it's not very cosy. I have client and server part inside GWT. The server has to communicate with MySQL database on localhost. So I've written an ant script to build a war that I can launch on Apache Tomcat server. It works perfectly there but I'm not able to debug the code effectively.
Do you have some advices how to perform this task? I was thinking of writing the clienty only in GWT and find some waz how to communicate my own server written outside the GWT. I've found Apache Thrift for GWT but this edited library of thrift seem not to work properly.
Thank you very much for your answers:)
It is possible to communicate with a database from a GWT application. The client side has to call the methods of the server via GWT-RPC, which can communicate with any database.
Maybe Jetty does not support it (have not tested it personally) but you can develop your web application using Apache too. There you can access the database the same way as from any web application:
You will need the mysql-connector-java-5.1.20-bin.jar file (downloadable from: http://dev.mysql.com/downloads/connector/j/ ), and restart the server added to the $CATALINA_HOME/common/lib directory.
OR added to the WEB-INF/lib folder of your web application.
You can find tutorials online of how to develop an application using Tomcat instead of Jetty. For example: https://wiki.auckland.ac.nz/display/BeSTGRID/Deploying+GWT+to+Tomcat+in+Eclipse
Reshi, stop and think about how applications really work. Nobody provides web pages with javascript to read/write databases, that would be crazy and unsecure. Servers are always in themiddle of all communication in this case. You need to create services that run inside your server, one of these services will be a database layer.
Javascript cant create network connections and read/write binary data, thus it would be insane to attempt to get the gwt compiler to compile any jdbc drvier and more.
Jetty does NOT stop us from connecting to a database. All you have to do is to follow the MVP model way. Although MVP is better bet against all hurdles, at a minimal point, you would have to try not having SQL code on the client package.
I have an existing Java application running on a linux box that monitors the state of relays and network information. This is a stand-alone device currently.
What I would like to further develop is a web server for the device to remotely monitor, configure, and control the device via an Ethernet connection. To achieve this, I need some way to interface between the web server, which could be configured as enabled/disabled, and the master device, which is always running.
Is there a good way to do this? I have looked at apache w/ tomcat and similar web servers but not sure if this is what I need. The key here is that the web server needs to be able to access the existing java application without interfering with its always running services.
You either develop a webapp, use your Java application's API inside the webapp, and deploy this webapp inside a web container. Or you can do the reverse and embed a web server inside your application (see here for documentation to embed Jetty).
If you want to keep the webapp and the original application in two separate JVMs, you'll need some wey to communicate between both, like sockets, RMI, or even files, but it will be more complex.
You might want to take a look at JMX http://docs.oracle.com/javase/tutorial/jmx/overview/index.html
I want to run a standalone java application on a remote server. It would not be accessible to clients, but would do background calculations and interact with a database and Secure Socket connection to a third party site. It would also interact with a php site.
Do I have to deploy this with JSP, or can I write a standalone application? If so, how would I deploy a standalone java application (jar file) on a remote server? I understand that I must have them install a jvm on the server (not a problem) but then how would I deploy it (if possible). Would I start it with a command line?
I know I have much to learn, but I am not sure how I would access the command line on a remote server. Through the cPanel?
Thanks.
First of all, you'll want to set up some firewall rules to allow access to that server. I'm hoping that you don't expose that server naked to the Internet.
If all you need is database access exposed on the Internet, I don't see why it can't be a secured web app deployed on a servlet/JSP engine and accessed via a web server. You can leverage basic auth for security, JDBC access to the database from the server, and servlets as controllers to accept requests in a nice REST API.
It'll save you the complications of sockets and inventing your own protocol (use HTTP), starting and stopping the application (now it's just a web server/servlet engine), and deployment (send a WAR file).
Does it really must be a 'standalone' application? I think that in your case the best match would be to use Spring container to load your application within some server (tomcat?) and expose services via standard controllers - with Spring you only have to add some annotations on services methods actually.
Then, your php site can interact with these controllers using for example ajax requests.
If your application is written already, you can easily transform it to run within Spring container. It's very non-invasive and promotes usage of POJOs.