My requirement is to serve a web service request using Apache Spark.
I developed a XML RPC server which will be triggered by a servlet code and it will in-turn run a spark-submit application. XML RPC will then wait for spark-submit to complete and send the result back to servlet.
My questions are:
Can I write a XML RPC server inside my spark job, and serve all request directly from my servlet with using standalone XML RPC server?
What is the correct method to trigger Apache spark for each web service request?
Integrating spark with servlets can be challenging due to dependency issues, but leaving that aside, you might be better suited to having a long running driver program with a shared spark context since creating a spark context can take time (and you can't share cached data between different spark contexts). The Ooyala Spark job server can be useful for this, IBM also has a project intended for doing something similar called the Spark Kernel. Hope one of these projects
Related
I have had lots of trobule trying to get my head around how to solve this scenario:
We have an integration application that uses Camel for integration. This application also has a REST Api that exposes some services providing information about the application, for instance listing the active routes etc.
I have created a user interface for this using AngularJS that connects to these rest services. My main problem is how can I package this application as a self contained jar-file that provides the user interface and all the camel integration.
My working theory: Use a separate Jetty server to serve the Angular JS files and let Camel expose the REST services. The problem with this is CORS since the REST services reside on another port than the jetty server serving the Web UI.
Some requirements for the solution:
Must be a single self contained jar-file.
The camel integration is the main purpose, the Web UI is secondary
and only used for trouble shooting. No need for a high performance
web container since the Web ui is used by only a handful of users.
I have been struggling with this for a couple of days now and it feels like I am over complicating the solution. Help on how to solve this is greatly appreciated.
You could take a look at hawtio
http://hawt.io/
as that is how we do that, hawtio is a web console for java, and has plugins for Camel. Its built using angularjs, and uses REST to communicate with the local or remote Java JVMs. To make the REST calls easier we use Jolokia.
Jolokia requires an agent to be embedded in the JVM, eg where Camel runs. Then that helps with CORS et all. http://jolokia.org/reference/html/security.html#d0e2490
I am new to neo4j and based on the reading I have done so far it seem there are two ways to interact with neo4j using Neo4j REST and Embedded. Where I am a little confused is does the Embedded option only give you the ability use the native Neo4j API to manipulate the datastore or can you also embed Neo4j and package it with your java application and if so how would I go about doing it?
As far as I know, Embedded term coined out to integrate neo4j with your application. In embedded mode, your db is locked and your application is solely authorized to access it. You can not access your db from any where else as far as your application is running and accessing it.
Where as in Neo4j Rest or Say Neo4j Server support REST API through which you can perform all the data store related operation via API call. In Rest API mode, you can handle your db externally using Neo4j GUI console along with your application.
Performance wise, I found embedded mode is much faster than Server mode.
does the Embedded option only give you the ability use the native Neo4j API to manipulate the datastore
You can use either of mode (Server REST API mode or embedded mode) to manipulate datastore.
Package with Java Application
it depends on your application configuration, in embedded mode you generally don't need external neo4j server running. You just need to explicitly mention your db path along with other configuration (I have used Spring data neo4j). Where as in Neo4j Server mode, you will require neo4j server running.
You can have look on this thread as well.
I have this project that involves both a client and a server. I developped both parts of the application in java and I want to test it in a hadoop cluster, since the server side is a simulation of a cloud, so by using hadoop I want to give my application a real sense of cloud environment. I started by creatin a multi-node Hadoop Cluster but I don"t know what should be the next step.
I would appreciate any enlightening.
The proper way to accomplish this would be to use a restful interface to send the commands.
For instance, on the computer that is the JobTracker, you could host a tomcat rest server. (Make sure that the hadoop dameons are running in the background). You could use a Spring/Hibernate based servlet in order to process the requests to the server. Finally, in the servlet, you could either include the hadoop-jars and call hadoop through the hadoop API, or you can call hadoop through the console (./hadoop runjar blah).
In order to upload files to the Server, you can use an sftp interface, or possibly directly upload files to the hdfs.
If you would like, I can share some code with you, because I have done a similar project.
Also, you can look into Apache Oozie. They host a restful job flow api for hadoop.
I've read some articles on the Internet that this is not possible. To communicate own SQL database that is located on other server from GWT application. Jetty doesn't allow it.
I found a way how to perform it but it's not very cosy. I have client and server part inside GWT. The server has to communicate with MySQL database on localhost. So I've written an ant script to build a war that I can launch on Apache Tomcat server. It works perfectly there but I'm not able to debug the code effectively.
Do you have some advices how to perform this task? I was thinking of writing the clienty only in GWT and find some waz how to communicate my own server written outside the GWT. I've found Apache Thrift for GWT but this edited library of thrift seem not to work properly.
Thank you very much for your answers:)
It is possible to communicate with a database from a GWT application. The client side has to call the methods of the server via GWT-RPC, which can communicate with any database.
Maybe Jetty does not support it (have not tested it personally) but you can develop your web application using Apache too. There you can access the database the same way as from any web application:
You will need the mysql-connector-java-5.1.20-bin.jar file (downloadable from: http://dev.mysql.com/downloads/connector/j/ ), and restart the server added to the $CATALINA_HOME/common/lib directory.
OR added to the WEB-INF/lib folder of your web application.
You can find tutorials online of how to develop an application using Tomcat instead of Jetty. For example: https://wiki.auckland.ac.nz/display/BeSTGRID/Deploying+GWT+to+Tomcat+in+Eclipse
Reshi, stop and think about how applications really work. Nobody provides web pages with javascript to read/write databases, that would be crazy and unsecure. Servers are always in themiddle of all communication in this case. You need to create services that run inside your server, one of these services will be a database layer.
Javascript cant create network connections and read/write binary data, thus it would be insane to attempt to get the gwt compiler to compile any jdbc drvier and more.
Jetty does NOT stop us from connecting to a database. All you have to do is to follow the MVP model way. Although MVP is better bet against all hurdles, at a minimal point, you would have to try not having SQL code on the client package.
I want to run a standalone java application on a remote server. It would not be accessible to clients, but would do background calculations and interact with a database and Secure Socket connection to a third party site. It would also interact with a php site.
Do I have to deploy this with JSP, or can I write a standalone application? If so, how would I deploy a standalone java application (jar file) on a remote server? I understand that I must have them install a jvm on the server (not a problem) but then how would I deploy it (if possible). Would I start it with a command line?
I know I have much to learn, but I am not sure how I would access the command line on a remote server. Through the cPanel?
Thanks.
First of all, you'll want to set up some firewall rules to allow access to that server. I'm hoping that you don't expose that server naked to the Internet.
If all you need is database access exposed on the Internet, I don't see why it can't be a secured web app deployed on a servlet/JSP engine and accessed via a web server. You can leverage basic auth for security, JDBC access to the database from the server, and servlets as controllers to accept requests in a nice REST API.
It'll save you the complications of sockets and inventing your own protocol (use HTTP), starting and stopping the application (now it's just a web server/servlet engine), and deployment (send a WAR file).
Does it really must be a 'standalone' application? I think that in your case the best match would be to use Spring container to load your application within some server (tomcat?) and expose services via standard controllers - with Spring you only have to add some annotations on services methods actually.
Then, your php site can interact with these controllers using for example ajax requests.
If your application is written already, you can easily transform it to run within Spring container. It's very non-invasive and promotes usage of POJOs.