I'm planning a web application where users will be able to upload and process their files. The specifics of the application are irrelevant to my questions, but lets assume that the application will deal with mp3 audio files. I'm going to split my application in two distinct parts: the front-end and the back-end.
The front-end application will be a usual web application serving html pages to users. Typically a user will upload his file and fill an html form to specify which operations he would like to perform on the file. The files will be initially uploaded to a storage facility, such as Amazon S3, and later processed by a back-end server. I'm using Play 2.0.4 framework to develop the front-end application and this is going very well for me. I managed to implement user authorization, drafted most of the UI and also implemented file upload to S3. The application is currently deployed on Heroku without any problems.
For my back-end server I'm considering to use Play 2 framework once again. The back-end server will receive notification (http request) from the front-end server about creation of a new job. Job specification will include a link to the original user file in the storage and arguments describing the job. The job should be added to a queue. Now the most important part is to delegate the actual processing job to a third party program, which most certainly will be a compiled command line utility, such as SoX for the case of audio processing, written by good people using a programming language of their choice. As far as I know it is possible to call an external program from java, pass command line arguments and collect the result. After processing is done, the back-end server will upload processed file back to storage, and send notification (http request) to the front-end application, which will store a link to the processed file and display it to the user at some later time. To be able to use command line utility I'm going to deploy the back-end application to a Amazon EC2 instance with a Typesafe stack installation.
Here are some questions about this basic plan:
Is Play 2 a reasonable choice for the back-end, or should I look into alternatives? One of them seems to be CGI, which according to Wikipedia "is a standard method for web server software to delegate the generation of web content to executable files." Unfortunately I don't have any experience with that.
There shouldn't be any problem implementing a job queue with Play?
Is it possible to install a command line utility on EC2 and call it from Play?
Should I expect any problems installing Typesafe stack on the EC2? This post briefly describes what I'm planning to do https://www.assembla.com/spaces/bufferine/wiki/Typesafe_stack_on_Amazon_EC2
Assuming that in the future the application will grow, how would I split the jobs among multiple instances on EC2? Should I create a separate job-balancing application in between my front-end and back-end?
I would appreciate any advice! Thanks!
Note: I'm using Java api for Play 2 framework, since I'm not familiar with Scala language.
You may consider Akka for processing and it's built in Play2. It will help you to manage tasks easily, and even saving hardware ressources if used with advanced features. There is a Java API that should cover all your needs. And it's not necessary in a backend APP, if you need more power you can scale even better with two same instancies. Play and Akka are stateless, you can just add new instances to scale. To make it run on EC2, just use the play dist command.
And yes, you can install whatever you want in EC2 and call it from your app.
You may like:
http://akka.io/
http://www.playframework.com/documentation/2.1.0/JavaAkka
http://www.playframework.com/documentation/2.1.0/ProductionDist
also, but in scala
http://blog.greweb.fr/2013/01/playcli-play-iteratees-unix-pipe/
http://blog.greweb.fr/2012/11/play-framework-enumerator-outputstream/
Related
We have several Java standalone applications (in form of Jar files) running on multiple servers. These applications mainly read and stream data between systems. We are using Java 8 mainly in our development. I was put in charge recently. My main function is to manage and maintain these apps.
Currently, I check these apps manually by accessing these servers, check if the app is running, and sometimes run some database queries to see if the app started pulling data. My problem is that in many cases, some of these apps fail and shutdown due to data issue or edge cases without anyone noticing. We need some monitoring and application recovery in place.
We don't have docker infrastructure in place. We plan to implement docker in the future, but for now this is not an option.
After research, the following are options I thought of or solutions I tried:
Have the apps create a socket client which sends a heartbeat to a monitoring app (which needs to be developed). I am keeping this as my last option.
I tried to use Eclipse Vertx to wrap the apps into Verticles. Then create a web view that can show me status and other info. After several tries, the apps fail to parse the data correctly (might be due to my lack of understanding to Vertx library).
Have a third party solution that does this, but I have no idea what solutions are out there. I am open for suggestions.
My requirements are:
Proper monitoring of the apps running and their status.
In case of failure, the app should start again while notifying the admin/developer.
I am willing to develop a solution or implement a third party one. I need you guidance on this.
Thank you.
You could use spring-boot-actuator (see health). It comes with a built-in endpoint that has some health checks(depending on your spring-boot project), but you can create your own as well.
Then, doing a http request to http://{host}:{port}/{context}/actuator/health (replace with yours), you could see those health checks status and also use the response status code to monitor your application.
Have you heard of Java Service Wrappers? Not a full management functionality, however it would monitor for JVM crashes and out of memory conditions and restart your application for sure. Alerting should also be possible.
There is a small comparison table here: https://yajsw.sourceforge.io/#mozTocId284533
So some basic monitoring and management is included already. If you need more, I suggest using JMX (https://www.oracle.com/java/technologies/javase/javamanagement.html) or Prometheus (https://prometheus.io/ and https://github.com/prometheus/client_java)
I have a complicated .jar file that I need to run on azure (C# ASP.NET). On my local system, I simply run java.exe and pass it the jar as an argument. I would like to do the same on the server, however, I don't know where java.exe is located.
I have had a look at the environment variables and found many jdk and jre references, so I assume it is possible.
I can not use ikvm, as the jar is too complex that it isn't running correctly.
So, as a summary: Where is the java.exe located on azure? And if it's not (and I can't do this), what else can I do?
EDIT:
To clarify more: I am developing a web app using ASP.NET. I have a .jar file that I have to run, and on the local machine I run it using:
processStartInfo = new ProcessStartInfo("java");
processStartInfo.Arguments = arguments;
//more options
Process process = new Process();
process.StartInfo = processStartInfo;
process.Start();
process.WaitForExit();
Now I am publishing this website to Microsoft's azure services, and I would like to do the same thing. Except, running it as is tells me that the process can't be run (ie they don't understand what "java" is). I want to find a way to be able to call java as a process. Obviously, if I know the path to java.exe, I simply run the path as a command and I'll be done (ie it'll execute java). That's what I need help with.
As derpirscher mentions in the comment you haven't specified what type of Azure service you want to use, and you haven't specified the nature of your Java code (does it listen for incoming connections on some port? does it talk to any external services? etc.). More info would help us give you a better answer.
That said... one option to start with would be Azure Web Jobs, which allow you to upload and run (among other options) a Java .jar file:
Azure Web Jobs overview
As the info at that link indicates, you can run on-demand, continuously, or on a periodic schedule. Some additional details found here:
Executing Java Web Jobs on Azure
For more general information about both running Java code on Azure and also interacting with Azure services from within Java code, see here:
Azure Java Dev Center
Specifically, here are some additional deployment options beyond Web Jobs:
Deploying Java code on Azure
Best of luck!
EDIT based on your additional feedback:
So if I'm understanding, you want to invoke a Java .jar file by spawning a new process from an ASP.NET application when a user inputs a certain query, etc.?
I can think of two potential options:
Host your ASP.NET application and the .jar on an Azure virtual machine that you customize with the correct version of Java, etc. This would allow you to configure Java how you like, on what path you want, etc.
Decouple the resources used to host your ASP.NET application from those used to invoke the Java code by (for instance) hosting your site as an Azure Web App and writing a message from there to an Azure storage queue each time the Java code should execute. On the receiving side of the queue, you'd have an Azure Web Job configured to listen on that queue and execute your .jar file whenever a new message arrives.
Triggering a Web Job from an Azure Queue
In general option 2 will be preferable from a scalability and pure design standpoint (allows you to separate the concerns of accepting queries vs. processing them, align costs most directly with actual resource consumption, etc.) but option 1 is perhaps easier from the perspective of someone unfamiliar with Azure or cloud architecture.
Just know that, depending on the nature of the processing you have to perform, number of expected concurrent users, etc. an acceptable VM-based solution may be more expensive than something similar to option 1 above. Like so many things in cloud, its ultimately a time vs. expense tradeoff that you have to make here.
Assumption that your application in C#/ASP.NET was running on Azure App Service like Azure WebApp. So you can access the Kudu console via the url https://<your-webapp-name>.scm.azurewebsites.net/DebugConsole, then you can command cd ..\"Program Files (x86)"\Java to move to the path of the collection of Java SDKs for different versions.
Please try to use the absolute path for java.exe (like D:\\Program Files (x86)\\Java\\jdk<version\\bin\\java.exe>) as the argument for the C# Class ProcessStartInfo.
However, I still recommend that you could try to deploy the application using Azure VM and run the app via configure the related environment variables on VM.
Our new start-up company is trying to build a mobile app with an accompanied website. We are trying to setup our application on Amazon Web Services.
We have Java code running in an EC2 instance, which will store data in S3. We want clients (iOS and Web for now) to communicate with the Java Backend via a REST API. Ideally the website would be hosted under the same AWS account.
The Java Code and REST API are already set up in a very basic form, but the setup of the Website is unclear, since this is new to us all. We also would like to evaluate beforehand if such a setup is even feasible.
Since I am in charge of the Website i have already spend hours researching this specific setup, but i simply lack experience in cloud/backend development to come to a conclusion.
Here are some questions we would like to answer:
Where would the HTML files and accompanied JavaScript etc. files be stored?
How can data (images etc.) that is stored within S3 by the JAVA code be accessed from the Website directly?
How could something like bootstrapping of data within HTML files be achieved (in JSON format preferably)?
How could the server be set up to compress certain files like CSS or JavaScript?
Please point me into the right direction, any comment is appreciated.
Where would the HTML files and accompanied JavaScript etc. files be
stored?
Either on the same AWS EC2 box or a different one, just give it a static IP and link that IP to the domain you want, done. Just remember to have port 80 open as a firewall rule.
How can data (images etc.) that is stored within S3 by the JAVA code
be accessed from the Website directly?
The files will have some url that you can link to directly in your html so it's essentially just a url.
How could something like bootstrapping of data within HTML files be
achieved (in JSON format preferably)?
You have a number of choices here. You could potentially create some JSP files to generate the HTML and load the JSP files on access and cache them so they load up super fast. You could argue however, this is overkill and in some ways, the REST endpoint should be robust enough to handle the requests.
Part of me thinks you should endeavor to use the REST API for this information so you just have to manage one endpoint, why make an extra endpoint or over engineered solution for the HTML that you then have to maintain? Build once and reuse.
How could the server be set up to compress certain files like CSS or
JavaScript?
During the build process, run the files through a minify process. This is built into maven so you could do it automatically or by hand using something like jscompress. This Minify plugin shows how to automatically minify your resources. Consider you'll need to be using Maven though as your build tool.
in a student project we are currently developing a website which service is also accessibly via native Android and Windows Phone Apps.
The mobile apps access the service through a public RESTful API written in JAVA which is running on the same server as the website. The website is written in PHP and independent from the API, but they both use the same database (MySQL).
We wanted to extend the functionality of the API and allow registration for the service in the mobile apps.
The problem is that the user receives an email with a confirmation link as soon as he registers for the service.
What is the best approach to ensure that the emails sent by the API are identical to the ones sent by the website?
The easiest way we figured out doing this would be just using the same templates for both, website and API, but in that case we need to manually keep those templates in sync.
Is there a better way than the one above?
Templates need not be in flat file model in each environment. You can store it at one place but should be read by a common intra-web-api say local RPC.
I.e. Write a script in current website environment that returns either a template or duly filled in as the requirement case may be.
And the same API should be called from both web-site php scripts and from java API.
This process will not alter the output in both the environments. The output would always be the same when on a later date you change the template.
I have a webapp with an architecture I'm not thrilled with. In particular, I have a servlet that handles a very large file upload (via commons-fileupload), then processes the file, passing it to a service/repository layer.
What has been suggested to me is that I simply have my servlet upload the file, and a service on the backend do the processing. I like the idea, but I have no idea to go about it. I do not know JMS.
Other details:
- App is a GWT app split into the recommended client/server/shared subpackages, using an MVP architecture.
- Currently, I am only running in GWT hosted mode, but am planning to move to Tomcat in the very near future.
I'm perfectly willing to learn whatever I need to in order to get this working (in fact, that's the point of writing the app). I'm not expecting anyone to write code for me, but can someone point me in the right direction to get started?
There are many options for this scenario, but the simplest may be just copying the uploaded file to a known location on the file system, and have a background daemon monitor the location and process when it finds it.
#Jason, there are many ways to solve your problem.
i) Have dump you file data into Database with column type BLOB. and have a DB polling thread(after a particular time period) polls table for newly inserted file .
ii) Have dump file into file system and have a file montioring process.
Benefit of i) over ii) is that DB is centralized and fast resource where as file systems are genrally slow and non-centalized in nature.
So basically servlet would dump either to DB or file system. Now about who will process that dumped file:- a) It could be either montioring process as discussed above or b) you can use JMS which is asynchronous in nature what it means servlet would put a trigger event in queue which will asynchronously trigger new processing thread.
Well don't introduce JMS in your system unnecessarily if you are ok with monitoring process.
This sounds interesting and familiar to me :). We do it in the similar way.
We have our four projects, all four projects includes file upload and file processing (Image/Video/PDF/Docs) etc. So we created a single project to handle all file processing, it is something like below:
All four projects and File processor use Amazon S3/Our File Storage for file storage so file storage is shared among all five projects.
We make request to File Processor providing details in XML via http request which include file-path on S3/Stoarge, aws authentication details, file conversion/processing parameters. File Processor does processing and puts processed files on S3/Storage, constructs XML with processed files details and sends XML via response.
We use Spring Frameowrk and Tomcat.
Since this is foremost a learning exercise, you need to pick an easy to use JMS provider. This discussion suggested FFMQ just one year ago.
Since you are starting with a simple processor, you can keep it simple and use a JMS Queue.
In the simplest form, each message send by the servlet has to correspond to a single job. You can either put the entire payload of the upload in the message, or just send a filename as reference to the content in the message. These are details you can refactor later.
On the processor side, if you are using Java EE, you can use a MessageBean. If you are not, then I would suggest a 3 JVM solution -- one each for Tomcat, the JMS server, and the message processor. This article includes the basics of a message consuming client.