There are a couple of Continuous Integration servers available for Java (Hudson, Jenkins, TeamCity).
In my understanding these are quite 'UI'-centric in that the main way to interact with these solutions is through a web interface. Also, they require a Servlet container to run and start a couple of processes on the server.
I would like to know whether there is a Continuous Integration solution available for Java/Maven/git that can run without a server and can be controlled from the command line. For instance, by issuing a command:
CLIIntegrationSolution path/to/git_repo
Which checks if there are any updates on the repo, builds the project if required, runs the tests, generates reports, and sends messages in case of any build or test errors.
I know this can be hand-crafted but it might take some time to get it right.
[could have been a comment, but just too long ]
You should go back check what you really wants to do. Jenkins has embedded web server , does not require an app server. It also exposes command line to do almost every thing from start stop and reconfigure build. It also exposes a web service layer for programmatic query
In short jenkins will do what ever you want. The GUI part is only to setup, not for actually performing the task. They are all done by scheduled/scm poll /manual triggers.
Related
I've got a Java code to perform some interactions with web pages and used Selenium for it.
Now I'd like to get this code executed every hours and I've thought it's a great occasion to discover the cloud world.
I've created an account on Google Cloud.
Because my app need to have a driver to use Selenium (gecko driver for Firefox), I'll have to create an docker image to set everything it need inside it.
In Google Cloud services, there is the "Cloud Scheduler" which can allow me to run a code when I want to.
But here are my questions :
What kind of target should I configure (HTTP, Pub/Sub, HTTP App Engine)?
Because I'm not using the Google Cloud Functions, my container will always be up, it doesn't seems as a great idea for a pricing reason? I would have like to have my container up only the time of the execution.
Also I was thinking to use Quarkus framework to wrap my application since I've since it was made for the cloud and very quick to start, is that the best option for me?
I'll be very glade if someone can help me to see this a little better. I'm not a total beginner I work as a Java / JavaScript developer for 5 years now and dockerized some application but everything about the cloud is a big piece, not easy to know where to start.
So you:
are using docker images
run your workload occasionally
aren't willing to use Cloud Function
==> Cloud Run is your best bet. Here is Google Cloud Run Quick start : https://cloud.google.com/run/docs/quickstarts/prebuilt-deploy
Keep in mind that your containerised application needs to be listening to HTTP requests so take a look at Cloud Run Container runtime contract
Finally you can indeed trigger Cloud Run from Cloud Scheduler, and here a detailed documentation on how to do it https://cloud.google.com/run/docs/triggering/using-scheduler
As #MBHAPhoenix says, Cloud Run is your best option. You can then trigger the job from Cloud Scheduler. We have this exact scenario currently running for one of our projects but our container is Python. We wrote an article about it here
You should note that to trigger your Cloud Run job from Cloud Scheduler, you'll have to 'secure it'. This means means you won't be able to just type the URL in a web browser. A service account will be responsible for running the Cloud Run job and you'll then need to grant your Cloud Scheduler service access to this service account so it can invoke the Cloud Run Job. I've been meaning to put up a post about the exact steps for doing this (will try to get it done this weekend).
In terms of cost, we have this snippet from our article
...Cloud Run only runs when it receives an HTTP request. It plays dead and comes alive to execute your code when an HTTP request comes in. When it is done executing the request, it goes 'dead' again till the next request comes in. This means you're not paying for time spent idling i.e. when it is not doing anything.....
We have several Java standalone applications (in form of Jar files) running on multiple servers. These applications mainly read and stream data between systems. We are using Java 8 mainly in our development. I was put in charge recently. My main function is to manage and maintain these apps.
Currently, I check these apps manually by accessing these servers, check if the app is running, and sometimes run some database queries to see if the app started pulling data. My problem is that in many cases, some of these apps fail and shutdown due to data issue or edge cases without anyone noticing. We need some monitoring and application recovery in place.
We don't have docker infrastructure in place. We plan to implement docker in the future, but for now this is not an option.
After research, the following are options I thought of or solutions I tried:
Have the apps create a socket client which sends a heartbeat to a monitoring app (which needs to be developed). I am keeping this as my last option.
I tried to use Eclipse Vertx to wrap the apps into Verticles. Then create a web view that can show me status and other info. After several tries, the apps fail to parse the data correctly (might be due to my lack of understanding to Vertx library).
Have a third party solution that does this, but I have no idea what solutions are out there. I am open for suggestions.
My requirements are:
Proper monitoring of the apps running and their status.
In case of failure, the app should start again while notifying the admin/developer.
I am willing to develop a solution or implement a third party one. I need you guidance on this.
Thank you.
You could use spring-boot-actuator (see health). It comes with a built-in endpoint that has some health checks(depending on your spring-boot project), but you can create your own as well.
Then, doing a http request to http://{host}:{port}/{context}/actuator/health (replace with yours), you could see those health checks status and also use the response status code to monitor your application.
Have you heard of Java Service Wrappers? Not a full management functionality, however it would monitor for JVM crashes and out of memory conditions and restart your application for sure. Alerting should also be possible.
There is a small comparison table here: https://yajsw.sourceforge.io/#mozTocId284533
So some basic monitoring and management is included already. If you need more, I suggest using JMX (https://www.oracle.com/java/technologies/javase/javamanagement.html) or Prometheus (https://prometheus.io/ and https://github.com/prometheus/client_java)
What I am looking for is a "best practices" description or example by which testing can be automated for components that are deployed to a Weblogic server.
I am not expecting anyone to present a step by step solution to this problem.
I am looking for a resource (book, manual, website, etc.) that can describe a path to this integration and testing goal.
The situation is that we have a pair of (Maven) Project deployments (in Eclipse) which are managed/reviewed/maintained through: Git, Stash, and Jenkins.
The first component is providing Web Services (RESTful services as well as Stateful and Stateless services). It is connected to the second component. The second component exposes Stateless and RESTful services that provide access services (CRUD: Create Read Update Delete) to an Oracle SQL Database.
Currently, the Jenkins Service is testing the Client UI through Jasmine Zzzzz.spec.js tests. This is all well and good for the "front-end hipsters", but not helpful for the Java service component developers.
What I would like to do is to be able to write (?JUnit?) tests to evaluate Service component operations that can be automatically executed by Jenkins continuous integration components. What I would like to avoid doing is mocking up everything to the point that the tests become trivial and pointless.
What needs to happen is:
1. Developer completes a Work Product (JIRA Task) to add functionality to a Service hosted by a Weblogic Server.
2. Work Product contains a Test (?JUnit?).
3. Work Product (including test) is pushed by Git to Stash.
4. Work Product Test is added to Integration Tests.
5. Stash and Jenkins execute and evaluate Work Product JUnit Test as part of [Integration Testing].
Integration Testing will:
1. Start a (configured) Weblogic Server (if one is not already started).
2. Compile and Publish Deployment containing Work Product.
3. Deployment will connect to a Configured Datasource.
4. Start [Work Product JUnit Test].
Work Product JUnit Test will:
1. Connect and authenticate to Weblogic Service Deployment.
2. Call tested Service methods.
3. Evaluate test results
Yes, that is a tall order with a hive full of buzzwords. However, I am having difficulty finding a worthwhile resource that isn't trying to direct me to mock up the very components that I am trying to test.
What you have described is a pretty standard CI setup; the sort of thing that is notable via its absence rather than its existence.
In that vein, it's probably appropriate for you and your team to read up on the fundamentals.
Continuous Integration (Fowler series)
Continuous Delivery (Fowler series)
The DevOps Handbook
then if you really want, you can pick up books on the specifics of Jenkins:
Jenkins: the definitive guide (oreilly)
Jenkins CI Cookbook
I was wondering if someone could point me to a good tutorial or blog post on writing a spring application that can be all run in a single process for integration testing locally but when deployed will deploy different subsystems into different processes/dynos on heroku.
For example, I have services for User management, Job processing, etc. all in my web application. I want to run it just as a web application locally. But when I deploy to heroku I want to deploy just the stateless web front end to TWO dynos and then have worker dynos that I can select different services to run on. I may decide to group 2 of these services into one process or decide that each should run in its own process. Obviously when the services run in their own process they will need to transparently add some kind of transport like REST or RabbitMQ or AKKA or some such.
Any pointers on where to start looking to learn how to do this? Or am I thinking about this incorrectly and you'd like to suggest a different approach? I need to figure out how to setup the application and also how to construct maven and intellij to achieve this.
Thanks.
I can't point you to a prefabricated article or post, but I can share the direction I started down to solve a similar problem. Essentially, the proposed approach was similar to yours - put specific services with potentially long-running logic in worker dynos and pass messages via Jesque (Java port of Resque) on a RedisToGo instance (Heroku add-on). I never got the separate web vs. worker Spring contexts fully ironed out (moved on to other priorities) but the gist of it was 1) web tier app context would be configured to post messages and 2) worker app context configured to consume.
That said, I used foreman locally to simulate the Heroku environment to debug scaling (foreman start --formation="web=2" + Apache mod_proxy_http). Big Spring gotcha when you scale to 2+ dynos - make sure you are using Redis or Memcache for session storage when using webapp-runner. Spring uses HttpSession by default to store the security context... no session affinity or native Tomcat session replication.
Final caveat - in our case, none of our worker processing needed to be reflected to the end user. That said, we were using Pusher for other features (also a Heroku add-on). If you need to update the user when an async task completes, I recommend looking at it.
I am creating a Java service which will run within a web servlet container (probably Tomcat). One portion of the server will run on its own and will not be initiated by HTTP. I know that when an HTTP call causes an exception, the web container can call it again.
I want to be sure that the part of the server which runs continuously will continue to run, even if it fails. I will handle whichever failures I can manually, but if it all fails I want something to restart it all. Are there any tools that can accomplish this easily? I am already using Spring and Tomcat, so if those can provide it, that is ideal. If not, then how about a good design pattern?
Edit: To clarify, I have a web service which will run in Tomcat. I want to run a separate thread within that service and set it up such that when the thread ends or an un-handled exception occurs, Tomcat (or something else) detects the failure and restarts the web service. I know that typically web containers have threads start from some external call and thus handle failures from those threads. What I want is something which handles a background worker thread.
Not quite clear on the design you have in mind, but it seems to me you need some sort of health check.
You can implement such a mechanism in many ways e.g. open a socket from this process that runs all time and periodically send a message.
If there is no reply then the process failed.
You could restart tomcat or implement a mechanism to restart that process.
Can not tell you more details since you do not specify much on what you are trying to do.
UPDATE:
I think that you should use JMX. It is offered by Spring and Tomcat that you already use.
Just make the process you want to monitor a managed resource and another module can check if it is alive.
If you are running inside a Servlet then as per J2EE spec, you cannot restart the container but, you can use ScheduledExecutorService to continuously monitor that your service is running and if not, then re-start it.
EDIT. More details below
You can call isTerminated() to check if the service still running and add more tasks to it, if the queue is empty.
I may be misunderstanding your problem here, but you might be over-thinking it.
There's nothing stopping you from running multiple Tomcat instances on a single machine. You could then have Server A connect to Server B to pull down information (via a web service of your choosing). This would alleviate the need for an outage on server A to cause an outage on server B (which is what I'm assuming you're trying to avoid).
This is a common way to isolate production environments simply by binding to a separate port. If Tomcat doesn't fit the bill for the service you can always run the application as a service on [insert operating system of choice] and connect to it via a proprietary protocol. Your operating system can handle restarts in that case. Typically I think the multiple Tomcat containers is the easiest approach as it is simple to install and relatively easy to set up.
Good luck, it seems like a fun system administration problem. You also might be interested in checking out Quartz job scheduling as that might fit the bill for an intermittent service.
edit: a little more detail might provide some more detailed answers.
See this post. It's a simple tomcat-watchdog shell script.