What do you use to automatically deploy applications for various kinds of server applications (web, socket, daemon) that uses various technologies (different DBs, languages, etc)?
Here we use Python, Java and Ruby, and may use other languages as well in the future.
Update: I ended up using Puppet to manage all server setup, configs AND deploy. To trigger the deploy of newer versions I ended up using Python Fabric scripts.
I use Puppet for some of the deployements / initial configuration of server. Maven and Ant for Java based projects.
You could use Ant, Makefile, or a batch script. Or a combination of them.
Go for KWateeSDCM. It comes with a straightforward web GUI and does not require obscure scripting and integrates nicely with your build chain via a REST API.
We use Maven and pull out to the ant-plugin when required. In turn the ant-plugin very occasionally calls out to some native scripting language/application/packager/whatever but we're finding that over time we can generally find a Maven or ANT plugin/task to do the trick.
You might want to look up Continuous Deployment, it's a pretty hot topic in the build and CI space right now.
For Python you could use Fabric or Paver
For Ruby, there's capistrano and 'vlad the deployer'
For Java, it's the Ant.
For PHP/Python projects I also use Peritor Webistrano, a neat frontend for capistrano. It involves changing a few of the default recipes to remove the rails-specific stuff, but it's worth it once you get it setup correctly.
I think the best choice you could do is using maven. Even if maven is mostly used for Java projects, you can with the right plugins (or the one you write) deploy anything anywhere.
you can use maven, ant, ivy along with hudson for java projects.
We use Ant or Maven for different projects we have in house (depending on the need and how old the project is too...). We tend to use Jenkins (formerly known as Hudson) as our build and deployment tool.
And then we encourage developers to write code that does not hard code to DB's, URL's, etc. We try to abstract via the container (ie, Data Sources, pure JMS API's, etc) when running within an app server. And we tend to abstract environment specifics via properties files that we look up at runtime. The path to the properties file is to be defined as a variable on the server. This way we can be flexible enough to build code once and move it through to all our environments.
Related
I'm primarily a Python developer working in an environment where just about every important batch is run by a Hudson or Jenkins server. I've written a python library which uses Hudson's python API which allows me trivially write Hudson clients in Python.
I'm looking for a Java class which will help me do the same kinds of things, but without any Python.
The neat thing about my home-made python library is that it allows me to work at a very high-level. Consumers of my library are entirely shielded from the complexities of what goes on within Hudson. Most of the things I need to do can be expressed in one-line functions:
For example I can search for and download artifacts. I can find jobs whose results match certain criteria. I can even remotely trigger jobs or query the status of jobs. The only problems with my library are that for boring legal reasons it must remain closed-source, and being Python I've no hope of being able to integrate it into Ant.
I'd like to make a lot of my Hudson magic work from Ant scripts - and for this I need to re-implement some of my Hudson stuff as an ant-task in Java. For that I'd need a java library which exposes a lot of what Hudson does at a high level. Does such a project exist? If so can you let me know where I can find it?
Updated: I already built a Python based solution Python Jenkins API - I'm after a Java equivalent of this.
jenkins cli
is that what you need?
Is it possible to write Jenkins plugins using Python (via Jython), and if so, where would I get started? I don't know Java (and don't have a real interest in learning it at the moment) so being able to use Python would be nice for me.
There's no docs on that. Frankly your problem with be you must understand how the interfaces and extensions map from java to jython to write it.
Here are links on writing Jenkins plugins
Let's you embed either Jython or Python scripts and run them from a plugin If all you nee to do is run a simple script, that be what you need.
If your goal is to execute Jython code within Jenkins, you may want to have a look at the Jython Plugin.
Starting version 1.6, you can actually install Jython packages (say if you have your own library you'd like to use), and it'll automatically sync up the packages across all Jenkins slaves.
Since Jython can produce Java classes I see no reason why this should not work.
Note that you will need to learn at least some things about Java (mostly the platform, not necessarily the language) in order to be able to write plugins for Jenkins.
I am making my first foray into scala for a production app.
The app is currently packaged as a war file.
My plan is to create a jar file of the scala compiled artifacts and add that into the lib folder for the war file.
My enhancement is a mysql-backed app exposed via Jersey & will be integrated with a 3rd party site via HttpClient invocations. I know how to do this via plain java. But when doing it in scala, there are several decision points that I am pussyfooting on.
scala 2.7.7 or 2.8 RC ?
JDBC via querulous Is this API ready for production ?
sbt vs maven. I am comfortable with maven.
Is there a scala idiomatic wrapper for HttpClient (or should I use it just like in java) ?
I'd love to hear your comments and experiences on starting out with scala.
I would use 2.8.0. There are just too many useful features in 2.8. Besides, 2.8 is closing in on a final release. If you're just starting out, why not start out with that? FWIW, I've been using 2.8.0 since Beta1, in various tools and libraries that I use daily. While there have been bugs, they haven't been enough to make me fall back to 2.7.7. YMMV, though.
This isn't going to make your decision any easier, but there are other possibilities for database access. I've been using SQueryL, for instance; I like it. ORBroker is another option.
If you're comfortable with Maven, then use it, by all means. Personally, I prefer SBT. I get the full power of a real programming language, when I need to implement special build logic. Just as useful, I don't have to deal with XML configuration files. (XML is good for data, but it's a crappy format for a human-edited configuration file.)
You might try Databinder Dispatch. See this article for a nice overview.
If you're only going to start development, Scala 2.8 GA will probably be available by the time you go production. Even if it's not, I would choose the freshest 2.8RC pack rather than sticking to 2.7.7. 2.8 not only has a number of great features, but also contains lots of 2.7.7 bugfixes.
There're not too many production-ready ORMs designed for Scala these days. I would probably choose Lift Persistence, because of the team of professionals and friendly community behind Lift Framework. But if you don't want to risk, you should consider using old good proven Java ORMs: Hibernate, JPA, iBatis (that was recently renamed to myBatis), etc.
You should give SBT a try! It's compatible with maven POMs, so migration to SBT shouldn't be too painful for you. Benefits from using SBT:
It's designed for Scala, so you will be relieved from the burden of maintaining countless number of plugins for Maven to make it work with Scala consistently
You will be able to write build scripts in Scala (it's an amazing experience compared with XMLs)
SBT has a killer feature - continuous whatever (building, testing, deploying). SBT monitors your code, detects when its changed, and triggers an action (test, re-deployment, etc.).
what's the point of using ant, maven, and buildr? won't the using build in eclipse or netbeans work fine? i'm just curious what the purpose and benefit of extended build tools are.
Dependency Management: The build tools follow a component model that provides hints on where to look for dependencies. In Eclipse / Netbeans, you have to depend on a JAR and you don't really know if this JAR has been updated or not. With these build tools, they 'know' updates in dependencies (generally because of a good integration with your source control repository), recalculate transitive dependencies and ensure that everything is always built with the latest versions.
Access Control: Java, apart from class level access control, has no higher abstraction. With these build tools you can specify exactly which projects you want to depend on you and control visibility and access at a higher level of granularity.
Custom Control: The Eclipse / Netbeans build always builds JAR files. With custom build mechanisms, you could build your own custom (company-internal) archive with extra metadata information, if you so wish.
Plugins: There are a variety of plugins that come with build tools which can do various things during build. From something basic like generating Javadocs to something more non-trivial like running tests and getting code coverage, static analysis, generation of reports, etc.
Transport: Some build systems also manage transport of archives - from a development system to a deployment or production system. So, you can configure transport routes, schedules and such.
Take a look at some continuous integration servers like CruiseControl or Hudson. Also, the features page of Maven provides some insight into what you want to know.
On top of all the other answers. The primary reason I keep my projects buildable without being forced to use NetBeans or Eclipse is that it makes it so much easier to setup automated (and continuous) builds.
It would be rather complicated (in comparison) to set up a server that somehow starts eclipse, updates the source from the repository, build it all, sends a mail with the result and copies the output to somewhere on a disk where the last 50 builds are stored.
If you are a single developer or a very small group, it can seem that a build system is just an overhead. As the number of developers increases though it quickly becomes difficult to track all changes and ensure developers are keeping in sync. A build system reduces the rate of increase of those overheads as your team grows. Consider the issues of building all the code in Eclipse once you have 100+ developers working on the project.
One compelling reason to have a separate build system is to ensure that what has been delivered to your customers is compiled from a specific version of the code checked into your SCM. This eliminates a whole class of "works on my box" issues and in my opinion this benefit is worth the effort on its own in reduced support time. Isolated builds (say on a CI server) also highlight issues in development, e.g. where partial or breaking changes have been committed, so you have a chance to catch issues early.
A build in an IDE builds whatever happens to be on the box, whereas a standalone build system will produce a reproducible build directly from the SCM. Of course this could be done within an IDE, but AFAIK only by invoking something like Ant or Maven to handle all the build steps.
Then of course there are also the direct benefits of build systems. A modular build system reduces copy-paste issues and handles dependency resolution and other build related issues. This should allow developers to focus on delivering code. Of course every new tool introduces its own issues and the learning curve involved can make it seem that a build system is a needless overhead (just Google I hate Maven to get some idea).
The problem with building from the IDE, is that there are tons of settings affecting the build. When you use a build tool all the settings a condensed in a more or less readable form into a small set of scripts or configuration files. This allows in the ideal case anybody to execute a build with hardly any manual setup.
Without the build tool it might become next to impossible to even compile your code in let's say a year, because you'll have to reverse engineer all the settings
Different features. For example Maven can scan your dependencies and go download them, and their dependencies so you don't have to. For even a medium sized project there may be a very large number of dependencies. I don't think Eclipse can do that.
#anonymous,
Why do you I assume that me, a member
of your team, is using an IDE all the
time? I might want to build the code
on a headless build server, is that
ok?
Would you also deny me the right of
using a continuous integration
engine?
May I fetch dependencies from a central repository please? How can I do that?
Would you tie me to a specific IDE? I can't run Eclipse easily on my very old laptop, but I'll buy a new one.
Maybe I should also uninstall subversion and use patches or just zip folders on a sftp/ftp/Samba share.
The build tools allow you to do a build automatically, without human invention, which is essential if you have a code base being able to build many applications (like we do).
We want to be certain that each and everyone of our applications can build correctly after any code base changes. The best way to check that is to let a computer do it automatically using a Continouos integration tool. We just check in code, and the CI server picks up there is a change and rebuilds all modules influenced by that change. If anything breaks the responsible person is mailed directly.
It is extremely handy being able to automate things.
To expand on Jens Schauder's answer, a lot of those build options end up in some sort of .project file. One of the evils of Eclipse is that they store absolute path names in all of it's project files, so you can't copy a project file from one machine to another, which might have its workspace in a different directory.
The strongest reason for me, is automated builds.
IDEs just work on a higher abstraction layer.
NetBeans nativly uses Ant as its underlying build tool and recently can directly open maven projects in NetBeans. Hence, your typical NetBeans project can be compiled with ant and your maven project already is a NetBeans project.
As with every GUI vs CLI discussion, IDEs seem easier for beginners but once you get the idea it becomes cumbersome to do complex things.
Changing the configuration with an IDE means clicking somewhere which is easy for basic things but for complex stuff you need to find the right place to click. Furthermore IDEs seem to hide the importent information. Clicking a button to add a library is easy but you may still not know where the library is, etc.
In contrast, using a CLI isn't easy to start with but becomes quickly easy. It allows to do complex things more easily.
Using Ant or Maven means that every one can choose his/her own IDE to work one the code. Telling someone to install IDE X to compile it is much more overhead than telling "run <build command> in your shell". And of course your can't explain the former to an external tool.
To sum up, the IDE uses a build tool itself. In case of NetBeans Ant (or Maven) is used so you can get all the advantages and disadvantages of those. Eclipse uses its own thing (as far as I know) but also can integrate ant scripts.
As for the build tools themselves Maven is significantly different from Ant. It can download specified dependencies up to the point of downloading a web server to run your project.
In all projects, developers will often manually invoke the Build process.but it is not Suitable for large Projects, Where it is very difficult to keep track of what needs to be built, in what sequence and what dependencies there are in the building process.Hence we Use Build Tools for Our Projects.
Build Tools Done varieties of the task in the Application which will do by the Developer in their daily life.
They are
1.Downloading dependencies.
2.Compiling source code into binary code.
3.Packaging that binary code.
4.Running tests.
5.Deployment to production systems.
I currently have Apache HTTP Server, but I'm guessing I'll need Tomcat (and then have to configure it in a way that makes it not open to the public), a Java JDK (which I already have, but should probably update), and an IDE (I have Eclipse). But what else should I have or know before starting?
Lets see... you'll need:
A JDK. Preferably the Standard Edition, unless you plan on using GlassFish as a J2EE server. If you do use the standard edition, you need to include the jars from your servlet container/J2EE stack to access J2EE libraries.
A Servlet container. Apache Tomcat is a popular one, and Eclipse already has integration support for it. Keep in mind that Tomcat is not a complete J2EE stack... you need something like JBoss Application Server or Apache Geronimo for that.
(Semi-Optional) A web server. Apache Web Server surprisingly works well with Apache Tomcat or Apache Geronimo... it's almost like they were made by the same people! Eclipse has built-in support for Tomcat and doesn't appear to require a web server because of it. I could be wrong, though.
(Optional) An IDE. If you use Eclipse, get the version for Java EE Developers.
(Optional) A Database. MySQL and HSQLDB are popular ones for Java, at least for small to medium applications. Keep in mind that you also need to download the JDBC drivers for your database.
(Optional) A revision control system. Even on a single-developer project, a revision control system can save your hide if you accidentally remove code or a file that you shouldn't. There are several choices here; Subversion is my personal choice, along with an Eclipse plugin for it, such as Subclipse.
I'd recommend having a database server installed as well- you can go pretty far with the various pure Java implementations, but anything significant will probably benefit from having a copy of MySQL around.
I'd also get some sort of version control going- tortoiseSVN works really well for windows. Even if you're just goofing around for yourself, getting in the habit of using it will save time and heartache down the road.
If you're trying to stay pretty simple you have everything you need; a servlet container.
The IDE of course helps.
I would recommend Eclipse's Web tools project as it will allow you to code, and then hit a button that will put that code on tomcat all from the IDE.
http://www.eclipse.org/webtools/
As Bogdan mentioned maven is a dependency management tool but depending on your level of comfort I'm not sure if you want to muddy the water with that yet.
Another project that uses Maven is appfuse (http://appfuse.org/display/APF/Home) and will help you get off the ground quickly for generating a webapp. It will give you a webapp, a database, unit tests, dependency management, and a good skeleton to structure your project off of.
So, if you're just wanting to play around with some .jsps and a servlet or two I would stick with the IDE, but if you're comfortable with web applications, MVC, and build tools I'd recommend appfuse.
That's all you need from the tools.
Then, you need to google some tutorial on using servlets and jsp pages. Even documentation bundled with tomcat is good enough.
You'll probably also want a database like MySQL or HSQLDB. You could replace Tomcat with Jetty which is often easier to get class changes to reload without restarting the server.
You can do everything from within NetBeans if you're willing to switch IDEs (it's a relatively painless switch, but you might be able to create a similar setup in Eclipse). There's a Tomcat plugin for NetBeans that allows you to create, edit, run, and debug servlet and JSP web apps all from within NetBeans. There's a starter tutorial here. When you install the Tomcat plugin, you get lots of good example servlets and JSPs as a bonus.
I would go for maven.
It will give you a quick start in configuring your project (by using archetypes) and it will manage your dependencies.
Install it and run the archetype command to create your project.
mvn archetype:create -DgroupId=com.mycompany.app -DartifactId=my-webapp -DarchetypeArtifactId=maven-archetype-webapp
After that just use the maven eclipse plugin to configure your eclipse environment for that project (http://maven.apache.org/plugins/maven-eclipse-plugin/).
I currently have Apache HTTP Server, but I'm guessing I'll need Tomcat (and then have to configure it in a way that makes it not open to the public)
For simple Java based web applications, you need not have Apache installed unless you are too concerned about performance and want to gain by letting Apache serve static resources.
But what else should I have or know before starting?
A good IDE is most recommended, otherwise the development will become painful. Use a simple database as people have already suggested. I would go with MySQL because its neither too complex for a webapp nor too trivial.
Instead of Tomcat I recommend Jetty, that is also a Servlet-Container. In my experience it's easier to setup and maintain. Good enough to test your application.
Other than that you need a JDK (naturally). A database is optional, but if your web-app wants to save data it's the best option.
Step 1:you need to have J2EE installed in your system if not download it from http://www.oracle.com/technetwork/java/javaee/downloads/index.html here.
Step 2: Should have Apache TomCat server to run your web project on your system locally(local Server).
installation steps https://www.ntu.edu.sg/home/ehchua/programming/howto/Tomcat_HowTo.html go to this website.
Step 3:set Class path for java JRE.