We are using k8s for deploying our application and it works awesome.But there is a small issue.We have moved from http layer communication to tcp layer.And the communication between different micro-services is through the service (k8s service) name and it works great but the developer can't test the same code locally as the service name will be resolved inside the cluster only.So here are some solutions that I have :-
1.Provided them a different name space where they can test the app with small changes.
The issue with this is that the developers use some break points and test some small changes in code and debug that will be hard by this method.
2.They can implement minikube in local but that doesn't sound good to even me.
3.They can run the container for ms locally and enter the ip of container in /etc/hosts corresponding to the k8s service name.In this the same code will work.
Any other better solutions are welcomed.
😔😔ðŸ˜ðŸ˜
Did you consider using spring boot profiles for this purpose? We are using it effectively for long across our teams. For this purpose, you'll have to extract the service(s) host as separate properties in application.yml (or application.properties) and use this host in rest of the properties as a variable. Following snippet explains this
application.yml
----------------
serviceA:
host: service-A-Name
api-one-endpoint: http://${serviceA.host}/api/v1/one
api-two-endpoint: http://${serviceA.host}/api/v1/two
api-three-endpoint: http://${serviceA.host}/api/v1/three
api-four-endpoint: http://${serviceA.host}/api/v1/four
In production (any hosted/managed environment for that matter), you provide appropriate value for spring property serviceA.host. In your use case, you'll be using this value AS-IS and provide k8s service name binding instead.
For local dev environment, you only need to override single property. For simple use case (say you need to override only single property), you can pass it as an agrument to your spring boot launcher (e.g. "--serviceA.host=localhost"). If you have many services (you likely do) even then you'll need to override well known few host name properties only. Using a dedicated dev profiles is much better in this case. Following example illustrate same scenario
application-dev.yml
-------------------
serviceA:
host: mylocalhost:9090
Then you use this profile in your eclipse/intellij launcher configuraiton for execution or debugging purpose by adding "--spring.profiles.active=dev" as and additional argument and spring boot will use updated host from dev profile. In fact combining these two approaches gives you even more flexibility for advance cases. If you agree on a common port convention across team then you can even check-in application-dev.yml for usage by everyone pretty much as-is.
spring boot profiles is much more powerful feature, I'll strongly recommend to go through it's documentation and few tutorial (like this one) to understand it fully and exploit it effectively for use cases like this one.
We are developing a spring based web application that needs to be deployed in various of environments:
1) developers local machines in order to develop new features \ fix bugs etc.
2) QA Env#1 - for the QA team to test our releases
3) QA Env#2 - for the QA team to test our releases
4) QA Env#3 - for the QA team to test our releases
5) Production
Now, each deployment for one of these environments requires to fill a properties file, this file contains many parameters :
1) tomcat ips
2) mysql ip
3) mongo ip
4) load balancer ip
5) ehcache multi cast ports and addresses.
6) many many more
Our question is:
Where this properties file should be defined? in our code base? (git, in a folder for each environment?) outside the our webapp project (this way, every environment should be updated with the correct properties file once, and then the deployment is straight forward with no configurations at all)?
Bear in mind we are working on a build\deploy machine that will automatically deploy our project to all of these envs, so this is something to keep in mind when deciding the correct way to handle this.
In general, I would not keep properties files in a repository, even though we are talking about a private one. Some reasons that come to my mind are:
Versioning is not needed. I personally see no point in keeping the version history of deployment or QA property files.
Customer privacy. You might not want to share deployment configuration with all people having access to the repository (developers, QA team, ...).
Preventing errors. Errors happens. A developer committing by mistake some changes to the deployment or QA property files might cause some serious troubles to your build/deploy chain.
What you can do is add to your repository some templates for the property files that contain a default configuration for your application (for instance the local developer configuration).
You can then have different property files for different environments that you distribute only within the involved teams (QA configuration to QA team, deployment configuration only for the deployed system).
Your build\deploy machine will most likely have access to all different property files and can retrieve the correct one depending on the target environment. With such configuration you have no benefit from keeping such files in the repository, it is enough to store them in a location available to people entitled to change them and to the build\deploy machine.
I am developing application which is embedded within the cluster environment in Websphere AS. I am using several nodes and sometimes I would like to change configuration settings on the fly and propagate it to all nodes within the cluster. I don't want to hold the config in the db or at least I would like to cache it on the node level and trigger config refresh action which forces each node to refresh the config from some common ground (i.e. db or net drive)
to avoid constant round-trips to the config storage.
More over some configuration can't be stored in db i.e. log level needs to be applied on the logger object in each node separately.
I was thinking about using JMS Topics and publish/subscribe approach to achive that goal.
The idea is that each node could subscribe to each Topic and no matter which nodes initate the config change modification would be propagated to all nodes within the cluster.
Has anyone ever tried to do that in WAS and whether there are any obstacles with this approach. If there are or if you have any other suggestion on how to solve that problem I would be very greatfull for your help.
Tx in advance,
Marcin
Here are a few options to consider as alternatives to JMS -
Use Java EE environment entries. These are scoped to the application, and WAS will automatically propagate any changes to all servers against which the application is deployed. This is a good approach since it is the standard Java EE approach to application configuration, if it is robust enough to meet your use case.
Use a WebSphere Shared Library. This allows you to link your applications to static files external to your application (i.e. on the filesystem), such that they are available on your classpath. Although these files are located on the node file systems, there is a way that you can place these files in WebSphere's centralized configuration repository such that they are automatically propagated to all WAS nodes. For more details on this, see this answer.
Both of these options are optimized for static configuration; in other words, configuration settings that are intended to be set at assembly-time, deployment-time, or to be changed by system administrators, but they are not typically used for values that change frequently, nor are they generally changed programmatically at runtime. WAS does allow your applications to pick these configuration settings in a rolling fashion, such that no application downtime is required though.
Currently we solved the problem with maybe not the most pretty approach but with the most simple one. Since we are using only 2 nodes we have possibility to enter web interface of specific node where we modify settings per each node. Maybe it is not very pretty but for now it is the easiest way. The config is stored in DB and we are planning to trigger config reload in each node and change the log level per node as well.
We build 3-tier enterprise solutions that typically consists of several webapp and ejbjar modules that all talk to a db and have several external integration points.
Each module typically needs its own configurations that can change over the solution's life time.
Deploying it becomes a nightmare because now we have 18 property files that must be remembered to copied over and configured also setting up data-sources, queues, memory requirements etc.
I'm hopeful but not optimistic that there can be a better way.
Some options we've considered/used, each with it's pros and cons:
Use multiple maven projects and continuous integration (eg. hudson or jenkins) to build a configuration jar that includes all the property files for each environment (dev, qa, prod) and then bundle everything up as an EAR. But then things can't be easily changed in production when needed.
Put most of the settings in the DB and have a simple screen to modify it. Internally we can have a generic configuration service EJB that can read and modify the values. Each module can have a custom extended version that have specific getters and setter.
Version control all the property files then check it out on production and check it into a production branch after making changes.
With all of these you still need to configure data-sources and queues etc. in a container specific way :(
Сonsider binding a custom configuration object to JNDI. Then lookup this object in your apps to configure them. Benefits - you can use custom configuration object instead of rather generic Map or Properties.
Another way is to use JMX to configure applications you need. Benefits - you can bind objects you have to configure directly to MBean Server and then use such a well-known tools as jconsole or visualvm to configure components of your application.
Both ways support dynamic reconfiguration of your applications at runtime. I would prefer using JMX.
I've gone through several cycles of finding ways to do this. I still don't have a definite answer.
The last cycle ended up with a process based on properties files. The idea was that each server instance was configured with a single properties file that configured everything. That file was read by the startup scripts, to set memory parameters, by the app server, and by the application itself.
The key thing, though, was that this file was not managed directly. Rather, it was a product of the build process. We had a range of files for different purposes, kept in version control, and a build step which merged the appropriate ones. This lets you factor out commonalities that are shared along various axes.
For example, we had development, continuous integration, QA, UAT, staging, and production environments, each with its own database. Servers in different environments needed different database settings, but each server in a given environment used the same settings. So, there was something like a development-db.properties, qa-db.properties, and so on. In each environment, we had several kinds of servers - web servers, content management servers, batch process servers, etc. Each had JVM settings, for heap size and so on, that were different to other kinds of servers, but consistent between servers across environments. So, we had something like web-jvm.properties, cms-jvm.properties, batch-jvm.properties, and so on. We also had a way to have overrides for specific systems - production-cms-jvm.properties sort of thing. We also had a common.properties that set common properties, and sensible defaults which could be overridden where needed.
Our build process was actually a bit more complicated than just picking the right options from each set; we had a master file for each server in each environment which specified which other files to include. We allowed files to specify other files to include, so we could build a graph of imports to maximise reuse.
It ended up being quite complicated. Too complicated, i think. But it did work, and it did make it very, very easy to make changes affecting many servers in a controlled way. We even merged a set of input files from development, and another from operations, which contained sensitive information. It was a very flexible approach.
I know this has already been answered and my answer is not necessarily generic, but here's my take on things:
Note, here I'm only considering system/resource properties, not application settings. In my view, application settings (such as a payment threshold or other settings should be stored in a database, so that the system can be reconfigured without having to restart a service or cause downtime by re-deploying or re-reading a properties file).
For settings that impact on how different parts of a system connect with each other (such as web service endpoints, etc), I would make use of the JNDI tree.
Database connectivity and JMS connectivity would then be set-up using the Websphere console and can be managed by the Websphere administrators. These can also be created as JACL scripts which can be put into version control if necessary.
In addition to the JNDI resources, for additional properties, such as usernames for web service calls to a backend, etc, I would use Websphere "Name Space Bindings". These bindings can be edited using the Websphere console and accessed via JNDI using the "cell/persistent/mypassword" name.
So I could create the "mypassword" binding (a string), and the management for it falls to the Websphere admin (away from developer eyes or other people who should not have access to production systems), while the same EAR file can be used on dev, test, preproduction and production (which is preferable to have different EAR files for different systems, as the likelihood of other differences creeping in is reduced).
The Java code would then use a simple JNDI lookup (and possibly cache the value in memory).
Advantages over properties files:
Not having a "vulnerable" file that would need to be secured because system properties contain passwords.
Not having to add Java security policies to allow access to that file location
Advantages over database properties:
Not tied to having one database tied to an application server.
Hope that helps
Use multiple maven projects and continuous integration (eg. hudson or
jenkins) to build a configuration jar that includes all the property
files for each environment (dev, qa, prod) and then bundle everything
up as an EAR. But then things can't be easily changed in production
when needed.
I think the config should be in the database of the application instance. Your local machine config may be diffrent to dev and to QA, PROD , DR etc.
What you need is a way of getting the config out the database in a simple way.
I create a separate project with a provided dependency of Apache commons-configuration
It has many ways of storing data, but I like databases and the configurations lives in the database environment.
import javax.sql.DataSource;
import org.apache.commons.configuration.DatabaseConfiguration;
public class MYConfig extends DatabaseConfiguration {
public MYConfig(DataSource datasource) {
super(datasource, "TABLE_CONFIG", "PROP_KEY", "PROP_VALUE");
}
}
Put most of the settings in the DB and have a simple screen to modify
it. Internally we can have a generic configuration service EJB that
can read and modify the values. Each module can have a custom extended
version that have specific getters and setter.
Commons configurations as a simple API, you may then write the GUI as you wish.
You can do the interface in anyway you wish. Or as a quick win have no interface.
Version control all the property files then check it out on production
and check it into a production branch after making changes.
Version control is great. Add another DatabaseConfiguration using composition. The class you extends is the active config and the composed one being the audit. There is another constructor can can have a version. Just overload the right methods to get the desired effect.
import javax.sql.DataSource;
import org.apache.commons.configuration.DatabaseConfiguration;
public class MYConfig extends DatabaseConfiguration {
final DatabaseConfiguration audit;
public MYConfig(DataSource datasource) {
super(datasource, "TABLE_CONFIG", "PROP_KEY", "PROP_VALUE");
audit = new DatabaseConfiguration("TABLE_CONFIG_AUDIT", "PROP_KEY", "PROP_VALUE");
}
#Override
public void addProperty(String key, Object value) {
Object wasValue = super.getProperty(key);
super.addProperty(key, value);
audit.put(key,wasValue);//add version code
}
}
http://commons.apache.org/proper/commons-configuration/
User a simple database table (Section, Key, Value). Add "Version" if you need it, and wrap the entire thing in a simple ConfigurationService class with methods like getInt(String section, String key)
Not a lot of work, and it makes the application code very neat, and tweaking with the configuration very easy.
Interesting alternative config file format: write a scala trait. Your config file can then just be a scala file that you compile and evaluate when the server starts.
http://robey.lag.net//2012/03/26/why-config.html
There is a team develops enterprise application with web interface: java, tomcat, struts, mysql, REST and LDAP calls to external services and so on.
All configuration is stored in context.xml --tomcat specific file that contains variables available via servlet context and object available via JNDI resources.
Developers have no access to production and QA platforms (as it should be) so context.xml is managed by support/sysadmin team.
Each release has config-notes.txt with instructions like:
please add "userLimit" variable to context.xml with value "123", rename "DB" resource to "fooDB" and add new database connection to our new server (you should know url and credentials) named "barDb"
That is not good.
Here is my idea how to solve it.
Each release has special config file with required variable names, descriptions and default values (if any): even web.xml could be used.
Here is pseudo example:
foo=bar
userLimit=123
barDb=SET_MANUAL(connection to our new server)
And there is a special tool that support team runs against deployment artifact.
Look at it (text after ">" is typed by support guy):
Config for version 123 of artifact "mySever".
Enter your config file location> /opt/tomcat/context/myServer.xml
+"foo" value "bar" -- already exists and would not be changed
+"userLimit" value "123" -- adding new
+"barDb"(connection to our new server) please type> jdbc:mysql:host/db
Saving your file as /opt/tomcat/context/myServer.xml
Your environment is not configured to run myServer-123.
That will give us ability to deploy application on any environment and update configuration if needed.
Do you like my idea? What do you use for environment configuration management? Does there is ready-to-use tools for that?
There are plenty of different strategies. All of them are good and depends on what suit you best.
Build a single artifact and deploy configs to a separate location. The artifact could have placeholder variables and, on deployment, the config could be read in. Have a look at Springs property placeholder. It works fantastically for webapps that use Spring and doesn't involve getting ops involved.
Have an externalised property config that lives outside of the webapp. Keep the location constant and always read from the property config. Update the config at any stage and a restart will be up the new values.
If you are modifying the environment (i.e. application server being used or user/group permissions) look at using the above methods with puppet or chef. Also have a look at managing your config files with these tools.
As for the whole should devs be given access to prod, it really depends on a per company basis. For smaller companies where the dev is called every time there is a problem, regardless of whether that problem is server or application related, then obviously devs require access to the box.
DevOps is not about giving devs access to the box, its about giving devs the ability to use infrastructure as a service, the ability to spawn new instances with application X with config Y and to push their applications into environments without ops. In a large company like ours, what it allows is the ability for devs to manage the application they put on a server. Operations shouldn't care what version is on their, thats our job, their job is all about keeping the server up and running.
I strongly disagree with your remark that devs shouldn't have access to prod or staging environments. It's this kind of attitude that leads to teams working against each other instead of with eath other.
But to answer your question: you are thinking about what is typically called continuous integration ( http://en.wikipedia.org/wiki/Continuous_integration ) and moving towards devops. Ideally you should aim for the magic "1 click automated deployment". The guys from Flickr wrote a lot of blogs (and books) about how they achieved that.
Anyhow .. there's a lot of tools around that sector. You may want to have a look a things like Hudson/Jenkins or Puppet/Chef.