Disclaimer: I am a noob in Spring. What I am asking may be very "odd" as I don't even know what I don't know.
I am trying to create a batch data movement/manipulation tool (may I say a ETL tool) using Java. Someone suggested me to check out spring-batch which I really liked as it has many libraries for data reading/writing and processing.
But my trouble is- my data sources (flatfile or table) are not fixed. There is a fronted where user will select which flatfile or database table(s) they want to load and the program will automatically load that. This means, usual things like:
Source / target entity structures
source or target database URL/DSN
Job parameters etc.
are not pre-determined in my case. They are determined in runtime. But, so far, whatever spring-batch examples I have seen - they have configured these information in XML. I can't do that as that will make these information static.
My Question is - If I do not want to use Spring Container (and all its XML based bean configuration) but still want to use spring-batch to take advantage of it's batch processing libraries, will that be possible/viable?
No, you need to use the Spring Container for using spring batch and all its XML or annotation based bean configuration. However, what you are trying is achievable, you just need to find way to make it configurable by using parameters in Spring batch. You can take anyone example from internet and start working on it to make it configurable.
Like you can utilize file reader from Spring by simply writing custom mapper. You save the effort to create and maintain file reading logic.
You can have writer which can query which you create dynamically based on your table and file at run time.
Examples shows everything in xml for making simple to understand, how ever if you explore little bit almost everything can be done at runtime.
Related
In my spring project I use the #Entity annotation and let hibernate create the database tables automatically. Now I need some trigger on my database. Since you cannot combine letting Hibernate create the schema and initializing the schema via schema.sql I don't know how to create database trigger. I cannot create trigger in the schema.sql because the table is created afterwards by hibernate.
Since I dont want to rewrite the whole project, I was wondering HOW TO CREATE DATABASE TRIGGER ON JAVA LEVEL.
What would be the professional way? Would be nice if you could provide a simple code example
I tried #EntityListener but you cannot inject crudrepositories, this option might not be the best. Thats why I'm looking for other solutions.
There are several options if you are absolutely sure that triggers are what you need.
First of all, there is no "Java way" for creating triggers, at least not that I am aware of. You could implement something like this though, but it would be much more complicated than simply maintaining the triggers definition in SQL statements files and applying them whenever is needed.
Second, why the need to create triggers with Spring? Triggers are not more than code that represent encapsulate some business logic. As such, it might be a good idea to maintain them in separate SQL files and apply them whenever trigger's code is updated. If what you are looking for is to apply them automatically, you can should look for tools like Liquibase that enables these kind of automated tasks.
If you insist on applying triggers with Spring, then you might consider using the automatic database initialization provided by Spring, which can run automatically SQL files containing DDL/DML statements. For example, if you use MySQL you might have file called schema-mysql.sql under Spring src/main/resources folder with your triggers definitions. Note that this will execute the SQL files everytime the application starts, so you will have to control such cases with the specifics statements of your database, like DROP TRIGGER IF EXISTS my_trigger; for the case of MySQL.
In my pasts experiences, whenever we needed to use triggers we simply maintained them in separate SQL files, and apply them automatically using Liquibase, but this was in rare occasions given that using triggers highly couples you to the database vendor, which brings to the table other kinds of problems.
I was about to start writing a program in Java which includes hibernate.
I am confused which should I choose between Annotations and XML Configuration.
I personally like XML configuration because I have been using this technique for sometime.
But I had one doubt:
Is annotation faster than XML configuration in the scale of time taken to execute the program?
I personally think that if I use XML, then JVM would have to parse the file and then validate it as well. This would take some time.
While, in case of annotations, the container will directly start performing the operations using the JARs available.
Can someone provide me with a clearer understanding on this.
Thanks in advance.
I've got a framework that i wrote, which uses dependency injection in order to instantiate a service with a particular data processing workflow. I don't want to have to write a huge spring file every time i want to use it so i would like to be able to draw the workflow in a diagraming tool where i can set properties and such and then take that diagram and generate the spring config from it. it's akin to the some IDE's where you draw stuff and set properties. I realise there may not be something built for this but if you could point me in the direction of a collection of tools that i can use to accomplish this or terminology that i can search the web for or anything that would be useful in making this happen, then i would really appreciate it.
This questions has been asked a few times, but most of the answers are a few years old so I'll post this hoping someone has figured out an answer.
Problem: We have a production live system that is currently working great with hibernate 4/jboss/etc. The problem is a few of our production sites are integrated with legacy systems that are using the same database, and creating data in new tables.
We've been given a requirement to action against these tables from our system. Unfortunately, the way these legacy systems work, I can't define a default entity type to match against these tables, and the table names/columns vary widely.
What I can do is find the name of the tables.
What I can't do is get hibernate to manage these tables like I want to. I've spent the last week looking for a solution, but the biggest problem I have is that the entity manager is immutable after instantiated.
Older stackoverflow questions:
Adding Tables and Columns in Hibernate on The Fly?
Has anyone have any hints on where I can possibly make this requirement achievable?
Thanks in advance.
Here's an outline of a way that this could work. Basically, you'd use Hibernate's reverse engineering tools to generate class files, then compile them on the fly, and then create a new Configuration with them and use that to generate a new EntityManager.
Hibernate reverse engineering
There's a set of IDE and Ant tools that Hibernate provides for reverse engineering and code generation which, among other thing, can generate code based on a database definition. Generally, their expectation is that people are going to generate code/mapping files based on a database either once, or during the build. What you want to do is generate those artifacts from your running program, which is definitely off-label usage. I'd start with the Ant task. End goal of this step is to have generated POJOs representing the contents of the tables in your database.
In process compilation
Now that you've got your Java files, you need to compile them. See this question for an example of using Process to run javac on your files. If you compile them to your classpath, they'll be available to your process.
Build a new configuration
Using the ejb3=true flag when you generated the POJOs, you should know be able to add them to a Configuration object already annotated. I'm not sure what the best way will be for you to actually identify the newly generated classes so that you can .addAnnotatedClass(). It may be best to capture output from the hbm2java, or perhaps to scan the output folder for the new Java files, or perhaps there's another way that your program can 'know' about the new entities. In any case, you can then use that Configuration to create a new EntityManager that maps to your new database structures.
I have an application that consists of approx. 20 java components.
About half of the components are servers and the other half batch programs.
Almost all of them talk directly to an oracle database (jdbc via some of our infrastructure code jars) the other couple of components talk to some of the servers which talk to the database.
Anyway, each component is configured with numerous XML configuration files.
These are becoming almost impossible to maintain.
Some of the configuration is specific to a component others are similar (database URLs, connectors etc)
What is worse is that the application is not installed in many environments - in fact only about 10 environments (qa, dev, production etc etc).
But the people who own these environments don't seem able to maintain the configs correctly.
In particular whenever there is an upgrade there is invariably configuration errors.
I have even started checking in some of the environments configurations into SVN along with the code.
I tried an xml schema validator at one point (it consisted of defining the valid XML in .xsd files and then throwing an error if the schema rules were breached but that didnt work)
I'm thinking I am missing something basic here - perhaps there is a tool to manage this or perhaps I should be storing the configuration in the database.
The application was largely designed by a colleague but I feel myself that it's overly configurable - in fact many of the config actually refers to classes - i.e. one can choose handlers and parsers etc - the XML config almost looks like code.
Any advice greatly appreciated
Peter
Substituting XML for code is usually a bad idea; things that are declarative are probably OK, but things that are procedural probably aren't.
If all that configuration was defined in Java code, a lot of the upgrade issues would turn into compilation issues. The compiler would pick them out for you, and you could correct them.
So you've got a multi-part problem. You need to rationalize your configuration information into a set of partitions (per-component, per-installation, global). You need to try to verify configuration information at compile-time, where possible. And you need to write validation for the loaded configurations, to sanity check them.
To the extent possible, shift configuration relatively static stuff into Guice (at least, it's what I prefer). A lot of things happen in a nice, type-safe way with it.
Consider running a WebDAV server for each instance of the app, and storing configuration into it. Each can hit a simple URL to pull the current versions of the configuration files.
Or, stand up a lightweight XML database like BaseX with its REST capability, then store and load your configuration information there. Use JSLP or something like it to have your components find the central configuration repository.
An additional advantage to using an XML DB is that you'll be able to do a lot of sanity checking and updating by querying across the set of all configuration files. For example, if a given instance of the application should have the same JDBC parameters in each configuration file, a simple xquery will tell you if that's true.
If you don't have the ability to modify the applications that are pulling the configuration file (the config file format is fixed), then consider writing a query servlets for the XML database that assemble the required configuration information, from nested blocks or templates. That will allow you to figure out what's common between the configuration files and dynamically generate parameterized versions of those blocks.
Sounds like the key here is making incremental improvements. Allow the old way to configure, but have the configuration load look for a central config source first.
I don't think that the syntax of the configuration files is at the heart of the problem: using Java properties files instead of XML would leave you with exactly the same issues. There may be an issue that the configuration information is too dispersed - it's hard to tell. The main issue seems to be that the whole thing is too fragile - the application is too dependent on manual configuration, and it seems that the configuration for each environment needs to be different. You should try to focus on reducing the number of configuration parameters that need to be set to make the system work (without necessarily reducing the options available for diagnostics etc for use when they are really needed.), on having intelligent defaults and self-configuration. Perhaps even invest in creating an installation wizard.
As you have some Oracle databases handy why not store your configuration in there?
Then you only need one or two configuration parameters to point to an Oracle database suitable for that environment and download the rest of the configuration from the database.
The contents of the configuration table should be pretty static for any given environment so there should be no need to amend anything except the jdbc connection when you migrate your software through its life cycle.