To avoid creating SQL statements as strings in a class I've placed them as .sql files in the same package and read the contents to a string in the static constructor. The reason for this is the SQL is very complex due to an ERP system that the SQL is querying.
There's no problem with this method, though since the SQL reading mechanism quite simply just reads the whole file any comments within that file may cause the read to fail if they are at the end of the line, as when reading it first removes excess whitespace and removes new-lines. Full commented lines (i.e. lines beginning with -- are removed).
I could enhance the simple reading to read the file and remove commented lines etc, though I have to wonder if there is something already available that could read an SQL file and clean it up.
I've seen this same problem solved in a project I've worked on by storing queries in XML, and loading the XML into a custom StoredQueriesCache object at runtime. To get a query, we would call a method on the StoredQueriesCache object and just pass the query name (which is defined in the XML), and it would return the query.
Writing something like this is fairly simple. The XML would look something like this below...
<Query>
<Name>SomeUniqueQueryName</Name>
<SQL>
SELECT someColumn FROM someTable WHERE somePredicate
</SQL>
</Query>
You would have one element for every stored query. The XML would be loaded into memory at application startup from file, or depending on your needs it could be lazy loaded from file. Then your StoredQueriesCache object that holds the XML would have methods to return individual queries by name. In my experience, having comments in the query has never caused any issue since linebreaks are part of the XML node's innertext, but if you want your StoredQueriesCache methods that retrieve the queries could parse comments out.
I've found this to be the most organized way of storing queries without embedding them in code, and without using stored procedures. There should honestly be a library that does this for you; maybe I'll write one!
Related
We have an event system producing database events for change data capture.
The system sends an event which contains the INSERT or UPDATE statement with ? placeholders and an array of the ordered values matching each question mark.
I want to use this for per hour backup files so if I get a statement like:
insert into T0(a,b,c) VALUES(?,?,?)
with an array of values 1, 2 and it's his then I write the a line to the backup file for that hour as
insert into T0(a,b,c) VALUES(1,2,'it\'s his');
A few things:
Is it only strings that need escaping? We don't have or allow binary columns
Is there a Java library that can do this already (from the Spring eco-system, Apache or otherwise)?
I've seen the Postgres JDBC code for escaping https://github.com/pgjdbc/pgjdbc/blob/master/pgjdbc/src/main/java/org/postgresql/core/Utils.java - is that sufficient?
I was also thinking of creating a SQLite database for each hour, writing to SQLite and then dumping it to the hr.sql text file. This has the advantage of capitalising on all the hardwork and thought already put into SQLite handling escaping but feels like overkill if there's a way to do the toString in Java then append a line to the file.
There's a performance consideration in using SQLite as well furthering my hesitation to that that route.
Found some options.
Postgres JDBC driver is this https://github.com/pgjdbc/pgjdbc/blob/master/pgjdbc/src/main/java/org/postgresql/core/Utils.java and other impl. is even more simple https://github.com/p6spy/p6spy/blob/master/src/main/java/com/p6spy/engine/common/Value.java#L172 literally doing
ESPECIAL_CHARACTER_PATTERN.matcher(stringValue).replaceAll("''")
Where private static final Pattern ESPECIAL_CHARACTER_PATTERN = Pattern.compile("'");
In both cases, only strings need this as I thought and binary is handled separately but we don't have/need binary.
Digging further I rediscovered ESAPI https://github.com/ESAPI/esapi-java-legacy
They have a lib for escaping SQL https://cheatsheetseries.owasp.org/cheatsheets/SQL_Injection_Prevention_Cheat_Sheet.html#defense-option-4-escaping-all-user-supplied-input
https://github.com/ESAPI/esapi-java-legacy/blob/develop/src/main/java/org/owasp/esapi/codecs/MySQLCodec.java
I'm using Knime and I need to be able to cross reference a value within a csv file against a value I get from an Oracle DB.
Specifically, I need to match a ZIP code I get from the DB to a CSV file I have that contains zip codes and their corresponding counties.
I'm not really sure how to approach it. I've tried Joins and Cross joins but the data ends up looking garbled and I'm unable to make any sense of it. Worst case scenario I end up manually looking things up.
If you are adding only one column (county in this example) I prefer Cell Replacer node with option to Append column. Easier to configure and faster than Joiner node.
I'm still fairly new to hibernate. I am uploading an sql script and auditing each statement in to a db. So, every statement will be saved as a string in to the database. however this file could contain up to 50,000+ statements. I've been reading up on hibernate batching, but i'm wondering what would be the best way to design and implement this.
So far, the file is uploading fine, i am creating a List out of each statement in the script, and then i save each object through hibernate individually. Obviously not great for performance!
I am wondering if i should still make a gigantic List of 50,000+ objects from the script - on controller side then pass it on to DAO, or should i parse through the file, say 100 rows at a time, and create a List of 100 objects, passing each list through to service->DAO.. and do so continuously until end of file.
How would the experts handle this design??
Thanks!
Take a look at spring-batch: with a job composed by 2 steps (file upload + data read/write) you'll solve your problem
I am having an xml file which contains data from different tables.
These tables are linked to each other.
I want to access Records from the xml.
Can i write SQL select query on Xml file.
No, you cannot use SQL for XML files. Either move the data to a relational store or use a hierarchical query language.
All you can expect after such a vague question is a bunch of random keywords (like XQuery, eXist, XPath, Oracle XML Db, MarkLogic, Jaxen). Probably none of them is relevant to whatever problem you might have at hand.
I'm quite new to Java Programming and am writing my first desktop app, this app takes a unique isbn and first checks to see if its all ready held in the local DB, if it is then it just reads from the local DB, if not it requests the data from isbndb.com and enters it into the DB the local DB is in XML format. Now what im wondering is which of the following two methods would create the least overhead when checking to see if the entry all ready exists.
Method 1.) File Exists.
On creating said DB entry the app would create a seperate file for every isbn number named isbn number.xml (ie. 3846504937540.xml) and when checking would use the file exists method to check if an entry all ready exists using the user provided isbn .
Method 2.) SAX XML Parser.
All entries would be entered into a single large XML file and when checking for existing entries the SAX XML Parser would be used to parse the file and then the user provided isbn would be checked against those in the XML DB for a match.
Note :
The resulting entries could number in the thousands over time.
Any information would be greatly appreciated.
I don't think either of your methods is all that great. I strongly suggest using a DBMS to store the data. If you don't have a DBMS on the system, or if you want an app that can run on systems without an installed DBMS, take a look at using SQLite. You can use it from Java with SQLiteJDBC by David Crawshaw.
As far as your two methods are concerned, the first will generate a huge amount of file clutter, not to mention maintenance and consistency headaches. The second method will be slow once you have a sizable number of entries because you basically have to read (on the average) half the data base for every query. With a DBMS, you can avoid this by defining indexes for the info you need to look up quickly. The DBMS will automatically maintain the indexes.
I don't like too much the idea of relying on the file system for that task: I don't know how critical is your application, but many things may happen to these xml files :) plus, if the folder gets very very big, you would need to think about splitting these files in some hierarchcal folder structure, to have decent performance.
On the other hand, I don't see why using an xml file as a database, if you need to update frequently.
I would use a relational database, and add a new record in a table for each entry, with an index on the isbn_number column.
If you are in the thousands records, you may very well go with sqlite, and you can replace it with a more powerful non-embedded DB if you ever need it, with no (or little :) ) code modification.
I think you'd better use DBMS instead of your 2 methods.
If you want least overhead just for checking existence, then option 1 is probably what you want, since it's direct look up. Parsing XML each time for checking requires you to to pass through the whole XML file in worst case. Although you can do caching with option 2 but that gets more complicated than option 1.
With option 1 though, you need to beware that there is a limit of how many files you can store under a directory, so you probably have to store the XML files by multiple layer (for example /xmldb/38/46/3846504937540.xml).
That said, neither of your options is good way to store data in the long run, you will find them become quite restrictive and hard to manage as data grows.
People already recommended using DBMS and I agree. On top of that I would suggest you to look into document-based database like MongoDB as your database.
Extend your db table to not only include the XML string but also the ISBN number.
Then you select the XML column based on the ISBN column.
Query: Java escaped, "select XMLString from cacheTable where isbn='"+ isbn +"'"
A different approach could be to use an ORM like Hibernate.
In ORM instead of saving the whole XML document in one column you use different different columns for each element and attribute and you could even split upp your document over several tables for a simpler long term design.