We Keep Coding

Different response for same method of DatabaseMetaData pointing to different databases?

When I invoke the method getIndexInfo( catalog, schema, table, true, false ), I receive a ResultSet slightly different from what is described:
With MySQL (5.7), I receive a ResultSet containing:
one row corresponding to the primary key column description
n rows corresponding to the unique columns description
With SQL Server (14.00), I receive a ResultSet containing:
one row corresponding to the tableIndexStatistic of the primary key
one row corresponding to the primary key column description
n rows corresponding to the unique columns description
m rows corresponding to the index columns description
Due to a project choice all the primary keys are auto-increment, so there aren't case where a primary key column is also a unique column.
I'm searching to write a solution database-independent, since it will be used for both MySQL and SQL Server databases;
MySQL use the MySQL-AB JDBC Driver 5.1.20, SQL Server use the Microsoft JDBC Driver 6.4.
Initially I "resolved" this problem retrieving from the session the driver name, in order to apply a specific filter for each database;
for MySQL i found that the column INDEX_NAME for the Primary Key is always 'PRIMARY', meanwhile for SQL Server I found that the column TYPE is:
0 for the tableIndexStatistic
1 for our SQL Server Primary Keys (tableIndexClustered )
2 (not found in my case yet, but is for tableIndexHashed )
3 for the Unique keys ( tableIndexOther )
A difference between MySQL and SQL Server is that the Primary Keys are respectively of TYPE 3 and 1.
Filter example:
String driver = session.getConfiguration().getDatabaseId();
DatabaseMetaData metadata = session.getConnection().getMetaData();
ResultSet result = metadata.getIndexInfo(catalog, schema, table, true, false);
while( result.next() ){
if( "mysql".equals(driver) ){
if( !"PRIMARY".equals((String) result.getObject("INDEX_NAME"))){
... code to save the result ...
}
} else if ( "sqlserver".equals(driver) ){
if( 3 == (short) result.getObject("TYPE")){
... code to save the result ...
}
} else {
throw new Exception();
}
}
This code worked for a bit, until I discovered on SQL Server a table with an index; in this case, as per documentation linked before, the indexes are part of the tableIndexOther so they have the column TYPE with the value 3.
At this point I've noticed that the column NON_UNIQUE is true for the Unique columns descriptions and false for the Index columns descriptions.
So I was thinking to proceed expanding the SQL Server filter including the NON_UNIQUE column but, against as described in the documentation, when I retrieve a tableIndexStatistic I'll get null instead of false.
I'm a bit confused of how I should approach all those inconsistencies with the documentation, since my main goal is to retrieve the same result of unique keys from those two databases.

Read a table from a SAP system using JCo

I am trying to read a table from an SAP system and I am always getting this error:
Exception in thread "main" com.sap.conn.jco.JCoRuntimeException: (127)
JCO_ERROR_FIELD_NOT_FOUND: Field EMPLOYEE is not a member of INPUT
at com.sap.conn.jco.rt.AbstractMetaData.indexOf(AbstractMetaData.java:404)
at com.sap.conn.jco.rt.AbstractRecord.setValue(AbstractRecord.java:4074)
at testConf.StepServer.main(StepServer.java:50)
And here is my code :
public static void main(String[] args) {
// This will create a file called mySAPSystem.jcoDestination
System.out.println("executing");
String DESTINATION_NAME1 = "mySAPSystem";
Properties connectProperties = new Properties();
connectProperties.setProperty(DestinationDataProvider.JCO_ASHOST, "xxx.xxx.x.xxx");
connectProperties.setProperty(DestinationDataProvider.JCO_SYSNR, "xx");
connectProperties.setProperty(DestinationDataProvider.JCO_CLIENT, "xxx");
connectProperties.setProperty(DestinationDataProvider.JCO_USER, "username");
connectProperties.setProperty(DestinationDataProvider.JCO_PASSWD, "test");
connectProperties.setProperty(DestinationDataProvider.JCO_LANG, "en");
createDestinationDataFile(DESTINATION_NAME1, connectProperties);
// This will use that destination file to connect to SAP
try {
JCoDestination destination = JCoDestinationManager.getDestination("mySAPSystem");
System.out.println("Attributes:");
System.out.println(destination.getAttributes());
System.out.println();
destination.ping();
} catch (JCoException e) {
e.printStackTrace();
}
try{
//here starts the problem
JCoDestination destination = JCoDestinationManager.getDestination(DESTINATION_NAME1);
JCoFunction function = destination.getRepository().getFunction("RFC_READ_TABLE");
JCoParameterList listParam = function.getImportParameterList();
listParam.setValue("EMPLOYEE", "EMPLOYEE"); // I have found this in an example and I don't understand exactly what should I put there
// I was thinking maybe is the column name but I am not sure
function.execute(destination);
JCoTable table = function.getTableParameterList().getTable("ZEMPLOYEES");//name of my table from SAP
System.out.println(table);
}
catch (JCoException e)
{
System.out.println(e.toString());
return;
}
}
The error is clear when it says JCO_ERROR_FIELD_NOT_FOUND: Field EMPLOYEE is not a member of INPUT but the employee is a field in my table.
The documentation doesn't help too much, it only says:
Sets the object as the value for the named field.
Parameters:
value - the value to set for the field
name - the name of the field to set
Witch, in my opinion, I have already done.
Should I make any additional modification in sap, in order to read this new table from java? All I have done is to create a new table following this tutorial (Create a simple table in SAP).
Maybe someone with more experience can tell me how should I configure this sample code in order to work.

General use of RFC_READ_TABLE
I never used JCo, but as far as I know its interface is very similar to NCo, the .Net connector. This is basically NCo code with some guesswork added to it, but it should work.
// get the table parameter FIELDS that should be in the parameter list
// the parameter table has several fields, only the field FIELDNAME has to be set before calling the function module
JCOTable inputTableParam = function.getTableParameterList().getTable("FIELDS");
// add a row to the FIELDS table parameter
inputTableParam.appendRow();
// set values for the new row
inputTableParam.setValue("FIELDNAME", "EMPLOYEE");
// just for fun, add another field to retrieve
inputTableParam.appendRow();
inputTableParam.setValue("FIELDNAME", "SURNAME");
// now we have to set the non-table parameters
JCoParameterList inputParamList = function.getImportParameterList();
// parameter QUERY_TABLE, defines which table to query
inputParamList.setValue("QUERY_TABLE", "ZEMPLOYEES");
// parameter DELIMITER - we get a single string as the return value, the field values within that string are delimited by this character
inputParamList.setValue("DELIMITER", ";");
// execute the function
function.execute(destination);
// the parameter table DATA contains the rows
JCoTable table = function.getTableParameterList().getTable("DATA");
in the end, your variable table will hold a table object with a single field called WA. That field contains the contents of the fields you selected in input parameter table FIELDS. You can iterate over table and get the values row by row.
Queries with RFC_READ_TABLE
RFC_READ_TABLE doesn't really allow queries, it only allows you to define WHERE clauses. The TABLE parameter OPTIONS has a single field TEXT, 72 characters wide, that can only take ABAP compliant WHERE clauses.
to extend the example above, we'll add a where clause to only select entries from table ZEMPLOYEES with SURNAME = "SMITH" and FORNAME = "JOHN".
JCOTable optionsTableParam = function.getTableParameterList().getTable("OPTIONS");
// add a row to the FIELDS table parameter
optionsTableParam.appendRow();
optionsTableParam.setValue("TEXT", "SURNAME EQ 'SMITH' AND FORNAME EQ 'JOHN');
the field TEXT is only 72 characters long, so if you want to add a longer clause, you manually have to break your conditions into several rows. RFC_READ_TABLE is a bit crude and limited.
Complex joins between tables can be achieved by creating a view within the SAP system (transaction SE11) and then query that view with RFC_READ_TABLE.
If you want to call function modules from JCo, it would be very helpful if you made yourself familiar with the basic function module properties. You can look at a function module definition in transaction SE37. There you can see the IMPORT, EXPORT, CHANGING and TABLE parameters. The parameters you have to fill and the parameters that contain the results depend on the function module you call - RFC_READ_TABLE has different ones from, say, BAPI_DELIVERY_GETLIST.
Here is the documentation for JCoFunction and one of the differences between JCo and NCo, JCo has individual functions to get and set the different parameter types: https://help.hana.ondemand.com/javadoc/com/sap/conn/jco/JCoFunction.html

You are trying to call the function RFC_READ_TABLE and you try to pass a value to its parameter named "EMPLOYEE". This is NOT a parameter of RFC_READ_TABLE, hence the error.
RFC_READ_TABLE has 3 important input parameters :
QUERY_TABLE : the name of the database table you want to query
OPTIONS : the WHERE clause (you may pass an empty value)
FIELDS : the list of columns from the database table you want to query
RFC_READ_TABLE has 1 return parameter :
DATA : the contents of the table
See this example : https://vishalmasih.wordpress.com/2014/10/31/sap-jco-searching-for-a-user-in-the-usr04-table/

Spark read() works but sql() throws Database not found

I'm using Spark 2.1 to read data from Cassandra in Java.
I tried the code posted in https://stackoverflow.com/a/39890996/1151472 (with SparkSession) and it worked. However when I replaced spark.read() method with spark.sql() one, the following exception is thrown:
Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `wiki`.`treated_article`; line 1 pos 14;
'Project [*]
+- 'UnresolvedRelation `wiki`.`treated_article`
at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
I'm using same spark configuration for both read and sql methods
read() code:
Dataset dataset =
spark.read().format("org.apache.spark.sql.cassandra")
.options(new HashMap<String, String>() {
{
put("keyspace", "wiki");
put("table", "treated_article");
}
}).load();
sql() code:
spark.sql("SELECT * FROM WIKI.TREATED_ARTICLE");

Spark Sql uses a Catalogue to look up database and table references. When you write in a table identifier that isn't in the catalogue it will throw errors like the one you posted. The read command doesn't require a catalogue since you are required to specify all of the relevant information in the invocation.
You can add entries to the catalogue either by
Registering DataSets as Views
First create your DataSet
spark.read().format("org.apache.spark.sql.cassandra")
.options(new HashMap<String, String>() {
{
put("keyspace", "wiki");
put("table", "treated_article");
}
}).load();
Then use one of the catalogue registry functions
void createGlobalTempView(String viewName)
Creates a global temporary view using the given name.
void createOrReplaceTempView(String viewName)
Creates a local temporary view using the given name.
void createTempView(String viewName)
Creates a local temporary view using the given name
OR Using a SQL Create Statement
CREATE TEMPORARY VIEW words
USING org.apache.spark.sql.cassandra
OPTIONS (
table "words",
keyspace "test",
cluster "Test Cluster",
pushdown "true"
)
Once added to the catalogue by either of these methods you can reference the table in all sql calls issued by that context.
Example
CREATE TEMPORARY VIEW words
USING org.apache.spark.sql.cassandra
OPTIONS (
table "words",
keyspace "test"
);
SELECT * FROM words;
// Hello 1
// World 2
The Datastax (My employer) Enterprise software automatically registers all Cassandra tables by placing entries in the Hive Metastore used by Spark as a Catalogue. This makes all tables accessible without manual registration.
This method allows for select statements to be used without an accompanying CREATE VIEW

I cannot think of a way to make that work off the top of my head. The problem lies in that Spark doesn't know the format to try, and the location that this would be specified is taken by the keyspace. The closest documentation for something like this that I can find is here in the DataFrames section of the Cassandra connector documentation. You can try to specify a using statement, but I don't think that will work inside of a select. So, your best bet beyond that is to create a PR to handle this case, or stick with the read DSL.

JOOQ create Table and insert values DSLContext

I'm trying to generate a Database using JOOQ
i do create a Table with this code:
CreateTableAsStep<Record> table = create.createTable("TestTable");
CreateTableColumnStep step = table.column("testColumn", SQLDataType.Integer);
step.execute();
this works fine, but when it comes to inserting data, i run into a problem
the doc includes the following example:
create.insertInto(AUTHOR)
.set(AUTHOR.ID, 100)
.set(AUTHOR.FIRST_NAME, "Hermann")
.set(AUTHOR.LAST_NAME, "Hesse")
.newRecord()
.set(AUTHOR.ID, 101)
.set(AUTHOR.FIRST_NAME, "Alfred")
.set(AUTHOR.LAST_NAME, "Döblin")
.execute();
here AUTHOR is not a simple String it expects a org.jooq.Table<R extends Record>
i thought there might be a return type when creating the table, but i did not find it. Google did not help as Table is not the best word to search for ;-)
Question: how can i get to an instance of a Table - i do have its name as String?

You can always create Table references via DSL.table(String) or DSL.table(Name). For example:
// Assuming this:
import static org.jooq.impl.DSL.*;
create.insertInto(table(name("TestTable")))
.set(field(name("testColumn")), 1)
.execute();
Notice also my usage of DSL.field(Name).
Plain SQL vs. Name references
It's worth reading up on the difference between creating dynamic table / field objects at runtime either with plain SQL strings (as in DSL.table(String)) or with name references (as in DSL.table(Name)). Essentially:
Plain SQL strings are case-insensitive and subject to SQL injection
Name references are case-sensitive by default
In your case, as you probably created case sensitive table/column names, you should prefer the latter. More info can be found here:
http://www.jooq.org/doc/latest/manual/sql-building/plain-sql
http://www.jooq.org/doc/latest/manual/sql-building/names

How to design an exception logging table using HyperTable and access it via the Java client?

If I have the following table schema to log an exception (in standard SQL schema):
Table: ExceptionLog
Columns: ID (Long),
ExceptionClass (String),
ExceptionMessage (String),
Host (String),
Port (Integer),
HttpHeader (String),
HttpPostBody (String),
HttpMethod (String)
How would I design the same thing in HyperTable (specifically, what is the best approach for efficiency)? And, how would I code it using the HyperTable Java client?

What is ID - is it an auto-incremented unique ID? Hypertable does not have auto-increments, but uses random GUIDs for unique IDs if you don't want to provide your own ID. Here's a link with more information:
http://hypertable.com/documentation/developer_guide/#unique-cells
I would maybe combine Host and Port into a single column ("hypertable.com:8080"), but that's my personal preference.
Everything else looks fine. You can simply translate this to a CREATE TABLE statement in HQL:
CREATE TABLE ExceptionLog (ID, ExceptionClass, ExceptionMessage, Host, Port, HttpHeader, HttpPostBody, HttpMethod);
You might also want to have secondary indices, i.e. on ExceptionClass, if you have frequent queries like
SELECT ExceptionClass FROM ExceptionLog WHERE ExceptionClass = "Segfault";
Secondary indices are documented here: http://hypertable.com/documentation/developer_guide/#secondary-indices
Here's a sample which shows how to use the Java client. It's fairly simple: https://github.com/cruppstahl/hypertable/blob/v0.9.6/src/java/ThriftClient/org/hypertable/thrift/BasicClientTest.java