I'm querying a table in athena using the athena APIs.
The table has a field of type struct.
How can I map this field into a Map<String, String> record?
All records in Datum type and only way to get the values is using getVarCharValue
Datum record_value_datum = row_data.get(i);
String value = record_value_datum.getVarCharValue()
And the values are not quoted so the code above would give me sth in this format:
key1=value1, key=value2, key3=valeu3
Related
I'm trying migrate from postgres db to mongodb. I've a field JSON in postgres table, which has key and value pairs. I need to fetch the value based on JSON key and convert those values(values are basically postgres id) to Mongo objectID and then map those to mongo document.
JSON field looks like this
"B":["956c5b0a5341d14c23ceed071bf136f8"]}
I've written a function in java to convert postgres ID column to mongoID.
public String convertIDToMongo(String colName){
---encoding logic
}
This is working when the field is explicitly available in the table and also if the field datatype is not an array. In case of JSON and array, is there a way to fetch set of values from JSON field and convert them into Mongo Object ID?
"Select json::json->'A' from table" ```
gives me the value ["06992a6fef0bcfc1ede998ac7da8f08b","7d1d55e1078191e53244c41d6b55b3a3","31571835c6600517f4e4c15b5144959b"] But I need some help to convert this list of IDs to Mongo Object IDs. Below is how I convert for non JSON field with one postgres ID value to Mongo ID.
"Select " + convertIDToMongo('colName') + " as "_id", \n" +
Any help is greatly appreciated. Thanks.
You need to parse the JSON array, then call your converter for each item in the array. There are many JSON parsing options available.
I send an SQL query via apache camel to a database (SQL Server):
sql:select *from [myDataTable]?dataSource=someDataBase
Then the answer to this query from the database above is sent via apache camel to a method of some class. Like that:
public static void someMethod(#Body List< Map< String, Object > queryResultFromMyDataTable>{...}
Question: how does the database know which data type it has to produce to put in the method above?
It, obviously, cannot produce String because the method expects List<Map<String, Object>>?
I'm not an Apache Camel expert but from what I understand you get your result in kind of standard way. Every row of the myDataTable is returned as an element of the returned List. Each element contains all values of that row represented as a Map<String,Object> where the key of the map is the name of the column and the value of the map is the value retrieved from the DB for the coresponding column.
You can find more in the documentation
outputType (common)
Make the output of consumer or producer to SelectList as List of Map,
or SelectOne as single Java object in the following way: a) If the
query has only single column, then that JDBC Column object is
returned. (such as SELECT COUNT( ) FROM PROJECT will return a Long
object. b) If the query has more than one column, then it will return
a Map of that result. c) If the outputClass is set, then it will
convert the query result into an Java bean object by calling all the
setters that match the column names. It will assume your class has a
default constructor to create instance with. d) If the query resulted
in more than one rows, it throws an non-unique result exception.
StreamList streams the result of the query using an Iterator. This can
be used with the Splitter EIP in streaming mode to process the
ResultSet in streaming fashion. There are 3 enums and the value can be
one of: SelectOne, SelectList, StreamList
Good day. I am deploying a streaming job to insert data from Spark (Scala) to Postgres.
df.select("col1","col2").write.mode(SaveMode.Append).jdbc(url, "tableName", connectionProperties)
Here col2 is having uuid values in the dataframe df, but it is a string datatype. When it tries to insert in to the table that has col2 column defined as type uuid its failing with the Column is of type uuid but expression is of type character varying . Can someone help me with a workaround for this. ?
Thank you.
When you have column column in dataframe that has string data type but had values of type UUID then you can set the specific property in your connection properties and that will allow you to save your dataframe. If you don't set that property it will not allow you to save the data in the sql/postgres table, if the column has datatype of UUID.
You can set this property in the Connection Properties
connectionProps.setProperty("stringtype", "unspecified")
Setting the above property will allow you to save the string type column having UUID values in the dataframe to a column of type UUID in the postgres database.
I'm trying to fetch custom columns data from sa360 into MySQL DB. So I have a column having the name Blended KPI 85/10/5. So I have saved the column name in the DB as well Blended KPI 85/10/5.
So first the data gets stored in a CSV file and then I'm reading the records from CSV file and capturing it in a List<Map<String, Object>> and later these records are to be stored into the DB. Since I'm having 5000+ records, I'm using batch insert. So I'm facing some issue syntax type error. Please see the below code snippet and the error.
I did try handling escape characters but with no success.
Values inside dailyRecords:
{account_id=2, brand_id=2, platform_id=1, campaign_id=71700000028596159, Blended_KPI_85/10/5=0.0, CPB_(85/10/5)=0.0}
Code:
String sql = "INSERT INTO campaign_table (`account_id` ,`brand_id` ,`platform_id` ,`campaign_id` , `Blended KPI 85/10/5` , `CPB (85/10/5)` ) VALUES (:account_id, :brand_id, :platform_id, :campaign_id, :Blended_KPI_85/10/5 , :CPB_(85/10/5))"
namedParameterJdbcTemplate.batchUpdate(sql, dailyRecords.toArray(new Map[dailyRecords.size()]));
On executing, I'm getting the below error:
No value supplied for the SQL parameter 'Blended_KPI_85': No value registered for key 'Blended_KPI_85'
You cannot use the characters of /,(,) for a placeholder name because they are reserved characters for the SQL syntax. A quick workaround is to change the names of placeholders in the SQL statement and also change the keys as well in your data.
You can easily modify the keys of the Map inside your data by the help of collection streams if your Java version is 8 or above:
String sql = "INSERT INTO campaign_table (`account_id` ,`brand_id` ,`platform_id` ,`campaign_id` ,`Blended KPI 85/10/5` ,`CPB (85/10/5)`) VALUES (:account_id, :brand_id, :platform_id, :campaign_id, :Blended_KPI_85_10_5 , :CPB_85_10_5)"
Map[] params = dailyRecords.stream().map(m -> {
m.put("Blended_KPI_85_10_5", m.get("Blended_KPI_85/10/5"));
m.put("CPB_85_10_5", m.get("CPB_(85/10/5)"));
return m;
}).toArray(Map[]::new);
namedParameterJdbcTemplate.batchUpdate(sql, params);
Note that I removed these characters and changed the placeholder names in your sql statement as below:
:Blended_KPI_85/10/5 => :Blended_KPI_85_10_5
:CPB_(85/10/5) => :CPB_85_10_5
Hope this helps. Cheers!
I have a Cassandra CQL3 column family with the following structure
CREATE TABLE mytable(
A text,
B text,
C text,
mymap map<text,text>,
D text,
PRIMARY KEY (A,B,C)
);
I am trying to insert a bunch of data into it using Astyanax.
The version of Cassandra that I am working with is 1.2, so I can't use BATCH insert.
I know that I can run CQL3 commands in a for loop using Prepared Statements.
I wanted to know if it's possible to use Astyanax mutation batch to insert the data into the above column family? I realize that this will make use of the Astyanax Thrift interface to insert into a CQL3 column family but for the sake of performant writes, is this a viable option?
I took a look at the structure of the column family in cassandra-cli and it looks something like this
ColumnFamily: mytable
Key Validation Class: org.apache.cassandra.db.marshal.UTF8Type
Default column value validator: org.apache.cassandra.db.marshal.BytesType
Cells sorted by: org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.ColumnToCollectionType(6974656d73:org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type)))
While I can insert data into the other columns (i.e A, B, C, D) by creating a POJO with #Component on the various fields, I'm not sure how to go about dealing with the map insert i.e. inserting into the mymap column.
A sample POJO that I created is
public class TestColumn {
#Component(ordinal = 0)
String bComponent;
#Component(ordinal = 1)
String cComponent;
#Component(ordinal = 2)
String columnName;
public TestColumn() {
}
}
The insertion code is as follows
AnnotatedCompositeSerializer columnSerializer = new AnnotatedCompositeSerializer(TestColumn.class);
ColumnFamily columnFamily = new ColumnFamily("mytable", StringSerializer.get(), columnSerializer);
final MutationBatch m = keyspace.prepareMutationBatch();
ColumnListMutation columnListMutation = m.withRow(columnFamily, "AVal");
columnListMutation.putColumn(new TestColumn("BVal", "CVal", null), ByteBufferUtil.EMPTY_BYTE_BUFFER,
timeToLive);
columnListMutation.putColumn(new ApiAvColumn("BVal", "CVal", "D"), "DVal",
timeToLive);
m.execute;
How exactly should I modify the above code so that I can insert the map value as well?
We solved this by using the DataStax Java Driver instead of Astyanax.