Hazelcast jet data loading - java

I'm trying to load database data into hazelcast jet map instance
public void sourceStage(JetInstance jet, Pipeline pipeLine){
SimpleDriverDataSource dataSource = new SimpleDriverDataSource();
dataSource.setDriver(new org.postgresql.Driver());
dataSource.setUrl("jdbc:postgresql://localhost/development");
dataSource.setUsername("postgres");
dataSource.setPassword("root");
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
String sql1 = "SELECT id1 , id2 FROM public.tbl_data where name='india'";
jdbcTemplate.query(sql1, new RowMapper<Object>() {
#Override
public Object mapRow(ResultSet rs, int rowNum) throws SQLException {
jet.getMap("t1").put(rs.getString("id1"), rs.getString("id2"));
return null;
}
});
}
and again trying to use same map as source in the pipe line
pipeLine.readFrom(Sources.map("t1")).map(map -> {
System.err.println(map.getKey() + "---" + map.getValue());
return map;
}).writeTo(Sinks.logger());
and I see I'm not able to read the data as source.
but if check jet.getMap("t1").size()
I'm getting the size value.

To load database data please use a pipeline with the JDBC source and IMap Sink. Mixing both to a single stage breaks the parallel execution model of the Jet engine.

Related

Bad SQL grammer jdbcTemplate spring boot

I am getting a bad SQL grammer error when trying to run this.
public List<UserTemp> findByID(Integer PatientNumber) {
String sql = "SELECT PatientNumber,FirstName,Surname from paitents WHERE 'PatientNumber' = ?,PatientNumber";
return jdbcTemplate.query(sql,
(rs, rowNum) ->
new UserTemp(
rs.getInt("PatientNumber"),
rs.getString("FirstName"),
rs.getString("Surname")
));
}
I am trying to select a patient based on the PatientNumber
Is your table called paitents or patients?

Is there any way to write custom or native queries in Java JPA (DocumentDbRepository) while firing a query to azure-cosmosdb?

Connected to azure-cosmosdb and able to fire default queries like findAll() and findById(String Id). But I can't write a native query using #Query annotation as the code is not considering it. Always considering the name of the function in respository class/interface. I need a way to fire a custom or native query to azure-cosmos db. ?!
Tried with #Query annotation. But not working.
List<MonitoringSessions> findBySessionID(#Param("sessionID") String sessionID);
#Query(nativeQuery = true, value = "SELECT * FROM MonitoringSessions M WHERE M.sessionID like :sessionID")
List<MonitoringSessions> findSessions(#Param("sessionID") String sessionID);
findBySessionID() is working as expected. findSessions() is not working. Below root error came while running the code.
Caused by: org.springframework.data.mapping.PropertyReferenceException: No property findSessions found for type MonitoringSessions
Thanks for the response. I got what I exactly wanted from the below link. Credit goes to Author of the link page.
https://cosmosdb.github.io/labs/java/technical_deep_dive/03-querying_the_database_using_sql.html
public class Program {
private final ExecutorService executorService;
private final Scheduler scheduler;
private AsyncDocumentClient client;
private final String databaseName = "UniversityDatabase";
private final String collectionId = "StudentCollection";
private int numberOfDocuments;
public Program() {
// public constructor
executorService = Executors.newFixedThreadPool(100);
scheduler = Schedulers.from(executorService);
client = new AsyncDocumentClient.Builder().withServiceEndpoint("uri")
.withMasterKeyOrResourceToken("key")
.withConnectionPolicy(ConnectionPolicy.GetDefault()).withConsistencyLevel(ConsistencyLevel.Eventual)
.build();
}
public static void main(String[] args) throws InterruptedException, JSONException {
FeedOptions options = new FeedOptions();
// as this is a multi collection enable cross partition query
options.setEnableCrossPartitionQuery(true);
// note that setMaxItemCount sets the number of items to return in a single page
// result
options.setMaxItemCount(5);
String sql = "SELECT TOP 5 s.studentAlias FROM coll s WHERE s.enrollmentYear = 2018 ORDER BY s.studentAlias";
Program p = new Program();
Observable<FeedResponse<Document>> documentQueryObservable = p.client
.queryDocuments("dbs/" + p.databaseName + "/colls/" + p.collectionId, sql, options);
// observable to an iterator
Iterator<FeedResponse<Document>> it = documentQueryObservable.toBlocking().getIterator();
while (it.hasNext()) {
FeedResponse<Document> page = it.next();
List<Document> results = page.getResults();
// here we iterate over all the items in the page result
for (Object doc : results) {
System.out.println(doc);
}
}
}
}

Cannot capture XML returned by SQL Server stored procedure using Java

I am trying to run a SQL Server 2014 stored procedure from Java (Spring) code and get some xml results.
When I run this in a SQL client e.g. RazorSQL I get a bunch of xmls (which is expected because the there are multiple stored procedures within it, that returns those xml).
Here is the Exec call from my SQL client:
EXEC [dbo].[sp_GetType]
#TRAN_ID = 42
#QUAL_ID = 0
GetType does a RETURN 0 at the end (so basically if all steps succeed, it returns 0)
This opens multiple tabs in my client with the xmls.
And one example stored procedure within GetType has these lines:
SELECT TOP 1 ModifiedBy = CASE WHEN #IS_ID = 1 FROM TABLE23.dbo.TRX
TrxId WITH (NOLOCK) WHERE #DD_ID = #TRAN_ID
FOR XML AUTO, ELEMENTS
My goal is to capture all the xmls returned by GetType into a List of objects.
Here is my dao:
private final JdbcTemplate jdbcTemplate;
#Autowired
public TransactionDao(JdbcTemplate jdbcTemplate) {
this.jdbcTemplate = jdbcTemplate;
}
#Transactional(readOnly = true)
public List<Object> getTransaction(Integer tranId, Integer qualId) {
Object dt = new Object();
List<Object> resultList = (List<Object>) jdbcTemplate.execute(
new CallableStatementCreator() {
public CallableStatement createCallableStatement(Connection con) throws SQLException {
String storedProc = "{call sp_GetType(?,?)}";
CallableStatement cs = con.prepareCall(storedProc);
cs.setInt(1, tranId);
cs.setInt(2, qualId);
return cs;
}
}, new CallableStatementCallback() {
public Object doInCallableStatement(CallableStatement > cs) throws SQLException,
DataAccessException {
List<Object> results = new ArrayList<Object>();
cs.execute();
if(cs.getMoreResults())
{
ResultSet rs = cs.getResultSet();
while(rs.next()) //rs has only one record
{
InputStream in = null;
Clob clob = rs.getClob(1);
in = clob.getAsciiStream();
}
rs.close();
}
return results;
}
});
return resultList;
}
I'm using the jtds 1.3.1 driver (I tried connecting using mssql-jdbc but no luck).
Any help is much appreciated.

Can't get results from flink SQL query

I'm facing a problem in which I don't get results from my query in Flink-SQL.
I have some informations stored in two Kafka Topics, I want to store them in two tables and perform a join between them in a streaming way.
These are my flink instructions :
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
// configure Kafka consumer
Properties props = new Properties();
props.setProperty("bootstrap.servers", "localhost:9092"); // Broker default host:port
props.setProperty("group.id", "flink-consumer"); // Consumer group ID
FlinkKafkaConsumer011<Blocks> flinkBlocksConsumer = new FlinkKafkaConsumer011<>(args[0], new BlocksSchema(), props);
flinkBlocksConsumer.setStartFromEarliest();
FlinkKafkaConsumer011<Transactions> flinkTransactionsConsumer = new FlinkKafkaConsumer011<>(args[1], new TransactionsSchema(), props);
flinkTransactionsConsumer.setStartFromEarliest();
DataStream<Blocks> blocks = env.addSource(flinkBlocksConsumer);
DataStream<Transactions> transactions = env.addSource(flinkTransactionsConsumer);
tableEnv.registerDataStream("blocksTable", blocks);
tableEnv.registerDataStream("transactionsTable", transactions);
Here is my SQL query :
Table sqlResult
= tableEnv.sqlQuery(
"SELECT block_timestamp,count(tx_hash) " +
"FROM blocksTable " +
"JOIN transactionsTable " +
"ON blocksTable.block_hash=transactionsTable.tx_hash " +
"GROUP BY blocksTable.block_timestamp");
DataStream<Test> resultStream = tableEnv
.toRetractStream(sqlResult,Row.class)
.map(t -> {
Row r = t.f1;
String field2 = r.getField(0).toString();
long count = Long.valueOf(r.getField(1).toString());
return new Test(field2,count);
})
.returns(Test.class);
Then, I print the results :
resultStream.print();
But I don't get any answers, my program is stuck...
For the schema used for serialization and deserialization, here is my test class which stores the result of my query (two fields a string and a long for respectively the block_timestamp and the count) :
public class TestSchema implements DeserializationSchema<Test>, SerializationSchema<Test> {
#Override
public Test deserialize(byte[] message) throws IOException {
return Test.fromString(new String(message));
}
#Override
public boolean isEndOfStream(Test nextElement) {
return false;
}
#Override
public byte[] serialize(Test element) {
return element.toString().getBytes();
}
#Override
public TypeInformation<Test> getProducedType() {
return TypeInformation.of(Test.class);
}
}
This is the same principle for BlockSchema and TransactionsSchema classes.
Do you know why I can't get the result of my query ? Should I test with BatchExecutionEnvironment ?

Use custom creds while using DropWizard JDBI

I'm developing a webservice using Dropwizard JDBI framework.
Now, instead of having a db configurations in yaml file, I want to use 'user specified params' what i mean to say is, the db configs will be provided through the endpoint url.
Is having custom creds possible through dropwizard jdbi?
if yes, what changes should i be thinking to do in the code while referring this ? ->
http://dropwizard.readthedocs.org/en/latest/manual/jdbi.html
I understand, in normal flow, the service method gets the config details in the run method -
-- Config Class
public class ExampleConfiguration extends Configuration {
#Valid
#NotNull
#JsonProperty
private DatabaseConfiguration database = new DatabaseConfiguration();
public DatabaseConfiguration getDatabaseConfiguration() {
return database;
}
}
-- Service Class
#Override
public void run(ExampleConfiguration config,
Environment environment) throws ClassNotFoundException {
final DBIFactory factory = new DBIFactory();
final DBI jdbi = factory.build(environment, config.getDatabaseConfiguration(), "postgresql");
final UserDAO dao = jdbi.onDemand(UserDAO.class);
environment.addResource(new UserResource(dao));
}
-- and yaml
database:
# the name of your JDBC driver
driverClass: org.postgresql.Driver
# the username
user: pg-user
# the password
password: iAMs00perSecrEET
# the JDBC URL
url: jdbc:postgresql://db.example.com/db-prod
But in this case, I might get the config details in the Resource level...
smthing like -
#GET
#Path(value = "/getProduct/{Id}/{dbUrl}/{dbUname}/{dbPass}")
#Produces(MediaType.APPLICATION_JSON)
public Product getProductById(#PathParam(value = "Id") int Id,
#PathParam(value = "dbUrl") String dbUrl,
#PathParam(value = "dbUname") String dbUname,
#PathParam(value = "dbPath") String dbPass) {
//I have to connect to the DB here! using the params i have.
return new Product(); //should return the Product
}
I'd appreciate if someone can point me a direction.
Why not just use JDBI directly?
#GET
#Path(value = "/getProduct/{Id}/{dbUrl}/{dbUname}/{dbPass}")
#Produces(MediaType.APPLICATION_JSON)
public Product getProductById(#PathParam(value = "Id") int id,
#PathParam(value = "dbUrl") String dbUrl,
#PathParam(value = "dbUname") String dbUname,
#PathParam(value = "dbPass") String dbPass) {
DataSource ds = JdbcConnectionPool.create(dbUrl, dbUname, dbPass);
DBI dbi = new DBI(ds);
ProductDAO dao = dbi.open(ProductDao.class);
Product product = dao.findById(id);
dao.close();
ds.dispose();
return product;
}
#RegisterMapper(ProductMapper.class)
static interface ProductDao {
#SqlQuery("select id from product_table where id = :id") // Whatever SQL query you need to product the product
Product findById(#Bind("id") int id);
#SqlQuery("select * from product_table")
Iterator<Product> findAllProducts();
}
static class ProductMapper implements ResultSetMapper<Product> {
public Product map(int index, ResultSet r, StatementContext ctx) throws SQLException {
return new Product(r.getInt("id")); // Whatever product constructor you need
}
}
There's a notion in the spring world of using a database router (reference: https://spring.io/blog/2007/01/23/dynamic-datasource-routing/).
You likely could setup a proxy for the database connection factory passed to DBI. That proxy would then get the credentials from a thread local (perhaps) and return the real connection giving you what you're after and still let you use the run type proxies.

Categories