Having trouble modeling DB Table, whose key (in key-value) is configurable.
So Basically i want to read battery data, which has key=ParamName, KeyValue=value of param name. Also, each battery has its own IP address.
Examples of ParamName would be 'BatteryStatus', 'BatteryTemperature', 'BatteryCurrent' etc.
My Java program would read the battery information using IP address, and get the values of all required ParamNames.
Now I could have easily defined the table with each ParamName as a column:
IP | BatteryStatus | BatteryTemperature| etc
But the problem is the ParamNames are defined in a configurable file, and I should be able to add new ParamNames or delete existing ParamNames without touchig code or DB. So I cannot use a fixed table structure then?
If I create something like below, it will duplicate the IP's
IP | ParamName | ParamValue
102.103.123.1 | BatteryStatus | "normal"
102.103.123.1 | BatteryTemperature| 32
102.103.123.1 | BatteryCurrent | 220
102.103.123.2 | BatteryStatus | "normal"
102.103.123.2 | BatteryTemperature| 35
etc.
As you can see, im trying to store a Key-{Key-Value} pair in DB. Any ideas how to do this effectively?
Having a triplet table as in your example is pretty close, assuming proper index on IP and ParamName.
The first optimisation I'd do is to replace ParamName with an INT column, with either another table that ParamName would be a foreign key to, or a lookup hashtable in your client code.
You could do the same for IP (a linked table, not the lookup hashtable), or you could just translate them to the numeric value and use it directly (What type should I store IP addresses for MySQL?), which is optimal both space- and performance-wise, just not (directly) very human-readable.
Hello friends i have complicated issue , please help me to solve this, let me explain problem :
Cassandra Table
DeviceID | companyId | data |
abc123 : xyz1 :{"Temperature":{"x":"67.824"},"Humidity":117.828,"vibration":{"y":"2.276","x":"72.995"},"date":1487641956515}
"date column data type is text"
now i am trying to get data in java using this query
ResultSet results = inSession.execute("SELECT data from keyspace.tablename where companyid = 4d91f767-2312-4f32-a25a-4674e8bae244 limit 24");
and getting data successfully like this...
{"Temperature":{"x":"67.824"},"Humidity":117.828,"vibration":{"y":"2.276","x":"72.995"},"date":1487641956515}{"Temperature":{"x":"73.981"},"Humidity":58.561,"vibration":{"y":"87.482","x":"87.131"},"date":1487641951512}{"Temperature":{"x":"62.747"},"Humidity":88.611,"vibration":{"y":"137.792","x":"36.363"},"date":1487641946512}{"Temperature":{"x":"36.072"},"Humidity":55.819,"vibration":{"y":"60.062","x":"2.779"},"date":1487641941508}{"Temperature":{"x":"36.724"},"Humidity":68.209,"vibration":{"y":"49.323","x":"64.822"},"date":1487641936507}{"Temperature":{"x":"31.777"},"Humidity":131.955,"vibration":{"y":"68.690","x":"6.737"},"date":1487641931503}{"Temperature":{"x":"41.768"},"Humidity":81.847,"vibration":{"y":"74.360","x":"60.438"},"date":1487641926499}{"Temperature":{"x":"49.538"},"Humidity":57.258,"vibration":{"y":"34.688","x":"81.397"},"date":1487641921496}{"Temperature":{"x":"98.013"},"Humidity":61.1,"vibration":{"y":"121.482","x":"93.721"},"date":1487641916492}{"Temperature":{"x":"98.307"},"Humidity":63.377,"vibration":{"y":"106.067","x":"98.968"},"date":1487641911487}{"Temperature":{"x":"92.119"},"Humidity":70.677,"vibration":{"y":"66.953","x":"59.440"},"date":1487641906481}{"Temperature":{"x":"41.627"},"Humidity":73.739,"vibration":{"y":"54.557","x":"82.876"},"date":1487641901475}{"Temperature":{"x":"74.684"},"Humidity":125.163,"vibration":{"y":"77.522","x":"96.560"},"date":1487641896471}{"Temperature":{"x":"50.228"},"Humidity":53.3,"vibration":{"y":"58.011","x":"26.710"},"date":1487641891468}{"Temperature":{"x":"61.710"},"Humidity":75.869,"vibration":{"y":"67.637","x":"69.842"},"date":1487641886465}{"Temperature":{"x":"61.908"},"Humidity":43.106,"vibration":{"y":"6.975","x":"15.009"},"date":1487641881461}{"Temperature":{"x":"75.157"},"Humidity":61.452,"vibration":{"y":"39.608","x":"58.490"},"date":1487826732069}{"Temperature":{"x":"77.562"},"Humidity":65.951,"vibration":{"y":"102.782","x":"24.761"},"date":1487826731069}{"Temperature":{"x":"60.483"},"Humidity":57.307,"vibration":{"y":"96.702","x":"86.667"},"date":1487826730068}{"Temperature":{"x":"85.893"},"Humidity":58.953,"vibration":{"y":"49.167","x":"86.790"},"date":1487826729067}{"Temperature":{"x":"84.073"},"Humidity":142.27,"vibration":{"y":"94.980","x":"65.363"},"date":1487826728065}{"Temperature":{"x":"81.733"},"Humidity":145.871,"vibration":{"y":"81.889","x":"57.215"},"date":1487826727064}{"Temperature":{"x":"41.944"},"Humidity":139.18,"vibration":{"y":"62.525","x":"74.986"},"date":1487826726063}{"Temperature":{"x":"85.298"},"Humidity":80.534,"vibration":{"y":"47.796","x":"74.527"},"date":1487826725062}
now my problem is i need to create list or array using key value , means
Temperature [24] or Temperature class list , is it possible ??? how ???
I am trying to use Hazelcast's map-reduce feature to perform an aggregate operation, which needs to access the co-located entries. The co-location is controlled using data-affinity.
Imagine the classic customer/order model as used in the Hazelcast documentation on data-affinity. In my example I want to return a customer summary that has the customer and the sum of all their orders, for example, given this data set:
customer_id | name
------------------
1 | Dave
2 | Kate
order_id | customer_id | value
------------------------------
1 | 1 | 5
2 | 1 | 10
3 | 2 | 12
I want to return:
customer_id | name | value
--------------------------
1 | Dave | 15
2 | Kate | 12
This is simple enough, however the reason for using data-affinity is to be able to perform the summing logic within the respective partition holding the data by simply getting all the orders within that partition and therefore avoiding any cross JVM communication.
And so my question, from within a Mapper or similar, how do you get the co-located entries in another cache?
EDIT:
After #noctarius' answer and comments, here's some code (I've tried to make it as brief as possible) that highlights the point at which I only want the orders from the current partition.
The order key class looks like this:
public class OrderKey implements PartitionAware<CustomerIdentity>
{
...
#Override
public CustomerIdentity getPartitionKey()
{
return this.customerIdentity;
}
...
}
And the Mapper like this:
public class OrderSumMapper implements Mapper<CustomerKey, Customer, CustomerKey, CustomerOrderTotal>, HazelcastInstanceAware
{
...
#Override
public void map(CustomerKey customerKey, Customer customer, Context<CustomerKey, CustomerOrderTotal> context)
{
Predicate ordersForCustomer = new OrdersForCustomerPredicate(customerKey);
int totalValue = 0;
//******************************************************************
//
// Given orders are co-located with the customer, how do you ensure
// this call to get the orders only runs in the current partition?
//
//******************************************************************
for (Order order : hazelcastInstance.getMap("orders").values(ordersForCustomer))
{
totalValue += order.getValue();
}
context.emit(customerKey, new CustomerOrderTotal(customer, total));
}
...
}
The highlighted call hazelcastInstance.getMap("orders").values(ordersForCustomer) would ordinarily hit all nodes in the cluster, but because the data is co-located this is an unncessary overhead.
And so back to me original question, how you I get the orders such that only those in the current partition are returned?
You just inject the current node's HazelcastInstance into your Mapper and retrieve the second data structure to read the data.
See a basic example here:
https://github.com/noctarius/hazelcast-mapreduce-presentation/blob/master/src/main/java/com/hazelcast/examples/tutorials/impl/SalaryMapper.java
I've worked it out, which will hopefully this prove useful to others (and so I'm shamelessly answering and accepting my own question).
After some experimentation it is possible to get to the objects held in another map within the partition from a Mapper running in that partition.
The first thing is to have the Mapper implement NodeAware, which causes Hazelcast to inject a reference to the Node the Mapper is running in.
Once you have the Node, you can write a method like this to access data in other maps within a given partition like this:
private Collection<Order> getCustomerOrders(CustomerKey customerKey)
{
List<Order> orders = new ArrayList<>();
MapService mapService = node.getClusterService().getNodeEngine().getService(MapService.SERVICE_NAME);
RecordStore recordStore = mapService.getRecordStore(node.getPartitionService().getPartitionId(customerKey), "orders");
for (Data key : recordStore.keySet())
{
OrderKey orderKey = mapService.getSerializationService().toObject(key);
if (customerKey.equals(orderKey.getCustomerKey()))
{
orders.add(mapService.getSerializationService().toObject(recordStore.get(key)));
}
}
return orders;
}
There's a bit of a deserialization overhead, but that'd be the case using a Predicate and working this way keeps all the processing performed by the Mapper in the JVM containing the data being mapped therefore avoiding any expensive process/network hops - basically it 'should' be faster and will definately cut down on the network traffic caused by inter-node communication.
I'm actually working on a talend job. I need to load from an excel file to an oracle 11g database.
I can't figure out how to break a field of my excel entry file within talend and load the broken string into the database.
For example I've got a field like this:
toto:12;tata:1;titi:15
And I need to load into a table, for example grade:
| name | grade |
|------|-------|
| toto |12 |
| titi |15 |
| tata |1 |
|--------------|
Thank's in advance
In a Talend job, you can use tFileInputExcel to read your Excel file, and then tNormalize to split your special column into individual rows with a separator of ";". After that, use tExtractDelimitedFields with a separator of ":" to split the normalized column into name and grade columns. Then you can use a tOracleOutput component to write the result to the database.
While this solution is more verbose than the Java snippet suggested by AlexR, it has the advantage that it stays within Talend's graphical programming model.
for(String pair : str.split(";")) {
String[] kv = pair.split(":");
// at this point you have separated values
String name = kv[0];
String grade = kv[1];
dbInsert(name, grade);
}
Now you have to implement dbInsert(). Do it either using JDBC or using any higher level tools (e.g. Hivernate, iBatis, JDO, JPA etc).
Is it possible index a complete database without mentioning the table names explicitly in the data-config.xml as new tables are added everyday and I cannot change the data-config.xml everyday to add new tables.
Haven table names based on the date smells like there is something wrong in your Design. But given this requirement in your question you can add Data to your solr server without telling you have a DB. You just have to make sure you hav a unique ID for the data record in you solr Server with whom you can identify the corresponding record in your DB, something like abcd_2011_03_19.uniqueid. You can post the data to solr in Java in solrj or just plain xml or json.
Example:
--------------
| User Input |
--------------
|post
V
-----------------------------------
| My Backend (generate unique id) |
-----------------------------------
|post(sql) |post (e.g. solrj)
V V
------ --------
| DB | | solr |
------ --------
My ascii skillz are mad :D