As I want to store data on HDFS, so need to access the HBase, so how could I connect to HBase using Java APIs.
Please suggest.
Thanks.
HBase has Java API. Have a look at http://hbase.apache.org/apidocs/index.html
Two important classes are
1) HBaseAdmin
2) HTable
HBaseAdmin is admin API used to create/delete/alter tables
HTable is the client API used to put/get/scan records.
I write a simple framework to operate on hbase.
https://github.com/zhang-xzhi/simplehbase/
Simplehbase is a lightweight ORM framework between java app and hbase.
The main feature of it are following:
data type mapping: mapping java type to hbase's bytes back and forth.
hbase operation wrapping: warpping hbase's put get scan operation to simple java interface.
hbase query language: using hbase filter, simplehbase can use sql-like style to operate on hbase.
dynamic query: like myibatis, simplehbase can use xml config file to define dynamic query to operate on hbase.
insert update support: provide insert, update on top of checkAndPut.
multiple version support: provide interface to operation on hbase's multiple version.
hbase batch operation support.
hbase native interface support.
HTablePool management.
HTable count and sum.
Use HBase as source using using TableMapper class and store in hdfs
Related
I'm testing a service that accesses dynamo DB tables, using test containers.
The necessary tables are being created in java test class before the tests run (partial code below) and everything works fine.
CreateTableRequest request = new CreateTableRequest()
.withTableName(TABLE_NAME)
.withKeySchema(keySchema)
.withAttributeDefinitions(attributeDefinitions)
.withProvisionedThroughput(new ProvisionedThroughput()
.withReadCapacityUnits(5L)
.withWriteCapacityUnits(6L));
Table table = dynamoDB.createTable(request);
table.waitForActive();
However, the "real" tables are deployed to AWS via a cloudformation template (in YAML).
My question is: Is there any way to use that template on tests? I mean, import and create those tables from it and not with the code above?
Maybe via an AWS CLI command or some library that I could use to read the YAML file, and create the tables used for testing based on the template.
Searched a lot about this, and can't find anything in Java.
Thanks in advance.
Your question - "that I could use to read the YAML file, and create the tables "
To read YAML from Java, you would need to use a supported API - for example:
https://dzone.com/articles/read-yaml-in-java-with-jackson
Once you are able to read YAML, you can use the Java SDK for Java (preferably V2) to interact with the DynamoDB service.
I have a folder with *.DDF and *.DAT files that are a pervasive/btrieve database. I am able to open and see the content of the database with DDF Periscope (ddf-periscope.com).
I can export data from each table individually using ddf periscope, and I would like to do the same thing using Java. Access the data in the DB and export them to a CSV file, POJOs or any way I can manipulate the data.
Is this possible?
You can use either JDBC or the JCL interfaces to access the data. You do still need the Pervasive engine but you can use Java. Here is a simple sample for the JDBC driver.
I don't have a JCL sample but there should be one in the Pervasive / Actian Java Class Library SDK.
I have been fetching data from database to my JAVA program by using JDBC till now and I am doing so by creating a connection class and in that class I will use the DriverManager.getConnection() fnc, but as I am moving towards JPA I have learnt that a persistence.xml file is needed to connect to mysql. In the project where I am using JPA am i supposed to create this connection class again and should I place the connector jar file in the project where I will be using jpa.
Excuse my unawarness of the concept I am still in the learning phase.Any help is appreciated as I am new to this.
Here Need to clear for you.
JPA is a specification. Different ORM technology uses it. Like hibernate implemented JPA specification. Specification defines how it works.
Hibernate is ORM technology. It binds Your Plain Java Object (Entity) to your database tables. And table's column will be Entity's field. Like Table has Id column of number type, in entity it will be Long id; . table name will be entity name or others that is defined in hibernate docs.
3rd One is Database connector. Yes there is different types of connector for every database. like for mysql connector is using to connect your implemented code with databases. You can think Its a communication Layer for database and your code. Your code is communicating with database through this connector.
Hope now you get concept why mysql connector is needed to connect. Happy Coding :)
I directly made a neo4j dataset by using neo4j console ("localhost:7474")
(as you knew "graph.db")
I wanna execute Cypher Query (by Java) Using this data.
I already saw the example from
https://github.com/neo4j/neo4j/blob/master/community/cypher/src/test/java/org/neo4j/cypher/javacompat/JavaQuery.java
I just wanna use this way but directly use existing data.
How can I do this?
If you are using Neo4j in embedded mode i.e. it runs within the same jvm as your application, you can access it using:
GraphDatabaseService graphDb = new
GraphDatabaseFactory().newEmbeddedDatabase(DBPATH)
where DBPATH is the path to the database you created using Webadmin. You can find that path in your neo4j install directory/conf/neo4j-server.properties
(The property name is org.neo4j.server.database.location)
Once you have instantiated your graphDb, you can execute Cypher queries as described in http://docs.neo4j.org/chunked/stable/tutorials-cypher-java.html
If you are not using Neo4j in embedded mode and want to connect to the existing server running on port 7474, you can use the java rest binding: https://github.com/neo4j/java-rest-binding/
I wrote this few months ago. It's very minimalist in order to be easy to understand !
https://github.com/bendaizer/neo4j_cypher_java_template
You just need to give the path to your database directory (with you data), and your cypher query. I didn't had time to write something better, so you need to recompile for each new query !
I periodically receive data that I use to update my database with. The external structure differs from my internal structure so what I end up doing is running the import and then running alter table commands. I do this manually. After I format it to my liking, I export the data and then import it into my existing schema.
My questions are:
1. How can I isolate the external SQL so that it does not adversely affect my database? Ideally, I would like to run it as another user in another database / workspace. Should I create a database temporarily and then drop it once this operation is complete?
Should I connect directly using JDBC to run all these queries since there will be a large sum of data? I am using Hibernate along with C3P0 to manage the primary connection.
Lastly, is there an API to automate/simplify exporting to SQL? If I go the JDBC route, I can iterate through each row and create the insert statements from that.
Any ideas?
Thanks,
Walter
IMO, its better to do that outside of Hibernate, using simple JDBC. Just create a connection for this thing, and execute all SQL statements. In the end close the connection. This way its handy to make a connection to another temporary database, if you choose this route. You will not need to configure all that into your Hibernate configuration.
Other way is to go with Hibernate and let it create the schema for you using entity objects and their mappings. This way you don't need to manually come up with the database structure required, it will be automatically created by Hibernate.