We Keep Coding

What is the best way to query data stored in HDFS using Spark?

I would like to create a Java app that queries data in HDFS using Spark. Until now, I've tested doing this in 2 ways:
- making SQL queries to the JDBC endpoint exposed by the Thrift server (started with the default configurations)
- using the spark dataset api
My question is, being completely new to hadoop/spark, which of the 2 ways would be most efficient and easier to set up ( without the default configurations)?
From what I understand until now, using Thrift server requires configuration and maintainance of Thrift and Hive. On the other hand, I expect that using the dataset API would be slower and with more limitations, keeping the data in memory.

The thrift server does require a slight bit more configuration and required a hive metastore to keep table definitions, you youre able to query everything using sql. At the end of the day, the performance between running a thrift server query and a query using the untyped dataset api is basically the same, functionally you have more flexibility with the dataset api. The strongly typed dataset apis are less performant than the untyped dataset api due to the codegen spitting out bad code (especially pre-spark 2.2).

I want to create one standalone java application product. is is feasible to use HSQL DB ? cause customer may have very historical data

I want to create a one stanalone product, i.e a standalone java application, and I need a database for that. So I was thinking to use HSQL db.
Can HSQLDB is fesible to very historical data like 5 years, 8 years?
Is HSQLDB is suitable for that? Or do you know any alternative? (I know MYSQL but I don't want to use MYSQL, I want to use embedded database)

SQL Queries to be used to find path between two points

For the past year I have been working on an Android based
Transit application that runs off a Restful XML based web service
written in PHP.
I have a little trouble properly querying GTFS Data from my MySQL
Database. I have imported the feed files exactly into the Database as
they are in .cvs form (including Field Names).
Does anyone have proper SQL Queries to be used to find path between two points

If that could be done in SQL (non procedural) it would be very interesting. I imagine it would use connect by syntax, or recursive CTEs. But since those aren't suppported by MySQL or SQLite (not sure which one you're asking about), I would go with doing simple queries and doing the processing in Java.

Using SQLXMLBulkload via Java

Is it possible to use the SQLXMLBulkload component of SQL Server using Java? I am using SQL Server 2005 database for my Java EE application.
If not then is there any similar API for Java?

Ultimately I found out that there is no nice way of doing it. Persisting XML to Relation database is still a challenge.
For my project I end up marshaling XML to Java and then persisted Java objects to DB with the help of Hiberante:
I understand that performance is a big concern with this approach but I didn't hit any performance issue yet. Plus this approach give me flexibility for example saving attribute values in separate columns etc.

Does scala play 1.2.4 have compatibility with postgres?

There are so many horror stories about postgres and Play framework here on SO and on blogs.
So I am wondering does Play 1.2.4, scala and anrom(magic) work with postgres or not.
In my test case I get the following when switching to Postgres.
The db is correctly created by 1.sql in evolutions. And the log prints Connected to jdbc:postgresql://localhost/postgres_play

Anorm works just fine with postgresql, we're currently converting a complex postgresql based app from PHP to play with scala and have had no DB related issues at all. As to your error, I'm not 100% certain but it may be simply that it is looking for the table "Player" and you have the table "player" in your database. Show us the code where you are defining the Player object that extends magic.