We Keep Coding

BigQuery Client bigquery.listTableData with Filter condition

I am using the below lines of code to fetch the data from Google Bigquery.
public class BQTEST {
public static void main(String... args) throws Exception {
String datasetName = "mydataset";
String tableName = "mytable";
String projectId = "id-gcs";
String query =
"SELECT id, " +
"qtr, " +
"sales, " +
"year " +
"FROM `id-gcs.mydataset.mytable` " +
"where MOD(id,2) = 0";
BigQuery bigquery = BigQueryOptions.newBuilder().setProjectId(projectId)
.setCredentials(
ServiceAccountCredentials.fromStream(new
FileInputStream("gcs.json"))
)
.build().getService();
TableId tableId = TableId.of(projectId, datasetName, tableName);
QueryJobConfiguration queryConfig = QueryJobConfiguration
.newBuilder(query)
.setPriority(QueryJobConfiguration.Priority.BATCH)
.build();
try {
bigquery.query(queryConfig);
} catch (InterruptedException e) {
e.printStackTrace();
throw new RuntimeException(e.getMessage());
}
TableResult results = bigquery.listTableData(
tableId,
BigQuery.TableDataListOption.pageSize(1)
);
for (FieldValueList row : results.iterateAll()) {
System.out.printf(
"ID: %s qtr: %s sales: %s year: %s\n", row.get(0).getValue(), row.get(1).getValue(), row.get(2).getValue(), row.get(3).getValue());
}
}
}
I have 12 records in a source table starting id value from 1,2,3...12. Since I applied Mod on ID the result set should be with the id value as 2,4,6,8,10,12.
Instead, it's returning the whole data as a resultset.
So, the condition in where clause is not applied.
Seeking help on this .

You're doing two unrelated things here: running a query, and then attempting to read the rows directly from the source table you just queried. The reason you're not getting filtered results is you're not reading rows from the query results, you're reading rows directly from the source table.
Upon further review, it looks like this is based on some sample code that's misleading; I'll get that addressed.
A short example that may be more illuminating: https://cloud.google.com/bigquery/docs/samples/bigquery-query#bigquery_query-java
Notable, see how the result iterator is returned from bigquery.query(). That's where your filtered results are available, not by iterating against the source table.

Android - sqlite in clause using string values from arraylist?

public void DBSearchCategory(String tableName) {
// 1st way
String inClause = s1.ListViewCategory.toString();
inClause = inClause.replace("[", "(");
inClause = inClause.replace("]", ")");
// Cursor cursor = database.rawQuery("SELECT CATEGORY FROM " + tableName
// + " WHERE CATEGORY NOT IN " + inClause
// + " ORDER BY RANDOM() LIMIT 1 ", null);
// 2nd way
try {
StringBuilder sb = new StringBuilder("");
for (String param : s1.ListViewCategory) {
sb.append(",").append('"').append(param).append('"');
}
params = sb.toString().substring(1);
Log.v("Tag", "params value is " + params);
} catch (StringIndexOutOfBoundsException e) {
}
Cursor cursor = database.rawQuery("SELECT CATEGORY FROM " + tableName
+ " WHERE CATEGORY NOT IN (?) "
+ " ORDER BY RANDOM() LIMIT 1 ", new String[]{params});
while (cursor.moveToNext()) {
category = cursor.getString(cursor.getColumnIndex("CATEGORY"));
s1.keyCategory = category;
}
cursor.close();
}
s1.ListViewCategory is a String type ArrayList in Singleton class s1, and it has values of categories: "game","country","city","subway","actor","pet" // In Database there are total 33 categories, and I want to exclude these 6 categories that are in s1.ListViewCategory
In rawQuery, I want to exclude categories that are in s1.ListViewCategory, so I tried 2 ways of cursor refering to these 2 stackoverflow questions:
Android - sqlite in clause using string values from array?
///Android - sqlite in clause using values from array
I used WHERE and NOT IN statement to exclude these 6 categories
When I tried 2nd way cursor, I got no error. However, the Sql query did not work. It had to exclude categories that are in String[params], but it did not work. So I used log to see what param is and I got this
2020-01-09 09:16:47.233 8978-8978/com.kj.word V/Tag: params value is
"game","country","city","subway","actor","pet"
When I tried 1st Cursor Category, I got error logcat:
Error Code : 1 (SQLITE_ERROR)
Caused By : SQL(query) error or missing database.
(no such column: game (code 1): , while compiling: SELECT CATEGORY FROM KeyWordDB WHERE CATEGORY
NOT IN (game, country, city, subway, actor, pet) ORDER BY RANDOM() LIMIT 1)
#################################################################
at android.database.sqlite.SQLiteConnection.nativePrepareStatement(Native Method)
at android.database.sqlite.SQLiteConnection.acquirePreparedStatement(SQLiteConnection.java:1008)
at android.database.sqlite.SQLiteConnection.prepare(SQLiteConnection.java:573)
at android.database.sqlite.SQLiteSession.prepare(SQLiteSession.java:588)
at android.database.sqlite.SQLiteProgram.<init>(SQLiteProgram.java:59)
at android.database.sqlite.SQLiteQuery.<init>(SQLiteQuery.java:37)
at android.database.sqlite.SQLiteDirectCursorDriver.query(SQLiteDirectCursorDriver.java:44)
at android.database.sqlite.SQLite
I confirmed that there is a database, so I guess it is probably sql query problem ...
So my question is How can I fix 1st or 2nd cursor to exclude categories that are in s1.ListViewCateogry?
I've searched real hard, but I wasn't able to find answer... Ill be real grateful, if someone answers this question

Change the double quotes with single quotes inside the loop that constructs the comma delimited list:
for (String param : s1.ListViewCategory) {
sb.append(",").append("'").append(param).append("'");
}
params = sb.toString().substring(1);
This code constructs a list like:
'game', 'country', 'city', 'subway', 'actor', 'pet'
If you use it as a parameter in the rawQuery() method then this list will be treated as a string literal and not a list of values.
So do this instead:
String sql = "SELECT CATEGORY FROM " + tableName
+ " WHERE CATEGORY NOT IN (?) "
+ " ORDER BY RANDOM() LIMIT 1 ";
sql = sql.replace("?", params);
Cursor cursor = database.rawQuery(sql, null);
Note that this method is prone to sql injection.
Another way is to create a list of ? placeholders instead of 1 placeholder and pass the list of values as an array of strings like this:
for (String param : s1.ListViewCategory) {
sb.append(",?");
}
String[] array = ListViewCategory.toArray(new String[s1.ListViewCategory.size()]);
params = sb.toString().substring(1);
String sql = "SELECT CATEGORY FROM " + tableName
+ " WHERE CATEGORY NOT IN (#) "
+ " ORDER BY RANDOM() LIMIT 1 ";
sql = sql.replace("#", params);
Cursor cursor = database.rawQuery(sql, array);

Loading x rows from database at a time, showing them, loading next x rows and so on

I have a database with multiple tables, each having millions of rows of data. In my Java appliaction, my task is to display certain data from certain tables depending on the arguments which I get from the user. I have successfully been able to load all of this data at once, but when there are 10 million rows, it takes a long time. I have found out about connection.setAutoCommit(false) and statement.setFetchSize(x), but these don't seem to work.
public static int getTableResults(Result_Controller okno){
int i = 0;
try {
Connection con = getConnection();
con.setAutoCommit(false);
String query = "SELECT h.\"Nazov hotela\", k.\"Nazov krajiny\", m.\"Nazov mesta\", h.\"Adresa hotela\", h.\"Hviezdicky\", p.\"Cena pobytu\", i.\"Typ izby\", h.\"Pocet izieb\" " +
"FROM hotel h " +
"inner JOIN izba i ON i.\"ID hotela\" = h.\"ID hotela\" " +
"inner JOIN krajina k ON k.\"ID krajiny\" = h.\"ID krajiny\" " +
"inner JOIN mesto m ON m.\"ID mesta\" = h.\"ID mesta\" " +
"inner JOIN pobyt p ON p.\"ID hotela\" = h.\"ID hotela\" " +
"WHERE h.\"Nazov hotela\" = ? " +
"AND k.\"Nazov krajiny\" = ? " +
"AND h.\"Hviezdicky\" = ? " +
"AND i.\"Pocet posteli\" >= ? " +
"AND p.\"Cena pobytu\" <= ? " +
"ORDER BY h.\"Nazov hotela\"";
PreparedStatement pst = con.prepareStatement(query, ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY);
pst.setFetchSize(150);
pst.setString(1, okno.getText_nazov().getText());
pst.setString(2, (String) okno.getKrajina().getValue());
int pocet_hviezdiciek = Integer.parseInt((String) okno.getHviezdicky().getValue());
pst.setInt(3, pocet_hviezdiciek);
int pocet_osob = Integer.parseInt((String) okno.getOsoby().getValue());
pst.setInt(4, pocet_osob);
double cena_pobytu = Double.parseDouble(okno.getText_cena().getText());
pst.setDouble(5, cena_pobytu);
ResultSet rs = pst.executeQuery();
System.out.println(pst);
while (rs.next()) {
okno.getOblist().add(new Vysledok_hladania(rs.getString("Nazov hotela"),
rs.getString("Nazov krajiny"),
rs.getString("Nazov mesta"),
rs.getString("Adresa hotela"),
rs.getInt("Hviezdicky"),
rs.getDouble("Cena pobytu"),
rs.getString("Typ izby"),
rs.getInt("Pocet izieb")));
System.out.println(i++);
}
okno.getTable().setItems(okno.getOblist());
} catch (SQLException ex) {
Logger.getLogger(Result_Controller.class.getName()).log(Level.SEVERE, null, ex);
}
return i;
}
All in all I want to get these results and display them in batches, not all at the same time.

You don't show the execution plan, but the problem must be the ORDER BY. PostgreSQL needs the complete result before it can begin to sort.
You may be able to get a “fast start-up” plan with a nested loop if you create the following index:
CREATE INDEX ON hotel ("Nazov hotela");
In addition, you need to have an index for the join conditions (the ID columns) on all other tables:
CREATE INDEX ON izba ("ID hotela");
CREATE INDEX ON krajina ("ID krajiny");
CREATE INDEX ON mesto ("ID mesta");
CREATE INDEX ON pobyt ("ID hotela");

Fetch all comma separated values as an unique element without affecting performance

Following is my data base table named feeds
column_name | data_type
-------------+-----------------------------
id | integer
category | text
question | text
answer | text
thumb | text
time | timestamp without time zone
The categories will be of comma separated values. I would like to get all comma separated values into an array that has got unique elements (No same categories will be present in the array).
To achieve this, I have created a separate table called category. When the user publishes a feed, i will populate both the feeds table and also category table at the same time as follows.
#POST
#Path("/post")
#Consumes(MediaType.APPLICATION_JSON)
public Response postStrMsg(String msg) {
String songString = "requesting";
JSONObject jsonObj = new JSONObject(msg);
try {
Connection connection = DbConnection.getConnection();
PreparedStatement stmt = connection.prepareStatement(
"INSERT INTO feeds ( CATEGORY , TOPIC , DESCRIPTION , THUMB , TIMEFRAME , AUTHOR ) VALUES( ?, ? , ? , ? , now() , ? )");
stmt.setString(1, jsonObj.getString("category"));
stmt.setString(2, jsonObj.getString("question"));
stmt.setString(3, jsonObj.getString("answer"));
stmt.setString(4, jsonObj.getString("thumb"));
stmt.setString(5, jsonObj.getString("author"));
stmt.executeUpdate();
String query = "INSERT INTO CATEGORY (SECTION) VALUES ('" + jsonObj.getString("category") + "');";
PreparedStatement stmts = connection.prepareStatement(query);
stmts.executeQuery();
songString = Utilities.constructJSON( "Posted Successfully",true);
} catch (Exception ex) {
songString = Utilities.constructJSON("Failure", false, ex.getMessage());
}
return Response.status(200).entity(songString).build();
}
And I used the following code to retrieve the categories when the user wants to know all categories. This will parse all comma separated values and filter all duplicate elements and it will provide the corresponding result.
Query :
"select min(c.id) as id, t.name from category c " + " cross join "
+ "unnest(string_to_array(c.category, ',')) AS t(name) " + "group by t.name order by 1";
Code
#GET
#Path("getCategories")
#Produces(MediaType.APPLICATION_JSON)
public String getCategories() {
ArrayList<Category> output = null;
try {
Connection connection = getConnection();
Statement stmt = connection.createStatement();
String query = "select min(c.id) as id, t.name from category c " + " cross join "
+ "unnest(string_to_array(c.category, ',')) AS t(name) " + "group by t.name order by 1";
ResultSet rs = stmt.executeQuery(query);
output = new ArrayList<Category>();
while (rs.next()) {
output.add(new Category(rs.getString("name")));
}
} catch (Exception ex) {
ex.printStackTrace();
}
JSONArray jsArray = new JSONArray(output);
return jsArray.toString();
}
The issue is that,
Since two different tables are populated at the same time, the performance is being degraded. Is it possible to achieve the same functionality without creating the same table?
No results were returned by the query while posting the feed

I think you need two more tables. One for categories and the other one to combine categories with feed. You should create a category table as follows:
column_name | data_type
-------------+-----------------------------
id | integer
category | text
And feedCategory table as follows:
column_name | data_type
-------------+-----------------------------
id | integer
category_id | integer
feed_id | integer
One feed can have more than one category. So you can populate the last table based on that.
It is not the best way to parse category text each time you will retrieve from database. How about querying feeds by category? Are you gonna parse category column for each feed and see if you have it in there?

Updating SQL Table with Java

I am developing an application that can update my database... However, I can't get my Java method working correctly. It gives me the following error: Must declare the scalar variable "#P0WHERE". Any suggestions?
The method I am using:
public void updateTable() throws SQLException
{
Scanner input = new Scanner(System.in);
System.out.println("Update the following:\n" + this);
this.getReservationInfo(input);
DataConnection connection = new DataConnection();
String query = "UPDATE " + TableName;
query += "Set Name = ?";
query += "WHERE Person_ID = " + id;
connection.updateData(query, person_name);
connection.closeConnection();
}

Add spaces before 'SET' and 'WHERE', otherwise it will not work.
String query = "UPDATE " + TableName;
query += " SET Name = ?";
query += " , Age = ?";
query += " , Col1 = ?"; //And other cols
query += " WHERE Person_ID = " + id;
EDIT: Changed query to update multiple columns.

I think so. You seem to be missing spaces. After the TableName and after the ?.
String query = "UPDATE " + TableName;
query += " Set Name = ?"; // tableSet not good, and
// ?WHERE is not valid add spaces.
query += " WHERE Person_ID = " + id;