I have inherited a system with a rather wierd bug that occurs maybe once every 6 months, where the application suddenly loses track of Database data.
The system has redundancy with 2 servers that are scheduled to run the same function at the same time. They both get the same input data to the function and they both talk to the same postgres database, however the behaviours is different on the 2 machines.
The function that is being executed is calling the database and checking if there is a row with the specified id as supplied by the input parameter and if there is it executes A(), otherwise B()
Problem is that one server executes A() and the other B(). I have searched everywhere and there is no code writing to this table or deleting from it. So within all reason I think that they should both execute the same code.
This is the code that is fetching from the database:
#PersistenceContext(unitName = "backend-persistence")
private EntityManager em;
public Optional<OfferEntity> getOfferFromOfferId(final long offerId, final String countryAlias, final String langauageAlias) {
CriteriaBuilder cb = em.getCriteriaBuilder();
CriteriaQuery<OfferEntity> cq = cb.createQuery(OfferEntity.class);
Root<OfferEntity> from = cq.from(OfferEntity.class);
cq.select(from);
cq.where(cb.and(cb.equal(from.get(OfferEntity_.offerId), offerId),
cb.equal(from.get(OfferEntity_.country), countryAlias),
cb.equal(from.get(OfferEntity_.language), langauageAlias)));
try {
return Optional.of(em.createQuery(cq).getSingleResult());
} catch (NoResultException nre) {
return Optional.empty();
}
}
And I am getting an empty optional from one of the servers but not the other.
So I guess as a tl;dr, am I missunderstanding NoResultException and in what concrete situations can this be thrown? besides if there are no rows matching the query.
You can only use getSingleResult() when you are sure that you will get exactly one result. In all other cases you have to use getResultList()
From the API doc of javax.persistence.Query getSingleResult():
java.lang.Object getSingleResult()
Execute a SELECT query that returns a single untyped result.
Returns:
the result
Throws:
NoResultException - if there is no result
NonUniqueResultException - if more than one result
IllegalStateException - if called for a Java Persistence query language UPDATE or DELETE statement
QueryTimeoutException - if the query execution exceeds the query timeout value set and only the statement is rolled back
TransactionRequiredException - if a lock mode has been set and there is no transaction
PessimisticLockException - if pessimistic locking fails and the transaction is rolled back
LockTimeoutException - if pessimistic locking fails and only the statement is rolled back
PersistenceException - if the query execution exceeds the query timeout value set and the transaction is rolled back
Related
I'm currently working in a legacy application that uses an Stored Procedure in a Sybase DB to generate some IDs incrementally. The procedure is defined as:
CREATE PROC getId
(#val int = -1 output)
AS
BEGIN
UPDATE ID_TABLE SET LAST_VALUE = LAST_VALUE + 1
SELECT #val = LAST_VALUE FROM ID_TABLE
RETURN #val
END
This procedure is being called from a Java application using Spring's TransactionTemplate to handle the transaction in a declarative manner.
public Integer getId() {
TransactionTemplate txTemplate = new TransactionTemplate(txManager); // txManager is an autowired instance of PlatformTransactionManager.
txTemplate.setIsolationLevel(TransactionDefinition.ISOLATION_SERIALIZABLE);
txTemplate.setTimeout(-1);
return (Integer) txTemplate.execute((TransactionCallback) status -> idDao.generateId());
}
Internally, idDao uses a JdbcTemplate to call the Stored Procedure using a CallableStatementCreatorFactory. Nothing too out of the ordinary there.
The Stored Procedure is called ~10k times/day. From time to time, we see some ID collisions. My understanding was that the isolation level being set to SERIALIZABLE should prevent this from happening, and I can't seem to reproduce this even calling getId simultaneously with several threads. Does anybody have a hint on what might be happening here?
I can't speak to the java/spring coding but on a purely SQL basis it looks and sounds like the update/select is not being performed within a transaction.
Without a transaction wrapper (eg, explicit begin/commit tran, autocommit=false, set chained on) the update/select is a prime target for a race condition and duplicate key errors.
NOTE: the isolation level has no effect if the update/select is performed outside of a transaction wrapper
Tracking down a 'missing transaction wrapper' issue is a good idea as there could be other pieces of SQL code suffering from poor/missing transaction management. Having said that ...
For this particular case one easy fix would be to add a transaction wrapper in the proc, eg:
CREATE PROC getId
(#val int = -1 output)
AS
BEGIN
begin tran
UPDATE ID_TABLE SET LAST_VALUE = LAST_VALUE + 1
SELECT #val = LAST_VALUE FROM ID_TABLE
commit tran
RETURN #val
END
Even better would be a rewrite of the update that eliminates the need for the select, eg:
CREATE PROC getId
(#val int = -1 output)
AS
BEGIN
UPDATE ID_TABLE SET #val=LAST_VALUE+1, LAST_VALUE = LAST_VALUE + 1
RETURN #val
END
NOTE: as a standalone operation this eliminates the need for a transaction wrapper and should eliminate the generation of duplicate keys (assuming this proc is the root cause of the duplicate key issue)
Assuming this is Sybase ASE I'd want to look at making sure the table is configured to use datarows locking; with allpages locking, and to a limited extent datapages locking, there's a chance of a race condition on index updates (which in turn could lead to a deadlock scenario).
I have this method that is called from another service:
#Transactional(propagation = Propagation.REQUIRES_NEW, rollbackFor = Exception.class)
public void execute(String sql) {
Query query = entityManager.createNativeQuery(sql);
query.executeUpdate();
}
Basically the client loads multiple sql files and run each sql file in a new transaction, in order to not impact other files execution.
For example this is an example of an sql file, that is cleaning up some data:
begin;
delete from table t where t.created_at < current_date - interval '2 month';
commit;
What I'm trying to do is to log, the outcome of each transaction. For example here, I want to display how many records were deleted. How can I do that from Spring ? I know that you can log something more specific with:
logging.level.org.springframework.transaction=TRACE
, but still I cannot see any outcome. This reveals information about sql that will run and when transaction started/ended.
Second solution was to check the result of:
int count = query.executeUpdate();
, but count is 0, even though the sql code got executed and deletes hundreds of rows.
Thanks upfront for the suggestions !
The problem is as #XtremeBaumer correctly pointed out your script. If you just run executeUpdate with a delete statement it will return the number of affected rows.
But that is not what you are doing. You are executing a code block delimited by begin and end. There might be a way for such a code block to return a value, but that would need to be coded into the code block and is probably highly database specific.
I am trying to insert into a db that I have, and I'd like to do so through parameters. I am connecting to a postgres db using java.
I can connect to the db just fine. I know that because I have various operations that I am using that are already working were I can see, and update existing rows in my db. I am having trouble with INSERT.
I have the following:
private String _update_rentals = "INSERT into rentals (cid, mid) values (?,?)";
private PreparedStatement _update_rentals_statement;
private String _update_movie_status = "UPDATE Movie SET checkedout = true WHERE mid = ?";
private PreparedStatement _update_movie_status_statement;
And I initialize them:
_update_movie_status_statement = _customer_db.prepareStatement(_update_movie_status);
_update_rentals_statement = _customer_db.prepareStatement(_update_rentals);
And
while (movieAvail.next()){
System.out.println(movieAvail.getBoolean(1));
if (movieAvail.getBoolean(1) == false){
//Do chekcout
_update_rentals_statement.clearParameters();
_update_rentals_statement.setInt(1, cid);
_update_rentals_statement.setInt(2, mid);
_update_rentals_statement.executeQuery();
_update_movie_status_statement.clearParameters();
_update_movie_status_statement.setInt(1, mid);
_update_movie_status_statement.executeQuery();
System.out.println("Enjoy your movie!");
}
}
I am getting an error with both of the executeQuery() calls. For some reason I am getting the following error with both:
Exception in thread "main" org.postgresql.util.PSQLException: No results were returned by the query.
I looked at other posts, and I believed that I was following syntax for both insert/ update correctly, so maybe I am overlooking some aspect of this.
This is all part of a larger code base, so I did not want to include the methods these pieces of code are in. But these are the isolated instances which play a part with this code.
In general, when you execute a query, you are willing to retrieve some kind of information from the database. This is usually the case when you are executing SELECT queries. However, with INSERT and UPDATE statements, you are not querying the database, you are simply executing an update or inserting new rows. In the documentation of PreparedStatement you can see in which cases an exception is being thrown when you try to call executeQuery:
Throws: SQLException - if a database access error occurs; this method
is called on a closed PreparedStatement or the SQL statement does not
return a ResultSet object
So in your case the problem is that your statements do not return a ResultSet. You should use execute or executeUpdate instead. The former simply executes the update, while the latter does the same, but also returns the number of affected rows.
I think the main issue is that you are calling executeQuery(), which expects a result to be returned, but Insert/Update are not queries and don't return a result. Try just calling execute().
We use java datastax cassandra driver 2.1.2. Cassandra version we use is 2.0.9.
We have statement which we build with QueryBuilder and we are setting consistency level to statement on TWO explicitly.
Select selectStatement = QueryBuilder.select().from(ARTICLES);
selectStatement.where(eq(ORGANIZATION_ID, organizationId));
selectStatement.setConsistencyLevel(ConsistencyLevel.TWO);
final ResultSet rs = session.execute(selectStatement);
//call to all() will be removed since it is enough to iterate over result set
//and then you get pagination for free instead of loading everything in
//memory
List<Row> rows = rs.all();
for (final Row row : rows) {
//do something with Row, convert to POJO
}
We get exception like this:
com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ALL (3 responses were required but only 2 replica responded)
com.datastax.driver.core.exceptions.ReadTimeoutException.copy (ReadTimeoutException.java:69)
com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException (DefaultResultSetFuture.java:259)
com.datastax.driver.core.ArrayBackedResultSet$MultiPage.prepareNextRow (ArrayBackedResultSet.java:279)
com.datastax.driver.core.ArrayBackedResultSet$MultiPage.isExhausted (ArrayBackedResultSet.java:239)
com.datastax.driver.core.ArrayBackedResultSet$1.hasNext (ArrayBackedResultSet.java:122)
com.datastax.driver.core.ArrayBackedResultSet.all (ArrayBackedResultSet.java:111)
I know that calling all() on ResultSet makes it load all articles for organization in memory and work with it and creates load on cassandra. This will be removed as noted in comments. This can cause read timeout but I am still puzzled why in exception message there is ALL.
Question is why exception is telling that consistency level ALL is used when we set it to TWO for original statement. Is all() internally doing something with query and using CL ALL by default?
Your problem is almost certainly https://issues.apache.org/jira/browse/CASSANDRA-7947 . You are seeing an error message from failing to perform read repair. It is unrelated to your original consistency level. This is fixed in 2.1.3+.
How is it possible?
We are executing EJBQL on Toplink(DB is Oracle) and query.getResultList is empty.
But!
When i switched log level to FINE and received Sql query, that TopLink generates, i tried to execute this query on database and (miracle!) i got a non-empty result!
What could be the reason and how is it treated?
Thanks in advance!
P.S. No exceptions.
UPDATE:
Query log:
SELECT DISTINCT t0.ID, t0.REG_NUM, t0.REG_DATE, t0.OBJ_NAME, t1.CAD_NUM, t1.CAD_NUM_EGRO, t2.ID, t2.DICT_TYPE, t2.ARCHIVE_DATE, t2.IS_DEFAULT, t2.IS_ACTUAL, t2.NAME, t0.INVENTORY_NUM FROM CODE_NAME_TREE_DICTIONARY t3, DEFAULTABLE_DICTIONARY t2, IMMOVABLE_PROP t1, ABSTRACT_PROPERTY t0 WHERE ((t3.ID IN (SELECT DISTINCT t4.ID FROM CODE_NAME_TREE_DICTIONARY t5, CODE_NAME_TREE_DICTIONARY t4, type_property_parents t6 WHERE (((t5.ID = ?) AND (t4.DICT_TYPE = ?)) AND ((t6.type_property_id = t4.ID) AND (t5.ID = t6.parent_id)))) AND ((t1.ID = t0.ID) AND (t0.PROP_TYPE_DISCR = ?))) AND ((t3.ID = t0.PROP_TYPE) AND ((t2.ID (+) = t1.STATUS_ID) AND (t2.DICT_TYPE = ?)))) ORDER BY t0.REG_NUM ASC
bind => [4537, R, R, realty_status]|#]
This query returns 100k rows, but toplink believes that it is not...
With log level to FINE can you verify that you are connecting to the same database? How simple is your testcase; can you verify that it is this exact JPQL that is being translated to that SQL?
VPD (http://download.oracle.com/docs/cd/B28359_01/network.111/b28531/vpd.htm)? Policies?
Is something of this flavor defined on the schema? These features transparently add dynamic where clauses to the statement that is executed in the database session, so the query results depend on the state of the session in this case.
When reformatting the query the following conditions seemed strange:
AND t2.ID (+) = t1.STATUS_ID
AND t2.DICT_TYPE = ?
The (+) indicates an outer join of t2 (DEFAULTABLE_DICTIONARY), but this table seems to be non-optional since it has to have a non-null DICT_TYPE for the second condition.
On closer looking, the bind parameters also seem to be off, the fields are in order
CODE_NAME_TREE_DICTIONARY.ID
CODE_NAME_TREE_DICTIONARY.DICT_TYPE
ABSTRACT_PROPERTY.PROP_TYPE_DISCR
DEFAULTABLE_DICTIONARY.DICT_TYPE
With the given parameters (4537, R, R, realty_status), the first DICT_TYPE would be 'R' while the second is the string "realty_status" which seems inconsistent.
Transactions? Oracle never gives you a "dirty read" which database speak for access to uncommitted data. If you send data on one connection you cannot access it on any other connection until it is committed. If you try the query later by hand, the data has been committed and you get the expected result.
This situation can arise if you are updating the data in more than one connection, and the data manipulation is not set to "auto commit". JPA defaults to auto-commit, but flushing at transaction boundaries can give you a cleaner design.
I can't tell exactly, but I am a little surprised that the string parameters are not quoted. Is it possible that interactively there are some automatic conversions, but over this connection instead of the string 'R' it was converted to the INT ascii for R?
I found the reason!
The reason is Oracle! I've tried the same code on Postgres and its worked!
I dont know why, but in some magic cases oracle ignores query parameters and query returns empty result.