We Keep Coding

MongoTemplate pull query

I am trying to remove one Item from an array of an embedded field in mongoDb. Array is type of string like the below one.
{
_id: 1,
fruits: [ "apples", "pears", "oranges", "grapes", "bananas" ],
vegetables: [ "carrots", "celery", "squash", "carrots" ]
}
{
_id: 2,
fruits: [ "plums", "kiwis", "oranges", "bananas", "apples" ],
vegetables: [ "broccoli", "zucchini", "carrots", "onions" ]
}
I just wanted to remove the carrots from the embedded array vegetables. using mongo shell, the below query works
db.stores.update(
{ },
{ $pull: { vegetables: "carrots" },
{ multi: true }
)
Now I needed to perform this using mongoTemplate in spring,
I tried with the below answer but it wont remove the element
MongoTemplate pull subdocument.
Can anyone suggest How I to achieve this using mongoTemplate in spring project

You can simply use the below query to pull any string in the array of vegetables object:
mongoTemplate.updateMulti(new Query(), new Update().pull("vegetables", "squash"), "stores");
In the above query, squash will be pulled after execution. For details, I have also added find query on stores collections after and before the above query as :
List<Stores> storesList = mongoTemplate.find(new Query(), Stores.class);
for(Stores stores : storesList) {
System.out.println(stores.toString());
}
mongoTemplate.updateMulti(new Query(), new Update().pull("vegetables", "squash"), "stores");
List<Stores> afterModificationStoresList = mongoTemplate.find(new Query(), Stores.class);
for(Stores stores : afterModificationStoresList) {
System.out.println(stores.toString());
}
The output is as below :
2019-11-26 10:19:57.947 DEBUG 7321 --- [ main] o.s.data.mongodb.core.MongoTemplate : find using query: { } fields: Document{{}} for class: class sample.data.mongo.models.Stores in collection: stores
Stores{id='1.0', fruits=[apples, pears, oranges, grapes, bananas], vegetables=[celery, squash]}
Stores{id='2.0', fruits=[plums, kiwis, oranges, bananas, apples], vegetables=[broccoli, zucchini, onions]}
2019-11-26 10:19:57.975 DEBUG 7321 --- [ main] o.s.data.mongodb.core.MongoTemplate : Calling update using query: { } and update: { "$pull" : { "vegetables" : "squash" } } in collection: stores
2019-11-26 10:19:57.985 DEBUG 7321 --- [ main] o.s.data.mongodb.core.MongoTemplate : find using query: { } fields: Document{{}} for class: class sample.data.mongo.models.Stores in collection: stores
Stores{id='1.0', fruits=[apples, pears, oranges, grapes, bananas], vegetables=[celery]}
Stores{id='2.0', fruits=[plums, kiwis, oranges, bananas, apples], vegetables=[broccoli, zucchini, onions]}
For code details, kinldy visit by Github repo: https://github.com/krishnaiitd/learningJava/blob/master/spring-boot-sample-data-mongodb/src/main/java/sample/data/mongo/main/Application.java#L126

Problem on multiple updates on same document in Multi-Doc Transactions of MongoDB

I'm facing lot of troubles in using the multi-document transactions. One of which happens to be - Whenever , there are multiple updates on the same document [in the same transaction], Mongo throws a weird error "com.mongodb.MongoCommandException: Command failed with error 251 (NoSuchTransaction): 'Transaction 1 has been aborted.' on server .....". I took it in the direction of visibility and adjusted the options for Read and write concerns like
WriteConcern.MAJORITY , ReadPreference.PRIMARY , ReadConcern.SNAPSHOT.
I'm using the java driver and the snippet for transaction start is something like below
TransactionOptions transactionOptions = TransactionOptions.builder()
.writeConcern(WriteConcern.MAJORITY)
.readPreference(ReadPreference.primary())
.readConcern(ReadConcern.SNAPSHOT)
.build();
ClientSessionOptions clientSessionOptions = ClientSessionOptions
.builder()
.defaultTransactionOptions(transactionOptions)
.build();
this.session = mongo.startSession(clientSessionOptions);
this.session.startTransaction(transactionOptions);
Any one has an idea on this error?

In order to get more context about the error, you need to check the logs of Mongodb.
In Mongo shell mongo, run show log global and you will see why the transaction has failed.
In my case, it was failing for this reason:
[startPeriodicThreadToAbortExpiredTransactions] Aborting transaction with
txnNumber 1 on session ea5e8c8d-ecc2-47ca-8901-55a43143a809 because it has
been running for longer than 'transactionLifetimeLimitSeconds'
This looks like a bug in MongoDB since it should throw atransactionLifetimeLimitSeconds error instead, or at least tell you what's the problem.

The transaction might include an operation that modifies the database catalog. I encountered the same error caused due to an insert operation on a non-existent collection. The 251 error happens when committing the transaction.
my-rs:PRIMARY> use mytest
switched to db mytest
my-rs:PRIMARY> show collections
my-rs:PRIMARY> session.startTransaction()
my-rs:PRIMARY> session.getDatabase('mytest').newcollection.insert({"a":"b"});
WriteCommandError({
"operationTime" : Timestamp(1554822412, 1),
"ok" : 0,
"errmsg" : "Cannot create namespace mytest.newcollection in multi-document transaction.",
"code" : 263,
"codeName" : "OperationNotSupportedInTransaction",
"$clusterTime" : {
"clusterTime" : Timestamp(1554822412, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
})
my-rs:PRIMARY> session.commitTransaction()
2019-04-09T15:07:04.695+0000 E QUERY [js] Error: command failed: {
"errorLabels" : [
"TransientTransactionError"
],
"operationTime" : Timestamp(1554822422, 1),
"ok" : 0,
"errmsg" : "Transaction 8 has been aborted.",
"code" : 251,
"codeName" : "NoSuchTransaction",
"$clusterTime" : {
"clusterTime" : Timestamp(1554822422, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
} :
_getErrorWithCode#src/mongo/shell/utils.js:25:13
doassert#src/mongo/shell/assert.js:18:14
_assertCommandWorked#src/mongo/shell/assert.js:536:17
assert.commandWorked#src/mongo/shell/assert.js:620:16
commitTransaction#src/mongo/shell/session.js:942:17
#(shell):1:1
my-rs:PRIMARY>
Please refer to https://docs.mongodb.com/manual/core/transactions/#restricted-operations

how to repeat this on Oracle? (Select for update + order by + LIMIT 1 + skip locked)

I have controler:
#GetMapping("/old")
public Product getOld() {
Product omeOld = productService.getOneOld();
log.info(String.valueOf(omeOld.getId()));
return omeOld;
}
Service:
#Override
#Transactional
public Product getOneOld() {
Product aNew = productsRepository.findTop1ByStatusOrderByCountAsc("NEW");
try {
Thread.sleep(5000L);
} catch (InterruptedException e) {
e.printStackTrace();
}
return aNew;
}
And repository:
#Repository
public interface ProductsRepository extends JpaRepository<Product, Long> {
Product findTop1ByStatusOrderByCountAsc(String status);
}
I start JMeter and send 5 request in 5 threads. In result I get 5 response after 5 seconds. Each request was processed by seconds. But in log I see next:
2018-09-14 14:04:35.524 INFO 9048 --- [nio-8080-exec-1] c.e.l.demo.controller.ProductController : 1
2018-09-14 14:04:35.525 INFO 9048 --- [nio-8080-exec-2] c.e.l.demo.controller.ProductController : 1
2018-09-14 14:04:35.532 INFO 9048 --- [nio-8080-exec-3] c.e.l.demo.controller.ProductController : 1
2018-09-14 14:04:35.534 INFO 9048 --- [nio-8080-exec-4] c.e.l.demo.controller.ProductController : 1
2018-09-14 14:04:35.534 INFO 9048 --- [nio-8080-exec-6] c.e.l.demo.controller.ProductController : 1
Each thread select the same row and process it. I need that first thread select first row, second thread select second row and etc. I try use #Lock(LockModeType.PESSIMISTIC_WRITE) :
#Lock(LockModeType.PESSIMISTIC_WRITE)
Product findTop1ByStatusOrderByCountAsc(String status);
Now when I start JMeter I have next behavior:
first thread worck 5 sec, after that second thread work 5 sec and etc. 25 secons all 5 threads. And in log:
2018-09-14 14:11:40.564 INFO 13724 --- [nio-8080-exec-5] c.e.l.demo.controller.ProductController : 1
2018-09-14 14:11:45.566 INFO 13724 --- [nio-8080-exec-4] c.e.l.demo.controller.ProductController : 1
2018-09-14 14:11:50.567 INFO 13724 --- [nio-8080-exec-2] c.e.l.demo.controller.ProductController : 1
2018-09-14 14:11:55.568 INFO 13724 --- [nio-8080-exec-1] c.e.l.demo.controller.ProductController : 1
2018-09-14 14:12:00.570 INFO 13724 --- [nio-8080-exec-3] c.e.l.demo.controller.ProductController : 1
All threads select the same row(if I change this roe in first thread - it will not select in second thread if the conditions do not match).
I try this:
#Query(value = "Select * from products where status = ?1 order by count asc LIMIT 1 for update", nativeQuery = true)
Product findTop1ByStatusOrderByCountAsc(String status);
the result is the same.
But I need - first thread select first row and block it/ Second thread select next not blocked row and process. I try next:
#Query(value = "Select * from products where status = ?1 order by count asc LIMIT 1 for update of products skip locked", nativeQuery = true)
Product findTop1ByStatusOrderByCountAsc(String status);
And it work fine! :
2018-09-14 14:25:00.355 INFO 7904 --- [io-8080-exec-10] c.e.l.demo.controller.ProductController : 4
2018-09-14 14:25:00.355 INFO 7904 --- [nio-8080-exec-4] c.e.l.demo.controller.ProductController : 3
2018-09-14 14:25:00.355 INFO 7904 --- [nio-8080-exec-9] c.e.l.demo.controller.ProductController : 1
2018-09-14 14:25:00.358 INFO 7904 --- [nio-8080-exec-5] c.e.l.demo.controller.ProductController : 5
2018-09-14 14:25:00.359 INFO 7904 --- [nio-8080-exec-2] c.e.l.demo.controller.ProductController : 6
Each select in each thread select one row from non blocked rows!
But how can I repeat this with Oracle? In oracle I can not write LIMIT 1 and if I use ROWNUM = 1 each thread select same row always.

spring-boot-starter-integration 1.4.3 performance degradation

I noticed a significant performance degradation when upgrading Spring-Boot from 1.4.2 to 1.4.3 while using spring-integration.
I could reproduce the problem with the following program:
pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<!--<version>1.4.2.RELEASE</version>-->
<version>1.4.3.RELEASE</version>
</parent>
<artifactId>performance-test</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-integration</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
</project>
Application.java:
package performance.test;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
#SpringBootApplication
public class Application {
public static void main(final String[] args) {
SpringApplication.run(Application.class, args);
}
}
UsersManager.java:
package performance.test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.integration.annotation.MessageEndpoint;
import org.springframework.integration.annotation.ServiceActivator;
import org.springframework.messaging.handler.annotation.Payload;
#MessageEndpoint
public class UsersManager {
private static final Logger LOGGER = LoggerFactory.getLogger(UsersManager.class);
#ServiceActivator(inputChannel = "testChannel")
public void process(#Payload String payload) {
LOGGER.debug("payload: {}", payload);
}
}
IntegrationTest.java:
package performance.test;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.SpringApplicationConfiguration;
import org.springframework.messaging.MessageChannel;
import org.springframework.messaging.support.MessageBuilder;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
#RunWith(SpringJUnit4ClassRunner.class)
#SpringApplicationConfiguration(classes = Application.class)
public class IntegrationTest {
private static final Logger LOGGER = LoggerFactory.getLogger(IntegrationTest.class);
#Autowired
private MessageChannel testChannel;
#Test
public void basic_test() {
for (int i = 1; i <= 10; i++) {
final long start = System.currentTimeMillis();
test(1000);
final long end = System.currentTimeMillis();
LOGGER.info("test run: {} took: {} msec", i, end - start);
}
}
private void test(int count) {
for (int i = 0; i < count; i++) {
testChannel.send(MessageBuilder.withPayload("foo").setHeader("monkey", "do").build());
}
}
}
When I use Spring-Boot 1.4.2, I get the following results:
2016-12-27 14:55:39.331 INFO 4939 --- [ main] performance.test.IntegrationTest : Started IntegrationTest in 0.975 seconds (JVM running for 1.52)
2016-12-27 14:55:39.468 INFO 4939 --- [ main] performance.test.IntegrationTest : test run: 1 took: 123 msec
2016-12-27 14:55:39.552 INFO 4939 --- [ main] performance.test.IntegrationTest : test run: 2 took: 83 msec
2016-12-27 14:55:39.632 INFO 4939 --- [ main] performance.test.IntegrationTest : test run: 3 took: 80 msec
2016-12-27 14:55:39.696 INFO 4939 --- [ main] performance.test.IntegrationTest : test run: 4 took: 63 msec
2016-12-27 14:55:39.763 INFO 4939 --- [ main] performance.test.IntegrationTest : test run: 5 took: 67 msec
2016-12-27 14:55:39.842 INFO 4939 --- [ main] performance.test.IntegrationTest : test run: 6 took: 78 msec
2016-12-27 14:55:39.895 INFO 4939 --- [ main] performance.test.IntegrationTest : test run: 7 took: 53 msec
2016-12-27 14:55:39.950 INFO 4939 --- [ main] performance.test.IntegrationTest : test run: 8 took: 55 msec
2016-12-27 14:55:40.004 INFO 4939 --- [ main] performance.test.IntegrationTest : test run: 9 took: 54 msec
2016-12-27 14:55:40.057 INFO 4939 --- [ main] performance.test.IntegrationTest : test run: 10 took: 53 msec
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.776 sec - in performance.test.IntegrationTest
However, when I use Spring-Boot 1.4.3, there is significant performance degradation:
2016-12-27 14:57:41.705 INFO 5122 --- [ main] performance.test.IntegrationTest : Started IntegrationTest in 1.002 seconds (JVM running for 1.539)
2016-12-27 14:57:41.876 INFO 5122 --- [ main] performance.test.IntegrationTest : test run: 1 took: 156 msec
2016-12-27 14:57:42.031 INFO 5122 --- [ main] performance.test.IntegrationTest : test run: 2 took: 153 msec
2016-12-27 14:57:42.251 INFO 5122 --- [ main] performance.test.IntegrationTest : test run: 3 took: 220 msec
2016-12-27 14:57:42.544 INFO 5122 --- [ main] performance.test.IntegrationTest : test run: 4 took: 293 msec
2016-12-27 14:57:42.798 INFO 5122 --- [ main] performance.test.IntegrationTest : test run: 5 took: 254 msec
2016-12-27 14:57:43.111 INFO 5122 --- [ main] performance.test.IntegrationTest : test run: 6 took: 312 msec
2016-12-27 14:57:43.544 INFO 5122 --- [ main] performance.test.IntegrationTest : test run: 7 took: 432 msec
2016-12-27 14:57:44.112 INFO 5122 --- [ main] performance.test.IntegrationTest : test run: 8 took: 567 msec
2016-12-27 14:57:44.807 INFO 5122 --- [ main] performance.test.IntegrationTest : test run: 9 took: 695 msec
2016-12-27 14:57:45.620 INFO 5122 --- [ main] performance.test.IntegrationTest : test run: 10 took: 813 msec
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.989 sec - in performance.test.IntegrationTest
I have no idea what could be the cause for this.
Are the other persons having the same issue?

This might be related to SPR-14929.
In this case, removing #Payload from the message parameter solves the issue (it is not needed in this case - you only need a payload annotation if there are multiple parameters).

why java concurrency test fails?

I have a simple class:
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class DummyService {
private final Logger logger = LoggerFactory.getLogger(getClass());
private boolean dataIndexing = false;
public boolean isDataIndexing() {
logger.info("isDataIndexing: {}", dataIndexing);
return dataIndexing;
}
public void cancelIndexing() {
logger.info("cancelIndexing: {}", dataIndexing);
dataIndexing = false;
}
public void createIndexCorp() {
logger.info("createIndexCorp: {}", dataIndexing);
createIndex();
}
public void createIndexEntr() {
logger.info("createIndexEntr: {}", dataIndexing);
createIndex();
}
private void createIndex() {
logger.info("createIndex: {}", dataIndexing);
if(dataIndexing)
throw new IllegalStateException("Service is busy!");
dataIndexing = true;
try {
while(dataIndexing) {
Thread.sleep(100);
logger.debug("I am busy...");
}
logger.info("Indexing canceled");
} catch (InterruptedException e) {
logger.error("Error during sleeping", e);
} finally {
dataIndexing = false;
}
}
}
and a unit test, with which i want to test object behavior:
public class CommonUnitTest
{
#Test
public void testCreateIndexWithoutAsync() throws InterruptedException {
final long sleepMillis = 500;
final DummyService indexService = new DummyService();
assertFalse(indexService.isDataIndexing());
new Thread(() -> {
indexService.createIndexCorp();
}
).start();
Thread.sleep(sleepMillis);
assertTrue(indexService.isDataIndexing());
// TaskExecutor should fails here
new Thread(() -> {
indexService.createIndexEntr();
logger.error("Exception expected but not occurred");
}
).start();
assertTrue(indexService.isDataIndexing());
indexService.cancelIndexing();
Thread.sleep(sleepMillis);
assertFalse(indexService.isDataIndexing());
}
}
The behaviour of object must be: If the method createIndexCorp or createIndexEntr is called by one thread, then another thread must get exception by trying to call one of this methods. But this does not happens! Here is the log:
2015-10-15 17:15:06.277 INFO --- [ main] c.c.o.test.DummyService : isDataIndexing: false
2015-10-15 17:15:06.318 INFO --- [ Thread-0] c.c.o.test.DummyService : createIndexCorp: false
2015-10-15 17:15:06.319 INFO --- [ Thread-0] c.c.o.test.DummyService : createIndex: false
2015-10-15 17:15:06.419 DEBUG --- [ Thread-0] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:06.524 DEBUG --- [ Thread-0] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:06.624 DEBUG --- [ Thread-0] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:06.724 DEBUG --- [ Thread-0] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:06.818 INFO --- [ main] c.c.o.test.DummyService : isDataIndexing: true
2015-10-15 17:15:06.820 INFO --- [ main] c.c.o.test.DummyService : isDataIndexing: true
2015-10-15 17:15:06.820 INFO --- [ Thread-1] c.c.o.test.DummyService : createIndexEntr: true
2015-10-15 17:15:06.820 INFO --- [ main] c.c.o.test.DummyService : cancelIndexing: true
2015-10-15 17:15:06.820 INFO --- [ Thread-1] c.c.o.test.DummyService : createIndex: true
2015-10-15 17:15:06.824 DEBUG --- [ Thread-0] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:06.921 DEBUG --- [ Thread-1] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:06.924 DEBUG --- [ Thread-0] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:07.021 DEBUG --- [ Thread-1] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:07.024 DEBUG --- [ Thread-0] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:07.121 DEBUG --- [ Thread-1] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:07.124 DEBUG --- [ Thread-0] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:07.221 DEBUG --- [ Thread-1] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:07.224 DEBUG --- [ Thread-0] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:07.321 DEBUG --- [ Thread-1] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:07.321 INFO --- [ main] c.c.o.test.DummyService : isDataIndexing: true
You can see that second thread can start process, but it should get the exception. Also the last assertion in the test code fails. How can that happen ? I dont understand this behavior. I tried to use volatile and synchronized keyword, but nothing helps. What is wrong with DummyService ?

You have 3 threads, t0, t1 and tm (main).
The order of operations is like this:
tm starts t0
t0 checks dataIndexing flag - false, goes into the loop, sets flag to true
tm sleeps
tm starts t1
tm sets indexing flag to false
t1 checks dataIndexing flag - false, goes into the loop, sets flag to true
t0 continues the loop because it missed that brief period when indexing was cancelled
If you sleep in the main tm before setting indexing flag to false, then t1 will get the exception.
You need to synchronize access to variables shared between multiple threads. I.e. checking the state of the flag and changing it needs to be done while holding a mutex.

It seems you're hitting the difference between logging and the actual execution. The threads can conceivably run cancel and create index in the space between logging and and the exception, thus the second thread slipping by and preventing cancelling of the first and the second.
It is not advisable to allow simultaneous changes to shared resource, namely private boolean dataIndexing. There are two solutions (at least):
1.A syncronized method to allow for change of the shared resource (thus limiting access to only one thread at a time)
private synchronized void setDataIndexing(boolean value) {
dataIndexing = value;
}
2.Guarding each change of this value in a syncronized section (in both the = true and the = false places):
syncronized (this) {
dataIndexing = /* the relevant value */;
}
I would advise a separate method, but good to know the alternatives.

Not an answer to your question, but this is completely unsynchronized:
if (dataIndexing)
throw new IllegalStateException("Service is busy!");
dataIndexing = true;
Is the service busy if your execution reaches the throw statement? Not necessarily! Another thread could have changed the value of dataIndexing from true to false in between the test and the throw.
What's worse, maybe much worse, is that two threads might both reach the statement after the throw at the same time:
Thread A Thread B
tests dataIndexing, finds it to
be false.
Tests dataIndexing, finds it to be false.
sets dataIndexing = true; sets dataIndexing = true;
... ...
Also, this is unreliable, and it takes time.
Thread.sleep(sleepMillis);
assertTrue(indexService.isDataIndexing());
Better to design your classes for testability. If your test needs to wait until isDataIndexing(), then your class should provide a means for the test to wait()...
Also, don't underestimate the importance of making tests that complete in the least amount of time possible. When you have a system that has thousands or tens of thousands of test cases, the seconds really start to add up.