We Keep Coding

What does "synchronization actions are totally ordered" mean?

I am reading Java Concurrency in Practice, in "16.1.3 The Java Memory Model in 500 words or less", it says:
The Java Memory Model is specified in terms of actions, which include reads and writes to variables, locks and unlocks of monitors, and starting and joining with threads. The JMM defines a partial ordering called happens-before on all actions within the program. To guarantee that the thread executing action B can see the results of action A (whether or not A and B occur in different threads), there must be a happens-before relationship between A and B. In the absence of a happens-before ordering between two operations, the JVM is free to reorder them as it pleases.
Even though actions are only partially ordered, synchronization actions—lock acquisition and release, and reads and writes of volatile variables—are totally ordered. This makes it sensible to describe happens-before in terms of “subsequent” lock acquisitions and reads of volatile variables.
About "partial ordering", I have found this and this, but I don't quite understand "Even though actions are only partially ordered, synchronization actions—lock acquisition and release, and reads and writes of volatile variables—are totally ordered.". What does "synchronization actions are totally ordered" mean?

Analyzing the statement "synchronization actions are totally ordered":
"synchronization actions" is a set S of program operations (actions)
we have a relation R over set S : it is the happens-before relation. That is, given program statements a and b, aRb if and only if a happens-before b.
Then what the statement says, is "relation R is total over S".
"relation R is total over S", means that for every two operations a,b from set S (with a!=b), either aRb, or bRa. That is, either a happens-before b, or b happens-before a.
If we define the set S as the set of all lock acquisitions and lock releases performed on the same lock object X; then the set S is totally ordered by the happens-before relation: let be a the acquisition of lock X performed by thread T1, and b the lock acquisition performed by thread T2. Then either a happens-before b (in case T1 acquires the lock first. T1 will need to release the lock first, then T2 will be able to acquire it); or b happens-before a (in case T2 acquires the lock first).
Note: not all relations are total.
In example, the relation <= is total over the real numbers. That is, for every pair a,b of real numbers, it is true that either a<=b or b<=a. The total order here means that given any two items, we can always decide which comes first wrt. the given relation.
But the relation P: "is an ancestor of", is not a total relation over the set of all humans. Of course, for some pairs of humans a,b it is true that either aPb (a is an ancestor of b), or bPa (b is an ancestor of a). But for most of them, neither aPb nor bPa is true; that is, we can't use the relation to decide which item comes "first" (in genealogical terms).
Back to program statements, the happens-before relation R is obviously partial, over the set of all program statements (like in the "ancestor-of" example): given un-synchronized actions a,b (any operations performed by different threads, in absence of proper synchronization), neither aRb nor bRa holds.

Atomicity of Reads and Writes for Variables in Java

this is a followup on another question of mine.
#templatetypedef answered the question (appreciated), and in his answer he wrote:
As a note - atomicity does not mean "all other threads will be blocked
until the value is ready. It means all other threads will either see
the state purely before the operation is done or purely after the
operation is done, but nothing else.
I have a confusion regarding this, and here’s why:
It says here:
Atomic actions cannot be interleaved, so they can be used without fear
of thread interference.
What I infer from this, is that it contradicts what he wrote.
If we have 2 int variables i1 and i2, and we do the atomic operation i1=i2; and this operation is executed by threadX.
Then if atomic actions cannot be interleaved as indicated above, it means that during this atomic operation (executed by threadX), no other threadY is allowed to access (for either read or write) that same variable i2, hence, no other threadY, is allowed to access that same variable during the atomic operation, so some form of blocking does exist.
Did I get this right?
Thanks...

To the best of my knowledge there is no atomic i1 = i2 operation. You can atomically read an int and you can atomically write to one, but you can't do both in the same operation with synchronization. So i1 = i2 is two different atomic operations, a read followed by a write. You are guaranteed that nothing will interleave the read operation so you won't see partial updates to i2 when you read it, and you are guaranteed that nothing will interleave the write to i1, but there's no guarantee that nothing will happen in between those two atomic operations.
Lets say Thread t1 is going to do:
i2 = 10
i1 = i2
And thread t2 is going to do:
i1 = 7
i2 = 18
System.out.println(i1)
What you are guaranteed is that t1 will end up assigning either 10 or 18 to i1 but you can't know which. However, you are guaranteed it can't be any other value because the read of i2 and the write to i1 are atomic so you can't end up seeing some of the bits of i2 while it's being modified. Similarly, t2 is guaranteed to print either 10, 18, or 7 and it can't print anything else. However, without synchronization there's no way to know which of those 3 values it will end up printing.

... it means that during this atomic operation (executed by threadX), no other threadY is allowed to access (for either read or write) that same variable i2, hence, no other threadY, is allowed to access that same variable during the atomic operation, so some form of blocking does exist.
No you didn't get it right.
Atomic operations mean that threads can not see values in a partial state. The assignment is atomic depending on the underlying architecture running your JVM and the data size of i1 and i2. I believe that Java says that int fields are atomically assigned but long (and double) may not be because it may take multiple operations by the CPU.
Atomic actions cannot be interleaved, so they can be used without fear of thread interference.
This is right. If i1 is 1 and i2 is 2 and threadX executes the assignment, then any other thread will either see the value of i1 as 1 (the old value) or 2 (the new value). ThreadY won't see it be some sort of half-way between 1 or 2 because that assignment is atomic even if multiple threads are updating the value of i1.
But what is really confusing the matter is that there are two concepts going on here: atomicity and memory synchronization. With threads, each CPU has its own memory cache so that memory operations are first made to the local memory and then these changes are written to main memory. A thread might see an old copy of i1 in its local cached memory even though another thread has updated main memory already. Even worse is when two threads have updated the value of i1 in their local memory and depending on their order of operations (which is highly random) one thread's value will overwrite the other thread's write to main memory. It's extremely hard to know which one will win the race condition.
As a note - atomicity does not mean "all other threads will be blocked until the value is ready.
Right. This is trying to let you know that there is no locking involved at all here. There are no guarantees as to the value that ThreadY will see. ThreadY could also be updating i1 at the same exact time to the value 3 and then other threads could see it as 1, 2, or 3 depending on the order of operations and whether or not those threads cross memory barriers when the cache flushing and updating is enforced.
The way we control fields and objects that are shared between threads is with the synchronized keyword, which gives a thread unique access to a resource. There are also Locks and other mechanisms to provide mutex. We also can force memory barriers by adding a volatile keyword to a field which means that any read or write to the field will be made to main memory. Both synchronized and volatile ensure proper publishing of data and ordering of operations.

Volatile Vs Atomic [duplicate]

This question already has answers here:
What is the difference between atomic / volatile / synchronized?
(7 answers)
Closed 9 years ago.
I read somewhere below line.
Java volatile keyword doesn't means atomic, its common misconception
that after declaring volatile, ++ operation will be atomic, to make
the operation atomic you still need to ensure exclusive access using
synchronized method or block in Java.
So what will happen if two threads attack a volatile primitive variable at same time?
Does this mean that whosoever takes lock on it, that will be setting its value first. And if in meantime, some other thread comes up and read old value while first thread was changing its value, then doesn't new thread will read its old value?
What is the difference between Atomic and volatile keyword?

The effect of the volatile keyword is approximately that each individual read or write operation on that variable is made atomically visible to all threads.
Notably, however, an operation that requires more than one read/write -- such as i++, which is equivalent to i = i + 1, which does one read and one write -- is not atomic, since another thread may write to i between the read and the write.
The Atomic classes, like AtomicInteger and AtomicReference, provide a wider variety of operations atomically, specifically including increment for AtomicInteger.

Volatile and Atomic are two different concepts. Volatile ensures, that a certain, expected (memory) state is true across different threads, while Atomics ensure that operation on variables are performed atomically.
Take the following example of two threads in Java:
Thread A:
value = 1;
done = true;
Thread B:
if (done)
System.out.println(value);
Starting with value = 0 and done = false the rule of threading tells us, that it is undefined whether or not Thread B will print value. Furthermore value is undefined at that point as well! To explain this you need to know a bit about Java memory management (which can be complex), in short: Threads may create local copies of variables, and the JVM can reorder code to optimize it, therefore there is no guarantee that the above code is run in exactly that order. Setting done to true and then setting value to 1 could be a possible outcome of the JIT optimizations.
volatile only ensures, that at the moment of access of such a variable, the new value will be immediately visible to all other threads and the order of execution ensures, that the code is at the state you would expect it to be. So in case of the code above, defining done as volatile will ensure that whenever Thread B checks the variable, it is either false, or true, and if it is true, then value has been set to 1 as well.
As a side-effect of volatile, the value of such a variable is set thread-wide atomically (at a very minor cost of execution speed). This is however only important on 32-bit systems that i.E. use long (64-bit) variables (or similar), in most other cases setting/reading a variable is atomic anyways. But there is an important difference between an atomic access and an atomic operation. Volatile only ensures that the access is atomically, while Atomics ensure that the operation is atomically.
Take the following example:
i = i + 1;
No matter how you define i, a different Thread reading the value just when the above line is executed might get i, or i + 1, because the operation is not atomically. If the other thread sets i to a different value, in worst case i could be set back to whatever it was before by thread A, because it was just in the middle of calculating i + 1 based on the old value, and then set i again to that old value + 1. Explanation:
Assume i = 0
Thread A reads i, calculates i+1, which is 1
Thread B sets i to 1000 and returns
Thread A now sets i to the result of the operation, which is i = 1
Atomics like AtomicInteger ensure, that such operations happen atomically. So the above issue cannot happen, i would either be 1000 or 1001 once both threads are finished.

There are two important concepts in multithreading environment:
atomicity
visibility
The volatile keyword eradicates visibility problems, but it does not deal with atomicity. volatile will prevent the compiler from reordering instructions which involve a write and a subsequent read of a volatile variable; e.g. k++.
Here, k++ is not a single machine instruction, but three:
copy the value to a register;
increment the value;
place it back.
So, even if you declare a variable as volatile, this will not make this operation atomic; this means another thread can see a intermediate result which is a stale or unwanted value for the other thread.
On the other hand, AtomicInteger, AtomicReference are based on the Compare and swap instruction. CAS has three operands: a memory location V on which to operate, the expected old value A, and the new value B. CAS atomically updates V to the new value B, but only if the value in V matches the expected old value A; otherwise, it does nothing. In either case, it returns the value currently in V. The compareAndSet() methods of AtomicInteger and AtomicReference take advantage of this functionality, if it is supported by the underlying processor; if it is not, then the JVM implements it via spin lock.

As Trying as indicated, volatile deals only with visibility.
Consider this snippet in a concurrent environment:
boolean isStopped = false;
:
:
while (!isStopped) {
// do some kind of work
}
The idea here is that some thread could change the value of isStopped from false to true in order to indicate to the subsequent loop that it is time to stop looping.
Intuitively, there is no problem. Logically if another thread makes isStopped equal to true, then the loop must terminate. The reality is that the loop will likely never terminate even if another thread makes isStopped equal to true.
The reason for this is not intuitive, but consider that modern processors have multiple cores and that each core has multiple registers and multiple levels of cache memory that are not accessible to other processors. In other words, values that are cached in one processor's local memory are not visisble to threads executing on a different processor. Herein lies one of the central problems with concurrency: visibility.
The Java Memory Model makes no guarantees whatsoever about when changes that are made to a variable in one thread may become visible to other threads. In order to guarantee that updates are visisble as soon as they are made, you must synchronize.
The volatile keyword is a weak form of synchronization. While it does nothing for mutual exclusion or atomicity, it does provide a guarantee that changes made to a variable in one thread will become visible to other threads as soon as it is made. Because individual reads and writes to variables that are not 8-bytes are atomic in Java, declaring variables volatile provides an easy mechanism for providing visibility in situations where there are no other atomicity or mutual exclusion requirements.

The volatile keyword is used:
to make non atomic 64-bit operations atomic: long and double. (all other, primitive accesses are already guaranteed to be atomic!)
to make variable updates guaranteed to be seen by other threads + visibility effects: after writing to a volatile variable, all the variables that where visible before writing that variable become visible to another thread after reading the same volatile variable (happen-before ordering).
The java.util.concurrent.atomic.* classes are, according to the java docs:
A small toolkit of classes that support lock-free thread-safe
programming on single variables. In essence, the classes in this
package extend the notion of volatile values, fields, and array
elements to those that also provide an atomic conditional update
operation of the form:
boolean compareAndSet(expectedValue, updateValue);
The atomic classes are built around the atomic compareAndSet(...) function that maps to an atomic CPU instruction. The atomic classes introduce the happen-before ordering as the volatile variables do. (with one exception: weakCompareAndSet(...)).
From the java docs:
When a thread sees an update to an atomic variable caused by a
weakCompareAndSet, it does not necessarily see updates to any other
variables that occurred before the weakCompareAndSet.
To your question:
Does this mean that whosoever takes lock on it, that will be setting
its value first. And in if meantime, some other thread comes up and
read old value while first thread was changing its value, then doesn't
new thread will read its old value?
You don't lock anything, what you are describing is a typical race condition that will happen eventually if threads access shared data without proper synchronization. As already mentioned declaring a variable volatile in this case will only ensure that other threads will see the change of the variable (the value will not be cached in a register of some cache that is only seen by one thread).
What is the difference between AtomicInteger and volatile int?
AtomicInteger provides atomic operations on an int with proper synchronization (eg. incrementAndGet(), getAndAdd(...), ...), volatile int will just ensure the visibility of the int to other threads.

So what will happen if two threads attack a volatile primitive variable at same time?
Usually each one can increment the value. However sometime, both will update the value at the same time and instead of incrementing by 2 total, both thread increment by 1 and only 1 is added.
Does this mean that whosoever takes lock on it, that will be setting its value first.
There is no lock. That is what synchronized is for.
And in if meantime, some other thread comes up and read old value while first thread was changing its value, then doesn't new thread will read its old value?
Yes,
What is the difference between Atomic and volatile keyword?
AtomicXxxx wraps a volatile so they are basically same, the difference is that it provides higher level operations such as CompareAndSwap which is used to implement increment.
AtomicXxxx also supports lazySet. This is like a volatile set, but doesn't stall the pipeline waiting for the write to complete. It can mean that if you read a value you just write you might see the old value, but you shouldn't be doing that anyway. The difference is that setting a volatile takes about 5 ns, bit lazySet takes about 0.5 ns.

Instruction reordering & happens-before relationship [duplicate]

This question already has answers here:
How to understand happens-before consistent
(5 answers)
Closed 4 years ago.
In the book Java Concurrency In Practice, we are told several time that the instructions of our program can be reordered, either by the compiler, by the JVM at runtime, or even by the processor. So we should assume that the executed program will not have its instructions executed in exactly the same order than what we specified in the source code.
However, the last chapter discussing Java Memory Model provides a listing of happens-before rules indicating which instruction ordering are preserved by the JVM. The first of these rules is:
"Program order rule. Each action in a thread happens before every action in that thread that comes later in the program order."
I believe "program order" refers to the source code.
My question: assuming this rule, I wonder what instruction may be actually reordered.
"Action" is defined as follow:
The Java Memory Model is specified in terms of actions, which include reads and writes to variables, locks and unlocks of monitors, and starting and joining with threads. The JMM defines a partial ordering called happens before on all actions within the program. To guarantee that the thread executing action B can see the results of action A (whether or not A and B occur in different threads), there must be a happens before relationship between A and B. In the absence of a happens before ordering between two operations, the JVM is free to reorder them as it pleases.
Other order rules mentionned are:
Monitor lock rule. An unlock on a monitor lock happens before every subsequent lock on that same monitor lock.
Volatile variable rule. A write to a volatile field happens before every subsequent read of that same field.
Thread start rule. A call to Thread.start on a thread happens before every action in the started thread.
Thread termination rule. Any action in a thread happens before any other thread detects that thread has terminated, either by successfully return from Thread.join or by Thread.isAlive returning false.
Interruption rule. A thread calling interrupt on another thread happens before the interrupted thread detects the interrupt (either by having InterruptedException thrown, or invoking isInterrupted or interrupted).
Finalizer rule. The end of a constructor for an object happens before the start of the finalizer for that object.
Transitivity. If A happens before B, and B happens before C, then A happens before C.

The key point of the program order rule is: in a thread.
Imagine this simple program (all variables initially 0):
T1:
x = 5;
y = 6;
T2:
if (y == 6) System.out.println(x);
From T1's perspective, an execution must be consistent with y being assigned after x (program order). However from T2's perspective this does not have to be the case and T2 might print 0.
T1 is actually allowed to assign y first as the 2 assignements are independent and swapping them does not affect T1's execution.
With proper synchronization, T2 will always print 5 or nothing.
EDIT
You seem to be misinterpreting the meaning of program order. The program order rule boils down to:
If x and y are actions of the same thread and x comes before y in program order, then hb(x, y) (i.e. x happens-before y).
happens-before has a very specific meaning in the JMM. In particular, it does not mean that y=6 must be subsequent to x=5 in T1 from a wall clock perspective. It only means that the sequence of actions executed by T1 must be consistent with that order. You can also refer to JLS 17.4.5:
It should be noted that the presence of a happens-before relationship between two actions does not necessarily imply that they have to take place in that order in an implementation. If the reordering produces results consistent with a legal execution, it is not illegal.
In the example I gave above, you will agree that from T1's perspective (i.e. in a single threaded program), x=5;y=6; is consistent with y=6;x=5; since you don't read the values. A statement on the next line is guaranteed, in T1, to see those 2 actions, regardless of the order in which they were performed.

How to understand happens-before consistent

In chapter 17 of JLS, it introduce a concept: happens-before consistent.
A set of actions A is happens-before consistent if for all reads r in A, where W(r) is the write action seen by r, it is not the case that either hb(r, W(r)) or that there exists a write w in A such that w.v = r.v and hb(W(r), w) and hb(w, r)"
In my understanding, it equals to following words:
..., it is the case that neither ... nor ...
So my first two questions are:
is my understanding right?
what does "w.v = r.v" mean?
It also gives an Example: 17.4.5-1
Thread 1 Thread 2
B = 1; A = 2;
r2 = A; r1 = B;
In first execution order:
1: B = 1;
3: A = 2;
2: r2 = A; // sees initial write of 0
4: r1 = B; // sees initial write of 0
The order itself has already told us that two threads are executed alternately, so my third question is: what does left number mean?
In my understanding, the reason of both r2 and r1 can see initial write of 0 is both A and B are not volatile field. So my fourth quesiton is: whether my understanding is right?
In second execution order:
1: r2 = A; // sees write of A = 2
3: r1 = B; // sees write of B = 1
2: B = 1;
4: A = 2;
According to definition of happens-before consistency, it is not difficult to understand this execution order is happens-before consistent(if my first understanding is correct).
So my fifth and sixth questions are: does it exist this situation (reads see writes that occur later) in real world? If it does, could you give me a real example?

Each thread can be on a different core with its own private registers which Java can use to hold values of variables, unless you force access to coherent shared memory. This means that one thread can write to a value storing in a register, and this value is not visible to another thread for some time, like the duration of a loop or whole function. (milli-seconds is not uncommon)
A more extreme example is that the reading thread's code is optimised with the assumption that since it never changes the value, it doesn't need to read it from memory. In this case the optimised code never sees the change performed by another thread.
In both cases, the use of volatile ensures that reads and write occur in a consistent order and both threads see the same value. This is sometimes described as always reading from main memory, though it doesn't have to be the case because the caches can talk to each other directly. (So the performance hit is much smaller than you might expect).
On normal CPUs, caches are "coherent" (can't hold stale / conflicting values) and transparent, not managed manually. Making data visible between threads just means doing an actual load or store instruction in asm to access memory (through the data caches), and optionally waiting for the store buffer to drain to give ordering wrt. other later operations.

happens-before
Let's take a look at definitions in concurrency theory:
Atomicity - is a property of operation that can be executed completely as a single transaction and can not be executed partially. For example Atomic operations[Example]
Visibility - if one thread made changes they are visible for other threads. volatile before Java 5 with happens-before
Ordering - compiler is able to change an ordering of operations/instructions of source code to make some optimisations.
For example happens-before which is a kind of memory barrier which helps to solve Visibility and Ordering issue. Good examples of happens-before are volatile[About], synchronized monitor[About]
A good example of atomicity is Compare and swap(CAS) realization of check then act(CTA) pattern which should be atomic and allows to change a variable in multithreading envirompment. You can write your own implementation if CTA:
volatile + synchronized
java.util.concurrent.atomic with sun.misc.Unsafe(memory allocation, instantiating without constructor call...) from Java 5 which uses JNI and CPU advantages.
CAS algoritm has thee parameters(A(address), O(old value), N(new value)).
If value by A(address) == O(old value) then put N(new value) into A(address),
else O(old value) = value from A(address) and repeat this actions again
Happens-before
Official doc
Two actions can be ordered by a happens-before relationship. If one action happens-before another, then the first is visible to and ordered before the second.
volatile[About] as an example
A write to a volatile field happens-before every subsequent read of that field.
Let's take a look at the example:
// Definitions
int a = 1;
int b = 2;
volatile boolean myVolatile = false;
// Thread A. Program order
{
a = 5;
b = 6;
myVolatile = true; // <-- write
}
//Thread B. Program order
{
//Thread.sleep(1000); //just to show that writing into `myVolatile`(Thread A) was executed before
System.out.println(myVolatile); // <-- read
System.out.println(a); //prints 5, not 1
System.out.println(b); //prints 6, not 2
}
Visibility - When Thread A changes/writes a volatile variable it also pushes all previous changes into RAM - Main Memory as a result all not volatile variable will be up to date and visible for another threads
Ordering:
All operations before writing into volatile variable in Thread A will be called before. JVM is able to reorder them but guarantees that no one operation before writing into volatile variable in Thread A will be called after it.
All operations after reading the volatile variable in Thread B will be called after. JVM is able to reorder them but guarantees that no one operation after reading a volatile variable in Thread B will be called before it.
[Concurrency vs Parallelism]

The Java Memory Model defines a partial ordering of all your actions of your program which is called happens-before.
To guarantee that a thread Y is able to see the side-effects of action X (irrelevant if X occurred in different thread or not) a happens-before relationship is defined between X and Y.
If such a relationship is not present the JVM may re-order the operations of the program.
Now, if a variable is shared and accessed by many threads, and written by (at least) one thread if the reads and writes are not ordered by the happens before relationship, then you have a data race.
In a correct program there are no data races.
Example is 2 threads A and B synchronized on lock X.
Thread A acquires lock (now Thread B is blocked) and does the write operations and then releases lock X. Now Thread B acquires lock X and since all the actions of Thread A were done before releasing the lock X, they are ordered before the actions of Thread B which acquired the lock X after thread A (and also visible to Thread B).
Note that this occurs on actions synchronized on the same lock. There is no happens before relationship among threads synchronized on different locks

In substance that is correct. The main thing to take out of this is: unless you use some form of synchronization, there is no guarantee that a read that comes after a write in your program order sees the effect of that write, as the statements might have been reodered.
does it exist this situation (reads see writes that occur later) in real world? If it does, could you give me a real example?
From a wall clock's perspective, obviously, a read can't see the effect of a write that has not happened yet.
From a program order's perspective, because statements can be reordered if there isn't a proper synchronization (happens before relationship), a read that comes before a write in your program, could see the effect of that write during execution because it has been executed after the write by the JVM.

Q1: is my understanding right?
A: Yes
Q2: what does "w.v = r.v" mean?
A: The value of w.v is same as that of r.v
Q3: What does left number mean?
A: I think it is statement ID like shown in "Table 17.4-A. Surprising results caused by statement reordering - original code". But you can ignore it because it does not apply to the conent of "Another execution order that is happens-before consistent is: " So the left number is shit completely. Do not stick to it.
Q4: In my understanding, the reason of both r2 and r1 can see initial write of 0 is both A and B are not volatile field. So my fourth quesiton is: whether my understanding is right?
A: That is one reason. re-order can also make it. "A program must be correctly synchronized to avoid the kinds of counterintuitive behaviors that can be observed when code is reordered."
Q5&6: In second execution order ... So my fifth and sixth questions are: does it exist this situation (reads see writes that occur later) in real world? If it does, could you give me a real example?
A: Yes. no synchronization in code, each thread read can see either the write of the initial value or the write by the other thread.
time 1: Thread 2: A=2
time 2: Thread 1: B=1 // Without synchronization, B=1 of Thread 1 can be interleaved here
time 3: Thread 2: r1=B // r1 value is 1
time 4: Thread 1: r2=A // r2 value is 2
Note "An execution is happens-before consistent if its set of actions is happens-before consistent"