ScappShots

Recently I have been looking into .NET-friendly in-memory data grid (IMDG) software. One of the products I have been experimenting with is Alachisoft’s NCache.

When I analyze a distributed caching product, I like to focus first on concurrency control. Before examining performance, scalability, or advanced architectural features, it is valuable to confirm that the software can be used to construct applications that remain reliable when data is shared and subject to updates. To do this I usually construct a test case involving extreme data hot spots, and exercise the features offered by the product for managing data access in these heavy contention situations.

If the product I’m working with offers features for dividing work into ACID style transactions then I explore those features as part of my analysis of concurrent data access. NCache does not provide features for using ACID transactions, so my work with NCache has focused on its locking features.

NCache provides both optimistic and pessimistic concurrency control. The pessimistic locking model turns out to be unusual, at least in my experience. Although the documentation on this feature is quite sparse, with help from the (very responsive) tech. support team at Alachisoft I was able to get an understanding of how NCache’s pessimistic locking works and how to use the NCache API to control it.

Here are some key points about NCache’s pessimistic locking:

1. The NCache data access API provides specialized methods for acquiring locks, releasing locks and accessing items in the cache under locking constraints.

2. Each lock represents the state of an item of data in the cache.

3. Locks are not granted to a thread or a process. Every lock is equally visible to and manipulable by all cache client threads, regardless of which thread originally requested the lock.

4. Locking is completely cooperative. If locking is used to control access to a data item then, to ensure that exclusive access to that item will be maintained until the routine that requested the lock relinquishes it, every potentially concurrent access to that item must be made using the locking form of the NCache data access API. Using a non-locking API method to access a locked item will bypass the lock.

5. In at least one case, using a non-locking API method will not only bypass the lock but will remove it.

6. NCache’s locks will never cause a client thread to block. Data access methods that access a locked item in the cache will return right away. Clients must include logic to interpret the results of the method invocation and retry it if appropriate.

7. In some cases, to determine why a method has not been successful the side effects of the method must be examined.

This final point requires a fuller explanation. Most of the the pessimistic locking API methods take a reference to an object called a LockHandle as one of their arguments. A LockHandle stores the information that identifies a lock on a data item. The values stored in the LockHandle you pass in may be changed by the method. Depending on which method you use and the state (locked or unlocked) of the item you are trying to access, to interpret the results of your locking call it may be necessary to inspect the values in the LockHandle you passed in. Here is an example:

The Get() method takes a key and returns the associated cached item if there is one. The locking form of the Get() method takes three additional arguments, one of which is a LockHandle reference. Depending on the values of those arguments, the method can be used to lock an object and retrieve it in a single call. In this case the method will return null if there is no object in the cache with the specified key value, or if the object exists but is already locked. To distinguish between these two cases the caller must examine the post-call values in the LockHandle.

NCache’s concurrency features work as intended, and with them I was able to maintain the data consistency required by my test scenario. Both the optimistic and the pessimistic features delivered correct results when data items were subject to vigorous concurrent access. However, when compared with the concurrency features provided by well known competitors, NCache’s concurrency controls are quirky and not particularly developer-friendly. Its vulnerability to pessimistic locks being accidentally bypassed or removed by clients that don’t conform to the cooperative lock management protocol makes applications that depend on exclusive data access prone to bugs. The non-blocking approach may be welcome in some situations, but the lack of blocking calls is likely to lead to increased client code complexity and reduced performance in many cases. The lack of support for ACID transactions poses a significant obstacle to building applications that must maintain strict data consistency. For some use cases NCache’s pessimistic locking or its optimistic concurrency features may suffice, but developers designing a system with shared access to writable data should study its concurrency control feature set carefully before choosing NCache.

If you are familiar with Terracotta then you know that its hallmark feature is  data sharing between Java programs running on different virtual machines.  Making use of this very powerful capability requires that the the programs that  share the data use Java synchronization to prevent conflicts between operations that access the shared data from corrupting the data or returning incomplete results.  Terracotta data sharing also requires that Terracotta be configured to detect and honour the Java synchronization instructions. In this post we refer to the combination of Java synchronization and Terracotta configuration as a “locking strategy”.

The choice of locking strategy can have a profound impact on a Terracotta application’s  performance.  Even in a very simple application, there may be several points of data contention at which locking strategies are required, and several possible locking strategies for each contention point.   This post provides a very brief illustration of how locking strategies are implemented with Terracotta, and of the impact that a small change in locking strategy can have on  application performance.

This post barely scratches the surface of Terracotta’s remarkable data sharing capabilities, and it completely bypasses many other powerful and important features of the product.  Readers are advised to regard  this post as a very simple illustration of some basic principles of working with Terracotta, and nothing more.

Readers should also understand that this post demonstrates use of Terracotta’s low level concurrency features.  Many Terracotta users will find themselves using Terracotta’s integration modules (TIMS), which provide out-of-the-box integration with productivity frameworks such as Hibernate and Spring.  The intent of the developers of these integration modules seems to be to shield users of the module as much as possible from the kinds of low level concurrency concerns that feature prominently in this post.

As a starting point, we create two Java programs from three classes:
A – a data POJO.
TCLockingExampleMain – a program that creates and instance of A and updates A’s data field.
TCLockingExampleReporter – a program that indicates whether or not our data is shareable.

Here is the source code for all three classes prior to implementing any locking strategy:

package tcLockingExample;

public class A {
int primInt;

public void primIntInc() {
this.primInt++;
}
}

package tcTLockingExample;

public class TCLockingExampleMain {
static A aInstance = new A();

/**
* @param args
*/
public static void main(String[] args) {
for (int x = 0; x < 10; x++ ) {
aInstance.primIntInc();
System.out.println(“aInstance.primInt: ” + aInstance.primInt);
}
}
}

package tcTLockingExample;

import java.util.Date;

public class TCLockingExampleReporter {

/**
* @param args
*/
public static void main(String[] args) {
System.out.println(new Date() + ” TCLockingChoicesMain.aInstance.primInt:” + TCLockingExampleMain.aInstance.primInt);
}
}

We will be tinkering with the first two classes as the example progresses.

The first step is to establish that the programs work as expected when run without Terracotta.  As the following output illustrates, both programs run:

aInstance.primInt: 1
aInstance.primInt: 2
aInstance.primInt: 3
aInstance.primInt: 4
aInstance.primInt: 5
aInstance.primInt: 6
aInstance.primInt: 7
aInstance.primInt: 8
aInstance.primInt: 9
aInstance.primInt: 1

but, of course, there is no data sharing between them:

Sun Jun 07 17:48:16 BST 2009 TCLockingChoicesMain.aInstance.primInt:0

Next we run each prgram as a Terracotta application, but without configuring Terracotta to share data between them.  The results are the same:

2009-06-07 18:06:48,029 INFO – Terracotta 3.0.0, as of 20090410-200435 (Revision 12431 by cruise@su10mo5 from 3.0)
2009-06-07 18:06:48,334 INFO – Configuration loaded from the file at ‘/home/dan/workspace/TCLockingExample/tc-config.xml’.
2009-06-07 18:06:48,494 INFO – Log file: ‘/home/dan/workspace/TCLockingExample/terracotta/client-logs/terracotta-client.log’.
2009-06-07 18:06:49,907 INFO – Connection successfully established to server at 192.168.1.20:9510
aInstance.primInt: 1
aInstance.primInt: 2
aInstance.primInt: 3
aInstance.primInt: 4
aInstance.primInt: 5
aInstance.primInt: 6
aInstance.primInt: 7
aInstance.primInt: 8
aInstance.primInt: 9
aInstance.primInt: 10

and from TCLockingExampleReporter:

2009-06-07 18:09:03,467 INFO – Terracotta 3.0.0, as of 20090410-200435 (Revision 12431 by cruise@su10mo5 from 3.0)
2009-06-07 18:09:03,772 INFO – Configuration loaded from the file at ‘/home/dan/workspace/TCLockingExample/tc-config.xml’.
2009-06-07 18:09:03,905 INFO – Log file: ‘/home/dan/workspace/TCLockingExample/terracotta/client-logs/terracotta-client.log’.
2009-06-07 18:09:05,199 INFO – Connection successfully established to server at 192.168.1.20:9510
Sun Jun 07 18:09:05 BST 2009 TCLockingChoicesMain.aInstance.primInt:0

Next we configure Terracotta to be aware of all three classes, and set a trap for ourselves by establishing the instance of A  in TCLockingExampleMain as a root class (a Terracotta root is an object that is identified as shared by the application’s Terracotta configuration):

<dso>
<instrumented-classes>
<include>
<class-expression>tcTLockingExample.A</class-expression>
</include>
<include>
<class-expression>tcTLockingExample.TCLockingExampleMain</class-expression>
</include>
<include>
<class-expression>tcTLockingExample.TCLockingExampleReporter</class-expression>
</include>
</instrumented-classes>
<roots>
<root>
<field-name>tcTLockingExample.TCLockingExampleMain.aInstance</field-name>
</root>
</roots>
</dso>

This is a trap because, having established  TCLockingExampleMain.aInstance as a shared object, we are obliged to implement a locking strategy wherever we write to it in our code.  Because we have not done this,   TCLockingExampleMain fails:

2009-06-07 18:13:48,927 INFO – Terracotta 3.0.0, as of 20090410-200435 (Revision 12431 by cruise@su10mo5 from 3.0)
2009-06-07 18:13:49,240 INFO – Configuration loaded from the file at ‘/home/dan/workspace/TCLockingExample/tc-config.xml’.
2009-06-07 18:13:49,371 INFO – Log file: ‘/home/dan/workspace/TCLockingExample/terracotta/client-logs/terracotta-client.log’.
2009-06-07 18:13:50,845 INFO – Connection successfully established to server at 192.168.1.20:9510
com.tc.object.tx.UnlockedSharedObjectException:
*********************************************************************
Attempt to access a shared object outside the scope of a shared lock.
All access to shared objects must be within the scope of one or more
shared locks defined in your Terracotta configuration.

Caused by Thread: main in VM(0)
Shared Object Type: tcTLockingExample.A

The cause may be one or more of the following:
* Terracotta locking was not configured for the shared code.
* The code itself does not have synchronization that Terracotta
can use as a boundary.
* The class doing the locking must be included for instrumentation.
* The object was first locked, then shared.

For more information on how to solve this issue, see:

http://www.terracotta.org/usoe

*********************************************************************

at com.tc.object.tx.ClientTransactionManagerImpl.getTransaction(ClientTransactionManagerImpl.java:360)
at com.tc.object.tx.ClientTransactionManagerImpl.fieldChanged(ClientTransactionManagerImpl.java:653)
at com.tc.object.TCObjectImpl.objectFieldChanged(TCObjectImpl.java:317)
at com.tc.object.TCObjectImpl.intFieldChanged(TCObjectImpl.java:357)
at tcTLockingExample.A.__tc_setprimInt(A.java)
at tcTLockingExample.A.primIntInc(A.java:7)
at tcTLockingExample.TCLockingExampleMain.main(TCLockingExampleMain.java:11)
Exception in thread “main” com.tc.object.tx.UnlockedSharedObjectException:
*********************************************************************
Attempt to access a shared object outside the scope of a shared lock.
All access to shared objects must be within the scope of one or more
shared locks defined in your Terracotta configuration.

Caused by Thread: main in VM(0)
Shared Object Type: tcTLockingExample.A

The cause may be one or more of the following: . . .

To fix this we will implement our first locking strategy, by synchronizing the method in A that writes to the data member of the shared instance:
package tcLockingExample;

public class A {
int primInt;

synchronized public void primIntInc() {
this.primInt++;
}
}

and configuring Terracotta to apply its locking to that method:

<dso>
<instrumented-classes>
<include>
<class-expression>tcLockingExample.A</class-expression>
</include>
<include>
<class-expression>tcLockingExample.TCLockingExampleMain</class-expression>
</include>
<include>
<class-expression>tcLockingExample.TCLockingExampleReporter</class-expression>
</include>
</instrumented-classes>
<roots>
<root>
<field-name>tcLockingExample.TCLockingExampleMain.aInstance</field-name>
</root>
</roots>
<locks>
<autolock>
<method-expression>void tcLockingExample.A.primIntInc()</method-expression>
<lock-level>write</lock-level>
</autolock>
</locks>
</dso>

Now TCLockingExampleMain works:

2009-06-07 18:24:22,965 INFO – Terracotta 3.0.0, as of 20090410-200435 (Revision 12431 by cruise@su10mo5 from 3.0)
2009-06-07 18:24:23,284 INFO – Configuration loaded from the file at ‘/home/dan/workspace/TCLockingExample/tc-config.xml’.
2009-06-07 18:24:23,414 INFO – Log file: ‘/home/dan/workspace/TCLockingExample/terracotta/client-logs/terracotta-client.log’.
2009-06-07 18:24:25,324 INFO – Connection successfully established to server at 192.168.1.20:9510
aInstance.primInt: 1
aInstance.primInt: 2
aInstance.primInt: 3
aInstance.primInt: 4
aInstance.primInt: 5
aInstance.primInt: 6
aInstance.primInt: 7
aInstance.primInt: 8
aInstance.primInt: 9
aInstance.primInt: 10

and TCLockingExampleReporter shows that the data is being shared:

2009-06-07 18:27:21,486 INFO – Terracotta 3.0.0, as of 20090410-200435 (Revision 12431 by cruise@su10mo5 from 3.0)
2009-06-07 18:27:21,804 INFO – Configuration loaded from the file at ‘/home/dan/workspace/TCLockingExample/tc-config.xml’.
2009-06-07 18:27:21,933 INFO – Log file: ‘/home/dan/workspace/TCLockingExample/terracotta/client-logs/terracotta-client.log’.
2009-06-07 18:27:23,271 INFO – Connection successfully established to server at 192.168.1.20:9510
Sun Jun 07 18:27:23 BST 2009 TCLockingChoicesMain.aInstance.primInt:10

Next we try a different locking strategy.  We remove the synchronization from A’s method, and instead synchronize TCLockingExampleMain’s write operation on the root object:

package tcLockingExample;

public class TCLockingExampleMain {
static A aInstance = new A();

/**
* @param args
*/
public static void main(String[] args) {
for (int x = 0; x < 10; x++ ) {
synchronized(aInstance) {
aInstance.primIntInc();
}
System.out.println(“aInstance.primInt: ” + aInstance.primInt);
}
}
}

We also update the Terracotta configuration, applying a Terracotta autolock (which means that Terracotta will add its locking wherever it sees Java synchronization) to TCLockingExampleMain’s main() method instead of A’s incrementer method:

<dso>
<instrumented-classes>
<include>
<class-expression>tcLockingExample.A</class-expression>
</include>
<include>
<class-expression>tcLockingExample.TCLockingExampleMain</class-expression>
</include>
</instrumented-classes>
<roots>
<root>
<field-name>tcLockingExample.TCLockingExampleMain.aInstance</field-name>
</root>
</roots>
<locks>
<autolock>
<method-expression>void tcLockingExample.TCLockingExampleMain.main(java.lang.String[])</method-expression>
<lock-level>write</lock-level>
</autolock>
</locks>
</dso>

This strategy also works:
2009-06-07 18:45:39,099 INFO – Terracotta 3.0.0, as of 20090410-200435 (Revision 12431 by cruise@su10mo5 from 3.0)
2009-06-07 18:45:39,425 INFO – Configuration loaded from the file at ‘/home/dan/workspace/TCLockingExample/tc-config.xml’.
2009-06-07 18:45:39,555 INFO – Log file: ‘/home/dan/workspace/TCLockingExample/terracotta/client-logs/terracotta-client.log’.
2009-06-07 18:45:41,441 INFO – Connection successfully established to server at 192.168.1.20:9510
aInstance.primInt: 1
aInstance.primInt: 2
aInstance.primInt: 3
aInstance.primInt: 4
aInstance.primInt: 5
aInstance.primInt: 6
aInstance.primInt: 7
aInstance.primInt: 8
aInstance.primInt: 9
aInstance.primInt: 10

and data sharing is enabled:
2009-06-07 18:44:17,498 INFO – Terracotta 3.0.0, as of 20090410-200435 (Revision 12431 by cruise@su10mo5 from 3.0)
2009-06-07 18:44:17,816 INFO – Configuration loaded from the file at ‘/home/dan/workspace/TCLockingExample/tc-config.xml’.
2009-06-07 18:44:17,951 INFO – Log file: ‘/home/dan/workspace/TCLockingExample/terracotta/client-logs/terracotta-client.log’.
2009-06-07 18:44:19,710 INFO – Connection successfully established to server at 192.168.1.20:9510
Sun Jun 07 18:44:19 BST 2009 TCLockingChoicesMain.aInstance.primInt:10

Finally we’ll take a quick look at how the choice of locking strategy can affect performance.  To see this, we change  TCLockingExampleMain so it increments A’s integer member a million times instead of ten as in previous executions.  We also add some code to tell us how long the program took to do the million iterations:

package tcLockingExample;

import java.util.Date;

public class TCLockingExampleMain {
static A aInstance = new A();

/**
* @param args
*/
public static void main(String[] args) {
Date startTime = new Date();
for (int x = 0; x < 1000000; x++) {
aInstance.primIntInc();
}
Date endTime = new Date();
System.out.println(“startTime: ” + startTime + ” endTime: ” + endTime
+ ” elapsed: ”
+ ((endTime.getTime() – startTime.getTime()) / 1000)
+ ” seconds”);
System.out.println(“aInstance.primInt: ” + aInstance.primInt);
}
}

When we run this program we see that the million iterations take around 26 seconds:

2009-06-07 19:16:09,091 INFO – Terracotta 3.0.0, as of 20090410-200435 (Revision 12431 by cruise@su10mo5 from 3.0)
2009-06-07 19:16:09,410 INFO – Configuration loaded from the file at ‘/home/dan/workspace/TCLockingExample/tc-config.xml’.
2009-06-07 19:16:09,544 INFO – Log file: ‘/home/dan/workspace/TCLockingExample/terracotta/client-logs/terracotta-client.log’.
2009-06-07 19:16:11,022 INFO – Connection successfully established to server at 192.168.1.20:9510
startTime: Sun Jun 07 19:16:11 BST 2009 endTime: Sun Jun 07 19:16:37 BST 2009 elapsed: 26 seconds
aInstance.primInt: 1000000

Next we try our second locking strategy with a million iterations:

package tcLockingExample;

import java.util.Date;

public class TCLockingExampleMain {
static A aInstance = new A();

/**
* @param args
*/
public static void main(String[] args) {
Date startTime = new Date();
for (int x = 0; x < 1000000; x++) {
synchronized (aInstance) {
aInstance.primIntInc();
}
}
Date endTime = new Date();
System.out.println(“startTime: ” + startTime + ” endTime: ” + endTime
+ ” elapsed: ”
+ ((endTime.getTime() – startTime.getTime()) / 1000)
+ ” seconds”);
System.out.println(“aInstance.primInt: ” + aInstance.primInt);
}
}

It takes about the same amount of time:

2009-06-07 19:21:08,279 INFO – Terracotta 3.0.0, as of 20090410-200435 (Revision 12431 by cruise@su10mo5 from 3.0)
2009-06-07 19:21:08,602 INFO – Configuration loaded from the file at ‘/home/dan/workspace/TCLockingExample/tc-config.xml’.
2009-06-07 19:21:08,737 INFO – Log file: ‘/home/dan/workspace/TCLockingExample/terracotta/client-logs/terracotta-client.log’.
2009-06-07 19:21:10,266 INFO – Connection successfully established to server at 192.168.1.20:9510
startTime: Sun Jun 07 19:21:10 BST 2009 endTime: Sun Jun 07 19:21:37 BST 2009 elapsed: 26 seconds
aInstance.primInt: 1000000

For our last test we move the synchronization statement outside of the loop, meaning that only one lock is required instead of a million (one per iteration):

package tcLockingExample;

import java.util.Date;

public class TCLockingExampleMain {
static A aInstance = new A();

/**
* @param args
*/
public static void main(String[] args) {
Date startTime = new Date();
synchronized (aInstance) {
for (int x = 0; x < 1000000; x++) {
aInstance.primIntInc();
}
}
Date endTime = new Date();
System.out.println(“startTime: ” + startTime + ” endTime: ” + endTime
+ ” elapsed: ”
+ ((endTime.getTime() – startTime.getTime()) / 1000)
+ ” seconds”);
System.out.println(“aInstance.primInt: ” + aInstance.primInt);
}
}

Execution time drops from 26 seconds to less than one second:

2009-06-07 19:23:52,409 INFO – Terracotta 3.0.0, as of 20090410-200435 (Revision 12431 by cruise@su10mo5 from 3.0)
2009-06-07 19:23:52,727 INFO – Configuration loaded from the file at ‘/home/dan/workspace/TCLockingExample/tc-config.xml’.
2009-06-07 19:23:52,861 INFO – Log file: ‘/home/dan/workspace/TCLockingExample/terracotta/client-logs/terracotta-client.log’.
2009-06-07 19:23:54,335 INFO – Connection successfully established to server at 192.168.1.20:9510
startTime: Sun Jun 07 19:23:54 BST 2009 endTime: Sun Jun 07 19:23:55 BST 2009 elapsed: 0 seconds
aInstance.primInt: 1000000

As these very simple examples show, there are often several choices for how to implement locking in a Terracotta application, and the selection of a strategy can have profound implications for application performance.  For a developer who is new to Terracotta, selecting appropriate locking strategies may involve significant amounts of trial and error.  As the developer’s understanding of Terracotta grows with experience, locking strategy selection becomes easier and less experimentation is required.