ScappShots

Last month I gave a five minute talk about GigaSpaces and Terracotta at London’s CloudCamp, an event which was well organized, well attended and well received. If you care to see my talk, follow this link:

http://skillsmatter.com/podcast/cloud-grid/refreshment-break-select-breakout-session

(videos generously hosted by the folks at Skills Matter).  Click around and you’ll find a lot more good content.

If you are familiar with Terracotta then you know that its hallmark feature is  data sharing between Java programs running on different virtual machines.  Making use of this very powerful capability requires that the the programs that  share the data use Java synchronization to prevent conflicts between operations that access the shared data from corrupting the data or returning incomplete results.  Terracotta data sharing also requires that Terracotta be configured to detect and honour the Java synchronization instructions. In this post we refer to the combination of Java synchronization and Terracotta configuration as a “locking strategy”.

The choice of locking strategy can have a profound impact on a Terracotta application’s  performance.  Even in a very simple application, there may be several points of data contention at which locking strategies are required, and several possible locking strategies for each contention point.   This post provides a very brief illustration of how locking strategies are implemented with Terracotta, and of the impact that a small change in locking strategy can have on  application performance.

This post barely scratches the surface of Terracotta’s remarkable data sharing capabilities, and it completely bypasses many other powerful and important features of the product.  Readers are advised to regard  this post as a very simple illustration of some basic principles of working with Terracotta, and nothing more.

Readers should also understand that this post demonstrates use of Terracotta’s low level concurrency features.  Many Terracotta users will find themselves using Terracotta’s integration modules (TIMS), which provide out-of-the-box integration with productivity frameworks such as Hibernate and Spring.  The intent of the developers of these integration modules seems to be to shield users of the module as much as possible from the kinds of low level concurrency concerns that feature prominently in this post.

As a starting point, we create two Java programs from three classes:
A – a data POJO.
TCLockingExampleMain – a program that creates and instance of A and updates A’s data field.
TCLockingExampleReporter – a program that indicates whether or not our data is shareable.

Here is the source code for all three classes prior to implementing any locking strategy:

package tcLockingExample;

public class A {
int primInt;

public void primIntInc() {
this.primInt++;
}
}

package tcTLockingExample;

public class TCLockingExampleMain {
static A aInstance = new A();

/**
* @param args
*/
public static void main(String[] args) {
for (int x = 0; x < 10; x++ ) {
aInstance.primIntInc();
System.out.println(“aInstance.primInt: ” + aInstance.primInt);
}
}
}

package tcTLockingExample;

import java.util.Date;

public class TCLockingExampleReporter {

/**
* @param args
*/
public static void main(String[] args) {
System.out.println(new Date() + ” TCLockingChoicesMain.aInstance.primInt:” + TCLockingExampleMain.aInstance.primInt);
}
}

We will be tinkering with the first two classes as the example progresses.

The first step is to establish that the programs work as expected when run without Terracotta.  As the following output illustrates, both programs run:

aInstance.primInt: 1
aInstance.primInt: 2
aInstance.primInt: 3
aInstance.primInt: 4
aInstance.primInt: 5
aInstance.primInt: 6
aInstance.primInt: 7
aInstance.primInt: 8
aInstance.primInt: 9
aInstance.primInt: 1

but, of course, there is no data sharing between them:

Sun Jun 07 17:48:16 BST 2009 TCLockingChoicesMain.aInstance.primInt:0

Next we run each prgram as a Terracotta application, but without configuring Terracotta to share data between them.  The results are the same:

2009-06-07 18:06:48,029 INFO – Terracotta 3.0.0, as of 20090410-200435 (Revision 12431 by cruise@su10mo5 from 3.0)
2009-06-07 18:06:48,334 INFO – Configuration loaded from the file at ‘/home/dan/workspace/TCLockingExample/tc-config.xml’.
2009-06-07 18:06:48,494 INFO – Log file: ‘/home/dan/workspace/TCLockingExample/terracotta/client-logs/terracotta-client.log’.
2009-06-07 18:06:49,907 INFO – Connection successfully established to server at 192.168.1.20:9510
aInstance.primInt: 1
aInstance.primInt: 2
aInstance.primInt: 3
aInstance.primInt: 4
aInstance.primInt: 5
aInstance.primInt: 6
aInstance.primInt: 7
aInstance.primInt: 8
aInstance.primInt: 9
aInstance.primInt: 10

and from TCLockingExampleReporter:

2009-06-07 18:09:03,467 INFO – Terracotta 3.0.0, as of 20090410-200435 (Revision 12431 by cruise@su10mo5 from 3.0)
2009-06-07 18:09:03,772 INFO – Configuration loaded from the file at ‘/home/dan/workspace/TCLockingExample/tc-config.xml’.
2009-06-07 18:09:03,905 INFO – Log file: ‘/home/dan/workspace/TCLockingExample/terracotta/client-logs/terracotta-client.log’.
2009-06-07 18:09:05,199 INFO – Connection successfully established to server at 192.168.1.20:9510
Sun Jun 07 18:09:05 BST 2009 TCLockingChoicesMain.aInstance.primInt:0

Next we configure Terracotta to be aware of all three classes, and set a trap for ourselves by establishing the instance of A  in TCLockingExampleMain as a root class (a Terracotta root is an object that is identified as shared by the application’s Terracotta configuration):

<dso>
<instrumented-classes>
<include>
<class-expression>tcTLockingExample.A</class-expression>
</include>
<include>
<class-expression>tcTLockingExample.TCLockingExampleMain</class-expression>
</include>
<include>
<class-expression>tcTLockingExample.TCLockingExampleReporter</class-expression>
</include>
</instrumented-classes>
<roots>
<root>
<field-name>tcTLockingExample.TCLockingExampleMain.aInstance</field-name>
</root>
</roots>
</dso>

This is a trap because, having established  TCLockingExampleMain.aInstance as a shared object, we are obliged to implement a locking strategy wherever we write to it in our code.  Because we have not done this,   TCLockingExampleMain fails:

2009-06-07 18:13:48,927 INFO – Terracotta 3.0.0, as of 20090410-200435 (Revision 12431 by cruise@su10mo5 from 3.0)
2009-06-07 18:13:49,240 INFO – Configuration loaded from the file at ‘/home/dan/workspace/TCLockingExample/tc-config.xml’.
2009-06-07 18:13:49,371 INFO – Log file: ‘/home/dan/workspace/TCLockingExample/terracotta/client-logs/terracotta-client.log’.
2009-06-07 18:13:50,845 INFO – Connection successfully established to server at 192.168.1.20:9510
com.tc.object.tx.UnlockedSharedObjectException:
*********************************************************************
Attempt to access a shared object outside the scope of a shared lock.
All access to shared objects must be within the scope of one or more
shared locks defined in your Terracotta configuration.

Caused by Thread: main in VM(0)
Shared Object Type: tcTLockingExample.A

The cause may be one or more of the following:
* Terracotta locking was not configured for the shared code.
* The code itself does not have synchronization that Terracotta
can use as a boundary.
* The class doing the locking must be included for instrumentation.
* The object was first locked, then shared.

For more information on how to solve this issue, see:

http://www.terracotta.org/usoe

*********************************************************************

at com.tc.object.tx.ClientTransactionManagerImpl.getTransaction(ClientTransactionManagerImpl.java:360)
at com.tc.object.tx.ClientTransactionManagerImpl.fieldChanged(ClientTransactionManagerImpl.java:653)
at com.tc.object.TCObjectImpl.objectFieldChanged(TCObjectImpl.java:317)
at com.tc.object.TCObjectImpl.intFieldChanged(TCObjectImpl.java:357)
at tcTLockingExample.A.__tc_setprimInt(A.java)
at tcTLockingExample.A.primIntInc(A.java:7)
at tcTLockingExample.TCLockingExampleMain.main(TCLockingExampleMain.java:11)
Exception in thread “main” com.tc.object.tx.UnlockedSharedObjectException:
*********************************************************************
Attempt to access a shared object outside the scope of a shared lock.
All access to shared objects must be within the scope of one or more
shared locks defined in your Terracotta configuration.

Caused by Thread: main in VM(0)
Shared Object Type: tcTLockingExample.A

The cause may be one or more of the following: . . .

To fix this we will implement our first locking strategy, by synchronizing the method in A that writes to the data member of the shared instance:
package tcLockingExample;

public class A {
int primInt;

synchronized public void primIntInc() {
this.primInt++;
}
}

and configuring Terracotta to apply its locking to that method:

<dso>
<instrumented-classes>
<include>
<class-expression>tcLockingExample.A</class-expression>
</include>
<include>
<class-expression>tcLockingExample.TCLockingExampleMain</class-expression>
</include>
<include>
<class-expression>tcLockingExample.TCLockingExampleReporter</class-expression>
</include>
</instrumented-classes>
<roots>
<root>
<field-name>tcLockingExample.TCLockingExampleMain.aInstance</field-name>
</root>
</roots>
<locks>
<autolock>
<method-expression>void tcLockingExample.A.primIntInc()</method-expression>
<lock-level>write</lock-level>
</autolock>
</locks>
</dso>

Now TCLockingExampleMain works:

2009-06-07 18:24:22,965 INFO – Terracotta 3.0.0, as of 20090410-200435 (Revision 12431 by cruise@su10mo5 from 3.0)
2009-06-07 18:24:23,284 INFO – Configuration loaded from the file at ‘/home/dan/workspace/TCLockingExample/tc-config.xml’.
2009-06-07 18:24:23,414 INFO – Log file: ‘/home/dan/workspace/TCLockingExample/terracotta/client-logs/terracotta-client.log’.
2009-06-07 18:24:25,324 INFO – Connection successfully established to server at 192.168.1.20:9510
aInstance.primInt: 1
aInstance.primInt: 2
aInstance.primInt: 3
aInstance.primInt: 4
aInstance.primInt: 5
aInstance.primInt: 6
aInstance.primInt: 7
aInstance.primInt: 8
aInstance.primInt: 9
aInstance.primInt: 10

and TCLockingExampleReporter shows that the data is being shared:

2009-06-07 18:27:21,486 INFO – Terracotta 3.0.0, as of 20090410-200435 (Revision 12431 by cruise@su10mo5 from 3.0)
2009-06-07 18:27:21,804 INFO – Configuration loaded from the file at ‘/home/dan/workspace/TCLockingExample/tc-config.xml’.
2009-06-07 18:27:21,933 INFO – Log file: ‘/home/dan/workspace/TCLockingExample/terracotta/client-logs/terracotta-client.log’.
2009-06-07 18:27:23,271 INFO – Connection successfully established to server at 192.168.1.20:9510
Sun Jun 07 18:27:23 BST 2009 TCLockingChoicesMain.aInstance.primInt:10

Next we try a different locking strategy.  We remove the synchronization from A’s method, and instead synchronize TCLockingExampleMain’s write operation on the root object:

package tcLockingExample;

public class TCLockingExampleMain {
static A aInstance = new A();

/**
* @param args
*/
public static void main(String[] args) {
for (int x = 0; x < 10; x++ ) {
synchronized(aInstance) {
aInstance.primIntInc();
}
System.out.println(“aInstance.primInt: ” + aInstance.primInt);
}
}
}

We also update the Terracotta configuration, applying a Terracotta autolock (which means that Terracotta will add its locking wherever it sees Java synchronization) to TCLockingExampleMain’s main() method instead of A’s incrementer method:

<dso>
<instrumented-classes>
<include>
<class-expression>tcLockingExample.A</class-expression>
</include>
<include>
<class-expression>tcLockingExample.TCLockingExampleMain</class-expression>
</include>
</instrumented-classes>
<roots>
<root>
<field-name>tcLockingExample.TCLockingExampleMain.aInstance</field-name>
</root>
</roots>
<locks>
<autolock>
<method-expression>void tcLockingExample.TCLockingExampleMain.main(java.lang.String[])</method-expression>
<lock-level>write</lock-level>
</autolock>
</locks>
</dso>

This strategy also works:
2009-06-07 18:45:39,099 INFO – Terracotta 3.0.0, as of 20090410-200435 (Revision 12431 by cruise@su10mo5 from 3.0)
2009-06-07 18:45:39,425 INFO – Configuration loaded from the file at ‘/home/dan/workspace/TCLockingExample/tc-config.xml’.
2009-06-07 18:45:39,555 INFO – Log file: ‘/home/dan/workspace/TCLockingExample/terracotta/client-logs/terracotta-client.log’.
2009-06-07 18:45:41,441 INFO – Connection successfully established to server at 192.168.1.20:9510
aInstance.primInt: 1
aInstance.primInt: 2
aInstance.primInt: 3
aInstance.primInt: 4
aInstance.primInt: 5
aInstance.primInt: 6
aInstance.primInt: 7
aInstance.primInt: 8
aInstance.primInt: 9
aInstance.primInt: 10

and data sharing is enabled:
2009-06-07 18:44:17,498 INFO – Terracotta 3.0.0, as of 20090410-200435 (Revision 12431 by cruise@su10mo5 from 3.0)
2009-06-07 18:44:17,816 INFO – Configuration loaded from the file at ‘/home/dan/workspace/TCLockingExample/tc-config.xml’.
2009-06-07 18:44:17,951 INFO – Log file: ‘/home/dan/workspace/TCLockingExample/terracotta/client-logs/terracotta-client.log’.
2009-06-07 18:44:19,710 INFO – Connection successfully established to server at 192.168.1.20:9510
Sun Jun 07 18:44:19 BST 2009 TCLockingChoicesMain.aInstance.primInt:10

Finally we’ll take a quick look at how the choice of locking strategy can affect performance.  To see this, we change  TCLockingExampleMain so it increments A’s integer member a million times instead of ten as in previous executions.  We also add some code to tell us how long the program took to do the million iterations:

package tcLockingExample;

import java.util.Date;

public class TCLockingExampleMain {
static A aInstance = new A();

/**
* @param args
*/
public static void main(String[] args) {
Date startTime = new Date();
for (int x = 0; x < 1000000; x++) {
aInstance.primIntInc();
}
Date endTime = new Date();
System.out.println(“startTime: ” + startTime + ” endTime: ” + endTime
+ ” elapsed: ”
+ ((endTime.getTime() – startTime.getTime()) / 1000)
+ ” seconds”);
System.out.println(“aInstance.primInt: ” + aInstance.primInt);
}
}

When we run this program we see that the million iterations take around 26 seconds:

2009-06-07 19:16:09,091 INFO – Terracotta 3.0.0, as of 20090410-200435 (Revision 12431 by cruise@su10mo5 from 3.0)
2009-06-07 19:16:09,410 INFO – Configuration loaded from the file at ‘/home/dan/workspace/TCLockingExample/tc-config.xml’.
2009-06-07 19:16:09,544 INFO – Log file: ‘/home/dan/workspace/TCLockingExample/terracotta/client-logs/terracotta-client.log’.
2009-06-07 19:16:11,022 INFO – Connection successfully established to server at 192.168.1.20:9510
startTime: Sun Jun 07 19:16:11 BST 2009 endTime: Sun Jun 07 19:16:37 BST 2009 elapsed: 26 seconds
aInstance.primInt: 1000000

Next we try our second locking strategy with a million iterations:

package tcLockingExample;

import java.util.Date;

public class TCLockingExampleMain {
static A aInstance = new A();

/**
* @param args
*/
public static void main(String[] args) {
Date startTime = new Date();
for (int x = 0; x < 1000000; x++) {
synchronized (aInstance) {
aInstance.primIntInc();
}
}
Date endTime = new Date();
System.out.println(“startTime: ” + startTime + ” endTime: ” + endTime
+ ” elapsed: ”
+ ((endTime.getTime() – startTime.getTime()) / 1000)
+ ” seconds”);
System.out.println(“aInstance.primInt: ” + aInstance.primInt);
}
}

It takes about the same amount of time:

2009-06-07 19:21:08,279 INFO – Terracotta 3.0.0, as of 20090410-200435 (Revision 12431 by cruise@su10mo5 from 3.0)
2009-06-07 19:21:08,602 INFO – Configuration loaded from the file at ‘/home/dan/workspace/TCLockingExample/tc-config.xml’.
2009-06-07 19:21:08,737 INFO – Log file: ‘/home/dan/workspace/TCLockingExample/terracotta/client-logs/terracotta-client.log’.
2009-06-07 19:21:10,266 INFO – Connection successfully established to server at 192.168.1.20:9510
startTime: Sun Jun 07 19:21:10 BST 2009 endTime: Sun Jun 07 19:21:37 BST 2009 elapsed: 26 seconds
aInstance.primInt: 1000000

For our last test we move the synchronization statement outside of the loop, meaning that only one lock is required instead of a million (one per iteration):

package tcLockingExample;

import java.util.Date;

public class TCLockingExampleMain {
static A aInstance = new A();

/**
* @param args
*/
public static void main(String[] args) {
Date startTime = new Date();
synchronized (aInstance) {
for (int x = 0; x < 1000000; x++) {
aInstance.primIntInc();
}
}
Date endTime = new Date();
System.out.println(“startTime: ” + startTime + ” endTime: ” + endTime
+ ” elapsed: ”
+ ((endTime.getTime() – startTime.getTime()) / 1000)
+ ” seconds”);
System.out.println(“aInstance.primInt: ” + aInstance.primInt);
}
}

Execution time drops from 26 seconds to less than one second:

2009-06-07 19:23:52,409 INFO – Terracotta 3.0.0, as of 20090410-200435 (Revision 12431 by cruise@su10mo5 from 3.0)
2009-06-07 19:23:52,727 INFO – Configuration loaded from the file at ‘/home/dan/workspace/TCLockingExample/tc-config.xml’.
2009-06-07 19:23:52,861 INFO – Log file: ‘/home/dan/workspace/TCLockingExample/terracotta/client-logs/terracotta-client.log’.
2009-06-07 19:23:54,335 INFO – Connection successfully established to server at 192.168.1.20:9510
startTime: Sun Jun 07 19:23:54 BST 2009 endTime: Sun Jun 07 19:23:55 BST 2009 elapsed: 0 seconds
aInstance.primInt: 1000000

As these very simple examples show, there are often several choices for how to implement locking in a Terracotta application, and the selection of a strategy can have profound implications for application performance.  For a developer who is new to Terracotta, selecting appropriate locking strategies may involve significant amounts of trial and error.  As the developer’s understanding of Terracotta grows with experience, locking strategy selection becomes easier and less experimentation is required.

Chapter 4 of The Definitive Guide to Terracotta says:

When you traverse a reference to a clustered object that isn’t currently instantiated on the local physical heap, Terracotta will automatically request that object’s data from the Terracotta server, instantiate it on the local heap, and wire up the reference you are traversing, and your application will be none the wiser.

Likewise, Terracotta can artificially add null references to clustered objects as memory pressure increases, which lets arbitrarily large clustered object graphs fit within a constrained physical heap space.

I ran a couple of experiments to help me understand how these capabilities translate into application behaviour. You can download the source code for the simple Java programs I used in these experiments here.

The first question I wanted to answer was whether or not Terracotta’s ability to page heap objects in and out would allow a Terracotta client to create an object graph that was too big for its physical heap. The book states that a program can traverse an object graph that is too large for its heap, but it does not claim that the virtualization effect applies to object creation. Nevertheless I thought the possibility was worth investigating.

I designed the experiment to test this as follows:

1. Determine how many instances of a class of a fixed size (just over 3 kB) can be created and added to a HashMap by a non-Terracotta Java application running with a heap space of a known size before the application runs out of heap space.

2. Determine how many instances of the same fixed size class can be created by the same Java application running as a Terracotta client application with the HashMap holding the object instances configured as a Terracotta root, and with a heap space of the same known size.

I foresaw two possible outcomes:

1. The numbers of objects created by the two programs would be about the same, indicating that Terracotta’s ability to relieve memory pressure does not apply to pressure caused by the creation of new objects.

2. The Terracotta client would be able to create many more instances of the test object than its plain Java counterpart did, indicating that Terracotta’s object paging feature detects and handles memory pressure caused by the creation of new objects.

The actual results matched the second of the scenarios I had envisioned. When I ran the test with a heap size of 100 megabytes, the plain Java client created 2,988 instances of the test object before running out of memory.

dan@scapps1:~/workspace/bigHeap$ java -Xms100m -Xmx100m -classpath ./bin bigHeap.BigHeapMain 2988

Done so far: 1000 total memory: 100532224 free memory: 68213168

Done so far: 2000 total memory: 100532224 free memory: 36038896

dan@scapps1:~/workspace/bigHeap$

Running as a Terracotta client, the program successfully instantiated more than 11,000 instances running with a server provisioned with 512 megabytes of heap:

dan@scapps1:~/workspace/bigHeap$ dso-java.sh -Xms100m -Xmx100m -classpath ./bin bigHeap.BigHeapMain 11010

Starting BootJarTool…

2009-06-01 00:08:30,423 INFO – Terracotta 3.0.0, as of 20090409-180411 (Revision 12431 by cruise@su10mo5 from 3.0)

2009-06-01 00:08:30,748 INFO – Configuration loaded from the file at ‘/home/dan/workspace/bigHeap/tc-config.xml’.

Starting Terracotta client…

2009-06-01 00:08:32,353 INFO – Terracotta 3.0.0, as of 20090409-180411 (Revision 12431 by cruise@su10mo5 from 3.0)

2009-06-01 00:08:32,673 INFO – Configuration loaded from the file at ‘/home/dan/workspace/bigHeap/tc-config.xml’.

2009-06-01 00:08:32,804 INFO – Log file: ‘/home/dan/workspace/bigHeap/terracotta/client-logs/terracotta-client.log’.

2009-06-01 00:08:34,786 INFO – Connection successfully established to server at 192.168.1.20:9510

Done so far: 1000 total memory: 93257728 free memory: 39313736

Done so far: 2000 total memory: 93257728 free memory: 34530984

Done so far: 3000 total memory: 93257728 free memory: 33617040

Done so far: 4000 total memory: 93323264 free memory: 36051464

Done so far: 5000 total memory: 93257728 free memory: 24922912

Done so far: 6000 total memory: 93061120 free memory: 10904496

Done so far: 7000 total memory: 93257728 free memory: 39410008

Done so far: 8000 total memory: 93257728 free memory: 32656648

Done so far: 9000 total memory: 93585408 free memory: 28937584

Done so far: 10000 total memory: 93257728 free memory: 23353536

Done so far: 11000 total memory: 93257728 free memory: 29487976

dan@scapps1:~/workspace/bigHeap$

My second experiment was designed to confirm the book’s claim that a client program can operate on an object graph that is too big for the client’s heap. To test this I used a program that iterates over all of the objects in the HashMap created by the program I used for the first experiment. With a 19 megabyte heap the program successfully reads all of the objects in the HashMap, the actual size of which is on the order of 33 megabytes:

dan@scapps1:~/workspace/bigHeap$ dso-java.sh -Xms19m -Xmx19m -classpath ./bin bigHeap.CountEm

Starting BootJarTool…

2009-06-01 00:22:37,684 INFO – Terracotta 3.0.0, as of 20090409-180411 (Revision 12431 by cruise@su10mo5 from 3.0)

2009-06-01 00:22:38,010 INFO – Configuration loaded from the file at ‘/home/dan/workspace/bigHeap/tc-config.xml’.

Starting Terracotta client…

2009-06-01 00:22:39,880 INFO – Terracotta 3.0.0, as of 20090409-180411 (Revision 12431 by cruise@su10mo5 from 3.0)

2009-06-01 00:22:40,204 INFO – Configuration loaded from the file at ‘/home/dan/workspace/bigHeap/tc-config.xml’.

2009-06-01 00:22:40,334 INFO – Log file: ‘/home/dan/workspace/bigHeap/terracotta/client-logs/terracotta-client.log’.

2009-06-01 00:22:42,349 INFO – Connection successfully established to server at 192.168.1.20:9510

Found: 11010

dan@scapps1:~/workspace/bigHeap$

These simple experiments demonstrate that Terracotta’s virtual heap can be used to allow client programs to manage datasets that are too large for their physical heaps. The virtual heap is just one of Terracotta’s distinctive and powerful features. I’ll examine others in subsequent posts.

I’ve mentioned my DICE study – a comparison of eight different solutions to a simple distributed computing problem using GigaSpaces and Terracotta – in several posts.  It is finally available for download.  If you want to take a look, please follow the link to the scapps website.  Access to the paper is quick and easy.

This DICE study I have been working on is as much about scalability – what happens to throughput when you add an additional unit of compute power – as about raw performance.  Nevertheless, testing eight different solutions to the same problem did provide some insight into overall performance characteristics.  I’ll tell you how the eight approaches stacked up in terms of speed, but first let me summarize the problem and the eight solutions.

The problem was to update the objects (“counters”) in a shared data area.  Each update was represented by an object (“updaters”) containing a reference to the counter object to be updated.  The updater objects also had to be written to the shared data area.  High ratios of updaters to counters promoted contention for access to the counters, creating classic hot spots.

Three solutions used Terracotta. The first had several instances of a simple program that processed a list of updaters.  The instances tussled for access to the counters.  The second solution was like the first except that each instance of the program had a list of updaters that referred to a distinct subset of the counters.  With this approach there was no competition for the Counters.  The third Terracotta solution had each instance of an updating program fed by a private queue.  Each queue contained updaters that referred to a distinct set of counters so, again, there was no competition for the counters.

Among these three approaches, the second proved to be the fastest overall, achieving over 5,000 updates per second when run with two or four instances of the updating program.  The first (and simplest) approach was the second fastest.  It’s best performance came with only a single updating program running, when it achieved over 3,400 updates per second.  Overall throughput declined precipitously as additional instances were added.

The third approach was the slowest, but got faster consistently as the number of instances was increased from one to two to four to eight (about the limit of my test environment).  With one instance of  the updating program running throughput was 576 updater per second.  At eight updaters throughput was 1,282 updates per second.

The five GigaSpaces solutions worked as follows:

1. A simple non-PU client that connects to a partitioned space and executes reads  and writes against the space.

2. Clients send updater objects to a partitioned space to be processed.  Updates performed by PUs against local space instances (space-based architecture) that use  a polling containers to detect the arrival of new updater objects.  Clients use writeMultiple() to improve throughput.

3.  Just like no. 2, but using FIFO features to preserve ordering of updates per counter.

4. Clients invoke remote methods advertised by PUs to update counters.  Updaters are passed as arguments.  PUs do work against local space instances.

5. Clients send update requests to spaces as Task objects.  Spaces execute the tasks.

Among these five approached, numbers two and three, both of which use writeMultiple() and polling containers, were the fastest by substantial margins.  Number two delivered over 30,000 updates per second with two clients and four updaters.  Number three came close to 17,000 updates per second with one client and two updaters.

Next fastest was number five at about 3,800 updates per second with one client and two updaters.  Number four peaked at around 2,400 updaters per second with two clients and two updaters.  Slowest of the five was number one, which reached between 1,000 and 1,100 updates per second in a variety of configurations.

Analyzing these results and explaining the differences in performance are topics too large for this post.  A few things are clear  however,  from even the simple set of results presented above:

1. The concept of locality – which decomposes into the related concepts of proximity and exclusivity – is profoundly important in designing solutions to this class of problem.

2. A very wide range of results is possible depending on the solution architecture.

3. For raw speed, GigaSpaces’ polling container construct offers a significant advantage over any of the other choices examined here.

For weeks I’ve been working on a comparison of techniques for updating a distributed data set using Terracotta and GigaSpaces.   This weekend I finally got a draft out for review.  It is mostly ok,  but I got results that I can’t explain when I tested two of the GigaSpaces implementations (there are eight implementations in total – three with Terracotta and five with GigaSpaces).  I’ve asked my friends at the two vendor organizations to take a look at the draft and give me their comments.  Maybe my GigaSpaces contact can help me resolve those two mysteries.  Either way the paper is about done and I’m starting to move on to some new work.

Next on the agenda:

• I have spent some time fiddling with transactional techniques with GigaSpaces.  I’m planning to take that work a bit further and write up my results.

• GemFire has been on my list of products to investigate for awhile.  As of today I have it installed on my test network and I expect to start working with it next week.

Let me know if you are interested in the DICE paper or the GigaSpaces transactional work.