Recently, I've been looking into several caching solutions for our product. After a little research we concluded that we use ehcache for our caching needs. But given the fact that we have a cluster of servers communicating with each others there is also a need for searching a clustering solution as well.
Luckily, Ehcache also supports clustering in its own way using either
- RMI
- JMS
- JGroups
All these methods work on the concept called replication. Ehcache keeps a copy in each of the cache node and replicates each and every mutable cache operation on every node in the cluster. Even though this solves the cluster problem to some extent, its in-efficient in a sense that I have a copy
of cache everywhere and I've to figure out how to keep things consistent and at the same time scalable to any sizes of caches.
Alternatives for the approach mentioned above were are also available.. see
here.
However, what I was looking for is a true clustered solution where a single copy of cache is made visible to all the nodes in the cluster as a single cache and at the same time I should be able to scale the cache to whichever size I want at runtime.
Terracotta is the answer to this question.
The installation worked like a breeze with absolutely no issues on windows. But I had to download the generic tar.gz file instead of a installer jar file for installing it in solaris.
I've used "The definitive Guide to Terracotta" book as a guide for my quest and I'd recommend everyone to read this book for a better understanding of how terracotta works internally. You can read it online ( a limited version ofcourse ) at
google books.
The book has one HelloClusteredWorld example in it which explains the basic working of Terracotta using a simple java program. It works in the strightforward way and so I don't want to mention it here again. I want to show another experiment that I conducted my own using the ehcache integration module ( the
ehcache TIM for Terracotta ) .
Following is the sample groovy program that I used for testing this.
import net.sf.ehcache.Element
import java.io.InputStreamReader
import net.sf.ehcache.CacheManager
/**
*
*/
url = getClass().getResource("ehcache-config.xml");
cacheMgr = new CacheManager(url)
def choice = "Y"
cache = cacheMgr.getCache("userCache");
inreader=new InputStreamReader(System.in)
println "Current cache size is : ${cache.getSize()}"
while ( true ){
print " what do you want to do ? "
choice = new InputStreamReader(System.in).readLine()
switch ( choice ){
case "flush":
println "Flushing the cache."
cacheMgr.flush()
break;
case "add":
print "Enter the key element you want to add : "
key = new InputStreamReader(System.in).readLine()
print "Enter the value element you want to add : "
val = new InputStreamReader(System.in).readLine()
cache.put(new Element(key, val))
break;
case "remove":
print "Enter key of the element to remove : "
key = inreader.readLine()
cache.remove(key)
case "rall":
print "Removing everything !"
cache.removeAll()
break;
case "size":
print "Current cache size is ${cache.getSize()}"
break;
default:
println "I don't understand what you're saying ? "
System.exit(0)
}
}
The purpose of this program is simple, It initializes a ehcache using its cache manager and tries to do CRUD operations in it. Without clustering this program works in a strightforward way waiting for user's input to do the corresponding action.
Following is the Ehcache XML to use...
<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="ehcache.xsd">
<defaultCache
maxElementsInMemory="10000"
eternal="false"
timeToIdleSeconds="120"
timeToLiveSeconds="120"
overflowToDisk="true"
diskSpoolBufferSizeMB="30"
maxElementsOnDisk="10000000"
diskPersistent="false"
diskExpiryThreadIntervalSeconds="120"
memoryStoreEvictionPolicy="LRU"
/>
<cache name="userCache"
maxElementsInMemory="10"
eternal="true"
overflowToDisk="true"
diskSpoolBufferSizeMB="20"
timeToIdleSeconds="300"
timeToLiveSeconds="600"
diskPersistent="false"
memoryStoreEvictionPolicy="LFU">
</cache>
</ehcache>
Now, I tried to convert this program into a cluster-aware program using terracotta. I used following tc-config.xml file.
<?xml version="1.0" encoding="UTF-8"?>
<!--
All content copyright Terracotta, Inc., unless otherwise indicated. All rights reserved.
-->
<!--
This is a Terracotta configuration file that has been pre-configured
for use with DSO. All classes are included for instrumentation,
and all instrumented methods are write locked.
For more information, please see the product documentation.
-->
<tc:tc-config xmlns:tc="http://www.terracotta.org/config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-4.xsd">
<servers>
<!-- Tell DSO where the Terracotta server can be found. -->
<server host="localhost">
<data>%(user.home)/terracotta/server-data</data>
<logs>%(user.home)/terracotta/server-logs</logs>
<dso>
<persistence>
<mode>permanent-store</mode>
</persistence>
</dso>
</server>
</servers>
<!-- Tell DSO where to put the generated client logs -->
<clients>
<logs>%(user.home)/terracotta/client-logs</logs>
<modules>
<module name="tim-ehcache-1.4.1" version="1.3.2"/>
</modules>
</clients>
<application>
<dso>
<roots>
<root>
<field-name>SubscriberCreator.cacheMgr</field-name>
</root>
<root>
<field-name>SubscriberCreator.cache</field-name>
</root>
</roots>
<!-- Start by including just the classes you expect to get added to the shared
graph. These typically include domain classes and shared data structures.
If you miss classes, Terracotta will throw NonPortableOjectExceptions
telling you more about what needs to be added. -->
<instrumented-classes>
<include>
<class-expression>SubscriberCreator</class-expression>
</include>
</instrumented-classes>
</dso>
</application>
</tc:tc-config>
But when I ran this program, i got this error.
Starting Terracotta client...
2009-07-01 12:23:44,758 INFO - Terracotta 3.0.1, as of 20090514-130552 (Revision 12704 by cruise@su1
0mo5 from 3.0)
2009-07-01 12:23:45,476 INFO - Configuration loaded from the file at 'd:\mPortal\Workspace_research\
HelloClusteredWorld\tc-config.xml'.
2009-07-01 12:23:45,711 INFO - Log file: 'C:\Documents and Settings\Admin\terracotta\client-logs\ter
racotta-client.log'.
2009-07-01 12:23:49,320 INFO - Connection successfully established to server at 127.0.0.1:9510
2009-07-01 12:23:49,570 WARN - The root expression 'SubscriberCreator.cacheMgr' meant for the class
'SubscriberCreator' has no effect, make sure that it is a valid expression and that it is spelled co
rrectly.
2009-07-01 12:23:49,570 WARN - The root expression 'SubscriberCreator.cache' meant for the class 'Su
bscriberCreator' has no effect, make sure that it is a valid expression and that it is spelled corre
ctly.
com.tc.exception.TCNonPortableObjectError:
*******************************************************************************
Attempt to share an instance of a non-portable class by assigning it to a root. This unshareable
class is a JVM- or host machine-specific resource. Please ensure that instances of this class
don't enter the shared object graph.
For more information on this issue, please visit our Troubleshooting Guide at:
http://terracotta.org/kit/troubleshooting
Thread : main
JVM ID : VM(22)
Non-portable root name: ALL_CACHE_MANAGERS
Unshareable class : java.util.concurrent.CopyOnWriteArrayList
Action to take:
1) Change your application code
* Ensure that no instances or subclass instances of java.util.concurrent.CopyOnWriteArrayList
are assigned to the DSO root: ALL_CACHE_MANAGERS
*******************************************************************************
at com.tc.object.ClientObjectManagerImpl.throwNonPortableException(ClientObjectManagerImpl.java:786)
at com.tc.object.ClientObjectManagerImpl.checkPortabilityOfRoot(ClientObjectManagerImpl.java:690)
at com.tc.object.ClientObjectManagerImpl.lookupOrCreateRoot(ClientObjectManagerImpl.java:656)
at com.tc.object.ClientObjectManagerImpl.lookupOrCreateRoot(ClientObjectManagerImpl.java:642)
at com.tc.object.bytecode.ManagerImpl.lookupOrCreateRoot(ManagerImpl.java:321)
at com.tc.object.bytecode.ManagerImpl.lookupOrCreateRoot(ManagerImpl.java:300)
at com.tc.object.bytecode.ManagerUtil.lookupOrCreateRoot(ManagerUtil.java:96)
at net.sf.ehcache.CacheManager.__tc_setALL_CACHE_MANAGERS(CacheManager.java)
at net.sf.ehcache.CacheManager.(CacheManager.java:61)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:164)
at SubscriberCreator.class$(SubscriberCreator.groovy)
at SubscriberCreator.$get$$class$net$sf$ehcache$CacheManager(SubscriberCreator.groovy)
at SubscriberCreator.run(SubscriberCreator.groovy:9)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:86)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:234)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1062)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:893)
at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:744)
at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:727)
at org.codehaus.groovy.runtime.InvokerHelper.runScript(InvokerHelper.java:383)
at org.codehaus.groovy.runtime.InvokerHelper$runScript.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:129)
at SubscriberCreator.main(SubscriberCreator.groovy)
Exception in thread "main" com.tc.exception.TCNonPortableObjectError:
*******************************************************************************
Attempt to share an instance of a non-portable class by assigning it to a root. This unshareable
class is a JVM- or host machine-specific resource. Please ensure that instances of this class
don't enter the shared object graph.
For more information on this issue, please visit our Troubleshooting Guide at:
http://terracotta.org/kit/troubleshooting
Thread : main
JVM ID : VM(22)
Non-portable root name: ALL_CACHE_MANAGERS
Unshareable class : java.util.concurrent.CopyOnWriteArrayList
Action to take:
1) Change your application code
* Ensure that no instances or subclass instances of java.util.concurrent.CopyOnWriteArrayList
are assigned to the DSO root: ALL_CACHE_MANAGERS
*******************************************************************************
at com.tc.object.ClientObjectManagerImpl.throwNonPortableException(ClientObjectManagerImpl.java:786)
at com.tc.object.ClientObjectManagerImpl.checkPortabilityOfRoot(ClientObjectManagerImpl.java:690)
at com.tc.object.ClientObjectManagerImpl.lookupOrCreateRoot(ClientObjectManagerImpl.java:656)
at com.tc.object.ClientObjectManagerImpl.lookupOrCreateRoot(ClientObjectManagerImpl.java:642)
at com.tc.object.bytecode.ManagerImpl.lookupOrCreateRoot(ManagerImpl.java:321)
at com.tc.object.bytecode.ManagerImpl.lookupOrCreateRoot(ManagerImpl.java:300)
at com.tc.object.bytecode.ManagerUtil.lookupOrCreateRoot(ManagerUtil.java:96)
at net.sf.ehcache.CacheManager.__tc_setALL_CACHE_MANAGERS(CacheManager.java)
at net.sf.ehcache.CacheManager.(CacheManager.java:61)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:164)
at SubscriberCreator.class$(SubscriberCreator.groovy)
at SubscriberCreator.$get$$class$net$sf$ehcache$CacheManager(SubscriberCreator.groovy)
at SubscriberCreator.run(SubscriberCreator.groovy:9)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:86)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:234)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1062)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:893)
at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:744)
at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:727)
at org.codehaus.groovy.runtime.InvokerHelper.runScript(InvokerHelper.java:383)
at org.codehaus.groovy.runtime.InvokerHelper$runScript.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:129)
at SubscriberCreator.main(SubscriberCreator.groovy)
Okay, so there're two errors in the trace I pasted above. The first one is kind of a warning message from terracotta that it is unable to identify the member variables I declared in the XML file in the SubscriberCreator class file.
2009-07-01 12:23:49,570 WARN - The
root expression 'SubscriberCreator.cacheMgr' meant for the class'SubscriberCreator' has no effect, make sure that it is a valid expression and that it is spelled correctly.
2009-07-01 12:23:49,570 WARN -
The root expression 'SubscriberCreator.cache' meant for the class 'SubscriberCreator' has no effect, make sure that it is a valid expression and that it is spelled correctly.
The reason for this error is because the class file I used to launch the terracotta client was generated from a groovy program. When Groovy compiler compiles the groovy file it converts the fields that we declare in the class as properties inside the class. So, whatever member variables that we declare in the groovy file is not in fact a member variable but internally its represented as a key in a HashTable.
Instead of worrying about fixing this, I just converted the groovy program into a java program and this problem has vanished since then. Following is the equivalent java program...
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URL;
import net.sf.ehcache.Cache;
import net.sf.ehcache.CacheManager;
import net.sf.ehcache.Element;
public class SubscriberCreator {
private URL url;
private CacheManager cacheMgr;
private Cache cache;
public SubscriberCreator() {
url = getClass().getResource("ehcache-config.xml");
cacheMgr = new CacheManager(url);
cache = cacheMgr.getCache("userCache");
}
public static void main(String[] args) throws IOException {
SubscriberCreator creator = new SubscriberCreator();
System.out.println("Current cache size is : "+creator.cache.getSize());
do {
System.out.println("\nWhat do you want to do ? \n\t 1) Add \n\t 2) remove \n\t 3) flush \n\t 4) clear cache \n\t 5) get \n\t 6) size");
System.out.print("Enter your choice : ");
String choice = new BufferedReader(new InputStreamReader(System.in)).readLine();
if ( choice == null ){
System.out.println (" Bye Bye ");
System.exit(0);
}
int choiceint;
try {
choiceint = Integer.parseInt(choice);
} catch (NumberFormatException e) {
System.out.println("Bye bye ");
break;
}
String key;
String val;
switch (choiceint){
case 1:
System.out.print ("Enter the key element you want to add : ");
key = new BufferedReader(new InputStreamReader(System.in)).readLine();
System.out.print("Enter the value element you want to add : ");
val = new BufferedReader(new InputStreamReader(System.in)).readLine();
creator.cache.put(new Element(key, val));
break;
case 2:
System.out.print ("Enter the key element you want to remove : ");
key = new BufferedReader(new InputStreamReader(System.in)).readLine();
creator.cache.remove(key);
System.out.println("key '"+key+"' removed Successfully !");
break;
case 3:
System.out.println("Flushing the cache now !");
creator.cache.flush();
System.out.println("Flushed the cache successfully ");
break;
case 4:
System.out.println("Clearning the cache !");
creator.cache.removeAll();
System.out.println("Cleared the cache successfully ");
break;
case 5:
System.out.print ("Enter the key element you want to retrieve : ");
key = new BufferedReader(new InputStreamReader(System.in)).readLine();
System.out.println("Element requested is : "+creator.cache.get(key));
break;
case 6:
System.out.println ("Cache size is : "+creator.cache.getSize());
break;
default:
{
System.out.println("I don't understand your request..");
System.exit(0);
}
}
}while ( true );
}
}
Coming to the next half of the problem, it complains about a JDK class called
java.util.concurrent.CopyOnWriteArrayList. It appears that terracotta 3.1 or earlier versions of it doesn't have the ability to instrument all the classes in the java.util.concurrent package.
I later realized that from ehcache 1.6 beta3 they have migrated to the JDK concurrent package from the third party concurrent package. So, I had to go down the ehcache release line and re-tried the same example with ehcache 1.5 stable release. It works like a charm !
What did I learnt from this exercise ? Terracotta is a well tested software , but there's definitely a learning curve involved in order to understand what to share/monitor/instrument etc. Without this the whole concept looks unclear and likely to confuse you than do any better.
Here're the minimum requirements for the terracotta 3.1
java version : JDK 1.5 +
ehcache TIM version : 1.4.1 or 1.3 both works fine.
ehcache core version : ehcache 1.5.x ( 1.6 is not yet supported, hope the new version of terracotta fixes these short comings )
Operating systems : Windows XP or higher, Solaris ( I tested only on these machines, I see no reason it doesn't work on other operating systems. )
