I was working on a C# project recently, and we were discussing my architecture. He was struck by my use of factory classes and told me that in the .NET world, developers simply do not create factory classes. Instead, they will create static methods or extension methods, such as a static Create() method on a domain object. He pointed out that this has the benefit of simplicity: the maintainer of the code only has to get their head around one class (the object created with the factory method) rather than two classes (an object and its factory). Also, you get the benefit of your code editor: a developer using your code does not even need to know about your factory method initially but will discover it via Intellisense.

These are certainly reasonable points, but of course, by using static factory methods, we lose the benefit of polymorphism. What if we want a class to be provided a factory so it can create objects, but the factory that it is provided could be one of several variants? For example, say I wanted a Worker class to create a IContentProvider according to some IContentProviderFactory that is injected into it. There could be multiple implementations of IContentProvider: FileContentProvider, DbContentProvider, etc. Static Create() methods on the FileContentProvider, DbContentProvider, etc. would force the Worker class to be aware of the various types of content providers rather than be agnostic of them.

In this case, if we are just producing one object dynamically at runtime, we could provide a delegate or lambda property or constructor parameter on our object. The method passed in could create the desired object and therefore serve as a factory.

However, if we have something more like an abstract factory pattern, in which we wish to have a factory that creates a series of related objects and you have more than one implementation of this factory, then this technique will not work very cleanly. In this case, we really DO need a factory class.

Any thoughts?

It should be noted that you can have more than these 3 factors, and this will add additional security. But at the very least, supplying these three fundamental factors should be a basis for your security model.

I have not always been directly involved (but have been involved with many of the design decisions), but the isuses involved are quite fascinating and I still wanted to share it with you, my faithful blog reader. By the way, the chief developer on this is Christian Jungers: www.CM3Consulting.com. (I am not a representative or employee of CM3 Consulting, and my opinions do
not necessarily reflect theirs, or theirs reflect mine.)

Some interesting design decisions we made:

The Factors We Chose

Who You Are

What We Use

• We are allowing any arbitrary user to access the site, so we don’t want to require users to provide biometric data or use other means to assure that we are dealing with a particular user.
• Thus we are using the machine that the user is using as a “proxy” for the user themselves.
• Specifically we use the MAC address of the machine.

How We Obtain it

• How do we determine this from a browser? We bring back the ancient concept of a Java applet
• We make use of a small amount of Javascript to grab the MAC address from the applet and place it in a hidden form field which can then be submitted to the server as part of a user login.
• By logging in, we add the MAC address to a list of MAC addresses associated with the user and store this in persistent storage. In this way, we effectively register the machine and associate it with the user.

Preventing Man-in-the-Middle Attacks

• We don’t want to send the MAC address in the clear to the server, because otherwise a “man in the middle” could steal the MAC address and then pretend to be that machine (by spoofing its own MAC address as that address)!
• Hence, we need to encrypt the MAC address sent to the user.
• Is it sufficient to simply use an HTTPS connection to the server? No!
• The connection always uses the same public key to encrypt content sent over the wire all the time (based on the certificate from the server), so the man in the middle can intercept the encrypted MAC address and then use this to authenticate themselves to the server as if they were the original server.
• Thus, we generate a new public/private key pair for each session and use this to encrypt the MAC address. This makes it very hard for the man-in-the-middle to intercept the encrypted MAC address and do something useful with it.

A Sample of How We Foil Man-in-the-Middle

1. User Angel goes to login page.
2. Among the values passed to the page is a public key that is used to encrypt the MAC address which has been generated by the server for Angel’s session.

   Let us call the public/private key pair generated public key A and private key A, respectively.

3. The MAC address portlet runs and gets the MAC address of the user’s machine.
4. The MAC address is encrypted with public key A and is sent to what it thinks is the server.
5. The man-in-the-middle, named User Mallory (the malicious user), is listening/sniffing the connection and sees the encrypted MAC address on the line.
6. Mallory goes to the login site but performs some spoofing magic (by tweaking the page and what is sent) so as to send the encrypted MAC address as if it were her own MAC address.
7. When Mallory goes to the login page, the server generates a new public/private key pair, let us call these public key B and private key B.
8. When the server receives the encrypted MAC address, she attempts to decrypt it with the private key B. However, it was originally encrypted with public key A, and thus the decryption fails to match the MAC address that user A registered.
9. Thus Mallor is unable to login as if she were Angel.

Efficiency Measures – Generating Keys on Server Startup

• Generating a separate key for each session is time consuming (about 0.5 a second on our dual core desktops). Of course, this will be considerably faster on the server but this still leaves us vulnerable to denial of service attacks.
• Instead we can generate a number of keys (say 100 or 1000) on server startup, and then each time we start a new session, we will randomly grab a key out of the pool.
• This makes user login much quicker, but it really doesn’t compromise security in any meaningful way.
• It still would be very hard for an eavesdropper to determine when a key is reused and therefore he would most likely not be able to hijack somebody else’s session).

Dealing with the Several MAC Addresses in Your Machine

• Actually, your machine does not have just one MAC address but several (corresponding to the various network cards/ network interfaces that your machine has.
• Our heuristic: we store all of these MAC addresses but consider a user’s machine a match as long as at least one of these MAC addresses match

Allowing the User to Login with Multiple Machines

• We store a number (8 to be specified) of MAC addresses for each user; if the user logs in with more than the allowed number of machines, roll the oldest machine off the list
• Should we have instead not let the user log in with more than 8 machines?

What You Have

What We Use

First Logging onto a New Machine

• When the user is first created, the user is emailed a temporary password and they must login with it (and then be forced to change the password immediately)
• When the user signs in on a new machine, they will be emailed an Auth code which the user will have to provide before being allowed to login with that machine.

Subsequent Logins – Provide a Cookie

• A revision to what I said above: actually we don’t just sign the MAC address. The user is also provided a random “seed” that comes down to the user via a cookie.
• The seed and MAC address are combined, signed, and then stored on the server as one of the machines that is associated with the user.
• This cookie thus has to be provided with each subsequent request from that machine.
• Thus, whether the user is logging in for the first time, using a machine for the first time, or just logging in other scenarios, the user always has to provide something.

What You Know

What We Currently Use

• The user generally has to provide a password.
• Right now, the user has to provide their email address when they log in which we are considering “something they know” when they login for first time. But this is a weak factor because it is generally publicly known or at least easily known.
• The email address would be the only “thing they know” when they log in for first time (because the temporary password is provided via email so it is something they have, not something they know).

What We Should Use

• Probably better to provide for a series of challenge questions that a call center representative can obtain when initially registering the user.
• Then the user will be required to provide answers to them when they log in (thus showing what they know).

I know, very involved, but also hopefully very informative of some of the things going through our minds as we implemented this.

The presentation, which I had previously given, is located here.

For this presentation, I had developed a sample LINQ provider that queries Yahoo Finance for some basic stock and stock option information and forms simple associations between them. I cheated a little bit and used Google Finance for one small piece of this as well.

Lessons learned:

• Writing a LINQ provider is quite hard. LINQ basically is providing you something like a framework for compilation of a fairly restricted language, but you are building part of a compiler nonetheless, which is never easy.

  Specifically the .NET compiler compiles your LINQ query into something called an expression tree, which is a static representation of the LINQ query code, e.g. what filters are applied by the query, what kinds of objects is it selecting, etc. So the parsing of the query is done for you. But you are responsible for the translation and execution of the expression tree, i.e. the actual calls to the real data source, whether this is a web service, database, API, or anything else.

• Using a LINQ provider is really easy. This is the most important part. You learn LINQ once, and then you know how to use pretty much any LINQ provider.

  Look how easy it is to use my new LINQ provider:

  from stockOption in optionContext.Query
  join stock in stockContext.Query on stockOption.UnderlyingStock equals stock
  where (stock.Symbol == "IBM" && stockOption.Strike > 35
                  && stockOption.Premium > 5)
  select new {stockOption.Symbol, stockOption.Premium,
                  StockSymbol = stock.Symbol,
                  stockOption.Strike})
  

  First a small bit of financial knowledge. An option is a right but not an obligation to buy or sell a certain amount of some underlying
  instrument, in this case stock for a certain price. It is time limited (it must be exercised by a certain date or it expires). So for
  a stock option, the underlying instrument is a stock. The strike price of an option is the price at which you now have to right to buy or sell the stock. So if I have an option to buy IBM stock for $50 by November 1, and the price of IBM stock rises to $65 on October 15th, I may choose to exercise the stock option. This would allow me to buy the IBM stock for $50, rather than the $65
  current price of the stock. I could also choose to wait it out, thinking that IBM stock might dip below $50, in which case I could buy
  it for cheaper than $50. However, if the option expires (we pass November 1st) and I have not executed it, the option expires worthless. One other note: the option also has a premium, which is the price I have to pay to obtain the option. Even if the
  option becomes worthless, the most I could pay out of pocket is the premium.

  It is easily explained what the query is doing above. I am finding all options currently available with the underlying stock “IBM” and then finding all stock options for IBM that have a strike price greater than $35 and for which the premium is greater than $5. I then return a list of anonymous objects, something like a tuple, that contains the stock symbol, the stock option symbol, and the stock option premium and strike price. I return one result for each stock option that matches the criteria.

  The key thing here is that while you had to have a little bit of business/domain knowledge to understand the query, you really did not require technical knowledge beyond basic LINQ knowledge to understand it. Furthermore, you didn’t even really need LINQ knowledge to understand it, because it looks so much like standard SQL.

  So it is clear (at least anecdotally) that LINQ is designed with the assumption that a few developers will experience the pain of creating providers with the understanding that the masses will benefit from simplicity of use.

So in one of my mapping files, I had the following tag:

<cacheModel id="objCache"
        type="com.ibatis.sqlmap.engine.cache.memory.MemoryCacheController"
        readOnly="true" serialize="false">
 </cacheModel>

I then applied this cache model to one of my result maps:

<select id="getSomething" parameterClass="..."
		resultMap="keyValueMap" cacheModel="objCache">
....
<select>

And I noticed an unexpected result: my query results were not actually being cached! In other words, despite the caching, the query was being executed against the database every single time.

The problem had to do with a setting on the IBatis MemoryCacheController: strong vs. weak caching. This has to do with a concept that has been there since the Java 1.3 days: weak references. Here’s one link that explains the various types of Java references:

We are used to strong references: when we have a reference to an object, the object will not be garbage collected until the reference is released. Objects referred to by weak references, on the other hand, can be reclaimed assuming there are no other references to the object .

By default, the MemoryCacheController uses weak caching by default. However, because our application only held temporary references to the cached objects (other than the cache itself), the cached objects were reclaimed by the garbage collector. Hence, the next time IBatis attempted to check the cache associated with a particular query, it did not find anything cached and ran the query again! Repeat this process, ad infinitum, and you understand why caching seemed to be nonexistent.

So I changed this:

<select id="getSomething" parameterClass="..."
		resultMap="keyValueMap" cacheModel="objCache">
    	<property name="referenceType" value="STRONG"/>
  	</cacheModel>

Note I am now resetting the reference type to STRONG, which causes the cache to use strong references rather than weak references. This solved the problem, because now the reference to the cached object itself will prevent the cached object from being garbage collected.

It is interesting to note that before I solved the problem, I noticed something strange: this problem did not occur when I ran the debugger. In other words, the debugger caused the cache to work correctly! The cached objects were not being garbage collected, and thus the IBatis cache did not have to run the queries again. This is not an application of the Heisenberg uncertainty principle (in which watching a quantum phenomenon changes its progression), but actually was due to the compiler itself holding references to the cached objects, whcih caused the objects to remain cached.

This touched on the following areas:

• Tomcat connectors
• Application server session management, specifically the management of the JSESSIONID cookie
• Cookie domains
• Virtual hosting and clustering
• Liferay public and private pages

Let me step you through it.

Liferay and Virtual Hosts 

We started out by making use of Liferay virtual hosts. Liferay organizations represent different deployments of a portal site to discrete groups of users or roles. Virtual hosting in Liferay specifically is a feature in Liferay that enables one to map domain names to Liferay organizations. This became essential in our project, because we turned on clustering, and thus we needed a mechanism to map a common URL to whatever host we are using to serve a user’s request. One important note to make is that we are making use of sticky sessions (no session replication for now), so all requests made by a given client will be routed to the same server. We also wanted to provide different domain names for different Liferay organizations, so as to make these appear to be completely different sites. We could have accomplished some of these goals via Apache URL mappings, but we wanted to leverage the Liferay support for this (for reasons beyond the scope of this article which I am not even sure are justified).

Liferay has a concept of public and private pages for a given organization. Public pages are pages that can be accessed before a user logs into Liferay, while private pages are only accessible once a user has logged into Liferay. This seems like an inappropriate security architecture to me, in that traditionally, one would provide something like an access control list for each page or groups of pages that defines what access (if any) that a user has to those pages. But this is an issue for another day, another post. I bring this up because Liferay has a weird restriction with their virtual hosting: Liferay seems to make you enter a different virtual host for public pages than for private pages. The rest of our troubles began with that.

 The Problem: Liferay, Virtual Hosts, and Session Sharing

In our application, when a user was logging in, we are storing information in the session. We are also causing a “Liferay login”, i.e. logging in as a Liferay user, which causes Liferay to allow the user to then access the private pages. The problem we started to notice was that when a user logged in, the information that was added to the session was no longer available once the user logged in. More specifically, it appeared like the session information added while on the public virtual host was not available on the private virtual host.

The Diagnosis: Sharing Sessions Across Different Hosts

The first naive but incorrect answer is that a different server is processing the request to initially log in the user from the request issued after the user is logged in. However, remember this is actually not correct, because we are making use of sticky sessions, so all requests by a given user are handled by the same server. So even though Liferay switches the virtual host being used when the user logs in, all requests are being served by the same underlying server. So this is not being caused by having different sessions on different servers. Probing further …

So what actually happened? Remember that a Java application server such as Tomcat creates a JSESSIONID cookie on the client’s desktop with the session id of the current session (unless, of course, we make use of URL rewriting, but we weren’t). The domain of the JSESSIONID cookie is the entire virtual host name, i.e. if our public virtual host is “login.xyz.com” and the private virtual host is “secure.xyz.com”, then while on the public site, one JSESSIONID was being stored for domain “login.xyz.com” and then on the private site, another JSESSIONID cookie was being stored for domain “secure.xyz.com”. Now you can start to see the problem. We are creating two sessions, one on the public virtual host “login.xyz.com” and one on the private virtual host “secure.xyz.com”. Why?  When the user hits the public site, a JSESSIONID with the domain “login.xyz.com” is created. When the user is then redirected to the private site, the domain of the existing JSESSIONID is not available because the domain of that cookie does not match the new host “secure.xyz.com”. Thus, it appears as if no session is available, and a new session is created on the server. Correspondingly, a new JSESSIONID cookie is created on the client as well for host secure.xyz.com.

First Solution Attempt: An Incorrect Custom Tomcat Connector Implementation 

How can solve this? Well, if I could create a JSESSIONID cookie that had a domain of “xyz.com”, then such a cookie would match both the public and private sites, because both host names are suffixed with xyz.com. But how can I do this? I initially started by trying to create my own JSESSIONID cookies, but this turned out to be complicated in that Tomcat continued to create JSESSIONID cookies on its own. So what I really needed was to somehow hook into Tomcat to change the way it creates JSESSIONID.

How to proceed? Well, my colleague found this: http://issues.liferay.com/browse/LEP-6976. One of the comments on this post indicates a possible fix:   make use of the Liferay class PortalStandardService, an extension of the Tomcat class StandardService, configure this in server.xml, and everything will just start to work. In theory this should have worked.   Basically, what this class does is that it overrides the method addConnector. Normally this method is called for each Tomcat connector configured in server.xml (or a default one, if none are specified). The Liferay version overrides this to wrap the Connector passed to it by Liferay with its own version: PortalConnector. PortalConnector, in turn, (a) copies a series of attributes from the Tomcat Connector it wraps and (b) overrides createRequest() method to create Liferay’s own version of the HttpServletRequest returned by Tomcat. The CreateRequest() override does something similar to what we are seeing so far: it wraps the HttpServletRequest created by Tomcat with a PortalRequest (again a Liferay class).

The Liferay class PortalRequest is where the magic happens. This class overrides a method configureSessionCookie, a method exposed by the Tomcat Request class (which in turn extends HttpServletRequest). This method receives the JSESSIONID Cookie from each request, and gives us an opportunity to set fields on it. In this case, PortalRequest takes the opportunity to modify the domain of the cookie to the common domain (“xyz.com”) by removing the first level of the domain. This, in theory, makes it possible to share the same JSESSIONID cookie across all hosts that are suffixed with “xyz.com”, so we can share the same session between the public and private sites.

So what’s wrong? It doesn’t work due to a bug in the Liferay class PortalConnector. In order to understand this bug, it is important to note that the Tomcat class Connector makes use of something called a ProtocolHandler, which it instantiates dynamically based on the protocol of the connector specified. Dynamic attributes set on the Connector are passed through, via property reflection, to the ProtocolHandler. The problem with the Liferay implementation of Connector, PortalConnector, was that it did not properly copy over these attributes, and hence the underlying ProtocolHandler created for PortalConnector did not contain the attributes/property values from the originally wrapped Tomcat Connector (and its internal ProtocolHandler). The key here is the the original Tomcat Connector creates its own ProtocolHandler(s) via reflection and our wrapping PortalConnector does the same (because it extends the Tomcat Connector). Hence, there needs to be a separate step to copy over the attributes from the original ProtocolHandler (of the original Tomcat Connector) to the Liferay wrapping connector, which is not being performed. Specifically, among the attributes that needed to be copied were the address and port, which because they were not assigned on the new ProtocolHandler, they are set to defaults, e.g. Port was set to 8080. However, the server already had something running on that port and received an “Address already in use” exception.

The Real Solution: The Correct Tomcat Connector For Creating Cross Domain Sessions

The solution was to provide our own implementation of the Tomcat StandardService. All of the components of our implementation were the same, except we retrieved the ProtocolHandler from the wrapped Connector and assigned it to the appropriate member variable of our subclass of Connector. This worked, because rather than using the newly instantiated (via reflection) ProtocolHandler that was created for us when we created PortalConnector, we instead reassign the protocolHandler member variable to use the Connector that Tomcat originally created from the server.xml configuration. So rather than copying over the attributes from one ProtocolHandler to another, we simply change the ProtocolHandler that our new PortalConnector is using, so it will use the correct attributes. So the new ProtocolHandler now has the correct port, and now we can finally make use of the magic that correctly sets the JSESSIONID domain, and everything works.

Related Scenarios

These issues are not restricted to the scenario delineated above. Other scenarios include

• Redirecting from http to https (non-secure to secure site)
• Intercepting application server operations that run on every request (through custom connector mechanism described above)
• Maintaining session across different sites in the same domain

I had a class to load assemblies upon AssemblyResolve events in a new AppDomain:

public abstract class AssemblyLoader : IAssemblyLoader
{
     public void Start(MyRequest request)
    {
               AppDomain.CurrentDomain.AssemblyResolve += (
                         (sender, resolveEventArgs)
                         => CurrentDomainAssemblyResolve(sender,
                                           resolveEventArgs,
                                           request));
     }

     public void Stop(MyRequest request)
     {
              AppDomain.CurrentDomain.AssemblyResolve -= ((sender, resolveEventArgs)
                         => CurrentDomainAssemblyResolve(sender,
                                           resolveEventArgs,
                                           request));
     }

     protected abstract Assembly CurrentDomainAssemblyResolve(object sender,
                   ResolveEventArgs args,
                   MyRequest request);
}

So this abstract class exposes an abstract method CurrentDomainAssemblyResolve which gets passed the standard AssemblyResolve arguments plus a custom request object.

I then created a child class to load the assemblies from disk:

public class DirectoryAssemblyLoader : AssemblyLoader
{
       protected override Assembly CurrentDomainAssemblyResolve(
                 object sender,
                 ResolveEventArgs args,
                 MyRequest request)
      {
             var assemblyNames = GetAssemblyNames(request);
             foreach (var assemblyName in assemblyNames)
            {
                  if (args.Name.Split(',')[0] != assemblyName) continue;

                  var assemblyPath =  GetAssemblyPath(assemblyName);
                  var assemblyBytes = File.ReadAllBytes(assemblyPath);
                  Assembly assembly = Assembly.Load(assemblyBytes);

                  return assembly;
              }

             return null;
       }
}

A couple of things about this code:

- First of all, this code was an intermediate step.  I eventually wanted to get the bytes for my assembly not from the file system but rather from another form of persistent storage which would store the bytes of the assembly.  That is why I first read the file containing the assembly from the file system, getting an array of bytes and then call Assembly.Load with the bytes (as opposed to using Assembly.LoadFrom file system.)

- This code checks that the assembly requested corresponds to the MyRequest argument, which contains configuration/metadata about which assemblies can be used satisfy the request.  The loop in the middle of code deals with this.  Basically we don’t want to load arbitrary assemblies that do not correspond to the request.    We are trying to find which assembly in the request metadata corresponds to the assembly which is currently being requested (which we find in the ResolveEventArgs argument)

- I noticed that this code was broken if I tried to resolve the same or equivalent assembly name (as specified by ResolveEventArgs.Name) more than once.  Even though these multiple resolution attempts are requesting the same assembly name, the assemblies returned are considered to be different assemblies.

- Because of this, if I try to load the same type more than once using this DirectoryAssemblyLoader class, it will consider the types loaded to be different types, leading to exceptions at runtime.

- The following post provides details about the different assembly load contexts:  http://blogs.msdn.com/suzcook/archive/2003/05/29/57143.aspx.  This example would be considered neither Load context or LoadFrom context.

BTW, if I had used the LoadFrom context instead (used Assembly.LoadFrom), this all would have worked, because if an assembly is loaded in this way more than once from the same place, the LoadFrom context recognizes these assemblies loaded from the same place as the same assembly.  In fact, the second resolution attempt will result in return of the already loaded assembly.  (although if an assembly with the same strong name was already loaded in the Load context via Assembly.Load, it may return that one instead, but that was not the case here)

The solution here was to cache the loaded assemblies by AssemblyName , so that the next time we attempt to access the assembly, we first look in the cache to see if we have already found the assembly:

     var aName = new AssemblyName(args.Name);

     // Here I am re-using the assemblies that I already loaded so
     // that the types resolve properly.

    foreach (var pair in assemblyCache)
        if (AssemblyName.ReferenceMatchesDefinition(aName, pair.Key))

    return pair.Value;

..........

      Assembly assembly = Assembly.Load(assemblyBytes);

      assemblyCache.Add(assembly.GetName(), assembly);

      return assembly;

So now the code first sees if an equivalently named assembly has been stored in our cache, and if so, it returns it.  If not, it loads the assembly as before but then adds it to the cache.

Hope this really interesting problem sheds light on how the different Assembly loading contexts work.

We have just been involved with extensive load testing on our current project, a portlet based project running on Liferay Portal Server (which in turn runs on top of
Tomcat), and making heavy use of Spring Portlet MVC for front end development.  The following considerations came into play:

Understand The Top Command: Average Load versus CPU Percentage

Our Tomcat/Liferay portlet web app runs on a LINUX server.  Therefore, for a high level understanding of how our load was being handled by the server, we made use of the “top” command.  We initially used the CPU percentage as an indicator of server load but soon discovered this is a high misleading measurement.  Take a look at http://www.linuxjournal.com/article/9001 to understand why CPU percentage is an inaccurate proxy for the load.  It really only captures snapshots of how frequently some task is running on the processor but it does not properly capture how many tasks are queuing up to run on the available processors, how much contention/context switching is occuring as a result of the contention for the processors, etc.  Also, note that the load can and likely will exceed the number of available processors. Finding the point at which the average load (for a time analogous to the period of the test) equals four or five times the number of available processors, in the case of largely I/O bound processes, or closer to the number of processors for processes that are largely CPU-bound,  can help indicate the  sweet spot at which the server is smoothly processing requests without major contention.

Use the Right Load Average

top provides 1, 5, and 15 minute running averages.  For quick burst tests, these measures will not at all be accurate.  So it is important to run tests that run at least a couple of minutes (in order to get an accurate 1 minute running average) which ramp up load, peak load in the middle of the testing period, and then bring load back down. 

Perform Baseline Observations

Make sure to perform baseline observations of the server (to see what basic load is on the server before the test as well as memory usage).  It probably pays to take several measurements of this to make sure that your app server is not currently running other tasks with unpredictable load or performance.  Then perform observations under load, and then additional observations after the test is completed to make sure that load drops back to baseline and that memory usage as well (i.e. there are no memory leaks).

Make Sure Connection Pool Settings Are Correct

Liferay, the Java portal server we are using, uses connection pooling internally by default for its metadata, but its defaults are set very high, and we ended up using an excessive  number of connections to the database, especially when we started to ramp up our load tests.  Try to guard against this by setting minimum and maximum pool sizes to reasonable amounts, both for any web apps with which you are testing as well as for application server metadata database connections.

Set Idle Connection Settings Appropriately

As an extension to the previous point, it is important that the settings include reclamation of idle connections.  We are using c3p0 for connection pooling, and it has a setting for an idle connection timeout.  Make use of these settings, but also make sure your Spring configuration specifies a destroy-method on the connection pool (“close” method in the case of the C3P0 pool class com.mchange.v2.c3p0.ComboPooledDataSource), so that the connections are reclaimed (at least down to min pool size) when the server shuts down.  Note this works even if server is shut down ungraceful, but obviously it will not work under abrupt machine shutdown.

Identify Proper User Cases

This can be a challenge in itself.  Ideally, these can be identified by existing performance data, perhaps by web server or HTTP server log analysis, looking at query execution times, etc.  However, this is not always possible because such data does not exist before you begin load testing. Using a profiling tool might help here, because it is better to measure than guess.  However, often it is helpful to find some use cases that seem like good candidates for profiling, given that there might be a large number of cases out there, and it is not possible to profile every one, or perhaps differences in performance do not become apparent until you hit the application with load. 

We identified such cases by combining knowledge of

(i) the use cases most commonly exercised by users or some proxy for this (perhaps the links at the front of your site)

(ii) queries with particularly complex joins or contain conditions that are not well indexed,

(iii) calls to other external systems for which network communication latency would be the bottleneck. 

Existing load data was not available, or we would have made use of this instead.  We tried to first identify those use cases that were most likely to fit these critieria, and then we further categorized these so as to narrow down to a small set of use cases that will realistically exercise the most likely problem points of the application. 

Clearly Identify Your Load Testing Goals

Are you trying to find the breaking load for the application, e.g. find out how far the app can scale?  Are you looking at response times under realistic load?  Are you trying to identify use cases with performance problems?  This identification can affect which use cases you choose and how you perform the tests.

Determine Appropriate Load

Exactly what load should you use in your load test?  If your users have some idea of what the load will be, maybe because there is a previous web site that you are now replacing, use this as a starting point, but you will likely have to double, triple, or use an even greater scale factor to determine the real load for testing, given that load can increase unexpectedly, especially if you succeed in improving the site significantly.  To be safe, you probably should try to find the breaking point for the test, and then try to determine if that maximum load could ever be realistically exceeded.

Build Up Loads Gradually

If you perform a test that submits requests at a set time interval, note load can increase quickly, due to fact that previous requests might not have been completed before new requests are submitted.  This is why it pays to start out with minimal loads (say 1 user) and then build up to higher, more realistic loads to get an idea how the app behaves under load.

Account for Database Caching

One trickiness with load testing:  if you perform multiple requests but each request retrieves the same read only data, caching may come into play, which can skew your results.  See if there are ways you can introduce variability into your load testing scripts.

Perfrom Several Load Testing Phases

1) Simple tests of our automated scripts and will generally be run against a highly controllable environment (maybe a desktop server) with a load perhaps of only 1 user. 
 
2) We ran more extensive tests against our test server machine, first with individual test cases and then combining multiple test
cases, gradually stepping up load, to see whether server performed adequately under load.  Specifically, we were trying to see whether server could handle expected loads (with a scaling factor for expected increase in load once server is in production) with reasonable response times, no timeouts, reasonable server load, no memory leaks, etc.
Any issues we identified we investigated using a Java profiling tool. 

3) Rinse and repeat for future iterations if changes need to be made to meet performance targets (in our case we met performance targets so this was not necessary). 

4) Following this we will need to perform iterations to determine maximum possible server load (try to break the
application by increasing load until requests start to fail or response times become unacceptable) in order to understand the limits of the app. 

5) In addition, we will need to perform another few iterations to tweak settings to achieve maximum performance and scalaiblity, i.e. adjust Java heap size, play with connection pool settings, app server
settings, etc. 

6) It may be a requirement on your project, as it is on ours, to also run iterations in which real clients join the load so they can get a personal feel for how the app performs under load.