Running Old Benchmark Tests (pre 2.1.2 / 2.2)

Running the Alfresco Benchmark Tests

Introduction

This article is an example of how to set up an Alfresco server purely for benchmarking purposes. It is possible to alter the configuration to use existing Tomcat-expanded installations or otherwise-installed versions of Alfresco. In this example, we show how to set the environment up to execute against a completely separate system, both in terms of configuration as well as data output.

Required Files

• An Alfresco war distribution
• An Alfresco benchmark distribution
• Structure:

  /home/alfresco
                /repo
                /benchmark

Extracting Files

• Extract alfresco.war from war distribution to /tmp
• Explode /tmp/alfresco.war into /home/alfresco/repo

  /home/alfresco/repo/WEB-INF
                             /classes
                             /lib

• Copy database JDBC connector to lib folder

  /home/alfresco/repo/WEB-INF/lib/xxx.jar

• Explode benchmark distribution into /home/alfresco/benchmark

  /home/alfresco/benchmark
                          /config
                          /lib

Edit Configuration Files

• cd /home/alfresco/benchmark

The benchmark tests use the conventional Alfresco override mechanism. Unless otherwise changed, extensions to the default configurations should be put into ./config/alfresco/extension.

• Copy the sample configuration files into ./config/alfresco/extension and remove the .sample extensions:

./config/alfresco/extension
                           /custom-repository-context.xml.sample
                           /custom-hibernate-dialect.properties.sample
                           /custom-repository.properties.sample

• Set repository data locations
  • vi ./config/alfresco/extension/custom-repository.properties

  db.url=jdbc:mysql:///alfrescoperf
  db.pool.initial=100
  db.pool.max=170
  dir.root=/home/alf_data_perf

• Put required upload sample files into ./data/content, e.g. sample image or text files.
• Update benchmark classpath
  • vi ./config/classpath.config
  • Change ${TOMCAT_HOME}/webapps to /home/alfresco/repo
• Initialize the benchmark runtime paths
  • ant generate-classpath
• Adjust JVM memory and other arguments
  • vi ./build-jvm-args.xml
  • For Solaris, remove -Xss option
• Set up database
  • mysql -u root -p
  • create database alfrescoperf;
  • grant all on alfrescoperf.* to 'alfresco'@'localhost' identified by 'alfresco' with grant option;

Bootstrap the Repository

This optional step will allow the repository to be usable by the regular clients (CIFS, FTP and WebClient) during or after the benchmark tests.

• ant run-repo -Duser=admin -Dpwd=admin

The default configuration for benchmarking disables several non-essential services, with the exception of FTP.

• FTP to the localhost to check it out (this can be done during benchmarking as well). If this step is not performed, FTP will not be available once the benchmark initial structure is created.
  • ftp localhost
  • cd Alfresco

Create Initial Load Structure

The benchmark read and write tests require an initial data structure against which to execute the tests. A basic structure to start with has 5 levels of folders with a 6th level of 100,000 files.

• ant load-benchmark-data -DrepositoryProfile="10,0,10,0,10,0,10,0,10,0,0,1"

Check that the /home/alf_data_perf folder has been created. FTP into the server and you will see the files and folders being created.

The command to generate the initial structure for the JCR model is slightly different:

• ant load-jcr-benchmark-data -DrepositoryProfile="10,0,10,0,10,0,10,0,10,0,0,1"

In this case, FTP will not recognise the jcr nodes.

When you see the totals messages, you can kill the process:

The data has been loaded into folder workspace://SpacesStore/... :
 - Folder count = ...
 - Content count = ...
 - Total count = ...

Execute Concurrency Tests

• ant -Dconfig=./config/native-stduser-0512threads.xml

Bulk Load 1M documents

• ant -Dconfig=./config/native-bulk-0100per-04threads-1M.xml

Analyzing the results

Results can be found in subdirectories of ./data/output, in a file name summaryReport.html.

The test framework records the response times (the time for a thread to complete the task) for all individual operations.

Definitions:

• In the config file:
  • threads: The number of threads that will perform the operation(s)
  • iterations: The number of times a thread must execute the operation
  • delay: The base delay (in seconds) between executions as experienced by each thread
  • variation: The variation (in seconds) on the base delay such that thread wait time = delay +- variation
• In the results
  • Test Type: This is the type of test performed. A particular test may have threads perform different types of operations based on a percentage distribution.
  • Test Count: This is the total number of operations performed by the test for the given test type.
  • Total Duration (s): The sum of all individual response times, regardless of the number of threads called.
  • Average (seconds): The average time taken (in seconds) to complete the operation. This is is the average reponse time for the operation.
  • RPS (runs per second): This is the average number of operations that could, theoretically, have been completed by a single thread in a second.

For bulk load performance, the total server throughput is:

  Server Throughput = RPS x (Concurrent thread count) x (File and Folder count per iteration)

With the mixed tests, particularly tests with delays, it is more useful to consider the individual thread response times.

If you wish to measure the maximum throughput of the server for a given operation, then the simplest is to execute with a single operation type and remove all wait times between individual runs.

e.g. 100% createContent, 4 threads, 80 iterations, 0 delay, 0 variation

Benchmark Summary Report

Test Type	Test Count	Total Duration (Sec)	Average	RPS
createContent	320	52.595032324	0.1643594760125	6.084224799572537

The server processed 24 individual content creations per second and each individual operation took 0.16s on average. This doesn't necessarily denote a maximum throughput, which can only be found by increasing the number of concurrent threads until the response times is unacceptable.

How Many Files Are There?

You can count the nodes of a particular type with:

• select distinct(type_qname) from alf_node;
• select count(*) from alf_node where type_qname = 'some-type';

Sample Configurations

Definitions

Standard User Tests

• Mixed read/write tests with delays
• One operation per transaction

Concurrent User Tests

• Mixed read/write tests without delays
• One operation per transaction

Bulkload Tests

• Write tests only
• Create multiple files and/or folders per transaction

Standard User Test with 512 Threads

Config

• cat ./config/native-stduser-0512threads.xml

<?xml version="1.0" encoding="UTF-8"?>
<testSuite name="ParsingPerformance" xmlns="http://www.sun.com/japex/testSuite">
   <driver name="JAXPDriver">
       <param name="japex.driverClass" value="org.alfresco.benchmark.alfresco.AlfrescoInternalAPIDriver"/>
       <param name="description" value="Alfresco Internal API"/>
       <param name="japex.numberOfThreads" value="512"/>
       <param name="japex.threadRampupTime" value="5"/>
       <param name="japex.includeWarmupRun" value="false"/>
   </driver>

   <testCase name="AllOperations">
       <param name="alfresco.usageProfile.readContent" value="30"/>
       <param name="alfresco.usageProfile.readProperties" value="50"/>
       <param name="alfresco.usageProfile.createContent" value="15"/>
       <param name="alfresco.usageProfile.createFolder" value="5"/>
       <param name="alfresco.numberOfAvailableUsers" value="512"/>
       <param name="alfresco.loadDepth" value="6"/>
       <param name="japex.warmupIterations" value="0"/>
       <param name="japex.runIterations" value="10"/>
       <param name="japex.runDelay" value="15"/>
       <param name="japex.runDelayVariation" value="10"/>
   </testCase>
</testSuite>

Output

• cat ./data/output/native-stduser-0512threads_AllOperations_1164112303868/summaryReport.htm

Summary Report

Test Type	Test Count	Total Duration (Sec)	Average	RPS
readContent	1457	236.37968	0.16223725463280714	6.163812388611407
readProperties	2914	463.56798	0.15908304049416608	6.286025190954734
createContent	570	170.680193	0.2994389350877193	3.339579068790952
createFolder	179	76.100419	0.4251420055865922	2.352155248974384

Bulkload Test of 1 million Files with 4 Threads Loading 100 Files per Transaction

Configuration

• cat ./config/native-bulk-0100per-04threads-1M.xml

<?xml version="1.0" encoding="UTF-8"?>
<testSuite name="ParsingPerformance" xmlns="http://www.sun.com/japex/testSuite">
   <driver name="JAXPDriver">
       <param name="japex.driverClass" value="org.alfresco.benchmark.alfresco.AlfrescoBulkLoadDriver"/>
       <param name="description" value="Alfresco Internal API"/>
         <param name="japex.numberOfThreads" value="4"/>
         <param name="japex.threadRampupTime" value="0"/>
         <param name="japex.includeWarmupRun" value="false"/>
   </driver>

   <testCase name="Bulk">
       <param name="alfresco.numberOfAvailableUsers" value="50"/>
       <param name="alfresco.loadDepth" value="6"/>
       <param name="japex.warmupIterations" value="0"/>
         <param name="japex.runIterations" value="2500"/>
         <param name="alfresco.fileCount" value="100"/>
         <param name="alfresco.folderCount" value="0"/>
         <param name="japex.runDelay" value="0"/>
         <param name="japex.runDelayVariation" value="0"/>
   </testCase>
</testSuite>

Output

• cat ./data/output/native-bulk-0100per-04threads-1M_Bulk_1164068059553/summaryReport.htm

Summary Report

Test Type	Test Count	Total Duration (Sec)	Average	RPS
	10000	17791.364113	1.7791364112999999	0.5620704481391107

In this case, the total server throughput was 0.56*100*4=224 files/second