Views
WCM Test Suite
From alfrescowiki
|
Warning: AVM Deprecation
|
NOTE: This document describes features to be found in Alfresco v3.1 onwards.
Contents |
What is the WCM Test Suite?
With the 3.1 version of the product, Alfresco is committed to a new test harness for the WCM application. This test harness will emphasize user concurrency and will ensure performance of Alfresco in a variety of load-stress scenarios.
These tests will be executed inside of a test harness that is external to that of our nightly builds. This test harness will be operable in a standalone mode and will be provided freely to our community and enterprise users. That way, our stakeholders can verify performance and reliability of Alfresco WCM in their own environments.
Test Scenarios
Each test scenario defined below is meant to define a user's interactions with the Alfresco WCM application.
In the case of a test that models interactions with the JSF Client, the test scenario would describe click paths that the user takes, from start to finish. In the case of CIFS interaction, it would describe the exact file interactions, touches and operations against the CIFS drive.
These test scenarios are therefore meant to emulate a user and are meant to operate on a single thread. With many different scenarios established, it should be possible to launch many users at once, each performing their own scenario.
In this way, it will be possible to produce homogeneous tests and heterogeneous tests which provide as much variance in user activity as possible. Through sufficient variance, longevity of test and repeated trials, we will be able to minimize the risk to our end users of a WCM application fault.
The idea here would be to run different combinations of all of the following basic operations concurrently. This might be done via some kind of statistical model eg. "content creation via Web Form occurs 60% of the time, Submission to workflow occurs 20% of the time, Web Project Creation occurs 2% of the time, etc. etc.", while also varying the level of concurrency (2, 3, 5, 10, 20 concurrent operations) to try to tickle rarer (N-way) concurrency issues.
The following test scenarios are defined:
Common Interface Tests
CIFS Bulk Upload 1 (bulkcifs1)
- User opens a connection to Alfresco via CIFS
- User navigates to www/avm_webapps/ROOT
- User creates a folder "bulkcifs1_<user>"
- User bulk uploads 10,000 documents of various types and sizes into that folder
- User bulk uploads another 10,000 documents of various types and sizes spread out across a subdirectory structure of 100 folders to that folder
- User closes the connection
CIFS Bulk Modification (bulkcifs2)
- User opens a connection to Alfresco via CIFS
- User navigates to www/avm_webapps/ROOT
- User creates a folder "bulkcifs2_<user>1"
- User bulk uploads 5,000 documents of various types and sizes
- User deletes 2,500 of the documents
- User uploads once again half of the deleted documents
- User replaces 2,500 of the documents (1,250 from the original upload, 1,250 that were deleted and then re-uploaded) with new versions
- User creates a folder "bulkcifs2_<user>2"
- User moves 1,250 documents to "bulkcifs2_<user>2"
- User closes the connection
CIFS File Operations 1 (opscifs1)
- User opens a connection to Alfresco via CIFS
- User navigates to www/avm_webapps/ROOT
- User creates a folder "opscifs1_<user>1"
- User creates a folder "opscifs1_<user>2"
- User navigates to www/avm_webapps/ROOT/opscifs1_<user>1
- User bulk uploads 1,000 documents of various types and sizes
- User deletes 250 of the documents
- User moves 500 of the documents to "opscifs1_<user>2"
- User navigates to ../opscifs1_<user>2
- User renames 125 of the documents
- User navigates to www/avm_webapps/ROOT
- User removes folder "opscifs1_<user>1"
- User renames folder "opscifs1_<user>2" to "opscifs1_<user>1"
- User uploads 250 documents to "opscifs1_<user>1"
- User inplace edits the 250 documents, changing the text
- User closes the connection
FTP Bulk Upload 1 (bulkftp1)
- User opens a connection to Alfresco via FTP
- User navigates to www/avm_webapps/ROOT
- User creates a folder "bulkftp1_<user>"
- User bulk uploads 10,000 documents of various types and sizes to that folder
- User bulk uploads another 10,000 documents of various types and sizes spread out across a subdirectory structure of 100 folders to that folder
- User closes the connection
FTP Bulk Modification (bulkftp2)
- User opens a connection to Alfresco via FTP
- User navigates to www/avm_webapps/ROOT
- User creates a folder "bulkcifs2_<user>1"
- User bulk uploads 5,000 documents of various types and sizes
- User deletes 2,500 of the documents
- User uploads once again half of the deleted documents
- User replaces 2,500 of the documents (1,250 from the original upload, 1,250 that were deleted and then re-uploaded) with new versions
- User creates a folder "bulkcifs2_<user>2"
- User moves 1,250 documents to "bulkcifs2_<user>2"
- User closes the connection
JSF User Interface Tests
Promote to Staging 1 (pts1)
- User logs in to the Alfresco JSF Client
- User navigates to the web project
- User navigates to their sandbox
- User creates a folder
- User navigates to the folder
- User adds an image (.jpg)
- User adds an XML file (.xml)
- User adds a PDF file
- User navigates to their web project splash page
- User promotes the files to staging
- User logs off
Further Exploration for 3.1
Web Project Creation, Update and Delete
Sandbox Creation, Update and Delete
Workflow Submit
Deployment
- ASR vs FSR
- Single target vs Multi-target
- Multi-target within FSR
- Different concurrent deployment scenarios (multiple users concurrently deploying to different servers in the same Web Project, multiple users concurrently deploying to different servers in different Web Projects, multiple users concurrently deploying to the same servers in the same Web Project (should probably be prevented, but good to test anyway), etc. etc.)
- Test server vs live server deployment
Preview
- Virt Server based preview
- Preview via "Sandboxed" Web Scripts
Delivery Tier (ASR) Request Performance/Scalability
- Basically a question of SearchService (Lucene) query performance/scalability
- Would also be worth testing raw AVM asset retrieval performance. While the best design is to pull raw assets from a filesystem populated via an FSR rather than streaming them out of an ASR, there are cases where it may make sense to pull assets from within the ASR (ie. via a Web Script), so that needs to be performant and scalable too.
- Investigated and test by Brian Robinson who saw a 51% improvement for his test case when moving from 3.1 to 3.2, refer to:
