The purpose of operational self testing is to gather sufficient documented evidence to enable a conclusion and testimony whether or not the controls as documented are operating in practice.

There are two types of self testing:

1.    Supervisory testing: This involves the ongoing testing of controls through the ongoing documented detailed review of a control by a supervisor.

2.    Sample based self-testing. This involves the selection of samples (for each control tested) from the entire population of the particular control being tested, and the performance of specific test procedures on the selected sample

The purpose of this document is to provide detailed guidance to enable business people to perform Self-testing.

Supervisory testing is when a supervisor reviews in detail the operation of a control and records the results. For example a change control manager may review and approve in detail a monthly list of change requests. Provided the review is in sufficient detail, and the results of the review adequately recorded then there is no requirement for a separate sample based test of the control to determine operating effectiveness. In essence the test of operating effectiveness is 'embedded' in the day-to-day operation of the function and into the role and activities of the supervisor.

Controls appropriate for supervisory testing

Controls for which supervisory testing may be appropriate will involve the operation or performance of a control by one member of staff, and then require a detailed review and approval by a more senior member of staff.

Impact of samples on supervisory testing

Samples, and sample selection will not ordinarily be appropriate for those controls which are subject to supervisory testing as every instance of the operation of the control will be tested. However, management may want to consider the frequency with which a key control operates for reporting purposes. For example change requests may be performed on a daily basis. However, for reporting purposes management may determine that the key instance of change control is performed on a monthly basis, as the daily changes are prepared only for operational purposes, i.e. to enable the system to perform. In this situation, only the monthly operation of a control would be required, and therefore the change managers' detailed review would only be required for this instance of the control.

Evidence of supervisory testing

It is important that the supervisors test is as detailed as a sample based testing. It is also fundamental that evidence of the performance of supervisors test is prepared and maintained. This is because a lack of evidence of the supervisors test would prevent an external auditor from concluding whether the control had been tested. The evidence of a supervisors' test can be obtained through a suitable checklist.  For example the change request could be prepared by a process operative, the review checklist attached, and then the change request is reviewed and, the supervisory test checklist is completed and signed off by the manager.

Developing a supervisory test checklist

The checklist for supervisory testing would need to be in sufficient detail equivalent to that of a detailed test script.

A checklist should focus on those elements of a control that supervisor would actually review for.

Each control should be classified between those that are tested through Sample Based testing (SBT), or the those tested through management on-going supervisory testing (MST).

               Prepare test scripts against actual controls

Testing requires accurately documented controls that are tested to ensure conformance to a requirement and, therefore, compliance.

The PCI Audit, for example, requires Merchants to undergo an annual onsite review.  This audit (for the top tiers of merchant providers) consists of assessment and reporting conducted by a Qualified Security Assessor (QSA). Otherwise a merchant who processes a lesser amount of transactions will provide answers to the PCI certification questionnaire, with the scope of compliance validation focused on any system(s) or system component(s) related to authorization and settlement where cardholder data is stored, processed, or transmitted. The PCI security standards council provide a security audit procedures document or a self-assessment questionnaire. Reporting requires a completed PCI Report on Compliance (ROC) document or self assessment questionnaire with answers suitable for presentation to the PCI certification committee.

Each control documented as design effective can be tested, and therefore an actual test script is developed for each actual control. The development of a test script for each actual control should take into consideration the following 3 components:

A.      Direction of the test, and coverage of the full population

B.      Evidence that the control has been performed

C.      Evidence of the quality of the control that has been performed

Applying the 'ABC' approach forces an embedded level of quality in the test scripts. Each of these components is described in more detail below.

             'A' - Direction of the test and covers the full population

The main purpose of the 'A' component is to ensure the test starts from the correct point in the flow of the process and that the test has the right direction, and that it covers the full population.

            Test Direction

The test will either start from the initiating documents within a process such as purchase order / requisitions, for the Purchasing process or the test starts from the end of the process, i.e. the records in the accounting system. This flow of the test is determined by the assertions that need to be addressed.

For  Sarbanes Oxley, assertions are claims that management make when they produce the financial statements. That is, once financial statements have been produced, management make the following claims:

• That the financial statements are complete
• That all assets / liabilities included in the financial statements actually do exist
•  That all transactions recorded actually did occur
• That the transaction values are measured accurately
• That the assets and liabilities are valued accurately
• That the information in the accounts is presented and disclosed as required
•  That the company has rights over the reported assets and that it has obligations to pay off the reported liabilities

The words highlighted above are the assertions that need to be tested via Self-testing for Sarbanes Oxley.

For other compliance efforts like PCI/DSS or ISO 27001/2 then the assertions will be different but they will exist and you should list them.  The general rule is that all assertions other than 'completeness' require the testing to be undertaken from the system back to source documents; 'completeness' on the other hand requires testing to be undertaken from source documents to the system. Where there are multiple assertions, i.e. completeness, and say existence and occurrence, the test direction and sample selection should be based on the 'completeness' assertion. However, the test script should involve checking that the details on the originating document agree to the system records to cover off the 'existence or occurrence' assertion. For example, relating to the 'existence or occurrence' assertion, 'All updates to master data for SAP are requested on standard forms that are reviewed and approved by a delegated authority who does not have access to update SAP Master Data'.

The population is all updates that are in the system, and evidence should then be obtained that those updates in the system have actually been made as the result of a properly completed and appropriately authorised form. Hence, here the testing is from the system back to source documents.

Alternatively;  'A checklist of leavers is prepared, confirmed with appropriate personnel where necessary, e.g., HR, to confirm that they have notified the appropriate person of staff leavers. The signed-off checklist is reviewed and approved by a delegated authority within Information Security'. This control also covers off the assertion of 'completeness' as well as 'existence and occurrence'.

The 'completeness' assertion means that a leavers checklist should have been prepared even if there was not a leaver to record. As 'completeness' is a more difficult assertion to test than the other assertions the 'completeness' assertion must take priority in the determination of source of the test and the test direction. Therefore, assuming the leavers' checklist is prepared on a monthly basis then the sample should be based on selecting months, and then ensuring that the checklist had been prepared and approved, and that the leavers as detailed on the checklist are actually dealt with (existence or occurrence) accurately. Hence, here the testing goes from the source documents to the system.

            Completeness of the Full Population

Consideration should be given to ensure that tests cover the full population.  For example, 'All orders are assigned a unique number by the system.  A review is performed to ensure that all orders received are entered in the system with a unique number Here the assertion is 'completeness' and therefore testing will start from source  i.e., orders. When performing testing and choosing a sample, consideration must be given to the possibility that orders could potentially not be processed at all. Therefore assessors should give consideration to obtaining full coverage of the population by making enquiries of the relevant people to ensure that they (assessors) understand the entire population, i.e. they understand the entire manner in which orders can be received, e.g. by web, e-mail, in the post etc. This enquiry could be supplemented by observation within the orders department of the receipt of orders on a particular day, together with inspection of the date that orders were actually received compared to the date that they were actually processed. The overall aim of this is to evidence that the population from which samples are to be picked is a complete population.

             'B'- Evidence that the control has been performed

Once the sample has been selected from the complete population, evidence must be obtained that the control has been performed. For example, for a manual authorization control this evidence will be the signature of the person who performs that control.

Documented evidence must be obtained to ascertain that the control has been performed as it has been designed. For manual controls; the evidence that the control has been performed should be available through physical records created when these controls have operated. 

For system controls, the evidence of the control will be obtained through obtaining appropriate reports and screen shots to prove that the system configuration, system access, and system reports are as documented within the design.

             'C' - Evidence of the quality of the control has been performed

System controls, once established either work, or they do not. Evidence gathered to prove that a system control operated also proves that the control operated consistently and effectively. Therefore system embedded application controls would not have a 'C' element.

Manual controls however, are subject to human error, and therefore we should test the quality of the control to gain assurance that the control has operated consistently and effectively. For example a signature on a User Access request does not necessarily mean that the person has carefully reviewed it. The signature itself does not provide sufficient evidence that the control has operated as intended; therefore we also need to test that the control has been performed correctly.

This would involve selecting a sample of the user access request process that is being tested (sampling guidance is provided below for 'sample within a sample') and inspecting that the details on user access requests followed the process, so as to provide after the fact evidence that the individual carefully reviewed the user access request before approving it and was authorised to do so.

In some circumstances it may be appropriate to re-perform the control. For example in the testing of a manual depreciation calculation it will be necessary to recalculate the accuracy of the depreciation calculation, i.e. to undertake the same activity as the individual who performed the control.

Another example would be the review and approval of an exception report. The signature in itself does not provide evidence that the items on the report were carefully reviewed. Therefore a sample of the items on the exception report should be followed up to provide the evidence that the control has been performed properly. This would involve discussion with the person who reviewed the report, an inspection of any annotations on the report as to why items were accepted, or corrected, and a follow up for any adjustments.

Test scripts should be made as specific as possible, for example:

•  If the standard form (A123) is used to originate an update to master data, then the test wording must include that form A123;
•  If an identified individual/function reviews form A123, then the test script wording must include that that individual/function has signed off form A123;

All tests scripts should include the test method: i.e. enquire, observe, inspect, re-performance but it is not necessary to have all four test methods for each control.

All tests scripts should be phrased in the style of a closed question.

Sample Sizes

The size of the sample is dependent on control frequency and the type of control.

The table below is the minimum sample sizes for the different control types and control frequencies that must be achieved in order to conclude on the operating effectiveness of the control. The samples that are used must be from the year of assessment, i.e. tests of controls relating to 2008 cannot be used to conclude on the effectiveness of controls in 2009.

The above sample sizes for 'system' controls (application embedded controls) assume that IT general controls are operating effectively  i.e. on the basis of a sample of 1, it can be confirmed that the application embedded control has been set up correctly and will continue to do so assuming IT general controls operate effectively.  However, where IT general controls are not found to be operating effectively, a report of changes to the settings to the relevant application embedded controls should be obtained to determine the nature of any changes made to them.

              Remediated Controls

In order to conclude a remediated control is operating effectively, the full sample size must be tested. For example, suppose a monthly control was remediated in March. It would require a further month of operation to have the required sample size to test (March and April). Therefore testing may not realistically be able to be performed on the control until May.

Where a remediated control is tested, it should be retested in full, and not just the elements of the control that were found to be deficient in previous testing. For example, suppose a control fails a test on one of five elements, the subsequent test of the control must test all five elements not just the element that failed. However, care should be taken where controls have been combined to ensure that if one control fails, only the control that was found not to be operating effectively is retested.

No Transaction

If there is no transaction in the year, then the control cannot be tested. In these circumstances, then the control should be assessed as 'not tested'.

Multi-location Environment

In certain circumstances, controls for example, over application software such as role reviews, where the significant controls upon which reliance is placed are operated in a number of locations, e.g. at different operating units. In such situations consider whether or not there are multiple populations that will need to be tested. Where the control operated at each operating unit is the same using a common process and system, then the population of the controls operating individually can be considered a homogenous population, and therefore sample selection can be based on the entire population. For example, changes to access rights, can be performed frequently at different operating units. The control is exercised using a common procedure and system, and therefore can be considered as homogenous. Therefore a selection of units can be visited and samples selected at each unit to ensure the required sample size is met. (E.g. 5 units (based on the table below), with each testing a sample of 5 changes, (making up the minimum sample size of 25 for the control type and frequency) If half of the units were on one system, using one procedure, and the other half were on another system using a different procedure, then there are potentially two homogenous populations, each of which would need to be tested using two full samples of 25 each.

Non routine / Per Transaction controls

Some controls operate on a per transaction basis, e.g. opening an account for a new starter.

For 'Per Transactions' the total population in the year should be considered. For example, if there was 1 instance, in the first half of the year,  and then 55 in the second half of the year, the total population that existed would be 56. The table in below can be used to determine the overall sample size in the year, i.e. the sample would be 5.

Sample within a Sample

For the testing of some controls such as testing of manual data entry, a sample of data would be selected from the system and then traced to the supporting manual data list (element 'A'). If the manual entry control operated many times a day, then the sample size would be 25. Testing would involve inspection for the authorised signature (element 'B'). Testing would also involve inspecting the details on the list. The number of individual data items entered could be significant, and therefore the table below is used to determine the extent of the 'C' element. For example the first list tested contained 250 items, then 20 items should be selected to agree to. If the second journal tested contained 30 line items then 3 line items should be agreed to.

Multiple Instances

Some controls such as user review within application software management may involve a number of instances, i.e. there are several applications that review users on a monthly basis. For example, suppose that there are 5 applications, and each reviews a user list monthly. The total population of reviews would be 60, and therefore based on the table below, 5 user reviews should be tested.

Extension of Sample Sizes

In the case of either a daily control or a control that operates many times a day, where only one (i.e. a single) exception is discovered, then the sample can be extended to 40 for a daily control and 50 for a many times a day control. If no further exceptions are found, then the control can be deemed to be operating effectively. In no other instance of control frequency (i.e. 'weekly' through to 'annually') can the sample sizes be extended.  This single exception can be for more than one element in a test for a single control.

Sample Selection Method

There are different sampling methods to choose between depending on the characteristics of the population. The possible options have been explained as follows:

Regular Interval

If the population has no bias i.e. there are no differences in the characteristics of the population and there are a large number of transactions (>200) then regular interval sample selection is preferable. This would involve identifying the total population size and select every 'n'th item to spread the test across the population.

For example, the total number of user sessions in the year might be 25,000. If the sample size is 25, the sample selected for testing can be every 1000^th (i.e. 25,000 / 25) user sessions starting from a random point in the first 1000 items of the population. This starting point can be selected by using a random number generator (e.g. using the =RAND() function in excel).

While this is the preferred method when there is a large population (>200), for practical reasons it may be easier to apply random sampling.

Random

This method can be used if the population has no bias i.e. there are no differences in the characteristics of the population. This sampling technique involves selecting the sample using either a random number generator, or by manually selecting items at random. Either way is acceptable as long as the tester is being objective; and hence testers should determine the most pragmatic approach in any instance. For example, the total number of user reviews prepared in the year may be 12, and therefore 2 months could be selected at random (i.e. any two months in the year) rather than using a random number generator.

This method should be applied when there is smaller population from which to select samples (<200).

Judgemental Sampling

This involves actively selecting samples to test, in combination with the other methods, so as to ensure appropriate focus in testing a particular population. In the example above of multiple instances of the user reviews, covering 5 different applications, the appropriate sample size is 5. If 2 applications handled the vast majority of users, then focus should be given to those applications by selecting 2 months (sample size for a monthly control) for each of the two main applications at random, and then 1 at random for any of the other applications.

This method should only be used when there are non-routine transactions, or the population from which samples are selected have a significantly different volume.

Key controls are those controls identified in a risk assessment as fundamental and where if not delivered the risk assessment would deteriorate; therefore these controls are tested with sufficient and appropriate evidence being obtained for each. Testing is time consuming and therefore we need to try as much as possible to combine or group our tests. The purpose of combining tests is to aid efficient execution of the test work. This is achieved by using a single sample to test multiple controls.

When we are looking at control combinations we are considering whether the test scripts related to the controls can be combined together such that the time consuming activities of selecting a sample and picking sample documentation are performed a minimal number of times. However, before identifying control combinations it is important to have the actual test scripts for each control fully documented using the A-B-C approach. Following this, control combinations can be identified by addressing the following two issues / questions:

• Do the test scripts that we are trying to combine use the same sample source?
• Do the controls whose tests we are trying to combine form part of a single process flow.

If the answer to either of the above questions is yes, a control combination is possible. The above logic has been implemented and illustrated in version 2 of the generic test scripts. These test scripts should be used along with the guidance in this section to assess if control combinations are possible with actual controls. For illustration purposes, an example of a control combination has been provided as follows:

The test workbook and associated test scripts should be reviewed by the control owner during the initial self-testing effort to confirm that the approach will enable the tester to conclude that the control is operating effectively and demonstrate compliance to the scope. The reviewer should also highlight areas where tests could be more efficient and discuss with the control owners any controls that do not appear to fully mitigate the risk..

This QA function is also a service offering from Compliancetutorial.com.

Management needs to be able to place reliance on the work performed and therefore must consider the objectivity and the competence of the people to perform the testing. This will involve assessing the ability of the people performing the testing to make objective, unbiased and accurate assessments of the operating effectiveness of the controls.

The fundamental rule of objectivity is that the person who performs the testing must not be the person who has responsibility for the operation or ownership of the control. For example, the Accounts Payable Manager, or any of his / her team should not be involved in performing the testing of the operating effectiveness over the ERP Accounts Payable processes. However, they (Accounts Payable Manager and his/her team) can and should provide input to towards the development of the test scripts.

There are four test methods - Enquiry, Observation, Inspection and Re-performance. The test methods enable the tester to gather evidence on the operating effectiveness of risk controls.

Enquiry:

Ask the control owner about the operation of control and use example scenarios to enquire how the control owner behaves, in order to establish control effectiveness. If enquiry is used, then it should be confirmed with enquiry of other individuals.

Observation:

Watch the control owner performing the control.

Inspection:

Check documents that evidence the performance of a control, e.g. inspect a check exists together with supporting documentation as necessary.

Re-performance:

Redo the activities involved in execution of the control, e.g. this may involve checking the items of a check back to supporting documentation, and obtaining supporting documentation.

Enquiry and Observation alone, or in conjunction with each other do not provide sufficient evidence on the operating effectiveness of a control and should be complemented by evidence gathered from Re-performance or Inspection tests.  This is because the testing process must be capable of being audited, and therefore requires a high standard of physical evidence.

Specific considerations when performing testing

Apart from the areas covered above, the following independent issues need to be considered when planning and performing Self-testing:

• The test scripts should create a detailed set of instructions that must be completed in full for each and every sample item. If a particular sample item selected cannot be found, then there is an exception. For example, a user record has been selected, but cannot be found, then there is an exception. 
•  It may be that each and every element of the control being tested has an exception. In these circumstances, all parts of the control are deficient and testing further samples will not change the conclusion. For example, if a control has 4 elements, and after testing 5 sample items (out of 25) it can be concluded that 3 of the 4 elements are not working, testing execution should continue to determine if the 4th element is effective. If with sample item 9 (out of 25), the 4th element is found to not work, then all elements of the control require remediation, and the remainder of the sample items should not be tested as further testing will not change the conclusion that all elements of the control require remediation. Where this line of reasoning is followed to stop the testing, there must be clear documentation of this reasoning in the testing work paper. As described above, once remediation has been completed on the elements that failed, all elements of the control will need to be retested using the full sample size.
• The sample sizes for 'system controls' (embedded application controls) are dependent upon the effective operation of the IT general controls. Therefore, the IT general controls should be tested to ensure that they are operating effectively before the testing of the embedded application controls.

A generic test workpaper is available from compliancetutorial.com

Samples should be referenced and retained. Where exceptions are found during testing, documentation related to these should be retained in hardcopy in the testing file and if necessary, used to agree a remediation action with the control owner

A conclusion on a test of risk control requires judgement. While all individual elements of the control tested are either effective, or they are not, consideration will need to be given to each of the elements individually, and when taken as a whole, based on judgement and experience to determine whether or not a control is operating effectively.

Therefore consideration should be given to which element of the control has been found to be ineffective. If a signature for the performance of a control is not found ('B' element), then this does not necessarily mean that the control is ineffective, provided that there is sufficient evidence as to the quality of the control ('C' element). This situation is most likely to apply to detect control where the review is after the fact, and the signature required after the operation of a control, for example the approval of the change committee. For example, if the report was satisfactory, but no signature was found , then it could be concluded that overall the control was effective.

In some circumstances it may be the signature ('B' element), forms the only real evidence as to the operation of a control. For example, in the review of an edit report to updates to the master file, it may be there are few updates, all of which can be traced to supporting documentation by the tester. The updates were all performed correctly and appeared on the edit report, and so there was no other evidence of the review by the control owner as no individual item on the edit report update required correction i.e. no ('C' element). Without the signature of the control operator, it cannot be determined whether the control actually took place, as it may have just been chance that no items that would have required correction actually occurred. In these circumstances it should be concluded the control was ineffective.

When a control requires specific approval as part of a prevent control, for example the approval of a role change before sending it to the security administration, then the absence of a signature will indicate that the control has not operated effectively.

If it is concluded that a control in a single instance is not effective, then the respective control has failed, and a remediation action should be developed and agreed with the control owner. However, if for a daily control, or a control that operates many times a day, and a single exception has been discovered, then the sample can be extended to 40 for a daily control and 50 for a many times a day control, and if no further exception is found in the increased sample, then the control can be deemed to be operating effectively. 

If any control is to be deemed design effective, it should be tested, using the minimum sample sizes above and be working for the minimum amount of time required. For example, a monthly control must work in November and December, and be tested before the control can be classified as effective as at 31^st December. If the control is not operating effectively at the as at date, i.e. 31 December, then the control is still ineffective.

Management will need to take responsibility for testing, such that the individuals who sign-off on the operating effectiveness of the controls are confident of the results of testing. In order to allow for this reliance to be placed on the testing work, management will need to take responsibility for the co-ordination and review of the testing.

It is important that an appropriate review process is put in place to ensure that the testing has been properly performed, accurately documented, and the results and assessment are in accordance with the prescribed testing methodology. In order to achieve this, the review process should be performed by a more experienced member of staff than the individual performing the testing and should be evidenced by a sign-off on every working paper.

The review approach should not be limited to an after-the-fact review of the working paper, but should also utilise other techniques such de-briefing the staff who executed the testing and include a review in detail through one or more of the items tested with the individuals that performed the testing.

This review is a service provided by compliancetutorial.com