This paper details part of a strategy suitable for most regulatory compliance programmes such as PCI/DSS accreditation, SOX 404 certification, the various EU statutory audit directives and also for enterprise architecture development and security and governance standards like ISO 27001/2, ITIL, CobIT. These standards or legislations provide a scope for self testing.

To help embed a culture and practice of compliance to a standard, to reinforce accountability and to make compliance sustainable, organisations can implement self-testing as a function of their business processes.

Self-testing is performed by implementation teams, IT support groups, control owners etc.  Self testing enables the organisation to prepare for the assessment from an external auditor, collect the evidence, remediate and embed new working practises (and the collection of evidence) into the business process and, critically, reduce the time and cost it takes to get and stay certified.

Self testing is preceded by some sort of documentation of the current processes and a risk assessment that identifies key controls. Key controls are those controls that have the most weight on any judgement of risk mitigation. This state is the current governance architecture design.

A walkthrough of a business process and the risk controls within it can help evaluate its design effectiveness for compliance. Performing a walkthrough of the relevant functions or transactions and tracing them all the way through the complete process, from instigation, through authorisation, recording, processing and reporting will assist with the identification or existence of control activities to establish whether control activities are being performed (i.e. are in place), appraisal of the design of the risk controls, as well as substantiating the accuracy of process documentation.

A walkthrough is a end to end evaluation, step-by-step of a process and its controls to verify and validate understanding on the operation of the process and its associated controls and to evaluate whether the actual controls, if operated as designed can effectively mitigate risk to an acceptable level.

This is not a test of whether the control is operating effectively, which is reviewed during operational self-testing.  This distinction is illustrated in the below example.

A control can often consist of a reconciliation process. This 'reconciliation' can be technical, for example, a process looking at incidents from an intrusion detection system or a financial reconciliation within a core financial accounting business process - Reconciliations are performed by an employee, and are reviewed & approved by a supervisor. Actions to clear reconciling items are initiated within 30 days of completing the reconciliation.

In conducting the walkthrough it would be ensured sufficient evidence exists that reconciliations are being prepared by the nominated personnel (i.e. a reconciliation statement together with documentary evidence of the balance, and documentation intended to explain/justify/evidence clearance of 'reconciling items') and that these are being reviewed (i.e. supervisor's signature).  Where there is such evidence it can be concluded that the control has been placed in operation and (assuming that it is properly mitigating the related risk) considered 'design effective'.

However, a supervisor's signature on a bank reconciliation statement does not necessarily mean that the person has carefully reviewed it.  The signature itself does not provide sufficient evidence that the control has been operated as intended.  It is therefore necessary to test whether the control has operated effectively, which is conducted in self-testing.  In doing so, it would be necessary to confirm that the documentation supporting a reconciliation meaningfully demonstrated confirmation of the reported balance and evidenced the prompt/appropriate clearance of reconciling items.  The self-test would therefore entail a more detailed examination of the relevant documentation to confirm, in the case of the above example,  that the supervisor's signature on the bank reconciliation evidenced that the control was operating as intended (i.e. effectively) in a sample.

In order to carry out a definitive walkthrough the following should be utilised during a walkthrough, this list is by no mean exhaustive and should be supplemented with additional materials if appropriate:

• Process Diagrams/Narratives
• Policies
• Procedures
• Risk Assessments
• Recent audit findings and recommendations

Inquiries should include requesting owners to illustrate the control activities performed. Such descriptions would commonly comprise the following:

•  Accurate explanation of the stages involved in performing the control activity, as well as whether the control is automated or manual or a combination
• Reports and other information used, in addition how such information is derived and employed.
• What the owner is seeking to determine if there is an error and what kinds of errors have been recognised. In addition, whether errors are automatically recognised by application software.
• Procedures employed when an error is recognised and how the error was resolved.
• Procedures employed when the owner is not in attendance or is absent.
• Procedures employed in connection with unusual dealings.
• Amendments to the control activities during the period under review.
• Changes in personnel who perform the control activities.
• Whether the owner has ever been requested to override or ignore the process or controls.
•  If other people have performed activities or made entries, and if so, to describe the circumstances, why it transpired and what the outcome was.

The walkthrough should include both the manual and automated steps of the process, from the point at which the major process steps are initiated to the end of the recording process.  A walkthrough comprises the following steps:

• Observe the process in operation and confirm understanding through inquiry.  Question personnel at each point where important processing controls or procedures are prescribed.
• Confirm flow of information and transactions.  Follow the flow of data and file information through the automated processes; identify what happens at each stage. Use the documents (inputs and outputs of each stage of the process) that are typical of the process being reviewed, providing an initial test of the evidence of the control being operated.
• Examine, gather and file documentation.  Review the quality of evidence obtain the 'show me' test for the operation of controls, is it adequate? Is it complete? Is it readily retrievable if required?
• Confirm understanding and operation of controls.  If the walkthrough does not confirm the preliminary understanding of controls, the process should be re-assessed and discussed with the Process Owner.  Further walkthroughs are carried out until full understanding is achieved.
• If possible the walkthrough ought to be a forward walkthrough of a process. If this were not possible (complex, time consuming) then walking back through an historic transaction (process) would be adequate.

Evaluation of design effectiveness is critical because only properly designed controls are capable of operating effectively.  A control deficiency exists when the design or operation of a control, or group of controls, does not allow management or employees, in the normal course of performing their assigned roles and responsibilities, to prevent or detect failures on a timely basis.

The control should be properly constructed to achieve the related control objectives.  Ask if the control is used as directed, will it accomplish its objective? If the control is deemed deficient, what are its specific shortcomings?

In evaluating control design you need to:

• Determine whether the controls that you have documented for the process meet ALL of the relevant control objectives
• Whether there is an appropriate balance of preventative and detective controls
• Whether there is an appropriate balance of manual and automated controls
• Whether individual controls meet the requirements of PCI/DSS/SOX/ISO 27001 and so on i.e. are they sufficiently well evidenced?

To determine whether there is an overall deficiency in the design of the actual control(s) versus the risk, consider:

• Evidence and facts gathered
• Segregation of duties
• Combination of controls
• Continuity of controls
• Nature/resolution of exceptions
• Policies and Procedures
•  Appropriately designed to detect errors of importance
•  Is there a history of detecting/preventing errors

The walkthrough should be documented, including evidence of the assessment made, when the walkthrough was carried out, and who did it.  This enables an assessment of the reliability of the the walkthrough to be made, e.g. walkthrough testing done in January is less reliable for an assessment at end-December than a walkthrough done in November; equally, the independence and knowledge of the reviewer doing the walkthrough could mean that management's reliance on the results of the walkthrough may be increased. The results of the walkthrough should at a minimum contain the following:

•  Evidence of the assessment made
• Conclusions of the walkthrough
• Details of any fix required
• Date the walkthrough was carried out
• Type of Walkthrough (i.e. forward, historical)
• Who performed the walkthrough
• Location of the walkthrough

If the walkthrough provides confirmation of the process and controls, as documented or discussed, mitigates the risk and meets the assertion required by the regulation, then it follows that remedial action is not required at this stage. Otherwise remediation would be undertaken and further walkthroughs performed until rectified.

The following classification of the reasons for (i.e. the root cause) for deficiencies can be used (combined for both design effectiveness and operating effectiveness).  This classification passes no judgement on either severity of a deficiency or its projected remediation effort.

DE = design effectiveness

OE = operating effectiveness

Where deficiencies are identified in the design of the controls, due to missing controls or ineffective controls, an action plan to correct the control deficiency should be made and followed up. 

In this respect, a remediation action is considered as closed when walkthrough is reperformed and recorded

Timing of changes to controls

When considering changes to the control structure, due to remediation work or for business reasons, e.g. acquisitions, the requirements of the required accreditation must be taken into account.  Consider:

Frequency of control operation

some controls are operated infrequently, e.g. quarterly.  If testing of the new controls can only take place once before the assessment is made, the level of assurance gained will not be high and a qualified report is more likely depending upon the significance of the controls.  

New IT systems

when new IT systems are being planned, adequate resourcing should be made available to ensure that the documentation of the new processes and controls is made, prior to the system being implemented so that the control design can be assessed as effective.  Timing of system implementations should ensure that there is sufficient time to adequately test the new controls and to gather adequate evidence of their operation, such that compliance can be demonstrated.