Personal Note: It’s a great opportunity for a QA/Tester to be a CMMI Appraisal Team Member (ATM). ATM gets extensive exposure of a complete CMMI model. If your organization is going for appraisal and you get an opportunity to be an ATM, grab it. It’s an achievement you will like to mention in your resume.
To Readers: Appraisal is complex and time taking process. I have tried to describe it as simple as possible. This article will give you brief idea about CMMI Appraisal. It’s not about CMMI model but it will tell what happens during appraisal! There may be understanding problems initially (If you are not familiar with CMMI terminology) until you arrive at an example below, Please keep reading until then.
Also read => CMM levels for QA Processes
What is CMMI Appraisal?
An appraisal is an activity that helps us to identify strengths and weaknesses of our organization’s processes and verify how closely the processes relate to CMMI best practices.
Flash Back: When any org decides to go for a maturity level appraisal (Commonly known as CMMI Appraisal) they contact an authorized organization from Software Engineering Institute (SEI) Partner network to do the appraisal. Authorized agency will appoint consultant to help/guide org to understand CMMI, creating documents and organization policies etc. This is a long process. It starts 1 year or 6-7 months (Depends maturity level of organizational process) before the final appraisal. After preparation, LA (Lead Appraiser) will be invited to do the appraisal.
What You Will Learn:
Preparation at organization level:
Before CMMI implementation, organization has to:
- Identify staff to attend Intro to CMMI course
- Create SEPT or SEPG (Software Engineering Process Team/Group) (Responsibility of SEPT/SEPG explained at the end of this article.)
- Create organization policies.
- Employ CMMI consultants
ATM Journey: After Intro to CMMI course you will have Appraisal Team Member (ATM) training. LA provides training of “SCAMPI Appraisal Method Training” to ATM. After training, an exam would be held and you have to pass. A reference document called Method Definition Document (MDD) will be provided to all the ATMs, this document describes the SCAMPI. (Topics of Intro to CMMI Training course is covered at the end of this article)
Now what is “SCAMPI”, who is “LA” and what ATM does:
SCAMPI: Standard CMMI Appraisal Method for Process Improvement.
Lead Appraiser (LA): Software Engineering Institute (SEI) Authorized SCAMPI Lead Appraiser. (Commonly known as LA. The LA announces the final findings report and signs the Maturity Level certificate. The word certificate isn’t generally used; SEI discourages the word “certification” and instead it refers to a company as having “achieved CMMI Level 5”).
What ATM does?
ATMs support the Lead Appraisers in collecting and validating evidences for Document Review, Interview of FAR (Functional Area Representative) members and preparation of other important artifacts. (More about ATM responsibility mentioned at the end of this article.)
Little more about CMMI: Capability Maturity Model Integration is a process improvement model developed by the Software Engineering Institute (SEI), Carnegie Mellon University.
CMMI is comprised of a set of “Process Areas.” Each Process Area is intended to be adapted to the culture and behaviors of your own company.
There are 22 Process Area in CMMI.
Within the Process Areas in the CMMI, there are multiple “Specific Goals (SGs)” and Specific Practices (SPs).” These practices define the expected behaviors of projects and organizations.
Image: Process Area components
SG: A specific goal describes the unique characteristics that must be present to satisfy the process area.
GG: Generic goals are called “generic” because the same goal statement applies to multiple process areas.
Every ATM will be responsible for Process Area allocated by LA to him/her. Suppose there are 22 ATMs, every ATM will have 1 process areas of all the projects selected for Appraisal. Generally LA forms small group of ATMs. Suppose 20 ATMS are there LA will create 10 groups (each group with two ATMs) and each group will be responsible for a few PA of all the projects.
Let me give an example to make you understand Specific Goal and Specific Practices and what is ATM’s role in appraisal process.
Take example of Process Area, Validation. (keeping in mind most of the reader here are Tester/QA)
Validation (VAL) is an Engineering process area at Maturity Level 3.
Purpose of this Process Area: The purpose of Validation is to demonstrate that a product or product component fulfills its intended use when placed in its intended environment.
There are two SG (Specific Goals) in this Process Area.
- SG 1 Prepare for Validation
- SG 2 Validate Product or Product Components
Let’s see Specific Practices by Goal. Specific Practices in SG1 mentioned below.
SG 1 Prepare for Validation
- SP 1.1 Select products and product components to be validated and validation methods to be used.
- SP 1.2 Establish and maintain the environment needed to support validation.
- SP 1.3 Establish and maintain procedures and criteria for validation.
It is not possible to elaborate all Specific Practices here; I will consider SP1.1 to give you an idea of appraisal process.
- SP 1.1 “Select products and product components to be validated and validation methods to be used”: STP (Software Test Plan) is the most suitable artifact for this SP. All Features/Component/Capability needs to be tested and the method is captured in this document.
- First ATM will verify if the STP is approved.
- Approval mail or any supporting artifact for approval is checked for. E.g. affirmation from approver in MoM. If it’s in MoM, approver presence in that meeting will be checked in attendance sheet on that particular date for this meeting.
- Review plan present in the PMP Annexure? Was Review done? Who reviews? Review date! Whether review points are captured and closed in review log? Closure date?
- If the review is conducted, there should be review-checklist.
- Was STP baseline? All review points were closed before the baseline! What is baseline date?
- Whether validation method mentioned in the STP?
- What is reference document for preparing STP? Requirement capture document (RCD) or Software Requirement Specification (SRS) or any other reference doc; suppose RCD is reference document, ATM checks Bi-Directional Traceability Matrix (BTM) for proper mapping of RCD to STP. All requirements should be covered properly in STP for validation.
- Guideline for preparing STP present? Does organization have standard template for STP? What is version number of STP? (Guideline and template creation comes under different PA; I have included this point just for reference.)
Verifying Date plays important role! Any document will be baseline after review, so baseline date will come after review date. Similarly date in checklist, review date, review point closure date will be compared; date on respective documents should appear in correct sequence.
I now hope you must be having some clarity on ATM responsibility. Cross checking all documents prepared during Software Development Life Cycle, dependency on each other followed by cross verification, covers the core responsibility of an ATM. Other than this ATM participates in the interview of FAR members. I shall cover this topic in future articles.
Now few more examples:
These are generic examples to give more insight on appraisal process.
Let’s see a scenario: – While going through a weekly status meeting MoM (Minutes of meeting) of a particular date ATM ensures on verifying two points.
- Risk discussed in the team meeting.
- One new member in the team and PM ensuring on subsequent training.
1) Risk : Risks are covered in the PA called Risk Management (RSKM).
ATM will check if this risk is mentioned in Risk Log? If present in the Risk Log, there will be Risk Mitigation Plan or Contingency Plan? If Mitigation/ Contingency Plan are present, there may be some action item against the Plan. Now if Action item is present, would Monitoring be needed? Monitoring comes under Project Monitoring and Control (PMC). ATM must have an idea as to what to check and where to check.
This example shows how process areas are connected to each other and how many artifacts ATM will go through to make sure all practices are followed properly.
All training related activity comes under Process Area called Organizational Training (OT).
Project Manager sends a request to HR department regarding training. HR will find an Internal/External trainer for the resource and conduct training. This is the general scenario, what else happens in OT?
Let’s see how OT works, suppose this financial year organization has allocated XX amount of money for training. Now HR will identify training needs, will collect the data from managers, resources and will prioritize it, along with this HR will also identify organization’s future needs. For instance, if organization decides to start working on communication projects along with software, either HR will hire new Resource/Managers/TL or will identify within organization to work on this new domain.
ATM Verification Areas:
- Identification of Training needs by HR
- Identification of Trainee
- Identification of Trainer
- Intimation to the manager and trainee to block their calendar and subsequent affirmation in advance.
- Date of request to Admin and IT team for Infrastructure (Training room, Laptop, Software, food, vehicle etc.)
- Attendance sheet maintained?
- After training feedback was taken from trainee?
- HR updated Org Skill matrix after training?
- If some training is pending from previous year, a plan is made to execute the same in current year and reason for HR not conducting the training last year. Etc.
For sure this is not complete list of activity for Organizational Training; other activities are also there, these are just quotes from my memory.
- Responsibility of SEPT/SEPG (Software Engineering Process Team/Group)
- Study organization processes
- Write process standards
- Develop SDLC stages and details
- Create meeting procedures
- Create estimation procedures
- Create planning procedures
- Create Quality Assurance procedures
- Create Configuration Management Procedures
- Intro to CMMI Training course covers following topics:-
- Introduction to the Model
- Model-based process improvement
- Overview of CMMI components
- Institutionalization of Processes across the Organization
- Process areas of CMMI Model
- Structure for the continuous and staged representations
- ATM responsibility
- Each ATM is responsible for understanding the CMMI model, understanding the organization context, and understanding and applying the SCAMPI methodology to evaluate implementation.
- Each ATM is responsible for participating in data review, participating in interviews (e.g. identifying interview questions, taking notes, asking questions, tagging the answers to the model practices), participating in all group discussions and consensus and participating in creating the final findings report.
- Each Appraisal Team Member will review and evaluate objective evidence of practice implementation and participate with the other Appraisal Team Members (including the Lead) in coming to consensus on how adequately practices are implemented at the organizational level and whether or not goals, practices, and maturity levels are satisfied.
Appraisal Validity: The findings and results from a SCAMPI A Appraisal are valid for a period of not more than three years from delivery of the appraisal findings. That means after every three years organization have to go for appraisal.
Now see the difference in a Tester’s activity from Pre CMMI to Post CMMI.
- Pre CMMI: Dev team sends a mail “All defects closed, Defect sheet attached” and requests for testing.
- Post CMMI:
- Tester checks for code baseline followed by upload in repository.
- CR raised for code change and accepted by SEPT/SEPG?
- Availability of Updated peer test report?
- Correct Version number offered for testing?
- The above are the prerequisites for testing.
About the author: This is a guest post by Kumar Amit Anand. Author is having more than 8 years of software testing experience in Finance and Defense domain. Having SCAMPI appraisal method training certificate, he has also participated in SEI L4 and L5 final appraisals. Author also helps team members to understand CMMI and documentation.
I hope this was an interesting read, I appreciate if you could spare few minutes and post your valuable comments. This would help me come up with more interesting topics going forward.