1. What is ‘Software Quality Assurance’?
2. What is ‘Software Testing’?
3. What are some recent major computer system failures caused by software bugs?
4. Does every software project need testers?
5. Why does software have bugs?
6. How can new Software QA processes be introduced in an existing organization?
7. What is verification? Validation?
8. What is a ‘walkthrough’?
9. What’s an ‘inspection’?
10. What kinds of testing should be considered?
11. What are 5 common problems in the software development process?
12. What are 5 common solutions to software development problems?
13. What is software ‘quality’?
14. What is ‘good code’?
15. What is ‘good design’?
16. What is the ‘software life cycle’?
17. What makes a good Software Test engineer?
18. What makes a good Software QA engineer?
19. What’s the role of documentation in QA?
20. What’s the big deal about ‘requirements’?
21. What’s a ‘test plan’?
22. What’s a ‘test case’?
23. What should be done after a bug is found?
24. What is ‘configuration management’?
25. Is programming knowledge required to do QA software testing training
Software QA involves the entire software development PROCESS – monitoring and improving the process, making sure that any agreed-upon standards and procedures are followed, and ensuring that problems are found and dealt with. It is oriented to ‘prevention’.
Testing involves operation of a system or application under controlled conditions and evaluating the results (e.g, ‘if the user is in interface A of the application while using hardware B, and does C, then D should happen’). The controlled conditions should include both normal and abnormal conditions. Testing should intentionally attempt to make things go wrong to determine if things happen when they shouldn’t or things don’t happen when they should. It is oriented to ‘detection’.
- Organizations vary considerably in how they assign responsibility for QA and testing. Sometimes they’re the combined responsibility of one group or individual. Also common are project teams that include a mix of testers and developers who work closely together, with overall QA processes monitored by project managers. It will depend on what best fits an organization’s size and business structure.
- In February of 2009 users of a major search engine site were prevented from clicking through to sites listed in search results for part of a day.
- A large health insurance company was reportedly banned by regulators from selling certain types of insurance policies in January of 2009 due to ongoing computer system problems that resulted in denial of coverage for needed medications and mistaken overcharging or cancelation of benefits.
- A news report in January 2009 indicated that a major IT and management consulting company was still battling years of problems in implementing its own internal accounting systems, including a 2005 implementation that reportedly “was attempted without adequate testing”.
- In August of 2008 it was reported that more than 600 U.S. airline flights were significantly delayed due to a software glitch in the U.S. FAA air traffic control system.
- Software bugs caused the bank accounts of 823 customers of a major U.S. bank to be credited with $924,844,208.32 each in May of 1996, according to newspaper reports.
- Software bugs in a Soviet early-warning monitoring system nearly brought on nuclear war in 1983, according to news reports in early 1999.
While all projects will benefit from testing, some projects may not require independent test staff to succeed.
Which projects may not need independent test staff? The answer depends on the size and context of the project, the risks, the development methodology, the skill and experience of the developers, and other factors. For instance, if the project is a short-term, small, low risk project, with highly experienced programmers utilizing thorough unit testing or test-first development, then test engineers may not be required for the project to succeed.
In some cases an IT organization may be too small or new to have a testing staff even if the situation calls for it. In these circumstances it may be appropriate to instead use contractors or outsourcing, or adjust the project management and development approach (by switching to more senior developers and agile test-first development, for example). Inexperienced managers sometimes gamble on the success of a project by skipping thorough testing or having programmers do post-development functional testing of their own work, a decidedly high risk gamble.
For non-trivial-size projects or projects with non-trivial risks, a testing staff is usually necessary. As in any business, the use of personnel with specialized skills enhances an organization’s ability to be successful in large, complex, or difficult tasks. It allows for both a) deeper and stronger skills and b) the contribution of differing perspectives. For example, programmers typically have the perspective of ‘what are the technical issues in making this functionality work?’. A test engineer typically has the perspective of ‘what might go wrong with this functionality, and how can we ensure it meets expectations?’. Technical people who can be highly effective in approaching tasks from both of those perspectives are rare, which is why, sooner or later, organizations bring in test specialists.
Miscommunication or no Communication – as to specifics of what an application should or shouldn’t do (the application’s requirements).
Software complexity – the complexity of current software applications can be difficult to comprehend for anyone without experience in modern-day software development.
Programming errors – programmers, like anyone else, can make mistakes.
Changing requirements (whether documented or undocumented) – the end-user may not understand the effects of changes, or may understand and request them anyway – redesign, rescheduling of engineers, effects on other projects, work already completed that may have to be redone or thrown out, hardware requirements that may be affected, etc.
Time pressures – scheduling of software projects is difficult at best, often requiring a lot of guesswork. When deadlines loom and the crunch comes, mistakes will be made.
Poorly documented code – it’s tough to maintain and modify code that is badly written or poorly documented; the result is bugs.
Software development tools – visual tools, class libraries, compilers, scripting tools, etc. often introduce their own bugs or are poorly documented, resulting in added bugs.
- A lot depends on the size of the organization and the risks involved. For large organizations with high-risk (in terms of lives or property) projects, serious management buy-in is required and a formalized QA process is necessary.
- Where the risk is lower, management and organizational buy-in and QA implementation may be a slower, step-at-a-time process. QA processes should be balanced with productivity so as to keep bureaucracy from getting out of hand.
- For small groups or projects, a more ad-hoc process may be appropriate, depending on the type of customers and projects. A lot will depend on team leads or managers, feedback to developers, and ensuring adequate communications among customers, managers, developers, and testers.
- The most value for effort will often be in (a) requirements management processes, with a goal of clear, complete, testable requirement specifications embodied in requirements or design documentation, or in ‘agile’-type environments extensive continuous coordination with end-users, (b) design inspections and code inspections, and (c) post-mortems/retrospectives.
- Other possibilities include incremental self-managed team approaches such as ‘Kaizen’ methods of continuous process improvement, the Deming-Shewhart Plan-Do-Check-Act cycle, and others.
Verification typically involves reviews and meetings to evaluate documents, plans, code, requirements, and specifications. This can be done with checklists, issues lists, walkthroughs, and inspection meetings. Validation typically involves actual testing and takes place after verifications are completed. The term ‘IV & V’ refers to Independent Verification and Validation.
A ‘walkthrough’ is an informal meeting for evaluation or informational purposes. Little or no preparation is usually required.
An inspection is more formalized than a ‘walkthrough’, typically with 3-8 people including a moderator, reader, and a recorder to take notes. The subject of the inspection is typically a document such as a requirements spec or a test plan, and the purpose is to find problems and see what’s missing, not to fix anything. Attendees should prepare for this type of meeting by reading thru the document; most problems will be found during this preparation. The result of the inspection meeting should be a written report.
Black box Testing: not based on any knowledge of internal design or code. Tests are based on requirements and functionality.
Regression Testing – re-testing after fixes or modifications of the software or its environment.
Acceptance Testing – final testing based on specifications of the end-user or customer, or based on use by end-users/customers over some limited period of time.
Load Testing – testing an application under heavy loads, such as testing of a web site under a range of loads to determine at what point the system’s response time degrades or fails.
Stress Testing – term often used interchangeably with ‘load’ and ‘performance’ testing. Also used to describe such tests as system functional testing while under unusually heavy loads, heavy repetition of certain actions or inputs, input of large numerical values, large complex queries to a database system, etc.
Performance Testing – term often used interchangeably with ‘stress’ and ‘load’ testing. Ideally ‘performance’ testing (and any other ‘type’ of testing) is defined in requirements documentation or QA or Test Plans.
Usability Testing – testing for ‘user-friendliness’.
User acceptance Testing – determining if software is satisfactory to an end-user or customer.
Poor requirements: If requirements are unclear, incomplete, too general, and not testable, there may be problems.
Unrealistic schedule: If too much work is crammed in too little time, problems are inevitable.
Inadequate Testing: No one will know whether or not the software is any good until customers complain or systems crash.
Featuritis: Requests to add on new features after development goals are agreed on.
Miscommunication – if developers don’t know what’s needed or customer’s have erroneous expectations, problems can be expected.
Solid Requirements: Clear, complete, detailed, cohesive, attainable, testable requirements that are agreed to by all players.
Realistic Schedules: Allow adequate time for planning, design, testing, bug fixing, re-testing, changes, and documentation; personnel should be able to complete the project without burning out.
Adequate Testing: Start testing early on, re-test after fixes or changes, plan for adequate time for testing and bug-fixing. ‘Early’ testing could include static code analysis/testing, test-first development, unit testing by developers, built-in testing and diagnostic capabilities, automated post-build testing, etc.
Stick to initial requirements where feasible: Be prepared to defend against excessive changes and additions once development has begun, and be prepared to explain consequences. If changes are necessary, they should be adequately reflected in related schedule changes. If possible, work closely with customers/end-users to manage expectations. In ‘agile’-type environments, initial requirements may be expected to change significantly, requiring that true agile processes be in place and followed.
Communication: Require walkthroughs and inspections when appropriate; make extensive use of group communication Tools.
Quality software is reasonably bug-free, delivered on time and within budget, meets requirements and/or expectations, and is maintainable. However, quality is obviously a subjective term. It will depend on who the ‘customer’ is and their overall influence in the scheme of things.
A wide-angle view of the ‘customers’ of a software development project might include end-users, customer acceptance testers, customer contract officers, customer management, the development organization’s management/accountants/testers/salespeople, future software maintenance engineers, stockholders, magazine columnists, etc.
‘Good code’ is code that works, is reasonably bug free, and is readable and maintainable. Some organizations have coding ‘standards’ that all developers are supposed to adhere to, but everyone has different ideas about what’s best, or what is too many or too few rules.
‘Design’ could refer to many things, but often refers to ‘functional design’ or ‘internal design’. Good internal design is indicated by software code whose overall structure is clear, understandable, easily modifiable, and maintainable; is robust with sufficient error-handling and status logging capability; and works correctly when implemented.
The life cycle begins when an application is first conceived and ends when it is no longer in use. It includes aspects such as initial concept, requirements analysis, functional design, internal design, documentation planning, test planning, coding, document preparation, integration, testing, maintenance, updates, retesting, phase-out, and other aspects.
A good test engineer has a ‘test to break’ attitude, an ability to take the point of view of the customer, a strong desire for quality, and an attention to detail. Tact and diplomacy are useful in maintaining a cooperative relationship with developers, and an ability to communicate with both technical (developers) and non-technical (customers, management) people is useful. Previous software development experience can be helpful as it provides a deeper understanding of the software development process, gives the tester an appreciation for the developers’ point of view, and reduce the learning curve in automated test tool programming. Judgment skills are needed to assess high-risk or critical areas of an application on which to focus testing efforts when time is limited.
The same qualities a good tester has are useful for a QA engineer. Additionally, they must be able to understand the entire software development process and how it can fit into the business approach and goals of the organization. Communication skills and the ability to understand various sides of issues are important. In organizations in the early stages of implementing QA processes, patience and diplomacy are especially needed. An ability to find problems as well as to see ‘what’s missing’ is important for inspections and reviews.
Generally, the larger the team/organization, the more useful it will be to stress documentation, in order to manage and communicate more efficiently.
One of the most reliable methods of ensuring problems, or failure, in a large, complex software project is to have poorly documented requirements specifications.
Requirements are the details describing an application’s externally-perceived functionality and properties. Requirements should be clear, complete, reasonably detailed, cohesive, attainable, and testable.
A software project test plan is a document that describes the objectives, scope, approach, and focus of a software testing effort. The process of preparing a test plan is a useful way to think through the efforts needed to validate the acceptability of a software product. The completed document will help people outside the test group understand the ‘why’ and ‘how’ of product validation. It should be thorough enough to be useful but not so thorough that no one outside the test group will read it.
A test case describes an input, action, or event and an expected response, to determine if a feature of a software application is working correctly. A test case may contain particulars such as test case identifier, test case name, objective, test conditions/setup, input data requirements, steps, and expected results.
The bug needs to be communicated and assigned to developers that can fix it. After the problem is resolved, fixes should be re-tested, and determinations made regarding requirements for regression testing to check that fixes didn’t create problems elsewhere. If a problem-tracking system is in place, it should encapsulate these processes. A variety of commercial problem-tracking/management software tools are available
24. What is ‘configuration management’?
Configuration management covers the processes used to control, coordinate, and track: code, requirements, documentation, problems, change requests, designs, tools/compilers/libraries/patches, changes made to them, and who makes the changes.