dido:public:ra:1.4_req:2_nonfunc:20_maintainability:testability
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
dido:public:ra:1.4_req:2_nonfunc:20_maintainability:testability [2021/01/14 16:42] ian |
dido:public:ra:1.4_req:2_nonfunc:20_maintainability:testability [2021/10/03 13:23] (current) 50.19.247.197 ↷ Links adapted because of a move operation |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | ====== 4.2.3.5 Testability ===== | + | ====== 4.3.3.5 Testability ===== |
| [[dido:public:ra:1.4_req:2_nonfunc:20_maintainability| Return to Maintainability]] | [[dido:public:ra:1.4_req:2_nonfunc:20_maintainability| Return to Maintainability]] | ||
| - | |||
| - | * **<color #FF0000><todo @char>Please Review</todo></color>** | ||
| ===== About ===== | ===== About ===== | ||
| [[dido:public:ra:1.4_req:2_nonfunc:20_maintainability:testability| Return to Top]] | [[dido:public:ra:1.4_req:2_nonfunc:20_maintainability:testability| Return to Top]] | ||
| - | [[dido:public:ra:xapend:xapend.a_glossary:t:testability]], Testable, Testing and Test are not synonyms for each other. Just because there is testing using various tests on a system or a program does not mean that the the system or program is Testable. | + | //[[dido:public:ra:xapend:xapend.a_glossary:t:testability]]//, //Testable//, //Testing// and //Test// are not synonyms for each other. Just because a system or program is undergoing //testing// using various //tests// does not necessarily mean that the system or program is actually //Testable//. The following table provides definitions for each of these four terms, associates each with the appropriate Structured [[dido:public:ra:xapend:xapend.a_glossary:a:assurance|Assurance]] Case, as well as, level in the Cognitive Model's Science and Knowledge Management [[dido:public:ra:xapend:xapend.i_cog_model:start|DIKW]] (Data, Information, Knowledge and Wisdom) pyramid. |
| ^ DIKW pyramid level ^ Structured Assurance Case ^ Term ^ Description ^ | ^ DIKW pyramid level ^ Structured Assurance Case ^ Term ^ Description ^ | ||
| | Understanding | [[dido:public:ra:xapend:xapend.a_glossary:s:swassurance]] | Testability | <WRAP> | | Understanding | [[dido:public:ra:xapend:xapend.a_glossary:s:swassurance]] | Testability | <WRAP> | ||
| - | **Testability** is about the __documenting__ functionality and the [[dido:public:ra:xapend:xapend.a_glossary:r:requirement|requirements]] for the system or the program and verifying that the requirements are going-to or have-been met. [[dido:public:ra:xapend:xapend.a_glossary:f:funcreq|Functional requirements]] are generally not a problem since most of these are directly measurable or observable. Often Functional requirements are directly measured or observed such as a data field needing to be provided, a relationship exists between two pieces of data, or that the relationship is a one-to-many or a many=to-many. For example, every person must have a company [[dido:public:ra:xapend:xapend.a_glossary:i:id |ID]] number, they may have multiple phone numbers or people can belong to multiple organizations. | + | **Testability** is about __documenting__ the functionality and [[dido:public:ra:xapend:xapend.a_glossary:r:requirement|requirements]] for a system or program and verifying that these requirements will be or have been met. [[dido:public:ra:xapend:xapend.a_glossary:f:funcreq|Functional requirements]] are generally not a problem since most of these are directly measurable or observable. Functional requirements are often directly measured or observed: a data field needing to be provided, a relationship existing between two pieces of data, or a relationship that is one-to-many or a many-to-many. For example, every person must have a unique company [[dido:public:ra:xapend:xapend.a_glossary:i:id |ID]] number but they may have multiple phone numbers and also belong to multiple organizations. |
| - | Other functional requirements are not so definite, but are expressed in terms of a range of acceptable value. For example, a [[dido:public:ra:xapend:xapend.a_glossary:g:gui]] will respond in less than 5 seconds. The heart pulse rate is between 35 to 200. | + | Other functional requirements are not so definite, but expressed in terms of a range of acceptable values. For example, a [[dido:public:ra:xapend:xapend.a_glossary:g:gui]] will respond in less than 5 seconds or the heart pulse rate is between 35 to 200 beats per minute. |
| - | However, [[dido:public:ra:xapend:xapend.a_glossary:n:nonfuncreq|non-functional requirements]] are generally more abstract and relate to the quality of the system or program being delivered (i.e., portable, reliable, maintainable, secureable, scalable, et.) Often these are not directly measurable or observable but can be inferred from characteristics found in the delivered system's or product's architecture, design and implementation. These kinds of requirements require assurance. The requirements are specified as claims (i.e., the system is has a [[dido:public:ra:1.4_req:2_nonfunc:14_reliability:02_availability | High Availability]] ) which require [[dido:public:ra:xapend:xapend.a_glossary:s:subclaim|sub-claims]], [[dido:public:ra:xapend:xapend.a_glossary:a:argument|arguments]] (i.e., [[dido:public:ra:xapend:xapend.a_glossary:a:availability]] can be predicted by using the [[dido:public:ra:xapend:xapend.a_glossary:m:mttr]]) of 5 minutes, 15 seconds or less of [[dido:public:ra:xapend:xapend.a_glossary:d:downtime|downtime]] in a year for all components. These kinds of requirements are generally specified in a [[dido:public:ra:xapend:xapend.a_glossary:p:performancespec]] and their is tendency to only describe hardware specifications, but the [[dido:public:ra:xapend:xapend.a_glossary:p:performance|performance]] specification can also capture non-functional metrics as well. | + | In contrast, [[dido:public:ra:xapend:xapend.a_glossary:n:nonfuncreq|non-functional requirements]] are generally more abstract: they relate to the __quality__ of the system or program being delivered (i.e., portable, reliable, maintainable, securable, scalable, etc.) and are usually not directly measurable or observable but must be inferred from characteristics found in the delivered system's or product's architecture, design and implementation. These kinds of requirements require ways to characterize [[dido:public:ra:xapend:xapend.a_glossary:a:assurance|assurance]] and are specified in terms of [[dido:public:ra:xapend:xapend.a_glossary:c:claim|claims]] (i.e., the system has a [[dido:public:ra:1.4_req:2_nonfunc:14_reliability:02_availability | High Availability]]), [[dido:public:ra:xapend:xapend.a_glossary:s:subclaim|sub-claims]], and [[dido:public:ra:xapend:xapend.a_glossary:a:argument|arguments]] (i.e., [[dido:public:ra:xapend:xapend.a_glossary:a:availability]] can be predicted using a [[dido:public:ra:xapend:xapend.a_glossary:m:mttr]] of 5 minutes, 15 seconds or less of [[dido:public:ra:xapend:xapend.a_glossary:d:downtime|downtime]] in a year for all components). These kinds of requirements are generally specified in [[dido:public:ra:xapend:xapend.a_glossary:p:performancespec]]. These specifications tend to focus on hardware specifications; however, [[dido:public:ra:xapend:xapend.a_glossary:p:performancespec|performance specifications]] can also capture non-functional metrics. |
| - | * **Note** Testability metrics are not limited to operational systems or programs but can also use system or program level artifacts describing architecture, design, discussion papers, outside references, software and executables. | + | * **Note** Testability metrics are not limited to operational systems or programs but can also take advantage of system or program level artifacts that describe architecture, design, discussion papers, outside references, software and executables. |
| - | Another problem when reviewing requirements and trying to determine if they are actually "testable". Here is a list of some common "mistakes" found in requirement documents(( | + | Here is a list of some common "mistakes" found in requirement documents(( |
| __Achieving Requirements Testability__, | __Achieving Requirements Testability__, | ||
| ProlificsTesting, | ProlificsTesting, | ||
| Line 25: | Line 23: | ||
| Accessed on 9 August 2020 | Accessed on 9 August 2020 | ||
| [[https://www.prolifics-testing.com/news/achieving-requirements-testability]] | [[https://www.prolifics-testing.com/news/achieving-requirements-testability]] | ||
| - | )): | + | )) that can make it difficult to determine if requirements are actually "testable": |
| - | * **Noise:** Text containing no information relevant to any aspect of the problem. For example, A standalone application that does not need access to the [[dido:public:ra:xapend:xapend.a_glossary:e:ethernet|Ethernet]] | + | * **Noise:** Text containing no information relevant to any aspect of the problem. For example, a requirement on a standalone [[dido:public:ra:xapend:xapend.a_glossary:a:application|application]] that does not need access to the [[dido:public:ra:xapend:xapend.a_glossary:e:ethernet|Ethernet]] |
| * The system shall conform to IPV6 ... | * The system shall conform to IPV6 ... | ||
| - | * **Silence:** A feature not covered by any text within the Requirements documents or the specification | + | * **Silence:** A feature not covered by any text within the Requirements documents or specifications |
| - | * **Over-specification:** Description of the solution rather than of the problem. For example, | + | * **Over-specification:** Description of the solution rather than the problem. For example, |
| - | * The [[dido:public:ra:xapend:xapend.a_glossary:d:distsystem|distributed system]] must use blockchain. (blockchain is one of many distributed technologies used by Cryptocurrencies) | + | * The [[dido:public:ra:xapend:xapend.a_glossary:d:distsystem|distributed system]] must use [[dido:public:ra:xapend:xapend.a_glossary:b:blockchain|blockchain]]. (blockchain is one of many distributed technologies used by Cryptocurrencies) |
| * The system must use a checkbox to select the appropriate option | * The system must use a checkbox to select the appropriate option | ||
| * **Contradictory:** Mutually incompatible descriptions of the same feature. For example, | * **Contradictory:** Mutually incompatible descriptions of the same feature. For example, | ||
| * The system shall not record any personal information | * The system shall not record any personal information | ||
| - | * The system shall record all transactions and the parties participating in the transaction | + | * The system shall record all transactions and parties participating in the transaction |
| * **Ambiguity:** Text that can be interpreted more than one way | * **Ambiguity:** Text that can be interpreted more than one way | ||
| * The system shall support real-time operations (what is real-time?) | * The system shall support real-time operations (what is real-time?) | ||
| Line 40: | Line 38: | ||
| * The system shall publish all information on a [[dido:public:ra:xapend:xapend.a_glossary:t:topic|topic]] (but topic has not been officially defined yet) | * The system shall publish all information on a [[dido:public:ra:xapend:xapend.a_glossary:t:topic|topic]] (but topic has not been officially defined yet) | ||
| * **Wishful thinking:** Defining a feature that can’t be validated | * **Wishful thinking:** Defining a feature that can’t be validated | ||
| - | * The system shall initialize all values with intelligent default choices. | + | * The system shall initialize all values with intelligent default choices. (what's the metric for "intelligent"?) |
| - | * **Weak phrases:** Causing uncertainty (“adequate”, “usually”, “etc.”) For Example, | + | * **Weak phrases:** Causing uncertainty (“adequate”, “usually”, “etc.”) For example, |
| - | * When possible, the systems shall do ... | + | * //When possible//, the systems shall ... |
| * The system shall collect their data (whose data?) | * The system shall collect their data (whose data?) | ||
| * **Jigsaw puzzles:** Requirements distributed across a document and then cross-referenced | * **Jigsaw puzzles:** Requirements distributed across a document and then cross-referenced | ||
| - | * **Duckspeak:** Requirements only there to conform to standards | + | * **Duckspeak:** Requirements included merely to conform to standards that have no or little relationship to the problem at hand. Perhaps required as part of a boilerplate. |
| - | * **Terminology invention:** “user input/presentation function”; “airplane reservation data validation function”. For example, | + | * **Terminology invention:** “user input/presentation function”; “airplane reservation data [[dido:public:ra:xapend:xapend.a_glossary:v:validation|validation]] function”. For example, |
| * The system shall use a double blind logged journal entry (huh, what is that?) | * The system shall use a double blind logged journal entry (huh, what is that?) | ||
| * **Putting the onus on developers and testers:** to guess what the requirements really are. | * **Putting the onus on developers and testers:** to guess what the requirements really are. | ||
| Line 53: | Line 51: | ||
| </WRAP> | | </WRAP> | | ||
| | Knowledge | [[dido:public:ra:xapend:xapend.a_glossary:c:claim]] | Testable | <WRAP> | | Knowledge | [[dido:public:ra:xapend:xapend.a_glossary:c:claim]] | Testable | <WRAP> | ||
| - | A **Testable** __attribute__ of a system or program is a functional or nonfunctional requirement is testable or not. All requirements can not be tested. Some requirements can be directly tested for by running specific tests (i.e., [[dido:public:ra:xapend:xapend.a_glossary:u:unittesting]], [[dido:public:ra:xapend:xapend.a_glossary:i:integrationtesting|integration testing]], etc.) using test plans that exercise the portion of the system or program software responsible for providing the functionality. For example, the system is suppose to offer the choice of ''none'', ''one'', ''many''. Another example might be that when an option is selected, a message is sent out over the network. | + | A **Testable** __attribute__ of a system or program is a functional or nonfunctional requirement that may be testable or not. Some requirements can be directly tested for by running specific tests (i.e., [[dido:public:ra:xapend:xapend.a_glossary:u:unittesting]], [[dido:public:ra:xapend:xapend.a_glossary:i:integrationtesting|integration testing]], etc.) using test plans that exercise a portion of the system or program software responsible for providing specific functionality. For example, the system is supposed to offer the choice of ''none'', ''one'', and ''many''. Another example might be that when an option is selected, a message is sent out over the network. |
| - | + | ||
| - | Other requirements are not directly testable by design and therefore are untestable. Often, these requirements are met through the use of mathematical proofs or demonstrations. For example, the generation of a [[dido:public:ra:xapend:xapend.a_glossary:u:uuid]] can not be tested directly, but the algorithm must provide an explanation and a proof that no two sets of conditions can produce the same UUID. Often there is a risk of generating the same UUID, but the chances of the UUID being used in identical is even smaller. Another example would the [[dido:public:ra:xapend:xapend.a_glossary:r:recaptcha]] which shows a series of photos and asks the user to identify the one with green peas in it. The order of the photos and the thing it is asking you to identify are randomly assigned. | + | |
| + | By design, some requirements are not directly testable, i.e, are untestable. Often, these requirements are met through the use of mathematical proofs or demonstrations. For example, the generation of a [[dido:public:ra:xapend:xapend.a_glossary:u:uuid]] can not be tested directly; instead, the algorithm used to generate them must provide an explanation and proof that no two sets of conditions will produce the same UUID. Often there is a risk of generating the same UUID, but the chances of the same UUID being used in identical domains or environments is even smaller. Another example would be the [[dido:public:ra:xapend:xapend.a_glossary:r:recaptcha]], which shows a series of photos and asks the user to identify the ones with green peas in them. The order of the photos and the thing it is asking you to identify are randomly assigned. | ||
| </WRAP> | | </WRAP> | | ||
| | Information | [[dido:public:ra:xapend:xapend.a_glossary:a:argument]] | Testing | <WRAP> | | Information | [[dido:public:ra:xapend:xapend.a_glossary:a:argument]] | Testing | <WRAP> | ||
| - | Testing is a __process__ that generally involves the execution of the system or program under scripted, controlled situations. The scripts can be human instructions in documents or they can be captured in text files that a testing engine uses to drive the software. Sometimes, a Unit Test is used to test the individual modules before they are integrated into the system or program. There are various requirement conformity checks that can be done to verify functional requirements: | + | Testing is a __process__ that generally involves the execution of the system or program under scripted, controlled situations. The scripts can be human instructions in documents or they can be captured in text files that a testing engine uses to drive the software. Sometimes, a Unit Test is used to test individual modules before they are integrated into the system or program. Below are the various requirement conformity checks that can be performed to verify functional requirements: |
| - [[dido:public:ra:xapend:xapend.a_glossary:u:unittesting]] | - [[dido:public:ra:xapend:xapend.a_glossary:u:unittesting]] | ||
| - [[dido:public:ra:xapend:xapend.a_glossary:i:integrationtesting]] | - [[dido:public:ra:xapend:xapend.a_glossary:i:integrationtesting]] | ||
| Line 72: | Line 68: | ||
| - [[dido:public:ra:xapend:xapend.a_glossary:b:blackboxtesting]] | - [[dido:public:ra:xapend:xapend.a_glossary:b:blackboxtesting]] | ||
| - [[dido:public:ra:xapend:xapend.a_glossary:i:interfacetesting]] | - [[dido:public:ra:xapend:xapend.a_glossary:i:interfacetesting]] | ||
| + | - [[dido:public:ra:xapend:xapend.a_glossary:i:interoptesting]] | ||
| </WRAP> | | </WRAP> | | ||
| | Data | [[dido:public:ra:xapend:xapend.a_glossary:e:evidence]] | Test | <WRAP> | | Data | [[dido:public:ra:xapend:xapend.a_glossary:e:evidence]] | Test | <WRAP> | ||
| - | Is about __collecting__ the evidence used to support arguments, sub-claims and claims made about the system or program. There is not a one-to-one relationship between a Test, an [[dido:public:ra:xapend:xapend.a_glossary:a:argument]], Sub-Claim or a Claim. Instead, one piece of data can support multiple Arguments, an Argument can support multiple Sub-Claims or Claims. That is why it is so important to have a Structured Assurance Case Model. | + | //Test// refers to the act of __collecting__ the evidence used to support arguments, sub-claims and claims made about the system or program. There is not a one-to-one relationship between a Test, an [[dido:public:ra:xapend:xapend.a_glossary:a:argument]], a Sub-Claim or a Claim. Instead, one piece of data can support multiple Arguments and an Argument can support multiple Sub-Claims or Claims. That is why it is so important to have a Structured Assurance Case Model. |
| </WRAP>| | </WRAP>| | ||
| - | ===== DDS Specifics ===== | + | ===== DIDO Specifics ===== |
| [[dido:public:ra:1.4_req:2_nonfunc:20_maintainability:testability| Return to Top]] | [[dido:public:ra:1.4_req:2_nonfunc:20_maintainability:testability| Return to Top]] | ||
| + | : <wrap hi><color red> To be added/expanded in future revisions of the DIDO RA </color></wrap> | ||
| /**=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- | /**=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- | ||
dido/public/ra/1.4_req/2_nonfunc/20_maintainability/testability.1610660569.txt.gz · Last modified: 2021/01/14 16:42 by ian
