The RST course is over and I’m going through my notes. Here are my notes from day 2 (which was Wednesday) This post is a little less pithy than my notes from day 1. Another aspect of the course not covered in my notes are the testing activities and post-activity debriefs. These are the real meat of the course but it’s hard to capture by writing about them and I suspect that everyone gets something different out of them.
My notes:
Learn the things that you have to test – maybe don’t just jump in and do stuff with the product as your first move if you have time. Some ideas:
- Look at competing products
- Look at past versions of the same product
- Look at the spec if you have one
If time is tight, you may have to jump right in
- It will feel more disorganized
- It’s like an eating contest – messy
Eventually both approaches end up in the same place
Coverage
Frequently misunderstood to only mean code coverage.
There are other kinds: product coverage, risk coverage, requirements coverage, etc.
The mnemonic SFDIPOT helps us remember aspects of the product to consider (“The big 7”)
- Structure (developers most concerned)
- Function (novice testers most concerned)
- Data (experienced testers will consider)
- Interfaces (system, user, cloud)
- Platform (dependencies outside the scope of your project; things you can’t fix yourself)
- Operations (usage patterns
- Time elements (similar to data, experienced people will consider)
Subtitle heuristic: “No user would do that” == No user I can think of, who I like, would do that on purpose
Other product aspects:
Testability (about controllability and observability; helps developers too when they have to debug)
Availability
Simplicity
Stability
Heuristic Test Strategy Model
Developing the model had a huge effect on James’ career
Came from thinking about Guide words in the safety engineering field (HazOps)
Helps you have an agenda in your head when you start a new testing project
At a high level, you:
- Examine the product
- Against Quality Criteria
- Within a project environment
You chase risk – where is the fire likely to be burning?
Product relates to coverage
Quality criteria relates to oracles
Project environment relates to procedures
First two are concepts, last one relates to the real world
4 pillars:
- Diversification
- Cost vs. Value
- Skill (like muscle)
- Heuristics (like bones)
Skill without heuristics lacks reliability
Heuristics are mooring points for tacit knowledge
Rapid testing is about solving problems while minimizing waste
Not “best practices” thinking – about problems and solutions
#1 issue: finding the problem
No one style; You can have your own style
A tester needs to sweat the details
Sometimes we pick oracles that are too probably (Prior probability heuristic)
Disconnect from the automatic response
Debugging your mental model – one way is to restate things
Skepticism: refusal of certainty (not refusal of belief)
Coherence-based reasoning (pattern matching), the basis of stereotypes
Two types of thinking – System 1 and System 2 (Thinking, Fast and Slow; Kahneman)
System 1: reflexive, never asks for more data, always ready to go
System 2: reflective, rational, takes time and energy
We want both systems working and cooperating
Scientific skills
Feeling of ignorance: you need it
Makes room for learning
3 questions to ask:
- Huh? (you don’t understand)
- Really? (what you understand may not be true)
- So? (it may not matter or may matter less than you think)
Requirements statements
Make sure you know what every word means – take it apart
No one person needs to supply all the ideas (pairing is good, but may be exhausting)
Make use of visualization
Realize there are tacit requirements too
Inattentional blindness
Sometimes you have to see tests run more than one time to understand
Video can help
Know the limitations of our observational skills
Oracles are how you know there is a problem
Expected results are happening all the time (at every test step)
Machines can’t observer anything, can’t interrupt a test
Assumption: you treat something as a fact when you have no evidence that it is a fact
Assumptions can be dangerous
How to assess: How likely is it that the assumption is wrong? What is the impact if it is?
Facts rest on a bed of assumptions
Testers are allowed to make reasonable assumptions
It’s hard to make a list of all our expectations
We are always in the process of making sense of the world
It is a mental process not a list
We do memorize lists]
It’s not possible to write down a test
It is possible to write down things that are useful
Humans are always looking at more than you can write down
Testing depends on tacit knowledge and shared models
Also, conversations about them. Also socialization
Rapid testing is rapid learning
Don’t judge things by the first 15 minutes
Don’t be intimidated
Review by rewrite, review by testing, review by enactment
Description vs. Actual vs. Intent
Notice unintended consequences