The 7 (Cynical) Laws of Software Testing
Note: This post has been sitting in my draft folder for almost a year, and I don't know what to do with it. It started off as a serious post, but then moved to a cynical take on software testing. I planned to rewrite it one way or the other (100% serious or 100% jokey) but decided it was too much work. So here it goes.
Trigger Warning: Recovering testers may be triggered and left in tears by this post. Do not read
I still remember that one test case, though it has been many years since. I was asked to test a critical feature. I was straddling the lines between developer and software tester at the moment. It was a critical bug fix, the release had to go out the next day.
The developer checked in the fix, and went home, sending an email "My work is done, your problem now". Not those exact words, but that sentiment.
I checked out the code to test it. It wouldn't even compile.
The developer had forgotten to check in an important file. Why didn't he check I could even build the code?
Because the company had a "throw over the wall" problem. Bang out the code, and over the QA wall it goes. Your problem now, pal.
And this is my biggest gripe with many companies, that talk so much about "Quality" but do nothing about it. Let me give you 7 Cynical for Software Testing, written by and for people who hate testing and software in general.
The Throw it Over the Wall culture
Developers write code. Testers test. What's the problem?
The problem is it relegates testers to second class citizens (and let's be honest, they are. Just look at the salary difference. Can be up to half in some companies).
Sure, the developers might write unit tests. And those unit tests might even pass now and then. (Many companies don't run unit tests as part of CI, leaving it up to the developers to run them locally; at that point, they might as well not be there). Because testing is QA's job. As long as our code sort of works, QA will find any issues with it.
To be successful, testing has to be closely linked to development. This is what DevOps originally meant, a development style where coders, testers and deployment engineers worked together in the same team, to deliver the features to the very end.
But DevOps now just means Sysadmin on the cloud, "Oh yes sir, I know about the 300 AWS keywords, please hire me, sir. Can I have more soup, please?"
So here is our first law:
Law of QA Monkeys: Code monkeys code, QA monkeys QA. Throw the code over the fence and update Jira, your problem now
The good news is that this is improving. Last few years, in every company I've been in, the testers work in the same team as developers, often in the same agile team. But remains of the attitude remain "I've finished coding how you test it is your problem" do creep up now and then, which leads to the second problem.
Hostility between QA/test and developers
Story from a few years ago, at the same company I talked about above. The developers viewed testers with disdain, and the testers mirrored it.
It became so bad that it became a cold war. I was officially in the dev team, though I was doing some testing. When a full-time test automation/scripting team was created, I was seconded to it.
It was a horrible experience, one of the few times I thought about just walking off the job.
There was an atmosphere of hostility. Every code review was full of nitpicks, and the team fought about every small decision. Every chance was taken to make the coder's life difficult. Some of it was pure incompetency, and some of it was malice. The test automation team refused to fix/remove flaky tests but still insisted developers get 100% test pass. Which meant developers would keep running the tests again and again till they passed. One guy spent two months running the tests before he got the green.
Rather than being helpful guides, tests became a hurdle you had to overcome. And so the developers stopped caring. They would only write quick unit tests and then push their code, saying it was the tester's problem now. The testers fought back, failing release builds for small reasons.
2nd Law of Testing for Idiots: Poorly written/flaky tests are worse than no tests. Poor tests give you a false confidence that you are "doing" something when all you have done is <insert rude joke about jerking off>
At least if you have no tests, you can just throw it to the customer– a combination of Rule 1 and Rule 6 (see below, Shantnu's Law of Sharing and Caring)
A very shitty situation, and very rare( I think? I hope??) This problem was made worse because of how management viewed testing, which is my next point.
Throwing money at test, expecting guaranteed "results" and zero bugs
I've seen this attitude at a Fortune 500 company and a startup, so it's not rare. Here's how it works.
Developers/Project Managers say: "We need more testers/test infrastructure if you want us to ship reliable code."
After years of complaining, management finally listens.
"Here you go," they said, throwing millions of dollars on the table. "Now we expect no bugs."
And rather than improving testing, it makes it worse.
I don't know if there is a software law for this. I know Fred Brooks stated his famous law way back in 1975:
Adding people to a late project makes it even more late
I don't know if there is a testing version of this:
Shantnu's Law of Broken Window Testing: Throwing money at testing and hoping it will fix quality issues makes it worse – All underlying issues get magnified 100 times once more money is thrown at it.
(The Broken Window concept comes from here)
Why does it make it worse? Because management now expects results. "We gave this big dolla' amount, we expect no bugs!"
But bugs don't go away just because you throw money at them. In the big company I was talking about, all it did was lead to an increase in politics, with different managers fighting it out for the budget.
In another startup, the Vice President would drop in on daily scrum meetings, asking "Why wasn't this bug discovered by the QA team?"
My reply, that we had asked for 2 weeks but gotten 2 days, did not sit well. Hadn't they quadrupled our test team? (from 1 person, just me, to 4 people). I tried to point out the developer team had increased 10 times and we were bringing in open-source components to speed up development, but that meant more testing. Again, this didn't sit well.
"We are giving you so much money, we expect zero bugs at the customer's site!" Great in theory, but only works if the whole culture is aligned to it.
4th Idiot's Law to Testing: You want zero bugs and I want to be sitting on a beach with Gal Godot. Seems both of us will die unsatisfied.
Corollary to 4th Law: The Buddha was right– life is suffering (and lots of software bugs).
Expecting quality only at one point in the cycle (QA or similar)
Quality must be a cultural thing, it's not QA's job to find all possible bugs once developers have thrown their shit over the fence.
- Developers must write their own unit/integration tests
- Testers must work with them on integration/end-to-end tests
- Testers must be given enough time to finish testing
- The product must only be released when QA says it is ready, not when the VP of Sales thinks it is
And then you have the right to complain if the code still crashes at the customer site. Quality comes from a culture that values it, not where money is thrown at it as a way to fix underlying cultural problems.
5th Idiot's Law of Testing: QA/Testing isn't like Harry Potter, you can't just magic away the underlying issues just by having a QA team.
Only doing functional testing
Every company I've interviewed for asks questions about different ways to test- fuzz testing, security testing, UI testing. Every company I've worked for, from Fortune 500 ones to startups, only does the minimal functional testing to shovel the shit, sorry I meant product, out of the door.
There is no pretence to even look at checking if the UI is easy to use. At one place, I raised a bug that the UI was confusing and worked in non-expected ways. The bug was closed, as the developers felt "Come on, it's obvious once you start using it". 3 months later, a customer complained about the same thing and suddenly the devs were working weekends to fix the issue before a big release.
Most companies have the attitude of We built this shit this way, test it this way. Kthxbye
I see companies like Microsoft release products with horrible UI issues, so I see no hope for smaller companies.
The attitude still is:
6th Idiot Law: If we do all the testing, what will the customer do? We have to leave something for them! --Shantnu's Law of Sharing and Caring in Testing
Because remember, sharing is caring!!
Here are all the Laws of Idiot's Guide to Testing collected in one place:
So here is our first law:
1) Law of QA Monkeys: Code monkeys code, QA monkeys QA. Throw the code over the fence and update Jira, your problem now
2nd Law of Testing for Idiots: Poorly written/flaky tests are worse than no tests. Poor tests give you a false confidence that you are "doing" something when all you have done is <insert rude joke about jerking off>
3) Shantnu's Law of Broken Window Testing: Throwing money at testing and hoping it will fix quality issues makes it worse – All underlying issues get magnified 100 times once more money is thrown at it.
4th Idiot's Law to Testing: You want zero bugs and I want to be sitting on a beach with Gal Godot. Seems both of us will die unsatisfied.
Corollary to 4th Law: The Buddha was right– life is suffering (and lots of software bugs).
5th Idiot's Law of Testing: QA/Testing isn't like Harry Potter, you can't just magic away the underlying issues just by having a QA team.
6th Idiot Law: If we do all the testing, what will the customer do? We have to leave something for them! --Shantnu's Law of Sharing and Caring in Testing
And I can combine the laws to create a 7th Law of Testing, Called the ONE LAW
One Law to Rule Them All,
One Law to Combine them
and in the Darkness Bind them
One Law to Rule Them All
The 7th Idiot's Law of Testing : Always sell to big corporations and government organisations, where it takes 3 years to get your contract approved. Because even if you tests are shitty or don't exist, if you throw the testing to customers, what are they going to do? Waste another 3 years getting a new contract?
The 7th Law says: Just do enough "pretend" testing so you can throw shit over the wall and find a better job.