It Takes A Village: Training Leading and Inspiring a Young Testing Team

Have you all checked out the March issue of The Testing Planet yet? Its a great issue, as usual, and this one includes an article by yours truly called: It takes a Village.

I summarize some of my experiences growing a testing team and the lesson I learned about inspiring young testers by giving them room and responsibility to inspire themselves.

I hope everyone checks it out!

Print Friendly
Posted in Uncategorized | 1 Comment

Testing 101: Boundary Value Analysis

Boundary Value Analysis (BVA) is a core Black Box Testing technique that limits the number of test cases you need to write in order to cover the functionality. BVA is similar to Equivilance Partitioning (EP), but more specific. While Equivilance Partitioning says that any value in a given range (partition) is equivilant for testing purposes, Boundary Value Analysis states that the values on the edges of each equivilant partition provide the most valuable test cases. By testing those values you will catch most of the bugs in the functionality because you are testing the boundaries of logic behind those rules.

Lets look at the data we used in the Equivilance Paritioning example to see how Boundary Value Analysis would handle the same set of data.

 0   1  |    2   3    4   5     |    6   7   8  9...
Invalid (too low) |      Valid       |   Invalid (too high)

BVA would suggest looking at the data, and selecting tests at the boundry of each section. Thus, you would select the following values: 0, 1, 2, 5, 6.

  • 0 is the lowest invalid value
  • 1 is the highest of the first set of invalid values.
  • 2 is the lowest valid value
  • 5 is the highest valid value
  • and 6 is the first invalid value in the second set.

By testing these values, you have tested the values at the boundries of each set of valid and invalid data.

In this example offers two sets of valid data:

           0   |   1 2      3      |     4      5 6     |     7  8 9...
Invalid (too low) | Valid (message A) |  Valid (message B) |  Invalid (too high)

BVA would select the following values: 0, 1, 3, 4, 6, 7  – just the values at the edges of each equivilance partition.

Since bugs tend to cluster at these edges, it is a perfect place to do your testing. While Equivilance Partitioning is a good way to split up your data, Boundary Value Analyis offers a way to select meaningful test data from those partitions.

 

 

Print Friendly
Posted in Uncategorized | 1 Comment

Testing 101 – Equivalence Partitioning

Equivalence Partitioning (EP) is a common software testing technique that seeks to limit the sheer number of tests needed to completely cover a given set of inputs. It is often part of Black Box Testing, as the tester simply bases their tests off the requirements given for the field, instead of evaluating how the software handles the input.

The concept is basically this: Any number or character within any given range of inputs, both valid and invalid, is equivalent to others in the same range (partition).

Example:

  • A field allows only numeric characters
  • Length must be 2-5 characters long

You could could be partition those requirements like this:

             0 1  |   2  3  4  5   | 6  7  8 9
Invalid (too low) |      Valid       |   Invalid (too high)

According to EP the values in each partition are equivalent. So 0 and 1 are equivalent, as are 2 – 5 and then the third partition of 6 and over.

This means that if you were testing this field, you could test 0 and 8 as your invalid entries and 3 as your valid one. The idea is to test the algorithm behind the software by testing a value from each part of the rule, but not all values. This is a good idea in theory.

It makes a lot of sense with larger input fields as well. While it is easy to test inputs 2-5, it is harder to test inputs 2-500. It is necessary when designing test cases to find ways to reduce the number of test cases while still maintaining coverage in order to test effectively and efficiently.  However you must be careful when using this technique, because if you incorrectly assume that each value in the partition is equivalent, or if you miss a partition all together by not understanding the underlying logic, you can miss some key functionality.

My issue with this practice is that if you only test 3 as your valid value, you are trusting that the developer coded the valid range (2-5) in correctly. For example, 3 may always work, but perhaps in this code, 5 is the cut off, but instead of allowing 5 characters, it takes 5 and anything over and rejects it. Logically when looking at requirements, EP makes sense, but in the real world of finding bugs, I find it has some flaws when used stand alone. It is also only as effective as the partitions you decide on. Any error in creating the partitions can lead to lacking coverage.

Example:

  • An input field accepts 2-5 characters
  •  If the input is 2 or 3 it outputs message A
  •  If it the input is 4 or 5 it outputs message B (for example, it may be alerting you to the security of a password, and the longer passwords are deemed more secure.)

So when you start testing with Equivalence Partitioning, you see that values 2-5 are valid and then you partition accordingly, just like we did above.

          0   1 |   2   3    4   5    |   6   7   8  9
Invalid (too low) |      Valid       |   Invalid (too high)

With the above rules, you are never testing Message A. You need further partitioning, like this:

      -2  -1  0 1 |    2      3      |     4      5     | 6  7  8 9
Invalid (too low)|Valid (message A)|Valid (message B)|Invalid (too high)

So to test fully, you would need to select values from each of 4 partitions, 2 invalid entries and 2 valid entries.

When using this practice, be sure you both understand the full meaning of the requirements so you don’t miss functionality, and be sure you have considered if your selections from each partition are really offering you complete coverage.

To help with some of these risk areas, I would always suggest using Boundary Value Analysis as well when testing with Equivalence Partitioning.

Print Friendly
Posted in Uncategorized | 1 Comment

When the Tester is the Coder

I spent some time writing and testing my own scripts recently. I was struck by the drastic shift in mindset I had to have to become a tester again after spending time coding.

You are creative both times. First, when you write the script, even as you do basic debugging, you are thinking about what it should do. You think about how it could do it faster, or more efficiently. You worry about meeting all the requirements. You hope your test data is sufficient. Then it comes time for real testing…

Suddenly, you have to step back. You have to change your perspective. While I do want to verify that it does what it is supposed to do, I also want to verify that it doesn’t do what it isn’t supposed to. Thats a different type of creativity. You think about what someone who has never seen the requirements might try to do. You imagine what outside forces might cause it to fail- a browser or server crashes, a computer shuts down, a password is forgotten etc. Try to do two things at once. You try to do things slowly, quickly, and in different environments. You check for XSS and hacks.

In all – it makes you use a different part of your brain, a different thought process and in moving from coder to tester, it takes a conscious effort to stop being one and start being the other. And you know what? I found different bugs, in my own code as a “tester” than I could as a “coder” – bugs I never would have found if  I hadn’t switched my perspective. It really two roles, two jobs. Done by the same person or different people – a good project will have a QA role and a Programmer role.

Print Friendly
Posted in Uncategorized | Leave a comment

I enjoy testing because…

1. Testing is a puzzle.There is no one right way to break a system, there are countless ways. One way for every users and every hardware/software/browser configuration out there. You get to be each of those people, if only for a moment as you design their test case.

2. Testing is hard. Done right, really testing an application should be hard. It should make you think, make you frustrated, make you figure things out.

3. Testing is enlightening. Not just to you, to developers on the team too! You see how parts work together, find errors you never would have thought of if you hadn’t been searching every corner of the app and developers see the effects of their code on the users, they see its weak spots and trouble areas.

4. Testing is the first exercise of the system. Its like taking a new model car out for spin. You get to explore it and put it through its paces make suggestions as the site develops.

5. Testing is both creative and logical. There is basic logic patterns to follow- boundary testing, Equivalence Partitioning etc – but there is room for creativity, human error and curiosity. The best bugs often lay right where those two overlap.

6. Testing makes you see things differently. If you test your own code- you have to get your mind out of the building philosophy, step back and come at it from a different angle. You have to ask what doesn’t work, not just what does. You don’t prove you’ve made it correctly, you prove you haven’t left anything out. Somehow, it always makes me a little proud when I test my own code. You get busy writing it, but in testing it, you get a chance to really see it.

7. Testers need to be something to everyone. Advocates for the users, editors for the developers, verification for management, use case developers for clients and reality checks for project managers. Sometimes- even trainers for new users, technical guides for other QAs and coders of our own tests. It is such a varied and interesting position.

 

Why do you like testing? What excited you about your role and keeps you engaged with your field?

 

Print Friendly
Posted in Uncategorized | 8 Comments

What technology can learn from the humanities.

Much has been written about what non-engineers in tech firms can learn from computer science. Learn to program and script, understand basic architecture, learn to speak the engineers’ language (because, lets face it, there is one). But in all the clamor for technology companies to learn more tech, hire engineers in test instead of test analysts, have designers who also program and project managers with computer science majors, let us take a moment to consider what computer science could learn from the humanities.

In my first job, there was a push to hire people without engineering degrees. Why? Because they solved problems differently than engineers did, and thus, they looked at problems a little differently. For a QA – this led them to different bugs than other people would find. However, I have seen this same phenomenon in my other companies as well. So what does the study of humanities really have to offer a modern technological firm?  The answer is in the approach to thinking, collaborating, and to life.

To paraphrase what Alan Cooper so eloquently put- engineers feel at home in the state of problem solving and tinkering that breeds instant frustration in most people. In firms that are too one sided (toward the engineering side) cool new technology, endless iterations to pare down the code, and endless API options can obscure what they are really trying to do- make something easier. But user experience isn’t the only thing to learn from humanities.

Collaborative approach. Pair programming is gaining in popularity, and it is a great start. But the humanities has encouraged discussions, team work and communication from the beginning. English majors know how to smooth out the gaps in a paper to make it seem like it was written by one person. How many design specs were clearly written by more than one person? That contradict themselves or don’t take other sections into account?

Holistic problem solving. Engineering encourages you to get down in the weeds, and focus on minute problems. Humanities teaches you to step back, gleen the whole landscape and see the issue in perspective. That mindset may not clear a compiling error, but it may solve an error about the way the features are configured or what features are needed at all, or why they work together they do.

The humanities teach you to challenge the question when you face a problem. Are you asking the right question or solving the right problem? Are you too down in the weeds to see the big picture? Is there a different way to do this? Do you really need to do this? Sometimes just that conversation can pull people up to a different plane of thought and an answer appears.

Beauty. Thats right, beauty. Artists of course, think about the aesthetics of their designs and artwork but the humanities as a whole put a great value in the beauty of what they create. From a well written scholarly paper to the clear tone of high C in an aria, students of the humanities will keep working and reworking not just until something is accurate, but until it is beautiful. In a world of computers that is starting to wake up to the need for interaction design and UX testing, it would behoove engineers to take note from the emphasis the humanities put on beauty.

A team too heavily weighted toward one side or the other is weaker than one with a good balance. While the non-engineers in the company are reading basic programming books or cramming on technical terms and concepts in order to communicate more effectively, the engineers should consider looking to the humanities to help round out their team and their skills. Because really, learning to speak in a common language is one of the core drives not only of business (and technical business especially) but also of the humanities.

Print Friendly
Posted in Mindset, roles, Teamwork | Leave a comment

Quality in Training

When the dust settles from sprints, milestones and launches, there is still a phase of the release cycle that is uncompleted. That is the user’s first encounter with the program. When I worked with websites, there was no formal phase for this. Users could explore and express their reactions through various outlets (twitter, forums, emails) and we would eventually hear the feedback. There was no learning curve.

On larger, enterprise software, there is a different story. A formal training and educational period exists to introduce the new users to the program, mentor them, and send them on their way.

This means:

a) There is a lot more feedback and a chance to see how users react to tasks and functionality

b) You have more control over a user’s first experience

c) Since the users were not the ones who chose to purchase and use the software, there can be some hesitancy.

A good training course can use all of those points to both collect feedback and ensure that the transition goes as smoothly as possible- even gets users excited about some of the new functionality.

What makes for good training? What gets users engaged and learning, while providing practical examples and reference materials? We need to bring the same thought and quality that we bring to developing, configuring and testing software to training new users (or new team members for that matter).

For starters, having trainers who really understand the software is important. Trainers who are excited about the new features and can patiently is important, but the quality of examples is important too. Has anyone found any key requirements for a good training class?

Print Friendly
Posted in Uncategorized | Leave a comment

The Terminal is your friend: shell scripting for non-geeks

Too many smart people are afraid of the command line. Project Managers, QAs, even executives of tech companies stay well clear of anything with that menacing, blinking curser.

There is good news though: The terminal -at a basic level- is easy and certainly nothing to be intimidated by.

Now, I am no sys-admin. I know sys admins who can run four terminals simultaneously and seemingly control a different terminal with each hand (and foot?). However, fearing the command line because power users blow your mind is like being afraid to drive because you could never keep up witha race car driver. But don’t worry, you won’t find that person here. I am not a race car driver or a sys power user. Here you will find, simply, an introduction to the command line and a few, basic tasks – coming from someone who still remembers opening the terminal for the first time.

Maybe you have never needed the to use the terminal before. Maybe you have always been curious (and a little bit in awe) of people who could fire up the window and whisper commands right into their computer. Here is the secret,  for most of what anyone would want to do in the command line, it is simple. Not only does using the command line impress your programmer and sys admin coworkers (thus giving you some street cred) it will also make some basic tasks a little simpler.

So, first things first. Lets get started and introduce the basic navigation tasks. I am using OSX.

1. Open the terminal: it will be the black box with the carrot sign on your dock, or, if you don’t see it, search for terminal in finder.

Yup, that one in the middle, if you have it set in your dock.

Yikes! There it is… the smug blinking cursor, taunting you.

 Actually, that cursor is just blinking as a polite way of asking you what it can do for you. Thats right- you are the boss. Who knew? So, lets introduce ourselves:

Type: echo hello and press enter.

Look! It is even friendly.

Now, here is what you need to do about the command line:

- it will only do exactly what you tell it to do. If you are having trouble, make sure your script is correct. Any misspellings, un uppercase letter?

- There is no undo. Now, this isn’t something to be scared of, it is just something to be aware of.

Lets get started!

The first thing to do is find out where you are.

People use phrases like that a lot – and at first your response might be “I know exactly where I am- sitting here typing in my computer” but that isn’t quite true. You know how when you are reading a book, you may be in the library (or the restroom) but you are actually viewing far off lands? Exactly the same thing with the command line.

Where are you? You are strolling through your computer, viewing the items and files you have in various directories. From here, you can stroll through MyDocuments, or hop over to your Desktop. You can copy a file from your Downloads file into a new folder on your Desktop, or send it over to a different computer that you are directly connected to. You can search through log files and export the errors into a file to view them (I am talking to you, QA’s) or connect up to github (for example) and create new projects or pull the latest release and check it out (thats for you, Project Managers). All of that is no harder than typing in addresses in your internet browser.

Type: pwd. PWD stands for “Print Working Directory” or, in english “tell me where I am right now”

pwd : where am I?

Ok, so now you know that you are in the home folder for user: Arthur. But, you still can’t see anything around you. That is like knowing you are standing in a library, but all the lights are off and you can’t see any of the aisles or any of the books. Get ready for the most used command (that is not scientific, but I am pretty sure it is true).

Type: ls  - which means, list the files in the directory I am in, or “show me what files are here”

ls -or- what can I see in here?

With ‘ls’ you can turn on the lights in the directory you are in. By that I mean, it allows you to see the files around you, just like turning on the lights in that library lets you see the shelves and books around you.

Usually, on your first ‘ls‘ you will see some familiar files ‘Desktop‘ ‘Documents‘ ‘Downloads‘ and ‘Applications‘. So there you are, standing there in your computer looking around at your options and you probably have some idea of what lies behind each of those  ”doors.”

Lets say you want to grab a photo you just downloaded and move it to your desktop.  To do that you need to: move to your download folder, find the file, and copy it to your desktop. That is 3 commands. Nothing to sweat about.

To switch in to your download folder, you need another command (maybe the second most popular): cd.

Step 1-  type: cd Downloads - cd stands for ‘change directory’ or go to that place and by adding the name of the directory you want to go to, you tell it where to change to. You could also have written cd Documents or cd .. – did that look funny? cd .. means, “back up” and it moves you out of the current directory you are in back up one level. If you are on your desktop and you type cd photos then want to get out of that file, you would type ‘cd ..’ and find yourself back in your desktop.

Just like before, you want to ask yourself: Where am I? What can I see here? We already know those commands, in fact, I put the first one up in the screenshot above. Type pwd, like I did above and see that your working directory has changed to /users/Arthur/Downloads.

Step 2: The next thing to do, of course, is to find out what you can see in your new directory. To do that type: ls. Look for the photo in the list of downloaded files (in this case, photo.JPG) There are easier ways to search for things- but for now, just look through the list.

Step 3: Copy the file. Type: cp photo.JPG ../Desktop

This means copy the file photo.jpg back up one directory and over to your desktop.

Now, all you have to do is back out of Downloads yourself, head over to the Desktop and find your file. (cd ../Desktop/ and then ls). Your moved file will appear in the list for your desktop. If you go look at your GUI desktop, it will show up there too.

So, I know what you are thinking. What is the point? At basic levels- like browsing through files and changing your directory- you can do it through the UI (finder window, my computer etc) just the same.  Sure, for something like moving a single file, you can always just drag it yourself, that is what the UI is for. But what if you had to move a whole directory, and instead of moving it to your desktop, you had to move it to a remote machine?

The main reason to do some of the basic things in the command line is to become more comfortable with it. At first, navigating around in the command line feels strange and foreign, but given time, it can actually feel natural.

Using the terminal will make it easier, not only to understand what developers and sys admins are talking about, it will make it easier for you to work with them and even to complete tasks of your own. Being comfortable in the terminal is an important skill for those of us who work with computers because it opens up a whole new set of possibilities.

As you get more comfortable with the basics, trying more complicated things is manageable. It doesn’t take a computer geek to run useful commands in the terminal, but beware, using the terminal just may make you into a geek before you know it.

Print Friendly
Posted in tools | 1 Comment

In support of the Humble Error Message

A familiar beep. A small pop up with a cryptic message indicating a problem has occurred.  The user shuts the error message, maybe tries again, unsure of how to fix it, they grow frustrated and feel hopelessly dumb. In fact, despite using computers daily, many people feel overwhelmed or frustrated by error messages they don’t understand and they start to shut down. Either way, it does nothing to improve the relationship between the user and the computer.

Common error messages make the computer into a policeman, not a teacher. Strict and abrupt. They literally beep at you in a sound much like the whistle on a playground. It makes computers feel like a puzzle at best or, at worst, like a game of chess with the computer as your cunning opponent.  The error message is often a method of propagating that persona. This does not help to instill curiosity or a sense of collaboration in the user. Who would want to work with such an unforgiving partner? A partner that gives little information, is unyielding and offers no advice. A partner that spews out additional data in its own language instead of trying to speak its partner’s.  No one would want to join that team, or hire that helper. Yet we do, and we put up with it over and over. With computers in our pockets and on our nightstands, it is time to improve that relationship.

It is up to us, in the software field, to make the first steps toward reconsiliation. To teach software to appoligize and be helpful as it fails. Computers need to be seen as partners, not opponents.  In testing, early error messages were often a placeholder, with a final message slipped in at the last moment before launch. In building and testing software that is designed for the “go right” path, you rob your users of software that fails just as gracefully as it succeeds.

What is the point in planning the experience of failing? It makes all the difference between successfully solving the problem and getting frustrated or stuck.
A helpful error message is like saying “excuse me” when you step on someone’s foot, or apologizing when you want to help someone but are unable to.
The computer has bad manners, and a improving our error messages is etiquette training for software.

A good error message will empower people. Empower average people- from mommy bloggers to dentists entering information to an old lady on a library computer- to solve their own error and move on, as seamlessly and painlessly as possible. It shouldn’t feel like an error, but rather a guide helping you to do what you want. Maybe, even teaching you how to perform tasks without errors along the way.

Good Error messages should include the following things:

  1.  A (non scary)  notice that an error has occurred
  2.  A brief, not too-technical explanation of what has happened.
  3.  Clear steps for how to resolve the error
  4.  Point the user to more information if it is needed.

Most error message have some of those items, some even have all of them. However, very few achieve the not-scary and not-too-technical qualifications. The error should look different than an alert. People who are bombarded by warnings, messages and notifications all day often just mindlessly close error pop ups. The message should not be simply a scolding “sorry, and the system has encountered an error” (or worse “Error 513″) but should instead be a tool to help you understand what error 513 actually is.

 It should tell you what rule you broke to reach the error- perhaps you entered a wrong password, or file is missing, or the network connection has gone offline. Steps should be given, catered to a novice user, on possible trouble shooting tips (is your capslock on? Did you move a file or is it in your trashcan? Try restarting your router with the following steps etc). It should help you troubleshoot.

I can usually copy the error message and search the internet for it and recieve forums and articles that provide a lot more help than the error message did.  The information is out there, and it should be included in some way in the origional error message itself, thus allowing users to turn to the program for help, instead of the internet.

If you went to your doctor, and he told you were sick, but didn’t tell you how to treat it, you would never go back. If you had query friends and neighbors to see if they had ever resolved a similiar sickness and then keep returning to the doctor until he tells you you are well, you’d be furious. Doctors have to help us, as well as explain in English (not medical speak) what is wrong with us. Why should computer users not get the same courtesy?

When you are building soft Software, or testing it, or writing requirements documentation for it- make sure the error messages are not simply added on at the end. Think about them and try to improve them. Approach them as a user who has not combed the functionality of your software. As someone who is too busy track through paragraphs of garbled technical information to guess at the source of the error. Your error messages should guide people back to the right path, instead of simply telling them they have ventured off it. With thoughtful error messages, try to make your programs poliet, helpful teachers that will work with you to get your tasks done.

 

Print Friendly
Posted in Uncategorized | Leave a comment

Guest Post: Michael Larsen – Who Do You Want To Be Today?

  Today, I am happy to introduce a guest post from Michael Larsen!
For those of you who have not been following his blog (and you should be) he has been writing about his experiences in diving deeper into coding as well as testing.  I was very excited when he agreed to do a guest post here. Read below for an excellent article on his expanding role and what developers and testers can learn from each other.
 
And of course, be sure to check out his fantastic QA blog, TESTHEAD and follow him on twitter @mkltesthead.

 

Who Do You Want To Be Today?
By Michael Larsen
11/04/2011

 

I had an interesting chat a few days ago. I sat down with my director and we discussed my “Success Factors” for the upcoming year. My company was a standalone start-up a year ago. Now we have been acquired by a larger multi-national corporation, and as can be expected, many of the more informal and implicitly understood criteria used in those start-up style evaluations get more formalized. Hey, it happens.
As we were talking, I made some interesting observations. First, most of the areas we were discussing had to do with learning and implementing Ruby, being able to create and maintain a development station (bootstrapping from the ground up), get our release testing to a quicker turnaround, with an emphasis on pending automation performance and resolving spurious failures with a time and percentage goal related to each. As I sat down and discussed these options, I made a list on two sides of a piece of paper. One was development related, and one was testing related. The overwhelming success factor values fell on the development side of the “ledger”.
 
After we discussed these, I leaned back, thought for a second, and said what I’d been thinking for some time now… “so, based on this criteria, your goal is to have me be a Software Development Engineer in Test (SDET)”. He stopped at that, and asked me what I meant. I explained that most of what we were discussing were development initiatives, and initiatives that would require me to get some meat and potatoes development chops together. While I thought that was interesting and relevant, where was the role of testing in this set of initiatives? He made it clear that he hoped I would have a foot in both camps. While yes, he felt the need for more automation was an imperative, he also recognized that Exploratory testing and the ability to do “sapient testing” was also an imperative (thank you, James Bach, I am in the process of spreading that term as far and wide as I can at my company).
 
Now, to be fair, I’m not saying there is anything wrong with this list of initiatives at all. In fact, I think it’s an excellent challenge and I’m thrilled that my development team feels I am up to the challenge. I comment on it merely because this is the first time I’ve actually been encouraged to dive into it as a formal and specific part of my review process. At first glimpse, it might look as though they didn’t understand testing, but that’s not the case at all. My director actually has a lot of testing experience, and our team is based around Test Driven Development (for real, they practice what they preach in that regard). His point was that my Exploratory Testing was well established and appreciated, but without getting more of the mundane testing under control, getting the time to do that level of Exploration necessary will never come.
 
It may interest you to know that I agreed with the assessment of the director and accepted the challenge, but I also threw in a curveball of my own. In return for learning the ropes and getting more up to speed on developing testing frameworks (or more to the point, learning how to do that) I also wanted the opportunity to teach the developers (and other testers in the company) how to look at the non-automated side of testing, the “sapient testing” I am trying to have others understand and become proficient in using. It should be an interesting trade. On one side, I get to become more proficient in TDD, Ruby, Rspec,  Capybara, Cucumber and various other automated technologies. On the other, I will help developers become more familiar with ET, Heuristics and other tools of the trade that will help them see non automated testing in the light it deserves to be seen in. How will this all turn out? That remains to be seen, but I’m excited to learn what I can from the developers. Here’s hoping they’re as excited to see what I can show them.
 
 
Print Friendly
Posted in Uncategorized | Leave a comment