Quality in Training

Posted on January 10, 2012 by Devon

When the dust settles from sprints, milestones and launches, there is still a phase of the release cycle that is uncompleted. That is the user’s first encounter with the program. When I worked with websites, there was no formal phase for this. Users could explore and express their reactions through various outlets (twitter, forums, emails) and we would eventually hear the feedback. There was no learning curve.

On larger, enterprise software, there is a different story. A formal training and educational period exists to introduce the new users to the program, mentor them, and send them on their way.

This means:

a) There is a lot more feedback and a chance to see how users react to tasks and functionality

b) You have more control over a user’s first experience

c) Since the users were not the ones who chose to purchase and use the software, there can be some hesitancy.

A good training course can use all of those points to both collect feedback and ensure that the transition goes as smoothly as possible- even gets users excited about some of the new functionality.

What makes for good training? What gets users engaged and learning, while providing practical examples and reference materials? We need to bring the same thought and quality that we bring to developing, configuring and testing software to training new users (or new team members for that matter).

For starters, having trainers who really understand the software is important. Trainers who are excited about the new features and can patiently is important, but the quality of examples is important too. Has anyone found any key requirements for a good training class?

Posted in Uncategorized | Leave a comment

The Terminal is your friend: shell scripting for non-geeks

Posted on December 27, 2011 by Devon

Too many smart people be afraid of the command line. Project Managers, QAs, even executives of tech companies stay well clear of anything with that menacing, blinking curser.

There is good news though: The terminal -at a basic level- is easy and certainly nothing to be intimidated by.

Now, I am no sys-admin. I know sys admins who can run four terminals simultaneously and seemingly control a different terminal with each hand (and foot?). However, fearing the command line because power users blow your mind is like being afraid to drive because you could never keep up witha race car driver. But don’t worry, you won’t find that person here. I am not a race car driver or a sys power user. Here you will find, simply, an introduction to the command line and a few, basic tasks – coming from someone who still remembers opening the terminal for the first time.

Maybe you have never needed the to use the terminal before. Maybe you have always been curious (and a little bit in awe) of people who could fire up the window and whisper commands right into their computer. Here is the secret,  for most of what anyone would want to do in the command line, it is simple. Not only does using the command line impress your programmer and sys admin coworkers (thus giving you some street cred) it will also make some basic tasks a little simpler.

So, first things first. Lets get started and introduce the basic navigation tasks. I am using OSX.

1. Open the terminal: it will be the black box with the carrot sign on your dock, or, if you don’t see it, search for terminal in finder.

Yup, that one in the middle, if you have it set in your dock.

Yikes! There it is… the smug blinking cursor, taunting you.

 Actually, that cursor is just blinking as a polite way of asking you what it can do for you. Thats right- you are the boss. Who knew? So, lets introduce ourselves:

Type: echo hello and press enter.

Look! It is even friendly.

Now, here is what you need to do about the command line:

- it will only do exactly what you tell it to do. If you are having trouble, make sure your script is correct. Any misspellings, un uppercase letter?

- There is no undo. Now, this isn’t something to be scared of, it is just something to be aware of.

Lets get started!

The first thing to do is find out where you are.

People use phrases like that a lot – and at first your response might be “I know exactly where I am- sitting here typing in my computer” but that isn’t quite true. You know how when you are reading a book, you may be in the library (or the restroom) but you are actually viewing far off lands? Exactly the same thing with the command line.

Where are you? You are strolling through your computer, viewing the items and files you have in various directories. From here, you can stroll through MyDocuments, or hop over to your Desktop. You can copy a file from your Downloads file into a new folder on your Desktop, or send it over to a different computer that you are directly connected to. You can search through log files and export the errors into a file to view them (I am talking to you, QA’s) or connect up to github (for example) and create new projects or pull the latest release and check it out (thats for you, Project Managers). All of that is no harder than typing in addresses in your internet browser.

Type: pwd. PWD stands for “Print Working Directory” or, in english “tell me where I am right now”

pwd : where am I?

Ok, so now you know that you are in the home folder for user: Arthur. But, you still can’t see anything around you. That is like knowing you are standing in a library, but all the lights are off and you can’t see any of the aisles or any of the books. Get ready for the most used command (that is not scientific, but I am pretty sure it is true).

Type: ls  - which means, list the files in the directory I am in, or “show me what files are here”

ls -or- what can I see in here?

With ‘ls’ you can turn on the lights in the directory you are in. By that I mean, it allows you to see the files around you, just like turning on the lights in that library lets you see the shelves and books around you.

Usually, on your first ‘ls‘ you will see some familiar files ‘Desktop‘ ‘Documents‘ ‘Downloads‘ and ‘Applications‘. So there you are, standing there in your computer looking around at your options and you probably have some idea of what lies behind each of those  ”doors.”

Lets say you want to grab a photo you just downloaded and move it to your desktop.  To do that you need to: move to your download folder, find the file, and copy it to your desktop. That is 3 commands. Nothing to sweat about.

To switch in to your download folder, you need another command (maybe the second most popular): cd.

Step 1-  type: cd Downloads - cd stands for ‘change directory’ or go to that place and by adding the name of the directory you want to go to, you tell it where to change to. You could also have written cd Documents or cd .. – did that look funny? cd .. means, “back up” and it moves you out of the current directory you are in back up one level. If you are on your desktop and you type cd photos then want to get out of that file, you would type ‘cd ..’ and find yourself back in your desktop.

Just like before, you want to ask yourself: Where am I? What can I see here? We already know those commands, in fact, I put the first one up in the screenshot above. Type pwd, like I did above and see that your working directory has changed to /users/Arthur/Downloads.

Step 2: The next thing to do, of course, is to find out what you can see in your new directory. To do that type: ls. Look for the photo in the list of downloaded files (in this case, photo.JPG) There are easier ways to search for things- but for now, just look through the list.

Step 3: Copy the file. Type: cp photo.JPG ../Desktop

This means copy the file photo.jpg back up one directory and over to your desktop.

Now, all you have to do is back out of Downloads yourself, head over to the Desktop and find your file. (cd ../Desktop/ and then ls). Your moved file will appear in the list for your desktop. If you go look at your GUI desktop, it will show up there too.

So, I know what you are thinking. What is the point? At basic levels- like browsing through files and changing your directory- you can do it through the UI (finder window, my computer etc) just the same.  Sure, for something like moving a single file, you can always just drag it yourself, that is what the UI is for. But what if you had to move a whole directory, and instead of moving it to your desktop, you had to move it to a remote machine?

The main reason to do some of the basic things in the command line is to become more comfortable with it. At first, navigating around in the command line feels strange and foreign, but given time, it can actually feel natural.

Using the terminal will make it easier, not only to understand what developers and sys admins are talking about, it will make it easier for you to work with them and even to complete tasks of your own. Being comfortable in the terminal is an important skill for those of us who work with computers because it opens up a whole new set of possibilities.

As you get more comfortable with the basics, trying more complicated things is manageable. It doesn’t take a computer geek to run useful commands in the terminal, but beware, using the terminal just may make you into a geek before you know it.

Posted in tools | Leave a comment

In support of the Humble Error Message

Posted on December 9, 2011 by Devon

A familiar beep. A small pop up with a cryptic message indicating a problem has occurred.  The user shuts the error message, maybe tries again, unsure of how to fix it, they grow frustrated and feel hopelessly dumb. In fact, despite using computers daily, many people feel overwhelmed or frustrated by error messages they don’t understand and they start to shut down. Either way, it does nothing to improve the relationship between the user and the computer.

Common error messages make the computer into a policeman, not a teacher. Strict and abrupt. They literally beep at you in a sound much like the whistle on a playground. It makes computers feel like a puzzle at best or, at worst, like a game of chess with the computer as your cunning opponent.  The error message is often a method of propagating that persona. This does not help to instill curiosity or a sense of collaboration in the user. Who would want to work with such an unforgiving partner? A partner that gives little information, is unyielding and offers no advice. A partner that spews out additional data in its own language instead of trying to speak its partner’s.  No one would want to join that team, or hire that helper. Yet we do, and we put up with it over and over. With computers in our pockets and on our nightstands, it is time to improve that relationship.

It is up to us, in the software field, to make the first steps toward reconsiliation. To teach software to appoligize and be helpful as it fails. Computers need to be seen as partners, not opponents.  In testing, early error messages were often a placeholder, with a final message slipped in at the last moment before launch. In building and testing software that is designed for the “go right” path, you rob your users of software that fails just as gracefully as it succeeds.

What is the point in planning the experience of failing? It makes all the difference between successfully solving the problem and getting frustrated or stuck.
A helpful error message is like saying “excuse me” when you step on someone’s foot, or apologizing when you want to help someone but are unable to.
The computer has bad manners, and a improving our error messages is etiquette training for software.

A good error message will empower people. Empower average people- from mommy bloggers to dentists entering information to an old lady on a library computer- to solve their own error and move on, as seamlessly and painlessly as possible. It shouldn’t feel like an error, but rather a guide helping you to do what you want. Maybe, even teaching you how to perform tasks without errors along the way.

Good Error messages should include the following things:

  1.  A (non scary)  notice that an error has occurred
  2.  A brief, not too-technical explanation of what has happened.
  3.  Clear steps for how to resolve the error
  4.  Point the user to more information if it is needed.

Most error message have some of those items, some even have all of them. However, very few achieve the not-scary and not-too-technical qualifications. The error should look different than an alert. People who are bombarded by warnings, messages and notifications all day often just mindlessly close error pop ups. The message should not be simply a scolding “sorry, and the system has encountered an error” (or worse “Error 513″) but should instead be a tool to help you understand what error 513 actually is.

 It should tell you what rule you broke to reach the error- perhaps you entered a wrong password, or file is missing, or the network connection has gone offline. Steps should be given, catered to a novice user, on possible trouble shooting tips (is your capslock on? Did you move a file or is it in your trashcan? Try restarting your router with the following steps etc). It should help you troubleshoot.

I can usually copy the error message and search the internet for it and recieve forums and articles that provide a lot more help than the error message did.  The information is out there, and it should be included in some way in the origional error message itself, thus allowing users to turn to the program for help, instead of the internet.

If you went to your doctor, and he told you were sick, but didn’t tell you how to treat it, you would never go back. If you had query friends and neighbors to see if they had ever resolved a similiar sickness and then keep returning to the doctor until he tells you you are well, you’d be furious. Doctors have to help us, as well as explain in English (not medical speak) what is wrong with us. Why should computer users not get the same courtesy?

When you are building soft Software, or testing it, or writing requirements documentation for it- make sure the error messages are not simply added on at the end. Think about them and try to improve them. Approach them as a user who has not combed the functionality of your software. As someone who is too busy track through paragraphs of garbled technical information to guess at the source of the error. Your error messages should guide people back to the right path, instead of simply telling them they have ventured off it. With thoughtful error messages, try to make your programs poliet, helpful teachers that will work with you to get your tasks done.

 

Posted in Uncategorized | Leave a comment

Guest Post: Michael Larsen – Who Do You Want To Be Today?

Posted on November 7, 2011 by Devon
  Today, I am happy to introduce a guest post from Michael Larsen!
For those of you who have not been following his blog (and you should be) he has been writing about his experiences in diving deeper into coding as well as testing.  I was very excited when he agreed to do a guest post here. Read below for an excellent article on his expanding role and what developers and testers can learn from each other.
 
And of course, be sure to check out his fantastic QA blog, TESTHEAD and follow him on twitter @mkltesthead.

 

Who Do You Want To Be Today?
By Michael Larsen
11/04/2011

 

I had an interesting chat a few days ago. I sat down with my director and we discussed my “Success Factors” for the upcoming year. My company was a standalone start-up a year ago. Now we have been acquired by a larger multi-national corporation, and as can be expected, many of the more informal and implicitly understood criteria used in those start-up style evaluations get more formalized. Hey, it happens.
As we were talking, I made some interesting observations. First, most of the areas we were discussing had to do with learning and implementing Ruby, being able to create and maintain a development station (bootstrapping from the ground up), get our release testing to a quicker turnaround, with an emphasis on pending automation performance and resolving spurious failures with a time and percentage goal related to each. As I sat down and discussed these options, I made a list on two sides of a piece of paper. One was development related, and one was testing related. The overwhelming success factor values fell on the development side of the “ledger”.
 
After we discussed these, I leaned back, thought for a second, and said what I’d been thinking for some time now… “so, based on this criteria, your goal is to have me be a Software Development Engineer in Test (SDET)”. He stopped at that, and asked me what I meant. I explained that most of what we were discussing were development initiatives, and initiatives that would require me to get some meat and potatoes development chops together. While I thought that was interesting and relevant, where was the role of testing in this set of initiatives? He made it clear that he hoped I would have a foot in both camps. While yes, he felt the need for more automation was an imperative, he also recognized that Exploratory testing and the ability to do “sapient testing” was also an imperative (thank you, James Bach, I am in the process of spreading that term as far and wide as I can at my company).
 
Now, to be fair, I’m not saying there is anything wrong with this list of initiatives at all. In fact, I think it’s an excellent challenge and I’m thrilled that my development team feels I am up to the challenge. I comment on it merely because this is the first time I’ve actually been encouraged to dive into it as a formal and specific part of my review process. At first glimpse, it might look as though they didn’t understand testing, but that’s not the case at all. My director actually has a lot of testing experience, and our team is based around Test Driven Development (for real, they practice what they preach in that regard). His point was that my Exploratory Testing was well established and appreciated, but without getting more of the mundane testing under control, getting the time to do that level of Exploration necessary will never come.
 
It may interest you to know that I agreed with the assessment of the director and accepted the challenge, but I also threw in a curveball of my own. In return for learning the ropes and getting more up to speed on developing testing frameworks (or more to the point, learning how to do that) I also wanted the opportunity to teach the developers (and other testers in the company) how to look at the non-automated side of testing, the “sapient testing” I am trying to have others understand and become proficient in using. It should be an interesting trade. On one side, I get to become more proficient in TDD, Ruby, Rspec,  Capybara, Cucumber and various other automated technologies. On the other, I will help developers become more familiar with ET, Heuristics and other tools of the trade that will help them see non automated testing in the light it deserves to be seen in. How will this all turn out? That remains to be seen, but I’m excited to learn what I can from the developers. Here’s hoping they’re as excited to see what I can show them.
 
 
Posted in Uncategorized | Leave a comment

The state of being: software quality

Posted on November 1, 2011 by Devon

Thinking about Quality is pervasive. It stretches outside our chosen fields and becomes something we take with us. Testing software is not a skill you turn on and off in the office. It is one that settles down inside you and changes the way you look at software all together.

Before I started working in QA, I, like many people I still know, overlooked software’s shortcomings. Alan Cooper calls these people ‘apoligists.’ If something doesn’t work correctly, or isn’t designed very well, you muddle through it and maybe feel a little dumb yourself, but thats as far as you get. But working in QA wakes up a different part- a part that questions why software should be allowed out the door in the current condition. You want to know why the bugs you find weren’t important enough to find or fix. You question the design decisions that leave menu choses buried in outdated links or take you way too many clicks to get where you want to go. Quality Assurances stretches outside of the QA field and has you interested in design and user experience, it has you pondering what areas tend to show bugs and if the software engineer may not have paid close enough attention to that particular area. You become interested in programming so that you can understand some of those engineering decisions. You sit in with project managers on client calls so you can understand what the most important features are to the customer.

In short, working in QA sets you at the convergence of some of the most fascinating parts of the software business. It encourages you to learn about all of them, take an interest in all of them and adapt your job duties interact with all of them. I have heard that QA is a so often a jumping point for other jobs. However, perhaps that is because it puts people in such an exciting position and exposes them to so many different areas of the field.

Upon leaving the rigors of daily testing- does that mindset ever leave you? Or did it weave itself through the way you look at software so much that the idea of software quality, software ease of use, settle into you for good?

Posted in Uncategorized | 2 Comments

the 5 senses of testing

Posted on August 25, 2011 by Devon

Ok- maybe not 5- please dont smell your computer screen. Or taste it or that matter. But testing is not simply a visual activity.

It is easy to focus on what you can see: how the computer responds to expected- and unexpected- input, how the interface appears to the user, browser compatibilities, xss tests. But to really test, sometimes you have to remember to use your other senses too.

Once, I was testing a site that would have a live video stream. Before the streams went live, a test stream was broadcast. Everything seemed to run ok, video looked good, any needed bugs were filed. The test stream was a good representation of what the real stream would be like- except this: it did not come with sound. In the rush of preparing for launch- this seemed like something we could work with. We tested a sound stream separately and it worked fine.

Then we launched.

Suddenly, with the real video stream broadcasting with sound, there was such a bad echo you couldn’t hear anything. As it turns out- a second player was loading on the page behind an iframe, hidden completely from view. Since the sound was also off, we could not detect it at all on the page. When the sound came on, however, it was just a few seconds off from the main video and it caused an echo. At the last minute, we had to rip out the iframe and relaunch a modified version of the page.

What this whole experience taught me is that you have to engage all your senses when testing. You need to hear how a feature sounds in its natural environment. You have to feel what it feels like to scroll right on the page to reach the log in box, or to notice if menu options are too close together and make you accidentally click the wrong thing. You have to see if the System Under Test smells cheesy or if it projects a good image and a good user experience. Does it show good taste (in design, in quality, in content, etc)?

Lastly, when everything seems to be working. You have to check in with your sixth sense. Does this page seem like it is ready to go out there? Is this application giving me any nagging uneasiness that I can’t explain? Does something feel off?

Only after carefully consulting all your senses- not just your eyes- can you truly be confident in a quality launch.

Posted in Manual Testing, Mindset, Uncategorized | Leave a comment

The User’s Context

Posted on June 30, 2011 by Devon

All that really matters, in the end, is how satisfied the user is with your product. That is true if you are a project manager, a tester, a developer or an executive. That drives your business, your profits, and gives you that warm and fuzzy, accomplished feeling.

Naturally, we wonder how we can successfully provide exactly what the user wanted in a way they can understand and that feels natural to them. I’ve heard before “the users are idiot, you have to make your site idiot proof.” While I think the intention behind that sentiment comes from the right place, I do not agree with it. Your users are not idiots. They may not be power computer users, but they are busy people: students, professionals, doctors or dentists with patients waiting on them to figure out your feature, and parents- who don’t have the time or energy to sit down and decipher all the cool features of your product. You get paid to immerse yourself in it, they do not- and often, they pay you for the privilege of using your software.

Firstly, if you have a new product, or a new feature ( and have the chance to) – do user testing. It is absolutely worth it as it gives you a fresh sampling of what people really think when they view your site or try to get through a work flow. Even a few minutes on Skype or casually sitting next to people on a computer in the office can make a big difference. If you want a great introduction to user testing, I suggest Don’t Make Me Think by Steve Krug.

However, if you don’t have a chance to do user testing- really putting yourself in the right mindset to evaluate your program/website/application as the user would can make a big difference. You have a list of features or a suite of tests- and it is easy to get lost in the weeds. Your users  want to be able to use your software seamlessly, easily and without maneuvering through clumsy work flows. This requires a context shift. At some point your testing (or designing, etc) should step away from the “verifying all features/determining all features’ stage and into “How does this feel as a user?”

I have done this in two ways recently. For one release, I got everyone in the office to log in to our site every morning and spend 10-15 minutes using it. No rules, nothing to check. Just get on, do one thing and log off. What I got back from that was a surprising list of “this feels awkward” or “I wish I could do this” which was great feedback, but it also switched everyone on the team into a user for a few minutes- and that perspective stuck. The site looks different to a user than to a developer or a tester- even when the user used to be the developer or tester. There were a lot of little things that popped up that people on the team just “hadn’t realized” before. Because before- they were building and testing the site, not using it. Simply wanting to deliver a feature for the users is not the same as actually forcing yourself to be in their position.

The other thing I did was to change my own approach. While doing a user testing session, I realized that when a user came to the site, they approached it differently. They didn’t care if a feature was “cool” they cared if it was something they needed, if it was easy to find and easy to use. So when I sit down now, after doing the usual evaluations, I give myself a minute to think of a user stumbling on the site – and I ask myself – is this just cool? Or does it matter. This approach is good to do when fleshing out features- determine what the root desire of the client is before moving forward. But it is also good to remind yourself of at each stage of the process. There is no point in putting on flashy features if they get in the way of the site. The users don’t care that you get more ad revenue from that pop up ad, but it frustrates them if they can’t close it and get back to reading your content.

In the mine field of business requirements and functionality, it is good to take a moment and put yourself in the user’s context, and really try to deliver what makes sense for them, as let that guide you forward. In the end, it will be good for your company too.

Posted in Mindset | 1 Comment

ApacheBench Vs. Siege

Posted on June 23, 2011 by Devon

I want to like Apache-bench more than I do. Although very simple to use with excellent reporting and widespread support, I keep abandoning it and find myself load testing with Siege. The main reason for this is the use of a url file. With siege, I can quickly and easily hit a file of urls at random, thus getting a more complete test of my app.

ex:
siege -c 100 -t 10m -i -f links.txt
(with the links file containing as many urls as I need)

At first glance, there wasn’t an easy way to do that in Apache-bench. Then I ran into an article describing how to do just that. (and it is easy, no less!)

“ab -n 100 -c 10 http://127.0.0.1:8300/test.cfm > test1.txt &
ab -n 100 -c 10 http://127.0.0.1:8300/scribble.cfm > test2.txt &”

( quoted from article above, published on this site. )

I was totally excited to find a solution to the multiple url issue and ready to give Apache-bench another shot. Unfortunately, the results are reported separately and would need to be aggregated by hand, as far as I can tell. Does any one use Apache-bench for multiple urls and have a solution for the reporting issue? How do you evaluate load data for larger scale load tests?

Posted in Uncategorized | Leave a comment

Selenium Conference talk

Posted on June 1, 2011 by Devon
Lightening talk at Selenium conference 2011

Selenium Conference 2011

The video of my lightning talk with Matt Perry at the Selenium conference is up! It was a quick talk with my coworker about how we use selenium at Agora and the Gondola ruby gem we made to try and bridge the knowledge gap between basic selenium IDE scripts and more in depth selenium RC tests.

It was a topic that I kept coming back to during the conference- which was full of great ideas and interesting topics- but also -full of very advanced Selenium users with out a good representation of people who are just getting started or companies without QA engineers. I came home with a ton of good ideas about what I want to do next in our testing process, but also – a little unsure of how teams with varying degrees of programming skill, with interns, limited time etc- would be able to implement everything we had heard. Gondola had been one of our attempts to make the process a little less overwhelming  – but it certainly is a work in progress and I hope the process will continue evolving as our team does.

 

See the full video for lightening talks as well as other Selenium Conference speakers on Sauce Labs’ youtube page.

Learn more about Gondola here: http://agoragames.github.com/gondola/#1

 

Posted in Automated Testing, Goals, Processes, Software Testing | Tagged , , , , | Leave a comment

Ruby meets the QA team

Posted on March 25, 2011 by Devon

So as I’ve mentioned, I’ve been digging in to ruby these last few months. I think it is worthwhile to evaluate how that has effected the life on the QA team.

We’ve had ruby based Selenium tests, (on on  Sauce Labs via Gondola) for about a year now.  Editing ruby tests is a million times easier than editing the IDE based tests and I’ve been enjoying the benefit of that for months. Actually learning ruby (vs. learning ruby selenium commands) has brought about a new level of benefit and ease though.

Firstly, actually making commits to Gondola is pretty fun. (its error reporting is now in color!) But also, I feel like tweaking the tests, writing them from scratch and finding solutions just comes easier. We are also moving forward with some of the great ideas passed around at the Selenium Conference, like page objects. I know that the conversion to page objects feels a lot less overwhelming and a lot more do-able because of my  growing comfort with ruby.

And of course, there is a lot more I want to do- and a lot of other ways I want to start improving our process. Since I am still a “baby programmer” I am eagerly trying to learn more all the time. I’ve been taking a basic java course this summer at the community college to reinforce what I taught myself and to expose me to a new language while ruby is still fresh.

Has it been an earth shattering change in our testing process? No. The theories of testing, QA methodology, the mindset for building tests does not require any skill with programming. Those techniques are the basis of what we do every day, the core of how we approach our tests and contribute to the team.However, armed with a bit of ruby knowledge, and a growing level of confidence in it, I find it has filled the team with a little dose of creativity, and made me, personally, feel much more excited about digging in to Selenium every day. It has opened the doors to a new wealth of possibilities of things we can do with our testing, provided new ways to apply those core QA methodologies and made more advanced tweaks to our Selenium tests seem approachable.

Posted in Uncategorized | Leave a comment