When the dust settles from sprints, milestones and launches, there is still a phase of the release cycle that is uncompleted. That is the user’s first encounter with the program. When I worked with websites, there was no formal phase for this. Users could explore and express their reactions through various outlets (twitter, forums, emails) and we would eventually hear the feedback. There was no learning curve.

On larger, enterprise software, there is a different story. A formal training and educational period exists to introduce the new users to the program, mentor them, and send them on their way.

This means:

a) There is a lot more feedback and a chance to see how users react to tasks and functionality

b) You have more control over a user’s first experience

c) Since the users were not the ones who chose to purchase and use the software, there can be some hesitancy.

A good training course can use all of those points to both collect feedback and ensure that the transition goes as smoothly as possible- even gets users excited about some of the new functionality.

What makes for good training? What gets users engaged and learning, while providing practical examples and reference materials? We need to bring the same thought and quality that we bring to developing, configuring and testing software to training new users (or new team members for that matter).

For starters, having trainers who really understand the software is important. Trainers who are excited about the new features and can patiently is important, but the quality of examples is important too. Has anyone found any key requirements for a good training class?

Too many smart people be afraid of the command line. Project Managers, QAs, even executives of tech companies stay well clear of anything with that menacing, blinking curser.

There is good news though: The terminal -at a basic level- is easy and certainly nothing to be intimidated by.

Now, I am no sys-admin. I know sys admins who can run four terminals simultaneously and seemingly control a different terminal with each hand (and foot?). However, fearing the command line because power users blow your mind is like being afraid to drive because you could never keep up witha race car driver. But don’t worry, you won’t find that person here. I am not a race car driver or a sys power user. Here you will find, simply, an introduction to the command line and a few, basic tasks – coming from someone who still remembers opening the terminal for the first time.

Maybe you have never needed the to use the terminal before. Maybe you have always been curious (and a little bit in awe) of people who could fire up the window and whisper commands right into their computer. Here is the secret,  for most of what anyone would want to do in the command line, it is simple. Not only does using the command line impress your programmer and sys admin coworkers (thus giving you some street cred) it will also make some basic tasks a little simpler.

So, first things first. Lets get started and introduce the basic navigation tasks. I am using OSX.

1. Open the terminal: it will be the black box with the carrot sign on your dock, or, if you don’t see it, search for terminal in finder.

Yup, that one in the middle, if you have it set in your dock.

Yikes! There it is… the smug blinking cursor, taunting you.

 Actually, that cursor is just blinking as a polite way of asking you what it can do for you. Thats right- you are the boss. Who knew? So, lets introduce ourselves:

Type: echo hello and press enter.

Look! It is even friendly.

Now, here is what you need to do about the command line:

- it will only do exactly what you tell it to do. If you are having trouble, make sure your script is correct. Any misspellings, un uppercase letter?

- There is no undo. Now, this isn’t something to be scared of, it is just something to be aware of.

Lets get started!

The first thing to do is find out where you are.

People use phrases like that a lot – and at first your response might be “I know exactly where I am- sitting here typing in my computer” but that isn’t quite true. You know how when you are reading a book, you may be in the library (or the restroom) but you are actually viewing far off lands? Exactly the same thing with the command line.

Where are you? You are strolling through your computer, viewing the items and files you have in various directories. From here, you can stroll through MyDocuments, or hop over to your Desktop. You can copy a file from your Downloads file into a new folder on your Desktop, or send it over to a different computer that you are directly connected to. You can search through log files and export the errors into a file to view them (I am talking to you, QA’s) or connect up to github (for example) and create new projects or pull the latest release and check it out (thats for you, Project Managers). All of that is no harder than typing in addresses in your internet browser.

Type: pwd. PWD stands for “Print Working Directory” or, in english “tell me where I am right now”

pwd : where am I?

Ok, so now you know that you are in the home folder for user: Arthur. But, you still can’t see anything around you. That is like knowing you are standing in a library, but all the lights are off and you can’t see any of the aisles or any of the books. Get ready for the most used command (that is not scientific, but I am pretty sure it is true).

Type: ls  - which means, list the files in the directory I am in, or “show me what files are here”

ls -or- what can I see in here?

With ‘ls’ you can turn on the lights in the directory you are in. By that I mean, it allows you to see the files around you, just like turning on the lights in that library lets you see the shelves and books around you.

Usually, on your first ‘ls‘ you will see some familiar files ‘Desktop‘ ‘Documents‘ ‘Downloads‘ and ‘Applications‘. So there you are, standing there in your computer looking around at your options and you probably have some idea of what lies behind each of those  ”doors.”

Lets say you want to grab a photo you just downloaded and move it to your desktop.  To do that you need to: move to your download folder, find the file, and copy it to your desktop. That is 3 commands. Nothing to sweat about.

To switch in to your download folder, you need another command (maybe the second most popular): cd.

Step 1-  type: cd Downloads - cd stands for ‘change directory’ or go to that place and by adding the name of the directory you want to go to, you tell it where to change to. You could also have written cd Documents or cd .. – did that look funny? cd .. means, “back up” and it moves you out of the current directory you are in back up one level. If you are on your desktop and you type cd photos then want to get out of that file, you would type ‘cd ..’ and find yourself back in your desktop.

Just like before, you want to ask yourself: Where am I? What can I see here? We already know those commands, in fact, I put the first one up in the screenshot above. Type pwd, like I did above and see that your working directory has changed to /users/Arthur/Downloads.

Step 2: The next thing to do, of course, is to find out what you can see in your new directory. To do that type: ls. Look for the photo in the list of downloaded files (in this case, photo.JPG) There are easier ways to search for things- but for now, just look through the list.

Step 3: Copy the file. Type: cp photo.JPG ../Desktop

This means copy the file photo.jpg back up one directory and over to your desktop.

Now, all you have to do is back out of Downloads yourself, head over to the Desktop and find your file. (cd ../Desktop/ and then ls). Your moved file will appear in the list for your desktop. If you go look at your GUI desktop, it will show up there too.

So, I know what you are thinking. What is the point? At basic levels- like browsing through files and changing your directory- you can do it through the UI (finder window, my computer etc) just the same.  Sure, for something like moving a single file, you can always just drag it yourself, that is what the UI is for. But what if you had to move a whole directory, and instead of moving it to your desktop, you had to move it to a remote machine?

The main reason to do some of the basic things in the command line is to become more comfortable with it. At first, navigating around in the command line feels strange and foreign, but given time, it can actually feel natural.

Using the terminal will make it easier, not only to understand what developers and sys admins are talking about, it will make it easier for you to work with them and even to complete tasks of your own. Being comfortable in the terminal is an important skill for those of us who work with computers because it opens up a whole new set of possibilities.

As you get more comfortable with the basics, trying more complicated things is manageable. It doesn’t take a computer geek to run useful commands in the terminal, but beware, using the terminal just may make you into a geek before you know it.

Thinking about Quality is pervasive. It stretches outside our chosen fields and becomes something we take with us. Testing software is not a skill you turn on and off in the office. It is one that settles down inside you and changes the way you look at software all together.

Before I started working in QA, I, like many people I still know, overlooked software’s shortcomings. Alan Cooper calls these people ‘apoligists.’ If something doesn’t work correctly, or isn’t designed very well, you muddle through it and maybe feel a little dumb yourself, but thats as far as you get. But working in QA wakes up a different part- a part that questions why software should be allowed out the door in the current condition. You want to know why the bugs you find weren’t important enough to find or fix. You question the design decisions that leave menu choses buried in outdated links or take you way too many clicks to get where you want to go. Quality Assurances stretches outside of the QA field and has you interested in design and user experience, it has you pondering what areas tend to show bugs and if the software engineer may not have paid close enough attention to that particular area. You become interested in programming so that you can understand some of those engineering decisions. You sit in with project managers on client calls so you can understand what the most important features are to the customer.

In short, working in QA sets you at the convergence of some of the most fascinating parts of the software business. It encourages you to learn about all of them, take an interest in all of them and adapt your job duties interact with all of them. I have heard that QA is a so often a jumping point for other jobs. However, perhaps that is because it puts people in such an exciting position and exposes them to so many different areas of the field.

Upon leaving the rigors of daily testing- does that mindset ever leave you? Or did it weave itself through the way you look at software so much that the idea of software quality, software ease of use, settle into you for good?

Ok- maybe not 5- please dont smell your computer screen. Or taste it or that matter. But testing is not simply a visual activity.

It is easy to focus on what you can see: how the computer responds to expected- and unexpected- input, how the interface appears to the user, browser compatibilities, xss tests. But to really test, sometimes you have to remember to use your other senses too.

Once, I was testing a site that would have a live video stream. Before the streams went live, a test stream was broadcast. Everything seemed to run ok, video looked good, any needed bugs were filed. The test stream was a good representation of what the real stream would be like- except this: it did not come with sound. In the rush of preparing for launch- this seemed like something we could work with. We tested a sound stream separately and it worked fine.

Then we launched.

Suddenly, with the real video stream broadcasting with sound, there was such a bad echo you couldn’t hear anything. As it turns out- a second player was loading on the page behind an iframe, hidden completely from view. Since the sound was also off, we could not detect it at all on the page. When the sound came on, however, it was just a few seconds off from the main video and it caused an echo. At the last minute, we had to rip out the iframe and relaunch a modified version of the page.

What this whole experience taught me is that you have to engage all your senses when testing. You need to hear how a feature sounds in its natural environment. You have to feel what it feels like to scroll right on the page to reach the log in box, or to notice if menu options are too close together and make you accidentally click the wrong thing. You have to see if the System Under Test smells cheesy or if it projects a good image and a good user experience. Does it show good taste (in design, in quality, in content, etc)?

Lastly, when everything seems to be working. You have to check in with your sixth sense. Does this page seem like it is ready to go out there? Is this application giving me any nagging uneasiness that I can’t explain? Does something feel off?

Only after carefully consulting all your senses- not just your eyes- can you truly be confident in a quality launch.

All that really matters, in the end, is how satisfied the user is with your product. That is true if you are a project manager, a tester, a developer or an executive. That drives your business, your profits, and gives you that warm and fuzzy, accomplished feeling.

Naturally, we wonder how we can successfully provide exactly what the user wanted in a way they can understand and that feels natural to them. I’ve heard before “the users are idiot, you have to make your site idiot proof.” While I think the intention behind that sentiment comes from the right place, I do not agree with it. Your users are not idiots. They may not be power computer users, but they are busy people: students, professionals, doctors or dentists with patients waiting on them to figure out your feature, and parents- who don’t have the time or energy to sit down and decipher all the cool features of your product. You get paid to immerse yourself in it, they do not- and often, they pay you for the privilege of using your software.

Firstly, if you have a new product, or a new feature ( and have the chance to) – do user testing. It is absolutely worth it as it gives you a fresh sampling of what people really think when they view your site or try to get through a work flow. Even a few minutes on Skype or casually sitting next to people on a computer in the office can make a big difference. If you want a great introduction to user testing, I suggest Don’t Make Me Think by Steve Krug.

However, if you don’t have a chance to do user testing- really putting yourself in the right mindset to evaluate your program/website/application as the user would can make a big difference. You have a list of features or a suite of tests- and it is easy to get lost in the weeds. Your users  want to be able to use your software seamlessly, easily and without maneuvering through clumsy work flows. This requires a context shift. At some point your testing (or designing, etc) should step away from the “verifying all features/determining all features’ stage and into “How does this feel as a user?”

I have done this in two ways recently. For one release, I got everyone in the office to log in to our site every morning and spend 10-15 minutes using it. No rules, nothing to check. Just get on, do one thing and log off. What I got back from that was a surprising list of “this feels awkward” or “I wish I could do this” which was great feedback, but it also switched everyone on the team into a user for a few minutes- and that perspective stuck. The site looks different to a user than to a developer or a tester- even when the user used to be the developer or tester. There were a lot of little things that popped up that people on the team just “hadn’t realized” before. Because before- they were building and testing the site, not using it. Simply wanting to deliver a feature for the users is not the same as actually forcing yourself to be in their position.

The other thing I did was to change my own approach. While doing a user testing session, I realized that when a user came to the site, they approached it differently. They didn’t care if a feature was “cool” they cared if it was something they needed, if it was easy to find and easy to use. So when I sit down now, after doing the usual evaluations, I give myself a minute to think of a user stumbling on the site – and I ask myself – is this just cool? Or does it matter. This approach is good to do when fleshing out features- determine what the root desire of the client is before moving forward. But it is also good to remind yourself of at each stage of the process. There is no point in putting on flashy features if they get in the way of the site. The users don’t care that you get more ad revenue from that pop up ad, but it frustrates them if they can’t close it and get back to reading your content.

In the mine field of business requirements and functionality, it is good to take a moment and put yourself in the user’s context, and really try to deliver what makes sense for them, as let that guide you forward. In the end, it will be good for your company too.

I want to like Apache-bench more than I do. Although very simple to use with excellent reporting and widespread support, I keep abandoning it and find myself load testing with Siege. The main reason for this is the use of a url file. With siege, I can quickly and easily hit a file of urls at random, thus getting a more complete test of my app.

ex:
siege -c 100 -t 10m -i -f links.txt
(with the links file containing as many urls as I need)

At first glance, there wasn’t an easy way to do that in Apache-bench. Then I ran into an article describing how to do just that. (and it is easy, no less!)

“ab -n 100 -c 10 http://127.0.0.1:8300/test.cfm > test1.txt &
ab -n 100 -c 10 http://127.0.0.1:8300/scribble.cfm > test2.txt &”

( quoted from article above, published on this site. )

I was totally excited to find a solution to the multiple url issue and ready to give Apache-bench another shot. Unfortunately, the results are reported separately and would need to be aggregated by hand, as far as I can tell. Does any one use Apache-bench for multiple urls and have a solution for the reporting issue? How do you evaluate load data for larger scale load tests?

Selenium Conference 2011

The video of my lightning talk with Matt Perry at the Selenium conference is up! It was a quick talk with my coworker about how we use selenium at Agora and the Gondola ruby gem we made to try and bridge the knowledge gap between basic selenium IDE scripts and more in depth selenium RC tests.

It was a topic that I kept coming back to during the conference- which was full of great ideas and interesting topics- but also -full of very advanced Selenium users with out a good representation of people who are just getting started or companies without QA engineers. I came home with a ton of good ideas about what I want to do next in our testing process, but also – a little unsure of how teams with varying degrees of programming skill, with interns, limited time etc- would be able to implement everything we had heard. Gondola had been one of our attempts to make the process a little less overwhelming  – but it certainly is a work in progress and I hope the process will continue evolving as our team does.

See the full video for lightening talks as well as other Selenium Conference speakers on Sauce Labs’ youtube page.

Learn more about Gondola here: http://agoragames.github.com/gondola/#1

So as I’ve mentioned, I’ve been digging in to ruby these last few months. I think it is worthwhile to evaluate how that has effected the life on the QA team.

We’ve had ruby based Selenium tests, (on on  Sauce Labs via Gondola) for about a year now.  Editing ruby tests is a million times easier than editing the IDE based tests and I’ve been enjoying the benefit of that for months. Actually learning ruby (vs. learning ruby selenium commands) has brought about a new level of benefit and ease though.

Firstly, actually making commits to Gondola is pretty fun. (its error reporting is now in color!) But also, I feel like tweaking the tests, writing them from scratch and finding solutions just comes easier. We are also moving forward with some of the great ideas passed around at the Selenium Conference, like page objects. I know that the conversion to page objects feels a lot less overwhelming and a lot more do-able because of my  growing comfort with ruby.

And of course, there is a lot more I want to do- and a lot of other ways I want to start improving our process. Since I am still a “baby programmer” I am eagerly trying to learn more all the time. I’ve been taking a basic java course this summer at the community college to reinforce what I taught myself and to expose me to a new language while ruby is still fresh.

Has it been an earth shattering change in our testing process? No. The theories of testing, QA methodology, the mindset for building tests does not require any skill with programming. Those techniques are the basis of what we do every day, the core of how we approach our tests and contribute to the team.However, armed with a bit of ruby knowledge, and a growing level of confidence in it, I find it has filled the team with a little dose of creativity, and made me, personally, feel much more excited about digging in to Selenium every day. It has opened the doors to a new wealth of possibilities of things we can do with our testing, provided new ways to apply those core QA methodologies and made more advanced tweaks to our Selenium tests seem approachable.