Measuring satisfaction

I've always felt that post-call customer surveys are methodologically unsound -- or at least as sound as the online poll surveys. The people who bother to take the surveys are self-selecting, and chances are that the people who respond were really happy about the service and were moved enough to trouble themselves with hanging on and providing their feedback, or -- which is more likely -- were really pissed and needed to vent off their frustration.

But there is another layer still that makes these surveys suspect: agent interference. That is, agents selecting which callers to nudge to the survey, and when nudging, trying to influence them to give them good marks.

Check out this article from Service Untitled. It talks about agents "begging" callers to give them good marks!

Such surveys can still be useful if used, for instance, to identify which calls to listen to for training purposes (listen to the ones that got real bad or real good marks). They can also be used to route callers to manager in case the marks entered are really bad, etc.

The Automation Quadrant

Here's a high-level visualization I've come up with of the interplay between task complexity, levels of task automation, and the possible resulting combinations of agent and user satisfaction levels.

#1: An easy task that is automated makes the agent happy because they don't have to deal with mundane, mindless tasks (What is my balance? What's the status of my order?).

#2: The user is happy because they can get through their call quickly, without having to wait for an agent. No waiting and the task is completed quickly.

#3: Trying to automate a very complex task probably means getting rid of agents.

#4: Having users go through complex tasks with an automated system rarely makes the user happy.

#5: The agent is unhappy because they are having to deal with mundane, mindless tasks (what is my balance, what's the status of my order). Their job is demoralizing, soul-deadening.

#6: The user is made to wait for an agent to do something that should take just a few seconds. And they get to interact with a demoralized agent.

#7: The agent is being used for complex tasks that cannot be easily automated: a fulfilling job.

#8: The user is glad that a human is helping them through a difficult task.

Barge-in: Part II

Here are 6 guidelines for when to turn barge-in on or off.

1. Turn off barge-in during the opening prompt

You usually want to turn your barge-in off at the very beginning of your application (unless it is used mainly by repeat users), where you are greeting the user and preparing them for the interaction. The same holds for the beginning of a new section in the call – e.g., a section in a survey.

2. Turn on barge-in when playing a menu prompt

Usually, the barge-in should be turned on while listing options, thus giving the user the ability to speak or enter their selection as soon as they hear it. However, in situations where it is important that the user hear all the options before making their selection, barge-in should be turned off. In either case, it is always a good idea to alert the user that they can either speak as soon as they hear the option they want to select, or that they will need to listen to the whole prompt before making their selection.

3. Turn off barge-in during transitions

Transitions are also sensitive points in an exchange, since they signal both the end of a phase and the beginning of the next phase. You do not want the marking of this important moment in the exchange to be marred by an unintentional barge-in.

4. Turn off barge-in when requesting confirmation

By definition, a systems requests confirmation when the consequences of making a recognition error are significant. Minimize the likelihood of interruptions when requesting confirmation.

5. Turn off barge-in when providing information

Minimize user frustration by reducing the likelihood of having the system be inadvertently interrupted by the user when the prompt being played contains information that is of interest to the user.

6. Turn off barge-in in error prompts

For instance, if a user thinks the system is expecting a date when in fact it is expecting a zip code, switching barge-in off so that the user can hear, "Sorry, I didn't get that. Please give me your zip code," would help the user recover. If barge-in is on, the user is liable to interrupt the prompt after, "Sorry, I didn't get that," and again mistakenly give the application a date.

Barge-in: Part I

Barge-in -- the ability by the user to interrupt a system prompt with voice or DTMF input -- is a very useful tool that the VUI designer can tap into to effectively adapt to various exchange settings the system may need to navigate. The challenge for the VUI designer is determining when to give the user the ability to interrupt and when to take it away from them.

As a rule, you should let the user interrupt system prompts with their input, unless a good reason presents for taking that ability away from them. Three broad parameters need to be taken into consideration when considering turning off barge-in.

User's level of experience with the system

The first and perhaps most obvious factor the designer must consider when debating whether to turn barge-in on or off is the user’s level of experience and familiarity with the application. If the vast majority of users are going to be repeat users and therefore very familiar with the call flow and the options available at any point in the call (e.g., employee check-in/check out), then the default setting for barge-in should be “on”. If the user is going to be a one-time or very infrequent user (e.g., a phone survey), then the barge-in setting should default to “off”.

Call environment

If your application is called from noisy environments (e.g., busy street, factory floor), consider doing two things: first, set the value of your speech recognizer’s sensitivity below the default setting (this will enable the system to tolerate a higher threshold of noise without taking it as input from the user), and second, turn barge-in off, thus at least ensuring that the system will be able to complete playing the prompt to the user.

Conversational context

The third dimension the VUI designer must keep in mind to decide whether to turn barge-in on or off is the conversational context – i.e., where in the structure of the application is the call.

In the next couple of posts, I will list 6 guidelines as to when to turn barge-in on or off.

Help fix Speech Technology Magazine's VUI

This month's SpeechTech Magazine lead editorial by David Myron, the magazine's editor, talks about how the magazine's IVR "sucks" and how Myron was alerted to that fact by a "known speech technology expert."

The expert apparently lambasted Myron by pointing out that the system opened with "a long, rambling monologue that provided virtually no value" and added that it "even had the ‘please listen carefully as our stuff has changed’ nonsense. Surprised that you didn’t have a Web hype or a couple of ‘Your call is important to us’ included. When I pressed 0, the system said that wasn’t valid and then turned around and told me to press 0 to reach an operator."

First, I give David credit for owning up to committing close to an egregious mistake by not making sure that the Magazine's IVR was a showcase of what speech technology can do.

I also give him credit for taking the next step to fix the problem:

[W]e’re turning to the readers of this magazine for help. I encourage independent VUI consultants to call 212-251-0608, navigate through our IVR system, and email me suggestions for improvement. The VUI designer with the best suggestions will be announced in the magazine, win a three-month, full-page ad placement in Speech Technology magazine (our production team will even create the ad for you), and we will also place the winner on our editorial advisory board for the next year. If this isn’t enough, we may even dance around the water cooler and chant your name.

One thing to note here is that David doesn't seem to understand something fundamental about design (VUI or otherwise): the need to talk to the customer and to understand what they want out of the IVR application when designing a system. Instead, he talks about the system as something out there that needs to be tweaked "objectively". I will ping him on this and maybe even offer to host their solution for free at Angel.com.

And by the way, I can bet my bottom dollar that the so-called "known speech technology expert" was none other than Walt Tetschner. He has three or four things he complains about constantly (and correctly I would say), and he touched on three of them here: length of opening prompt, the "please listen as our stuff has changed" and of course, pressing zero and getting to a human. But the real giveaway is the use of the word "nonsense" to describe behavior he doesn't like. A pretty predicatable character, I must say....

Dictionary of VUI Terms

Finally got around doing something I've been meaning to do for a year now: publish an on-line, interactive dictionary of VUI terms.

Check it out at: http://www.lingospace.com/VUI

The dictionary is a work in progress, and I will be feeding a bunch of terms in the next few days. Please feel free to add to it via: http://www.lingospace.com/VUI/addword.asp

Let sentences end in a preposition

If the more natural sounding way of asking a question or providing information has you ending a sentence with a preposition, so be it. Do not twist sentences into weird sounding phraseologies just to satisfy some English grammar rule that, in any case, is almost always disregarded in spoken speech.

Here is a not-so-natural sounding prompt:

System: From where will you be leaving?

Here is a more natural sounding prompt:

System: Where are you leaving from?