How accurate is Dragon Naturally Speaking 11? Here are some results.

We recently bought Dragon NaturallySpeaking 11 Premium. It’s hard not to got sucked in by their marketing.

Dragon NaturallySpeaking 11 claims on their box that you can just say words and watch them appear “three times faster than typing — with up to 99% recognition accuracy right out of the box.”

This caught our attention because we’re looking for authentic ways to bring technology into the classroom beyond interactive whiteboards and digital projectors:

• As a teacher, Dragon NaturallySpeaking 11 could make you more efficient. For example, we type out a lot of student feedback and if we really could work three times faster, then it means that we could cut down the time it takes us to mark essays and assignments significantly.
• If you’re a student, chances are you don’t type very quickly. If you have a learning disability, you might even have difficulty getting your ideas down on paper. The idea that you can simply speak out your ideas and the computer will capture them and type them out for you could be very useful for some people.
• (Nuance does provide an academic discount on their Dragon NaturallySpeaking 11 Premium to qualified students and teachers. Make sure you read these 10 things to know before you buy the Dragon NaturallySpeaking 11 premium education version.)

But just how accurate is Dragon NaturallySpeaking 11? We’ve been using Dragon NaturallySpeaking 11 every day for the past 17 days.

• How accurate is Dragon NaturallySpeaking after two weeks of work?
• How accurate is it when it comes out-of-the-box?

Before we bought Dragon 11, we did a fair bit of research online. (Originally we bought the student/teacher version of Dragon NaturallySpeaking 11 premium. But, we returned that and eventually bought Dragon NaturallySpeaking 11 premium wireless because it comes bundled with a Bluetooth headset.)

Google brings up a lots of sites repeating this idea that with Dragon NaturallySpeaking (DNS), you can get up to 99% accuracy.

Now that we have Dragon NaturallySpeaking, we’re curious to see how accurate it really is for the average user.

• Why people use the rainbow passage to test speech recognition software
• What we did to test the accuracy of Dragon NaturallySpeaking 11
• Rainbow Passage Sample used to test Dragon NaturallySpeaking
• Results: how accurate is Dragon NaturallySpeaking?
• What did we learn from all of this?

Why people like to use the Rainbow Passage to Test Speech Recognition Software

A few people are using the Rainbow Passage as a standardized way to compare the quality of various headsets and microphones.

The rainbow passage is popular because it apparently contains all the phonemes in the English language in roughly the same proportion as in everyday speech. It is one of the most common standard reading passages used in studying accents, reading comprehension, as a speech exercise and for testing speech recognition software.

Other examples of texts used by linguists include “Comma gets a Cure“, “Arthur the rat“, and “My Grandfather“.

Chuck Runquist, a former senior technical solutions PM for Dragon NaturallySpeaking, chimed in on the forum, agreeing that the rainbow passage was viable for microphone testing because it was, “short, it’s sweet, and it’s linguistically comprehensive” but points out that the overall accuracy of Dragon NaturallySpeaking will be higher the longer a person dictates without pausing.

In addition, the average user generally dictates in short choppy phrases of four words or less. This is not conducive to good accuracy. Learning to dictate in phrases consisting of nine words or more, complete sentences, or even paragraphs will always produce better accuracy. The biggest problem is that there’s a significant difference between reading the text of the document versus free-form dictation. The real accuracy attained by any user is measured by the degree of accuracy that the user gets during free-form dictation. Dragon NaturallySpeaking is specifically designed to be more accurate when reading documents for the reasons specified above.

It is important not to confuse the average user by failing to distinguish between reading a document and free-form dictation. The average user is left in a quandary because they attain a very high degree of accuracy reading the rainbow passage, but find that their accuracy when dictating e-mails or documents is noticeably lower, and even sometimes significantly lower.

In other words, Dragon NaturallySpeaking is most accurate when you’re reading to it from a book or newspaper and less accurate when you’re actually using it for real life stuff (i.e. sending e-mails or writing assignments.)

What We Did to Test the Accuracy of Dragon NaturallySpeaking 11

Out of curiosity, we decided to use the Rainbow Passage as one measure to see how accurate Dragon NaturallySpeaking 11 really was.

(As Runquist points out, there can be a significant difference between the accuracy you get when reading a longer passage, and doing a free-form dictation like writing an assignment. Free-form dictation tends to be choppier because you’re thinking about what to say next. This post was written using Dragon NaturallySpeaking 11 Premium Wireless. If you’re curious to see what kind of accuracy we got with this free-form dictation, go to the bottom of this post.)

It’s been over two weeks since we first started using Dragon NaturallySpeaking 11. We use the speech recognition software a little bit every day and when we get the chance, we use the general training module to read texts so that Dragon can get used to our voice.

Here’s what we did to test Dragon NaturallySpeaking 11:

1. We recorded the rainbow passage using Audacityso that we could use the same audio sample in our tests.In our initial recording, we said out loud all of the commands and punctuation in the rainbow passage. We saved this file as an MP3.We then used Audacity to edit out all of the commands and punctuation. So what we were left with was a simple recording of the rainbow passage. We were curious to see if the punctuation marks help Dragon to understand our text better.

   Here is the version of the rainbow passage that we used to test Dragon NaturallySpeaking 11 (source: http://speechempoweredcomputing.co.uk/Newsletter/?p=69 ):

   The Rainbow Passage

   When sunlight strikes raindrops in the air, they act like a prism and form a rainbow. The rainbow is a division of white light into many beautiful colors. These take the shape of a long round arch, with its path high above, and its two ends apparently beyond the horizon. There is, according to legend, a boiling pot of gold at one end. People look, but no one ever finds it. When a man looks for something beyond his reach, his friends say he is looking for a pot of gold at the end of the rainbow.

   Throughout the centuries men have explained the rainbow in various ways. Some have accepted it as a miracle without physical explanation. To the Hebrews it was a token that there would be no more universal floods. The Greeks used to imagine that it was a sign from the gods to foretell war or heavy rain. The Norse men consider the rainbow as a bridge over which the gods passed from Earth to their home in the sky. Other men have tried to explain the phenomena physically. Aristotle thought that the rainbow was caused by a reflection of the sun’s rays by the rain. Since then physicists have found that it is not reflection, but refraction by the raindrops which causes the rainbow. Many complicated ideas about the rainbow have been formed. The difference in the rainbow depends considerably upon the size of the water drops, and the width of the colored band increases as the size of the drops increases. The actual primary rainbow observed is said to be the effect of super position of a number of bows. If the red of the second bow falls upon green of the first, the result is to give a bow with an abnormally wide yellow band, since red and green lights when mixed formed yellow. This is a very common type of bow, one showing mainly red and yellow, with little or no green or blue.

2. We used the Plantronics Calisto headset (Bluetooth) that came bundled in the Dragon NaturallySpeaking 11 premium wireless edition. (According to their hardware compatibility chart, the Calisto headset scores 4 out of 5.)The premium version allows you to transcribe a digital recording or MP3 file. (The home version does not. Check out their edition comparison chart and the feature matrix PDF for more information.)
3. For the purposes of our test, we used our user profile with the Calisto Bluetooth headset that we normally used when working on the computer. (We didn’t create a new user profile for a different audio source to transcribe the MP3 files.)Although the premium version also allows you create multiple user profiles based on different audio sources, we were a little confused about how that worked. We know that Dragon learns when we correct our mistakes, but if you’re using your user profile for your Bluetooth headset, does it remember all of the mistakes that you corrected when you used your user profile for your digital recorder? Or your USB headset?
4. We created a brand-new user profile using the same audio set up that we currently have to see how accurate Dragon NaturallySpeaking was in the beginning. It took us around 12 min. to create this profile.The only training that we did with this brand-new user profile was read the initial training text, “What to Expect from Speech Recognition.” It took 5 min. to read this text (with prompting), and the rest of the time was spent giving basic information about ourselves, and waiting for Dragon to prepare the user profile.
5. We used the comparison tools in Microsoft Word 2007 to find differences between the original rainbow passage text, and the transcription that Dragon produced.
6. We did not correct any mistakes Dragon NaturallySpeaking 11 made when transcribing the Rainbow Passage so that it would not learn from its mistakes. When we were writing this post, we copy and pasted the Rainbow Passage from the source so that we would not have to correct any mistakes to avoid accidentally training Dragon.

Results: How Accurate Is Dragon NaturallySpeaking 11

A few notes about the results:

If you click on the images, you can see the kinds of mistakes that Dragon NaturallySpeaking 11 made.

• Word errors are highlighted in yellow
• Punctuation or capitalization errors are in blue
• Mistakes that are not actually mistakes are in gray. (i.e. different spellings of the same word.)

It’s interesting to see that even though we used a recording of the Rainbow Passage, Dragon NaturallySpeaking thought we said different things based on the punctuation we used, or the amount of training we gave Dragon.

We wanted to show the accuracy rate as things you would want to fix in your document. That’s why we included word errors, as well as punctuation/capitalization errors in the accuracy rate listed in the table.

Here are the results:

User Profile	Test	No. of Word Errors	No. of Capitalization / Punctuation Errors	Total No. of Errors	Accuracy Rate
Brand New User	Rainbow Passage (Dictation only)	8	33	41	87.8%
	Rainbow Passage (Commands and Punctuation)	8	0	8	97.6%
After 17 days of using Dragon NaturallySpeaking 11(Auto-formatting for commas and periods is turned on)	Rainbow Passage (Dictation only)	7	8	15	95.5%
	Rainbow Passage (Commands and Punctuation)	8	0	8	97.6%

The rainbow passage has 335 words in it. With only 5 min. of required training to create a new user profile, Dragon NaturallySpeaking 11 made only eight word errors. That’s a word accuracy of 97.6% for Dragon NaturallySpeaking 11 straight out-of-the-box. If we include capitalization and punctuation errors, then the accuracy drops down to 87.8%

After 17 days of working with the voice-recognition software, Dragon NaturallySpeaking 11 still made around eight word errors, which makes us wonder how much Dragon NaturallySpeaking is actually improving. Then again, this is only a 335 word passage.

What did we learn from all of this?

Dragon NaturallySpeaking 11 seems to be pretty good without any training at all. A word accuracy of 95% means on average having to fix one mistake every 20 words. Most students would be able to live with that (especially if they had someone to help them correct their one mistake every 20 words.)

After 17 days of use (and perhaps more importantly, turning on the auto punctuation feature,) the number of capitalization and punctuation mistakes that Dragon NaturallySpeaking 11 made dropped from 33 mistakes to 8 mistakes when listening to a dictation only version of the Rainbow Passage.

As Runquist pointed out earlier, Dragon NaturallySpeaking performs better when you’re reading to it from a text as opposed to a free-form dictation like when you’re doing a writing assignment.

Having said that, we’ve used Dragon NaturallySpeaking 11 premium wireless to write several posts on this blog. We’ve had a word recognition accuracy rate of over 95% on each of those posts, including this one.

We would like to see if we can get a word recognition accuracy of 99%. That means only one word mistake in 100 words. We’ll see.

This post was written using Dragon NaturallySpeaking 11 Premium Wireless. There 2215 words in this post, but 335 words don’t count because we copy and pasted the Rainbow Passage to avoid training Dragon to understand the rainbow passage. So really, there are 1880 words in this post. Dragon made 60 word errors. So, we had an accuracy of 96.8% in this document. If you include punctuation and capitalization errors, Dragon made, 35 punctuation and capitalization errors. So, we had an accuracy of 94.9% in this document.