Is Dragon NaturallySpeaking 12 really more accurate than Dragon NaturallySpeaking 11?

Grade 8 Teacher

12 years ago

Dragon Naturally Speaking premium 12 just came out today, and the official nuance website says the updated speech recognition software is more accurate, straight out-of-the-box compared to Dragon 11. (Apparently, it’s up to a 20% more accurate than Dragon Naturally Speaking 11.) Maybe, but I’m not so sure.

After using Dragon Naturally Speaking (11.5) for 60 blog posts and 74,000+ words, I find the voice recognition engine generally gets around 97.6% of the words correct.

I used the rainbow passage to do a quick check to see if Dragon Naturally Speaking 12 was really better than Dragon Naturally Speaking 11. The cool thing about Dragon Naturally Speaking premium is that you can transcribe recordings from MP3 files. (Only your own voice, sorry!)

I recorded the rainbow passage as an MP3 and included punctuation and commands (i.e. “comma”, “period”, and “new paragraph”) “Recording #1”
I used audacity to trim out the punctuation and commands and I saved this as “Recording #2.”

This way I could be sure that any differences between Dragon 12 and Dragon 11 would be due to the voice-recognition software and not differences in the way that I read the rainbow passage.

I also used two different user profiles to compare Dragon Naturally Speaking 11 and Dragon Naturally Speaking 12.

I created a brand-new user profile in both Dragon Naturally Speaking 11, and Dragon Naturally Speaking 12. I went to the general training process, but I did not let Dragon analyze my emails and documents when I created these profiles.
I used my regular Dragon Naturally Speaking user profile that I’ve been training for almost two years. (I’ve used this voice profile to dictate more than 74,000 words with an average accuracy of 97.6%.) I also upgraded this user profile for Dragon Naturally Speaking 12 when I was testing it.

Finally, I saved each transcription into a plain text file and then I used tortoise merge to compare the original document to what Dragon Naturally Speaking transcribed. Usually, I go through and look at the type of mistakes Dragon Naturally Speaking made to separate word errors from punctuation and capitalization errors, but this time I kept things simple. (Check out the results at the bottom of this post.)

Red errors are incorrect words, or additional words that Dragon Naturally Speaking wrote down.
Green lines are words or punctuation that Dragon Naturally Speaking missed.

Overall, Dragon Naturally Speaking 12 seems less accurate than Dragon Naturally Speaking 11 for both new user and trained voice profiles. Here are the results:

Dragon Naturally Speaking 12 vs Dragon Naturally Speaking 11 Accuracy

The new version of Dragon Naturally Speaking (12) made more mistakes, even straight out-of-the-box (test #1 and test #2.) In fact, when I analyzed my MP3 of the rainbow passage with commands (test #1), Dragon NaturallySpeaking 12 made almost twice as many mistakes as Dragon NaturallySpeaking 11.

It’s also interesting to note that my trained user voice profile makes fewer mistakes than a brand-new profile. I guess Dragon really does improve over time!

When I used my regular profile and the rainbow passage with commands (test #3), Dragon Naturally Speaking made slightly fewer mistakes (5 or 10 mistakes) compared to a brand-new profile (8 or 15 errors.)
If you check out the rainbow passage with no commands (test #4), my regular user profile made less than half the mistakes (23 or 26 mistakes) that a brand-new profile heard with the same recording (60 or 65 mistakes).

Brand-new user profile

Test #1 using Rainbow Passage With Commands (i.e. “comma”, “period”, “new paragraph”) – Recording #1

Dragon Naturally Speaking	Red Errors	Green Errors / Lines	Total Number of Errors
DNS 11	7	1	8
DNS 12	13	2	15

Test #2 using Rainbow Passage With NO Commands – Recording #2

Dragon Naturally Speaking Version	Red Errors	Green Errors / Lines	Total Number of Errors
DNS 11	27	33	60
DNS 12	32	33	65

User Profile after 74,000 words with a general word recognition accuracy around 97.6%

Test #3 using Rainbow Passage With Commands (i.e. “comma”, “period”, “new paragraph”) – Recording #1

Dragon Naturally Speaking	Red Errors	Green Errors / Lines	Total Number of Errors
DNS 11	4	1	5
DNS 12	7	3	10

Test #4 using Rainbow Passage With NO Commands – Recording #2

Dragon Naturally Speaking Version	Red Errors	Green Errors / Lines	Total Number of Errors
DNS 11	12	11	23
DNS 12	15	11	26

A few general comments:

The point of this post is to quickly compare the latest version of Dragon Naturally Speaking to see how good it is. As a previous reader pointed out, some errors that I count are not actually errors:

For example, “Norse men” and “Norsemen” and “super position” and “superposition” are not actually errors, but variations of the same word.
sun’s and sun‘s is also not an error – it’s just the difference between a curly apostrophe and a straight apostrophe.

But given that I used the same recording of the rainbow passage in this study, we can still get a sense of the relative strengths of the voice-recognition software. Dragon Naturally Speaking is still hands-down, the best voice recognition software I’ve played with. I look forward to working with Dragon Naturally Speaking 12 over the next few weeks to see how it improves.

Did you upgrade to Dragon Naturally Speaking 12? How do you like it?

This blog post was dictated using Dragon NaturallySpeaking Premium 12 and Microsoft Word.

There were 678 words in the first draft of this post. This doesn’t include any of the tables or the stuff below this line.
Dragon Naturally Speaking 12 made 23 word errors which mean that it transcribed 96.6% of the words correctly.
The voice recognition software also made an additional 15 punctuation and capitalization errors meaning the total accuracy rate was 94.4%.

Click here to find out more about the Dragon NaturallySpeaking Student / Teacher version.