Monitor & Analyze

Measuring Success

Acquisition and Retention % of new and returning users (over the last 12 weeks). Active - what is my ‘high potential’ user base? Engaged - how much of my ‘high potential’ user base is actually using my Skill regularly? New users - do I need to do more to promote my Skill?

Usage Total number of sessions, successful sessions vs failed sessions, language used by users (Amazon refers to this as utterances), most popular features (i.e. intents), skill use/hour and peak usage time. See sessions over time and the number of successful/failed sessions.

Interaction Path Analysis Which flows are most popular and the extent to which users complete flows vs drop off which can tell us both about design implementation and the relative usefulness of different features. IPA allows us to analyse behaviour from Skill invocation through to exit via visual representation. We are able to consider behaviours such as: common flows, most popular intents and sticking points where users are dropping off.

Total Unique Customers On its own, total unique customers is a vanity metric which may tell you that your marketing is driving installs of your Skill but doesn’t tell you anything about the utility your Skill is providing to users. When used in conjunction with the other metrics mentioned, unique customers becomes powerful. It can provide an indication of the effectiveness of your marketing.

User Ratings and Reviews The humble rating and reviews system is of course a gold mine for gauging success and prioritising features for future releases. Usually goes one of two ways: “this is rubbish” or “why can’t it do X or Y?”. If your reviews are in the latter vein, your product has been implicitly validated.