More evidence of a consumerist approach to youth policy? Maybe, but the mystery shopper approach has been piloted and has involved a lot of young people and Connexions services in making sure that it's an appropriate form of evaluation.
What form did it take then? There were 21 mystery shoppers aged 14 to 19, all trained in awareness of Connexions Direct and in the techniques of mystery shopping.
Which are? Don't sound like you are reading from a prepared script. Twelve scenarios were developed for the shoppers to seek advice on. Of these, five were about careers and learning, three about relationships, two about health and one each about housing and rights. So they varied from how to become a plumber, to something requiring more nuanced answers such as concerns over a best friend's possible eating disorder. The shoppers were advised how to follow up the initial response and develop the enquiry using their own words.
Which routes did they try? All 12 scenarios were tried across all four of the Connexions Direct services - telephone, live web chat, email and text.
And our survey said? The encouraging thing was that across all formats more than 90 per cent of enquiries were answered with clear advice, with advisers remaining impartial. When it worked well, users said the advisers checked regularly with enquirers that everything was understood and that they were happy. Less positively, one in four enquirers was left feeling they weren't given enough information.
They were just referred on somewhere else? Young people didn't mind that as long as they felt they had been listened to. They didn't like to feel fobbed off. The "canned" response was particularly evident in text messages.
But its hrd 2 b deep if u r ritin like dis.
It's an inherent danger of the form but something for providers to note.
The text service got an overall score of six out of 10 from the shoppers, with web chat seven, and telephone and email eight. Email received very high ratings for impartiality and clarity, although it was quite refreshing to see one young person complaining that their adult adviser's "lack of punctuation made the email more difficult to read".
One cause for concern was that responses were received to only 65 per cent of emails - some further work is being put in to discover whether initial emails were received and if so whether responses were sent but not received.
Why should we care about all this? The Government loves technology. The youth green paper proposes an "innovative information and communications technology service" that will offer advice via email, text and telephone.
It wants to merge services. If Connexions Direct is only answering 65 per cent of emails, that must be sorted out before even more emphasis is put on technology.
FACT BOX
- Advisers were not always consistent in their data collection. In telephone calls, 86 per cent asked the caller's age but other required questions were as low as 51 per cent. Advisers commented that they don't always know how best to ask questions
- In rating how telephone enquiries were handled, 28 out of 139 calls were given 10 out of 10, but two were given one out of 10
- Connexions Direct: Mystery Shopping Evaluation Study can be found in the research section of the web site www.dfes.gov.uk.
