At the 2009 Nonprofit Technology Conference, we gave our session evaluation system a major tech overhaul by highlighting a mobile text messaging option in addition to our normal paper evaluations -- as well as an online option that was added at the last minute. At the 2010 NTC in Atlanta, we built on what we learned and took this one step further by moving to a completely paperless evaluation system.
How It Worked
One of our main concerns in implementing a paperless system was excluding attendees who weren't comfortable with text messaging or had limited text plans (a complaint we heard numerous times in 2009), so in addition to offering the Text Messaging option provided by Mobile Commons, we teamed up with Precision Polling to offer a Call-In option and an Online option. To keep the three systems unified and to limit confusion, we assigned a 3-digit identifying code to each of our sessions that could be used for any of the three evaluation methods.
Text: From the user's perspective, this system was almost identical to what attendees saw at the 2009 NTC. "Text [3-digit Eval Code] to 69866" were the only directions needed, at which point the system would text back with the first question in the survey and prompt the user to text back their answer. The user's reply would trigger the next question and so on until the survey was complete (for a total of 5 sent and 5 received text messages). On the back end, Mobile Commons had made several improvements to the already simple system since the 2009 NTC, making the campaign setup much quicker and allowing us even easier access to the results as they came in.
Call-In: This system was surprisingly simple to use both from a user's point of view as well as from an administrator's. We had local Atlanta phone number for attendees to call, at which point they were prompted to enter the Eval Code for whatever session they were evaluating. This was followed by each of the evaluation questions (pre-recorded by NTEN's own Holly Ross) and a prompt for their answers. Behind the scenes, the setup was very intuitive and customizable -- with a simple voice recording system and the ability to add branching based on a user's response -- and accessing the data as it came in couldn't have been easier.
Online: On the backend, this system was nearly identical to the Call-In option, as the two were just different input options for the same Precision Polling system. For the user, they could simply navigate to nten.org/ntc-eval, at which point they would enter the Eval Code for the session they wanted to rate, and then proceed through the rest of the questions in a simple online survey format.
What Worked Well
In 2009, some of the highlights of our Mobile Evaluation experiment were "paper use cut by 95%", "staff time devoted to evaluations cut by 90%", and "no decline in completion rate". By going paperless at the 2010 NTC, as well as by implementing all of the systems well in advance of the conference, we were able to improve on all of these.
Paper use cut by 100%: As "Paperless Evaluations" would suggest, of course, but still worth mentioning.
Staff time devoted to evaluations cut by 100%: If you count the pre-conference setup, this isn't entirely accurate, but during the conference, when NTEN's 6 person staff is really stretched, the removal of paper evaluations combined with improved pre-conference communication with session designers meant there was no additional evaluation setup work required once the conference got going.
Post conference data consolidation cut by 100%: There was still the data evaluation that took the normal amount of time, but we were able to output all the data directly into useable spreadsheets from the different systems.
Evaluation completion rate maintained: Just like in 2009, one of our main concerns with making a major change to our evaluation system was that we would cause confusion and see our completion rate drop significantly, so seeing it instead increase slightly (from 900 to 914) was a great result.
What Didn't Work Well
While the improved pre-conference planning did smooth out a lot of the issues we ran into at the 2009 NTC, we were still left with many areas to improve on.
Lack of buy-in and understanding by speakers: This was probably our biggest issue in 2009, and despite an effort to improve our communication to session designers about the new evaluation system, there were still several sessions where the speakers seemed unaware of the system, and still only a handful of sessions where attendees were actively encouraged to use one of the systems. In fact, just over half the sessions received more than 4 evaluations. At this point, I'm not sure what the solution for this is other than trying to concentrate our efforts on educating our attendees instead.
Directions for submitting evaluations were not widely available: We did a much better job this year making the session eval codes easily available to our attendees, but where we still missed the mark was in successfully publishing the directions on how to use these codes. Our main distribution point for these directions was the session designers, and while this is can be very successful if the speakers help promote the evaluations, it's becoming clear that this can't be our only point of communication with attendees. We also printed the directions in the program book, but unless you read the book cover to cover or were looking for them, they were much too easy to pass over.
Call-In option was mostly ignored: This was actually one of the more surprising results we found. The final numbers for evaluation distribution were 61% Online, 37% by Text, and only 2% by Phone. Comparing to the 2009 NTC distribution (52% Online, 39% Text, and 9% Paper), one interpretation is that the paper evaluations were replaced largely by online submissions, and then a handful of text users gave the Call-In option a try. I'm still a bit stumped though as to why the Call-In option didn't gain much traction as in our testing phases, I found it to be quite easy to use.
Changes Planned for Next Year
Overall, our experience with paperless evaluations was extremely positive: we maintained our evaluation completion rate while at the same time saving paper and minimizing staff time devoted to the program. Of course, now that we have no more room to improve in either the saving paper or minimizing staff time (at the conference), we'll need to start focusing more on improving our completion rate.
Get more buy-in from conference attendees: One of the big lessons learned this year was that while speaker buy-in is a great way to increase completion rate for some sessions, it's not going to be effective for ALL sessions. The only way to improve completion rates across the board will be to generate more awareness of the evaluation systems among our attendees.
Focus on just two systems: While we're still not sure why the Call-In option didn't take off this year, it seems our attendees largely prefer the Online and Text versions of our session evaluations, so in an effort to further streamline and simplify the evaluation process, we will likely just focus on these two systems at the 2011 NTC in DC.
Your Feedback
While we have plenty of lessons learned and ideas for how to improve this system for the 2011 NTC next March, we'd love to hear more direct feedback from NTC attendees (or anyone) on their experience with any of the systems and why they did (or did not) use a specific one. Either leave a comment here or you can send an email to karl@nten.org. Thanks!