The risk of choosing the best

It would seem to be common sense that given a choice of several options, you should choose the best one. But, the determination of ‘best’ is often a tricky business.

When I was selecting a survey vendor for my employer, SurveyGizmo rose to the top pretty quickly. It had the features and the plan that made sense for us. Really it was not much of a contest – they had features the competitors just didn’t offer.

Right choice, right? Well maybe not.

If I choose them because they are the most desirable, then so will everyone else. In the world of web-based services where service is offered at some level for free, choosing the best means taking the risk that you may not have the product available when you need it.

As the best choice is discovered, the load on their resources goes up, and availability of the product goes down. Today I find that I often cannot get to my data in a timely manner, sometimes can’t even get to the site, and the response from SurveyGizmo is that server response is getting slower during business hours and I should run reports in the evening.

While this phenomenon isn’t unique to the web or cloud offerings, it’s in this industry where it is felt most sharply.

More on SurveyGizmo vs. LimeSurvey

After using SurveyGizmo for quite a long time, I have some more insight to offer on it. Truth be told, I’m getting very tempted to jump to LimeSurvey. Here’s why:

  • Lack of access to the database. This is HUGE. The only way to get data out of SG is to download a csv file. The problem is that they format it in crosstab fashion – that is, with a column for every question. This makes it unusable for any database application. So if you’re planning to pull the results into another database, well, you end up writing some software to do pretty non-trivial conversions.
  • No support for recurring surveys. At least, not without a lot of extra steps. Most of the surveys I do are recurring, and the results are analyzed as time series. Having them in separate surveys is a pain.
  • SG has a strange architecture that leads to some performance problems that confuse users. The surveys are by default save-continue, but if you leave and come back too soon it looks like your results are gone. Only they aren’t.

Out of the box SG has some advantages that are making the choice to jump a little less clear:

  • The reporting, while riddled with oddities, is competent and allows me to do one-off surveys quickly without any database work.
  • Some question types work better, nearly all look better.
  • I don’t have to administer a server that’s ‘off the grid’. Like many companies, ours is Microsoft-only and having a Linux server goes against the grain.

SurveyGizmo vs. LimeSurvey

We recently automated surveys at my employer. For many years we sent word documents out to our membership, who would edit them and either email or fax them back to us. With the current economy some positions were eliminated, including the person who did all of this survey work. This meant the time for automation was at hand, but how?

The first choice is a survey system that would be integrated into our other business systems. However this is a complex undertaking and will take some time to develop. In the mean time I still needed something that would get the job done.

After much searching, I settled on SurveyGizmo and Limesurvey. Both have similar features, but in the end I chose SurveyGizmo mainly because it was hosted, which avoided additional load on our IT staff. Limesurvey is open source, is web-based, and runs on Apache/PHP/MySQL. We don’t have any servers able to provide this environment, so running Limesurvey entailed extra complexity.

I have spent quite a bit of time in SurveyGizmo, and while I’m mostly happy with my choice there are a few things that anyone else in the similar position should consider:

Email invitations don’t do any good unless they reach their recipient. The first thing I learned about SurveyGizmo is that our email filters blocked their emails. We can unblock them for us, but what about our members? With this observation in mind I contacted SG to find out what could be done to ensure emails would get through. Their solution is to create an SPF record for our domain that would authorize their servers as senders of email on our behalf, but there are reasons why this isn’t such a peachy idea.

The invitations sent from Limesurvey didn’t have trouble getting through.

We actually send the invitations through a different system in practice, so it is less of an issue for us but that means losing a lot of neat functionality for doing reminders and tracking response.

With Limesurvey you have direct database access to the data, and this is awesome if you use the data to create custom reports, time series across many surveys, or combinations of survey results. SurveyGizmo doesn’t allow this, and their data import is strictly in cross-tab format. That is, multiple columns for each question. This works if you don’t intend to put the data into a database, but really, really sucks if you do. I’m working around it with a series of Excel macros to convert the data, but I don’t want to do this long term. SurveyGizmo has quoted providing the data in a different format, but the fees involved exceed those for an integrated survey system. Not a reasonable choice!

The ability to enter a survey on behalf of an invitee – say one of the people who just loves to fax the results in – is much easier with Limesurvey. In SG you have to download the entire list of invitees to get the survey link, paste that into a browser, and then take the survey. Not a huge deal, but it could be a factor if you have a lot of manual returns.

It’s not all bad though.

SurveyGizmo has phone support, a much nicer interface, much clearer documentation, and snazzier question styles. Ranking, in particular, is spiffy with little up/down arrows for each choice.

Piping survey questions or pages of questions from a single question, which allows parts to be repeated multiple times depending on previous answers, is easy to understand and implement.

Pages, which are analogous to groups in Limesurvey, are automatically defined which eliminates a layer of complexity.

Changing question order is drag & drop – even between pages.In Lime it’s more laborious.

Overall I would probably choose SurveyGizmo again, in my current situation. If I had more time, and if we had other sites running on Apache/PHP/MySQL Limesurvey would be more attractive. We run mainly repeats of a given set of surveys. If we did more one-off kind of surveys, I think SurveyGizmo’s better interface/design environment would carry more weight.