The JyMob Blog

Musings of the founder of a failed startup

On Designing Questionnaires That Are Conclusive

One of the strongest motivations for us at JyMob is the quality of its questionnaires. It’s our real IP. We strongly believe that to improve hiring inefficiencies, companies need to employ strategies that simply:

  1. Screen-in the relevant candidates, and
  2. Screen-out the irrelevant (not bad) candidates

This is, of course, easier said than done. It can only happen if the question-setters are committed to the worthwhileness of the questions. They should be conclusive. If a correct or incorrect answer to a question is inconclusive of whether the candidate understands and has solved it correctly, then, in our opinion, the question has failed.

Unfortunately, such questions form a vast majority of questions asked in online candidate assessment products. At JyMob, therefore, we are quite sensitive to forming questions that are worthwhile. There’s no point in further complicating an already complex process such as ‘hiring’. From an ‘objective assessment’ standpoint, there’s nothing wrong in having a question that is multiple-choice or multiple-select, but as it turns out, creating a challenging, fun and clear multiple-choice, multiple-select question is often time-consuming, if not difficult. Our experts, we believe know how to do it and we are constantly look for experts who share this thought process.

For example, take a look at this question that appears in JyMob’s problem database: a good multiple-choice question.

Here are the criteria for a good multiple-choice question satisfied by this one:

  • It makes it (almost) imperative for the candidate to attempt this question seriously. Of course, they can apply brute force and choose one of the 5 options provided. But then again, JyMob makes it clear that the hiring team for this job may ask you to provide actual code (demonstrating your thought process) that resulted in your selection of a given option.
  • It’s suited for an ‘online’ assessment in that it makes candidates do something before resorting to online havens like Google, Stack Overflow, Wikipedia etc.
  • It’s not based on some rather obscure feature of computing. If you are working on data interchange, you need to know how the data is interpreted on various computers!
  • It’s conclusive enough. By providing an “indirection” in terms of an attachment, the designer of this question clearly made the candidate work.

I will write later about our philosophy of various ‘testing models’ we are playing with in order to provide an environment where candidates might be assessed in the most appropriate manner. But for now, we can see the power of simple multiple-choice questions designed with the end in mind – being able to conclusively assess a candidate.

Candidate’s comfort level is another thing we, as question setters have in our minds. We don’t want to annoy candidates that you would love to interview. Thus, we want to make the question as clear as possible. We believe we have been extremely diligent in our endeavor.

If you are, or know anyone who loves to design such questions, please let us know (or let them know that we, at JyMob, are interested in using their skills by compensating them well for their creativity).