Stars or Likes? The 8 Best Crowdsourcing Review Systems

Choosing the wrong rating or review system can kill your product. Before spending dev time building out any particular one, consider all your options! Here are the top eight:

No alt text provided for this image

1. Detailed survey

Many don’t know how Rotten Tomatoes works. Critics give a review based on a certain set of criteria, and an algorithm ends up giving the final score. With a complex questionnaire, the rating will be most accurate. However, there will be very few scores given. If there are few scores given, it will feel less “crowdsourced” and more “curated.” Therefore, this rating system is only ideal if there are a few amount of items to review, but lots of passionate reviewers.

No alt text provided for this image

2. Binary: creating a set list

As opposed to a simple binary voting system, websites like US News give a limited amount of spots (e.g. Top 30) and have users vote on whether or not an item belongs on that list. This is ideal if the purpose of the list is a comparison, the set point would otherwise be ambiguous, and there should be a small amount of list items. On the down side, this review system may exclude interesting ideas that few people would think of. For example, my favorite city in the Philippines was Malapascua- but few people have been there so they wouldn’t be able to make a comparison.

No alt text provided for this image

3. Binary: net positive votes

As opposed to the percentage of positive to negative votes, this binary list simply gives whatever has upvotes (Note: Reddit actually also factors in how fast votes are given, so that it also accounts for “trending”). Reddit has an incredibly high amount of active users, so there are a lot of reviews coming in, and they are able to keep social proof when users are seeing a high number of votes casted. The benefit of this is that it gives a high importance to what is disliked. For example, when something that should not ethically be posted is posted, there can be enough down-votes for the post to be shoved towards the bottom. This review system incredibly useful when there is lots of disagreement over answers, and the benefits of a simple binary system are desired.

Add alt textNo alt text provided for this image

4. Classic five stars

This is the most obvious choice for a review system. The five star rating system is great because it provides enough qualitative information to be meaningful and its definition has been almost universally learned. It should be used on ecommerce platforms, hotel websites, and other straightforward digital products. However, the most common reason the star system cannot be used is that users will not be willing to go out of their way to write a review. Five star systems require high cognitive load for the complex decision, so they should not be used when the bulk of the items in the listing have slim to no reviews.

Add alt textNo alt text provided for this image

6. Binary: explanation of responses

One of the main problems that come up when creating a review system is that upvotes and downvotes may be unclear in their meaning. In the case of the website FMLife, a downvote could mean that the reader doesn’t like the poster’s story, or that the reader agrees that they wouldn’t like what happened to the poster either. In this case, writing out the meaning of a upvote and downvote would be important.

Add alt textNo alt text provided for this image

7. Bubbles as opposed to stars

While Trip Advisor has an interesting algorithm that comes up with what they consider “best,” it is also interesting that they chose to use bubble ratings instead of standard star ratings. The reason why they chose bubbles over stars is that stars generally mean “quality” and they wanted people to say “satisfaction.” Sometimes adventure may not technically be quality- but it’s important that the adventure was enjoyable!

Add alt textNo alt text provided for this image

8. One upvote

Upvotes can be used when negativity should not be expressed. In the case of Product Hunt, it’s not nice to be mean to entrepreneurs who spent time building their companies. Similarly, it’s not nice to dislike somebody’s comment on Facebook. Therefore, upvotes are useful when only positivity should be shown. However, even in those ecosystems, there is the lowest qualitative information provided by users. Therefore, there should be a balance made between the need for positivity and the need for qualitative information.

Leave a comment

Your email address will not be published. Required fields are marked *