SemantAPI.Human is an application that uses Amazon’s Mechanical Turk service to have humans assign a sentiment score to submitted documents. The application uses the same output as the SemantAPI.Robot tool, and adds the Mechanical Turk output to the same CSV file as the baseline reference for sentiment analysis accuracy.

While human scoring is very subjective (trained humans only agree on sentiment scores 80% of the time at best), it’s still the best option available as a reference.

MechanicalTurk to contrast with API results

The SemantAPI.Human application offers a variety of options to precisely configure the Mechanical Turk HITs. The number of assignments and the reward amount are the most important options. Out of the box, the tool requests an analysis of each document 3 times and finally calculates the average sentiment score and polarity based on the humans’ scores. The “MechanicalTurk Score” column represents a confidence score related to the provided “Sentiment Polarity”. So, if two people said that the sentiment is negative but one person said positive, the tool will return a Negative “Sentiment Polarity” with 0.67(2/3) confidence score.

Before you begin

1. You will need to register for and configure a Mechanical Turk profile. This service is available for a fee.

2. Create a .txt or .csv file of text data you’d like to analyze

How to use SemantAPI.Human

1. Download the SemantAPI toolkit

Download the SemantAPI toolkit by clicking the button below.

Download SemantAPI

2. Install SemantAPI

Double-click on the file to begin the installation process. If you have already installed the SemantAPI toolkit, skip to step 4

3. Follow the installation wizard

Screen Shot 2013-09-23 at 3.27.08 PM

4. Open SemantAPI.Human

  1. Click on the Start button.
  2. Click on All Programs
  3. Navigate and click on SemantAPI
  4. Click on SemantAPI.Human

5. Enter your API keys and verification

Insert your Mechanical Turk keys

6. Select your source file to analyze

If you’re looking to compare human sentiment to sentiment derived from SemantAPI.Robot, select the same source document used in SemantAPI.Robot.
***Note: Your source file must be in either .csv or .txt format.

Select your source file

7. Modify the parameters of the HITs. (Optional)

HIT Parameters

8. Process!

When you’re ready, send it off on its merry way!

Process the HITs

9. Wait for the HITs...

 Don’t worry, you’ll be getting your results soon enough!

10. Open your output file and view your results!




Help us evolve SemantAPI!

Both SemantAPI.Robot and SemantAPI.Human are written in C# and based on Microsoft’s .Net framework 3.5 platform.

A redistributable package of the SemantAPI toolkit can be downloaded here. The source code is available on GitHub.