Customize Consent Preferences

We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.

The cookies that are categorized as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site.... 

Always Active

Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

No cookies to display.

Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

No cookies to display.

Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

No cookies to display.

Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.

No cookies to display.

Advertisement cookies are used to provide visitors with customized advertisements based on the pages you visited previously and to analyze the effectiveness of the ad campaigns.

No cookies to display.

In this chapter of Advancing Natural Language Processing in Educational Assessment, Director of Research Polina Harik, PhDMeasurement Scientist Janet Mee, PhD, Senior Measurement Scientist Christopher Runyon, PhD and Distinguished Research Scientist Brian Clauser, PhD describe INCITE, a natural language processing (NLP)-based system for scoring free-text responses.

The system was developed by NBME to score responses produced as part of the USMLE® Step 2 Clinical Skills Examination. The specifics of that application make our system unique in several ways because:

  1. The INCITE was intended for use as part of physician licensure and must therefore function with a specialized medical vocabulary.
  2. The scores for the written responses were based entirely on the content of the response, as well-structured and complete sentences are not required and spelling is secondary.
  3. The scores from the examination were used to make high-stakes decisions, hence accuracy is critical.
  4. The procedures used for scoring must be transparent so that it is possible to identify the specific scorable features of the text that the algorithm identified in each response