Reader Studies

A reader study enables you to have a set of readers answer a set of questions about a set of images.

Editors

You can add multiple editors to your reader study. An editor is someone who can edit the reader study settings, add other editors, add and remove readers, add images and edit questions.

Readers

A user who can read this study, creating an answer for each question and image in the study.

Cases

The set of images that will be used in the study.

Hanging List

How the each image will be presented to the user as a set of hanging protocols. For instance, you might want to present two images side by side and have a reader answer a question about both, or overlay one image on another.

Creating a Reader Study

A ReaderStudy can use any available Workstation. A WorkstationConfig can also be used for the study to customise the default appearance of the workstation.

Cases

Cases can be added to a reader study by adding Image instances. Multiple image formats are supported:

  • .mha

  • .mhd with the accompanying .zraw or .raw file

  • .tif/.tiff

  • .jpg/.jpeg

  • .png

  • 3D/4D DICOM support is also available, though this is experimental and not guaranteed to work on all .dcm images.

Defining the Hanging List

When you upload a set of images you have the option to automatically generate the default hanging list. The default hanging list presents each reader with 1 image per protocol.

You are able to customise the hanging list in the study edit page. Here, you are able to assign multiple images and overlays to each protocol.

Available image ports are: * main * secondary * tertiary * quaternary * quinary * senary * septenary * octonary * nonary * denary

Overlays can be applied to the image ports by using the image-port name with the suffix ‘-overlay’ (e.g. main-overlay).

Questions

A Question can be optional and the following answer_type options are available:

  • Heading (not answerable)

  • Bool

  • Single line text

  • Multiline text

The following annotation answer types are also available:

  • Distance measurement

  • Multiple distance measurements

  • 2D bounding box

To use an annotation answer type you must also select the image port where the annotation will be made.

Adding Ground Truth

To monitor the performance of the readers you are able add ground truth to a reader study by uploading a csv file.

If ground truth has been added to a ReaderStudy, any Answer given by a reader is evaluated by applying the scoring_function chosen for the Question.

The scores can then be compared on the leaderboard. Statistics are also available based on these scores: the average and total scores for each question as well as for each case are displayed in the statistics view.

class grandchallenge.reader_studies.models.Answer(*args, **kwargs)[source]

An Answer can be provided to a Question that is a part of a ReaderStudy.

Parameters:

Relationship fields:

Parameters:

Reverse relationships:

Parameters:
exception DoesNotExist
exception MultipleObjectsReturned
property api_url: str

API url for this Answer.

calculate_score(ground_truth)[source]

Calculate the score for this Answer based on ground_truth.

save(*args, calculate_score=True, **kwargs)[source]

Save the current instance. Override this in a subclass if you want to control the saving process.

The ‘force_insert’ and ‘force_update’ parameters can be used to insist that the “save” must be an SQL insert or update (or equivalent for non-SQL backends), respectively. Normally, they should not be set.

static validate(*, creator, question, answer, display_set, is_ground_truth=False, instance=None)[source]

Validates all fields provided for answer.

class grandchallenge.reader_studies.models.AnswerGroupObjectPermission(id, permission, group, content_object)[source]
Parameters:

id (AutoField) – Primary key: ID

Relationship fields:

Parameters:

  • permission (ForeignKey to Permission) – Permission (related name: answergroupobjectpermission)

  • group (ForeignKey to Group) – Group (related name: answergroupobjectpermission)

  • content_object (ForeignKey to Answer) – Content object (related name: answergroupobjectpermission)

exception DoesNotExist
exception MultipleObjectsReturned
class grandchallenge.reader_studies.models.AnswerType(*values)[source]
class grandchallenge.reader_studies.models.AnswerUserObjectPermission(id, permission, user, content_object)[source]
Parameters:

id (AutoField) – Primary key: ID

Relationship fields:

Parameters:

  • permission (ForeignKey to Permission) – Permission (related name: answeruserobjectpermission)

  • user (ForeignKey to User) – User (related name: answeruserobjectpermission)

  • content_object (ForeignKey to Answer) – Content object (related name: answeruserobjectpermission)

exception DoesNotExist
exception MultipleObjectsReturned
class grandchallenge.reader_studies.models.CategoricalOption(id, question, title, default)[source]
Parameters:

Relationship fields:

Parameters:

question (ForeignKey to Question) – Question (related name: options)

exception DoesNotExist
exception MultipleObjectsReturned
class grandchallenge.reader_studies.models.DisplaySet(id, created, modified, reader_study, order, title)[source]
Parameters:

Relationship fields:

Parameters:

Reverse relationships:

Parameters:
exception DoesNotExist
exception MultipleObjectsReturned
property api_url: str

API url for this DisplaySet.

workstation_url
class grandchallenge.reader_studies.models.DisplaySetGroupObjectPermission(id, permission, group, content_object)[source]
Parameters:

id (AutoField) – Primary key: ID

Relationship fields:

Parameters:

  • permission (ForeignKey to Permission) – Permission (related name: displaysetgroupobjectpermission)

  • group (ForeignKey to Group) – Group (related name: displaysetgroupobjectpermission)

  • content_object (ForeignKey to DisplaySet) – Content object (related name: displaysetgroupobjectpermission)

exception DoesNotExist
exception MultipleObjectsReturned
class grandchallenge.reader_studies.models.DisplaySetUserObjectPermission(id, permission, user, content_object)[source]
Parameters:

id (AutoField) – Primary key: ID

Relationship fields:

Parameters:

  • permission (ForeignKey to Permission) – Permission (related name: displaysetuserobjectpermission)

  • user (ForeignKey to User) – User (related name: displaysetuserobjectpermission)

  • content_object (ForeignKey to DisplaySet) – Content object (related name: displaysetuserobjectpermission)

exception DoesNotExist
exception MultipleObjectsReturned
class grandchallenge.reader_studies.models.ImagePort(*values)[source]
class grandchallenge.reader_studies.models.OptionalHangingProtocolReaderStudy(id, reader_study, hanging_protocol)[source]
Parameters:

id (AutoField) – Primary key: ID

Relationship fields:

Parameters:

  • reader_study (ForeignKey to ReaderStudy) – Reader study (related name: optionalhangingprotocolreaderstudy)

  • hanging_protocol (ForeignKey to HangingProtocol) – Hanging protocol (related name: optionalhangingprotocolreaderstudy)

exception DoesNotExist
exception MultipleObjectsReturned
class grandchallenge.reader_studies.models.Question(id, created, modified, overlay_segments, look_up_table, reader_study, question_text, help_text, answer_type, image_port, default_annotation_color, required, direction, scoring_function, order, interface, widget, interactive_algorithm, answer_max_value, answer_min_value, answer_step_size, answer_min_length, answer_max_length, answer_match_pattern, empty_answer_confirmation, empty_answer_confirmation_label)[source]
Parameters:

  • id (UUIDField) – Primary key: Id

  • created (DateTimeField) – Created

  • modified (DateTimeField) – Modified

  • overlay_segments (JSONField) – Overlay segments. The schema that defines how categories of values in the overlay images are differentiated. Example usage: [{“name”: “background”, “visible”: true, “voxel_value”: 0},{“name”: “tissue”, “visible”: true, “voxel_value”: 1}]. If a categorical overlay is shown, it is possible to show toggles to change the visibility of the different overlay categories. To do so, configure the categories that should be displayed. For example: [{“name”: “Level 0”, “visible”: false, “voxel_value”: 0].

  • question_text (TextField) – Question text

  • help_text (TextField) – Help text

  • answer_type (CharField) – Answer type

  • image_port (CharField) – Image port

  • default_annotation_color (HexColorField) – Default annotation color. Default color for displaying and creating annotations for this question

  • required (BooleanField) – Required

  • direction (CharField) – Direction

  • scoring_function (CharField) – Scoring function

  • order (PositiveSmallIntegerField) – Order

  • widget (CharField) – Widget

  • interactive_algorithm (CharField) – Interactive algorithm. Which interactive algorithm should be used for this question?

  • answer_max_value (SmallIntegerField) – Answer max value. Maximum value for answers of type Number. Can only be set in combination with the ‘Number input’ or ‘Number Range’ widgets.

  • answer_min_value (SmallIntegerField) – Answer min value. Minimum value for answers of type Number. Can only be set in combination with the ‘Number input’ or ‘Number Range’ widgets.

  • answer_step_size (DecimalField) – Answer step size. Step size for answers of type Number. Defaults to 1, allowing only integer values. Can only be set in combination with the ‘Number input’ or ‘Number Range’ widgets.

  • answer_min_length (PositiveSmallIntegerField) – Answer min length. Minimum length for answers of type Text. Can only be set in combination with the ‘Text Input’ or ‘Text Area’ widgets.

  • answer_max_length (PositiveSmallIntegerField) – Answer max length. Maximum length for answers of type Text. Can only be set in combination with the ‘Text Input’ or ‘Text Area’ widgets.

  • answer_match_pattern (RegexField) – Answer match pattern. Regular expression to match a pattern for answers of type Text. Can only be set in combination with the ‘Text Input’ or ‘Text Area’ widgets.

  • empty_answer_confirmation (BooleanField) – Empty answer confirmation. Require an explicit confirmation when saving an empty answer to this question.

  • empty_answer_confirmation_label (TextField) – Empty answer confirmation label. Label to show when confirming an empty answer.

Relationship fields:

Parameters:

  • look_up_table (ForeignKey to LookUpTable) – Look up table. The look-up table that is applied when an overlay image is first shown (related name: question)

  • reader_study (ForeignKey to ReaderStudy) – Reader study (related name: questions)

  • interface (ForeignKey to ComponentInterface) – Interface (related name: question)

Reverse relationships:

Parameters:
class AnswerType(*values)
class Direction(*values)[source]
exception DoesNotExist
class ImagePort(*values)
exception MultipleObjectsReturned
class ScoringFunction(*values)[source]
property api_url: str

API url for this Question.

calculate_score(answer, ground_truth)[source]

Calculates the score for answer by applying scoring_function to answer and ground_truth.

clean()[source]

Hook for doing any extra model-wide validation after clean() has been called on every field by self.clean_fields. Any ValidationError raised by this method will not be associated with a particular field; it will have a special-case association with the field defined by NON_FIELD_ERRORS.

property empty_answer_value

Returns the answer value which is considered to be empty

is_answer_valid(*, answer)[source]

Validates answer against ANSWER_TYPE_SCHEMA.

property is_fully_editable

True if no Answer has been given for this Question.

save(*args, **kwargs)[source]

Save the current instance. Override this in a subclass if you want to control the saving process.

The ‘force_insert’ and ‘force_update’ parameters can be used to insist that the “save” must be an SQL insert or update (or equivalent for non-SQL backends), respectively. Normally, they should not be set.

class grandchallenge.reader_studies.models.QuestionGroupObjectPermission(id, permission, group, content_object)[source]
Parameters:

id (AutoField) – Primary key: ID

Relationship fields:

Parameters:

  • permission (ForeignKey to Permission) – Permission (related name: questiongroupobjectpermission)

  • group (ForeignKey to Group) – Group (related name: questiongroupobjectpermission)

  • content_object (ForeignKey to Question) – Content object (related name: questiongroupobjectpermission)

exception DoesNotExist
exception MultipleObjectsReturned
class grandchallenge.reader_studies.models.QuestionUserObjectPermission(id, permission, user, content_object)[source]
Parameters:

id (AutoField) – Primary key: ID

Relationship fields:

Parameters:

  • permission (ForeignKey to Permission) – Permission (related name: questionuserobjectpermission)

  • user (ForeignKey to User) – User (related name: questionuserobjectpermission)

  • content_object (ForeignKey to Question) – Content object (related name: questionuserobjectpermission)

exception DoesNotExist
exception MultipleObjectsReturned
class grandchallenge.reader_studies.models.QuestionWidgetKindChoices(*values)[source]
class grandchallenge.reader_studies.models.ReaderStudy(*args, **kwargs)[source]

Reader Study model.

A reader study is a tool that allows users to have a set of readers answer a set of questions on a set of images (cases).

Parameters:

  • title (CharField) – Title

  • description (TextField) – Description

  • slug (AutoSlugField) – Slug

  • id (UUIDField) – Primary key: Id

  • created (DateTimeField) – Created

  • modified (DateTimeField) – Modified

  • view_content (JSONField) – View content

  • public (BooleanField) – Public. Should this reader study be visible to all users on the overview page? This does not grant all users permission to read this study. Users will still need to be added to the study’s readers group in order to do that.

  • access_request_handling (CharField) – Access request handling. How would you like to handle access requests?

  • logo (PictureField) – Logo

  • logo_width (PositiveSmallIntegerField) – Logo width

  • logo_height (PositiveSmallIntegerField) – Logo height

  • social_image (PictureField) – Social image. An image for this reader study which is displayed when you post the link on social media. Should have a resolution of 640x320 px (1280x640 px for best display).

  • social_image_width (PositiveSmallIntegerField) – Social image width

  • social_image_height (PositiveSmallIntegerField) – Social image height

  • help_text_markdown (TextField) – Help text markdown

  • shuffle_hanging_list (BooleanField) – Shuffle hanging list

  • is_educational (BooleanField) – Is educational. If checked, readers get the option to verify their answers against the uploaded ground truth. This also means that the uploaded ground truth will be readily available to the readers.

  • instant_verification (BooleanField) – Instant verification. In an educational reader study, enabling this setting will allow the user to go through the reader study faster. The ‘Save and continue’ button will be replaced by a ‘Verify and continue’ button which will show the answer verification pop up and allow the user to save and go to the next case upon dismissal.

  • case_text (JSONField) – Case text

  • allow_answer_modification (BooleanField) – Allow answer modification. If true, readers are allowed to modify their answers for a case by navigating back to previous cases. ‘Allow case navigation’ must be checked as well to enable this setting.

  • enable_autosaving (BooleanField) – Enable autosaving. If true, answers to questions are saved in the background while a user reads a case. ‘Allow Answer Modification’ must be enabled as well for this to work.

  • allow_case_navigation (BooleanField) – Allow case navigation. If true, readers are allowed to navigate back and forth between cases in this reader study.

  • allow_show_all_annotations (BooleanField) – Allow show all annotations. If true, readers are allowed to show/hide all annotations for a case.

  • roll_over_answers_for_n_cases (PositiveSmallIntegerField) – Roll over answers for n cases. The number of cases for which answers should roll over. It can be used for repeated readings with slightly different hangings. For instance, if set to 1. Case 2 will start with the answers from case 1; whereas case 3 starts anew but its answers will roll over to case 4. Setting it to 0 (default) means answers will not roll over.

  • leaderboard_accessible_to_readers (BooleanField) – Leaderboard accessible to readers. If checked, readers can see the leaderboard. Usernames and avatars will be hidden to protect other readers’ privacy.

  • end_of_study_text_markdown (TextField) – End of study text markdown. Text to show when a user has completed the reader study

  • max_credits (PositiveIntegerField) – Max credits. The maximum number of credits that may be consumed for this reader study. Leave blank to allow unlimited usage.

Relationship fields:

Parameters:

  • hanging_protocol (ForeignKey to HangingProtocol) – Hanging protocol. Indicate which sockets need to be displayed in which image port. E.g. {“main”: [“socket1”]}. The first item in the list of sockets will be the main image in the image port. The first overlay type socket thereafter will be rendered as an overlay. For now, any other items will be ignored by the viewer. (related name: readerstudy)

  • editors_group (OneToOneField to Group) – Editors group (related name: editors_of_readerstudy)

  • readers_group (OneToOneField to Group) – Readers group (related name: readers_of_readerstudy)

  • workstation (ForeignKey to Workstation) – Workstation (related name: readerstudy)

  • workstation_config (ForeignKey to WorkstationConfig) – Workstation config (related name: readerstudy)

  • workstation_sessions (ManyToManyField to Session) – Workstation sessions (related name: reader_studies)

  • optional_hanging_protocols (ManyToManyField to HangingProtocol) – Optional hanging protocols. Optional alternative hanging protocols for this reader study (related name: optional_for_reader_study)

  • publications (ManyToManyField to Publication) – Publications. The publications associated with this reader study (related name: readerstudy)

  • modalities (ManyToManyField to ImagingModality) – Modalities. The imaging modalities contained in this reader study (related name: readerstudy)

  • structures (ManyToManyField to BodyStructure) – Structures. The structures contained in this reader study (related name: readerstudy)

  • organizations (ManyToManyField to Organization) – Organizations. The organizations associated with this reader study (related name: readerstudies)

  • actor_actions (GenericRelation to Action) – Actor actions (related name: actions_with_reader_studies_readerstudy_as_actor)

  • target_actions (GenericRelation to Action) – Target actions (related name: actions_with_reader_studies_readerstudy_as_target)

  • action_object_actions (GenericRelation to Action) – Action object actions (related name: actions_with_reader_studies_readerstudy_as_action_object)

Reverse relationships:

Parameters:

  • readerstudyuserobjectpermission (Reverse ForeignKey from ReaderStudyUserObjectPermission) – All reader study user object permissions of this reader study (related name of content_object)

  • readerstudygroupobjectpermission (Reverse ForeignKey from ReaderStudyGroupObjectPermission) – All reader study group object permissions of this reader study (related name of content_object)

  • workstationsessionreaderstudy (Reverse ForeignKey from WorkstationSessionReaderStudy) – All workstation session reader studys of this reader study (related name of reader_study)

  • display_sets (Reverse ForeignKey from DisplaySet) – All display sets of this reader study (related name of reader_study)

  • questions (Reverse ForeignKey from Question) – All questions of this reader study (related name of reader_study)

  • readerstudypermissionrequest (Reverse ForeignKey from ReaderStudyPermissionRequest) – All reader study permission requests of this reader study (related name of reader_study)

  • optionalhangingprotocolreaderstudy (Reverse ForeignKey from OptionalHangingProtocolReaderStudy) – All optional hanging protocol reader studys of this reader study (related name of reader_study)

  • session_utilizations (Reverse ManyToManyField from SessionUtilization) – All session utilizations of this reader study (related name of reader_studies)

  • sessionutilizationreaderstudy (Reverse ForeignKey from SessionUtilizationReaderStudy) – All session utilization reader studys of this reader study (related name of reader_study)

exception DoesNotExist
exception MultipleObjectsReturned
add_editor(user)[source]

Adds user as an editor for this ReaderStudy.

add_reader(user)[source]

Adds user as a reader for this ReaderStudy.

answerable_question_count
answerable_questions

All questions for this ReaderStudy except those with answer type

clean()[source]

Hook for doing any extra model-wide validation after clean() has been called on every field by self.clean_fields. Any ValidationError raised by this method will not be associated with a particular field; it will have a special-case association with the field defined by NON_FIELD_ERRORS.

get_progress_for_user(user)[source]

Returns the percentage of completed hangings and questions for user.

property image_groups

Names of the images as they are grouped in the hanging list.

is_editor(user)[source]

Checks if user is an editor for this ReaderStudy.

is_reader(user)[source]

Checks if user is a reader for this ReaderStudy.

leaderboard
remove_editor(user)[source]

Removes user as an editor for this ReaderStudy.

remove_reader(user)[source]

Removes user as a reader for this ReaderStudy.

save(*args, **kwargs)[source]

Save the current instance. Override this in a subclass if you want to control the saving process.

The ‘force_insert’ and ‘force_update’ parameters can be used to insist that the “save” must be an SQL insert or update (or equivalent for non-SQL backends), respectively. Normally, they should not be set.

score_for_user(user)[source]

Returns the average and total score for answers given by user.

scores_by_user
statistics
study_image_names
class grandchallenge.reader_studies.models.ReaderStudyGroupObjectPermission(id, permission, group, content_object)[source]
Parameters:

id (AutoField) – Primary key: ID

Relationship fields:

Parameters:

  • permission (ForeignKey to Permission) – Permission (related name: readerstudygroupobjectpermission)

  • group (ForeignKey to Group) – Group (related name: readerstudygroupobjectpermission)

  • content_object (ForeignKey to ReaderStudy) – Content object (related name: readerstudygroupobjectpermission)

exception DoesNotExist
exception MultipleObjectsReturned
class grandchallenge.reader_studies.models.ReaderStudyPermissionRequest(*args, **kwargs)[source]

When a user wants to read a reader study, editors have the option of reviewing each user before accepting or rejecting them. This class records the needed info for that.

Parameters:

Relationship fields:

Parameters:

  • user (ForeignKey to User) – User. which user requested to participate? (related name: readerstudypermissionrequest)

  • reader_study (ForeignKey to ReaderStudy) – Reader study. To which reader study has the user requested access? (related name: readerstudypermissionrequest)

  • actor_actions (GenericRelation to Action) – Actor actions (related name: actions_with_reader_studies_readerstudypermissionrequest_as_actor)

  • target_actions (GenericRelation to Action) – Target actions (related name: actions_with_reader_studies_readerstudypermissionrequest_as_target)

  • action_object_actions (GenericRelation to Action) – Action object actions (related name: actions_with_reader_studies_readerstudypermissionrequest_as_action_object)

exception DoesNotExist
exception MultipleObjectsReturned
save(*args, **kwargs)[source]

Save the current instance. Override this in a subclass if you want to control the saving process.

The ‘force_insert’ and ‘force_update’ parameters can be used to insist that the “save” must be an SQL insert or update (or equivalent for non-SQL backends), respectively. Normally, they should not be set.

class grandchallenge.reader_studies.models.ReaderStudyUserObjectPermission(id, permission, user, content_object)[source]
Parameters:

id (AutoField) – Primary key: ID

Relationship fields:

Parameters:

  • permission (ForeignKey to Permission) – Permission (related name: readerstudyuserobjectpermission)

  • user (ForeignKey to User) – User (related name: readerstudyuserobjectpermission)

  • content_object (ForeignKey to ReaderStudy) – Content object (related name: readerstudyuserobjectpermission)

exception DoesNotExist
exception MultipleObjectsReturned
class grandchallenge.reader_studies.models.WorkstationSessionReaderStudy(id, reader_study, workstation_session)[source]
Parameters:

id (AutoField) – Primary key: ID

Relationship fields:

Parameters:

  • reader_study (ForeignKey to ReaderStudy) – Reader study (related name: workstationsessionreaderstudy)

  • workstation_session (ForeignKey to Session) – Workstation session (related name: workstationsessionreaderstudy)

exception DoesNotExist
exception MultipleObjectsReturned
grandchallenge.reader_studies.models.delete_reader_study_groups_hook(*_, instance, using, **__)[source]

Deletes the related groups.

We use a signal rather than overriding delete() to catch usages of bulk_delete.