EffiARA

EffiARA (Efficient Annotator Reliability Assessment) is a framework for distributing examples to annotators and computing reliability using inter- and intra-annotator agreement.

• Check out the code on Github

• Read the paper introducing EffiARA

Contents:

• Getting Started
  • Installation
  • Fundamentals
  • Minimum Working Example
• Advanced Usage
  • Allocating and Reallocating Samples to Annotators
  • Creating a LabelGenerator

API:

• API
  • Annotations
  • SampleDistributor
  • LabelGenerator
  • calculate_krippendorff_alpha_per_label()
  • cosine_similarity()
  • inter_annotator_agreement_krippendorff()
  • pairwise_agreement()
  • pairwise_cohens_kappa_agreement()
  • pairwise_cosine_similarity()
  • pairwise_fleiss_kappa_agreement()
  • pairwise_nominal_krippendorff_agreement()
  • pairwise_percentage_agreement()

Indices and tables

• Index

• Module Index

• Search Page