For full experimental methods and results, see:

Choi, J., Kilmer, D., Mueller-Smith, M., & Taheri, S. (2023) Hierarchical Approaches to Text-based Offense Classification. Science Advances. 9(9), 1-15.

Data

In order to develop the TOC tool, we pooled two novel sources of hand-coded offense description information and caseload count data from MFJ and the CJARS. The pooled set of data draws on multiple decades of electronic criminal justice records from across the United States. In total, we employed 313,209 hand-coded unique offense descriptions, representing 439,534,275 total criminal justice events that have occurred in the United States over recent decades, to train the model.

Distribution of total unique descriptions by their case counts in the full training data. Out of the 313,209 unique descriptions, 274,744 (87.7%) descriptions occur less than 25 times, and 18,202 descriptions occur more than 100 times with an average case count of 23,950. This figure shows that a fair number of unique descriptions happen quite infrequently in the data.