{"messages":[{"status":"ok","category":"all"}], "collection":[{"title":"Application of Large Language Models (LLM) for Automatic Classification of Work Accident Text Data: Verification of Accuracy and Practicality","authors":"Ando, H.; Matsugaki, R.; Yamakawa, S.; Ogami, A.","author_corresponding":"Hajime Ando","author_corresponding_institution":"Department of Work Systems and Health, Institute of Industrial Ecological Sciences, University of Occupational and Environmental Health, Japan","doi":"10.1101\/2025.10.02.25337141","date":"2025-10-03","version":"1","type":"PUBLISHAHEADOFPRINT","license":"cc_by_nc_nd","category":"occupational and environmental health","jatsxml":"https:\/\/www.medrxiv.org\/content\/early\/2025\/10\/03\/2025.10.02.25337141.source.xml","abstract":"BackgroundFalls are the most frequent type of occupational accident, making the development of effective countermeasures an urgent issue. Traditional accident analysis relies on manual classification of text data by experts, a process that is both time-consuming and labor-intensive. Large Language Models (LLMs) offer the potential to significantly streamline this analysis process without the need for task-specific pre-training.\n\nObjectiveThis study aims to automatically classify text data from occupational accidents using LLMs and to verify the accuracy and practicality of this approach.\n\nMethodsThe analysis targeted 2,619 fall-related injury cases in the health\/hygiene and retail sectors, extracted from the 2021 Survey on Industrial Accidents database. The results of manual classification performed by experts in a previous study were used as the \"ground truth.\" These were compared against the automatic classification results from four different LLMs (GPT-4.1, GPT-4.1 mini, GPT-4o mini, and o4-mini). Evaluation metrics included accuracy, precision, recall, F1-score, and Cohens kappa coefficient. The processing was conducted using OpenAIs Batch API, with processing time and costs also being measured.\n\nResultsNewer generation models demonstrated a high rate of agreement with expert classifications across most categories, with the exception of \"causal substance,\" generally achieving a Cohens kappa coefficient above 0.7. For the \"accident location (indoor\/outdoor)\" category, the accuracy reached over 91%. Even for \"causal substance,\" the category with the lowest accuracy, the reasoning model o4-mini achieved a kappa coefficient of 0.662. In terms of practicality, even when using the highest-performing model (o4-mini), the entire dataset was processed in approximately 90 minutes at a cost of about $11, demonstrating high cost-performance.\n\nConclusionThis study demonstrates that LLMs can classify occupational accident text data with an accuracy comparable to manual expert analysis, but at a lower cost and higher speed. This method is expected to facilitate large-scale accident analysis, which has been challenging in the past, and contribute to the rapid development of evidence-based preventive measures for occupational accidents.","published":"10.3390\/occuphealth1020016","server":"medRxiv"}]}