Transformer是一种基于注意力的架构,被证明是自然语言处理(NLP)中最先进的模型。为了降低在医学语言理解中开始使用基于变压器的模型的难度,并扩展scikit-learn工具包在深度学习中的能力,我们提出了一个简单易学的Python工具包,名为transformer-sklearn。通过将变压器的接口仅包装在三个函数中(即,fit,得分,并预测),变压器-sklearn结合了变压器和scikit-learn工具包的优点。
在变形金刚-sklearn中,实现了三个Python类,即,分类任务的BERTologyClassifier,命名实体识别(NER)任务的BERTologyNERClassifier,和BERTologyRegressor的回归任务。每个类包含三个方法,即,适合具有训练数据集的基于变压器的微调模型,用于评估微调模型性能的分数,和预测用于预测测试数据集的标签。transformers-sklearn是一个用户友好的工具包,它(1)可通过几个参数定制(例如,model_name_or_path和model_type),(2)支持多语言NLP任务,(3)需要较少的编码。输入数据格式由具有注释语料库的transformer-sklearn自动生成。新手只需要准备数据集。模型框架和训练方法在Transformer-sklearn中预定义。
我们收集了四个开源医学语言数据集,包括中医试验文本多标签分类,BC5CDR用于英文生物医学文本名称实体识别,用于中国糖尿病实体识别的DiabetesNER和用于英语生物医学句子相似性估计的BIOSSES。在四个医学NLP任务中,我们的脚本的平均代码大小是45行/任务,是变压器脚本大小的六分之一。实验结果表明,基于预训练的BERT模型的Transformer-sklearn分别获得了0.8225、0.8703和0.6908的宏F1分数,关于试验分类,BC5CDR和DiabetesNER任务以及BIOSSES任务上0.8260的Pearson相关性,这与变压器的结果是一致的。
拟议的工具包可以帮助新来者使用scikit-learn编码风格轻松解决医学语言理解任务。变压器sklearn的代码和教程可在https://doi.org/10.5281/zenodo.4453803获得。在未来,将支持更多的医学语言理解任务,以改善Transformer_sklearn的应用。
Transformer is an attention-based architecture proven the state-of-the-art model in natural language processing (NLP). To reduce the difficulty of beginning to use transformer-based models in medical language understanding and expand the capability of the scikit-learn
toolkit in deep learning, we proposed an easy to learn Python
toolkit named transformers-sklearn. By wrapping the interfaces of transformers in only three functions (i.e., fit, score, and predict), transformers-sklearn combines the advantages of the transformers and scikit-learn toolkits.
In transformers-sklearn, three Python classes were implemented, namely, BERTologyClassifier for the classification task, BERTologyNERClassifier for the named entity recognition (NER) task, and BERTologyRegressor for the regression task. Each class contains three methods, i.e., fit for fine-tuning transformer-based models with the training dataset, score for evaluating the performance of the fine-tuned model, and predict for predicting the labels of the test dataset. transformers-sklearn is a user-friendly
toolkit that (1) Is customizable via a few parameters (e.g., model_name_or_path and model_type), (2) Supports multilingual NLP tasks, and (3) Requires less coding. The input data format is automatically generated by transformers-sklearn with the annotated corpus. Newcomers only need to prepare the dataset. The model framework and training methods are predefined in transformers-sklearn.
We collected four open-source medical language datasets, including TrialClassification for Chinese medical trial text multi label classification, BC5CDR for English biomedical text name entity recognition, DiabetesNER for Chinese diabetes entity recognition and BIOSSES for English biomedical sentence similarity estimation. In the four medical NLP tasks, the average code size of our script is 45 lines/task, which is one-sixth the size of transformers\' script. The experimental results show that transformers-sklearn based on pretrained BERT models achieved macro F1 scores of 0.8225, 0.8703 and 0.6908, respectively, on the TrialClassification, BC5CDR and DiabetesNER tasks and a Pearson correlation of 0.8260 on the BIOSSES task, which is consistent with the results of transformers.
The proposed
toolkit could help newcomers address medical language understanding tasks using the scikit-learn coding style easily. The code and tutorials of transformers-sklearn are available at https://doi.org/10.5281/zenodo.4453803 . In future, more medical language understanding tasks will be supported to improve the applications of transformers_sklearn.