TCSI_pp_zh

CAPP-130: A Corpus of Chinese Application Privacy Policy Summarization and Interpretation.

Introduction

A privacy policy serves as an online internet protocol crafted by service providers, which details how service providers collect, process, store, manage, and use personal information when users engage with applications. However, these privacy policies are often filled with technobabble and legalese, making them 'incomprehensible'. As a result, users often agree to all terms unknowingly, even some terms may conflict with the law, thereby posing a considerable risk to personal privacy information. To tackle these challenges, we introduce a fine-grained CAPP-130 corpus and a TCSI-pp framework. CAPP-130 contains $130$ Chinese privacy policies from popular applications that have been carefully annotated and interpreted by legal experts, resulting in $52,489$ annotations and $20,555$ rewritten sentences. TCSI-pp first extracts sentences related to the topic specified by users and then uses a generative model to rewrite the sentences into comprehensible summarization. Built upon TSCI-pp, we construct a summarization tool TSCI-pp-zh by selecting RoBERTa from six classification models for sentence extraction and selecting mT5 from five generative models for sentence rewriting.

Code: here

Environment

Project dependencies can be installed in the following ways:

pip install -r requirements.txt

Equipment: A100 *2

Chinese Application Privacy Policy Corpus (CAPP-130)

CAPP-130 contains $130$ Chinese privacy policies from popular applications that have been carefully annotated and interpreted by legal experts, resulting in $52,489$ annotations and $19,570$ rewritten sentences.

Basic Statistics of Corpus CAPP-130

The guide for Paper and Annotation Guidelines (Chinese version, English version) explains the tags and the process of annotation, which can be found in the Documents. Currently, the Annotation Guidelines are available only in Chinese, but we are working on translating them into English. Table 1 shows the basic statistical information of CAPP-130, and Table 2 shows the pre-sliced data information used for TCSI-pp. They are stored in the CAPP-130 Corpus.

Table 1: Basic Statistics of Corpus CAPP-130.

Table 2: The pre-sliced data from CAPP-130 is used to train TCSI-pp.

Topic-Controlled Framework for Summarization and Interpretation of Privacy Policy (TCSI-pp)

we provide a Topic-Controlled Framework for Summarization and Interpretation of Privacy Policy (TCSI-pp). Unlike previous methods that only extract specific sentences, TCSI-pp first retrieves relevant sentences based on the topics chosen from data practice categories by users using a classification model. Then, a generative model is used to rewrite these sentences clearly and concisely for the understanding of the general public, with potentially risky sentences emphasized.

Information Extraction

These are specifically utilized for binary classification models such as "Important Identification" and "Risk Identification", as well as multi-classification models like "Topic Identification".

How to use

The model is placed in the XXX_pretain (where XXX is the model name) directory and each directory contains three files:

• pytorch_model.bin
• bert_config.json
• vocab.txt

Pre-trained model download address from here.

After decompression, put it in the corresponding directory according to the above, and confirm the file name is correct.

We have independently acquired three sets of classification benchmarks from six different models: RoBERTa, BERT, mBERT, sBERT, Pert, and ERNIE.

You can be used in the following ways:

# Train and test binary classification model:
python run.py --model 'model_name' --data 'data_name'

# Train and test multi-classification model:
python run_multi.py --model 'model_name' --data 'data_name'

Please note that the above code examples are for illustrative purposes only and you may need to make appropriate adjustments based on your specific situation.

Baselines

We provide classification baselines for "Important Identification", "Risk Identification", and "Topic Identification". They are respectively trained and tested on the 'important_identification_dataset', 'risk_identification_dataset', and 'topics_identification_dataset' in the sub-dataset. Table 3 displays the evaluation metrics of six models.

Table 3: Evaluation Metrics with F1 for Classification Models.

(New) We will upload all model parameters to here.

Rewritten Sentences

A generative model is used to rewrite these sentences clearly and concisely for the understanding of the general public, with potentially risky sentences emphasized.

How to use

For rewriting sentences, we fine-tuned the following models based on the transformer encoder-decoder architecture: mT5, Bert2Bert, Bert2gpt, RoBerta2gpt, and ERNIE2gpt. These models were initialized with parameters from publicly available models, such as mT5-small, Bert-base-Chinese, ernie-3.0-base-zh, chinese-roberta-wwm-ext, and gpt2-base-chinese. These models can be found on Hugging Face model repository.

You can be used in the following ways:

# train and test:
python model_name.py
#The model_name needs to be changed to mT5, Bert2Bert, Bert2gpt, RoBerta2gpt, or ERNIE2gpt.

Please note that the above code examples are for illustrative purposes only and you may need to make appropriate adjustments based on your specific situation.

Baselines

Table 4 displays the ROUGE, Bert-score, Bart-score, and Carburacy evaluation metrics for these models:

Table 4: Evaluation metrics for the rewrite models.

(New) We will upload all model parameters to here.

Chinese application privacy policy summary tool (TCSI-pp-zh)

we select the most effective RoBERTa and mT5 to implement the Chinese application privacy policy summary tool (TCSI-pp-zh). Experiments on real privacy policies show that TCSI-pp-zh is superior over GPT-4 and other models, demonstrating higher readability and reliability in the task of summarizing Chinese application privacy policies.

How to use

You can be used in the following ways:

# train and test:
python ./TCSI_pp_zh/TCSI_pp_zh.py --binary_model 'binary_model_name' --multi_model 'multi_model_name'  --rewrite_model 'rewrite_model_name' --topic_list 'choose_a_topic_list'  --data 'a_privacy_policy'

Please note that the above code examples are for illustrative purposes only and you may need to make appropriate adjustments based on your specific situation.

Effect Demonstration

Figure 1 displays the summarization of GPT-4 and TCSI-pp-zh in a privacy policy, where text having the same background color represents descriptions of the same part of the privacy policy generated by different algorithms; red text emphasizes incorrect content produced in the summary.

Figure 1: Summarization of GPT-4 and TCSI-pp-zh.

citation

If you use the data or code of this project, or if our work is helpful to you, please state the citation

@inproceedings{
zhu2023capp,
title={{CAPP}-130: A Corpus of Chinese Application Privacy Policy Summarization and Interpretation},
author={Pengyun Zhu and Long Wen and Jinfei Liu and Feng Xue and Jian Lou and Zhibo Wang and Kui Ren},
booktitle={Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2023},
url={https://openreview.net/forum?id=OyTIV57Prb}
}

Update

We will continue to update this repository on GitHub.