20220712-h07m20s32_example_conll2003

fxmarty/20220712-h07m20s32_example_conll2003

20220712-h07m20s32_example_conll2003 is a token classification model from fxmarty. Use it when you need labels on individual words, such as names.

task: token-classification Backend: sagemaker-training Backend args: {'instancetype': 'ml.g4dn.2xlarge', 'supportedinstructions': 'avx512vnni'} Number of evaluation samples: 1000

Downloads · 30 days

Access

Public

Updated Jul 12, 2022

Repo size

3 KB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.json31.2 KB · 75%

At a glance

Task: Token Classification
Access: Public
Created: Jul 12, 2022
Updated: Jul 12, 2022
SHA: f9866210

Try a prompt

Datasets

conll2003

Model card

From the Hugging Face model README

task: token-classification
Backend: sagemaker-training
Backend args: {'instance_type': 'ml.g4dn.2xlarge', 'supported_instructions': 'avx512_vnni'}
Number of evaluation samples: 1000

Fixed parameters:

model_name_or_path: elastic/distilbert-base-uncased-finetuned-conll03-english
dataset:
- path: conll2003
- eval_split: validation
- data_keys: {'primary': 'tokens'}
- ref_keys: ['ner_tags']
- calibration_split: train
node_exclusion: []
per_channel: False
calibration:
- method: minmax
- num_calibration_samples: 100
framework: onnxruntime
framework_args:
- opset: 11
- optimization_level: 1
aware_training: False

Benchmarked parameters:

quantization_approach: dynamic, static
operators_to_quantize: ['Add', 'MatMul'], ['Add']

Evaluation

Non-time metrics

quantization_approach	operators_to_quantize		precision (original)	precision (optimized)		recall (original)	recall (optimized)		f1 (original)	f1 (optimized)		accuracy (original)	accuracy (optimized)
`dynamic`	`['Add', 'MatMul']`	\|	0.937	0.937	\|	0.953	0.953	\|	0.945	0.945	\|	0.988	0.988
`dynamic`	`['Add']`	\|	0.937	0.937	\|	0.953	0.953	\|	0.945	0.945	\|	0.988	0.988
`static`	`['Add', 'MatMul']`	\|	0.937	0.074	\|	0.953	0.253	\|	0.945	0.114	\|	0.988	0.363
`static`	`['Add']`	\|	0.937	0.065	\|	0.953	0.186	\|	0.945	0.096	\|	0.988	0.340

Time metrics

Time benchmarks were run for 3 seconds per config.

Below, time metrics for batch size = 1, input length = 64.

quantization_approach	operators_to_quantize		latency_mean (original, ms)	latency_mean (optimized, ms)		throughput (original, /s)	throughput (optimized, /s)
`dynamic`	`['Add', 'MatMul']`	\|	57.64	12.30	\|	17.67	81.33
`dynamic`	`['Add']`	\|	43.51	29.42	\|	23.00	34.00
`static`	`['Add', 'MatMul']`	\|	43.05	21.11	\|	23.33	47.67
`static`	`['Add']`	\|	43.50	37.93	\|	23.00	26.67

Below, time metrics for batch size = 4, input length = 64.

quantization_approach	operators_to_quantize		latency_mean (original, ms)	latency_mean (optimized, ms)		throughput (original, /s)	throughput (optimized, /s)
`dynamic`	`['Add', 'MatMul']`	\|	119.50	39.92	\|	8.67	25.33
`dynamic`	`['Add']`	\|	119.62	107.42	\|	8.67	9.33
`static`	`['Add', 'MatMul']`	\|	120.23	56.94	\|	8.33	17.67
`static`	`['Add']`	\|	119.10	130.78	\|	8.67	7.67

Below, time metrics for batch size = 8, input length = 64.

quantization_approach	operators_to_quantize		latency_mean (original, ms)	latency_mean (optimized, ms)		throughput (original, /s)	throughput (optimized, /s)
`dynamic`	`['Add', 'MatMul']`	\|	165.84	75.45	\|	6.33	13.33
`dynamic`	`['Add']`	\|	214.65	211.41	\|	4.67	5.00
`static`	`['Add', 'MatMul']`	\|	166.53	129.00	\|	6.33	8.00
`static`	`['Add']`	\|	214.81	256.95	\|	4.67	4.00

Task: Token Classification
Created: Jul 12, 2022
Updated: Jul 12, 2022