Precision, Recall, and F1 Analysis for LMW Candidates from SpVar Model - Paper on SpVar

I. Introduction

In the previous study (AMIA paper), multiple MES and ES models are used to retrieved SpVar from the distilled MEDLINE n-gram set and then use as filter to retrieve LMWs. This model is OK with some issues:

• Performance and Frequency:
  Due to the complexity of the algorithm, the performance is very slow. Thus, we have to reduce the size of MEDLINE n-gram set by applying high frequency (WC = 150). With this reduction, it took weeks for the program to complete the process.
• The precision on the bench mark test on Lexicon.2015 is only 52.65% (even the recall reach 99.72%). This is OK to used as additonal filter to the high precision (ACR) matcher. However, it can not be used as stand long matcher to retrieve high precision LMW candidates.
• LMWs with WC less than 150 are missing

An improved model, M2CES, is developed to address these issues.

II. Development

• Compare to Table-3 in 2016 (ACR) paper final

  Case	Description	TP	FP	T. Retrieved	T. Relevant	FN	TN	Precision	Recall	F1	Accuracy
  31	Parenthetic Acronym - Gold Standard	14,805	1,870	16,675	14,805	0	0	0.8879	1.0000	0.9406	0.8879
  Filters or a single matcher
  33	SpVar Matcher - MES + ES, WC: 150	7,509	482	7,991	14,805	7,296	1,388	0.9397	0.5072	0.6588	0.5336
  33A	SpVar Matcher - M2CES, WC: 150	3,695	226	3,921	14,805	11,110	1,644	0.9424	0.2496	0.3946	0.3202
  33B	SpVar Matcher - M2CES, WC: 100	4,485	283	4,768	14,805	10,320	1,587	0.9406	0.3029	0.4583	0.3641
  33C	SpVar Matcher - M2CES, WC: 50	5,748	421	6,169	14,805	9,057	1,449	0.9318	0.3882	0.5481	0.4316
  33D	SpVar Matcher - M2CES, WC: 30	6,682	513	7,195	14,805	8,123	1,357	0.9287	0.4513	0.6075	0.4821
  Combination: SpVar + CUI + Distrilled
  36	SpVar - MES + ES, WC: 150	5,510	206	5,716	14,805	9,295	1,664	0.9640	0.3722	0.5370	0.4302
  36A	SpVar - M2CES, WC: 150	2,793	106	2,899	14,805	12,012	1,764	0.9634	0.1887	0.3155	0.2733
  36B	SpVar - M2CES, WC: 100	3,319	118	3,437	14,805	11,486	1,752	0.9657	0.2242	0.3639	0.3041
  36C	SpVar - M2CES, WC: 50	4,102	162	4,264	14,805	10,703	1,708	0.9620	0.2771	0.4302	0.3484
  36D	SpVar - M2CES, WC: 30	4,699	189	4,888	14,805	10,106	1,681	0.9613	0.3174	0.4772	0.3826
  Combination: SpVar + CUI + Distrilled
  37	SpVar - MES + ES, WC: 150	727	11	738	14,805	14,078	1,859	0.9851	0.0491	0.0935	0.1551
  37A	SpVar - M2CES, WC: 150	354	5	359	14,805	14,451	1,865	0.9861	0.0239	0.0467	0.1331
  37B	SpVar - M2CES, WC: 100	427	5	432	14,805	14,378	1,865	0.9884	0.0288	0.0560	0.1375
  37C	SpVar - M2CES, WC: 50	568	8	576	14,805	14,237	1,862	0.9861	0.0384	0.0739	0.1457
  37D	SpVar - M2CES, WC: 30	654	10	664	14,805	14,151	1,860	0.9849	0.0442	0.0846	0.1508