UNIVERSE++: Universal Score-based Speech Enhancement with High Content Preservation

This page contains audio samples of enhanced speech for the paper "Universal Score-based Speech Enhancement with High Content Preservation" to be presented at Interspeech 2024.

Abstract

We propose UNIVERSE++, a universal speech enhancement method based on score-based diffusion and adversarial training. Specifically, we improve the existing UNIVERSE model that decouples clean speech feature extraction and diffusion. Our contributions are three-fold. First, we make several modifications to the network architecture, improving training stability and final performance. Second, we introduce an adversarial loss to promote learning high quality speech features. Third, we propose a low-rank adaptation scheme with a phoneme fidelity loss to improve content preservation in the enhanced speech. In the experiments, we train a universal enhancement model on a large scale dataset of speech degraded by noise, reverberation, and various distortions. The results on multiple public benchmark datasets demonstrate that UNIVERSE++ compares favorably to both discriminative and generative baselines for a wide range of qualitative and intelligibility metrics.

Noise Suppression on Voicebank-DEMAND (16 kHz)

Voicebank-DEMAND (test, 16 kHz)

File	Unprocessed	StoRM	UNIVERSE	UNIVERSE++
p257_311	DNSMOS=1.11/PESQ-WB=1.05/ESTOI=0.45	DNSMOS=2.67/PESQ-WB=1.35/ESTOI=0.70	DNSMOS=2.67/PESQ-WB=1.59/ESTOI=0.71	DNSMOS=3.03/PESQ-WB=1.98/ESTOI=0.75
p232_407	DNSMOS=1.25/PESQ-WB=1.22/ESTOI=0.52	DNSMOS=2.86/PESQ-WB=2.07/ESTOI=0.69	DNSMOS=3.06/PESQ-WB=1.57/ESTOI=0.63	DNSMOS=3.15/PESQ-WB=1.76/ESTOI=0.65
p257_303	DNSMOS=1.54/PESQ-WB=1.08/ESTOI=0.59	DNSMOS=3.39/PESQ-WB=1.91/ESTOI=0.76	DNSMOS=3.55/PESQ-WB=1.92/ESTOI=0.75	DNSMOS=3.49/PESQ-WB=2.06/ESTOI=0.76
p232_242	DNSMOS=1.71/PESQ-WB=1.26/ESTOI=0.45	DNSMOS=3.05/PESQ-WB=1.61/ESTOI=0.67	DNSMOS=3.07/PESQ-WB=1.43/ESTOI=0.62	DNSMOS=2.99/PESQ-WB=1.55/ESTOI=0.65
p232_388	DNSMOS=1.98/PESQ-WB=1.48/ESTOI=0.75	DNSMOS=3.00/PESQ-WB=2.78/ESTOI=0.83	DNSMOS=3.08/PESQ-WB=2.38/ESTOI=0.80	DNSMOS=2.83/PESQ-WB=2.63/ESTOI=0.81
p232_323	DNSMOS=2.27/PESQ-WB=1.35/ESTOI=0.68	DNSMOS=3.03/PESQ-WB=2.06/ESTOI=0.86	DNSMOS=3.14/PESQ-WB=2.52/ESTOI=0.89	DNSMOS=3.25/PESQ-WB=2.75/ESTOI=0.91
p232_104	DNSMOS=2.28/PESQ-WB=1.29/ESTOI=0.77	DNSMOS=3.16/PESQ-WB=2.56/ESTOI=0.84	DNSMOS=3.31/PESQ-WB=2.74/ESTOI=0.86	DNSMOS=3.21/PESQ-WB=2.36/ESTOI=0.85
p257_133	DNSMOS=2.39/PESQ-WB=1.21/ESTOI=0.69	DNSMOS=2.88/PESQ-WB=2.16/ESTOI=0.87	DNSMOS=3.04/PESQ-WB=2.39/ESTOI=0.79	DNSMOS=3.24/PESQ-WB=2.57/ESTOI=0.85
p232_028	DNSMOS=2.48/PESQ-WB=1.45/ESTOI=0.58	DNSMOS=3.13/PESQ-WB=2.13/ESTOI=0.71	DNSMOS=3.18/PESQ-WB=2.43/ESTOI=0.71	DNSMOS=3.27/PESQ-WB=2.26/ESTOI=0.73
p257_149	DNSMOS=2.55/PESQ-WB=1.10/ESTOI=0.63	DNSMOS=3.13/PESQ-WB=2.39/ESTOI=0.82	DNSMOS=3.05/PESQ-WB=2.42/ESTOI=0.78	DNSMOS=3.14/PESQ-WB=2.47/ESTOI=0.80

Universal Speech Enhancement (24 kHz)

Voicebank-DEMAND (test, 24 kHz)

File	Unprocessed	BS-RNN	StoRM	UNIVERSE	UNIVERSE++	+finetune
p257_267	DNSMOS=1.12/PESQ-WB=1.13/ESTOI=0.66	DNSMOS=2.96/PESQ-WB=2.41/ESTOI=0.76	DNSMOS=2.90/PESQ-WB=2.19/ESTOI=0.77	DNSMOS=2.88/PESQ-WB=2.23/ESTOI=0.76	DNSMOS=2.98/PESQ-WB=2.35/ESTOI=0.76	DNSMOS=2.96/PESQ-WB=2.06/ESTOI=0.75
p232_306	DNSMOS=1.23/PESQ-WB=1.58/ESTOI=0.68	DNSMOS=2.81/PESQ-WB=2.29/ESTOI=0.77	DNSMOS=2.80/PESQ-WB=1.99/ESTOI=0.77	DNSMOS=2.89/PESQ-WB=2.41/ESTOI=0.71	DNSMOS=2.84/PESQ-WB=2.17/ESTOI=0.75	DNSMOS=2.78/PESQ-WB=2.31/ESTOI=0.74
p257_129	DNSMOS=1.45/PESQ-WB=1.17/ESTOI=0.54	DNSMOS=2.67/PESQ-WB=2.18/ESTOI=0.75	DNSMOS=2.99/PESQ-WB=1.91/ESTOI=0.72	DNSMOS=2.86/PESQ-WB=1.79/ESTOI=0.70	DNSMOS=2.97/PESQ-WB=1.95/ESTOI=0.76	DNSMOS=2.92/PESQ-WB=1.97/ESTOI=0.73
p232_182	DNSMOS=1.57/PESQ-WB=1.30/ESTOI=0.66	DNSMOS=3.00/PESQ-WB=1.99/ESTOI=0.85	DNSMOS=2.91/PESQ-WB=1.55/ESTOI=0.81	DNSMOS=3.16/PESQ-WB=1.84/ESTOI=0.79	DNSMOS=3.45/PESQ-WB=2.11/ESTOI=0.89	DNSMOS=3.19/PESQ-WB=1.99/ESTOI=0.86
p257_069	DNSMOS=1.90/PESQ-WB=1.17/ESTOI=0.45	DNSMOS=2.68/PESQ-WB=1.86/ESTOI=0.66	DNSMOS=3.11/PESQ-WB=1.74/ESTOI=0.64	DNSMOS=3.10/PESQ-WB=1.57/ESTOI=0.63	DNSMOS=2.85/PESQ-WB=1.91/ESTOI=0.69	DNSMOS=3.02/PESQ-WB=1.74/ESTOI=0.65
p232_051	DNSMOS=2.14/PESQ-WB=1.41/ESTOI=0.60	DNSMOS=2.93/PESQ-WB=2.19/ESTOI=0.81	DNSMOS=3.13/PESQ-WB=1.81/ESTOI=0.76	DNSMOS=3.07/PESQ-WB=2.59/ESTOI=0.69	DNSMOS=3.41/PESQ-WB=1.93/ESTOI=0.79	DNSMOS=3.42/PESQ-WB=1.72/ESTOI=0.74
p257_255	DNSMOS=2.26/PESQ-WB=1.18/ESTOI=0.52	DNSMOS=2.62/PESQ-WB=2.53/ESTOI=0.76	DNSMOS=2.99/PESQ-WB=1.98/ESTOI=0.74	DNSMOS=2.96/PESQ-WB=2.02/ESTOI=0.66	DNSMOS=3.10/PESQ-WB=2.41/ESTOI=0.79	DNSMOS=2.69/PESQ-WB=2.50/ESTOI=0.77
p232_138	DNSMOS=2.39/PESQ-WB=1.53/ESTOI=0.59	DNSMOS=3.02/PESQ-WB=2.51/ESTOI=0.71	DNSMOS=2.80/PESQ-WB=2.36/ESTOI=0.72	DNSMOS=3.05/PESQ-WB=2.68/ESTOI=0.69	DNSMOS=3.13/PESQ-WB=2.94/ESTOI=0.71	DNSMOS=3.02/PESQ-WB=2.73/ESTOI=0.72
p232_384	DNSMOS=2.48/PESQ-WB=1.86/ESTOI=0.69	DNSMOS=2.79/PESQ-WB=2.15/ESTOI=0.72	DNSMOS=2.83/PESQ-WB=2.46/ESTOI=0.73	DNSMOS=3.13/PESQ-WB=1.76/ESTOI=0.67	DNSMOS=3.13/PESQ-WB=1.80/ESTOI=0.75	DNSMOS=3.05/PESQ-WB=1.99/ESTOI=0.72
p232_349	DNSMOS=2.51/PESQ-WB=1.28/ESTOI=0.69	DNSMOS=3.14/PESQ-WB=2.34/ESTOI=0.89	DNSMOS=3.13/PESQ-WB=2.14/ESTOI=0.83	DNSMOS=3.37/PESQ-WB=1.95/ESTOI=0.76	DNSMOS=3.40/PESQ-WB=2.62/ESTOI=0.89	DNSMOS=3.30/PESQ-WB=2.72/ESTOI=0.89

Voicebank-DEMAND, low-pass filter at 4 kHz (test, 24 kHz)

File	Unprocessed	BS-RNN	StoRM	UNIVERSE	UNIVERSE++	+finetune
p257_215	DNSMOS=1.15/PESQ-WB=1.20/ESTOI=0.58	DNSMOS=2.68/PESQ-WB=2.14/ESTOI=0.73	DNSMOS=2.61/PESQ-WB=1.24/ESTOI=0.73	DNSMOS=2.84/PESQ-WB=1.91/ESTOI=0.68	DNSMOS=3.00/PESQ-WB=2.02/ESTOI=0.75	DNSMOS=3.00/PESQ-WB=1.94/ESTOI=0.73
p232_032	DNSMOS=1.21/PESQ-WB=1.15/ESTOI=0.47	DNSMOS=2.94/PESQ-WB=1.80/ESTOI=0.69	DNSMOS=2.30/PESQ-WB=1.11/ESTOI=0.59	DNSMOS=3.37/PESQ-WB=1.59/ESTOI=0.65	DNSMOS=3.47/PESQ-WB=1.94/ESTOI=0.73	DNSMOS=3.48/PESQ-WB=1.84/ESTOI=0.74
p257_061	DNSMOS=1.50/PESQ-WB=1.28/ESTOI=0.61	DNSMOS=2.95/PESQ-WB=2.23/ESTOI=0.70	DNSMOS=2.68/PESQ-WB=1.51/ESTOI=0.68	DNSMOS=3.18/PESQ-WB=1.81/ESTOI=0.62	DNSMOS=3.16/PESQ-WB=2.00/ESTOI=0.68	DNSMOS=3.20/PESQ-WB=1.81/ESTOI=0.68
p232_113	DNSMOS=1.94/PESQ-WB=1.35/ESTOI=0.33	DNSMOS=2.89/PESQ-WB=2.01/ESTOI=0.53	DNSMOS=2.11/PESQ-WB=1.24/ESTOI=0.49	DNSMOS=3.13/PESQ-WB=1.52/ESTOI=0.51	DNSMOS=3.10/PESQ-WB=1.56/ESTOI=0.58	DNSMOS=3.15/PESQ-WB=1.50/ESTOI=0.61
p232_154	DNSMOS=2.13/PESQ-WB=1.29/ESTOI=0.64	DNSMOS=3.09/PESQ-WB=1.90/ESTOI=0.75	DNSMOS=2.19/PESQ-WB=1.27/ESTOI=0.74	DNSMOS=3.24/PESQ-WB=1.80/ESTOI=0.71	DNSMOS=3.35/PESQ-WB=1.84/ESTOI=0.81	DNSMOS=3.31/PESQ-WB=2.02/ESTOI=0.78
p232_369	DNSMOS=2.28/PESQ-WB=1.41/ESTOI=0.75	DNSMOS=2.87/PESQ-WB=2.33/ESTOI=0.89	DNSMOS=2.40/PESQ-WB=1.30/ESTOI=0.87	DNSMOS=3.35/PESQ-WB=2.27/ESTOI=0.83	DNSMOS=3.47/PESQ-WB=2.73/ESTOI=0.91	DNSMOS=3.36/PESQ-WB=2.32/ESTOI=0.92
p232_275	DNSMOS=2.34/PESQ-WB=1.39/ESTOI=0.77	DNSMOS=2.82/PESQ-WB=2.32/ESTOI=0.86	DNSMOS=2.33/PESQ-WB=1.22/ESTOI=0.83	DNSMOS=3.22/PESQ-WB=2.19/ESTOI=0.81	DNSMOS=3.38/PESQ-WB=2.08/ESTOI=0.88	DNSMOS=3.33/PESQ-WB=2.11/ESTOI=0.87
p257_092	DNSMOS=2.44/PESQ-WB=1.53/ESTOI=0.55	DNSMOS=2.67/PESQ-WB=2.56/ESTOI=0.61	DNSMOS=2.77/PESQ-WB=1.45/ESTOI=0.59	DNSMOS=2.63/PESQ-WB=1.88/ESTOI=0.50	DNSMOS=2.78/PESQ-WB=2.80/ESTOI=0.59	DNSMOS=2.78/PESQ-WB=2.37/ESTOI=0.62
p257_430	DNSMOS=2.50/PESQ-WB=1.34/ESTOI=0.66	DNSMOS=2.70/PESQ-WB=2.60/ESTOI=0.84	DNSMOS=2.52/PESQ-WB=1.36/ESTOI=0.83	DNSMOS=2.88/PESQ-WB=1.83/ESTOI=0.78	DNSMOS=2.86/PESQ-WB=2.58/ESTOI=0.85	DNSMOS=2.74/PESQ-WB=2.44/ESTOI=0.85
p257_253	DNSMOS=2.55/PESQ-WB=1.53/ESTOI=0.71	DNSMOS=2.79/PESQ-WB=2.35/ESTOI=0.80	DNSMOS=2.70/PESQ-WB=1.42/ESTOI=0.80	DNSMOS=2.76/PESQ-WB=1.80/ESTOI=0.72	DNSMOS=2.97/PESQ-WB=1.92/ESTOI=0.80	DNSMOS=2.90/PESQ-WB=2.05/ESTOI=0.83

ICASSP 2023 Signal Enhancement Challenge (test, 24 kHz)

File	Unprocessed	BS-RNN	StoRM	UNIVERSE	UNIVERSE++	+finetune
98e98d0f-cccb-4f3f-a350-eaf60e5a4c72	DNSMOS=1.12	DNSMOS=2.87	DNSMOS=3.05	DNSMOS=3.13	DNSMOS=3.27	DNSMOS=3.33
c7bb89ae-4658-4ef1-a835-f6f3544277d8	DNSMOS=1.34	DNSMOS=2.99	DNSMOS=2.00	DNSMOS=2.74	DNSMOS=1.61	DNSMOS=2.21
8d842ceb-dc1c-4db7-b2d9-1932e77e31b0	DNSMOS=1.56	DNSMOS=1.85	DNSMOS=2.17	DNSMOS=2.83	DNSMOS=2.96	DNSMOS=3.17
43944f9f-9b33-4f0c-8bf5-fb579fa73f92	DNSMOS=1.62	DNSMOS=2.58	DNSMOS=2.17	DNSMOS=2.80	DNSMOS=2.82	DNSMOS=2.95
a5851937-85af-41cf-ad3b-79575d7c1c77	DNSMOS=1.70	DNSMOS=2.32	DNSMOS=2.47	DNSMOS=2.87	DNSMOS=2.57	DNSMOS=2.74
2bc71138-a372-4ea3-b99a-5c03ceb93b90	DNSMOS=1.75	DNSMOS=1.75	DNSMOS=1.80	DNSMOS=2.20	DNSMOS=2.23	DNSMOS=2.23
47ce4be3-dbb8-4b8e-87ac-5dcf7860fd7a	DNSMOS=1.86	DNSMOS=2.91	DNSMOS=2.74	DNSMOS=2.93	DNSMOS=2.41	DNSMOS=3.18
16d46d34-5147-46c6-ab5b-a7a8aa065fef	DNSMOS=1.89	DNSMOS=1.76	DNSMOS=1.98	DNSMOS=2.00	DNSMOS=2.16	DNSMOS=2.01
cfc489dc-9bea-430e-a637-b7c1f8724de7	DNSMOS=1.99	DNSMOS=2.56	DNSMOS=2.58	DNSMOS=2.73	DNSMOS=2.92	DNSMOS=2.95
7259eed1-75e1-4b37-86d7-311eecafc862	DNSMOS=2.07	DNSMOS=2.31	DNSMOS=2.47	DNSMOS=2.28	DNSMOS=2.45	DNSMOS=2.37

ICASSP 2024 Packet Loss Concealement Challenge (validation set, 24 kHz)

File	Unprocessed	BS-RNN	StoRM	UNIVERSE	UNIVERSE++	+finetune
plcchallenge2024_val_0482	DNSMOS=1.34/PESQ-WB=1.03/ESTOI=0.18	DNSMOS=1.21/PESQ-WB=1.03/ESTOI=0.24	DNSMOS=2.31/PESQ-WB=1.04/ESTOI=0.24	DNSMOS=1.89/PESQ-WB=1.04/ESTOI=0.19	DNSMOS=1.84/PESQ-WB=1.05/ESTOI=0.25	DNSMOS=1.93/PESQ-WB=1.05/ESTOI=0.24
plcchallenge2024_val_0078	DNSMOS=1.64/PESQ-WB=1.11/ESTOI=0.69	DNSMOS=2.55/PESQ-WB=1.48/ESTOI=0.78	DNSMOS=2.91/PESQ-WB=1.94/ESTOI=0.81	DNSMOS=2.82/PESQ-WB=1.37/ESTOI=0.63	DNSMOS=2.77/PESQ-WB=1.78/ESTOI=0.73	DNSMOS=3.00/PESQ-WB=1.69/ESTOI=0.74
plcchallenge2024_val_0286	DNSMOS=1.76/PESQ-WB=1.06/ESTOI=0.41	DNSMOS=2.44/PESQ-WB=1.10/ESTOI=0.52	DNSMOS=3.14/PESQ-WB=1.15/ESTOI=0.52	DNSMOS=3.04/PESQ-WB=1.15/ESTOI=0.50	DNSMOS=2.90/PESQ-WB=1.15/ESTOI=0.53	DNSMOS=3.01/PESQ-WB=1.12/ESTOI=0.52
plcchallenge2024_val_0382	DNSMOS=1.96/PESQ-WB=1.31/ESTOI=0.75	DNSMOS=2.74/PESQ-WB=1.98/ESTOI=0.80	DNSMOS=2.95/PESQ-WB=2.11/ESTOI=0.68	DNSMOS=2.99/PESQ-WB=1.60/ESTOI=0.58	DNSMOS=2.61/PESQ-WB=2.02/ESTOI=0.65	DNSMOS=2.96/PESQ-WB=2.02/ESTOI=0.73
plcchallenge2024_val_0080	DNSMOS=2.05/PESQ-WB=1.05/ESTOI=0.50	DNSMOS=2.30/PESQ-WB=1.40/ESTOI=0.80	DNSMOS=2.98/PESQ-WB=1.53/ESTOI=0.82	DNSMOS=2.98/PESQ-WB=1.64/ESTOI=0.78	DNSMOS=2.90/PESQ-WB=1.70/ESTOI=0.81	DNSMOS=3.18/PESQ-WB=1.63/ESTOI=0.83
plcchallenge2024_val_0798	DNSMOS=2.10/PESQ-WB=1.24/ESTOI=0.67	DNSMOS=2.93/PESQ-WB=1.64/ESTOI=0.76	DNSMOS=3.13/PESQ-WB=2.16/ESTOI=0.79	DNSMOS=3.03/PESQ-WB=1.94/ESTOI=0.72	DNSMOS=3.14/PESQ-WB=2.12/ESTOI=0.78	DNSMOS=3.15/PESQ-WB=2.02/ESTOI=0.78
plcchallenge2024_val_0200	DNSMOS=2.20/PESQ-WB=1.16/ESTOI=0.74	DNSMOS=2.73/PESQ-WB=1.55/ESTOI=0.80	DNSMOS=3.14/PESQ-WB=1.94/ESTOI=0.82	DNSMOS=3.28/PESQ-WB=1.35/ESTOI=0.56	DNSMOS=3.31/PESQ-WB=1.60/ESTOI=0.62	DNSMOS=3.32/PESQ-WB=1.89/ESTOI=0.75
plcchallenge2024_val_0163	DNSMOS=2.28/PESQ-WB=1.28/ESTOI=0.82	DNSMOS=3.19/PESQ-WB=2.02/ESTOI=0.92	DNSMOS=3.26/PESQ-WB=2.77/ESTOI=0.93	DNSMOS=3.09/PESQ-WB=2.74/ESTOI=0.89	DNSMOS=3.25/PESQ-WB=2.82/ESTOI=0.92	DNSMOS=3.25/PESQ-WB=2.81/ESTOI=0.93
plcchallenge2024_val_0414	DNSMOS=2.33/PESQ-WB=1.18/ESTOI=0.71	DNSMOS=2.62/PESQ-WB=1.50/ESTOI=0.80	DNSMOS=3.40/PESQ-WB=2.06/ESTOI=0.85	DNSMOS=3.22/PESQ-WB=1.77/ESTOI=0.77	DNSMOS=3.28/PESQ-WB=2.02/ESTOI=0.84	DNSMOS=3.21/PESQ-WB=2.02/ESTOI=0.84
plcchallenge2024_val_0580	DNSMOS=2.42/PESQ-WB=1.06/ESTOI=0.49	DNSMOS=2.73/PESQ-WB=1.10/ESTOI=0.56	DNSMOS=2.88/PESQ-WB=1.09/ESTOI=0.58	DNSMOS=2.99/PESQ-WB=1.11/ESTOI=0.53	DNSMOS=2.87/PESQ-WB=1.12/ESTOI=0.58	DNSMOS=3.08/PESQ-WB=1.12/ESTOI=0.57