Wav2Lip Noise Robustness Experiment

This repository investigates how noisy audio conditions affect Wav2Lip-generated lip synchronisation quality, using SyncNet evaluation, perceptual inspection, DeepFilterNet3 preprocessing, and a train-free Whisper-VAD hybrid mel-gating experiment.

Demo: Original Wav2Lip vs Whisper-VAD Hybrid

After testing multiple approaches, the best qualitative result was achieved with the Whisper-VAD Hybrid method. In this final approach, Whisper log-mel features are used only to detect speech and non-speech regions, while the original Wav2Lip mel spectrogram is kept as the actual model input. Non-speech regions detected by the Whisper-based VAD mask are suppressed before being passed to Wav2Lip.

The demo below compares the original Wav2Lip output under background people noise with the proposed train-free Whisper-VAD hybrid mel-gating method.

Original Wav2Lip with background noise	Whisper-VAD hybrid result

Full videos:

• Original noisy Wav2Lip output
• Whisper-VAD hybrid output

The original Wav2Lip still moves the mouth during background-noise-only regions. The Whisper-VAD hybrid method reduces this false mouth movement while preserving stronger mouth motion during speech.

Zoomed Lip Comparison (1-3 seconds)

The following zoomed-in GIFs show the lip region between 1 and 3 seconds, where background noise is present. This close-up makes it easier to see the difference in mouth behaviour under noisy conditions.

Original Wav2Lip lip region with background noise (1-3s)	Whisper-VAD hybrid lip region with background noise (1-3s)

Mel Spectrogram Comparison

Original Wav2Lip mel spectrogram

Final hybrid mel spectrogram

VAD speech / non-speech mask

Motivation

Wav2Lip can produce strong lip-sync results with clean speech, but it may also react to background speech, music, or non-speech noise in the conditioning audio. This repository evaluates noisy Wav2Lip behaviour using SyncNet scores, perceptual inspection, DeepFilterNet3 preprocessing, and a train-free Whisper-VAD hybrid mel-gating method.

Experimental setup

• Input: a single face video with clean, noisy, denoised, and Whisper-VAD hybrid audio-conditioning variants
• Pipelines: Wav2Lip generation for lip synthesis, SyncNet scoring, DeepFilterNet3 for denoising, and Whisper-VAD hybrid mel gating
• Scripts: reproducible steps are available in scripts/

Noise types used

• background_people_noise
• cat_meow
• guitar_music
• piano_music
• violin_music

Wav2Lip and SyncNet pipeline

• Run scripts/run_wav2lip.sh to generate videos for audio conditions
• Run scripts/run_syncnet.sh to score videos; logs placed in results/syncnet_logs/

DeepFilterNet preprocessing pipeline

• Run scripts/run_deepfilternet.sh to denoise noisy audio files and save into audio_samples/

Whisper-VAD Hybrid Mel Gating

A train-free Whisper-based VAD-style hybrid method was added to reduce false mouth movement during background-noise-only regions.

The method uses Whisper log-mel features only for speech/non-speech detection, while keeping the original Wav2Lip mel spectrogram as the actual model input. In detected non-speech regions, the original Wav2Lip mel is suppressed before being passed to Wav2Lip.

This produced the best qualitative result so far: strong mouth movement during speech while reducing background-noise-induced false mouth motion.

Method	Speech movement	Background-noise movement	Observation
Original Wav2Lip	Strong	High	Mouth moved during noise-only regions
Only Whisper log-mel + VAD	Weak/moderate	Low	Good noise suppression, but weaker mouth opening
Original Wav2Lip Mel + Hybrid Whisper-VAD	Strong	Low	Best train-free qualitative result

Original Wav2Lip Mel + Hybrid Whisper-VAD was the best train-free qualitative result because the method does not replace the audio representation that Wav2Lip was trained on. Instead, it keeps the original Wav2Lip mel spectrogram during detected speech regions and only suppresses non-speech regions identified by the Whisper-based VAD mask.

Note: Although the hybrid Whisper-VAD method gave the best train-free qualitative result, a more robust long-term solution may be to finetune Wav2Lip under noisy conditions using Whisper-based audio processing. In this setup , the training data does not need to change; instead, the audio conditioning pipeline can be modified so that Wav2Lip learns to map speech-aware, noise-robust audio features to appropriate mouth motion. This could make the model more robust to background speech, music, and non-speech noise without relying on hand-tuned VAD thresholds at inference time.

Details, visualisations, and implementation are available in:

• whisper_vad_hybrid.md
• scripts/inference_whisper_vad_hybrid.py
• results/whisper_vad_hybrid/demo/

Results and summaries

Full raw SyncNet logs are kept at results/syncnet_logs/all_syncnet_results.txt.

Structured result tables are available in results/tables/:

• results/tables/all_results_summary.csv
• results/tables/all_results_summary.md

Below is a concise, cleaned summary of the main SyncNet metrics from the experiments:

Main Wav2Lip Noise Robustness Results

Experiment	Condition	AV Offset	Min Dist	Confidence
Original video	Clean original video	2	8.159	5.387
Wav2Lip clean audio	Clean original audio → Wav2Lip	-2	7.463	7.159
Background speech noise	Noisy original audio → Wav2Lip	-2	8.656	4.592
Cat meow	Noisy original audio → Wav2Lip	-2	9.083	5.081
Guitar music	Noisy original audio → Wav2Lip	-2	8.445	5.367
Piano music	Noisy original audio → Wav2Lip	-2	8.224	5.248
Violin music	Noisy original audio → Wav2Lip	-2	8.146	5.693
Background speech + DeepFilterNet3	Denoised audio → Wav2Lip	-2	7.682	6.349
Cat meow + DeepFilterNet3	Denoised audio → Wav2Lip	-2	7.965	6.622
Guitar music + DeepFilterNet3	Denoised audio → Wav2Lip	-2	7.581	6.646
Piano music + DeepFilterNet3	Denoised audio → Wav2Lip	-2	7.427	6.719
Violin music + DeepFilterNet3	Denoised audio → Wav2Lip	-2	7.461	6.943
Hybrid Whisper-VAD + Original Wav2Lip Mel	Whisper-VAD gated original mel → Wav2Lip	-2	9.248	3.412

Although the hybrid Whisper-VAD method produced a lower SyncNet confidence score, qualitative inspection showed the best perceptual behaviour in background-noise-only regions. This is expected because the method intentionally suppresses mouth motion during detected non-speech regions, while SyncNet may still respond to acoustic energy from background noise.

Additional SyncNet Behaviour Results

Experiment	Condition	AV Offset	Min Dist	Confidence	Interpretation
Original video	Clean original video	2	8.159	5.387	Baseline SyncNet behaviour
Audio advanced 200 ms	Manual temporal shift	7	8.514	5.109	SyncNet offset changed under artificial misalignment
Audio delayed 200 ms	Manual temporal shift	-3	8.357	5.219	SyncNet offset changed in the opposite direction
Normal noise	Added background noise	2	8.098	5.000	Noise reduced confidence slightly without large offset change
Short overlap	Overlapping audio	1	9.523	3.943	Overlap mainly reduced confidence
Other speaker overlap	Competing speaker	2	9.730	3.782	Speaker interference reduced confidence
Loud other speaker overlap	Loud competing speaker	2	10.740	3.157	Strong competing speech further reduced confidence
Full overlap	Full competing overlap	2	12.215	1.642	Severe overlap caused very low confidence
Wav2Lip output with overlap speech	Wav2Lip generated video	-2	5.761	9.543	Very high confidence, consistent with SyncNet-style training objective

These additional SyncNet behaviour experiments suggest that artificial temporal shifts are reflected in the predicted AV offset, while noisy or overlapping audio mainly reduces SyncNet confidence. This supports the interpretation that SyncNet offset and confidence should be analysed together, and that SyncNet alone is not sufficient to judge perceptual lip-sync quality.

Key Findings

• Wav2Lip generated videos achieved higher SyncNet confidence than the original video itself when using clean audio

  • Original video confidence: 5.387
  • Wav2Lip with original audio: 7.159

  This behaviour is expected because Wav2Lip is trained using a SyncNet-style discriminator/loss, meaning the model is partially optimised for SyncNet-like audio-visual alignment objectives.

• Higher SyncNet scores did not always correspond to better perceptual visual quality

  • Some noisy videos still received moderate SyncNet confidence scores despite visibly unnatural mouth motion.

• Visual inspection showed that Wav2Lip can still react to background noise and non-speech audio

  • Background speech, music, and cat meow sounds sometimes influenced generated mouth movement.

• SyncNet alone is not sufficient to evaluate perceptual lip-sync quality

  • SyncNet measures audio-visual synchronisation, but not visual realism, mouth naturalness, identity preservation, or whether the mouth is responding to the correct speaker.

• Additional visual quality evaluation is needed

  • Human inspection or extra perceptual metrics are required alongside SyncNet for a more complete evaluation.

• Background speech noise was the most harmful noise condition

  • Confidence dropped from 7.159 → 4.592
  • Competing speech-like patterns strongly affected generated mouth movement.

• DeepFilterNet3 preprocessing consistently improved noisy Wav2Lip outputs

  • Denoising the background people noise audio improved from 4.592 → 6.349
  • All denoised conditions showed higher SyncNet confidence compared with their noisy counterparts.

• The experiments suggest that mel spectrogram conditioning is a major robustness bottleneck

  • Mel spectrograms encode acoustic energy but do not explicitly separate target speech, background speech, music, or non-speech acoustic events.

• The Whisper-VAD hybrid method improved perceptual behaviour but reduced SyncNet confidence

  • Hybrid result: AV offset -2, min dist 9.248, confidence 3.412.
  • Qualitatively, this was the best result because the mouth stayed more stable during background-noise-only regions.
  • This supports the finding that SyncNet confidence alone does not fully capture perceptual lip-sync quality or whether the mouth is responding to the correct sound source.

• The results indicate that robust audio-driven lip synchronisation likely requires speech-aware conditioning representations

  • Future directions can include HuBERT, WavLM, source separation, speech enhancement, and perceptual visual evaluation metrics.

Limitations

• Model checkpoints are excluded from the repository.
• Small experiment videos are included for reproducibility and qualitative inspection.
• Reproducibility requires external Wav2Lip and SyncNet repositories and model files.

Future work

• Evaluate HuBERT/WavLM conditioning.
• Integrate alternative denoisers and compare effect on SyncNet.
• Finetune Wav2Lip with Whisper Mel or train an adapter using noisy and overlapped speech examples.

Checkpoints

• wav2lip_gan.pth
• syncnet_v2.model
• s3fd.pth

How to run (example)

Set WAV2LIP_PATH and SYNCNET_PATH environment variables to point to local clones. Then:

bash scripts/generate_noisy_audio.sh audio_samples/original_audio.wav
bash scripts/run_wav2lip.sh audio_samples/ output_videos/ input_face.mp4
bash scripts/run_deepfilternet.sh audio_samples/ audio_samples/denoised/
bash scripts/run_wav2lip.sh audio_samples/denoised/ output_videos_denoised/ input_face.mp4
bash scripts/run_syncnet.sh output_videos/ results/syncnet_logs/

See scripts/ for detailed usage.

External repositories

Wav2Lip: https://github.com/Rudrabha/Wav2Lip

SyncNet: https://github.com/joonson/syncnet_python

DeepFilterNet: https://github.com/Rikorose/DeepFilterNet

Test Media and Audio Sources

The original input video clip was obtained from publicly available online video content and cropped to a short segment for research experimentation only.

Original video source:
https://www.youtube.com/watch?v=7ARBJQn6QkM&t=1175s

The additional reference audio sample used in some experiments was obtained from the NeuTTS project by Neuphonic:

dave.wav (used to generate overlap speech):
https://github.com/neuphonic/neutts/blob/main/samples/dave.wav

All media assets are used strictly for research and experimental evaluation purposes. If requested by the original rights holders, the relevant media files will be removed.