In modern speech recognition (such as OpenAI's Whisper), generative music models, and voice synthesis neural vocoders, models rarely train on raw waveforms or linear spectrograms. Instead, they use Mel-spectrograms.
The Limitation of Linear Hertz (Hz)
In physical acoustics, frequency is measured linearly in Hertz. However, human ears do not hear pitch linearly:
- A 100 Hz step from 100 Hz to 200 Hz is an entire musical octave.
- A 100 Hz step from 15,000 Hz to 15,100 Hz is practically imperceptible to human ears.
If an AI model trained on a linear spectrogram, it would allocate most of its parameters to ultra-high frequencies that carry almost no linguistic or musical meaning.
What is the Mel Scale?
The Mel scale is a perceptual scale of pitches judged by human listeners to be equal in distance from one another. Frequencies below 1,000 Hz follow linear spacing, while frequencies above 1,000 Hz follow logarithmic spacing.
m = 2595 × log10(1 + f / 700)Formula converting frequency in Hz (f) into perceptual Mel units (m).
How a Mel-Spectrogram is Generated
- Compute STFT: Generate a standard linear spectrogram using the Short-Time Fourier Transform.
- Apply Mel Filter Bank: Multiply the linear frequency bins by overlapping triangular filters spaced evenly along the Mel scale.
- Logarithmic Compression: Take the logarithm of filter bank energies (log-mel) to approximate human perception of loudness (decibels).