Seeing sound

A spectrogram is many spectra side by side: time across, frequency up, loudness as light.¹⁸ Each choice below changes what you can see.

Sound

1Frames

Cut the sound into overlapping frames, taper each, take its spectrum: one column a frame.² Long frames resolve pitch and smear time; short frames the reverse. No frame is sharp in both.¹ Try the tones and clicks.

Frame

2Windows

A frame's edges are cuts, and a cut rings: through a rectangular window one tone leaks into every frequency. Tapered windows keep it close.³ Padding the frame with zeros draws the same spectrum finer: smoother, not sharper.

3Frequency axis

A linear axis gives every hertz the same room; hearing doesn't. Octaves, mel^4,5 and ERB⁶ give the low end the space it has in the ear. The voice's harmonics, evenly spaced in hertz, crowd together up an octave axis.

4Light

Decibels become light. Grey, viridis and magma change evenly to the eye;⁷ jet's rainbow draws edges that aren't there.⁸ The range is how deep into the quiet it looks.

Range dB

5Reassignment

Each cell's energy moves to where its phase says it came from: its instantaneous frequency and its moment.^9,10 Tones become lines, clicks become edges, from the same frames. Synchrosqueezing moves in frequency only and can be undone.¹¹

6Many resolutions

Low notes need long frames, attacks short ones. Constant-Q gives each frequency a frame of so many of its own periods;¹² editors blend several frame lengths by band.¹³ Here, 8,192 samples under 200 Hz down to 512 above 3 kHz.

7Noise

The spectrum of noise from one frame is itself noise. Several orthogonal tapers over the same frame, averaged, steady it and keep the tone.^14,15

Tapers

8No frames at all

The Wigner–Ville distribution compares the signal with itself, reversed, at every moment: a chirp becomes a perfect line.¹⁶ But between any two components it draws a third that isn't there.¹⁷ A tone and a chirp:

9All at once

Every choice above, as controls that combine, with ink, a map from the panel through its accent, even in lightness like grey.¹⁹ Below, the frame under the pointer as one spectrum.

Method Frame Window Tapers Axis Top kHz Map Range dB Padding

10Where they live

audiothe REPL's spectrogram: reassigned, octaves, mel or linear, zoomed in time and frequency
spectrogramany recording's spectrogram, in the browser
@audio/stftthe frames, windows and FFT underneath
Sonic Visualiserphase-refined spectrograms, for study

D. Gabor, “Theory of communication,” J. IEE 93 (1946): a signal's duration and bandwidth can't both be small.
J. B. Allen, “Short term spectral analysis, synthesis, and modification by discrete Fourier transform,” IEEE Trans. ASSP 25 (1977).
F. J. Harris, “On the use of windows for harmonic analysis with the discrete Fourier transform,” Proc. IEEE 66 (1978).
S. S. Stevens, J. Volkmann & E. B. Newman, “A scale for the measurement of the psychological magnitude pitch,” JASA 8 (1937).
D. O'Shaughnessy, Speech Communication: Human and Machine (1987): mel = 2595 log₁₀(1 + f / 700).
B. R. Glasberg & B. C. J. Moore, “Derivation of auditory filter shapes from notched-noise data,” Hearing Research 47 (1990).
N. Smith & S. van der Walt, “A better default colormap for Matplotlib,” SciPy 2015: viridis; magma from the same work.
D. Borland & R. M. Taylor II, “Rainbow color map (still) considered harmful,” IEEE Computer Graphics and Applications 27 (2007).
K. Kodera, R. Gendrin & C. de Villedary, “Analysis of time-varying signals with small BT values,” IEEE Trans. ASSP 26 (1978).
F. Auger & P. Flandrin, “Improving the readability of time-frequency and time-scale representations by the reassignment method,” IEEE Trans. Signal Processing 43 (1995).
I. Daubechies, J. Lu & H.-T. Wu, “Synchrosqueezed wavelet transforms,” Applied and Computational Harmonic Analysis 30 (2011).
J. C. Brown, “Calculation of a constant Q spectral transform,” JASA 89 (1991).
iZotope RX, spectrogram settings: multi-resolution.
D. J. Thomson, “Spectrum estimation and harmonic analysis,” Proc. IEEE 70 (1982).
K. S. Riedel & A. Sidorenko, “Minimum bias multiple taper spectral estimation,” IEEE Trans. Signal Processing 43 (1995): sine tapers.
J. Ville, “Théorie et applications de la notion de signal analytique,” Câbles et Transmission 2 (1948).
L. Cohen, “Time-frequency distributions: a review,” Proc. IEEE 77 (1989).
W. Koenig, H. K. Dunn & L. Y. Lacy, “The sound spectrograph,” JASA 18 (1946): darker where louder, on paper.
B. Ottosson, A perceptual color space for image processing (2020): grey and ink, even in OKLab lightness.