I change mine 2x a year, regardless of whether it looks like it needs changing or not (usually May and November). Sometimes it's visibly dirty, sometimes it's not (but they're usually dirty after the fall season). I've found that when swapping them regularly, dirty or not, my vehicles maintain more of a "fresh" smell. When you change them, you'll immediately notice that it seems like you're breathing fresher air. Keep in mind that they filter microscopic particles, so even when they appear to be clean, airflow is probably reduced to some degree if they've not been changed in awhile. Vehicles also often sit in humid environments and our filters are permeated with humid air. Mold and pollen spores, visible or not, are also bound to collect.
Is this frequency of changing necessary? No, absolutely not. Some have a vehicle for years and never change them. In the end, it's personal preference, although clogged filters can be hard on blower fans and affect the overall efficiency of the system.