I hear you. We never did that with GEM however. There were Pascal releases where the "fixed" list was only 6 bugs, but if I looked at the GEM CMS library there might be 1000 changes. Some might have been very specific to a language (or feature), some might be latent support for future features, some might have been fixes to repair a regression from the week before. The effort required to track "substantive/possibly visible bugfixes" was too much even more a larger team. It just gets turned into "Various optmization bugfixes with volatile variables" or "with COMMON blocks" or "with parameter passing" or "with DEBUG generation".
I've tried (sometimes REALLY tried; sometimes barely tried) to keep track of those substantive changes.
For instance, come the day when we move from LLVM 10 to LLVM 'latest' (currently at 18.1.8, but will be larger by the time we catch up), I will not try to research all the changes/enhancements/fixes in the LLVM code base. You are welcome to head over to llvm.org and dig out the old release notes (they have them for every release - sometimes detailed; sometimes vague, depending on the author).
I'll try to track better AND standardize the release notes between compilers. They are little more alike than before, but not 100%. Heck, I was just happy to get all the /VERSION output strings to match!