It's certainly possible. I think the slowdown is caused because the Videl chip in the falcon uses quite a bit of CPU bandwidth, obviously using more in higher resolutions/colour depths. I remember being advised to run DSP music programs in mono or 16 colours because the SCSI transfers would be slowed down if I was in a higher colour mode.
This won't affect the TT, the Crazydots (and MatGraf matrix, blah blah) have their own graphics processors, and the amount of bus bandwidth they take up is fairly constant (Like with PCI, it never runs at more than 33MHz).
Check out my Advanced Computer Architecture assignment for more evidence of the effect of Cache (and NVDI, Blitter etc). I can't put it up right now, but sometime later on it'll be there at www.geocities.com/shalroth/aca4.doc (MS-Word 97 format)