GEM *IS* OPENSOURCED.. It was GPL'd by caldera sometime back.
You might want to check out the following FreeGEM sites..
http://www.retroarchive.org/cpm/archive/unofficial/gemworld.html
http://www.deltasoft.com/
They have these sources compiled under both pacific C and GCC. Considering GEM/1 is closest to ATARI GEM that would be the source to look at.
DO you think we need TOS compatibility now? or just GEM compatibility? Games are a big prob I guess. If we only need GEM we could use the above and try to bolt them on to a well documented open sourced kernal.. maybe something like uC-OS RTOS?