I have Tyrian 2000 on GOG as well and was feeling nostalgic. I could replicate the issues and here’s what I did to fix it.

• Installed the game with Heroic and added it (automatically) to Steam.
• Applied a community control layout named “Tyrian” to fix the mapping issue.
• Launch the game from Steam and use to onscreen keyboard to press 3 to go to “Game DOS settings”. I had to run it twice before the controls worked for some reason. Change the music device to “FM Sound” (I think the default was “Midi 330h”) and save.
• Return to the menu and press 1 to launch Tyrian 2000.

I usually like running DOS games in EmuDeck/Retroarch though.