Apparently the issue is games that have very many tiny files that aren't all inside of one larger file. This is a thread that sounds about right: https://www.reddit.com/r/SteamDeck/comments/tcomih/reserving_space_extremely_slow_on_sd_card/

So if a game is like GW2 and has nothing but a couple of extramassive files, it's very fast to allocate the space. If tha game has hundreds of thousands of very tiny files, it takes forever to allocate the space.