It’s more dangerous, if you’re about to pass a vehicle which limits your vision like a semi, box truck, delivery van, etc. and you can’t really see what’s ahead of you in the right lane, if you’re sitting in the left side of the vehicle, you won’t get good visibility until you are a decent amount of the way into the lane already.

By the time you’re far enough In to see what’s ahead of you it may be too late to switch back if there’s a slow vehicle and you can rear end them. Passing on the left you can see what’s ahead by just moving close to the left and craning your head a bit.