OK, had a little time to check this. If I switch of Steam Haptics this behaviour goes away. So its definitely that and not game rumble.

However I noticed something else. I only get the haptics feedback when I slightly move the joystick. If I move it fully in one direction the haptics feedback stops. So I guess it must be something in controller settings that changes stick behavior between full moves and moving them less than that. Maybe loops into what the other poster said.

Thanks again for the suggestion!