Moving the cursor without clicking or scrolling is hardly ever useful (except for in menu structures). So it is better to let go of the mouse if you do that for longer periods of time. If the system would recognise mouse motion as “valid” mouse action, you would soon start to learn shaking the mouse frequently, just to avoid the vibration. That is to be avoided.