It's more simple than you think.
They're not distracting.
Don't be disappointed in the answer. Just consider:
As human beings, we have an innate sense of when someone is being false. It's an adaptation to warn us of danger. Inherently, we can sense when the average person is hiding something or presenting a false front.
A good actor, therefore, is simply a person able to bypass that internal warning system.
If I, as a viewer, believe you...well, you're doing your job.
Simple as that.
Sanford Meisner called this principle, "behaving truthfully in imaginary circumstances."
I call it, "ability to bullshit."
Either way, it's all a good actor--or politician or salesman (but I repeat myself)--has to do.
If I am distracted by your performance? Well, somthing's off. Simple. Something isn't ringing true. And that takes me out of the story. And that's bad.
Betcha thought it was more complicated than that.
Sorry to disappoint you.