"I can't possibly do the shows that you deserve," he said after nixing two comedy performances in Virginia with his "Only ...