This line shows the Duke's male entitled pride: he won't lower himself to talk to his wife and confront her about what he fears, as it would show him as paranoid and weak. It also suggests there's no point arguing with a woman as she is so far below him, and incapable of rational thinking, AND he shouldn't have to explain it to her – she should be meek and obedient and submissive