Neil Oliver: ‘It’s hard to tell yourself you’ve been taken for a fool but open your eyes’