I am so sorry about this diagnosis. I've been there.
None of my girls has had a problem with vets, oddly, they have all been happy to go to the office. I have had a home care vet for the last eight years so their office visits are limited. If you decide to go the palliative care route, would a home care vet be an option? It is much less invasive, more like a nice visitor who happens to be checking him out while petting and visiting. Plus, there's no white coat to be afraid of!