The obvious answer is when the max dose of pain meds don’t control the pain. Don’t hesitate to ramp up the dose, BTW, if you see any signs of pain. That said, it isn’t always that clear. My two osteo dogs both broke their legs before I even knew there was anything wrong (and the screaming was the absolute worst thing I have ever heard), but I have had two other dogs with other types of cancer, and I could see they were in pain and tired of fighting and ready to go. I have also had pets, cats and one dog, that never gave me “the sign”, I just had to make the call, best guess.
i guess what I am saying is, other than the obvious pain thing, you just have to go with your knowledge of your dog. Sometimes you have to be the grown up and make the hard decision, and that really sucks.
edited to add, I have always believed in A day too soon is better than a day too late, and after 20+ pets over the years I have never had cause to not believe this.