This question is legit!
"what is it that tells you to let the image go and move on to something else or that it's worth putting in more work to salvage it?"
There is no simple answer to your question, because I think that ultimately the method you are talking about "doodling around to try to make it work" is like throwing a hail Mary at the Superbowl. I think everyone does this for a while, until they've done it enough with lackluster results, and eventually they just do the process: research, thumbnail, line art, values, color, render. The process is slightly different from artist to artist, but mostly follows this path. I was very reluctant to do full line art for a long time or thumbnail, because I thought it lost the spontaneity, and it felt like it took a long time, when I just wanted to paint. But, fixing a drawing is waaaaaay easier than fixing a painting, so now I just do it. I also feel less bad about detonating a sketch than a painting. Check out Lee's great video on "how to get a great illustration every time". You don't have to do 50 thumbnails every time, but at least do a handful. Sometimes, your first one will be the one you end up using, but usually it's not, and if it is, you feel better going forward because you played with different options. With practice too, you will notice when an illustration goes off the rails sooner, and catch yourself, when you are noodling and not making progress on an image.
IMHO, the objective answer is, "does the illustration tell the story you want it to tell?" If you, or someone looking at it is not sure, it's a no. Illustration is all about communication.
That may have been a bit of a rant, but I have, uh...feelings about this subject (aka, I have wasted a lot of hours), and I really appreciate that you are thinking about it. I think it just takes practice of knowing when to detonate, and when to salvage. And, do the process.
As far as your illustration goes, it's really for you to decide. What is the story you are trying to tell, and does it do that?