Reliably sap-stabbing an engineer who isn't asleep or a noob is a tricky proposition at best. It takes a master of backstabbing, appropriate and fast weapon switching depending on what goes down, and even watch usage to really pull it off say, 9 time out of 10. I can usually do it fairly consistantly on bad engineers, but career engineers like Max and Doom say are much harder to pull it off on.
Really, it comes down to this; being able to get to the sentry unnoticed and then switching to the knife and backstabbing the engineer before he notices the sentry is sapped and starts spinning around like crazy trying to spot you. If you can't get the backstab off, back away quickly, pull out your revolver and immediately start firing on him, otherwise you're one or two wrench hits away from defeat. At range, if you start firing first, you might be able to kill him before he has the time to pull out one of his guns and kill you. Plus, he probably wants to just break the sapper, if he does go immediately for the sapper, you've misjudged his reaction because you should have stayed close and tryed to backstab him again. Your only option now is to quickly cloak, hide, redisguise and wait for an other chance.
Pretty much any team support for the engineer makes this 10 times harder or at least, ensures that you're gonna die even if you successfully break the sentry and/or kill the engineer. This is why a pyro body guard is great to have when you're sentry camping a bottleneck. Thanks to the poofer, he can even repel Rocketman and Grenadier attacks to a degree.
My favourite sap-stab foul up ever? The time on pl_badwater when Niku or Soupy (can't recall) had a sentry on that first big roof that over looks the second point. I ran up for the sap-stab but failed to connect with the stab as my knife poked the engineer in his arm or something. Running for my life, I ended up falling off the roof. The startled engineer made the same mistep with his wrench flailing at me. He connected somewhere during the fall to the ground below and killed me with a crit-hit but with the roof now engineerless, my sappers did their job.