The Art of Random Encounters in RPGs
That's the crux of the problem in the new Paper Mario: The random encounters are completely pointless. They net you nothing but coins, and you can find coins just lying on the ground. Why bother fighting at all? The real challenge in designing random encounters is to make sure they're relevant throughout the entire game at every point. If you have to travel back to the beginning for some mission, there need to be some additional threats along the way. Otherwise, you are literally just taking a nature hike. At no point in an RPG should random encounters in the field suddenly be irrelevant.
On the other hand, Paper Mario did deliver a new wrinkle that I think a lot of games should consider with their random encounters. Before you start the battle, you can start a sequence of events on the field. For instance, jump on a koopa to trigger the fight, then in the battle they will be curled up in their shell. Use a jump attack and the shell will ricochet off other enemies, enabling a perfect strategic victory.
Some RPGs offer this, like Xenoblade's Shulk sneaking up behind enemies and using Back Slash, but never have I seen it done in so concrete and clever a way. Adding stealth tricks or other handicap-makers like in Hitman or the Batman Arkham games would be an inventive way to make random encounters more engaging. Not all the time, of course, but the game that finds the perfect balance between the simplicity of the Paper Mario system's approach and the more tried and true battle mechanics will be the one that keeps gamers hooked for a long time.
One thing that I think is underused, though, is the idea of actual surprise battles where there are visible monsters. Super Mario RPG was good at this, having enemies jump out from behind objects to attack you. That was 16-bit. Now we can have monsters swoop out of the sky when we're not looking, crash through walls, appear from the shadows. It would be like Resident Evil but with RPG trappings.
You could even use such a system to let regular enemies bypass whatever automatic buffs you have in place. Every time I play an RPG lately, starting a battle immediately triggers a handful of protection spells or debuffs for the enemy even before the first blow lands. That's great and all, and my little OCD heart loves to tinker with things like Final Fantasy XII's gambit system, but every once in a while it would rule to have to deal with something truly unexpected surprising you before you can get your shields up.
One way or another, the random-encounter system is still the basic building block of all RPGs, but it's a practice that continues to evolve and must evolve. One day someone will hit on the perfect combination, and I am looking forward to it.