I don't personally have any stories about the smaller towns, but a coworker of mine was off the beaten path and ended up socializing with a family for an afternoon, drinking tequila with a gentleman while his grandchild sat on his lap.
It was a very memorable side-trip that he couldn't have booked through a tour guide.
http://this-twisted-history.blogspot.com/