Malfunction of a protein has been linked to a form of mental retardation that affects up to one out of every 500 males, according to new research.