It was just a gradual development. At first, as you noted, Confession was only given for very serious sins (ie. the lapsy who renounced their faith during times of persecution). Then it gradually grew to those who committed very serious sins (i.e. murder, rape, incest), and it was only administered once on Good Thursday (the ashes on ash Wednesday formerly symbolized the beginning of the penitential process whereby the penitents would wear sackcloth and be covered in ashes).
Then the Irish monasteries began serving as spiritual and intellectual centers for those in Ireland. I forget the details, but somewhere in the process they developed a tariff penance system whereby those who committed a sin would go to them, receive spiritual direction and a penance, and he or she would presumably be forgiven once the penance was complete. We still have many, many copies of the penitential manuals from this period that very clearly indicate that confession was not as it is today.
Then during the reforms under Charlemagne it was decided that public sins would be confessed in public and private sins would be confessed in private or to a priest. During this period there were three divisions of penance: solemn public penance for serious sins, non-solemn public penance for less serious public sins, and private penance for private sins.
It wasn’t really until the second half of the twelfth century that Peter Abelard started emphasizing the subjective factors in human sin (i.e. the concept of inner sorrow or contrition being important). At this point, the role of the confessor shifted from merely assigning a penance (penitentials were becoming oh so confusing) and instead focuses on the contrition of the person confessing. It is also at this point in history that a declarative form of absolution developed (i.e. I absolve you from your sins in the name of the…).
Aquinas developed a rather sophisticated theology of penance, and by Lateran IV it was mandatory for all Christians once a year. It was also at this time that the relationship between confession and the Eucharist was officially affirmed.
I don’t remember which books I read to gather this information, but Remember that one was written by an Australian priest.