No, your confession was not invalid if you truly forgot to confess sins. The only way your confession would be invalid in this case would be if you purposely failed to confess mortal sins or lied about them. Otherwise all your sins, both mortal and venial, that you committed prior to confession have been forgiven. It would be adviseable to mention any mortal sins you failed to confess at your next confession as a sign of contrition, but again, they have already been forgiven.
I would advise you to perhaps try to make a more thorough examination of conscience prior to making your next confession and hopefully that way you will have less difficulty remembering your sins. As far as getting nervous, most people get a little nervous in the confessional (particularly if they are new to the sacrament) but there’s really no need to be. Remember, the priest has heard it all and is not going to chase you out. He is acting as a representative of Christ and the Lord is Love and Mercy itself. Likewise you must always remember that anything you confess is completely secret between you and the priest and the priest can never mention anything anyone confesses to anyone.
A good analogy I’ve heard many times on this site and elsewhere has been the idea of going to see the doctor (because, let’s face it, Christ is the Divine Physician). Now if you have a medical problem, and it’s really embarassing, and you don’t let your doctor know about it when you see him how is the doctor supposed to heal you? Clearly you must move beyond your embarassment and allow the True Doctor to do His job because He knows you better than you know yourself and wants you to be healed.
