I don’t know if this will help you in any way, but I had a similar issue with my own spiritual director. He’s very good about making time for me, is kind, and has the patience of a saint (people dealing with me do tend to need some level of patience - I’m very shy and getting personal details of my spiritual life is probably a bit like pulling teeth).
Despite his patience and good humor, I did not get the impression that he particularly cared for me. He seemed to forget things that I told him and asked some of the same questions repeatedly. One day, outside of spiritual direction, he was in a position where he was assigning tasks to different people and it became apparent that he didn’t know my name. I was hurt at the thought that I was sharing my spiritual journey with someone who hadn’t bothered to learn my name.
God had mercy on me and sent a friend who takes a lot of interest in my discernment. For a time, I compared my poor spiritual director against this friend, and found him wanting. Sadly, I held a grudge against him as a result.
Eventually, some level of decency got through my stubbornness. What did I ask of my spiritual director? I wanted him to help me grow in faith and discern God’s Will for my life. Was he doing that? Absolutely. 110%. Was it fair to ask him to remember every little thing I told him? No way. He has so many parishioners, so many things to worry about, that that would be ridiculous to expect. (I even realized later that the repeated questions were deliberate. They are how he’s keeping track of my spiritual growth). It wasn’t even entirely fair to ask him to know my name. My name is unusual, he is from a country where it is never used, and it’s only one of the many that go to him.
Eventually, I realized what I had in mind when I asked him to be my spiritual director was a combination of spiritual director and friend. But spiritual direction is a professional role. My spiritual director is doing everything he is supposed to and is a tremendous help. I am really quite blessed to have him. And if I want to share my spiritual journey with someone who has a particular interest - that’s what my friends are for. I had no reason to feel slighted by my spiritual director. He did nothing wrong.
Again, I don’t know if this helps you. If you feel that there are bad feelings between you, then maybe it’s best to change spiritual directors. But if I may suggest something, do something to show you appreciate what he’s done for you (you never know what’s going on outside spiritual direction. He may be offering many prayers on your behalf). Have a Mass said for him, give an hour in Eucharistic Adoration, volunteer at the parish - something like that. It may help him if he knows his efforts mean something to you.
Pray to God to know what is best. And ask Him to send you a spiritual friend.