Here are my two cents. I think that the main problem in this passage, specially indicated in Lk 9:49 is that the disciples were concerned not because the man was following Christ, but because “he followeth not with us.”
John is putting the emphasis not on the man following Jesus and doing good in His name, but on the man not being among them, not being part of the special “us” of the 12. Notice how John said “he followeth not WITH US”, he didn’t say “he does not follow YOU”.
Taking the verse in the context of the chapter starting with verse 41, we see that it all started with the disciples not being able two cast out a demon. Jesus tells his disciples before rebuking the evil spirit in the boy “O faithless and perverse generation” (Lk 9:41). Later we see in Lk 9:46 that “there entered a thought into them [the disciples], which of them should be greater” and Jesus saw “the thoughts of their hearts”. When we consider all these verses together, we see that Jesus saw the sin of pride in the disciples’ hearts! They felt they were better, the thought they deserved to be among the greatest in Heaven. (Lk 9:46). But Jesus told them instead they had to be like children to be the greatest in heaven (Lk 9:47-48). And what does it mean? That the disciples should leave aside their pride and become humble like little children!
But hold on. Even after this, John and his brother wanted God to destroy a city because they did not received THEM (Lk 9:52-54)! I can almost see James and John arguing on their way back to Jesus “how dare those Samaritans not to receive us! Don’t they know who we are?” See? It was all about THEM, over and over again is only about the disciples, not about Jesus and his message. That’s why Jesus responds to them in Lk 9:55 “you know not of what spirit you are.”
Once we have this context clear, we can now understand Jesus words in Lk 9:50 “Forbid him not: for he that is not against you is for you.”
Jesus is nipping in the bud the sin of pride among his disciples. The whole context of the passage does not show Jesus setting the foundation for a merely “spiritual church”. Besides, how did this man know how to cast out demons by the power of Jesus name? Why did this man know he could even do that? More over, why were not the disciples, that were so close to Jesus, able to cast out one demon and this complete stranger could cast out “demons”? Can we assume that this man did follow Jesus and his teachings, being part of those thousands in the crowd, but did not befriend the disciples? If the man was able to cast out demons in Jesus’ name that tells me that he had his heart in the right place. Otherwise his prayers would have been as ineffective as the disciples’ to cast out demons. Go figure the hit to the disciples pride! They looked like fools! The stranger man could cast out demons and they, the special ones closest to the Lord, could not!
Now, there are several places in Scriptures in favor of the visible church, as others have already pointed out, but two places in particular come to my mind: Eph 4:3-5 “Careful to keep the unity of the Spirit in the bond of peace. One body and one Spirit: as you are called in one hope of your calling.One Lord, one faith, one baptism” and then Jesus words in John 17:17-22 “Sanctify them in truth. Thy word is truth. As thou hast sent me into the world, I also have sent them into the world. And for them do I sanctify myself, that they also may be sanctified in truth. And not for them only do I pray, but for them also who through their word shall believe in me. That they all may be one, as thou, Father, in me, and I in thee; that they also may be one in us: that the world may believe that thou hast sent me. And the glory which thou hast given me, I have given to them: that, they may be one, as we also are one.”
For the world to believe it is necessary for the church to be visible! I’m not denying here the invisible dimension of the Church Suffering and the Church Triumphant, but the concept of just an invisible church will not give testimony of Christ as to move the non-believers to believe!
Think about this, the spiritual things might be concepts and realities easily accepted by believers, but maybe not for the non-believers. When they see such disparity and disagreement even among Christians about what we believe they are just going to think we are a bunch of crazy people, proclaiming we believe in the same Christ and reading the same Bible and sometimes having totally opposite interpretations of the same text!
if the church does not have a total and complete unity like it says in Ephesians, if we don’t have one hope, one faith, one spirit and if the world cannot visibly see this unity as the verses from John 17 say, how will the world believe that the Father sent Jesus?
And the sacraments? What are those and how many are there? 
Other than that I’m a simple poster on CAF.