The Church does consider demonic possession to be real. I understand, however, that from priest to priest there are different views on how the phenomenon actually presents itself. It has often been said that the greatest trick Satan ever played was to convince humanity that he doesn’t actually exist, and I certainly think it is unwise to put all cases down to mental illness (although I am sure it is a psychological problem most of the time).
You could begin to look at this topic as it is referred to in the Gospels. Even at a time when illness of the mind couldn’t have been diagnosed with the scientific accuracy that is possible today, there was a distinction made between illness and demonic possession. It would be so easy, in the absence of a medical explanation, to dismiss any disturbed person as possessed. In Matthew 4:24 - “…they brought to Him all sick people who were afflicted with various diseases & torments, and those who were demon-possessed, epileptics, and paralytics…”. Even today, the causes of epilepsy are not fully understood - why, in those days did they not simply class the epileptic as possessed, or that the paralytic was that way because of a demonic influence? I suggest that it was because there were other ‘symptoms’ which distinguished illness from possession - they are still used today to help identify true possession -eg.: abnormal strength for the person’s age, size; speaking a language they could not have known; knowledge of past events which the person could not have known in normal circumstances, becoming aggressive in the presence of religious persons, objects or relics & at the uttering of the name of God and the saints, etc.
Mark and Luke make similar distinctions to Matthew. In Mark 5 there is a story of a possessed man. Commanding the demon to speak, Christ is told: “My name is Legion; for we are many.” The demons cause the man to speak in the plural (Mk 5:12): “Send us to the swine, that we may enter them.”
While I am in favour of finding scientific and rational explanations for supposed possessions (since if a person who is mentally ill gets an exorcism rather than proper medical treatment that’s needed, there can be dangerous consequences), I firmly believe there are some things which science can’t answer - and true demonic possession is one of them. I have heard and read quite harrowing stories of exorcisms, which in many cases can go on for years. While the famous film was exaggerated in parts for dramatic effect, (it is based on a true story of a boy who was possessed) from what I have read in various documents, I think it gives a fundamentally accurate account of possession - I have even read of accounts of the possessed’s face changing form during an exorcism (as happens in the film).
As to whether one could actually invite the Devil to possess them, I’m sure they could - the question is would he take up their offer? I mean this in all seriousness, because as I see it such a person is already under the power of Satan. To me it stands to reason that likely ‘targets’ are actually those who so close to God - the ones Satan is most in danger of losing -Look at all the saints who were tormented by Satan: Padre Pio, the Cure of Ars & many others. Christ tells us that the only effecive tools against the power of Satan are prayer and fasting…