I was wondering why is the doctrine of Papal Infallibility such a big deal why is it so scary to non-Catholics?
Because it is not true. It was also voted at VaticalCouncil I under the most bizarre circumstances.
Denying the validity of Vatican I Council, Archbishop Peter Richard Kenrick refused to speak at any of the general sessions after June 4th, 1870. Bishop Joseph Strossmeyer of Diakovar told Lord Acton, There is no denying that the Council lacked freedom from beginning to end. To Professor Joseph Hubert Reinkens, Strossmeyer said, that the Vatican Council had not had the freedom necessary to make it a true Council and to justify its passing resolutions binding the conscience of the entire Catholic world. The proof of this was perfectly self-evident.
Bishop Francois Le Courtier spoke for many when he wrote, Our weakness at this moment comes neither from scripture nor the tradition of the Fathers nor the witness of the General Councils nor the evidence of history. It comes from our lack of freedom, which is radical. An imposing minority, is crushed beneath the yoke of a restrictive agenda, which contradicts conciliar traditions. It is crushed by commissions which have not been truly elected and which dare to insert undebated paragraphs in the text after debate has closed. It is crushed by the commission for postulates, which has been imposed from above. It is crushed by the absolute absence of discussion, response, objections, and the opportunity to demand explanations; The minority is crushed, above all, by the full weight of the supreme authority which oppresses it Furthermore, the opposing minority of about two hundred bishops objected to the short time allowed for studying the text on primacy and infallibility as well as to the practice adopted by the deputations of inserting new clauses at the last moment.
The minority bishops were not allowed to discuss the historical objections against Papal Infallibility with the deputation on the faith. In a letter Bishop Le Courtier complains, See what more than aught else destroys our liberty: it is crushed under the respect we have for our Head. Later in frustrated anger, Bishop Francois Le Courtier tossed his council documents into the river Tiber and left Rome. The papers were retrieved and brought to the attention of Vatican officials. The price for this gesture was extracted three years later, when he was dismissed as Bishop of Montpellier.
In spite of the unequal representation and Pius IX using the power and prestige of his office, there was still a large number - eighty-eight bishops - who voted against Papal Infallibility, which was enshrined in the constitution,
Pastor Aeternus. Sixty-two bishops, many of whom were
de facto opponents, voted with reservations, with only four hundred and fifty-one giving a clear yes - this is less than half of the one thousand and eighty-four prelates with voting privileges and less than two-thirds of the seven hundred bishops in attendance at the commencement of the Council. Over seventy-six bishops in Rome abstained from voting and fifty-five bishops informed the Pope that while maintaining their opposition to the definition that out of filial piety and reverence, which very recently brought our representatives to the feet of your Holiness, do not allow us in a cause so closely concerning Your Holiness to say
non placet (it is not pleasing) openly in the face of the Father. This statement alone speaks volumes for the subservience that these bishops had for the immense authority figure of the Pope - a presence unknown in the councils of the Early Church.
Thus lacking a moral unanimity or even a clear two-thirds majority, Papal Infallibility was now elevated as an article of faith equal to the Holy Trinity and the Incarnation.