We know Luther removed, in addition to the deuterocanon texts, texts everybody else sees as belonging in the canon.
If you are referring to the epistle of James & other NT books, this is historically incorrect. Luther questioned them - just as many early church fathers questioned some of these books (see Eusebius’ Ecclesiastical History for examples). However, unlike them, Luther included not only all 27 books of the NT, he also included the deuteros too, in his German translation, just in a separate uninspired section.
Another way to ask is: what made Luther more inspired and authoritative than the combined knowledge the Church fathers and theologians during the previous 1500 years?
The problem is that we don’t have an officially “defined” Biblical canon until 1546 at Trent. The ECFs in the first few centuries, including Doctors of the Church, did not universally agree on the Biblical canon. Irenaeus supported the “smaller” canon Protestants embrace, and he included Wisdom & the Shepherd of Hermas in his
New Testament canon, not the Old, which demonstrates as late as the late second century, Wisdom was still not in the Old Testament.
None of the fourth century councils included Baruch or the epistle of Jeremiah. They would not be added to later versions of the Vulgate until sometime in the 9th Century AD. The Councils of Hippo (397) & Carthage (397) “added” the additions of Ezra, which were not in the earlier Council of Rome (382). And Carthage removed Revelation, because it was rejected by many Eastern Church Fathers (which then got added back in the 5th Century Council of Carthage, because it was included in Jerome’s Latin Vulgate in 405 AD).
You don’t get an Ecumenical Council with the same books in Catholic Bibles until Florence (1441), and even then, there were many who questioned some of the Deuteros, like Sirach well-after Florence. Even after Trent, Cardinals like Cajetan & Ximenes (and even Erasmus) preferred the “smaller” Biblical canon of the Reformers, despite rejecting the Reformation movement.
So, the history of the Biblical canon isn’t as “consistent” throughout church history as one might think.