A bunch of excellent remarks from different posters, and this one is especially good. It points to the parallel between the theistic and atheistic beliefs. They both are founded on a basic principle: “there is some unexplained and unexplainable foundation for all existence”. The word “explanation” means to reduce someting to a “more” basic entity, and the ultimate foundation (obviously) cannot be reduced any further.
For the theists this basic explanation is “god” (not necessarily the Christian God), but some entity, which “transcends” or surpasses the “natural, observable world” - the usual word is “supernatural”, (though it could and should be called “unnatural”, after all there is no “sub-natural”, is there?

).
For the atheists it is the universe. The word “universe” is STEM - space, time, energy, matter (energy and matter are actually the same). For the atheists the questions: “where did the universe come from?”, or “what existed before the universe?”, or what exists outside the universe?" are nonsensical questions, just like the question: “what exists to the north from the North Pole?” would be a syntactically valid, but semantically meaningless utterance.
When comparing the theistic and atheistic “basic” stance, it is obvious that the atheistic one is more “parsimonous”. It looks at the “stuff” we all experience, and is content with it. Of course a parsimonous answer is not necessarily a valid answer. It may well be that the atheistic view is incorrect. However, there is nothing that would necessitate the assumption of a “transcendant” or “supernatural” explanation. Theists don’t like this. They insist on some “mystical” approach. For them the idea that “this is all there is” is “dirty materialism”.
But hard as they try, they cannot come up with anything that does not have a “natural” explanation. They try questions: “without God where does love (or justice, or morality, or whatever) come from?”. When there was no real science, everything was attributed to some “god”. The lightning was imagined as God’s flaming sword. Illnesses were imagined as something caused by “demons”. Etc… As science evolves, the so called “supernatural” explanations become unnecessary and superfluous. And observe, that the so-called supernatural “explanations” were anything but explanations: they all say: “an unknown being, using unimaginable means made it somehow happen” - which is the antithesis of an explanation.
When it comes to the Big Bang, and the so-called “beginning” of the universe, there is a false assumption, saying that the universe “started” with the Big Bang. That is yet another incorrect assumption. Rather, the current form of the universe started at that point. What was the nature of the singularity is unknown.