For hundreds of thousands of years, Neanderthals spanned the northern reaches of Eurasia, from the Atlantic to Mongolia. Though they were well adapted to their environment, it was a harsh terrain just south of the ice. They lived in small groups and their population was never that large. The ancestors of modern humans lived in the south where there was greater abundance and so a larger population. In addition, a cultural revolution among modern humans occurred ~50,000 years ago which allowed for them to expand rapidly into territory previously not occupied by other humans and rapidly increase their numbers. In short, the initial Neanderthals were probably overwhelmed by the numbers of our species.
Either there were many more of us, and we overwhelmed them. Or, the Neanderthal genes were selected against in mixed populations due to incompatibilities. Or a combination of both.
More than 500,000 years separate Neanderthals from humans, and our genomes have drifted apart. Incompatibilities between Neanderthal and human genes made it so that their’s were flushed out of our genome over time like they were impurities. Ultimately the number which remained after the great mixing and then the selection was the 2% (or less) you see today.