Maybe I'm misreading you so straighten me out if so. You seem to be making the point that people cared more about not going to war than about ending slavery. That's certainly true! And the reason why is obvious: they didn't need to go to war to end slavery. The free market was already tipping the scales further and further in the favor of freedom lovers. Three states in a row had been admitted as free (that is, three states in a row had a majority of Northerners that wanted to live in an explicitly free state), and in a fourth anti-slavery sentiment was so strong that people literally
went to war to fight for it. Immigration was surging with abolitionist leaning peoples such as the Germans. Couple that with the slave states that were about to flip to free, and the United States could have simply abolished slavery in Congress in short order.
That's the whole reason why the South seceded in the first place. They could see the writing on the wall, that they were becoming more and more outnumbered by people who explicitly did not want slavery. If the average Northerner really didn't care about slavery in the South, why secede at all? Where was the threat to provoke such an extreme response?