Not sure without knowing more of your background - provisional answer is "maybe." Some parts of Ch.1 are certainly math-heavy. The video lectures are quite accessible. Did you study, say, ~1 year of calculus?
yes, 1 year of calculus. differentiation and integrals were the only two calculus topics that I recognized from that chapter. at the time when I was trying to get through sicp I wasn't sure whether the pay off was worth the effort of covering all those math topics and I wanted to learn other programming related topics as well with my time.
At least up to 1.3.3, if you've already done single-variable calculus, the math isn't bad - it's stuff you likely saw before (if a while ago).
It seems heavier on math than it really is because (I believe) the authors want the reader to see the parallels between math notation (functional, declarative, concise) and Lisp implementation (also functional, declarative, and quite concise).
I agree, math is used as a domain for black box abstraction for the same reason banking is used for the example domain for concurrency and graphics for data abstraction - the domain provides good examples for trivial implementations.
A person no more needs extensive understanding of mathematics than they need such understanding of accounting or image processing.