It's more so the photograph of my house than knowing my street number.

What about that makes you uncomfortable exactly? I'm genuinely curious, as I've heard that sentiment expressed many times before.

It's not like your house is otherwise invisible...

My house isn't necessarily invisible, but having photographs of my (and every other house) sitting in some company's database makes me uncomfortable.

It's kind of like those cars that drive around recording everyone's license plate. Sure, that information is sitting out in the open anyway, but streamlining the collection of that information gives me the creeps.

I don't think most countries have this option.

America and Europe do at least. Use the "Report a Problem" link when looking at your house on Street View.