I believe there have been successful experiments to detect underwater mountains by their gravitational pull, but I don't think it's an operational system type thing. https://ieeexplore.ieee.org/document/5345843
and also, I wanted to mention another submarine navigation technique that the Soviet Russians pioneered. I thought it was called Crazy Ivan but I just looked that up and that's Soviet Russian submarinery, but not this one. The idea is to carefully map the ocean floor features in a particular area, and water currents, and know your boat well, and use that information to "blindly run a route" at high speed. I'm not an expert but something like, when submarines are traveling at speed, stuff like sonar (at least the quiet kind) is useless because of all the turbulence around the boat, so the Russians figured out they didn't need to be stealthier to evade US subs in the Baltic, they just needed to know where they were and then go fast. The goal was to get to the open sea were it's easier to get loast in the vastness. Maybe these Navy subs were trying something like that.
and also, I wanted to mention another submarine navigation technique that the Soviet Russians pioneered. I thought it was called Crazy Ivan but I just looked that up and that's Soviet Russian submarinery, but not this one. The idea is to carefully map the ocean floor features in a particular area, and water currents, and know your boat well, and use that information to "blindly run a route" at high speed. I'm not an expert but something like, when submarines are traveling at speed, stuff like sonar (at least the quiet kind) is useless because of all the turbulence around the boat, so the Russians figured out they didn't need to be stealthier to evade US subs in the Baltic, they just needed to know where they were and then go fast. The goal was to get to the open sea were it's easier to get loast in the vastness. Maybe these Navy subs were trying something like that.