Wouldn't the type system complain that you're returning nil in a function that is not declared/inferred as returning nil?

This is correct for any "normal" (a.k.a. "item level") function, since their signatures must always be explicit. But you're allowed to infer the signatures of closures, which is the only place where this problem could arise.

  // This function lives at the top level of the module, so its types must be annotated
  fn foo(bar: int) -> int {
      // Here's a closure without type annotations:
      let baz = |qux| { qux + 1 };
      // And a closure with type annotations:
      let baz = |qux: int| -> int { qux + 1 };
      // Which means that this would be a compile-time error:
      let baz = |qux: int| -> int { qux + 1; };  // returning nil, but expected int
      // But if you expect this to return int, then there are very few cases where
      // this might not be a compile-time error:
      let baz = |qux| { qux + 1; };
      log(error, baz(bar));  // will print "()", but did you want bar + 1?
      io::println(fmt!("%?", baz(bar)));  // same as previous
      io::println(fmt!("%d", baz(bar)));  // this one *is* a compile-time error
                                          // macros ftw
      baz(bar);  // this will also be a compile-time error
  }

What if foo() could return nil in some cases?

Then you need an algebraic data type having nil is one of its constructors, and that function returning that type.

AFAIK, Rust doesn't have nil. So, the closest thing is using the option type.

In Rust, nil is the name for the unit type (written as empty parentheses, like "()"), used whenever a function doesn't return anything meaningful. You're correct in that instead of null pointers it has the None variant of the Option enum.