Could be to do with where you're searching. The fact that ag skips everything in your .gitignore seems to have helped when I tested. Both of these were run at the root of my projects directory...
$ time grep -r f_admin .
35.87s user 6.50s system 65% cpu 1:04.47 total
$ time ag f_admin
1.51s user 4.94s system 196% cpu 3.284 total
It's a proprietary codebase, but it consists of around 500MB of mostly C and C++ files and their headers. There are a few other files (makefiles, a few perl/python scripts).