Are you sure that the difference is due to the allocations? I would expect this to be dominated by matrix multiplies or svds. Are you comparing this with the same blas/LAPACK?

Edit : OK, I see those are small matrices. Then Staticarrays should be a nice contender here, both for speed and readability.