Yeah I don't submit to journals often but have had similar experiences. They have also asked for addition of significant experiments with significant compute. Like they wanted us to try our technique on an additional 3-5 networks (which are all of the same general architecture we originally did) and it is very clear that one of the reviewers was an author of one of those (but adding the other two to "mask" themselves). It would have doubled the project's compute and they weren't happy when we responded that we simply don't have the budget and aren't convinced the experiments would be meaningful but we'd be happy to add them to our table for comparison. All three suggested that the paper did not have much value but the paper already had >200 citations via arxiv by this time too... (several works built on ours and even got published...)
It's a really weird political game and I just think we need to move to a system where we either have reviewing seen as a collaborative/allied effort (as opposed to adversarial) or we rethink the whole system entirely and let reviews happen naturally (i.e. submit to OR and allow comments). It's just very clear that a system can't scale if it relies on all members acting in good faith. Especially when there's high amounts of competition.
Edit: one thing I find interesting is I contextualize different than other reviewers. Like I see a experiment section where they have a node of 2080Tis and think "okay, students or small lab. Compute bound, so do the experiments they performed optimize under those conditions?" Whereas I think most reviewers don't contextualize in this manner and so I see many act as if every lab has access to many A100 nodes. I think this matters because the small labs can still do essential work, but we just need to recognize the signal may not be as strong. Their workload is probably higher due to lack of compute too (and I subsequently tend to see deeper analysis and less reliance on letting the numbers do all the talking). I don't think the GPU poor "aren't contributing" because they can't, I think they can't because gatekeeping. It's insane to expect academics to compete with big tech in a one-to-one comparison. If all you care about is benchmarks then compute always wins.
It's a really weird political game and I just think we need to move to a system where we either have reviewing seen as a collaborative/allied effort (as opposed to adversarial) or we rethink the whole system entirely and let reviews happen naturally (i.e. submit to OR and allow comments). It's just very clear that a system can't scale if it relies on all members acting in good faith. Especially when there's high amounts of competition.
Edit: one thing I find interesting is I contextualize different than other reviewers. Like I see a experiment section where they have a node of 2080Tis and think "okay, students or small lab. Compute bound, so do the experiments they performed optimize under those conditions?" Whereas I think most reviewers don't contextualize in this manner and so I see many act as if every lab has access to many A100 nodes. I think this matters because the small labs can still do essential work, but we just need to recognize the signal may not be as strong. Their workload is probably higher due to lack of compute too (and I subsequently tend to see deeper analysis and less reliance on letting the numbers do all the talking). I don't think the GPU poor "aren't contributing" because they can't, I think they can't because gatekeeping. It's insane to expect academics to compete with big tech in a one-to-one comparison. If all you care about is benchmarks then compute always wins.