I suspect the CPU is working harder to transfer info on all those objects, and is unable to keep the GPU busy. Your computer is "CPU-bound" (or limited) in this situation rather than "GPU-bound".

No expert myself, I base this on videos at the Hardware Unboxed YouTube channel.

You might try running a CPU performance meter and see what that shows.

Edit: You do have a powerful CPU, so I dunno... something to consider.