There are a bunch of different trails leaving from the Hidden Valley Nature Center. With back country campsites out near Little Dyer Pond, you'll most likely only see a few overnight hikers and have the rest of the trails to yourself.
I camped by the lake on a bike tour a few years ago. It was a lovely, quiet and clean spot, with a privy and picnic table. I took a dip in the pond, and charged my phone at the pavilion. Highly recommend!