You land on the Sound side at a pier beside the fort. The fort is worth doing first, while you are fresh and while the interior is cool, and it takes twenty minutes to half an hour unless a ranger talk is running.
Then the boardwalk. It is a few hundred yards of raised timber across the middle of the island, fully exposed, and it is where everybody's optimism about carrying four bags and a cool box meets reality. Take one trip, not two.
The Gulf side is where the day happens. The swim beach sits in front of the pavilions, with restrooms, outdoor showers and a small seasonal concession behind it. The water is clear green, deepens sooner than the mainland does and carries small surf, and there is a marked swim area with seasonal lifeguard cover. Beyond the marked area you are on an unguarded open coast twelve miles from a hospital, which is worth remembering before anybody swims out to look at something.
Realistically you get about four hours ashore, which is enough to see the fort, swim properly, eat and still be at the pier early rather than sprinting for it. The heat between noon and three in July is serious and the pavilions are the only relief. Plan to be under cover or in the water in that window, and do the walking at either end of the day.