"I dunno. I'll admit I'm no engineer, but I'd have second thoughts about building ships out of concrete."

It was difficult to disagree with my friend Perry as we plodded across Sunset Beach, the fine, airy sand spilling into our trainers. Concrete is hardly renowned for its buoyant qualities, a fact seemingly underlined by the sight of the SS Atlantus ahead of us, its broken grey hulk jutting from the choppy Atlantic waters. It was one of a dozen experimental vessels built of concrete due to a steel shortage during World War I. It didn't sink, however - the ship served to transport American troops back home from Europe and was to live on as a ferry before breaking its moorings during a storm and running aground off the coast.

Numerous attempts to refloat the Atlantus failed, and now the shattered, century-old hull is one of several curiosities and attractions found in the New Jersey resort of Cape May. Not all of the state is rusting industry and anonymous freeways, despite the sorry scenes that greet passengers at Newark International. Cape May City, some three hours' drive south of New York, is the USA's oldest seaside resort, a town with a population of barely 4,000 that swells by over 50,000 in the summer months.

The resort's reputation owes plenty to the buttercup-yellow Congress Hall, a 19th century hotel with two pillared wings of suites framing a palatial lawn that rolls out to the sea front. Before US presidents retreated to Camp David for rest and relaxation, they would reside at Congress Hall. President Benjamin Harrison lived at the hotel during the summer months and would conduct affairs of state from there.

Our day trip began with brunch at Congress Hall in the Blue Pig Tavern. Perry and his wife Devon regularly take road trips out of NYC and Cape May was a favourite destination. The tavern's sausage and biscuits in gravy, poached eggs and potatoes also came highly recommended by my two guides. We don't cherish gravy in the UK quite like our US brethren who'll willingly pour it on everything, and together with the rich sausage and the sweet biscuit, a little ketchup and a trickle of runny yolk, I felt a genuine pang of sadness when I realised there was only a mouthful left on my plate. The complimentary bowl of salt-water taffy, which originated further up the coast in Atlantic City, barely helped ease my pain.

Cape May offered plenty of opportunities to walk off the calories. The beaches are regularly voted among the best in the country and stretch for miles along and around the cape. We ploughed through the golden dunes to Cape May Point, our path eventually blocked by towers of grey-green concrete, abandoned and in disrepair, perilously close to the encroaching tide. The possibility of submarine attacks during the second world war led the United States Navy to establish a significant presence in Cape May.

Their horseshoe-shaped bunker served as a gun emplacement to protect coastal shipping and originally stood nearly 300 metres from the sea-front, but Cape May has suffered coastal erosion like few other towns. Shifting sands and violent storms destroyed the township of South Cape May in the early 20th century - only a handful of homes were saved and moved elsewhere inland. The pilings that the bunker rests upon were only ever meant to stabilise the structure - how long before thousands of tons of concrete collapses into the sea?

The bunker is part of Cape May Point State Park, as is the nearby Cape May Lighthouse. We paid our $7 entry fee and a series of grey-haired volunteers pointed the way to the 199 steps up the spiral staircase, where we were greeted by the bearded lighthouse keeper, clad in stereotypical navy sweater and peaked cotton cap and looking not disimilar to a villain from Scooby Doo. In fact there's a reasonable chance he was; Cape May City is famous for its hauntings, and the nearby freshwater Lake Lilly is not only a renowned birdwatching spot, but in the 19th century was a meeting place for pirates. Local legend has it that the treasure of Captain Kidd remains hidden beneath the mud and silt.

A further walk along the shore and we arrived at Sunset Beach and caught sight of the SS Atlantus. There was a scattering of other visitors, all of them eager treasure hunters probing and scooping up sand. Some involved their children, others worked in pairs, some stared intensely into the beach as if possessing x-ray vision. The lost treasure of Captain Kidd was still lost, after all.

The Sunset Beach gift shop was deserted, as were the streets of Cape May City when we eventually limped back into town. All that will change once summer breaths warmth back into the sea breeze. The broad, flat beaches and low-rise skyline guarantee sunshine from dawn til dusk. The marina offers festivals and whale-watching tours. The centre of town is the stuff of picture postcards; colourful Victorian houses and quaint local stores, trolley rides, clean streets, friendly locals. It may be the butt of jokes across the nation but Cape May proves there's real beauty to be found in New Jersey, and plenty more besides.