When the portrait painter Oswald (later Sir Oswald) Birley and his wife Rhoda acquired Charleston Manor in 1931, the house was in disrepair. With the Sussex architect Walter Godfrey, the energetic Birleys restored it. They also turned the barn into a concert hall and Lady Birley created the gardens. The Birley's had a lot of artistic friends and a number of them got involved in the summer seasons of talks and music started by Lady Birley in about 1935. Each weekend over a period of four or five weeks there would be recitals, gardening talks, poetry readings – even on one occasion a country dance for artists of the Russian Ballet.

Lady Birley's entertainments lasted well into the 1970s despite the death of Sir Oswald in 1952 and her own increasing ill health. Lady Birley died in 1980, and there followed a fallow period until the mid 1980s when the Manor's new owners, the Headlams, set up a new summer season of concerts. It was under their auspices in July 1986 that a 26 year old cellist called Robert Cohen made his Charleston debut, ‘As I stepped onto the lawn,' he says, ‘and caught my first glimpse of the magnificent barn in such fabulous surroundings, the excitement of the festival captured me. What a wonderful setting for music!' Other artists that year included Dame Janet Baker.