The 300-metre long fence topped by two strands of barbed wire was erected by local farmer Lynda Best alongside the Bybrook and opposite three large houses.
Locals say the fence is designed to stop village children playing in the river in front of the houses as they did last summer.
But conservationists Jan Herlinger, Alison Bond and Julie Taylor claim it is there to "preserve, protect and enhance" the flora and fauna next to the Bybrook.
Asking Box Parish Council for support, they said they were disappointed their project has been misinterpreted.
They added: "It is disappointing that this is seen as a negative rather than a positive for conservation."
Dr Brian Mathew, the Wiltshire councillor for Box and Colerne, has now written to them, saying he is "not impressed" with their action.
He added: "A 300m-long fence topped with two strands of barbed wire erected along the Bybrook does not speak 'conservation', it appears to say 'exclusion' and a lot of people old and young are upset and angered by it.
"The two signs erected at each end of the fence proclaiming it to be a 'conservation area' carry no authority under Wiltshire Council Planning.
"No attempt seems to have been made to engage with the community and, quite frankly, I am appalled by this scar on a beautiful valley in an area of outstanding natural beauty.
"A bystander who was with me saw a swan as it came in to land on the river, have to take avoiding action to miss the top of the barbed wire fence in its final approach to land on the water.
"My advice is that you remove this fence as soon as possible. If you are not prepared to do this then for goodness sake take the barbed wire off it before wildlife, such as a swan or indeed a child, gets impaled on it.
"This is a truly beautiful place, where many of Box's residents have come to walk their dogs and exercise on a daily basis for generations and especially during the Covid-19 lockdown, and for the young such a chance to be with nature is just so important."
Mrs Best, of Ditteridge, Box, has been approached for a comment but has not yet responded.
Box parish councillors voted to take no further action, as the fence does not require planning permission.
But local resident Varian Tye, 67, said: "I feel quite passionate about this as there was no consultation.
"In view of its significant impact it would have therefore been good if the wider community knew about its erection and the reason why before it was erected.
"However, sadly it does not require planning permission so the wider community were not able to express their concerns prior to its erection."