Baracoa with its fishing villages around accessible by walkways suspended a foot of the water made of bamboo its crumbling Baseball Stadium on the beach make this an intriguing place. There are treks galore and the main river Toa River provides many tours.
Baracoa was founded by the Spanish in 1512 by Diego Velazquez but I can imagine these villages were here first somehow. The region is beautiful and the town charming. There are some great Casas to be stayed in too.
A great guide to Baracoa can be found online here.