Grate the peel from one orange. Cut the remaining oranges in half. Scoop out the pulp to yield 2 cups of drained fruit.
Heat the oven to 350°F. In a medium saucepan, heat the brandy, brown sugar, evaporated milk, margarine and salt. Add to the sweet potatoes and orange pulp and mix well.
Sprinkle sweet potato mixture with grated orange peel. Bake for 30 minutes. Garnish with nutmeg.