AVAudioMixerNode pan or AVAudioUnitSampler stereoPan properties do not work for left / right audio balance change for AVAudioEngine

I have the following code that plays a single midi note, but I want to adjust the balance / panning so that it only plays from the left speaker or right speaker, or perhaps some combination. I thought that changing "sampler.stereoPan" or perhaps "engine.mainMixerNode.pan" would do the trick, but it doesn't seem to have any effect. Any ideas what I am doing wrong?

engine = AVAudioEngine()
sampler = AVAudioUnitSampler()

sampler.stereoPan = -1.0 // doesn't work
//engine.mainMixerNode.pan = -1.0 // doesn't work

engine.attachNode(sampler)
engine.connect(sampler, to: engine.mainMixerNode, format: engine.mainMixerNode.outputFormatForBus(0))

var error: NSError?
engine.startAndReturnError(&error)

sampler.startNote(65, withVelocity: 64, onChannel: 1)