Vector destructor problems for a vector of common ptrs

I am writing a sound manager to play sound with openal. Each object in the game is stored as a vector of shared pointers, each with a private audio object. When each object is defined individually and doesn't fit inside a vector, it works flawlessly. However, if you put it in a vector, it looks like they are not destroyed properly, causing crashes on exit. I also use a set (static member of the sound class) to keep track of the original IDs that open the return for loaded sounds.

loaded sounds/boundary.ogg. There are 1 sources. 
loaded sounds/player.ogg. There are 2 sources. 
// at this point, the application plays these sounds properly. 

// on app exit:
destroying sounds/boundary.ogg // deletes the source id for this, should be 1 sources left
there are 0 sources // set is now empty, maybe it got corrupted somehow?

      

        
        
        
      

    

The program crashes here. The debugger says that this line in xtree caused the crash:

while (!this->_Isnil(_Pnode))

      

        
        
        
      

    

If the objects are not in the vector, this works without issue. I have also tried to use a vector instead of a set to store the sources, but I am getting an error indicating that the vector is corrupted. Does anyone know what's going on?