I add the project if I want to have all the changes and the latest version of the library. Then I can update the library project when I update my whole solution.
If I want to be sure I'm using a specific version of the library, I'll just add the dll.
Edit: Except what Mitch said.
source
to share