Turns out I have to install tensor flow from source because the binary version only works with cuDNN version 4. I installed cuDNN version 5, and I was going to roll back to v4, but then I found out that version 4 does not work with the GTX 1080 cards. That needs version 5. So now I am trying to uninstall TensorFlow and then reinstall it from source.