iPhone:
Tap the "settings" tab at the bottom right corner of the Haystack app
Toggle 'Contact Syncing' to "off"
This will remove all Haystack cards' information from your phone's contacts.
Android:
Tap on the 'three lines' at the top > Settings
Toggle 'Contact Syncing' to "off"