iPhone:
Tap the Settings tab at the bottom right corner of the Haystack app
Toggle Contact Syncing to off
This will remove all Haystack cards' information from your phone's contacts.
Android:
Tap the three lines (menu) icon at the top
Tap Settings
Toggle Contact Syncing to off