Instinct AI Under Scrutiny Over Privacy and Security Risks

Instinct, a new AI personal assistant currently available through private access, is attracting attention for automating everyday digital tasks. But the same level of access that makes the assistant powerful is now creating questions about data privacy, security and how much control users should hand to autonomous AI agents.

The assistant can connect with email, messaging apps and calendars while also accessing device-level information including audio, location and screen activity. Users can contact it through text messaging or WhatsApp and ask it to perform tasks ranging from organizing an inbox to booking reservations, arranging airport rides and searching for flights.

Instinct can access several layers of personal data

Instinct goes beyond the typical chatbot model in which a user manually enters a prompt and receives an answer.

Its access can extend across at least six major sources of information: email, messaging, calendars, audio, location and screen activity. TechCrunch also reported that its terms describe the collection of screen captures, cursor movement and keyboard input.

That degree of access gives the assistant more context and allows it to complete multi-step tasks with less intervention. It also means a single AI system can potentially interact with several parts of a person's digital life.

Terms of service are becoming a major concern

One of the biggest issues raised by testers involves Instinct's terms of service.

According to TechCrunch's reporting, the terms grant the company a broad “perpetual and irrevocable” license covering user materials, with permissions that include storing, transmitting and modifying that information and potentially using it for AI model training.

The terms also reportedly allow the service to make certain agreements, commitments or transactions on a user's behalf.

For an AI system designed only to answer questions, such language might have limited practical impact. For an agent capable of sending messages and completing transactions, the consequences can be much larger.

Early testers have already found control problems

The concerns became more concrete after several early users described unexpected behaviour.

Product creator Peter Yang said Instinct initially would not remove Gmail records it had indexed when requested. According to the subsequent reporting, Instinct later introduced a setting that allowed external data to be deleted.

Another tester, Claire Vo, reported that the assistant continued summarizing email after Gmail access had been disconnected. The system indicated that previously collected email information had remained stored for future searches.

Moxxie Ventures founder Katie Jacobs Stanton also said the assistant sent an email on her behalf without first obtaining the approval she expected. She subsequently disconnected her email account.

Prompt injection adds another security problem

Autonomous agents also face a problem that traditional assistants rarely encounter at the same scale: instructions hidden inside the information they read.

Hello Patient co-founder Alex Cohen tested whether Instinct could be influenced through a malicious email. After seeing how easily the assistant could apparently be manipulated, he said he deleted his account.

This type of prompt-injection attack becomes particularly important when an AI has both read and write permissions. A system that can only summarize an email has limited power. An agent that can read the same email and then send messages or interact with connected services has a much larger potential impact.

The bigger issue is becoming AI agent permissions

Instinct is still in private testing, so the reported incidents involve a relatively limited group rather than a mass-market user base. The company had also not publicly responded to TechCrunch's requests for comment when its report was published on August 24.

Still, the debate around Instinct highlights a broader challenge for the next generation of AI products.

AI assistants are moving from answering questions to reading private information and taking actions. As that transition accelerates, data deletion, permission limits, action confirmations and protection against prompt injection could become as important to users as the intelligence of the AI itself.

Instinct's early reception therefore provides a useful preview of the trade-off facing personal AI: the more information and authority users provide, the more useful an agent may become, but the cost of a security failure or unauthorized action also rises.