AI Governance

Giving ChatGPT Direct Access to APIs Would Be A Security Disaster

Kognitos

Key Takeaways

Giving LLMs like ChatGPT direct access to APIs is a security disaster waiting to happen, this post warns. Because language models can be tricked into following an attacker’s instructions, adversaries can hide poisoned prompts in public webpages, emails, or any data the model reads, and a single exposure can be enough to hijack the model’s actions for that session, potentially stealing PII or PHI. The safer path is to keep people in the driver’s seat: rather than letting the LLM call systems directly, its plan should be presented as detailed English steps, reviewed by a human, and then executed by a separate non-AI interpreter that guarantees precise, correct, and safe actions. This human-in-the-loop model is how Kognitos customers automate business processes with generative AI and APIs safely and at scale. Explore the approach on the Kognitos platform.

APIs are essentially the set of protocols, routines, and tools used to build software applications. They help connect different forms of software and enable automation across applications.It is tempting to give ChatGPT and other LLMs direct access to these APIs. Doing so would be a security disaster waiting to happen. This is because LLMs can be easily tricked by an attacker to follow their instructions instead of the user’s. Attackers can use this to steal private information, takeover systems, or infect other automated LLMs.

They can place hidden poisoned prompts on public webpages, emails, or in any data that the LLM accesses. If the LLM looks at the poisoned data at all, that is often sufficient for the attacker to gain complete control of the LLM’s actions for that session. Within an enterprise this could wreak havoc, especially for enterprises who contain Personal Identifiable Information (PII) or Protected Health Information (PHI). But there is a better way that enterprises can use the power of Generative AI and LLMs to automate business processes and other activities without incurring major security risks.

Instead of giving ChatGPT and LLMs direct access to APIs, any time an LLM wants to call out to another system, its plan must be reviewed by a human first. The best way to do this would be to present the plan as detailed English steps, and then use a non-AI system to run the approved plan. This interpreter ensures that people remain in control, and can make certain that actions taken by AI are both precise, correct and safe for their business. This is what our customers at Kognitos use today to automate business processes using both LLMs and APIs in a safe, scalable way that empowers the business user.

In conclusion, while Language Models like ChatGPT have made significant strides in the field of natural language processing, we must not overlook the security risks associated with their access to APIs. It is imperative that we take necessary precautions and implement strict security measures to ensure that LLMs are not exploited by attackers. We must keep a watchful eye on this field and ensure that we prioritize security while advancing these technologies. Instead of giving direct access to APIs, platforms keeping people in the driver seat to approve the actions of LLMs is the best path forward for enterprises.

Frequently Asked Questions

Giving ChatGPT or other large language models direct access to APIs creates a serious security vulnerability because LLMs can be easily tricked by attackers into following malicious instructions. Attackers can embed hidden poisoned prompts in public webpages, emails, or any data the LLM accesses. If the LLM encounters poisoned data even once, an attacker can gain complete control of the LLM's actions for that session. In enterprise environments handling Personal Identifiable Information or Protected Health Information, this could lead to data theft, system takeover, or cascading infections across other automated LLMs.

Prompt injection attacks work by placing hidden instructions inside data that an LLM reads, such as a webpage, email, or document. When the LLM processes this data, it may interpret the hidden instructions as legitimate commands and execute them instead of following the user's original intent. The attacker's hidden prompt effectively hijacks the LLM's behavior for the duration of that session. Because LLMs cannot reliably distinguish between legitimate user instructions and maliciously injected content, these attacks are difficult to prevent at the model level alone.

The safer alternative is to require human review of any plan an LLM proposes before it is allowed to call external systems or APIs. The LLM generates a plan expressed as detailed English-language steps, which a human can review and approve. A separate non-AI interpreter then executes only the approved plan, ensuring that AI actions are precise, correct, and safe. This approach keeps people in control and prevents attackers from exploiting LLM vulnerabilities to take unauthorized actions.

Human-in-the-loop review is essential because LLMs are susceptible to manipulation, meaning their outputs cannot be trusted unconditionally in high-stakes enterprise workflows. By presenting the AI's intended actions as readable English steps before execution, businesses give authorized humans the opportunity to catch errors, detect tampering, or reject inappropriate actions. This oversight is especially critical in regulated industries where mistakes involving customer data could trigger GDPR, HIPAA, or SOC compliance violations. Maintaining human oversight also builds organizational confidence in AI automation by making the system's behavior transparent and auditable.

Kognitos provides a platform where LLMs generate automation plans in natural language, which business users review and approve before a non-AI interpreter executes the approved steps against real APIs and systems. This architecture separates the language understanding capability of LLMs from the execution layer, so a compromised or manipulated LLM cannot take unauthorized actions on its own. Enterprises using Kognitos can automate business processes in a scalable, safe manner that empowers non-technical business users while maintaining security controls. The approach combines the flexibility of generative AI with the determinism and auditability required for enterprise operations.

Enterprises should evaluate whether any proposed AI automation solution keeps humans in control of approving actions before they are executed against live systems. They should assess the attack surface created by giving LLMs direct API access and look for architectures that insert a non-AI execution layer between the LLM's plan and the actual API calls. Security considerations should include how the system handles sensitive data such as PII and PHI, and whether it meets compliance requirements like GDPR, HIPAA, SOC 2, and ISO 27001. Finally, enterprises should verify that the system provides clear, human-readable explanations of every automated action so that business users can confidently review and approve AI-generated workflows.

Kognitos

Giving ChatGPT Direct Access to APIs Would Be A Security Disaster

Key Takeaways

Frequently Asked Questions

Related Articles

Ready to automate?