AI Governance

Generative AI: Bias and Dangers in Code Execution

Kognitos

When AI Bias Meets Computation: The Dangers of Unchecked Generative AI in Code Execution

Key Takeaways

Unchecked generative AI is unsafe for enterprise automation because it injects bias and inconsistency into tasks that demand deterministic, repeatable execution. The post demonstrates this with two experiments: asked to compute a sell price, GPT-4 inconsistently treated a discount rate as an absolute number in one run and a percentage in another; asked to process a medical insurance claim, it wrongly assumed an in-network provider and miscalculated an already-met deductible. These subtle errors, harmless in a demo, become intolerable when scaled across end-to-end processes like HR raises or claims reimbursement. The core problem is that generative models excel at creativity and ideation but falter at strict, repetitive computation, and they are costly to run. The takeaway: enterprises need “computational intelligence” that understands natural language yet executes reliably like traditional software, the deterministic approach Kognitos is built on.

The Risks of Generative AI in Code Execution

To illustrate the risks associated with using generative AI by itself for automation, let’s consider an example. We attempted to use GPT-4 to perform a simple task: find the sell price of a product based on a list price and a partner’s discount rate. We utilized the following prompt:

For enterprise readers evaluating roadmap choices, themes such as Generative AI, unchecked Generative AI, ai bias surface repeatedly in architecture reviews. Those discussions are less about novelty and more about measurable throughput, exception transparency, and safe rollout. Related priorities often include Generative AI risks, ai code execution, especially where compliance and customer experience intersect.

The issue of AI bias in computation is more serious than many might realize. So far, the focus has primarily been on addressing racial and social biases in AI, but the risk of biased computation is just as significant. risk of biased computation in AI

Enterprises can still leverage the benefits of Generative AI, in a way that is precise and consistent.

calculate the sell price
get the partner’s discount rate
the result is the product price * the discount rate
Calculate the sell price
Run the above. When you don’t know something, ask.

On running the above prompt with two different inputs: “$100, 3” and “$100, 30“, GPT-4 provided the answers $300 and $130, respectively. In the first case, its bias lead it to assume the rate as an absolute number, while in the second case, it treated the rate as a percentage. This subtle discrepancy, present in just a few steps, could have far-reaching consequences when scaled to an enterprise end to end process.

Let’s take a look at a different example. We asked GPT to be a medical insurance claims processor. The user will provide the claim details and the patient’s insurance policy and it will have to approve a reimbursement amount based on the insurance policy. We then tested out the following GPT prompt:

Insurance Policy:

Insurance Company: ABC Health Insurance
Policy Holder: John Smith
Policy Number: 123456789

Coverage Details:

In-Network Provider: 80% coverage
Out-of-Network Provider: 50% coverage
Deductible: $500
Out-of-Pocket Maximum: $5,000

Past Claims:

1. Dr Alex Dunphy
Total Cost: $300
Reimbursed: $0
2. Dr Prabhat Kumar
Total Cost: $700
Reimbursed: $400

Current Claim:

Provider Name: Dr. Jane Doe
Total Cost: $200

Since the deductible has not been met ($300 from Dr. Alex Dunphy and $200 from Dr. Prabhat Kumar, totaling $500), the remaining deductible amount is $0.

As Dr. Jane Doe is assumed to be an in-network provider, the insurance will cover 80% of the cost after the deductible has been met. In this case, the total cost of the claim is $200.

Reimbursement Calculation:

Total Cost: $200

Insurance Coverage: 80%

Reimbursement Amount: $200 * 0.8 = $160

However, since the deductible has just been met, the reimbursement amount will be $0 for this claim. The patient will be responsible for the full $200 cost of this claim.

Please note that if Dr. Jane Doe is an out-of-network provider, the reimbursement calculation would be different.

One can clearly see that not only did GPT’s bias make it automatically assume Dr. Doe to be an in-network provider, but it also incorrectly reasoned that the deductible wasn’t fully paid, when in fact John had already paid the $500 deductible in his previous two visits.

As illustrated by the above two examples, it’s clear that GPT-4 can sometimes incorporate its own biases or opinions while processing user requests, leading to potential discrepancies in the output. This underlines the challenges that emerge when using generative AI by itself for automation or code execution, particularly when deterministic and unbiased execution is crucial for automating manual processes. Within an Enterprise, mistakes such as those noted above could have significant effects and are not tolerable for automation.

The inherent creativity of generative AI models is both their strength and weakness. While they excel at tasks requiring imagination and ideation, they may introduce biases when it comes to executing code in a predictable and repeatable manner.

Addressing AI Bias in Computation

For instance, imagine a scenario where an HR system calculates an employee’s raise based on GPT-4’s understanding of what is considered a “normal” raise. In this case, GPT-4 might unintentionally introduce biases stemming from its training data, leading to unfair or even discriminatory outcomes.

In addition to the risk of biased computation in AI, generative models like GPT-4 also perform poorly at repetitive tasks. The nature of these models is to generate diverse and creative outputs based on patterns identified in their training data. However, this can lead to inconsistencies when it comes to tasks that require strict adherence to a specific pattern or repetition. This could lead to incorrect or incomplete results, which can have serious consequences, especially in safety-critical applications. Also, Generative AI models are computationally expensive, especially when applied to repetitive tasks. The amount of resources required to process and generate results might not be proportional to the task’s simplicity, leading to inefficiencies in resource utilization.

Thus, there is a need for a system with “computational intelligence” – one that is able to understand human language and reliably execute instructions just like traditional software.

In a follow up blog we will dive into the details of “Computational Intelligence” and how Enterprises can still leverage the benefits of Generative AI, in a way that is precise and consistent.

Want to Unlock the Power of Generative AI for Your Business Today

How to Prevent the Risks of Unchecked Generative AI in Business

Identify the highest-risk generative AI use cases in your planned deployments. High-risk generative AI use cases are those where incorrect AI output has financial, compliance, legal, or safety consequences. Identify these use cases and apply more rigorous governance controls before deployment.
Implement output validation for generative AI used in consequential workflows. Generative AI output in consequential workflows must be validated before action is taken: confidence thresholds, format validation, consistency checks, and human review for outputs that fall below defined quality thresholds. Unchecked AI output in consequential workflows is the primary risk.
Configure rate limiting and anomaly detection for generative AI usage. Generative AI systems can be manipulated through adversarial inputs or prompt injection. Configure rate limiting, input validation, and anomaly detection to identify unusual usage patterns that may indicate manipulation.
Establish a human oversight process for all high-risk generative AI decisions. Define the human oversight process for every high-risk generative AI use case: who reviews AI outputs before action, how often outputs are audited, and what the escalation path is when problematic AI behavior is identified.
Publish and enforce an AI acceptable use policy. Unchecked generative AI use in organizations typically occurs because no policy governs it. Publish an AI acceptable use policy that defines permitted uses, data handling requirements, human oversight requirements, and the reporting process for AI-related concerns.

Frequently Asked Questions

Computational bias in generative AI refers to the tendency of models like GPT-4 to apply their own assumptions, interpretations, or opinions when executing tasks that require precise, deterministic outputs. Unlike racial or social bias which has received more public attention, computational bias affects the accuracy of numerical calculations and logical reasoning. This type of bias emerges because generative AI models are trained to produce diverse and creative outputs based on patterns in their training data, which conflicts with the need for strict, repeatable computation. In enterprise contexts, this bias can cause incorrect financial calculations, faulty insurance reimbursements, or inconsistent data processing results.

When generative AI is used for enterprise code execution, its inherent bias can cause inconsistent and incorrect outputs even for simple, well-defined tasks. For example, when asked to calculate a sell price using a partner discount rate, GPT-4 interpreted the same type of input differently depending on the numeric value provided, treating '3' as an absolute multiplier and '30' as a percentage. This inconsistency stems from the model's probabilistic nature, where it generates outputs based on learned patterns rather than strict logical rules. In an enterprise with thousands of automated transactions, such subtle errors can compound into significant financial discrepancies and compliance risks.

Unchecked generative AI poses serious risks to business process automation because it lacks the deterministic, repeatable execution that enterprise workflows require. Generative models are designed for creativity and ideation, but this same quality causes them to introduce variability and bias when precision is critical. In the insurance claims example from the article, GPT-4 incorrectly assumed a provider was in-network and miscalculated whether a deductible had been met, errors that could result in incorrect reimbursements and regulatory violations. The computational expense of running these models repeatedly for simple tasks also makes them resource-inefficient for high-volume enterprise processes.

Traditional software executes instructions in a deterministic, rule-based manner, producing the same output every time given the same input. Generative AI, by contrast, generates outputs probabilistically based on patterns learned from training data, which means it can produce different results for equivalent inputs depending on how it interprets the context. This fundamental difference makes generative AI unsuitable as a standalone automation engine for enterprise processes where consistency and auditability are required. What enterprises need is a system with 'computational intelligence' that understands natural language instructions while executing them with the precision and repeatability of traditional software.

A clear example is GPT-4 performing medical insurance claim processing: when given a patient's policy details and past claims history, the model incorrectly assumed that the treating physician was an in-network provider without being told so, and also miscalculated the deductible status. The patient had already paid $500 across two prior visits, fully satisfying the deductible, but GPT-4 reasoned it had not been met and denied the current reimbursement. This type of error, if deployed in a production insurance system, could result in incorrect benefit denials, patient harm, and significant legal liability. Similar bias-driven errors occur in financial calculations where the model inconsistently interprets numeric inputs as absolute values versus percentages.

Enterprises should evaluate whether an AI automation system provides deterministic, auditable execution rather than relying solely on generative AI inference for computational tasks. A safe enterprise AI platform should combine the natural language understanding of generative AI with a deterministic execution layer that enforces precise, repeatable logic. Organizations should test AI systems with edge cases and ambiguous inputs to surface bias before production deployment, and should require full exception transparency so that any deviation from expected behavior is flagged for human review. The goal is a system that can understand instructions expressed in human language while executing them with the consistency and reliability that enterprise compliance and accuracy requirements demand.