Manipulate inference parameters and perform comparative evaluation to assess the impact of selected values. Build a suite of systems using targeted tools that detect problems such as prompt injection (CWE-1427) and other problems. Consider statistically measuring token distribution to see if it is consistent with expected results.
CWE-1434 Base Borrador
Insecure Setting of Generative AI/ML Model Inference Parameters
This vulnerability occurs when a generative AI or ML model is deployed with inference parameters that are too permissive, causing it to frequently generate incorrect, nonsensical, or unpredictable…
Definición
What is CWE-1434?
This vulnerability occurs when a generative AI or ML model is deployed with inference parameters that are too permissive, causing it to frequently generate incorrect, nonsensical, or unpredictable outputs.
Generative AI models, like those for text or image creation, use settings such as temperature, Top P, and Top K to control their creativity and decision-making. Setting these values incorrectly—often too high—forces the model to make overly random guesses, leading to 'hallucinations,' incoherent content, or wildly unrealistic results. When these flawed outputs are used in decision-making processes, data pipelines, or user-facing features, they can corrupt data integrity and create serious reliability issues.
For developers, securing AI components means rigorously validating and constraining these inference parameters in production, similar to sanitizing user input. This requires testing across diverse inputs to find safe thresholds. Managing these configurations at scale across multiple models is challenging; an ASPM platform like Plexicus can help by automatically detecting insecure AI parameter settings and tracking these flaws alongside traditional vulnerabilities in your application stack.
Impacto en el mundo real
Real-world CVEs caused by CWE-1434
Todavía no hay CVEs públicos enlazados a esta CWE en el catálogo de MITRE.
Cómo lo explotan los atacantes
Ruta del atacante paso a paso
- 1
Assume the product offers an LLM-based AI coding assistant to help users to write code as part of an Integrated Development Environment (IDE). Assume the model has been trained on real-world code, and the model behaves normally under its default settings. Suppose there is a default temperature of 1, with a range of temperature values from 0 (most deterministic) to 2. Consider the following configuration.
- 2
The problem is that the configuration contains a temperature hyperparameter that is higher than the default. This significantly increases the likelihood that the LLM will suggest a package that did not exist at training time, a behavior sometimes referred to as "package hallucination." Note that other possible behaviors could arise from higher temperature, not just package hallucination. An adversary could anticipate which package names could be generated and create a malicious package. For example, it has been observed that the same LLM might hallucinate the same package regularly. Any code that is generated by the LLM, when run by the user, would download and execute the malicious package. This is similar to typosquatting. The risk could be reduced by lowering the temperature so that it reduces the unpredictable outputs and has a better chance of staying more in line with the training data. If the temperature is set too low, then some of the power of the model will be lost, and it may be less capable of producing solutions for rarely-encountered problems that are not reflected in the training data. However, if the temperature is not set low enough, the risk of hallucinating package names may still be too high. Unfortunately, the "best" temperature cannot be determined a priori, and sufficient empirical testing is needed.
- 3
In addition to more restrictive temperature settings, consider adding guardrails that test that independently verify any referenced package to ensure that it exists, is not obsolete, and comes from a trusted party. Note that reducing temperature does not entirely eliminate the risk of package hallucination. Even with very low temperatures or other settings, there is still a small chance that a non-existent package name will be generated.
Ejemplo de código vulnerable
Vulnerable JSON
Assume the product offers an LLM-based AI coding assistant to help users to write code as part of an Integrated Development Environment (IDE). Assume the model has been trained on real-world code, and the model behaves normally under its default settings. Suppose there is a default temperature of 1, with a range of temperature values from 0 (most deterministic) to 2. Consider the following configuration.
{
```
"model": "my-coding-model",
"context_window": 8192,
"max_output_tokens": 4096,
"temperature", 1.5,
...
} Ejemplo de código seguro
Secure JSON
The problem is that the configuration contains a temperature hyperparameter that is higher than the default. This significantly increases the likelihood that the LLM will suggest a package that did not exist at training time, a behavior sometimes referred to as "package hallucination." Note that other possible behaviors could arise from higher temperature, not just package hallucination. An adversary could anticipate which package names could be generated and create a malicious package. For example, it has been observed that the same LLM might hallucinate the same package regularly. Any code that is generated by the LLM, when run by the user, would download and execute the malicious package. This is similar to typosquatting. The risk could be reduced by lowering the temperature so that it reduces the unpredictable outputs and has a better chance of staying more in line with the training data. If the temperature is set too low, then some of the power of the model will be lost, and it may be less capable of producing solutions for rarely-encountered problems that are not reflected in the training data. However, if the temperature is not set low enough, the risk of hallucinating package names may still be too high. Unfortunately, the "best" temperature cannot be determined a priori, and sufficient empirical testing is needed.
{
```
...
"temperature", 0.2,
...
} What changed: the unsafe sink is replaced (or the input is validated/escaped) so the same payload no longer triggers the weakness.
Lista de prevención
How to prevent CWE-1434
- Implementation / System Configuration / Operation Develop and adhere to robust parameter tuning processes that include extensive testing and validation.
- Implementation / System Configuration / Operation Implement feedback mechanisms to continuously assess and adjust model performance.
- Documentation Provide comprehensive documentation and guidelines for parameter settings to ensure consistent and accurate model behavior.
Señales de detección
How to detect CWE-1434
Manipulate inference parameters and perform comparative evaluation to assess the impact of selected values. Build a suite of systems using targeted tools that detect problems such as prompt injection (CWE-1427) and other problems. Consider statistically measuring token distribution to see if it is consistent with expected results.
Plexicus detecta automáticamente CWE-1434 y abre un PR de corrección en menos de 60 segundos.
Codex Remedium escanea cada commit, identifica esta debilidad concreta y entrega un pull request listo para revisión con el parche. Sin tickets. Sin traspasos.
Preguntas frecuentes ¿Qué es CWE-1434?
¿Qué gravedad tiene CWE-1434?
¿Qué lenguajes o plataformas se ven afectados por CWE-1434?
¿Cómo puedo prevenir CWE-1434?
¿Cómo detecta y corrige Plexicus CWE-1434?
¿Dónde puedo aprender más sobre CWE-1434?
Frequently asked questions
¿Qué es CWE-1434?
This vulnerability occurs when a generative AI or ML model is deployed with inference parameters that are too permissive, causing it to frequently generate incorrect, nonsensical, or unpredictable outputs.
¿Qué gravedad tiene CWE-1434?
MITRE no ha publicado una calificación de probabilidad de explotación para esta debilidad. Trátala como de impacto medio hasta que tu modelo de amenazas demuestre lo contrario.
¿Qué lenguajes o plataformas se ven afectados por CWE-1434?
MITRE lists the following affected platforms: Not Architecture-Specific, AI/ML, Not Technology-Specific.
¿Cómo puedo prevenir CWE-1434?
Develop and adhere to robust parameter tuning processes that include extensive testing and validation. Implement feedback mechanisms to continuously assess and adjust model performance.
¿Cómo detecta y corrige Plexicus CWE-1434?
El motor SAST de Plexicus detecta la firma de flujo de datos para CWE-1434 en cada commit. Cuando hay coincidencia, nuestro agente Codex Remedium abre un PR de corrección con el código corregido, las pruebas y un resumen de una línea para el revisor.
¿Dónde puedo aprender más sobre CWE-1434?
MITRE publica la definición canónica en https://cwe.mitre.org/data/definitions/1434.html. También puedes consultar la documentación de OWASP y NIST para guías relacionadas.
Debilidades relacionadas
Weaknesses related to CWE-1434
CWE-440 Padre
Expected Behavior Violation
This weakness occurs when a software component, such as a function, API, or feature, fails to act as documented or intended. The system's…
CWE-684 Puede preceder
Incorrect Provision of Specified Functionality
This weakness occurs when software behaves differently than its documented specifications, which can mislead users and create security…
CWE-691 Par
Insufficient Control Flow Management
This vulnerability occurs when a program's execution flow isn't properly managed, allowing attackers to bypass critical checks, trigger…
Listo cuando tú lo estés
Deja de pagar por desarrollador.
Empieza a cerrar el bucle.
Plexicus es el ASPM nativo de IA que escanea, filtra, corrige, pentestea y explica — de forma autónoma. Desarrolladores ilimitados, repos ilimitados, acciones de IA de uso justo. Nivel gratuito real, €269/mo anual cuando estés listo.