Manipulate inference parameters and perform comparative evaluation to assess the impact of selected values. Build a suite of systems using targeted tools that detect problems such as prompt injection (CWE-1427) and other problems. Consider statistically measuring token distribution to see if it is consistent with expected results.
CWE-1434 Base Rascunho
Insecure Setting of Generative AI/ML Model Inference Parameters
This vulnerability occurs when a generative AI or ML model is deployed with inference parameters that are too permissive, causing it to frequently generate incorrect, nonsensical, or unpredictable…
Definição
What is CWE-1434?
This vulnerability occurs when a generative AI or ML model is deployed with inference parameters that are too permissive, causing it to frequently generate incorrect, nonsensical, or unpredictable outputs.
Generative AI models, like those for text or image creation, use settings such as temperature, Top P, and Top K to control their creativity and decision-making. Setting these values incorrectly—often too high—forces the model to make overly random guesses, leading to 'hallucinations,' incoherent content, or wildly unrealistic results. When these flawed outputs are used in decision-making processes, data pipelines, or user-facing features, they can corrupt data integrity and create serious reliability issues.
For developers, securing AI components means rigorously validating and constraining these inference parameters in production, similar to sanitizing user input. This requires testing across diverse inputs to find safe thresholds. Managing these configurations at scale across multiple models is challenging; an ASPM platform like Plexicus can help by automatically detecting insecure AI parameter settings and tracking these flaws alongside traditional vulnerabilities in your application stack.
Impacto no mundo real
Real-world CVEs caused by CWE-1434
Ainda não há referências CVE públicas associadas a este CWE no catálogo da MITRE.
Como os atacantes a exploram
Trajeto do atacante passo a passo
- 1
Assume the product offers an LLM-based AI coding assistant to help users to write code as part of an Integrated Development Environment (IDE). Assume the model has been trained on real-world code, and the model behaves normally under its default settings. Suppose there is a default temperature of 1, with a range of temperature values from 0 (most deterministic) to 2. Consider the following configuration.
- 2
The problem is that the configuration contains a temperature hyperparameter that is higher than the default. This significantly increases the likelihood that the LLM will suggest a package that did not exist at training time, a behavior sometimes referred to as "package hallucination." Note that other possible behaviors could arise from higher temperature, not just package hallucination. An adversary could anticipate which package names could be generated and create a malicious package. For example, it has been observed that the same LLM might hallucinate the same package regularly. Any code that is generated by the LLM, when run by the user, would download and execute the malicious package. This is similar to typosquatting. The risk could be reduced by lowering the temperature so that it reduces the unpredictable outputs and has a better chance of staying more in line with the training data. If the temperature is set too low, then some of the power of the model will be lost, and it may be less capable of producing solutions for rarely-encountered problems that are not reflected in the training data. However, if the temperature is not set low enough, the risk of hallucinating package names may still be too high. Unfortunately, the "best" temperature cannot be determined a priori, and sufficient empirical testing is needed.
- 3
In addition to more restrictive temperature settings, consider adding guardrails that test that independently verify any referenced package to ensure that it exists, is not obsolete, and comes from a trusted party. Note that reducing temperature does not entirely eliminate the risk of package hallucination. Even with very low temperatures or other settings, there is still a small chance that a non-existent package name will be generated.
Exemplo de código vulnerável
Vulnerable JSON
Assume the product offers an LLM-based AI coding assistant to help users to write code as part of an Integrated Development Environment (IDE). Assume the model has been trained on real-world code, and the model behaves normally under its default settings. Suppose there is a default temperature of 1, with a range of temperature values from 0 (most deterministic) to 2. Consider the following configuration.
{
```
"model": "my-coding-model",
"context_window": 8192,
"max_output_tokens": 4096,
"temperature", 1.5,
...
} Exemplo de código seguro
Secure JSON
The problem is that the configuration contains a temperature hyperparameter that is higher than the default. This significantly increases the likelihood that the LLM will suggest a package that did not exist at training time, a behavior sometimes referred to as "package hallucination." Note that other possible behaviors could arise from higher temperature, not just package hallucination. An adversary could anticipate which package names could be generated and create a malicious package. For example, it has been observed that the same LLM might hallucinate the same package regularly. Any code that is generated by the LLM, when run by the user, would download and execute the malicious package. This is similar to typosquatting. The risk could be reduced by lowering the temperature so that it reduces the unpredictable outputs and has a better chance of staying more in line with the training data. If the temperature is set too low, then some of the power of the model will be lost, and it may be less capable of producing solutions for rarely-encountered problems that are not reflected in the training data. However, if the temperature is not set low enough, the risk of hallucinating package names may still be too high. Unfortunately, the "best" temperature cannot be determined a priori, and sufficient empirical testing is needed.
{
```
...
"temperature", 0.2,
...
} What changed: the unsafe sink is replaced (or the input is validated/escaped) so the same payload no longer triggers the weakness.
Lista de verificação de prevenção
How to prevent CWE-1434
- Implementation / System Configuration / Operation Develop and adhere to robust parameter tuning processes that include extensive testing and validation.
- Implementation / System Configuration / Operation Implement feedback mechanisms to continuously assess and adjust model performance.
- Documentation Provide comprehensive documentation and guidelines for parameter settings to ensure consistent and accurate model behavior.
Sinais de deteção
How to detect CWE-1434
Manipulate inference parameters and perform comparative evaluation to assess the impact of selected values. Build a suite of systems using targeted tools that detect problems such as prompt injection (CWE-1427) and other problems. Consider statistically measuring token distribution to see if it is consistent with expected results.
O Plexicus deteta automaticamente o CWE-1434 e abre um PR de correção em menos de 60 segundos.
O Codex Remedium analisa cada commit, identifica esta fraqueza exata e entrega um pull request pronto para revisão com o patch. Sem tickets. Sem transferências.
Perguntas frequentes O que é o CWE-1434?
Qual a gravidade do CWE-1434?
Que linguagens ou plataformas são afetadas pelo CWE-1434?
Como posso prevenir o CWE-1434?
Como é que o Plexicus deteta e corrige o CWE-1434?
Onde posso saber mais sobre o CWE-1434?
Frequently asked questions
O que é o CWE-1434?
This vulnerability occurs when a generative AI or ML model is deployed with inference parameters that are too permissive, causing it to frequently generate incorrect, nonsensical, or unpredictable outputs.
Qual a gravidade do CWE-1434?
A MITRE não publicou uma classificação de probabilidade de exploração para esta fraqueza. Trate-a como impacto médio até o seu modelo de ameaças provar o contrário.
Que linguagens ou plataformas são afetadas pelo CWE-1434?
MITRE lists the following affected platforms: Not Architecture-Specific, AI/ML, Not Technology-Specific.
Como posso prevenir o CWE-1434?
Develop and adhere to robust parameter tuning processes that include extensive testing and validation. Implement feedback mechanisms to continuously assess and adjust model performance.
Como é que o Plexicus deteta e corrige o CWE-1434?
O motor SAST do Plexicus correlaciona a assinatura de fluxo de dados do CWE-1434 em cada commit. Quando é encontrada uma correspondência, o nosso agente Codex Remedium abre um PR de correção com o código corrigido, testes e um resumo de uma linha para o revisor.
Onde posso saber mais sobre o CWE-1434?
A MITRE publica a definição canónica em https://cwe.mitre.org/data/definitions/1434.html. Pode também consultar a documentação da OWASP e do NIST para orientações adjacentes.
Fraquezas relacionadas
Weaknesses related to CWE-1434
CWE-440 Pai
Expected Behavior Violation
This weakness occurs when a software component, such as a function, API, or feature, fails to act as documented or intended. The system's…
CWE-684 Pode preceder
Incorrect Provision of Specified Functionality
This weakness occurs when software behaves differently than its documented specifications, which can mislead users and create security…
CWE-691 Par
Insufficient Control Flow Management
This vulnerability occurs when a program's execution flow isn't properly managed, allowing attackers to bypass critical checks, trigger…
Pronto quando você estiver
Pare de pagar por desenvolvedor.
Comece a fechar o ciclo.
O Plexicus é o ASPM nativo de IA que verifica, filtra, corrige, pentesta e explica — de forma autónoma. Programadores ilimitados, repos ilimitados, ações de IA de utilização justa. Nível gratuito real, €269/mo anual quando estiver pronto.