How Large Language Models Are Changing Software Testing

Large language models (LLMs) are changing software testing by helping teams draft test cases, generate test data, explain failures, and maintain automation scripts. This guide shows how to add them to an existing testing process while keeping results reviewable and grounded in the application’s requirements; you’ll need a test environment, representative requirements or code, and a human reviewer.

What You’ll Need

• A test repository or a small, non-production feature to use as a pilot.
• Requirements, acceptance criteria, API documentation, or source code that you are permitted to share with your chosen model.
• An LLM tool approved for your organization’s data and security rules. If using an external service, check its data-handling terms before submitting code or test data.
• Your existing test runner, such as a unit-test framework, browser automation tool, or API test suite.
• A developer or tester who can review generated tests, run them, and correct inaccurate assumptions.

Step-by-Step

1. Choose a bounded pilot. Select one feature with clear expected behavior, such as a form’s validation rules or an API endpoint’s documented responses. Avoid starting with an entire application: a narrow scope makes it easier to check whether generated tests are useful. Define a measurable goal, such as finding missing boundary cases or reducing the time spent drafting test outlines, rather than assuming the model will improve quality automatically.
2. Gather reliable context. Give the model the relevant acceptance criteria, input constraints, expected outputs, and important dependencies. Include examples of existing tests and the project’s conventions if available. Remove secrets, personal data, credentials, and unrelated proprietary material. If requirements conflict or leave behavior unspecified, resolve that with the product or engineering owner before asking for test cases.
3. Ask for a test plan before code. Request scenarios grouped by behavior: normal inputs, boundaries, invalid inputs, error handling, and relevant state changes. Ask the model to state assumptions and identify gaps in the requirements. For example, for a password-reset endpoint, specify the documented response for a known account, an unknown account, malformed input, and repeated requests. Do not let the model invent security or product rules; mark unresolved cases for clarification.
4. Check coverage against the requirements. Map each proposed scenario to a requirement or risk. Look for omissions such as empty values, minimum and maximum values, unsupported formats, timeouts, and interactions between fields where applicable. Remove duplicate cases and flag tests that depend on unspecified behavior. Coverage tools can help show which code ran, but execution coverage alone does not prove that assertions check the right outcomes.
5. Generate tests in the project’s style. Provide a small example of an existing test and ask for code using the same framework, naming, fixtures, and assertion patterns. Request one test file or a few cases at a time. Keep generated changes in a reviewable branch or patch; do not let a model directly alter production systems or merge code without normal approval.
6. Review each assertion for meaning. Confirm that the test checks the requirement rather than merely repeating the implementation. Verify expected values, setup, cleanup, and whether the test is deterministic. A test that passes only because it asserts that a function returns anything, or that duplicates a defect in the current code, provides little protection.
7. Run tests locally and inspect failures. Use the project’s normal commands and environment. First distinguish syntax, dependency, or setup problems from genuine behavior failures. Run each new test more than once when it involves timing, randomness, or external services. Replace unstable dependencies with the project’s established fakes or mocks where appropriate, and avoid treating a passing run as proof that the scenario is correct.
8. Use the model to explain, not decide, failures. Share a sanitized failure message, relevant test, and expected behavior. Ask for possible causes and ways to verify them. Check each suggestion against logs, source code, and reproducible results. The model may misread stack traces or suggest changing an assertion to make a test pass; correct the application or test according to the documented behavior, not the easiest explanation.
9. Integrate with review and continuous integration. Submit generated tests through the same code review, linting, and CI checks as hand-written tests. Record the requirement each test covers and note any assumptions. Keep test generation optional and inspectable: reviewers should be able to understand and maintain the code without needing the original conversation.
10. Measure the pilot and refine your prompts. Track practical outcomes such as review time, useful edge cases found, flaky tests, and maintenance effort. Compare them with the existing process. If the output is generic, provide more precise requirements and examples; if it is verbose, ask for fewer scenarios. Expand to another feature only when the pilot produces tests that the team can explain and maintain.

Common Mistakes

• Accepting plausible output as fact: LLMs can produce incorrect APIs, unsupported framework syntax, or behavior that was never specified. Compile and run everything, and verify claims against project documentation.
• Using vague prompts: “Test this feature” often leads to generic happy-path cases. Supply constraints, expected behavior, and examples, then ask for assumptions explicitly.
• Sharing sensitive inputs: Source code, logs, and test data can contain secrets or personal information. Follow organizational policy, sanitize material, and use an approved service.
• Generating tests that mirror the code: If the model sees only an implementation, it may reproduce the same mistaken assumptions. Anchor tests in requirements and independent examples where possible.
• Measuring success by test count: More tests can mean more duplicates or fragile checks. Focus on meaningful requirement coverage, defect detection, and maintainability.

Tips

• Ask for a table of scenario, input, expected outcome, and requirement before asking for executable code.
• Use small prompts with one clear task, and include the exact framework or language version only when you have verified it.
• Request boundary cases and negative cases, but have a domain expert confirm that each one reflects intended behavior.
• Keep prompts and generated patches with the review record when useful, but do not make the conversation a substitute for test documentation.
• Use deterministic fixtures and stable test data; keep live services and production data out of routine generated tests.

Troubleshooting

The model returns generic scenarios. Add concrete acceptance criteria, field constraints, and an existing test example. Ask it to identify missing information instead of filling gaps with assumptions.

The generated code does not compile. Confirm the language, test framework, imports, and project conventions. Provide a small compiling example and ask for a minimal correction; then run the project’s formatter and test command.

Tests fail inconsistently. Look for clocks, random values, shared state, network calls, or unordered results. Make the test deterministic using established fixtures, controlled time, or mocks, and repeat the run.

Tests pass but bugs remain. Review whether assertions capture user-visible requirements and whether important branches or boundary conditions are missing. Add cases based on a defect’s actual cause, then consider mutation testing or code review to assess whether assertions would catch changed behavior.

The model suggests weakening a failing test. Do not remove an assertion just to get a green run. Recheck the requirement, reproduce the behavior, and decide whether the implementation or test is wrong before changing either.

Frequently Asked Questions

Can an LLM replace software testers?
No. It can speed up drafting and analysis, but people still need to define intended behavior, assess risk, investigate results, and approve changes.

Can LLMs write automated tests?
Yes. They can draft unit, API, or UI tests when given relevant context and examples. Every generated test still needs review and execution in the real project.

Are LLM-generated tests reliable?
Not automatically. Models can misunderstand requirements or invent details, so verify each scenario and assertion against authoritative project behavior.

Is it safe to submit source code to an LLM?
It depends on the tool and your organization’s rules. Check data retention, access, and approval requirements; remove secrets and sensitive data before sharing.

How should a team measure whether LLM testing helps?
Compare useful coverage, defects found, review effort, flakiness, and ongoing maintenance with the existing process. Test count alone is not a quality measure.

What is a good first use case?
Start with a small feature whose requirements are clear, such as generating boundary-value scenarios for a validator or drafting tests for a documented API response.

Related

• Ollama Structured Fields in Python: A Practical Guide
• Error 1606: Fix Could Not Access Network Location
• ChatGPT Not Working in Telegram? 7 Fixes