How to Automate API Testing and CI/CD with AI

Quick Overview

The demonstration shows how to automate API testing within a CI/CD pipeline by creating a GitHub Actions workflow file (postman.yaml) that runs a Postman collection against the User Permissions API, ultimately leading to 100% test passing after initial fixes were implemented via AI assistance.

Key Points: The user initialized a Postman workspace by selecting 'From Git repository' and connecting to a local directory containing API resources. The Postman collection for the 'User Permissions API' initially had failing tests, particularly concerning the 'Delete Users' endpoint returning HTML instead of JSON for a 404 error. The presenter used AI assistance within Postman to write comprehensive tests for each request, leading to a collection run summary showing 70 tests passed out of 87 before fixes. After implementing suggested fixes (including adjusting expected return formats for string arrays and ensuring JSON error responses), a subsequent collection run achieved 100% test pass rate across all requests. The final step involved creating a GitHub Actions workflow file (postman.yaml) in the .github/workflows folder to automatically run the Postman collection tests on every pull request targeting the main branch. The new workflow automatically installs Postman CLI, runs the collection tests using a secret POSTMANAPIKEY, and uploads the results as artifacts, demonstrating full CI/CD integration.

Context: This video provides a tutorial demonstrating the integration of Postman API testing with a Continuous Integration/Continuous Deployment (CI/CD) pipeline using GitHub Actions. The demonstration focuses on the 'User Permissions API' collection, showing how Postman's AI features can help in writing robust tests, diagnosing failures, and finally automating the execution of these tests within a standard GitHub workflow triggered by pull requests.

Detailed Analysis

The demonstration begins by setting up a new Postman workspace directly from a Git repository, indicating a local-first development workflow. The presenter then navigates to the 'User Permissions API' collection within this workspace, which already contains several requests (GET, PUT, POST, DELETE). Upon running the collection tests initially, several failures are noted, particularly around response structure validation and error handling for endpoints like 'Delete Users' returning HTML instead of expected JSON for 404 errors. The presenter leverages Postman's AI capabilities to generate more comprehensive test coverage, specifically adding tests for every request. After running the collection again, the summary indicates significant improvement, with many tests passing, but critical issues remain, such as structure mismatches and endpoint errors. The recommended fixes involve updating expected string array formats and ensuring proper error responses. Once these issues are addressed (implicitly guided by the AI suggestions and subsequent test runs), the collection passes all 87 tests successfully. The final and most crucial step involves automating this testing process: the presenter creates a GitHub Actions workflow file named 'postman.yaml' in the '.github/workflows' directory. This YAML file is configured to run on 'push' to the 'main' branch or on 'workflowdispatch'. The workflow job utilizes the 'postman-cli' to run the Postman collection, using the secret stored in GitHub. This setup ensures that the API tests execute automatically upon code changes, confirming the integration of testing into the CI/CD pipeline, and the final successful run validates the entire process.