NOIV

Appearance

noiv test ​

Execute comprehensive API test suites with detailed validation and reporting.

Syntax ​

bash
noiv test <config> [OPTIONS]

Description ​

The test command executes comprehensive API test suites with advanced validation capabilities:

  • Multi-format support - YAML, JSON, and Postman collections
  • Advanced validation - JSONPath, regex, and custom assertions
  • Environment management - Multiple environment configurations
  • Detailed reporting - Rich output with success/failure analysis
  • CI/CD integration - Perfect for automated testing pipelines

Arguments ​

<config> (required) ​

Test configuration file to execute.

bash
noiv test api_tests.yaml
noiv test user_tests.json
noiv test postman_collection.json

Options ​

--environment, -e ​

Specify environment to use for testing.

bash
noiv test api_tests.yaml --environment production
noiv test api_tests.yaml -e staging

--variable, -v ​

Set or override variables for test execution.

bash
noiv test api_tests.yaml --variable base_url=https://api.example.com
noiv test api_tests.yaml -v user_id=12345 -v auth_token=abc123

--filter, -f ​

Run only tests matching the filter pattern.

bash
noiv test api_tests.yaml --filter "user.*"
noiv test api_tests.yaml -f "auth|login"

--exclude, -x ​

Exclude tests matching the pattern.

bash
noiv test api_tests.yaml --exclude "slow.*"
noiv test api_tests.yaml -x "integration.*"

--parallel, -p ​

Enable parallel test execution with specified concurrency.

bash
noiv test api_tests.yaml --parallel 4
noiv test api_tests.yaml -p 8

--timeout, -t ​

Set global timeout for all requests (default: 30 seconds).

bash
noiv test api_tests.yaml --timeout 60
noiv test api_tests.yaml -t 120

--output, -o ​

Specify output file for test results.

bash
noiv test api_tests.yaml --output results.json
noiv test api_tests.yaml -o test_report.xml

--format ​

Output format: table (default), json, xml, html, or junit.

bash
noiv test api_tests.yaml --format json
noiv test api_tests.yaml --format junit
noiv test api_tests.yaml --format html --output report.html

--verbose, -V ​

Enable verbose output with detailed request/response information.

bash
noiv test api_tests.yaml --verbose
noiv test api_tests.yaml -V

--fail-fast ​

Stop execution on first test failure.

bash
noiv test api_tests.yaml --fail-fast

--retry ​

Number of retry attempts for failed tests (default: 0).

bash
noiv test api_tests.yaml --retry 3

--delay ​

Delay between test executions in milliseconds (default: 0).

bash
noiv test api_tests.yaml --delay 1000

--verify-ssl ​

Enable/disable SSL certificate verification (default: true).

bash
noiv test api_tests.yaml --no-verify-ssl

--follow-redirects ​

Enable/disable following HTTP redirects (default: true).

bash
noiv test api_tests.yaml --no-follow-redirects

Examples ​

Basic Test Execution ​

bash
noiv test api_tests.yaml

Output:

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                                            Test Execution Results                                             ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

🎯 Test Suite: User Management API Tests
🌐 Environment: development
📊 Total Tests: 8 | ✅ Passed: 7 | ❌ Failed: 1 | ⏭️  Skipped: 0

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                                               Test Results                                                    ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

✅ Create User - Valid Data                    234ms    201 Created
✅ Create User - Duplicate Email               198ms    409 Conflict  
✅ Get User - Valid ID                         167ms    200 OK
✅ Get User - Invalid ID                       145ms    404 Not Found
✅ Update User - Valid Data                    289ms    200 OK
✅ Update User - Invalid ID                    156ms    404 Not Found
❌ Delete User - Valid ID                      FAILED   500 Internal Server Error
✅ List Users - With Pagination                223ms    200 OK

🕒 Total Execution Time: 1.42 seconds
📈 Success Rate: 87.5% (7/8 tests passed)

Environment-Specific Testing ​

bash
noiv test api_tests.yaml --environment production

Filtered Test Execution ​

bash
# Run only authentication tests
noiv test api_tests.yaml --filter "auth.*"

# Run user tests excluding slow ones
noiv test user_tests.yaml --exclude ".*slow.*"

Parallel Test Execution ​

bash
noiv test api_tests.yaml --parallel 4 --verbose

Output:

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                                           Parallel Test Execution                                            ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

🔄 Running 12 tests across 4 parallel workers...

Worker 1: ✅ Create User - Valid Data          234ms
Worker 2: ✅ Get User - Valid ID               167ms  
Worker 3: ✅ Update User - Valid Data          289ms
Worker 4: ✅ List Users - With Pagination      223ms

Worker 1: ✅ Create User - Duplicate Email     198ms
Worker 2: ✅ Get User - Invalid ID             145ms
Worker 3: ❌ Delete User - Valid ID            FAILED
Worker 4: ✅ Update User - Invalid ID          156ms

🕒 Parallel Execution Time: 0.58 seconds (2.4x speedup)

CI/CD Integration Testing ​

bash
noiv test regression_tests.yaml \
  --environment $CI_ENVIRONMENT \
  --format junit \
  --output test-results.xml \
  --fail-fast \
  --retry 2

Test Configuration ​

YAML Test Suite ​

yaml
name: User Management API Tests
description: Comprehensive tests for user CRUD operations
base_url: https://api.example.com
version: "1.0"

environments:
  development:
    base_url: http://localhost:3000
    variables:
      api_key: "dev-key-123"
      timeout: 30
  
  staging:
    base_url: https://staging-api.example.com
    variables:
      api_key: "staging-key-456"
      timeout: 60
      
  production:
    base_url: https://api.example.com
    variables:
      api_key: "{{PROD_API_KEY}}"
      timeout: 90

variables:
  user_id: ""
  auth_token: ""
  test_email: "test@example.com"

setup:
  - name: Authenticate
    request:
      method: POST
      path: /auth/login
      body:
        email: "admin@example.com"
        password: "admin123"
    expect:
      status: 200
      body:
        token: "{{auth_token}}"
    extract:
      auth_token: "$.token"

tests:
  - name: Create User - Valid Data
    description: Test user creation with valid data
    request:
      method: POST
      path: /users
      headers:
        Content-Type: application/json
        Authorization: "Bearer {{auth_token}}"
      body:
        name: "John Doe"
        email: "{{test_email}}"
        age: 30
    expect:
      status: 201
      headers:
        Content-Type: application/json
      body:
        id: "{{user_id}}"
        name: "John Doe"
        email: "{{test_email}}"
        created_at: "regex:^\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}"
      response_time_ms: 1000
    extract:
      user_id: "$.id"
    
  - name: Get User - Valid ID
    depends_on: "Create User - Valid Data"
    request:
      method: GET
      path: /users/{{user_id}}
      headers:
        Authorization: "Bearer {{auth_token}}"
    expect:
      status: 200
      body:
        id: "{{user_id}}"
        name: "John Doe"
        email: "{{test_email}}"
    assert:
      - jsonpath: "$.age"
        condition: "gte"
        value: 18
      - jsonpath: "$.email"
        condition: "matches"
        value: "^[\\w.-]+@[\\w.-]+\\.[A-Za-z]{2,}$"

teardown:
  - name: Cleanup Created User
    request:
      method: DELETE
      path: /users/{{user_id}}
      headers:
        Authorization: "Bearer {{auth_token}}"
    expect:
      status: 204

JSON Test Suite ​

json
{
  "name": "E-commerce API Tests",
  "base_url": "https://shop-api.example.com",
  "environments": {
    "test": {
      "base_url": "http://localhost:3000",
      "variables": {
        "api_key": "test-key"
      }
    }
  },
  "tests": [
    {
      "name": "Search Products",
      "request": {
        "method": "GET",
        "path": "/products/search",
        "query": {
          "q": "laptop",
          "category": "electronics",
          "limit": 20
        }
      },
      "expect": {
        "status": 200,
        "body": {
          "products": {"type": "array"},
          "total": {"type": "number"},
          "page": 1
        }
      },
      "assert": [
        {
          "jsonpath": "$.products.length",
          "condition": "lte",
          "value": 20
        },
        {
          "jsonpath": "$.total",
          "condition": "gte",
          "value": 0
        }
      ]
    }
  ]
}

Advanced Validation ​

JSONPath Assertions ​

yaml
tests:
  - name: Complex Response Validation
    request:
      method: GET
      path: /users
    expect:
      status: 200
    assert:
      # Array length validation
      - jsonpath: "$.users.length"
        condition: "gte"
        value: 1
      
      # Nested field validation
      - jsonpath: "$.users[0].profile.settings.notifications"
        condition: "equals"
        value: true
      
      # Field existence
      - jsonpath: "$.pagination.total_pages"
        condition: "exists"
      
      # Regex pattern matching
      - jsonpath: "$.users[*].email"
        condition: "matches"
        value: "^[\\w.-]+@[\\w.-]+\\.[A-Za-z]{2,}$"
      
      # Numeric range validation
      - jsonpath: "$.users[*].age"
        condition: "between"
        min: 18
        max: 120

Response Header Validation ​

yaml
tests:
  - name: API Security Headers
    request:
      method: GET
      path: /secure-endpoint
    expect:
      status: 200
      headers:
        X-Content-Type-Options: nosniff
        X-Frame-Options: DENY
        X-XSS-Protection: "1; mode=block"
        Strict-Transport-Security: "regex:max-age=\\d+"

Response Time Validation ​

yaml
tests:
  - name: Performance Requirements
    request:
      method: GET
      path: /fast-endpoint
    expect:
      status: 200
      response_time_ms: 500  # Must respond within 500ms
    
  - name: Slow Endpoint Tolerance
    request:
      method: POST
      path: /heavy-computation
    expect:
      status: 200
      response_time_ms: 5000  # Allow up to 5 seconds

Custom JavaScript Assertions ​

yaml
tests:
  - name: Complex Business Logic Validation
    request:
      method: GET
      path: /order/{{order_id}}
    expect:
      status: 200
    assert:
      - javascript: |
          const order = response.body;
          const totalFromItems = order.items.reduce((sum, item) => sum + (item.price * item.quantity), 0);
          const expectedTotal = totalFromItems + order.tax + order.shipping;
          assert(Math.abs(order.total - expectedTotal) < 0.01, 
                 `Order total ${order.total} doesn't match calculated total ${expectedTotal}`);
      
      - javascript: |
          const order = response.body;
          assert(order.status === 'pending' || order.status === 'confirmed' || order.status === 'shipped',
                 `Invalid order status: ${order.status}`);

Test Dependencies ​

Sequential Dependencies ​

yaml
tests:
  - name: Setup - Create User
    request:
      method: POST
      path: /users
      body:
        name: "Test User"
        email: "test@example.com"
    extract:
      user_id: "$.id"
    
  - name: Test - Update User Profile
    depends_on: "Setup - Create User"
    request:
      method: PUT
      path: /users/{{user_id}}/profile
      body:
        bio: "Updated bio"
    
  - name: Cleanup - Delete User
    depends_on: "Test - Update User Profile"
    request:
      method: DELETE
      path: /users/{{user_id}}

Conditional Dependencies ​

yaml
tests:
  - name: Create Premium User
    condition: "{{user_type}} == 'premium'"
    request:
      method: POST
      path: /users/premium
    
  - name: Test Premium Features
    depends_on: "Create Premium User"
    condition: "{{user_type}} == 'premium'"
    request:
      method: GET
      path: /premium/features

Output Formats ​

Table Format (Default) ​

Human-readable table format with colors and status indicators.

JSON Format ​

bash
noiv test api_tests.yaml --format json --output results.json
json
{
  "summary": {
    "total": 8,
    "passed": 7,
    "failed": 1,
    "skipped": 0,
    "success_rate": 87.5,
    "execution_time": 1.42
  },
  "tests": [
    {
      "name": "Create User - Valid Data",
      "status": "passed",
      "duration": 234,
      "request": {
        "method": "POST",
        "url": "https://api.example.com/users",
        "headers": {...},
        "body": {...}
      },
      "response": {
        "status": 201,
        "headers": {...},
        "body": {...},
        "time": 234
      }
    }
  ],
  "environment": "development",
  "timestamp": "2025-07-24T10:30:00Z"
}

JUnit XML Format ​

bash
noiv test api_tests.yaml --format junit --output test-results.xml
xml
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="API Tests" tests="8" failures="1" time="1.42">
  <testsuite name="User Management API Tests" tests="8" failures="1" time="1.42">
    <testcase name="Create User - Valid Data" time="0.234" classname="UserAPI"/>
    <testcase name="Delete User - Valid ID" time="0.156" classname="UserAPI">
      <failure message="Expected status 204 but got 500">
        Response: {"error": "Internal server error"}
      </failure>
    </testcase>
  </testsuite>
</testsuites>

HTML Report ​

bash
noiv test api_tests.yaml --format html --output report.html

Generates a comprehensive HTML report with:

  • Interactive test results
  • Request/response details
  • Performance metrics
  • Failure analysis
  • Environment information

Error Handling ​

Connection Errors ​

bash
noiv test api_tests.yaml

Output for Connection Issues:

❌ Create User - Valid Data                   FAILED   Connection Error
   Error: Connection timeout after 30 seconds
   URL: https://api.example.com/users
   
❌ Get User - Valid ID                        SKIPPED  Previous test failed
   Reason: Depends on "Create User - Valid Data"

Validation Failures ​

❌ Get User - Valid ID                        FAILED   Validation Error
   Expected: status 200
   Actual: status 404
   
   Expected: body.name = "John Doe"  
   Actual: body.name = "Jane Doe"
   
   Response Body:
   {
     "id": "12345",
     "name": "Jane Doe",
     "email": "test@example.com"
   }

Assertion Failures ​

❌ Complex Validation Test                    FAILED   Assertion Error
   JSONPath: $.users.length
   Condition: gte 1
   Actual: 0
   
   JavaScript Assertion Failed:
   Order total 99.99 doesn't match calculated total 89.99

Integration Examples ​

GitHub Actions ​

yaml
name: API Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      
      - name: Install NOIV
        run: pip install noiv
        
      - name: Run API Tests
        run: |
          noiv test api_tests.yaml \
            --environment staging \
            --format junit \
            --output test-results.xml \
            --variable api_key=${{ secrets.API_KEY }}
            
      - name: Publish Test Results
        uses: dorny/test-reporter@v1
        with:
          name: API Test Results
          path: test-results.xml
          reporter: java-junit

Docker Testing ​

dockerfile
FROM python:3.9-slim
RUN pip install noiv
COPY tests/ /tests/
WORKDIR /tests
CMD ["noiv", "test", "api_tests.yaml", "--environment", "docker"]

Jenkins Pipeline ​

groovy
pipeline {
    agent any
    stages {
        stage('API Tests') {
            steps {
                sh '''
                    noiv test api_tests.yaml \
                        --environment ${BUILD_ENV} \
                        --format junit \
                        --output test-results.xml \
                        --variable build_number=${BUILD_NUMBER}
                '''
            }
            post {
                always {
                    junit 'test-results.xml'
                }
            }
        }
    }
}

Best Practices ​

1. Organize Tests Logically ​

tests/
├── auth/
│   ├── login_tests.yaml
│   └── registration_tests.yaml
├── users/
│   ├── crud_tests.yaml
│   └── profile_tests.yaml
└── integration/
    └── end_to_end_tests.yaml

2. Use Environment Variables ​

yaml
environments:
  production:
    base_url: https://api.example.com
    variables:
      api_key: "{{API_KEY}}"        # From environment
      database_url: "{{DB_URL}}"    # From environment

3. Include Cleanup ​

yaml
teardown:
  - name: Delete Test Data
    request:
      method: DELETE
      path: /test-data/{{test_run_id}}

4. Use Descriptive Test Names ​

yaml
tests:
  - name: "User Registration - Valid Email Format"
  - name: "User Registration - Duplicate Email Rejection"
  - name: "User Registration - Missing Required Fields"

5. Validate Business Logic ​

yaml
assert:
  - jsonpath: "$.order.total"
    condition: "equals"
    value: "{{calculated_total}}"
  - javascript: |
      assert(response.body.inventory.stock >= 0, 
             "Stock cannot be negative");

See Also ​

Released under the MIT License.

Copyright © 2025 NOIV Team