add docs

doc/README.md

@@ -0,0 +1,22 @@
+# TestArena Backend Documentation
+
+Welcome to the official documentation for the TestArena Backend. This system is designed to automate the build and test execution process for ESP32 projects using ESP-IDF and QEMU.
+
+## Table of Contents
+
+1.  **[Architecture & Design](architecture.md)**: Overview of the system components, technology stack, and design principles.
+2.  **[Usage Guide](usage.md)**: Instructions on how to deploy, configure, and use the system.
+3.  **[API Reference](api_reference.md)**: Detailed documentation of the available REST API endpoints.
+4.  **[Workflows & Flowcharts](flow_diagrams.md)**: Visual representations of the system's key processes.
+
+## Project Overview
+
+TestArena is a robust platform for managing and executing automated test scenarios. It provides a web-based dashboard for monitoring test queues, real-time logging of execution steps, and detailed HTML reports of test results.
+
+### Key Features
+
+*   **Automated Build & Test**: Automatically clones repositories, builds firmware, and executes tests in a QEMU environment.
+*   **Real-time Monitoring**: Live dashboard with search, sorting, and queue management (abort/delete).
+*   **Comprehensive Logging**: Timestamped and leveled logs for every step of the process.
+*   **Scalable Architecture**: Decoupled API and Worker services for better performance and reliability.
+*   **Easy Deployment**: Automated deployment script for quick setup on Ubuntu servers.

doc/api_reference.md

@@ -0,0 +1,111 @@
+# API Reference
+
+The TestArena Backend provides a RESTful API for managing test queues and monitoring system status.
+
+## Base URL
+`http://<server-ip>:8080/api`
+
+---
+
+## Endpoints
+
+### 1. Submit a New Queue
+`POST /queue`
+
+Submits a new set of tasks to the execution queue.
+
+**Request Body:**
+```json
+{
+    "source": "string",
+    "<queue_id>": [
+        "environment_name",
+        {
+            "<task_id>": "path/to/scenario.xml"
+        }
+    ]
+}
+```
+
+**Response:**
+```json
+{
+    "status": "Queue OK",
+    "queue_id": "string"
+}
+```
+
+---
+
+### 2. List All Queues
+`GET /queues`
+
+Returns a list of all queues in the system, ordered by creation date (newest first).
+
+**Response:**
+```json
+[
+    {
+        "id": "string",
+        "status": "Waiting|Running|Finished|Aborted",
+        "created_at": "ISO8601 Timestamp",
+        "environment": "string",
+        "source": "string"
+    }
+]
+```
+
+---
+
+### 3. Get Status
+`GET /status/{id}`
+
+Gets the status of a specific queue or task.
+
+**Response:**
+```json
+{
+    "id": "string",
+    "type": "queue|task",
+    "status": "string"
+}
+```
+
+---
+
+### 4. Abort Queue or Task
+`POST /abort/{id}`
+
+Aborts a waiting or running queue or a single task.
+
+**Response:**
+```json
+{
+    "id": "string",
+    "status": "Aborted"
+}
+```
+
+---
+
+### 5. Delete Queue
+`DELETE /delete/{id}`
+
+Permanently deletes a queue, its associated tasks, and all related files from the server.
+
+**Response:**
+```json
+{
+    "id": "string",
+    "status": "Deleted"
+}
+```
+
+---
+
+## Error Handling
+
+The API uses standard HTTP status codes:
+*   `200 OK`: Request successful.
+*   `404 Not Found`: The requested ID does not exist.
+*   `500 Internal Server Error`: An unexpected error occurred on the server.

doc/architecture.md

@@ -0,0 +1,64 @@
+# Architecture & Design
+
+This document describes the high-level architecture and design of the TestArena Backend.
+
+## System Architecture
+
+TestArena follows a decoupled architecture consisting of a RESTful API and a background worker.
+
+```mermaid
+graph TD
+    User((User)) -->|HTTP| API[FastAPI Web Server]
+    API -->|Read/Write| DB[(SQLite Database)]
+    API -->|Create| FS[Filesystem /testarena]
+    Worker[Background Worker] -->|Poll| DB
+    Worker -->|Execute| Scripts[Execution Scripts]
+    Scripts -->|Build/Test| QEMU[QEMU Emulator]
+    Scripts -->|Write Logs| FS
+    User -->|View| Dashboard[Web Dashboard]
+    Dashboard -->|API Calls| API
+```
+
+## Core Components
+
+### 1. FastAPI Web Server (`main.py`)
+The entry point for the system. It handles:
+*   Receiving test queue requests via REST API.
+*   Managing the SQLite database.
+*   Serving the web dashboard and static files.
+*   Providing endpoints for status monitoring, aborting, and deleting queues.
+
+### 2. Background Worker (`worker.py`)
+A dedicated process that continuously polls the database for "Waiting" queues. Its responsibilities include:
+*   Cloning and checking out the correct branch of the target repository.
+*   Orchestrating the build process using `idf.py build`.
+*   Running the QEMU simulation.
+*   Executing individual test scenarios.
+*   Managing real-time logging to the filesystem.
+
+### 3. Execution Scripts (`TPF/`)
+A set of specialized scripts for different stages of the workflow:
+*   `gitea_repo_controller.sh`: Handles Git operations (clone, checkout, pull).
+*   `scenario_execution.py`: Parses XML scenarios and runs test suites.
+*   `test_execution.sh`: Executes individual test commands and generates HTML logs.
+*   `scenario_exe_parser.py`: Helper script for parsing XML scenario files.
+
+## Technology Stack
+
+| Layer | Technology |
+| :--- | :--- |
+| **Backend Framework** | FastAPI (Python 3.12+) |
+| **Database** | SQLite with SQLAlchemy ORM |
+| **Frontend** | Vanilla HTML5, CSS3 (Modern Glassmorphism UI), JavaScript |
+| **Process Management** | Systemd (testarena-app, testarena-worker) |
+| **Web Server / Proxy** | Nginx |
+| **Build System** | ESP-IDF (Espressif IoT Development Framework) |
+| **Emulator** | QEMU (XTENSA) |
+| **Scripting** | Bash, Python |
+
+## Design Principles
+
+*   **Decoupling**: The API and Worker are separate processes, allowing the UI to remain responsive even during heavy test execution.
+*   **Real-time Visibility**: All subprocess output is streamed in real-time to both the console and log files.
+*   **Robustness**: Implemented global timeouts, automatic cleanup of orphaned processes, and detailed error handling.
+*   **Automation**: The entire deployment and execution flow is fully automated to minimize manual intervention.

doc/flow_diagrams.md

@@ -0,0 +1,75 @@
+# Workflows & Flowcharts
+
+This document provides visual representations of the key processes within the TestArena system.
+
+## 1. Test Queue Submission Flow
+
+This flowchart shows the process from when a user submits a queue until it is ready for the worker.
+
+```mermaid
+sequenceDiagram
+    participant User
+    participant API as FastAPI (main.py)
+    participant DB as SQLite
+    participant FS as Filesystem
+
+    User->>API: POST /api/queue (Payload)
+    API->>API: Extract Queue ID, Source, Tasks
+    API->>FS: Create /testarena/<queue_id> folder
+    API->>FS: Create queue_status.json
+    API->>DB: Insert Queue & Task records
+    API-->>User: 200 OK (Queue OK)
+```
+
+## 2. Worker Execution Workflow
+
+This diagram illustrates the lifecycle of a test queue as processed by the background worker.
+
+```mermaid
+flowchart TD
+    Start([Start Worker]) --> Poll{Poll DB for 'Waiting'}
+    Poll -- No --> Wait[Wait 5s] --> Poll
+    Poll -- Yes --> Running[Set Status to 'Running']
+    Running --> Clone[Clone/Checkout Repo]
+    Clone --> Build[Build Firmware: idf.py build]
+    Build --> QEMU[Start QEMU: idf.py qemu]
+    QEMU --> Loop[Loop through Tasks]
+    
+    subgraph Task Execution
+        Loop --> RunScript[Run scenario_execution.py]
+        RunScript --> Stream[Stream Output to Log]
+        Stream --> Parse[Parse Results]
+        Parse --> UpdateDB[Update Task Status & Result]
+    end
+    
+    UpdateDB --> Next{More Tasks?}
+    Next -- Yes --> Loop
+    Next -- No --> Finish[Set Queue to 'Finished']
+    Finish --> Wait
+```
+
+## 3. Real-time Logging Architecture
+
+How logs are captured and displayed to the user.
+
+```mermaid
+graph LR
+    Sub[Subprocess] -->|Stdout/Stderr| Pipe[Pipe]
+    Pipe -->|Read Line| Worker[worker.py]
+    Worker -->|Write| File[queue_log.txt]
+    Worker -->|Print| Console[Systemd Journal]
+    User -->|View| Dashboard[Web Dashboard]
+    Dashboard -->|Fetch| API[FastAPI]
+    API -->|Read| File
+```
+
+## 4. Delete Queue Workflow
+
+The process of cleaning up system data.
+
+```mermaid
+flowchart LR
+    Req[DELETE /api/delete/{id}] --> DB[Delete DB Records]
+    DB --> FS[Remove /testarena/{id} Directory]
+    FS --> Res[Return Success]
+```

doc/usage.md

@@ -0,0 +1,80 @@
+# Usage Guide
+
+This guide provides instructions on how to deploy, configure, and use the TestArena system.
+
+## Deployment
+
+The system is designed to be deployed on an Ubuntu server. An automated deployment script is provided.
+
+### Prerequisites
+*   Ubuntu 22.04 or later.
+*   ESP-IDF installed at `/home/asf/esp/esp-idf`.
+*   QEMU (XTENSA) installed and available in the system path.
+
+### Installation Steps
+1.  Clone the repository to `/home/asf/testarena_backend`.
+2.  Navigate to the directory: `cd /home/asf/testarena_backend`.
+3.  Run the deployment script with sudo:
+    ```bash
+    sudo ./deploy.sh
+    ```
+    This script will:
+    *   Install system dependencies (`nginx`, `sqlite3`, `perl`, etc.).
+    *   Set up a Python virtual environment and install requirements.
+    *   Configure Nginx as a reverse proxy.
+    *   Set up and enable systemd services for the API and Worker.
+
+## Managing Services
+
+Use standard `systemctl` commands to manage the TestArena services:
+
+```bash
+# Restart services
+sudo systemctl restart testarena-app testarena-worker
+
+# Check status
+sudo systemctl status testarena-app testarena-worker
+
+# View logs
+sudo journalctl -u testarena-worker -f
+```
+
+## Using the Dashboard
+
+Access the dashboard at `http://<server-ip>:8080/`.
+
+### Features:
+*   **Queue Monitor**: View all test queues, their status, and environment.
+*   **Search**: Filter queues by Queue ID using the search box.
+*   **Sorting**: Click on table headers (Queue ID, Environment, Status) to sort the data.
+*   **Actions**:
+    *   **Abort**: Stop a running or waiting queue.
+    *   **Delete**: Permanently remove a queue's data from the database and filesystem.
+*   **Live Logs**: View real-time system logs in the sidebar.
+
+## Submitting a Test Queue
+
+You can submit a new test queue by sending a POST request to `/api/queue`.
+
+### Example Payload:
+```json
+{
+    "source": "feature/new-sensor",
+    "QUEUE_12345": [
+        "Production_Env",
+        {
+            "TASK_001": "path/to/scenario_1.xml",
+            "TASK_002": "path/to/scenario_2.xml"
+        }
+    ]
+}
+```
+
+## Viewing Results
+
+Test results are stored in `/home/asf/testarena/<queue_id>/`.
+*   `queue_log.txt`: The full execution log for the entire queue.
+*   `<task_id>/execution_report.html`: A detailed HTML report for a specific task.
+*   `<task_id>/<case_id>-logging.html`: Individual logs for each test case.
+
+You can also browse results via the web interface at `http://<server-ip>:8080/results/`.