Adding docs

2026-07-11 10:41:54 +00:00 · 2025-04-16 22:54:53 -04:00
parent 41ceeea90a
commit 8eac7b90c6
6 changed files with 789 additions and 6 deletions
--- a/PROJECT.md
+++ b/PROJECT.md
@@ -0,0 +1,155 @@
 # Simple File Upload Server Project
 ## Project Overview
 The Simple File Upload Server is a Flask-based application designed for secure, temporary file sharing and mobile app log uploads. It follows a clean MVC architecture and implements modern web standards for both API and user interface interactions.
 ## Project Rules and Guidelines
 ### Code Organization
 1. **MVC Pattern**
   - Models: Place all data models in `src/models/`
   - Views: Store templates in `src/templates/`
   - Controllers: Keep business logic in `src/controllers/`
 2. **File Structure**
   - Source code belongs in `src/`
   - Configuration files in root directory
   - Temporary files in `tmp/uploads/`
   - Templates in `src/templates/`
 ### Development Rules
 1. **Code Style**
   - Follow PEP 8 guidelines for Python code
   - Use type hints where possible
   - Document all public functions and classes
   - Keep functions focused and single-purpose
 2. **Security Requirements**
   - All uploads must be authenticated with API key
   - File names must be sanitized
   - SSL required for production
   - Implement proper error handling
   - Clean up expired/abandoned files
 3. **API Guidelines**
   - Maintain backward compatibility
   - Document all endpoint changes
   - Include proper error responses
   - Use consistent response formats
   - Validate all input parameters
 4. **Feature Implementation**
   - Add tests for new features
   - Update documentation
   - Follow chunked upload protocol for large files
   - Maintain mobile app compatibility
   - Consider both WiFi and mobile network scenarios
 ### Testing Requirements
 1. **New Features**
   - Unit tests required
   - Test both success and failure cases
   - Include mobile network testing
   - Verify chunked upload functionality
 2. **Performance Testing**
   - Test with files >100MB
   - Verify timeout configurations
   - Test under various network conditions
   - Monitor memory usage
 ### Deployment Rules
 1. **Docker Requirements**
   - Use multi-stage builds
   - Include only necessary files
   - Set proper permissions
   - Configure appropriate timeouts
 2. **Production Setup**
   - SSL certificates required
   - Strong CLIENT_KEY required
   - Proper file cleanup configuration
   - Monitoring setup
 ## Documentation Structure
 ### Core Documentation
 1. **README.md**
   - Basic setup and configuration
   - API endpoint documentation
   - Docker deployment instructions
   - Security best practices
   [View README.md for details](README.md)
 2. **TASK.md**
   - Project roadmap
   - Feature implementation status
   - Future enhancements
   - Current objectives
   [View TASK.md for progress](TASK.md)
 3. **LARGE_UPLOAD.md**
   - Chunked upload implementation
   - Network handling strategies
   - Mobile optimization
   - Technical specifications
   [View LARGE_UPLOAD.md for implementation details](LARGE_UPLOAD.md)
 ## Version Control Rules
 1. **Branching Strategy**
   - Main branch: production-ready code
   - Develop branch: integration branch
   - Feature branches: feature/* prefix
   - Hotfix branches: hotfix/* prefix
 2. **Commit Guidelines**
   - Clear, descriptive commit messages
   - Reference issue numbers
   - Single responsibility per commit
   - Include relevant tests
 3. **Release Process**
   - Tag all releases (v*.*.*)
   - Update version.txt
   - Generate change logs
   - Test deployment process
 ## CI/CD Pipeline
 1. **Automated Processes**
   - Build testing
   - Unit test execution
   - Docker image building
   - Version tagging
   - DockerHub publishing
 2. **Quality Gates**
   - All tests must pass
   - Code coverage requirements
   - No security vulnerabilities
   - Proper version tagging
 ## Monitoring and Maintenance
 1. **Server Health**
   - Regular cleanup of expired files
   - Monitor disk usage
   - Track API usage
   - Log error rates
 2. **Performance Metrics**
   - Upload success rates
   - Download completion rates
   - Response times
   - Chunked upload efficiency
 ## Support and Updates
 1. **Issue Management**
   - Use GitHub issues for tracking
   - Label issues appropriately
   - Follow security disclosure process
   - Maintain changelog
 2. **Documentation Updates**
   - Keep API docs current
   - Update setup instructions
   - Document known issues
   - Maintain upgrade guides
--- a/TASK.md
+++ b/TASK.md
@@ -26,6 +26,11 @@
 - [x] Configurable limits
 - [x] API response format
 - [x] API key authentication
 - [x] Chunked upload support
  - [x] Upload session management
  - [x] Progress tracking
  - [x] Resume capability
  - [x] Chunk verification
 ### Download System
 - [x] Code validation
@@ -59,18 +64,21 @@
 ### Infrastructure
 - [ ] Add unit tests
 - [x] Set up CI/CD pipeline
  - [x] Automated Docker builds
  - [x] Version tagging
  - [x] DockerHub publishing
 - [x] Implement persistent storage
 - [ ] Add monitoring/metrics
 - [ ] Create backup system
 - [ ] Add health check endpoints
 ### Chunked Upload Support
- [ ] Implement new API endpoints for chunked uploads (/upload/chunk, /upload/start, /upload/complete)
+- [x] Implement new API endpoints for chunked uploads (/upload/chunk, /upload/start, /upload/complete)
- [ ] Add server-side chunk management and reassembly
+- [x] Add server-side chunk management and reassembly
- [ ] Configure Gunicorn with appropriate timeouts for large uploads
+- [x] Configure Gunicorn with appropriate timeouts for large uploads
- [ ] Implement temporary storage for in-progress chunked uploads
+- [x] Implement temporary storage for in-progress chunked uploads
- [ ] Add automatic cleanup for abandoned uploads
+- [x] Add automatic cleanup for abandoned uploads
- [ ] Create progress tracking for partially complete uploads
+- [x] Create progress tracking for partially complete uploads
 - [ ] Test with large files (>100MB) over various connection types
 ### UI/UX
--- a/docs/API.md
+++ b/docs/API.md
@@ -0,0 +1,188 @@
 # API Documentation
 ## Overview
 The Simple File Upload Server provides a RESTful API for file uploads and downloads, with special support for chunked uploads of large files. All upload endpoints require API key authentication.
 ## Authentication
 All upload endpoints require the `X-Client-Key` header with a valid API key.
 Example:
 ```http
 X-Client-Key: your-secret-key
 ```
 ## Endpoints
 ### File Upload
 #### POST /upload
 Upload a single file in one request.
 **Request:**
 - Method: `POST`
 - Content-Type: `multipart/form-data`
 - Authentication: Required
 **Parameters:**
 | Name | Type | Required | Description |
 |------|------|----------|-------------|
 | file | File | Yes | The file to upload |
 | max_downloads | Integer | No | Maximum number of downloads (default: 1) |
 | expiry_hours | Integer | No | Hours until file expires (default: 24) |
 **Response:**
 ```json
 {
    "code": "ABC123"
 }
 ```
 ### Chunked Upload
 #### POST /upload/start
 Initialize a chunked upload session.
 **Request:**
 - Method: `POST`
 - Content-Type: `multipart/form-data`
 - Authentication: Required
 **Parameters:**
 | Name | Type | Required | Description |
 |------|------|----------|-------------|
 | filename | String | Yes | Name of the file being uploaded |
 | total_chunks | Integer | Yes | Total number of chunks to expect |
 | max_downloads | Integer | No | Maximum number of downloads (default: 1) |
 | expiry_hours | Integer | No | Hours until file expires (default: 24) |
 **Response:**
 ```json
 {
    "upload_id": "abC123dEf456gHij"
 }
 ```
 #### POST /upload/chunk
 Upload a single chunk of a file.
 **Request:**
 - Method: `POST`
 - Content-Type: `multipart/form-data`
 - Authentication: Required
 **Parameters:**
 | Name | Type | Required | Description |
 |------|------|----------|-------------|
 | upload_id | String | Yes | Upload session ID from /upload/start |
 | chunk_index | Integer | Yes | Index of this chunk (0-based) |
 | chunk | File | Yes | The chunk data |
 **Response:**
 ```json
 {
    "success": true,
    "received_chunks": 5,
    "total_chunks": 10,
    "chunk_index": 4
 }
 ```
 #### POST /upload/verify
 Verify the status of uploaded chunks.
 **Request:**
 - Method: `POST`
 - Content-Type: `multipart/form-data`
 - Authentication: Required
 **Parameters:**
 | Name | Type | Required | Description |
 |------|------|----------|-------------|
 | upload_id | String | Yes | Upload session ID |
 **Response:**
 ```json
 {
    "total_chunks": 10,
    "received_chunks": 5,
    "missing_chunks": [5,6,7,8,9]
 }
 ```
 #### POST /upload/complete
 Complete a chunked upload session.
 **Request:**
 - Method: `POST`
 - Content-Type: `multipart/form-data`
 - Authentication: Required
 **Parameters:**
 | Name | Type | Required | Description |
 |------|------|----------|-------------|
 | upload_id | String | Yes | Upload session ID |
 **Response:**
 ```json
 {
    "code": "ABC123"
 }
 ```
 ### File Download
 #### GET /download/{code}
 Download a file using its 6-character code.
 **Request:**
 - Method: `GET`
 - Authentication: Not required
 **Parameters:**
 | Name | Type | Required | Description |
 |------|------|----------|-------------|
 | code | String | Yes | 6-character download code |
 **Response:**
 - Success: File download begins
 - Error: HTML error page with user-friendly message
 ### File Management
 #### DELETE /delete/{code}
 Delete a file using its code.
 **Request:**
 - Method: `DELETE`
 - Authentication: Not required
 **Parameters:**
 | Name | Type | Required | Description |
 |------|------|----------|-------------|
 | code | String | Yes | 6-character download code |
 **Response:**
 ```json
 {
    "message": "File deleted successfully"
 }
 ```
 ## Error Responses
 ### Common Error Format
 ```json
 {
    "error": "Error message description"
 }
 ```
 ### HTTP Status Codes
 - 200: Success
 - 400: Bad Request (missing parameters, invalid format)
 - 401: Unauthorized (invalid or missing API key)
 - 404: Not Found (invalid code, expired file)
 - 500: Server Error
 ## Rate Limits
 Currently not implemented. See future enhancements in TASK.md.
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -0,0 +1,206 @@
 # System Architecture
 ## Overview
 The Simple File Upload Server is built using a Model-View-Controller (MVC) architecture pattern in Flask, with clear separation of concerns and modular design. The system handles both traditional file uploads and chunked uploads for large files, with a focus on security and reliability.
 ## System Components
 ### 1. Core Architecture (MVC)
 #### Models (`src/models/`)
 - **FileMetadata**
  - Handles file metadata management
  - Tracks download counts, expiry times
  - Manages file paths and access codes
  - Implements cleanup for expired files
 #### Views (`src/templates/`)
 - **index.html**
  - Modern, responsive UI for file downloads
  - Client-side validation
  - Progress tracking
  - Error handling
 - **error.html**
  - User-friendly error pages
  - Contextual error messages
  - Clean, responsive design
 #### Controllers (`src/controllers/`)
 - **file_controller.py**
  - Upload/download logic
  - Chunked upload management
  - File cleanup operations
  - Access code generation
 ### 2. Data Flow
 ```mermaid
 graph TD
    A[Client] -->|Upload Request| B[Flask App]
    B -->|Validate| C[FileController]
    C -->|Store| D[FileMetadata]
    D -->|Save| E[Filesystem]
    A -->|Download Request| B
    B -->|Validate Code| C
    C -->|Check| D
    D -->|Retrieve| E
    E -->|Serve| A
 ```
 ### 3. File Storage System
 #### Directory Structure
 ```
 tmp/
 ├── uploads/
 │   ├── [regular files]
 │   └── chunks/
 │       └── [upload_session_folders]/
 ```
 #### Metadata Storage
 - `file_metadata.json`: Tracks regular file uploads
 - `chunk_metadata.json`: Manages chunked upload sessions
 ### 4. Security Architecture
 #### Authentication
 - API key authentication for uploads
 - 6-digit alphanumeric codes for downloads
 - SSL/TLS support for secure transmission
 #### File Security
 - Secure filename sanitization
 - Temporary storage with auto-cleanup
 - Download limit enforcement
 - File expiry system
 ### 5. Chunked Upload System
 #### Components
 1. **Session Management**
   - Unique upload session IDs
   - Chunk tracking and verification
   - Progress monitoring
 2. **File Assembly**
   ```mermaid
   graph LR
       A[Client] -->|Chunks| B[Temporary Storage]
       B -->|Assembly| C[Final File]
       C -->|Metadata| D[Database]
   ```
 3. **Cleanup Process**
   - Automatic cleanup of abandoned uploads
   - Temporary file management
   - Session expiry handling
 ### 6. Logging System (`src/utils/mLogger.py`)
 #### Logging Levels
 - INFO: Normal operations
 - WARNING: Potential issues
 - ERROR: Operation failures
 - DEBUG: Development information
 - VERBOSE: Detailed tracking
 #### Log Format
 ```
 timestamp|level|module::message
 ```
 ### 7. Error Handling
 #### Types of Errors
 1. **Client Errors**
   - Invalid file types
   - Missing parameters
   - Authentication failures
   - Invalid codes
 2. **Server Errors**
   - Storage issues
   - File assembly failures
   - Database errors
   - Network timeouts
 #### Error Response Format
 ```json
 {
    "error": "Error description",
    "details": "Additional information"
 }
 ```
 ### 8. Configuration System
 #### Environment Variables
 - `CLIENT_KEY`: API authentication
 - `UPLOAD_FOLDER`: File storage location
 - `SSL_CERT`: SSL certificate path
 - `SSL_KEY`: SSL key path
 #### Application Constants
 - Maximum file size
 - Chunk size limits
 - Session timeouts
 - Cleanup intervals
 ## Performance Considerations
 ### 1. Resource Management
 - Chunked upload for large files
 - Temporary storage cleanup
 - Memory-efficient file handling
 - Connection timeout management
 ### 2. Scalability
 - Stateless design
 - File system abstraction
 - Configurable worker processes
 - Independent upload sessions
 ### 3. Network Optimization
 - Adaptive chunk sizes
 - Progress tracking
 - Resume capability
 - Connection monitoring
 ## Deployment Architecture
 ### Docker Container
 ```mermaid
 graph TD
    A[Docker Container] -->|Runs| B[Gunicorn]
    B -->|WSGI| C[Flask App]
    C -->|Stores| D[Volume Mount]
    C -->|Logs| E[Log Volume]
 ```
 ### Process Management
 - Gunicorn worker processes
 - Configurable timeouts
 - SSL termination
 - Health monitoring
 ## Future Architecture Considerations
 ### Planned Improvements
 1. **Monitoring System**
   - API usage metrics
   - Storage utilization
   - Error rate tracking
   - Performance monitoring
 2. **Enhanced Security**
   - Rate limiting
   - IP whitelisting
   - File type restrictions
   - Access audit logging
 3. **High Availability**
   - Load balancing
   - Redundant storage
   - Session persistence
   - Backup management
--- a/docs/LARGE_UPLOAD.md
+++ b/docs/LARGE_UPLOAD.md
--- a/docs/SECURITY.md
+++ b/docs/SECURITY.md
@@ -0,0 +1,226 @@
 # Security Documentation
 ## Overview
 This document outlines the security measures implemented in the Simple File Upload Server and provides guidelines for secure deployment and operation.
 ## Security Features
 ### Authentication
 1. **API Key Authentication**
   - Required for all upload endpoints
   - Configured via CLIENT_KEY environment variable
   - Transmitted via X-Client-Key HTTP header
   - No default keys allowed in production
 2. **Download Security**
   - 6-digit alphanumeric codes
   - One-time use downloads (configurable)
   - Automatic expiration
   - No directory listing
 ### File Security
 1. **Upload Security**
   - Secure filename sanitization
   - No executable permissions
   - Temporary storage with cleanup
   - Chunked upload verification
 2. **Storage Security**
   - Files stored outside web root
   - Metadata separation from files
   - Automatic cleanup of expired files
   - Abandoned upload detection
 3. **Download Controls**
   - Maximum download limits
   - Expiry time enforcement
   - Force download headers
   - Content-Type verification
 ### Network Security
 1. **SSL/TLS**
   - Required for production
   - Modern cipher configuration
   - Perfect forward secrecy
   - SSL certificate validation
 2. **Headers**
   - Secure response headers
   - No version disclosure
   - XSS protection
   - Content security policy
 ## Secure Deployment
 ### Production Requirements
 1. **SSL Configuration**
 ```nginx
 ssl_protocols TLSv1.2 TLSv1.3;
 ssl_prefer_server_ciphers on;
 ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
 ```
 2. **Environment Setup**
 ```bash
 # Required settings
 export CLIENT_KEY="strong-random-key"
 export UPLOAD_FOLDER="/secure/path/uploads"
 # SSL settings
 export SSL_CERT="/path/to/cert.pem"
 export SSL_KEY="/path/to/key.pem"
 ```
 3. **File Permissions**
 ```bash
 # Storage directory
 chmod 750 /secure/path/uploads
 chown www-data:www-data /secure/path/uploads
 # SSL files
 chmod 600 /path/to/key.pem
 chmod 644 /path/to/cert.pem
 ```
 ### Docker Security
 1. **Container Configuration**
   - Non-root user
   - Read-only root filesystem
   - Limited capabilities
   - Resource limits
 2. **Volume Management**
   - Separate volumes for uploads
   - Proper permissions
   - Regular cleanup
   - Backup strategy
 ## Security Monitoring
 ### Logging
 1. **Access Logs**
   - Upload attempts
   - Download tracking
   - Error logging
   - Authentication failures
 2. **Audit Trail**
   - File operations
   - Configuration changes
   - System events
   - Cleanup activities
 ### Monitoring Requirements
 1. **System Monitoring**
   - Disk usage
   - Memory utilization
   - Network traffic
   - Error rates
 2. **Security Alerts**
   - Authentication failures
   - Unusual activity
   - Storage warnings
   - SSL certificate expiry
 ## Security Best Practices
 ### Server Configuration
 1. **Firewall Rules**
   - Allow only required ports
   - Rate limit connections
   - Block suspicious IPs
   - Monitor access patterns
 2. **Process Isolation**
   - Run as dedicated user
   - Minimal permissions
   - Resource limits
   - Process supervision
 ### Data Protection
 1. **File Handling**
   - Validate file types
   - Scan for malware
   - Enforce size limits
   - Secure deletion
 2. **Metadata Protection**
   - Regular backups
   - Access controls
   - Data validation
   - Secure storage
 ## Future Security Enhancements
 ### Planned Features
 1. **Rate Limiting**
   - Per-IP limits
   - Token bucket algorithm
   - Configurable thresholds
   - Automatic blocking
 2. **Advanced Authentication**
   - API key rotation
   - IP whitelisting
   - Two-factor auth option
   - Access tokens
 3. **Enhanced Monitoring**
   - Real-time alerts
   - Usage analytics
   - Threat detection
   - Performance tracking
 ### Security Roadmap
 1. **Short Term**
   - Implement rate limiting
   - Add file type restrictions
   - Enhance logging
   - Add health checks
 2. **Long Term**
   - Access audit system
   - Advanced monitoring
   - Automated backups
   - High availability
 ## Incident Response
 ### Response Plan
 1. **Detection**
   - Monitor logs
   - Track metrics
   - Alert on anomalies
   - User reports
 2. **Response**
   - Assess impact
   - Contain threat
   - Notify stakeholders
   - Document incident
 3. **Recovery**
   - Restore service
   - Update security
   - Review procedures
   - Implement fixes
 ### Security Contacts
 For security issues or vulnerabilities:
 1. Create a GitHub issue with [SECURITY] prefix
 2. For critical issues, contact maintainers directly
 3. Follow responsible disclosure guidelines
 4. Allow time for patches before public disclosure