Adding docs

PROJECT.md

@@ -0,0 +1,155 @@
+# Simple File Upload Server Project
+
+## Project Overview
+The Simple File Upload Server is a Flask-based application designed for secure, temporary file sharing and mobile app log uploads. It follows a clean MVC architecture and implements modern web standards for both API and user interface interactions.
+
+## Project Rules and Guidelines
+
+### Code Organization
+1. **MVC Pattern**
+   - Models: Place all data models in `src/models/`
+   - Views: Store templates in `src/templates/`
+   - Controllers: Keep business logic in `src/controllers/`
+
+2. **File Structure**
+   - Source code belongs in `src/`
+   - Configuration files in root directory
+   - Temporary files in `tmp/uploads/`
+   - Templates in `src/templates/`
+
+### Development Rules
+1. **Code Style**
+   - Follow PEP 8 guidelines for Python code
+   - Use type hints where possible
+   - Document all public functions and classes
+   - Keep functions focused and single-purpose
+
+2. **Security Requirements**
+   - All uploads must be authenticated with API key
+   - File names must be sanitized
+   - SSL required for production
+   - Implement proper error handling
+   - Clean up expired/abandoned files
+
+3. **API Guidelines**
+   - Maintain backward compatibility
+   - Document all endpoint changes
+   - Include proper error responses
+   - Use consistent response formats
+   - Validate all input parameters
+
+4. **Feature Implementation**
+   - Add tests for new features
+   - Update documentation
+   - Follow chunked upload protocol for large files
+   - Maintain mobile app compatibility
+   - Consider both WiFi and mobile network scenarios
+
+### Testing Requirements
+1. **New Features**
+   - Unit tests required
+   - Test both success and failure cases
+   - Include mobile network testing
+   - Verify chunked upload functionality
+
+2. **Performance Testing**
+   - Test with files >100MB
+   - Verify timeout configurations
+   - Test under various network conditions
+   - Monitor memory usage
+
+### Deployment Rules
+1. **Docker Requirements**
+   - Use multi-stage builds
+   - Include only necessary files
+   - Set proper permissions
+   - Configure appropriate timeouts
+
+2. **Production Setup**
+   - SSL certificates required
+   - Strong CLIENT_KEY required
+   - Proper file cleanup configuration
+   - Monitoring setup
+
+## Documentation Structure
+
+### Core Documentation
+1. **README.md**
+   - Basic setup and configuration
+   - API endpoint documentation
+   - Docker deployment instructions
+   - Security best practices
+   [View README.md for details](README.md)
+
+2. **TASK.md**
+   - Project roadmap
+   - Feature implementation status
+   - Future enhancements
+   - Current objectives
+   [View TASK.md for progress](TASK.md)
+
+3. **LARGE_UPLOAD.md**
+   - Chunked upload implementation
+   - Network handling strategies
+   - Mobile optimization
+   - Technical specifications
+   [View LARGE_UPLOAD.md for implementation details](LARGE_UPLOAD.md)
+
+## Version Control Rules
+1. **Branching Strategy**
+   - Main branch: production-ready code
+   - Develop branch: integration branch
+   - Feature branches: feature/* prefix
+   - Hotfix branches: hotfix/* prefix
+
+2. **Commit Guidelines**
+   - Clear, descriptive commit messages
+   - Reference issue numbers
+   - Single responsibility per commit
+   - Include relevant tests
+
+3. **Release Process**
+   - Tag all releases (v*.*.*)
+   - Update version.txt
+   - Generate change logs
+   - Test deployment process
+
+## CI/CD Pipeline
+1. **Automated Processes**
+   - Build testing
+   - Unit test execution
+   - Docker image building
+   - Version tagging
+   - DockerHub publishing
+
+2. **Quality Gates**
+   - All tests must pass
+   - Code coverage requirements
+   - No security vulnerabilities
+   - Proper version tagging
+
+## Monitoring and Maintenance
+1. **Server Health**
+   - Regular cleanup of expired files
+   - Monitor disk usage
+   - Track API usage
+   - Log error rates
+
+2. **Performance Metrics**
+   - Upload success rates
+   - Download completion rates
+   - Response times
+   - Chunked upload efficiency
+
+## Support and Updates
+1. **Issue Management**
+   - Use GitHub issues for tracking
+   - Label issues appropriately
+   - Follow security disclosure process
+   - Maintain changelog
+
+2. **Documentation Updates**
+   - Keep API docs current
+   - Update setup instructions
+   - Document known issues
+   - Maintain upgrade guides

TASK.md

@@ -26,6 +26,11 @@
 - [x] Configurable limits
 - [x] API response format
 - [x] API key authentication
+- [x] Chunked upload support
+  - [x] Upload session management
+  - [x] Progress tracking
+  - [x] Resume capability
+  - [x] Chunk verification
 
 ### Download System
 - [x] Code validation
@@ -59,18 +64,21 @@
 ### Infrastructure
 - [ ] Add unit tests
 - [x] Set up CI/CD pipeline
+  - [x] Automated Docker builds
+  - [x] Version tagging
+  - [x] DockerHub publishing
 - [x] Implement persistent storage
 - [ ] Add monitoring/metrics
 - [ ] Create backup system
 - [ ] Add health check endpoints
 
 ### Chunked Upload Support
-- [ ] Implement new API endpoints for chunked uploads (/upload/chunk, /upload/start, /upload/complete)
+- [x] Implement new API endpoints for chunked uploads (/upload/chunk, /upload/start, /upload/complete)
-- [ ] Add server-side chunk management and reassembly
+- [x] Add server-side chunk management and reassembly
-- [ ] Configure Gunicorn with appropriate timeouts for large uploads
+- [x] Configure Gunicorn with appropriate timeouts for large uploads
-- [ ] Implement temporary storage for in-progress chunked uploads
+- [x] Implement temporary storage for in-progress chunked uploads
-- [ ] Add automatic cleanup for abandoned uploads
+- [x] Add automatic cleanup for abandoned uploads
-- [ ] Create progress tracking for partially complete uploads
+- [x] Create progress tracking for partially complete uploads
 - [ ] Test with large files (>100MB) over various connection types
 
 ### UI/UX

docs/API.md

@@ -0,0 +1,188 @@
+# API Documentation
+
+## Overview
+The Simple File Upload Server provides a RESTful API for file uploads and downloads, with special support for chunked uploads of large files. All upload endpoints require API key authentication.
+
+## Authentication
+All upload endpoints require the `X-Client-Key` header with a valid API key.
+
+Example:
+```http
+X-Client-Key: your-secret-key
+```
+
+## Endpoints
+
+### File Upload
+
+#### POST /upload
+Upload a single file in one request.
+
+**Request:**
+- Method: `POST`
+- Content-Type: `multipart/form-data`
+- Authentication: Required
+
+**Parameters:**
+| Name | Type | Required | Description |
+|------|------|----------|-------------|
+| file | File | Yes | The file to upload |
+| max_downloads | Integer | No | Maximum number of downloads (default: 1) |
+| expiry_hours | Integer | No | Hours until file expires (default: 24) |
+
+**Response:**
+```json
+{
+    "code": "ABC123"
+}
+```
+
+### Chunked Upload
+
+#### POST /upload/start
+Initialize a chunked upload session.
+
+**Request:**
+- Method: `POST`
+- Content-Type: `multipart/form-data`
+- Authentication: Required
+
+**Parameters:**
+| Name | Type | Required | Description |
+|------|------|----------|-------------|
+| filename | String | Yes | Name of the file being uploaded |
+| total_chunks | Integer | Yes | Total number of chunks to expect |
+| max_downloads | Integer | No | Maximum number of downloads (default: 1) |
+| expiry_hours | Integer | No | Hours until file expires (default: 24) |
+
+**Response:**
+```json
+{
+    "upload_id": "abC123dEf456gHij"
+}
+```
+
+#### POST /upload/chunk
+Upload a single chunk of a file.
+
+**Request:**
+- Method: `POST`
+- Content-Type: `multipart/form-data`
+- Authentication: Required
+
+**Parameters:**
+| Name | Type | Required | Description |
+|------|------|----------|-------------|
+| upload_id | String | Yes | Upload session ID from /upload/start |
+| chunk_index | Integer | Yes | Index of this chunk (0-based) |
+| chunk | File | Yes | The chunk data |
+
+**Response:**
+```json
+{
+    "success": true,
+    "received_chunks": 5,
+    "total_chunks": 10,
+    "chunk_index": 4
+}
+```
+
+#### POST /upload/verify
+Verify the status of uploaded chunks.
+
+**Request:**
+- Method: `POST`
+- Content-Type: `multipart/form-data`
+- Authentication: Required
+
+**Parameters:**
+| Name | Type | Required | Description |
+|------|------|----------|-------------|
+| upload_id | String | Yes | Upload session ID |
+
+**Response:**
+```json
+{
+    "total_chunks": 10,
+    "received_chunks": 5,
+    "missing_chunks": [5,6,7,8,9]
+}
+```
+
+#### POST /upload/complete
+Complete a chunked upload session.
+
+**Request:**
+- Method: `POST`
+- Content-Type: `multipart/form-data`
+- Authentication: Required
+
+**Parameters:**
+| Name | Type | Required | Description |
+|------|------|----------|-------------|
+| upload_id | String | Yes | Upload session ID |
+
+**Response:**
+```json
+{
+    "code": "ABC123"
+}
+```
+
+### File Download
+
+#### GET /download/{code}
+Download a file using its 6-character code.
+
+**Request:**
+- Method: `GET`
+- Authentication: Not required
+
+**Parameters:**
+| Name | Type | Required | Description |
+|------|------|----------|-------------|
+| code | String | Yes | 6-character download code |
+
+**Response:**
+- Success: File download begins
+- Error: HTML error page with user-friendly message
+
+### File Management
+
+#### DELETE /delete/{code}
+Delete a file using its code.
+
+**Request:**
+- Method: `DELETE`
+- Authentication: Not required
+
+**Parameters:**
+| Name | Type | Required | Description |
+|------|------|----------|-------------|
+| code | String | Yes | 6-character download code |
+
+**Response:**
+```json
+{
+    "message": "File deleted successfully"
+}
+```
+
+## Error Responses
+
+### Common Error Format
+```json
+{
+    "error": "Error message description"
+}
+```
+
+### HTTP Status Codes
+- 200: Success
+- 400: Bad Request (missing parameters, invalid format)
+- 401: Unauthorized (invalid or missing API key)
+- 404: Not Found (invalid code, expired file)
+- 500: Server Error
+
+## Rate Limits
+Currently not implemented. See future enhancements in TASK.md.

docs/ARCHITECTURE.md

@@ -0,0 +1,206 @@
+# System Architecture
+
+## Overview
+The Simple File Upload Server is built using a Model-View-Controller (MVC) architecture pattern in Flask, with clear separation of concerns and modular design. The system handles both traditional file uploads and chunked uploads for large files, with a focus on security and reliability.
+
+## System Components
+
+### 1. Core Architecture (MVC)
+
+#### Models (`src/models/`)
+- **FileMetadata**
+  - Handles file metadata management
+  - Tracks download counts, expiry times
+  - Manages file paths and access codes
+  - Implements cleanup for expired files
+
+#### Views (`src/templates/`)
+- **index.html**
+  - Modern, responsive UI for file downloads
+  - Client-side validation
+  - Progress tracking
+  - Error handling
+- **error.html**
+  - User-friendly error pages
+  - Contextual error messages
+  - Clean, responsive design
+
+#### Controllers (`src/controllers/`)
+- **file_controller.py**
+  - Upload/download logic
+  - Chunked upload management
+  - File cleanup operations
+  - Access code generation
+
+### 2. Data Flow
+
+```mermaid
+graph TD
+    A[Client] -->|Upload Request| B[Flask App]
+    B -->|Validate| C[FileController]
+    C -->|Store| D[FileMetadata]
+    D -->|Save| E[Filesystem]
+    A -->|Download Request| B
+    B -->|Validate Code| C
+    C -->|Check| D
+    D -->|Retrieve| E
+    E -->|Serve| A
+```
+
+### 3. File Storage System
+
+#### Directory Structure
+```
+tmp/
+├── uploads/
+│   ├── [regular files]
+│   └── chunks/
+│       └── [upload_session_folders]/
+```
+
+#### Metadata Storage
+- `file_metadata.json`: Tracks regular file uploads
+- `chunk_metadata.json`: Manages chunked upload sessions
+
+### 4. Security Architecture
+
+#### Authentication
+- API key authentication for uploads
+- 6-digit alphanumeric codes for downloads
+- SSL/TLS support for secure transmission
+
+#### File Security
+- Secure filename sanitization
+- Temporary storage with auto-cleanup
+- Download limit enforcement
+- File expiry system
+
+### 5. Chunked Upload System
+
+#### Components
+1. **Session Management**
+   - Unique upload session IDs
+   - Chunk tracking and verification
+   - Progress monitoring
+
+2. **File Assembly**
+   ```mermaid
+   graph LR
+       A[Client] -->|Chunks| B[Temporary Storage]
+       B -->|Assembly| C[Final File]
+       C -->|Metadata| D[Database]
+   ```
+
+3. **Cleanup Process**
+   - Automatic cleanup of abandoned uploads
+   - Temporary file management
+   - Session expiry handling
+
+### 6. Logging System (`src/utils/mLogger.py`)
+
+#### Logging Levels
+- INFO: Normal operations
+- WARNING: Potential issues
+- ERROR: Operation failures
+- DEBUG: Development information
+- VERBOSE: Detailed tracking
+
+#### Log Format
+```
+timestamp|level|module::message
+```
+
+### 7. Error Handling
+
+#### Types of Errors
+1. **Client Errors**
+   - Invalid file types
+   - Missing parameters
+   - Authentication failures
+   - Invalid codes
+
+2. **Server Errors**
+   - Storage issues
+   - File assembly failures
+   - Database errors
+   - Network timeouts
+
+#### Error Response Format
+```json
+{
+    "error": "Error description",
+    "details": "Additional information"
+}
+```
+
+### 8. Configuration System
+
+#### Environment Variables
+- `CLIENT_KEY`: API authentication
+- `UPLOAD_FOLDER`: File storage location
+- `SSL_CERT`: SSL certificate path
+- `SSL_KEY`: SSL key path
+
+#### Application Constants
+- Maximum file size
+- Chunk size limits
+- Session timeouts
+- Cleanup intervals
+
+## Performance Considerations
+
+### 1. Resource Management
+- Chunked upload for large files
+- Temporary storage cleanup
+- Memory-efficient file handling
+- Connection timeout management
+
+### 2. Scalability
+- Stateless design
+- File system abstraction
+- Configurable worker processes
+- Independent upload sessions
+
+### 3. Network Optimization
+- Adaptive chunk sizes
+- Progress tracking
+- Resume capability
+- Connection monitoring
+
+## Deployment Architecture
+
+### Docker Container
+```mermaid
+graph TD
+    A[Docker Container] -->|Runs| B[Gunicorn]
+    B -->|WSGI| C[Flask App]
+    C -->|Stores| D[Volume Mount]
+    C -->|Logs| E[Log Volume]
+```
+
+### Process Management
+- Gunicorn worker processes
+- Configurable timeouts
+- SSL termination
+- Health monitoring
+
+## Future Architecture Considerations
+
+### Planned Improvements
+1. **Monitoring System**
+   - API usage metrics
+   - Storage utilization
+   - Error rate tracking
+   - Performance monitoring
+
+2. **Enhanced Security**
+   - Rate limiting
+   - IP whitelisting
+   - File type restrictions
+   - Access audit logging
+
+3. **High Availability**
+   - Load balancing
+   - Redundant storage
+   - Session persistence
+   - Backup management

docs/SECURITY.md

@@ -0,0 +1,226 @@
+# Security Documentation
+
+## Overview
+This document outlines the security measures implemented in the Simple File Upload Server and provides guidelines for secure deployment and operation.
+
+## Security Features
+
+### Authentication
+1. **API Key Authentication**
+   - Required for all upload endpoints
+   - Configured via CLIENT_KEY environment variable
+   - Transmitted via X-Client-Key HTTP header
+   - No default keys allowed in production
+
+2. **Download Security**
+   - 6-digit alphanumeric codes
+   - One-time use downloads (configurable)
+   - Automatic expiration
+   - No directory listing
+
+### File Security
+
+1. **Upload Security**
+   - Secure filename sanitization
+   - No executable permissions
+   - Temporary storage with cleanup
+   - Chunked upload verification
+
+2. **Storage Security**
+   - Files stored outside web root
+   - Metadata separation from files
+   - Automatic cleanup of expired files
+   - Abandoned upload detection
+
+3. **Download Controls**
+   - Maximum download limits
+   - Expiry time enforcement
+   - Force download headers
+   - Content-Type verification
+
+### Network Security
+
+1. **SSL/TLS**
+   - Required for production
+   - Modern cipher configuration
+   - Perfect forward secrecy
+   - SSL certificate validation
+
+2. **Headers**
+   - Secure response headers
+   - No version disclosure
+   - XSS protection
+   - Content security policy
+
+## Secure Deployment
+
+### Production Requirements
+
+1. **SSL Configuration**
+```nginx
+ssl_protocols TLSv1.2 TLSv1.3;
+ssl_prefer_server_ciphers on;
+ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
+```
+
+2. **Environment Setup**
+```bash
+# Required settings
+export CLIENT_KEY="strong-random-key"
+export UPLOAD_FOLDER="/secure/path/uploads"
+
+# SSL settings
+export SSL_CERT="/path/to/cert.pem"
+export SSL_KEY="/path/to/key.pem"
+```
+
+3. **File Permissions**
+```bash
+# Storage directory
+chmod 750 /secure/path/uploads
+chown www-data:www-data /secure/path/uploads
+
+# SSL files
+chmod 600 /path/to/key.pem
+chmod 644 /path/to/cert.pem
+```
+
+### Docker Security
+
+1. **Container Configuration**
+   - Non-root user
+   - Read-only root filesystem
+   - Limited capabilities
+   - Resource limits
+
+2. **Volume Management**
+   - Separate volumes for uploads
+   - Proper permissions
+   - Regular cleanup
+   - Backup strategy
+
+## Security Monitoring
+
+### Logging
+
+1. **Access Logs**
+   - Upload attempts
+   - Download tracking
+   - Error logging
+   - Authentication failures
+
+2. **Audit Trail**
+   - File operations
+   - Configuration changes
+   - System events
+   - Cleanup activities
+
+### Monitoring Requirements
+
+1. **System Monitoring**
+   - Disk usage
+   - Memory utilization
+   - Network traffic
+   - Error rates
+
+2. **Security Alerts**
+   - Authentication failures
+   - Unusual activity
+   - Storage warnings
+   - SSL certificate expiry
+
+## Security Best Practices
+
+### Server Configuration
+
+1. **Firewall Rules**
+   - Allow only required ports
+   - Rate limit connections
+   - Block suspicious IPs
+   - Monitor access patterns
+
+2. **Process Isolation**
+   - Run as dedicated user
+   - Minimal permissions
+   - Resource limits
+   - Process supervision
+
+### Data Protection
+
+1. **File Handling**
+   - Validate file types
+   - Scan for malware
+   - Enforce size limits
+   - Secure deletion
+
+2. **Metadata Protection**
+   - Regular backups
+   - Access controls
+   - Data validation
+   - Secure storage
+
+## Future Security Enhancements
+
+### Planned Features
+
+1. **Rate Limiting**
+   - Per-IP limits
+   - Token bucket algorithm
+   - Configurable thresholds
+   - Automatic blocking
+
+2. **Advanced Authentication**
+   - API key rotation
+   - IP whitelisting
+   - Two-factor auth option
+   - Access tokens
+
+3. **Enhanced Monitoring**
+   - Real-time alerts
+   - Usage analytics
+   - Threat detection
+   - Performance tracking
+
+### Security Roadmap
+
+1. **Short Term**
+   - Implement rate limiting
+   - Add file type restrictions
+   - Enhance logging
+   - Add health checks
+
+2. **Long Term**
+   - Access audit system
+   - Advanced monitoring
+   - Automated backups
+   - High availability
+
+## Incident Response
+
+### Response Plan
+
+1. **Detection**
+   - Monitor logs
+   - Track metrics
+   - Alert on anomalies
+   - User reports
+
+2. **Response**
+   - Assess impact
+   - Contain threat
+   - Notify stakeholders
+   - Document incident
+
+3. **Recovery**
+   - Restore service
+   - Update security
+   - Review procedures
+   - Implement fixes
+
+### Security Contacts
+
+For security issues or vulnerabilities:
+1. Create a GitHub issue with [SECURITY] prefix
+2. For critical issues, contact maintainers directly
+3. Follow responsible disclosure guidelines
+4. Allow time for patches before public disclosure