From 8eac7b90c65c9623e0284b0b090effc6d8ea2a2c Mon Sep 17 00:00:00 2001 From: Matt Hills Date: Wed, 16 Apr 2025 22:54:53 -0400 Subject: [PATCH] Adding docs --- PROJECT.md | 155 ++++++++++++++++ TASK.md | 20 ++- docs/API.md | 188 ++++++++++++++++++++ docs/ARCHITECTURE.md | 206 +++++++++++++++++++++ LARGE_UPLOAD.md => docs/LARGE_UPLOAD.md | 0 docs/SECURITY.md | 226 ++++++++++++++++++++++++ 6 files changed, 789 insertions(+), 6 deletions(-) create mode 100644 PROJECT.md create mode 100644 docs/API.md create mode 100644 docs/ARCHITECTURE.md rename LARGE_UPLOAD.md => docs/LARGE_UPLOAD.md (100%) create mode 100644 docs/SECURITY.md diff --git a/PROJECT.md b/PROJECT.md new file mode 100644 index 0000000..585a204 --- /dev/null +++ b/PROJECT.md @@ -0,0 +1,155 @@ +# Simple File Upload Server Project + +## Project Overview +The Simple File Upload Server is a Flask-based application designed for secure, temporary file sharing and mobile app log uploads. It follows a clean MVC architecture and implements modern web standards for both API and user interface interactions. + +## Project Rules and Guidelines + +### Code Organization +1. **MVC Pattern** + - Models: Place all data models in `src/models/` + - Views: Store templates in `src/templates/` + - Controllers: Keep business logic in `src/controllers/` + +2. **File Structure** + - Source code belongs in `src/` + - Configuration files in root directory + - Temporary files in `tmp/uploads/` + - Templates in `src/templates/` + +### Development Rules +1. **Code Style** + - Follow PEP 8 guidelines for Python code + - Use type hints where possible + - Document all public functions and classes + - Keep functions focused and single-purpose + +2. **Security Requirements** + - All uploads must be authenticated with API key + - File names must be sanitized + - SSL required for production + - Implement proper error handling + - Clean up expired/abandoned files + +3. **API Guidelines** + - Maintain backward compatibility + - Document all endpoint changes + - Include proper error responses + - Use consistent response formats + - Validate all input parameters + +4. **Feature Implementation** + - Add tests for new features + - Update documentation + - Follow chunked upload protocol for large files + - Maintain mobile app compatibility + - Consider both WiFi and mobile network scenarios + +### Testing Requirements +1. **New Features** + - Unit tests required + - Test both success and failure cases + - Include mobile network testing + - Verify chunked upload functionality + +2. **Performance Testing** + - Test with files >100MB + - Verify timeout configurations + - Test under various network conditions + - Monitor memory usage + +### Deployment Rules +1. **Docker Requirements** + - Use multi-stage builds + - Include only necessary files + - Set proper permissions + - Configure appropriate timeouts + +2. **Production Setup** + - SSL certificates required + - Strong CLIENT_KEY required + - Proper file cleanup configuration + - Monitoring setup + +## Documentation Structure + +### Core Documentation +1. **README.md** + - Basic setup and configuration + - API endpoint documentation + - Docker deployment instructions + - Security best practices + [View README.md for details](README.md) + +2. **TASK.md** + - Project roadmap + - Feature implementation status + - Future enhancements + - Current objectives + [View TASK.md for progress](TASK.md) + +3. **LARGE_UPLOAD.md** + - Chunked upload implementation + - Network handling strategies + - Mobile optimization + - Technical specifications + [View LARGE_UPLOAD.md for implementation details](LARGE_UPLOAD.md) + +## Version Control Rules +1. **Branching Strategy** + - Main branch: production-ready code + - Develop branch: integration branch + - Feature branches: feature/* prefix + - Hotfix branches: hotfix/* prefix + +2. **Commit Guidelines** + - Clear, descriptive commit messages + - Reference issue numbers + - Single responsibility per commit + - Include relevant tests + +3. **Release Process** + - Tag all releases (v*.*.*) + - Update version.txt + - Generate change logs + - Test deployment process + +## CI/CD Pipeline +1. **Automated Processes** + - Build testing + - Unit test execution + - Docker image building + - Version tagging + - DockerHub publishing + +2. **Quality Gates** + - All tests must pass + - Code coverage requirements + - No security vulnerabilities + - Proper version tagging + +## Monitoring and Maintenance +1. **Server Health** + - Regular cleanup of expired files + - Monitor disk usage + - Track API usage + - Log error rates + +2. **Performance Metrics** + - Upload success rates + - Download completion rates + - Response times + - Chunked upload efficiency + +## Support and Updates +1. **Issue Management** + - Use GitHub issues for tracking + - Label issues appropriately + - Follow security disclosure process + - Maintain changelog + +2. **Documentation Updates** + - Keep API docs current + - Update setup instructions + - Document known issues + - Maintain upgrade guides \ No newline at end of file diff --git a/TASK.md b/TASK.md index 4b137b4..e7657fd 100644 --- a/TASK.md +++ b/TASK.md @@ -26,6 +26,11 @@ - [x] Configurable limits - [x] API response format - [x] API key authentication +- [x] Chunked upload support + - [x] Upload session management + - [x] Progress tracking + - [x] Resume capability + - [x] Chunk verification ### Download System - [x] Code validation @@ -59,18 +64,21 @@ ### Infrastructure - [ ] Add unit tests - [x] Set up CI/CD pipeline + - [x] Automated Docker builds + - [x] Version tagging + - [x] DockerHub publishing - [x] Implement persistent storage - [ ] Add monitoring/metrics - [ ] Create backup system - [ ] Add health check endpoints ### Chunked Upload Support -- [ ] Implement new API endpoints for chunked uploads (/upload/chunk, /upload/start, /upload/complete) -- [ ] Add server-side chunk management and reassembly -- [ ] Configure Gunicorn with appropriate timeouts for large uploads -- [ ] Implement temporary storage for in-progress chunked uploads -- [ ] Add automatic cleanup for abandoned uploads -- [ ] Create progress tracking for partially complete uploads +- [x] Implement new API endpoints for chunked uploads (/upload/chunk, /upload/start, /upload/complete) +- [x] Add server-side chunk management and reassembly +- [x] Configure Gunicorn with appropriate timeouts for large uploads +- [x] Implement temporary storage for in-progress chunked uploads +- [x] Add automatic cleanup for abandoned uploads +- [x] Create progress tracking for partially complete uploads - [ ] Test with large files (>100MB) over various connection types ### UI/UX diff --git a/docs/API.md b/docs/API.md new file mode 100644 index 0000000..324aebd --- /dev/null +++ b/docs/API.md @@ -0,0 +1,188 @@ +# API Documentation + +## Overview +The Simple File Upload Server provides a RESTful API for file uploads and downloads, with special support for chunked uploads of large files. All upload endpoints require API key authentication. + +## Authentication +All upload endpoints require the `X-Client-Key` header with a valid API key. + +Example: +```http +X-Client-Key: your-secret-key +``` + +## Endpoints + +### File Upload + +#### POST /upload +Upload a single file in one request. + +**Request:** +- Method: `POST` +- Content-Type: `multipart/form-data` +- Authentication: Required + +**Parameters:** +| Name | Type | Required | Description | +|------|------|----------|-------------| +| file | File | Yes | The file to upload | +| max_downloads | Integer | No | Maximum number of downloads (default: 1) | +| expiry_hours | Integer | No | Hours until file expires (default: 24) | + +**Response:** +```json +{ + "code": "ABC123" +} +``` + +### Chunked Upload + +#### POST /upload/start +Initialize a chunked upload session. + +**Request:** +- Method: `POST` +- Content-Type: `multipart/form-data` +- Authentication: Required + +**Parameters:** +| Name | Type | Required | Description | +|------|------|----------|-------------| +| filename | String | Yes | Name of the file being uploaded | +| total_chunks | Integer | Yes | Total number of chunks to expect | +| max_downloads | Integer | No | Maximum number of downloads (default: 1) | +| expiry_hours | Integer | No | Hours until file expires (default: 24) | + +**Response:** +```json +{ + "upload_id": "abC123dEf456gHij" +} +``` + +#### POST /upload/chunk +Upload a single chunk of a file. + +**Request:** +- Method: `POST` +- Content-Type: `multipart/form-data` +- Authentication: Required + +**Parameters:** +| Name | Type | Required | Description | +|------|------|----------|-------------| +| upload_id | String | Yes | Upload session ID from /upload/start | +| chunk_index | Integer | Yes | Index of this chunk (0-based) | +| chunk | File | Yes | The chunk data | + +**Response:** +```json +{ + "success": true, + "received_chunks": 5, + "total_chunks": 10, + "chunk_index": 4 +} +``` + +#### POST /upload/verify +Verify the status of uploaded chunks. + +**Request:** +- Method: `POST` +- Content-Type: `multipart/form-data` +- Authentication: Required + +**Parameters:** +| Name | Type | Required | Description | +|------|------|----------|-------------| +| upload_id | String | Yes | Upload session ID | + +**Response:** +```json +{ + "total_chunks": 10, + "received_chunks": 5, + "missing_chunks": [5,6,7,8,9] +} +``` + +#### POST /upload/complete +Complete a chunked upload session. + +**Request:** +- Method: `POST` +- Content-Type: `multipart/form-data` +- Authentication: Required + +**Parameters:** +| Name | Type | Required | Description | +|------|------|----------|-------------| +| upload_id | String | Yes | Upload session ID | + +**Response:** +```json +{ + "code": "ABC123" +} +``` + +### File Download + +#### GET /download/{code} +Download a file using its 6-character code. + +**Request:** +- Method: `GET` +- Authentication: Not required + +**Parameters:** +| Name | Type | Required | Description | +|------|------|----------|-------------| +| code | String | Yes | 6-character download code | + +**Response:** +- Success: File download begins +- Error: HTML error page with user-friendly message + +### File Management + +#### DELETE /delete/{code} +Delete a file using its code. + +**Request:** +- Method: `DELETE` +- Authentication: Not required + +**Parameters:** +| Name | Type | Required | Description | +|------|------|----------|-------------| +| code | String | Yes | 6-character download code | + +**Response:** +```json +{ + "message": "File deleted successfully" +} +``` + +## Error Responses + +### Common Error Format +```json +{ + "error": "Error message description" +} +``` + +### HTTP Status Codes +- 200: Success +- 400: Bad Request (missing parameters, invalid format) +- 401: Unauthorized (invalid or missing API key) +- 404: Not Found (invalid code, expired file) +- 500: Server Error + +## Rate Limits +Currently not implemented. See future enhancements in TASK.md. \ No newline at end of file diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 0000000..1755b04 --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,206 @@ +# System Architecture + +## Overview +The Simple File Upload Server is built using a Model-View-Controller (MVC) architecture pattern in Flask, with clear separation of concerns and modular design. The system handles both traditional file uploads and chunked uploads for large files, with a focus on security and reliability. + +## System Components + +### 1. Core Architecture (MVC) + +#### Models (`src/models/`) +- **FileMetadata** + - Handles file metadata management + - Tracks download counts, expiry times + - Manages file paths and access codes + - Implements cleanup for expired files + +#### Views (`src/templates/`) +- **index.html** + - Modern, responsive UI for file downloads + - Client-side validation + - Progress tracking + - Error handling +- **error.html** + - User-friendly error pages + - Contextual error messages + - Clean, responsive design + +#### Controllers (`src/controllers/`) +- **file_controller.py** + - Upload/download logic + - Chunked upload management + - File cleanup operations + - Access code generation + +### 2. Data Flow + +```mermaid +graph TD + A[Client] -->|Upload Request| B[Flask App] + B -->|Validate| C[FileController] + C -->|Store| D[FileMetadata] + D -->|Save| E[Filesystem] + A -->|Download Request| B + B -->|Validate Code| C + C -->|Check| D + D -->|Retrieve| E + E -->|Serve| A +``` + +### 3. File Storage System + +#### Directory Structure +``` +tmp/ +├── uploads/ +│ ├── [regular files] +│ └── chunks/ +│ └── [upload_session_folders]/ +``` + +#### Metadata Storage +- `file_metadata.json`: Tracks regular file uploads +- `chunk_metadata.json`: Manages chunked upload sessions + +### 4. Security Architecture + +#### Authentication +- API key authentication for uploads +- 6-digit alphanumeric codes for downloads +- SSL/TLS support for secure transmission + +#### File Security +- Secure filename sanitization +- Temporary storage with auto-cleanup +- Download limit enforcement +- File expiry system + +### 5. Chunked Upload System + +#### Components +1. **Session Management** + - Unique upload session IDs + - Chunk tracking and verification + - Progress monitoring + +2. **File Assembly** + ```mermaid + graph LR + A[Client] -->|Chunks| B[Temporary Storage] + B -->|Assembly| C[Final File] + C -->|Metadata| D[Database] + ``` + +3. **Cleanup Process** + - Automatic cleanup of abandoned uploads + - Temporary file management + - Session expiry handling + +### 6. Logging System (`src/utils/mLogger.py`) + +#### Logging Levels +- INFO: Normal operations +- WARNING: Potential issues +- ERROR: Operation failures +- DEBUG: Development information +- VERBOSE: Detailed tracking + +#### Log Format +``` +timestamp|level|module::message +``` + +### 7. Error Handling + +#### Types of Errors +1. **Client Errors** + - Invalid file types + - Missing parameters + - Authentication failures + - Invalid codes + +2. **Server Errors** + - Storage issues + - File assembly failures + - Database errors + - Network timeouts + +#### Error Response Format +```json +{ + "error": "Error description", + "details": "Additional information" +} +``` + +### 8. Configuration System + +#### Environment Variables +- `CLIENT_KEY`: API authentication +- `UPLOAD_FOLDER`: File storage location +- `SSL_CERT`: SSL certificate path +- `SSL_KEY`: SSL key path + +#### Application Constants +- Maximum file size +- Chunk size limits +- Session timeouts +- Cleanup intervals + +## Performance Considerations + +### 1. Resource Management +- Chunked upload for large files +- Temporary storage cleanup +- Memory-efficient file handling +- Connection timeout management + +### 2. Scalability +- Stateless design +- File system abstraction +- Configurable worker processes +- Independent upload sessions + +### 3. Network Optimization +- Adaptive chunk sizes +- Progress tracking +- Resume capability +- Connection monitoring + +## Deployment Architecture + +### Docker Container +```mermaid +graph TD + A[Docker Container] -->|Runs| B[Gunicorn] + B -->|WSGI| C[Flask App] + C -->|Stores| D[Volume Mount] + C -->|Logs| E[Log Volume] +``` + +### Process Management +- Gunicorn worker processes +- Configurable timeouts +- SSL termination +- Health monitoring + +## Future Architecture Considerations + +### Planned Improvements +1. **Monitoring System** + - API usage metrics + - Storage utilization + - Error rate tracking + - Performance monitoring + +2. **Enhanced Security** + - Rate limiting + - IP whitelisting + - File type restrictions + - Access audit logging + +3. **High Availability** + - Load balancing + - Redundant storage + - Session persistence + - Backup management \ No newline at end of file diff --git a/LARGE_UPLOAD.md b/docs/LARGE_UPLOAD.md similarity index 100% rename from LARGE_UPLOAD.md rename to docs/LARGE_UPLOAD.md diff --git a/docs/SECURITY.md b/docs/SECURITY.md new file mode 100644 index 0000000..23f4d5d --- /dev/null +++ b/docs/SECURITY.md @@ -0,0 +1,226 @@ +# Security Documentation + +## Overview +This document outlines the security measures implemented in the Simple File Upload Server and provides guidelines for secure deployment and operation. + +## Security Features + +### Authentication +1. **API Key Authentication** + - Required for all upload endpoints + - Configured via CLIENT_KEY environment variable + - Transmitted via X-Client-Key HTTP header + - No default keys allowed in production + +2. **Download Security** + - 6-digit alphanumeric codes + - One-time use downloads (configurable) + - Automatic expiration + - No directory listing + +### File Security + +1. **Upload Security** + - Secure filename sanitization + - No executable permissions + - Temporary storage with cleanup + - Chunked upload verification + +2. **Storage Security** + - Files stored outside web root + - Metadata separation from files + - Automatic cleanup of expired files + - Abandoned upload detection + +3. **Download Controls** + - Maximum download limits + - Expiry time enforcement + - Force download headers + - Content-Type verification + +### Network Security + +1. **SSL/TLS** + - Required for production + - Modern cipher configuration + - Perfect forward secrecy + - SSL certificate validation + +2. **Headers** + - Secure response headers + - No version disclosure + - XSS protection + - Content security policy + +## Secure Deployment + +### Production Requirements + +1. **SSL Configuration** +```nginx +ssl_protocols TLSv1.2 TLSv1.3; +ssl_prefer_server_ciphers on; +ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384; +``` + +2. **Environment Setup** +```bash +# Required settings +export CLIENT_KEY="strong-random-key" +export UPLOAD_FOLDER="/secure/path/uploads" + +# SSL settings +export SSL_CERT="/path/to/cert.pem" +export SSL_KEY="/path/to/key.pem" +``` + +3. **File Permissions** +```bash +# Storage directory +chmod 750 /secure/path/uploads +chown www-data:www-data /secure/path/uploads + +# SSL files +chmod 600 /path/to/key.pem +chmod 644 /path/to/cert.pem +``` + +### Docker Security + +1. **Container Configuration** + - Non-root user + - Read-only root filesystem + - Limited capabilities + - Resource limits + +2. **Volume Management** + - Separate volumes for uploads + - Proper permissions + - Regular cleanup + - Backup strategy + +## Security Monitoring + +### Logging + +1. **Access Logs** + - Upload attempts + - Download tracking + - Error logging + - Authentication failures + +2. **Audit Trail** + - File operations + - Configuration changes + - System events + - Cleanup activities + +### Monitoring Requirements + +1. **System Monitoring** + - Disk usage + - Memory utilization + - Network traffic + - Error rates + +2. **Security Alerts** + - Authentication failures + - Unusual activity + - Storage warnings + - SSL certificate expiry + +## Security Best Practices + +### Server Configuration + +1. **Firewall Rules** + - Allow only required ports + - Rate limit connections + - Block suspicious IPs + - Monitor access patterns + +2. **Process Isolation** + - Run as dedicated user + - Minimal permissions + - Resource limits + - Process supervision + +### Data Protection + +1. **File Handling** + - Validate file types + - Scan for malware + - Enforce size limits + - Secure deletion + +2. **Metadata Protection** + - Regular backups + - Access controls + - Data validation + - Secure storage + +## Future Security Enhancements + +### Planned Features + +1. **Rate Limiting** + - Per-IP limits + - Token bucket algorithm + - Configurable thresholds + - Automatic blocking + +2. **Advanced Authentication** + - API key rotation + - IP whitelisting + - Two-factor auth option + - Access tokens + +3. **Enhanced Monitoring** + - Real-time alerts + - Usage analytics + - Threat detection + - Performance tracking + +### Security Roadmap + +1. **Short Term** + - Implement rate limiting + - Add file type restrictions + - Enhance logging + - Add health checks + +2. **Long Term** + - Access audit system + - Advanced monitoring + - Automated backups + - High availability + +## Incident Response + +### Response Plan + +1. **Detection** + - Monitor logs + - Track metrics + - Alert on anomalies + - User reports + +2. **Response** + - Assess impact + - Contain threat + - Notify stakeholders + - Document incident + +3. **Recovery** + - Restore service + - Update security + - Review procedures + - Implement fixes + +### Security Contacts + +For security issues or vulnerabilities: +1. Create a GitHub issue with [SECURITY] prefix +2. For critical issues, contact maintainers directly +3. Follow responsible disclosure guidelines +4. Allow time for patches before public disclosure \ No newline at end of file