mattintech/simplefileupload-server
1
0
Fork 0
mirror of https://github.com/mattintech/simplefileupload-server.git synced 2026-07-11 11:51:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

Adding docs

Browse Source
This commit is contained in:
Matt Hills
2025-04-16 22:54:53 -04:00
parent 41ceeea90a
commit 8eac7b90c6
6 changed files with 789 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

188
docs/API.md Normal file
View File

@@ -0,0 +1,188 @@
# API Documentation
## Overview
The Simple File Upload Server provides a RESTful API for file uploads and downloads, with special support for chunked uploads of large files. All upload endpoints require API key authentication.
## Authentication
All upload endpoints require the `X-Client-Key` header with a valid API key.
Example:
```http
X-Client-Key: your-secret-key
```
## Endpoints
### File Upload
#### POST /upload
Upload a single file in one request.
**Request:**
- Method: `POST`
- Content-Type: `multipart/form-data`
- Authentication: Required
**Parameters:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| file | File | Yes | The file to upload |
| max_downloads | Integer | No | Maximum number of downloads (default: 1) |
| expiry_hours | Integer | No | Hours until file expires (default: 24) |
**Response:**
```json
{
"code": "ABC123"
}
```
### Chunked Upload
#### POST /upload/start
Initialize a chunked upload session.
**Request:**
- Method: `POST`
- Content-Type: `multipart/form-data`
- Authentication: Required
**Parameters:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| filename | String | Yes | Name of the file being uploaded |
| total_chunks | Integer | Yes | Total number of chunks to expect |
| max_downloads | Integer | No | Maximum number of downloads (default: 1) |
| expiry_hours | Integer | No | Hours until file expires (default: 24) |
**Response:**
```json
{
"upload_id": "abC123dEf456gHij"
}
```
#### POST /upload/chunk
Upload a single chunk of a file.
**Request:**
- Method: `POST`
- Content-Type: `multipart/form-data`
- Authentication: Required
**Parameters:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| upload_id | String | Yes | Upload session ID from /upload/start |
| chunk_index | Integer | Yes | Index of this chunk (0-based) |
| chunk | File | Yes | The chunk data |
**Response:**
```json
{
"success": true,
"received_chunks": 5,
"total_chunks": 10,
"chunk_index": 4
}
```
#### POST /upload/verify
Verify the status of uploaded chunks.
**Request:**
- Method: `POST`
- Content-Type: `multipart/form-data`
- Authentication: Required
**Parameters:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| upload_id | String | Yes | Upload session ID |
**Response:**
```json
{
"total_chunks": 10,
"received_chunks": 5,
"missing_chunks": [5,6,7,8,9]
}
```
#### POST /upload/complete
Complete a chunked upload session.
**Request:**
- Method: `POST`
- Content-Type: `multipart/form-data`
- Authentication: Required
**Parameters:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| upload_id | String | Yes | Upload session ID |
**Response:**
```json
{
"code": "ABC123"
}
```
### File Download
#### GET /download/{code}
Download a file using its 6-character code.
**Request:**
- Method: `GET`
- Authentication: Not required
**Parameters:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| code | String | Yes | 6-character download code |
**Response:**
- Success: File download begins
- Error: HTML error page with user-friendly message
### File Management
#### DELETE /delete/{code}
Delete a file using its code.
**Request:**
- Method: `DELETE`
- Authentication: Not required
**Parameters:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| code | String | Yes | 6-character download code |
**Response:**
```json
{
"message": "File deleted successfully"
}
```
## Error Responses
### Common Error Format
```json
{
"error": "Error message description"
}
```
### HTTP Status Codes
- 200: Success
- 400: Bad Request (missing parameters, invalid format)
- 401: Unauthorized (invalid or missing API key)
- 404: Not Found (invalid code, expired file)
- 500: Server Error
## Rate Limits
Currently not implemented. See future enhancements in TASK.md.

206
docs/ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,206 @@
# System Architecture
## Overview
The Simple File Upload Server is built using a Model-View-Controller (MVC) architecture pattern in Flask, with clear separation of concerns and modular design. The system handles both traditional file uploads and chunked uploads for large files, with a focus on security and reliability.
## System Components
### 1. Core Architecture (MVC)
#### Models (`src/models/`)
- **FileMetadata**
- Handles file metadata management
- Tracks download counts, expiry times
- Manages file paths and access codes
- Implements cleanup for expired files
#### Views (`src/templates/`)
- **index.html**
- Modern, responsive UI for file downloads
- Client-side validation
- Progress tracking
- Error handling
- **error.html**
- User-friendly error pages
- Contextual error messages
- Clean, responsive design
#### Controllers (`src/controllers/`)
- **file_controller.py**
- Upload/download logic
- Chunked upload management
- File cleanup operations
- Access code generation
### 2. Data Flow
```mermaid
graph TD
A[Client] -->|Upload Request| B[Flask App]
B -->|Validate| C[FileController]
C -->|Store| D[FileMetadata]
D -->|Save| E[Filesystem]
A -->|Download Request| B
B -->|Validate Code| C
C -->|Check| D
D -->|Retrieve| E
E -->|Serve| A
```
### 3. File Storage System
#### Directory Structure
```
tmp/
├── uploads/
│ ├── [regular files]
│ └── chunks/
│ └── [upload_session_folders]/
```
#### Metadata Storage
- `file_metadata.json`: Tracks regular file uploads
- `chunk_metadata.json`: Manages chunked upload sessions
### 4. Security Architecture
#### Authentication
- API key authentication for uploads
- 6-digit alphanumeric codes for downloads
- SSL/TLS support for secure transmission
#### File Security
- Secure filename sanitization
- Temporary storage with auto-cleanup
- Download limit enforcement
- File expiry system
### 5. Chunked Upload System
#### Components
1. **Session Management**
- Unique upload session IDs
- Chunk tracking and verification
- Progress monitoring
2. **File Assembly**
```mermaid
graph LR
A[Client] -->|Chunks| B[Temporary Storage]
B -->|Assembly| C[Final File]
C -->|Metadata| D[Database]
```
3. **Cleanup Process**
- Automatic cleanup of abandoned uploads
- Temporary file management
- Session expiry handling
### 6. Logging System (`src/utils/mLogger.py`)
#### Logging Levels
- INFO: Normal operations
- WARNING: Potential issues
- ERROR: Operation failures
- DEBUG: Development information
- VERBOSE: Detailed tracking
#### Log Format
```
timestamp|level|module::message
```
### 7. Error Handling
#### Types of Errors
1. **Client Errors**
- Invalid file types
- Missing parameters
- Authentication failures
- Invalid codes
2. **Server Errors**
- Storage issues
- File assembly failures
- Database errors
- Network timeouts
#### Error Response Format
```json
{
"error": "Error description",
"details": "Additional information"
}
```
### 8. Configuration System
#### Environment Variables
- `CLIENT_KEY`: API authentication
- `UPLOAD_FOLDER`: File storage location
- `SSL_CERT`: SSL certificate path
- `SSL_KEY`: SSL key path
#### Application Constants
- Maximum file size
- Chunk size limits
- Session timeouts
- Cleanup intervals
## Performance Considerations
### 1. Resource Management
- Chunked upload for large files
- Temporary storage cleanup
- Memory-efficient file handling
- Connection timeout management
### 2. Scalability
- Stateless design
- File system abstraction
- Configurable worker processes
- Independent upload sessions
### 3. Network Optimization
- Adaptive chunk sizes
- Progress tracking
- Resume capability
- Connection monitoring
## Deployment Architecture
### Docker Container
```mermaid
graph TD
A[Docker Container] -->|Runs| B[Gunicorn]
B -->|WSGI| C[Flask App]
C -->|Stores| D[Volume Mount]
C -->|Logs| E[Log Volume]
```
### Process Management
- Gunicorn worker processes
- Configurable timeouts
- SSL termination
- Health monitoring
## Future Architecture Considerations
### Planned Improvements
1. **Monitoring System**
- API usage metrics
- Storage utilization
- Error rate tracking
- Performance monitoring
2. **Enhanced Security**
- Rate limiting
- IP whitelisting
- File type restrictions
- Access audit logging
3. **High Availability**
- Load balancing
- Redundant storage
- Session persistence
- Backup management

326
docs/LARGE_UPLOAD.md Normal file
View File

@@ -0,0 +1,326 @@
# Large File Upload Analysis and Improvements
## Current Implementation Analysis
### Client Side (Android)
The SimpleLogUpload Android application is designed to upload large log files (up to 220MB) to a simple file upload server. The current implementation uses:
- OkHttp3 for HTTP requests
- A single monolithic file upload in one HTTP request
- Fixed network timeouts:
- Connection timeout: 60 seconds
- Write timeout: 180 seconds (3 minutes)
- Read timeout: 60 seconds
- Basic retry on connection failure
Behavior by network type:
- **WiFi**: Uploads work successfully
- **5G/Mobile**: Socket timeouts occur during upload (`java.net.SocketException: Socket closed`)
### Server Side (Python/Flask)
The server is a simple Flask application with:
- Standard file upload endpoint (`/upload`)
- No explicit chunked upload support
- Uses Gunicorn as WSGI server without specific timeout configurations
- No specific optimizations for large files
## Problem Root Causes
1. **Mobile Network Limitations**:
- Mobile carriers may have connection restrictions for large uploads
- 5G networks may have more aggressive connection management
- Network fluctuations are more common in mobile networks
2. **Timeout Configuration**:
- Current timeout settings may be insufficient for large files over mobile networks
- Server and client timeouts aren't aligned
3. **Single Upload Approach**:
- The entire 220MB file is uploaded in a single HTTP request
- No resume capability if connection drops
- No progress tracking or partial file recovery
4. **Missing Network Adaptability**:
- No differentiation between WiFi and mobile network settings
- No adaptive timeout or retry mechanisms
- No connection monitoring during upload
## Recommended Solutions
### 1. Implement Chunked Uploading
Develop a proper chunked upload system with these features:
```java
// Example chunked upload strategy
public class ChunkedUploadManager {
// Adjust these based on testing
private static final int DEFAULT_CHUNK_SIZE = 5 * 1024 * 1024; // 5MB chunks
private static final int MAX_RETRY_ATTEMPTS = 3;
private static final int RETRY_DELAY_MS = 1000;
// File metadata
private File file;
private int chunkSize;
private int totalChunks;
private String uploadId;
// Network state
private boolean isWifiConnection;
private boolean isPaused;
// Tracking
private int currentChunkIndex;
private Set<Integer> uploadedChunks = new HashSet<>();
public ChunkedUploadManager(File file, boolean isWifiConnection) {
this.file = file;
this.isWifiConnection = isWifiConnection;
// Adjust chunk size based on network type
this.chunkSize = isWifiConnection ? DEFAULT_CHUNK_SIZE : (DEFAULT_CHUNK_SIZE / 2);
this.totalChunks = (int) Math.ceil((double) file.length() / chunkSize);
}
// Methods for initialization, chunk upload, tracking, etc.
}
```
**Server Changes Required**:
```python
# New endpoint for chunked uploads
@app.route('/upload/chunk', methods=['POST'])
@require_client_key
def upload_chunk():
# Get parameters
chunk_index = int(request.form.get('chunk_index'))
total_chunks = int(request.form.get('total_chunks'))
upload_id = request.form.get('upload_id')
filename = request.form.get('filename')
# Handle chunk file
chunk_file = request.files['chunk']
# Process chunk
# ...
return jsonify({'success': True})
# New endpoint to start chunked upload session
@app.route('/upload/start', methods=['POST'])
@require_client_key
def start_chunked_upload():
# Initialize a new upload session
upload_id = generate_unique_id()
# Store upload metadata
# ...
return jsonify({'upload_id': upload_id})
# New endpoint to complete chunked upload
@app.route('/upload/complete', methods=['POST'])
@require_client_key
def complete_chunked_upload():
# Combine chunks into final file
# ...
return jsonify({'code': result_code})
```
### 2. Adaptive Network Settings
Implement network-aware settings that adjust based on connection type:
```java
private void configureNetworkSettings() {
ConnectivityManager cm = (ConnectivityManager) context.getSystemService(Context.CONNECTIVITY_SERVICE);
NetworkInfo activeNetwork = cm.getActiveNetworkInfo();
boolean isWifi = activeNetwork != null && activeNetwork.getType() == ConnectivityManager.TYPE_WIFI;
// Adjust settings based on network type
if (isWifi) {
// WiFi settings - can be more aggressive
CONNECTION_TIMEOUT = 60; // seconds
WRITE_TIMEOUT = 180; // seconds
READ_TIMEOUT = 60; // seconds
CHUNK_SIZE = 10_485_760; // 10MB
} else {
// Mobile network settings - more conservative
CONNECTION_TIMEOUT = 30; // seconds
WRITE_TIMEOUT = 90; // seconds
READ_TIMEOUT = 30; // seconds
CHUNK_SIZE = 2_097_152; // 2MB
}
}
```
### 3. Network Monitoring and Handling
Implement network state monitoring to handle:
- Network type changes during upload
- Connection losses
- Bandwidth fluctuations
```java
private void registerNetworkCallback() {
ConnectivityManager cm = (ConnectivityManager) context.getSystemService(Context.CONNECTIVITY_SERVICE);
NetworkRequest.Builder builder = new NetworkRequest.Builder();
cm.registerNetworkCallback(builder.build(), new ConnectivityManager.NetworkCallback() {
@Override
public void onAvailable(Network network) {
// Resume upload if paused
resumeUploadIfPaused();
}
@Override
public void onLost(Network network) {
// Pause upload
pauseUpload();
}
@Override
public void onCapabilitiesChanged(Network network, NetworkCapabilities capabilities) {
// Adjust strategies based on new network capabilities
boolean isUnmetered = capabilities.hasCapability(NetworkCapabilities.NET_CAPABILITY_NOT_METERED);
adjustUploadParameters(isUnmetered);
}
});
}
```
### 4. Progressive Backoff and Retry Strategy
Implement a more sophisticated retry mechanism:
```java
private void uploadChunkWithRetry(int chunkIndex) {
int retryCount = 0;
int maxRetries = 5;
long backoffMs = 1000; // Start with 1 second
while (retryCount < maxRetries) {
try {
// Attempt to upload chunk
boolean success = uploadChunk(chunkIndex);
if (success) {
return; // Success, exit retry loop
}
} catch (IOException e) {
Log.e(TAG, "Chunk upload failed: " + e.getMessage());
}
retryCount++;
if (retryCount < maxRetries) {
// Calculate backoff with exponential increase and jitter
long jitter = (long) (Math.random() * 0.3 * backoffMs);
long sleepTime = backoffMs + jitter;
Log.d(TAG, "Retrying chunk " + chunkIndex + " after " + sleepTime + "ms (attempt " + retryCount + ")");
try {
Thread.sleep(sleepTime);
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
return;
}
// Increase backoff for next iteration
backoffMs = Math.min(backoffMs * 2, 30000); // Cap at 30 seconds
}
}
// If we get here, all retries failed
Log.e(TAG, "All retry attempts failed for chunk " + chunkIndex);
reportChunkFailed(chunkIndex);
}
```
### 5. Enhanced Progress Tracking and Resumability
Implement more granular progress tracking for better user experience and resumability:
```java
public class UploadProgress {
private long totalBytes;
private long uploadedBytes;
private int totalChunks;
private Set<Integer> completedChunks;
private boolean isComplete;
private String sessionId;
// Persist progress to disk for resumability
public void saveProgress() {
SharedPreferences prefs = context.getSharedPreferences("upload_progress", Context.MODE_PRIVATE);
SharedPreferences.Editor editor = prefs.edit();
// Store progress data
editor.putString("session_" + sessionId + "_chunks", TextUtils.join(",", completedChunks));
editor.putLong("session_" + sessionId + "_bytes", uploadedBytes);
editor.putBoolean("session_" + sessionId + "_complete", isComplete);
editor.apply();
}
// Load progress for resuming uploads
public static UploadProgress loadProgress(Context context, String sessionId) {
// Implementation details
}
}
```
### 6. Server-Side Optimizations
Modify the server to better handle large file uploads:
1. Implement explicit chunked upload support
2. Configure Gunicorn timeouts appropriately:
```
gunicorn wsgi:app --bind 0.0.0.0:7777 --timeout 300 --workers 4
```
3. Add support for resumable uploads with progress tracking
4. Implement temporary file cleanup for abandoned uploads
## Implementation Priority
1. **First Phase**:
- Implement basic chunked uploading
- Add network type detection and adaptive settings
- Configure server for higher timeouts
2. **Second Phase**:
- Add upload resumability
- Implement network state monitoring
- Enhanced progress tracking
- Progressive retry mechanism
3. **Final Phase**:
- Server-side optimizations for chunked uploads
- Background upload functionality
- Bandwidth control options
## Testing Strategy
1. Test uploading the same 220MB file under various network conditions:
- Stable WiFi
- Congested WiFi
- 5G with strong signal
- 5G with weak signal
- 4G fallback
- Network transitions (WiFi → Mobile)
2. Measure and compare:
- Success rate of uploads
- Total upload time
- Battery consumption
- Data usage efficiency
## Conclusion
The socket timeout issues on 5G networks stem from trying to upload large files in a single HTTP request without adapting to the characteristics of mobile networks. By implementing chunked uploads with resume capability, network-aware configurations, and robust error handling, we can create a more reliable upload solution for all network types.
The most critical issue to address is the lack of chunking, which leaves the application vulnerable to connection interruptions, particularly on mobile networks where connections may be less stable than WiFi.

226
docs/SECURITY.md Normal file
View File

@@ -0,0 +1,226 @@
# Security Documentation
## Overview
This document outlines the security measures implemented in the Simple File Upload Server and provides guidelines for secure deployment and operation.
## Security Features
### Authentication
1. **API Key Authentication**
- Required for all upload endpoints
- Configured via CLIENT_KEY environment variable
- Transmitted via X-Client-Key HTTP header
- No default keys allowed in production
2. **Download Security**
- 6-digit alphanumeric codes
- One-time use downloads (configurable)
- Automatic expiration
- No directory listing
### File Security
1. **Upload Security**
- Secure filename sanitization
- No executable permissions
- Temporary storage with cleanup
- Chunked upload verification
2. **Storage Security**
- Files stored outside web root
- Metadata separation from files
- Automatic cleanup of expired files
- Abandoned upload detection
3. **Download Controls**
- Maximum download limits
- Expiry time enforcement
- Force download headers
- Content-Type verification
### Network Security
1. **SSL/TLS**
- Required for production
- Modern cipher configuration
- Perfect forward secrecy
- SSL certificate validation
2. **Headers**
- Secure response headers
- No version disclosure
- XSS protection
- Content security policy
## Secure Deployment
### Production Requirements
1. **SSL Configuration**
```nginx
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
```
2. **Environment Setup**
```bash
# Required settings
export CLIENT_KEY="strong-random-key"
export UPLOAD_FOLDER="/secure/path/uploads"
# SSL settings
export SSL_CERT="/path/to/cert.pem"
export SSL_KEY="/path/to/key.pem"
```
3. **File Permissions**
```bash
# Storage directory
chmod 750 /secure/path/uploads
chown www-data:www-data /secure/path/uploads
# SSL files
chmod 600 /path/to/key.pem
chmod 644 /path/to/cert.pem
```
### Docker Security
1. **Container Configuration**
- Non-root user
- Read-only root filesystem
- Limited capabilities
- Resource limits
2. **Volume Management**
- Separate volumes for uploads
- Proper permissions
- Regular cleanup
- Backup strategy
## Security Monitoring
### Logging
1. **Access Logs**
- Upload attempts
- Download tracking
- Error logging
- Authentication failures
2. **Audit Trail**
- File operations
- Configuration changes
- System events
- Cleanup activities
### Monitoring Requirements
1. **System Monitoring**
- Disk usage
- Memory utilization
- Network traffic
- Error rates
2. **Security Alerts**
- Authentication failures
- Unusual activity
- Storage warnings
- SSL certificate expiry
## Security Best Practices
### Server Configuration
1. **Firewall Rules**
- Allow only required ports
- Rate limit connections
- Block suspicious IPs
- Monitor access patterns
2. **Process Isolation**
- Run as dedicated user
- Minimal permissions
- Resource limits
- Process supervision
### Data Protection
1. **File Handling**
- Validate file types
- Scan for malware
- Enforce size limits
- Secure deletion
2. **Metadata Protection**
- Regular backups
- Access controls
- Data validation
- Secure storage
## Future Security Enhancements
### Planned Features
1. **Rate Limiting**
- Per-IP limits
- Token bucket algorithm
- Configurable thresholds
- Automatic blocking
2. **Advanced Authentication**
- API key rotation
- IP whitelisting
- Two-factor auth option
- Access tokens
3. **Enhanced Monitoring**
- Real-time alerts
- Usage analytics
- Threat detection
- Performance tracking
### Security Roadmap
1. **Short Term**
- Implement rate limiting
- Add file type restrictions
- Enhance logging
- Add health checks
2. **Long Term**
- Access audit system
- Advanced monitoring
- Automated backups
- High availability
## Incident Response
### Response Plan
1. **Detection**
- Monitor logs
- Track metrics
- Alert on anomalies
- User reports
2. **Response**
- Assess impact
- Contain threat
- Notify stakeholders
- Document incident
3. **Recovery**
- Restore service
- Update security
- Review procedures
- Implement fixes
### Security Contacts
For security issues or vulnerabilities:
1. Create a GitHub issue with [SECURITY] prefix
2. For critical issues, contact maintainers directly
3. Follow responsible disclosure guidelines
4. Allow time for patches before public disclosure