mattintech/simplefileupload-android
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

doc organization

Browse Source
This commit is contained in:
Matt Hills
2025-04-16 21:48:00 -04:00
parent d32fd4690d
commit a86c7b74aa
5 changed files with 75 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

141
docs/CHUNKED_UPLOAD.md Normal file
View File

@@ -0,0 +1,141 @@
# Chunked Upload Implementation
This document describes the chunked upload implementation added to the SimpleLogUpload Android application to solve issues with large file uploads on mobile networks.
## Overview
The chunked upload system splits large files into smaller pieces (chunks), uploads them individually, and then reassembles them on the server. This approach provides several benefits:
1. Better reliability on unstable networks (especially 5G/mobile)
2. Ability to resume interrupted uploads
3. More accurate progress reporting
4. Adaptability to different network conditions
## Components
### 1. ChunkedUploadManager
The core component that handles:
- Splitting files into chunks
- Managing parallel uploads
- Tracking upload progress
- Handling network changes
- Implementing retry logic
- Verifying chunk status with server
- Automatically retrying missing chunks
```java
ChunkedUploadManager uploadManager = new ChunkedUploadManager(
context, file, maxDownloads, expiryHours);
```
### 2. UploadStrategySelector
Determines whether to use chunked uploads based on:
- File size
- Network type (WiFi vs. Mobile)
- Configurable thresholds
```java
boolean useChunked = UploadStrategySelector.shouldUseChunkedUpload(context, file);
```
### 3. Modified UploadController
The existing UploadController now:
- Selects between standard and chunked uploads
- Handles both upload methods with a unified API
- Provides progress updates
## API Requirements
The server implements four endpoints:
1. `/upload/start` - Initializes a chunked upload session
- Accepts: filename, filesize, chunk_size, total_chunks, max_downloads, expiry_hours
- Returns: upload_id
2. `/upload/chunk` - Handles individual chunk uploads
- Accepts: upload_id, chunk_index, total_chunks, chunk (binary)
- Returns: success/failure
3. `/upload/verify` - Verifies which chunks have been received
- Accepts: upload_id
- Returns: received_chunks, missing_chunks, total_chunks
4. `/upload/complete` - Finalizes an upload by reassembling chunks
- Accepts: upload_id, filename, total_chunks
- Returns: upload code (same as standard upload)
## Configuration
The chunked upload system is configured with the following parameters:
- WiFi chunk size: 10MB
- Mobile chunk size: 2MB
- WiFi threshold: 50MB (files larger than this use chunked upload)
- Mobile threshold: 10MB
- WiFi parallel uploads: 3
- Mobile parallel uploads: 1
These values can be adjusted in the `ChunkedUploadManager` and `UploadStrategySelector` classes.
## Enhanced Reliability Features
### Chunk Verification
The system implements robust verification to ensure all chunks are properly uploaded:
1. Client-side tracking of all uploaded chunks by index
2. Server endpoint to verify which chunks have been received
3. Automatic reupload of any missing chunks
4. Verification that files actually exist on disk before finalizing
### Network Handling
The implementation includes robust network handling:
- Detects network type (WiFi/Mobile/5G/LTE)
- Adjusts chunk size and parallel uploads accordingly
- Monitors network state changes
- Pauses uploads on network loss
- Resumes uploads when network is restored
- Adapts to network type changes during upload
### Progressive Retry
Implements intelligent retry logic:
- Specific retry of only failed chunks
- Server-client synchronization of chunk status
- Automatic recovery from partial upload failures
- Verification before finalization to catch missing chunks
## Testing
To test this implementation:
1. Ensure the server has the required endpoints implemented
2. Test with large files (100MB+) on both WiFi and mobile networks
3. Test with network transitions (WiFi -> Mobile)
4. Test interrupting uploads and resuming them
5. Compare performance with the previous implementation
## Troubleshooting
- Check logs for entries from `ChunkedUploadManager` and `UploadController`
- Look for network-related logs to identify connection issues
- Check server logs for chunk verification and missing chunk information
- Verify server-side chunk handling is working correctly
## Implementation Status
- ✅ Basic chunked upload system
- ✅ Network type detection and adaptivity
- ✅ Chunk verification and automatic retry
- ✅ Resume capability for interrupted uploads
- ✅ Parallel chunk uploading
- ✅ Network state monitoring
- ✅ Progressive backoff and retry strategy
- ✅ Enhanced progress tracking

316
docs/LARGE_UPLOAD.md Normal file
View File

@@ -0,0 +1,316 @@
# Large File Upload Analysis and Improvements
## Current Implementation Analysis
### Client Side (Android)
The SimpleLogUpload Android application is designed to upload large log files (up to 220MB) to a simple file upload server. The current implementation uses:
- OkHttp3 for HTTP requests
- A single monolithic file upload in one HTTP request
- Fixed network timeouts:
- Connection timeout: 60 seconds
- Write timeout: 180 seconds (3 minutes)
- Read timeout: 60 seconds
- Basic retry on connection failure
Behavior by network type:
- **WiFi**: Uploads work successfully
- **5G/Mobile**: Socket timeouts occur during upload (`java.net.SocketException: Socket closed`)
### Server Side (Python/Flask)
The server is a simple Flask application with:
- Standard file upload endpoint (`/upload`)
- No explicit chunked upload support
- Uses Gunicorn as WSGI server without specific timeout configurations
- No specific optimizations for large files
## Problem Root Causes
1. **Mobile Network Limitations**:
- Mobile carriers may have connection restrictions for large uploads
- 5G networks may have more aggressive connection management
- Network fluctuations are more common in mobile networks
2. **Timeout Configuration**:
- Current timeout settings may be insufficient for large files over mobile networks
- Server and client timeouts aren't aligned
3. **Single Upload Approach**:
- The entire 220MB file is uploaded in a single HTTP request
- No resume capability if connection drops
- No progress tracking or partial file recovery
4. **Missing Network Adaptability**:
- No differentiation between WiFi and mobile network settings
- No adaptive timeout or retry mechanisms
- No connection monitoring during upload
## Recommended Solutions
### 1. Implement Chunked Uploading
Develop a proper chunked upload system with these features:
```java
// Example chunked upload strategy
public class ChunkedUploadManager {
// Adjust these based on testing
private static final int DEFAULT_CHUNK_SIZE = 5 * 1024 * 1024; // 5MB chunks
private static final int MAX_RETRY_ATTEMPTS = 3;
private static final int RETRY_DELAY_MS = 1000;
// File metadata
private File file;
private int chunkSize;
private int totalChunks;
private String uploadId;
// Network state
private boolean isWifiConnection;
private boolean isPaused;
// Tracking
private int currentChunkIndex;
private Set<Integer> uploadedChunks = new HashSet<>();
public ChunkedUploadManager(File file, boolean isWifiConnection) {
this.file = file;
this.isWifiConnection = isWifiConnection;
// Adjust chunk size based on network type
this.chunkSize = isWifiConnection ? DEFAULT_CHUNK_SIZE : (DEFAULT_CHUNK_SIZE / 2);
this.totalChunks = (int) Math.ceil((double) file.length() / chunkSize);
}
// Methods for initialization, chunk upload, tracking, etc.
}
```
**Server Changes Implemented**:
```python
# New endpoint for chunk verification
@app.route('/upload/verify', methods=['POST'])
@require_client_key
def verify_chunks():
# Get upload ID from request
upload_id = request.form.get('upload_id')
# Get session metadata
session = metadata[upload_id]
# Calculate missing chunks
received_chunks = set(session['received_chunks'])
all_chunks = set(range(session['total_chunks']))
missing_chunks = list(all_chunks - received_chunks)
# Verify files exist on disk
for chunk_index in range(session['total_chunks']):
chunk_path = os.path.join(session['chunk_dir'], f'chunk_{chunk_index}')
if not os.path.exists(chunk_path) and chunk_index in received_chunks:
# Remove from received_chunks if file is missing
session['received_chunks'].remove(chunk_index)
# Return missing chunks information
return jsonify({
'total_chunks': session['total_chunks'],
'received_chunks': len(session['received_chunks']),
'missing_chunks': missing_chunks
})
```
### 2. Adaptive Network Settings
Implement network-aware settings that adjust based on connection type:
```java
private void configureNetworkSettings() {
ConnectivityManager cm = (ConnectivityManager) context.getSystemService(Context.CONNECTIVITY_SERVICE);
NetworkInfo activeNetwork = cm.getActiveNetworkInfo();
boolean isWifi = activeNetwork != null && activeNetwork.getType() == ConnectivityManager.TYPE_WIFI;
// Adjust settings based on network type
if (isWifi) {
// WiFi settings - can be more aggressive
CONNECTION_TIMEOUT = 60; // seconds
WRITE_TIMEOUT = 180; // seconds
READ_TIMEOUT = 60; // seconds
CHUNK_SIZE = 10_485_760; // 10MB
} else {
// Mobile network settings - more conservative
CONNECTION_TIMEOUT = 30; // seconds
WRITE_TIMEOUT = 90; // seconds
READ_TIMEOUT = 30; // seconds
CHUNK_SIZE = 2_097_152; // 2MB
}
}
```
### 3. Network Monitoring and Handling
Implement network state monitoring to handle:
- Network type changes during upload
- Connection losses
- Bandwidth fluctuations
```java
private void registerNetworkCallback() {
ConnectivityManager cm = (ConnectivityManager) context.getSystemService(Context.CONNECTIVITY_SERVICE);
NetworkRequest.Builder builder = new NetworkRequest.Builder();
cm.registerNetworkCallback(builder.build(), new ConnectivityManager.NetworkCallback() {
@Override
public void onAvailable(Network network) {
// Resume upload if paused
resumeUploadIfPaused();
}
@Override
public void onLost(Network network) {
// Pause upload
pauseUpload();
}
@Override
public void onCapabilitiesChanged(Network network, NetworkCapabilities capabilities) {
// Adjust strategies based on new network capabilities
boolean isUnmetered = capabilities.hasCapability(NetworkCapabilities.NET_CAPABILITY_NOT_METERED);
adjustUploadParameters(isUnmetered);
}
});
}
```
### 4. Progressive Backoff and Retry Strategy
Implement a more sophisticated retry mechanism:
```java
private void uploadChunkWithRetry(int chunkIndex) {
int retryCount = 0;
int maxRetries = 5;
long backoffMs = 1000; // Start with 1 second
while (retryCount < maxRetries) {
try {
// Attempt to upload chunk
boolean success = uploadChunk(chunkIndex);
if (success) {
return; // Success, exit retry loop
}
} catch (IOException e) {
Log.e(TAG, "Chunk upload failed: " + e.getMessage());
}
retryCount++;
if (retryCount < maxRetries) {
// Calculate backoff with exponential increase and jitter
long jitter = (long) (Math.random() * 0.3 * backoffMs);
long sleepTime = backoffMs + jitter;
Log.d(TAG, "Retrying chunk " + chunkIndex + " after " + sleepTime + "ms (attempt " + retryCount + ")");
try {
Thread.sleep(sleepTime);
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
return;
}
// Increase backoff for next iteration
backoffMs = Math.min(backoffMs * 2, 30000); // Cap at 30 seconds
}
}
// If we get here, all retries failed
Log.e(TAG, "All retry attempts failed for chunk " + chunkIndex);
reportChunkFailed(chunkIndex);
}
```
### 5. Enhanced Progress Tracking and Resumability
Implement more granular progress tracking for better user experience and resumability:
```java
public class UploadProgress {
private long totalBytes;
private long uploadedBytes;
private int totalChunks;
private Set<Integer> completedChunks;
private boolean isComplete;
private String sessionId;
// Persist progress to disk for resumability
public void saveProgress() {
SharedPreferences prefs = context.getSharedPreferences("upload_progress", Context.MODE_PRIVATE);
SharedPreferences.Editor editor = prefs.edit();
// Store progress data
editor.putString("session_" + sessionId + "_chunks", TextUtils.join(",", completedChunks));
editor.putLong("session_" + sessionId + "_bytes", uploadedBytes);
editor.putBoolean("session_" + sessionId + "_complete", isComplete);
editor.apply();
}
// Load progress for resuming uploads
public static UploadProgress loadProgress(Context context, String sessionId) {
// Implementation details
}
}
```
### 6. Server-Side Optimizations
Modify the server to better handle large file uploads:
1. Implement explicit chunked upload support
2. Configure Gunicorn timeouts appropriately:
```
gunicorn wsgi:app --bind 0.0.0.0:7777 --timeout 300 --workers 4
```
3. Add support for resumable uploads with progress tracking
4. Implement temporary file cleanup for abandoned uploads
## Implementation Priority
1. **First Phase**:
- Implement basic chunked uploading
- Add network type detection and adaptive settings
- Configure server for higher timeouts
2. **Second Phase**:
- Add upload resumability
- Implement network state monitoring
- Enhanced progress tracking
- Progressive retry mechanism
3. **Final Phase**:
- Server-side optimizations for chunked uploads
- Background upload functionality
- Bandwidth control options
## Testing Strategy
1. Test uploading the same 220MB file under various network conditions:
- Stable WiFi
- Congested WiFi
- 5G with strong signal
- 5G with weak signal
- 4G fallback
- Network transitions (WiFi → Mobile)
2. Measure and compare:
- Success rate of uploads
- Total upload time
- Battery consumption
- Data usage efficiency
## Conclusion
The socket timeout issues on 5G networks stem from trying to upload large files in a single HTTP request without adapting to the characteristics of mobile networks. By implementing chunked uploads with resume capability, network-aware configurations, and robust error handling, we can create a more reliable upload solution for all network types.
The most critical issue to address is the lack of chunking, which leaves the application vulnerable to connection interruptions, particularly on mobile networks where connections may be less stable than WiFi.

96
docs/RELEASE_PROCESS.md Normal file
View File

@@ -0,0 +1,96 @@
# Release Process for SimpleLogUpload
This document describes the process of creating new releases for the SimpleLogUpload Android application using GitHub Actions.
## Versioning Scheme
We use a `vYY.MM.PATCH` versioning format:
- `YY`: Year (last two digits, e.g., 25 for 2025)
- `MM`: Month (two digits, e.g., 04 for April)
- `PATCH`: Incremental patch number within the month (starting at 1)
Example: `v25.04.1` represents the first release in April 2025.
## Prerequisites
Before you can use the automated release process, you need to set up the signing configuration:
1. **Create a keystore** (if you don't already have one):
```bash
keytool -genkey -v -keystore simplelogupload.keystore -alias upload_key -keyalg RSA -keysize 2048 -validity 10000
```
2. **Encode your keystore file**:
```bash
# Make the script executable
chmod +x scripts/prepare_signing_key.sh
# Run the script with your keystore file
./scripts/prepare_signing_key.sh /path/to/your/simplelogupload.keystore
```
3. **Add the required secrets to your GitHub repository**:
- `SIGNING_KEY`: The Base64-encoded keystore file (output from step 2)
- `KEY_ALIAS`: The alias of your key (e.g., `upload_key`)
- `KEY_STORE_PASSWORD`: The password for your keystore
- `KEY_PASSWORD`: The password for your key
Add these at: https://github.com/YOUR-USERNAME/SimpleLogUpload/settings/secrets/actions
## Creating a New Release
1. **Update your code**:
- Make all necessary code changes
- Update version numbers in app/build.gradle if needed
- Commit and push your changes to the main branch
2. **Create and push a tag**:
```bash
# Get the latest changes
git pull origin main
# Create a new tag
git tag v25.04.1
# Push the tag to GitHub
git push origin v25.04.1
```
3. **Monitor the GitHub Actions workflow**:
- Go to the "Actions" tab in your repository
- You should see a workflow named "Android Release Build" running
- When complete, it will create a new release with the APK attached
4. **Verify the release**:
- Check the "Releases" section of your repository
- Ensure the APK file is attached
- Download and test the APK on a device
## Troubleshooting
If the automatic release process fails:
1. **Check the GitHub Actions logs**:
- Go to the "Actions" tab and click on the failed workflow
- Examine the logs for errors
2. **Common issues**:
- Missing GitHub secrets
- Incorrect keystore information
- Build errors in the codebase
3. **Manual release**:
If needed, you can build and release manually:
```bash
# Build the release APK
./gradlew assembleRelease
# Sign the APK using your keystore
# (Use apksigner from the Android SDK)
```
## Post-Release Steps
1. **Announce the release** to users or stakeholders
2. **Monitor for issues** after the release
3. **Plan the next release** and begin work on new features