> ## Documentation Index
> Fetch the complete documentation index at: https://docs.wearer.xyz/llms.txt
> Use this file to discover all available pages before exploring further.
# Development Guide
> Contribute to or extend Transfer Learning
# Development Guide
This guide provides information for developers who want to contribute to or extend Transfer Learning.
## Project Structure
The Transfer Learning project follows a modular architecture with clear separation of concerns:
```
transfer-learning/
├── data/ # Data storage directory
├── docs/ # Documentation
├── logs/ # Log files
├── metrics/ # Metrics collection
├── src/ # Source code
│ └── transfer_learning/
│ ├── core/ # Core processing modules
│ │ ├── video_processor.py
│ │ ├── audio_transcriber.py
│ │ ├── content_analyzer.py
│ │ └── frame_extractor.py
│ ├── guide/ # Guide generation
│ │ └── generator.py
│ ├── models/ # AI model interfaces
│ │ ├── openai.py
│ │ └── whisper.py
│ ├── monitoring/ # Monitoring and metrics
│ │ ├── logger.py
│ │ └── metrics.py
│ ├── utils/ # Utility functions
│ │ ├── downloader.py
│ │ ├── validation.py
│ │ └── path.py
│ ├── cli.py # CLI implementation
│ └── config.py # Configuration
├── tests/ # Test suite
├── .env # Environment variables
├── pyproject.toml # Project configuration
└── setup.py # Package setup
```
## Development Setup
### Installation Scripts
Transfer Learning provides automated installation scripts in the `scripts` directory:
* `install.sh` - For Unix-based systems (Linux, macOS)
* `install.bat` - For Windows systems
These scripts handle the complete setup process:
```bash theme={null}
# On Unix-based systems
chmod +x scripts/install.sh
./scripts/install.sh
# On Windows
scripts\install.bat
```
The installation scripts perform the following tasks:
* Verify Python version compatibility (3.9+)
* Create and activate a virtual environment
* Install UV package manager
* Install project dependencies
* Set up required directories (data, logs, metrics, cache)
* Create a default `.env` configuration file
* Install development dependencies (optional)
To modify or extend the installation process:
1. Edit `scripts/install.sh` or `scripts/install.bat`
2. Update the directory creation in the `create_directories` function
3. Modify the default configuration in the `create_env_file` function
4. Add new dependencies or setup steps as needed
```bash theme={null}
git clone https://github.com/Wearer-Lab/transfer-learning.git
cd transfer-learning
```
```bash theme={null}
python -m venv .venv
source .venv/bin/activate # On Windows, use `.venv\Scripts\activate`
```
```bash theme={null}
pip install uv
uv pip install -e ".[dev]"
```
Transfer Learning uses UV as the package manager for faster, more reliable dependency management. UV provides significant performance improvements over traditional pip, especially for large dependency trees.
Create a `.env` file in the project root directory:
```bash theme={null}
OPENAI_API_KEY=your-api-key-here
ENABLE_MONITORING=true
LOG_LEVEL=DEBUG
```
## Development Workflow
Run the test suite to ensure your changes don't break existing functionality:
```bash theme={null}
pytest
```
Run tests with coverage report:
```bash theme={null}
pytest --cov=transfer_learning
```
Check code style with Ruff:
```bash theme={null}
ruff check .
```
Format code with Black:
```bash theme={null}
black src tests
```
Run the CLI in development mode:
```bash theme={null}
python -m transfer_learning.cli process-video path/to/video.mp4
```
Or use the installed entry point:
```bash theme={null}
transfer-learning process-video path/to/video.mp4
```
You can also use the convenient `tl` command alias:
```bash theme={null}
tl process-video path/to/video.mp4
```
The `tl` command is available through shell scripts in the `scripts` directory:
* `scripts/tl` for Unix-like systems (macOS/Linux)
* `scripts/tl.bat` for Windows
## Extending Transfer Learning
### Adding a New Command
To add a new command to the CLI:
1. Open `src/transfer_learning/cli.py`
2. Add a new function with the `@app.command()` decorator:
```python theme={null}
@app.command()
def my_new_command(
arg1: str = typer.Argument(..., help="Description of arg1"),
option1: str = typer.Option("default", help="Description of option1"),
):
"""Description of my new command."""
# Command implementation
console.print(f"Running my new command with {arg1} and {option1}")
```
### Adding a New Module
To add a new module:
1. Create a new Python file in the appropriate directory
2. Implement your module functionality
3. Import and use it in the CLI or other modules
Example of a new utility module:
```python theme={null}
# src/transfer_learning/utils/my_utility.py
"""My utility module."""
def my_utility_function(param1: str, param2: int = 10) -> str:
"""
Description of my utility function.
Args:
param1: Description of param1
param2: Description of param2
Returns:
Description of return value
"""
return f"{param1} processed with {param2}"
```
### Enhancing Monitoring
To add new metrics to the monitoring system:
1. Open `src/transfer_learning/monitoring/metrics.py`
2. Add new fields to the `ProcessingMetrics` dataclass
3. Update the `to_dict` method to include the new fields
4. Use the new metrics in your code
## Documentation
Transfer Learning uses Mintlify for documentation. To update the documentation:
1. Edit the MDX files in the `docs` directory
2. Preview the documentation locally:
```bash theme={null}
npx mintlify dev
```
3. Submit a pull request with your documentation changes
## Contribution Guidelines
* Follow PEP 8 guidelines
* Use type hints for all functions
* Write docstrings for all modules, classes, and functions
* Keep functions small and focused
* Write tests for all new functionality
* Ensure all tests pass before submitting a PR
* Aim for high test coverage
* Include both unit and integration tests
* Update documentation for all new features
* Include examples in docstrings
* Keep the README and docs in sync
* Document configuration options
* Create a feature branch for your changes
* Keep PRs focused on a single feature or fix
* Include a clear description of changes
* Reference any related issues
## Development Checklist
Use this checklist when developing new features:
* [ ] Implement the feature
* [ ] Write tests
* [ ] Update documentation
* [ ] Check code style
* [ ] Run the test suite
* [ ] Update the README if necessary
* [ ] Submit a pull request
## Monitoring and Logging
Transfer Learning includes a comprehensive monitoring and logging system:
The `MetricsTracker` class in `src/transfer_learning/monitoring/metrics.py` provides metrics collection:
```python theme={null}
from transfer_learning.monitoring.metrics import MetricsTracker, Timer
# Initialize metrics tracker
metrics = MetricsTracker()
# Start tracking metrics for a video
metrics.start_processing("video_id")
# Use timer context manager to track duration
with Timer(metrics, "frame_analysis_duration"):
# Process frames
analyze_frames(frames)
# Update metrics manually
metrics.update_metrics(
frame_count=100,
processed_frames=100,
error_count=0
)
# End tracking and save metrics
metrics.end_processing()
```
The `setup_logger` function in `src/transfer_learning/monitoring/logger.py` provides logging:
```python theme={null}
from transfer_learning.monitoring.logger import setup_logger
# Set up logger
logger = setup_logger(
name=__name__,
log_file="logs/my_module.log",
level="DEBUG"
)
# Use logger
logger.info("Processing started")
logger.debug("Detailed debug information")
logger.warning("Warning message")
logger.error("Error message")
```
## Performance Optimization
When developing performance-critical components:
1. Use asynchronous processing for I/O-bound operations
2. Implement batching for API calls
3. Use caching to avoid redundant processing
4. Monitor memory usage and optimize as needed
Example of asynchronous processing:
```python theme={null}
import asyncio
from typing import List, Dict, Any
async def process_batch(batch: List[str]) -> List[Dict[str, Any]]:
"""Process a batch of items asynchronously."""
tasks = [process_item(item) for item in batch]
return await asyncio.gather(*tasks)
async def process_item(item: str) -> Dict[str, Any]:
"""Process a single item."""
# Implement processing logic
return {"item": item, "result": "processed"}
async def main():
"""Main processing function."""
batches = [["item1", "item2"], ["item3", "item4"]]
results = []
for batch in batches:
batch_results = await process_batch(batch)
results.extend(batch_results)
return results
```