Development Guide

This guide provides information for developers who want to contribute to or extend Transfer Learning.

Project Structure

The Transfer Learning project follows a modular architecture with clear separation of concerns:

transfer-learning/
├── data/               # Data storage directory
├── docs/               # Documentation
├── logs/               # Log files
├── metrics/            # Metrics collection
├── src/                # Source code
│   └── transfer_learning/
│       ├── core/       # Core processing modules
│       │   ├── video_processor.py
│       │   ├── audio_transcriber.py
│       │   ├── content_analyzer.py
│       │   └── frame_extractor.py
│       ├── guide/      # Guide generation
│       │   └── generator.py
│       ├── models/     # AI model interfaces
│       │   ├── openai.py
│       │   └── whisper.py
│       ├── monitoring/ # Monitoring and metrics
│       │   ├── logger.py
│       │   └── metrics.py
│       ├── utils/      # Utility functions
│       │   ├── downloader.py
│       │   ├── validation.py
│       │   └── path.py
│       ├── cli.py      # CLI implementation
│       └── config.py   # Configuration
├── tests/              # Test suite
├── .env                # Environment variables
├── pyproject.toml      # Project configuration
└── setup.py            # Package setup

Development Setup

Installation Scripts

Transfer Learning provides automated installation scripts in the scripts directory:

install.sh - For Unix-based systems (Linux, macOS)
install.bat - For Windows systems

These scripts handle the complete setup process:

# On Unix-based systems
chmod +x scripts/install.sh
./scripts/install.sh

# On Windows
scripts\install.bat

The installation scripts perform the following tasks:

Verify Python version compatibility (3.9+)
Create and activate a virtual environment
Install UV package manager
Install project dependencies
Set up required directories (data, logs, metrics, cache)
Create a default .env configuration file
Install development dependencies (optional)

To modify or extend the installation process:

Edit scripts/install.sh or scripts/install.bat
Update the directory creation in the create_directories function
Modify the default configuration in the create_env_file function
Add new dependencies or setup steps as needed

Clone the Repository

git clone https://github.com/Wearer-Lab/transfer-learning.git
cd transfer-learning

Set Up Virtual Environment

python -m venv .venv
source .venv/bin/activate  # On Windows, use `.venv\Scripts\activate`

Install Development Dependencies

pip install uv
uv pip install -e ".[dev]"

Transfer Learning uses UV as the package manager for faster, more reliable dependency management. UV provides significant performance improvements over traditional pip, especially for large dependency trees.

Set Up Environment Variables

Create a .env file in the project root directory:

OPENAI_API_KEY=your-api-key-here
ENABLE_MONITORING=true
LOG_LEVEL=DEBUG

Development Workflow

Running Tests

Run the test suite to ensure your changes don’t break existing functionality:

pytest

Run tests with coverage report:

pytest --cov=transfer_learning

Code Style

Check code style with Ruff:

ruff check .

Format code with Black:

black src tests

Running the CLI

Run the CLI in development mode:

python -m transfer_learning.cli process-video path/to/video.mp4

Or use the installed entry point:

transfer-learning process-video path/to/video.mp4

You can also use the convenient tl command alias:

tl process-video path/to/video.mp4

The tl command is available through shell scripts in the scripts directory:

scripts/tl for Unix-like systems (macOS/Linux)
scripts/tl.bat for Windows

Extending Transfer Learning

Adding a New Command

To add a new command to the CLI:

Open src/transfer_learning/cli.py
Add a new function with the @app.command() decorator:

@app.command()
def my_new_command(
    arg1: str = typer.Argument(..., help="Description of arg1"),
    option1: str = typer.Option("default", help="Description of option1"),
):
    """Description of my new command."""
    # Command implementation
    console.print(f"Running my new command with {arg1} and {option1}")

Adding a New Module

To add a new module:

Create a new Python file in the appropriate directory
Implement your module functionality
Import and use it in the CLI or other modules

Example of a new utility module:

# src/transfer_learning/utils/my_utility.py
"""My utility module."""

def my_utility_function(param1: str, param2: int = 10) -> str:
    """
    Description of my utility function.
    
    Args:
        param1: Description of param1
        param2: Description of param2
        
    Returns:
        Description of return value
    """
    return f"{param1} processed with {param2}"

Enhancing Monitoring

To add new metrics to the monitoring system:

Open src/transfer_learning/monitoring/metrics.py
Add new fields to the ProcessingMetrics dataclass
Update the to_dict method to include the new fields
Use the new metrics in your code

Documentation

Transfer Learning uses Mintlify for documentation. To update the documentation:

Edit the MDX files in the docs directory
Preview the documentation locally:

npx mintlify dev

Submit a pull request with your documentation changes

Contribution Guidelines

Code Style

Follow PEP 8 guidelines
Use type hints for all functions
Write docstrings for all modules, classes, and functions
Keep functions small and focused

Testing

Write tests for all new functionality
Ensure all tests pass before submitting a PR
Aim for high test coverage
Include both unit and integration tests

Documentation

Update documentation for all new features
Include examples in docstrings
Keep the README and docs in sync
Document configuration options

Pull Requests

Create a feature branch for your changes
Keep PRs focused on a single feature or fix
Include a clear description of changes
Reference any related issues

Development Checklist

Use this checklist when developing new features:

Monitoring and Logging

Transfer Learning includes a comprehensive monitoring and logging system:

Metrics
Logging

The MetricsTracker class in src/transfer_learning/monitoring/metrics.py provides metrics collection:

from transfer_learning.monitoring.metrics import MetricsTracker, Timer

# Initialize metrics tracker
metrics = MetricsTracker()

# Start tracking metrics for a video
metrics.start_processing("video_id")

# Use timer context manager to track duration
with Timer(metrics, "frame_analysis_duration"):
    # Process frames
    analyze_frames(frames)

# Update metrics manually
metrics.update_metrics(
    frame_count=100,
    processed_frames=100,
    error_count=0
)

# End tracking and save metrics
metrics.end_processing()

The setup_logger function in src/transfer_learning/monitoring/logger.py provides logging:

from transfer_learning.monitoring.logger import setup_logger

# Set up logger
logger = setup_logger(
    name=__name__,
    log_file="logs/my_module.log",
    level="DEBUG"
)

# Use logger
logger.info("Processing started")
logger.debug("Detailed debug information")
logger.warning("Warning message")
logger.error("Error message")

Performance Optimization

When developing performance-critical components:

Use asynchronous processing for I/O-bound operations
Implement batching for API calls
Use caching to avoid redundant processing
Monitor memory usage and optimize as needed

Example of asynchronous processing:

import asyncio
from typing import List, Dict, Any

async def process_batch(batch: List[str]) -> List[Dict[str, Any]]:
    """Process a batch of items asynchronously."""
    tasks = [process_item(item) for item in batch]
    return await asyncio.gather(*tasks)

async def process_item(item: str) -> Dict[str, Any]:
    """Process a single item."""
    # Implement processing logic
    return {"item": item, "result": "processed"}

async def main():
    """Main processing function."""
    batches = [["item1", "item2"], ["item3", "item4"]]
    results = []
    
    for batch in batches:
        batch_results = await process_batch(batch)
        results.extend(batch_results)
    
    return results

Get Started

CLI Commands

Advanced Usage

Development Guide

Development Guide

Project Structure

Development Setup

Installation Scripts

Development Workflow

Extending Transfer Learning

Adding a New Command

Adding a New Module

Enhancing Monitoring

Documentation

Contribution Guidelines

Code Style

Testing

Documentation

Pull Requests

Development Checklist

Monitoring and Logging

Performance Optimization

​Development Guide

​Project Structure

​Development Setup

​Installation Scripts

​Development Workflow

​Extending Transfer Learning

​Adding a New Command

​Adding a New Module

​Enhancing Monitoring

​Documentation

​Contribution Guidelines

Code Style

Testing

Documentation

Pull Requests

​Development Checklist

​Monitoring and Logging

​Performance Optimization

Development Guide

Project Structure

Development Setup

Installation Scripts

Development Workflow

Extending Transfer Learning

Adding a New Command

Adding a New Module

Enhancing Monitoring

Documentation

Contribution Guidelines

Development Checklist

Monitoring and Logging

Performance Optimization