Implement Memory-Efficient Masked SVD for Sparse Matrices

[ianfd]

Description

Develop an optimized SVD calculation approach for CSR matrices that operates on a subset of features without requiring matrix cloning.

Objectives

• Create a masked SVD implementation that minimizes memory overhead
• Optimize for sparse matrix operations on feature subsets
• Reduce memory usage for large-scale single-cell analyses

Key Components to Implement

Masked Matrix Views

• Implement efficient masked view functionality for CSR matrices
• Create adapters that present a logical subset without data duplication
• Support for both row (observation) and column (feature) masking

Sparse SVD Algorithms

• Adapt existing SVD algorithms to work with masked views
• Implement specialized iterative methods (e.g., Arnoldi/Lanczos)
• Add randomized SVD support for further efficiency

Memory Optimization

• Implement in-place operations where possible
• Add streaming computation options for extremely large matrices
• Create memory usage monitoring and optimization tools

Performance Features

• Add parallel processing capabilities
• Implement early stopping criteria for iterative methods
• Create benchmarking utilities for performance evaluation

Integration Points

• Must work with existing CSR/CSC matrix implementations
• Should support existing SVD interfaces for compatibility
• Consider integration with PCA implementation

Technical Notes

• Focus first on column masking for HVG selection use cases
• Consider using iterative methods that don't require materialization of large matrices
• Balance memory efficiency with computational performance