[NimaSarajpoor]

[WIP] This PR provides a notebook that explains the math behind overlap-add in convolution.

[gitnotebooks]

Found 1 changed notebook. Review the changes at https://app.gitnotebooks.com/stumpy-dev/sliding_dot_product/pull/43

[NimaSarajpoor]

@seanlaw
This PR adds a notebook that provides the math behind linear convolution, circular convolution, and overlap-add method. Before finishing the overlap-add method, I wanted to see if you can review the notebook and share your comments. Once I address those, I can resume the work and complete the "overlap-add" part.

[seanlaw]

Thanks @NimaSarajpoor. I will take a look

[seanlaw]

I haven't read through the "Option I" in detail but I do like how you're building up the story! It almost feels like we might want to add an "overarching goals"/"questions to be answered" section to the top of the document and explain why the goals/questions exist or why they are important (i.e., how are they related to the software that we are writing?) Then, when we address them, we refer back to the goals/questions and say "and now you see how this is related/solved".

At the end of the day, the document isn't only about the convolution concept. It is also about, say, "where does the 'range' come from and how/why does it change when we use a different approach?". I tend to notice that when a "smart" person has spent a long time standing knee deep in the space, everything about the topic seems obvious to them and the "connections" are clear. However, us "dumb" people need you to "state the obvious" in writing with excruciating detail (i.e., don't skip steps!) and to declare those "connections" out loud and then we will be forever grateful. ☺️

[NimaSarajpoor]

@seanlaw

to add an "overarching goals"/"questions to be answered" section to the top of the document and explain why the goals/questions exist or why they are important (i.e., how are they related to the software that we are writing?)

Sounds good! I will add something in the beginning to help readers understand the objectives and why they matter.

At the end of the day, the document isn't only about the convolution concept. It is also about, say, "where does the 'range' come from and how/why does it change when we use a different approach?"

"If you can't explain it simply, you don't understand it well enough" So, whenever something isn’t clearly elaborated, I take it as a reminder to make sure that not only is the communication clear, but that I actually understand the concept well enough to explain it clearly 😅

[NimaSarajpoor]

@seanlaw
I've made some changes but I need to get back to it after a couple of days to assess the flow with fresh eyes. Will let you know.

[NimaSarajpoor]

@seanlaw
This is ready for your review. I tried to stick to the math as much as possible and keep the steps easy to follow. I added an objectives section right at the start to give readers an idea of what they can expect to learn. Still, I need your fresh eyes to help me detect any remaining gaps.

As a side, I think the efficiency of the overlap-add method requires its own discussion. Still, I tried to add something at the end of the notebook to provide some initial ideas for later.

[seanlaw]

@NimaSarajpoor I am out of town this week but will try to find some time to review next week

[seanlaw]

@NimaSarajpoor I skimmed the document but I really, really like how thoroughly you've stepped through everything. The story is very clear (though, I haven't studied the math in detail). Eventually, I think you should be able reference different sections of this document from within your code and it will be a wonderful complementary document!

[NimaSarajpoor]

Update:

• Comments, including the equation number issue, were addressed.
• Added more tests, and added references to their corresponding sections

@seanlaw
I added a new section "Code" at the end of notebook and put all codes (functions / tests) there. I revised this new section several times to increase its clarity. Can you please just focus on the "Code" part and let me know if you feel you can follow the story it tries to tell?

[seanlaw]

I added a new section "Code" at the end of notebook and put all codes (functions / tests) there. I revised this new section several times to increase its clarity. Can you please just focus on the "Code" part and let me know if you feel you can follow the story it tries to tell?

This is actually quite lovely to have! Thanks for including it

[NimaSarajpoor]

Thanks!!

---

For me, there are two key takeaways from this notebook:

(1) A full linear convolution between two 1D arrays X and Y can be computed via circular convolution between those arrays when each is padded with zeros till their length becomes N=len(X) + len(Y) - 1. And if the valid portion is of interest only, then N can be set to max(len(X), len(Y)).

(2) Overlap-add method is like a divide-and-conquer approach. The full linear convolution is computed between one array and chunks of the other array, and then their outputs are combined.

In a GIF provided in the other notebook, I showed circular convolution between an array and each chunk of another array. However, I think it is easier to visualize it using flip-and-slide approach of full linear convolution. I will add a GIF to this notebook to demonstrate that.

[seanlaw]

(1) A full linear convolution between two 1D arrays X and Y can be computed via circular convolution between those arrays when each is padded with zeros till their length becomes N=len(X) + len(Y) - 1. And if the valid portion is of interest only, then N can be set to max(len(X), len(Y)).

(2) Overlap-add method is like a divide-and-conquer approach. The full linear convolution is computed between one array and chunks of the other array, and then their outputs are combined.

It might be good to reiterate this in a Conclusion/relevant section

[NimaSarajpoor]

It might be good to reiterate this in a Conclusion/relevant section

I added the conclusion section to highlight the two key takeaways. In addition, I added an animation for the overlap-add part.

[NimaSarajpoor]

@seanlaw
Btw, I wanted to bring your attention to the function overlap_add_convolve that I added to the notebook and I think it is a good start for overlap-add.

A simplified version of it is provided below:

import numpy as np


def naive_overlap_add(Q, T, T_chunks_range, mode='full'):
    """   
    Compute full or valid convolution between Q and T
    by computing FULL convolution between Q and chunks of T.

    T_chunks_range is a two column array where each row consists of
    a start and (exclusive) stop index pair. The first column contains
    the start indices and the second column contains the stop indices.
    No two rows have overlap, and the union of all rows
    should be the `range(0, len(T))`
    """
    m = len(Q)
    n = len(T)
    
    # compute full convolution between Q and T
    out = np.zeros(n + m - 1, dtype=np.float64)      
    for (start, stop) in T_chunks_range:
        tmp =  np.convolve(Q, T[start:stop], mode='full')
        out[start: start + len(tmp)] += tmp
    
    # apply mode       
    if mode == 'valid':
        return out[m-1:n]

    return out


if __name__ == '__main__':
    T_chunks_range = np.array([
        [0, 1],
        [1, 10],
        [10, 13],
    ])

    Q = np.random.rand(5)
    T = np.random.rand(13)

    ref = np.convolve(Q, T)
    comp = naive_overlap_add(Q, T, T_chunks_range)

    np.testing.assert_allclose(ref, comp)

And, to improve performance, we need to cover the following items:
(1) Replace np.convolve with circular convolution using (R)FFT / I(R)FFT
(2) Use chunks with same size to reduce the computation
(3) Find a good size for the chunks
(4) Vectorize the operations

Will add this to PR #35