Skip to content

Conversation

@ItzNotABug
Copy link
Member

Adds the CSV source handler.

@ItzNotABug ItzNotABug requested a review from abnegate April 9, 2025 12:55
@ItzNotABug ItzNotABug self-assigned this Apr 9, 2025
@ItzNotABug
Copy link
Member Author

@abnegate here's a quick benchmark -

  1. 3 documents in 0.0039 sec.
  2. 1,00,000 documents in 5.9281 sec
  3. 10,00,000 documents in 65.5917 sec
  4. 50,00,000 documents in 318.6609 sec


The controller froze on a CSV with 1,00,00,000 documents but I'll assume its way more than the last set.


@abnegate
If the insertion order doesn't matter, can we not use coroutines here so its not really sequential?

@ItzNotABug ItzNotABug requested a review from abnegate April 12, 2025 11:31
@ItzNotABug ItzNotABug requested a review from abnegate April 15, 2025 05:26
Copy link
Member

@abnegate abnegate left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add some tests as well

@ItzNotABug ItzNotABug requested a review from eldadfux April 16, 2025 07:51
@abnegate abnegate merged commit 545705e into main Apr 16, 2025
3 checks passed
@abnegate abnegate deleted the feat-csv branch April 16, 2025 07:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants