Add utilities for character level text generation #16
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces a new
textmodule and refactors the character-level LSTM example to use improved text utilities for vocabulary management, embedding, and sampling. It also adds a dedicated example demonstrating the use of these text utilities. The changes modernize the text generation pipeline, making it both simpler and more flexible.Text utilities integration and example:
textmodule to the library, providingTextVocabulary,CharacterEmbedding, and advanced sampling functions, and re-exported these for easy access. [1] [2]text_utils_example.rsthat demonstrates the usage of vocabulary encoding/decoding, character embeddings, LSTM/linear pipeline, and various sampling strategies.Refactor of character-level LSTM example:
CharacterLSTMintext_generation_advanced.rsto use the newTextVocabularyandCharacterEmbedding, replacing manual character-index mapping and custom embedding logic.