Use hash func to boost file creation and lookup #79
Conversation
|
How can you determine which hash function is the most suitable? |
hash.c
Outdated
| uint64_t fnv1a_64(const char *str) | ||
| { | ||
| uint64_t h = 0xcbf29ce484222325ULL; | ||
| while (*str) { | ||
| h ^= (unsigned char) (*str++); | ||
| h *= 0x100000001b3ULL; | ||
| } | ||
| return h; | ||
| } |
There was a problem hiding this comment.
Can you re-use the existing hash functions in Linux?
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
1 issue found across 8 files
Prompt for AI agents (all 1 issues)
Understand the root cause of the following 1 issues and fix them.
<file name="simplefs.h">
<violation number="1" location="simplefs.h:122">
Declaring `simplefs_file_vm_ops` as `static` in the header defines a zeroed vm_ops instance in every translation unit instead of referencing a single populated table. Please make this an extern declaration so consumers can use the real ops struct.</violation>
</file>
Since this is your first cubic review, here's how it works:
- cubic automatically reviews your code and comments on bugs and improvements
- Teach cubic by replying to its comments. cubic learns from your replies and gets better over time
- Ask questions if you need clarification on any suggestion
React with 👍 or 👎 to teach cubic. Mention @cubic-dev-ai to give feedback, ask questions, or re-run the review.
simplefs.h
Outdated
| extern const struct file_operations simplefs_file_ops; | ||
| extern const struct file_operations simplefs_dir_ops; | ||
| extern const struct address_space_operations simplefs_aops; | ||
| static const struct vm_operations_struct simplefs_file_vm_ops; |
There was a problem hiding this comment.
Declaring simplefs_file_vm_ops as static in the header defines a zeroed vm_ops instance in every translation unit instead of referencing a single populated table. Please make this an extern declaration so consumers can use the real ops struct.
Prompt for AI agents
Address the following comment on simplefs.h at line 122:
<comment>Declaring `simplefs_file_vm_ops` as `static` in the header defines a zeroed vm_ops instance in every translation unit instead of referencing a single populated table. Please make this an extern declaration so consumers can use the real ops struct.</comment>
<file context>
@@ -119,6 +119,7 @@ struct dentry *simplefs_mount(struct file_system_type *fs_type,
extern const struct file_operations simplefs_file_ops;
extern const struct file_operations simplefs_dir_ops;
extern const struct address_space_operations simplefs_aops;
+static const struct vm_operations_struct simplefs_file_vm_ops;
/* extent functions */
</file context>
| static const struct vm_operations_struct simplefs_file_vm_ops; | |
| extern const struct vm_operations_struct simplefs_file_vm_ops; |
visitorckw
left a comment
visitorckw
left a comment
There was a problem hiding this comment.
I saw that your PR description includes some performance benchmarks, but the commit message lacks any performance numbers to support your improvements. Please improve the commit message.
| return 0; | ||
|
|
||
| sbi->nr_free_blocks -= len; | ||
| struct buffer_head *bh; |
There was a problem hiding this comment.
Please refrain from making extraneous or unrelated changes. Even if such changes are necessary, please split them into separate patches and explain in detail why they are needed in the commit message.
| if (IS_ERR(bdev_file)) { | ||
| printk(KERN_ERR | ||
| "failed to open journal device unknown-block(%u,%u) %ld\n", | ||
| pr_err("failed to open journal device unknown-block(%u,%u) %ld\n", |
There was a problem hiding this comment.
Why do we need to change this to use pr_err()?
This also appears to be an unrelated change.
| /* Iterate over blocks in one extent */ | ||
| for (bi = 0; bi < eblock->extents[ei].ee_len && remained_nr_files; | ||
| for (bi = 0; | ||
| bi < eblock->extents[ei].ee_len && remained_nr_files && ei_nr; |
There was a problem hiding this comment.
IIUC, this optimizes efficiency by using nr_files to break out of the loop early, but it is unrelated to "hash lookup". It should be a separate patch. Please follow the "One Logical Change Per Patch" principle and split your changes.
RoyWFHuang
force-pushed
the
feature/op_perf
branch
from
b928f62 to
308bb4c
Compare
.config
Outdated
| # | ||
| # Automatically generated file; DO NOT EDIT. | ||
| # Configuration | ||
| # | ||
|
|
||
| # | ||
| # SimpleFS Configuration | ||
| # | ||
| # CONFIG_SIMPLEFS_HASH_HASH64 is not set | ||
| CONFIG_SIMPLEFS_HASH_FNV1A=y | ||
| # CONFIG_SIMPLEFS_HASH_FULLNAME is not set |
RoyWFHuang
force-pushed
the
feature/op_perf
branch
from
308bb4c to
1645d00
Compare
I’m not sure if "fnv" is the most suitable, but index in SimpleFS is relatively small, using a more complex algorithm might not provide significant benefits. I think fnv is a reasonable balance between simplicity and performance. |
RoyWFHuang
force-pushed
the
feature/op_perf
branch
from
1645d00 to
ca74c03
Compare
Introduce a hash-based mechanism to speed up file creation and lookup operations. The hash function enables faster access to extent and logical block extent index, improving overall filesystem performance. hash_code = file_hash(file_name); extent index = hash_code / SIMPLEFS_MAX_BLOCKS_PER_EXTENT block index = hash_code % SIMPLEFS_MAX_BLOCKS_PER_EXTENT;
RoyWFHuang
force-pushed
the
feature/op_perf
branch
from
ca74c03 to
51e0478
Compare
visitorckw
left a comment
visitorckw
left a comment
There was a problem hiding this comment.
You ignored many of my comments without making any changes or providing any replies. You still retained many irrelevant changes, making the review difficult. Additionally, a single-line commit message saying only "optimize the file search process" is way too vague. Please improve the git commit message.
|
|
||
|
|
||
| #if defined(CONFIG_SIMPLEFS_HASH_HASH64) | ||
| static uint64_t _hash64(const char *name, size_t len) |
There was a problem hiding this comment.
Is it really necessary for us to re-implement our own hash function? Upstream Linux already has many hash implementations, like full_name_hash(). Could we use those here instead?
3 participants
Previously, SimpleFS used a sequential insertion method to create files, which worked efficiently when the filesystem contained only a small number of files.
However, in real-world use cases, filesystems often manage a large number of files, making sequential search and insertion inefficient.
Inspired by Ext4’s hash-based directory indexing, this change adopts a hash function to accelerate file indexing and improve scalability.
Change:
Implemented hash-based file index lookup
Improved scalability for large directory structures
hash_code = file_hash(file_name);
extent index = hash_code / SIMPLEFS_MAX_BLOCKS_PER_EXTENT
block index = hash_code % SIMPLEFS_MAX_BLOCKS_PER_EXTENT;
Performance test
hash table:
legacy:
hash table:
legacy:
Use perf stat ls -la to measure the query time for each file and sum up all elapsed times to calculate the total lookup cost.
hash table:
min time: 0.00163 s
max time: 0.07574 s
avg time: 0.00247343 s
tot time: 75.686920 s
legacy:
min time: 0.00171 s
max time: 0.03799 s
avg time: 0.00423332 s
tot time: 129.539510 s
Summary by cubic
Switched SimpleFS to configurable hash-based directory indexing (FNV‑1a in tests) to speed up file creation and lookup by mapping filenames to extent/block slots. On 30.6k files: create ~33% faster, delete ~12% faster, lookup ~41% faster.
New Features
Refactors
Written for commit 51e0478. Summary will update automatically on new commits.