Skip to content

Commit 712f691

Browse files
committed
readme added.
1 parent dc8732e commit 712f691

File tree

1 file changed

+57
-0
lines changed

1 file changed

+57
-0
lines changed

README.md

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
# Rust external sort
2+
3+
`ext-sort` is a rust external sort algorithm implementation.
4+
5+
External sort algorithm implementation. External sorting is a class of sorting algorithms
6+
that can handle massive amounts of data. External sorting is required when the data being
7+
sorted do not fit into the main memory (RAM) of a computer and instead must be resided in
8+
slower external memory, usually a hard disk drive. Sorting is achieved in two passes.
9+
During the first pass it sorts chunks of data that each fit in RAM, during the second pass
10+
it merges the sorted chunks together.
11+
For more information see https://en.wikipedia.org/wiki/External_sorting.
12+
13+
## Features
14+
15+
* **Data agnostic:**
16+
`ext-sort` support all data types that that implement `serde` serialization/deserialization.
17+
* **Serialization format agnostic:**
18+
`ext-sort` use `MessagePack` serialization format by default, but it can be easily substituted by your custom one
19+
if `MessagePack` serialization/deserialization performance is not sufficient for your task.
20+
* **Multithreading support:**
21+
`ext-sort` support multithreading, which means data is sorted in multiple threads utilizing maximum CPU resources
22+
and reducing sorting time.
23+
24+
# Basic example
25+
26+
``` rust
27+
use std::fs;
28+
use std::io::{self, prelude::*};
29+
use std::path;
30+
31+
use bytesize::MB;
32+
use env_logger;
33+
use log;
34+
35+
use ext_sort::buffer::mem::MemoryLimitedBufferBuilder;
36+
use ext_sort::{ExternalSorter, ExternalSorterBuilder};
37+
38+
fn main() {
39+
env_logger::Builder::new().filter_level(log::LevelFilter::Debug).init();
40+
41+
let input_reader = io::BufReader::new(fs::File::open("input.txt").unwrap());
42+
let mut output_writer = io::BufWriter::new(fs::File::create("output.txt").unwrap());
43+
44+
let sorter: ExternalSorter<String, MemoryLimitedBufferBuilder> = ExternalSorterBuilder::new()
45+
.with_tmp_dir(path::Path::new("tmp"))
46+
.with_buffer(MemoryLimitedBufferBuilder::new(50 * MB))
47+
.build()
48+
.unwrap();
49+
50+
let sorted = sorter.sort(input_reader.lines()).unwrap();
51+
52+
for item in sorted.map(Result::unwrap) {
53+
output_writer.write_all(format!("{}\n", item).as_bytes()).unwrap();
54+
}
55+
output_writer.flush().unwrap();
56+
}
57+
```

0 commit comments

Comments
 (0)