@@ -1,2 +1,3 @@ | |||
/target | |||
/.vscode | |||
Cargo.lock |
@@ -1,10 +1,10 @@ | |||
[workspace] | |||
members = [ | |||
"asparit", | |||
"async-ecs", | |||
"async-ecs-derive", | |||
"parallel-iterator" | |||
] | |||
[patch.crates-io] | |||
asparit = { path = "asparit" } | |||
async-ecs-derive = { path = "async-ecs-derive" } | |||
parallel-iterator = { path = "parallel-iterator" } |
@@ -1,5 +1,5 @@ | |||
[package] | |||
name = "parallel-iterator" | |||
name = "asparit" | |||
version = "0.1.0" | |||
authors = ["Bergmann89 <info@bergmann89.de>"] | |||
edition = "2018" |
@@ -0,0 +1,315 @@ | |||
# Parallel Iterators | |||
These are some notes on the design of the parallel iterator traits. | |||
This file does not describe how to **use** parallel iterators. | |||
## The challenge | |||
Parallel iterators are more complicated than sequential iterators. | |||
The reason is that they have to be able to split themselves up and | |||
operate in parallel across the two halves. | |||
The current design for parallel iterators has two distinct modes in | |||
which they can be used; as we will see, not all iterators support both | |||
modes (which is why there are two): | |||
- **Pull mode** (the `Producer` and `UnindexedProducer` traits): in this mode, | |||
the iterator is asked to produce the next item using a call to `next`. This | |||
is basically like a normal iterator, but with a twist: you can split the | |||
iterator in half to produce disjoint items in separate threads. | |||
- in the `Producer` trait, splitting is done with `split_at`, which accepts | |||
an index where the split should be performed. Only indexed iterators can | |||
work in this mode, as they know exactly how much data they will produce, | |||
and how to locate the requested index. | |||
- in the `UnindexedProducer` trait, splitting is done with `split`, which | |||
simply requests that the producer divide itself *approximately* in half. | |||
This is useful when the exact length and/or layout is unknown, as with | |||
`String` characters, or when the length might exceed `usize`, as with | |||
`Range<u64>` on 32-bit platforms. | |||
- In theory, any `Producer` could act unindexed, but we don't currently | |||
use that possibility. When you know the exact length, a `split` can | |||
simply be implemented as `split_at(length/2)`. | |||
- **Push mode** (the `Consumer` and `UnindexedConsumer` traits): in | |||
this mode, the iterator instead is *given* each item in turn, which | |||
is then processed. This is the opposite of a normal iterator. It's | |||
more like a `for_each` call: each time a new item is produced, the | |||
`consume` method is called with that item. (The traits themselves are | |||
a bit more complex, as they support state that can be threaded | |||
through and ultimately reduced.) Unlike producers, there are two | |||
variants of consumers. The difference is how the split is performed: | |||
- in the `Consumer` trait, splitting is done with `split_at`, which | |||
accepts an index where the split should be performed. All | |||
iterators can work in this mode. The resulting halves thus have an | |||
idea about how much data they expect to consume. | |||
- in the `UnindexedConsumer` trait, splitting is done with | |||
`split_off_left`. There is no index: the resulting halves must be | |||
prepared to process any amount of data, and they don't know where that | |||
data falls in the overall stream. | |||
- Not all consumers can operate in this mode. It works for | |||
`for_each` and `reduce`, for example, but it does not work for | |||
`collect_into_vec`, since in that case the position of each item is | |||
important for knowing where it ends up in the target collection. | |||
## How iterator execution proceeds | |||
We'll walk through this example iterator chain to start. This chain | |||
demonstrates more-or-less the full complexity of what can happen. | |||
```rust | |||
vec1.par_iter() | |||
.zip(vec2.par_iter()) | |||
.flat_map(some_function) | |||
.for_each(some_other_function) | |||
``` | |||
To handle an iterator chain, we start by creating consumers. This | |||
works from the end. So in this case, the call to `for_each` is the | |||
final step, so it will create a `ForEachConsumer` that, given an item, | |||
just calls `some_other_function` with that item. (`ForEachConsumer` is | |||
a very simple consumer because it doesn't need to thread any state | |||
between items at all.) | |||
Now, the `for_each` call will pass this consumer to the base iterator, | |||
which is the `flat_map`. It will do this by calling the `drive_unindexed` | |||
method on the `ParallelIterator` trait. `drive_unindexed` basically | |||
says "produce items for this iterator and feed them to this consumer"; | |||
it only works for unindexed consumers. | |||
(As an aside, it is interesting that only some consumers can work in | |||
unindexed mode, but all producers can *drive* an unindexed consumer. | |||
In contrast, only some producers can drive an *indexed* consumer, but | |||
all consumers can be supplied indexes. Isn't variance neat.) | |||
As it happens, `FlatMap` only works with unindexed consumers anyway. | |||
This is because flat-map basically has no idea how many items it will | |||
produce. If you ask flat-map to produce the 22nd item, it can't do it, | |||
at least not without some intermediate state. It doesn't know whether | |||
processing the first item will create 1 item, 3 items, or 100; | |||
therefore, to produce an arbitrary item, it would basically just have | |||
to start at the beginning and execute sequentially, which is not what | |||
we want. But for unindexed consumers, this doesn't matter, since they | |||
don't need to know how much data they will get. | |||
Therefore, `FlatMap` can wrap the `ForEachConsumer` with a | |||
`FlatMapConsumer` that feeds to it. This `FlatMapConsumer` will be | |||
given one item. It will then invoke `some_function` to get a parallel | |||
iterator out. It will then ask this new parallel iterator to drive the | |||
`ForEachConsumer`. The `drive_unindexed` method on `flat_map` can then | |||
pass the `FlatMapConsumer` up the chain to the previous item, which is | |||
`zip`. At this point, something interesting happens. | |||
## Switching from push to pull mode | |||
If you think about `zip`, it can't really be implemented as a | |||
consumer, at least not without an intermediate thread and some | |||
channels or something (or maybe coroutines). The problem is that it | |||
has to walk two iterators *in lockstep*. Basically, it can't call two | |||
`drive` methods simultaneously, it can only call one at a time. So at | |||
this point, the `zip` iterator needs to switch from *push mode* into | |||
*pull mode*. | |||
You'll note that `Zip` is only usable if its inputs implement | |||
`IndexedParallelIterator`, meaning that they can produce data starting | |||
at random points in the stream. This need to switch to push mode is | |||
exactly why. If we want to split a zip iterator at position 22, we | |||
need to be able to start zipping items from index 22 right away, | |||
without having to start from index 0. | |||
Anyway, so at this point, the `drive_unindexed` method for `Zip` stops | |||
creating consumers. Instead, it creates a *producer*, a `ZipProducer`, | |||
to be exact, and calls the `bridge` function in the `internals` | |||
module. Creating a `ZipProducer` will in turn create producers for | |||
the two iterators being zipped. This is possible because they both | |||
implement `IndexedParallelIterator`. | |||
The `bridge` function will then connect the consumer, which is | |||
handling the `flat_map` and `for_each`, with the producer, which is | |||
handling the `zip` and its preecessors. It will split down until the | |||
chunks seem reasonably small, then pull items from the producer and | |||
feed them to the consumer. | |||
## The base case | |||
The other time that `bridge` gets used is when we bottom out in an | |||
indexed producer, such as a slice or range. There is also a | |||
`bridge_unindexed` equivalent for - you guessed it - unindexed producers, | |||
such as string characters. | |||
<a name="producer-callback"> | |||
## What on earth is `ProducerCallback`? | |||
We saw that when you call a parallel action method like | |||
`par_iter.reduce()`, that will create a "reducing" consumer and then | |||
invoke `par_iter.drive_unindexed()` (or `par_iter.drive()`) as | |||
appropriate. This may create yet more consumers as we proceed up the | |||
parallel iterator chain. But at some point we're going to get to the | |||
start of the chain, or to a parallel iterator (like `zip()`) that has | |||
to coordinate multiple inputs. At that point, we need to start | |||
converting parallel iterators into producers. | |||
The way we do this is by invoking the method `with_producer()`, defined on | |||
`IndexedParallelIterator`. This is a callback scheme. In an ideal world, | |||
it would work like this: | |||
```rust | |||
base_iter.with_producer(|base_producer| { | |||
// here, `base_producer` is the producer for `base_iter` | |||
}); | |||
``` | |||
In that case, we could implement a combinator like `map()` by getting | |||
the producer for the base iterator, wrapping it to make our own | |||
`MapProducer`, and then passing that to the callback. Something like | |||
this: | |||
```rust | |||
struct MapProducer<'f, P, F: 'f> { | |||
base: P, | |||
map_op: &'f F, | |||
} | |||
impl<I, F> IndexedParallelIterator for Map<I, F> | |||
where I: IndexedParallelIterator, | |||
F: MapOp<I::Item>, | |||
{ | |||
fn with_producer<CB>(self, callback: CB) -> CB::Output { | |||
let map_op = &self.map_op; | |||
self.base_iter.with_producer(|base_producer| { | |||
// Here `producer` is the producer for `self.base_iter`. | |||
// Wrap that to make a `MapProducer` | |||
let map_producer = MapProducer { | |||
base: base_producer, | |||
map_op: map_op | |||
}; | |||
// invoke the callback with the wrapped version | |||
callback(map_producer) | |||
}); | |||
} | |||
}); | |||
``` | |||
This example demonstrates some of the power of the callback scheme. | |||
It winds up being a very flexible setup. For one thing, it means we | |||
can take ownership of `par_iter`; we can then in turn give ownership | |||
away of its bits and pieces into the producer (this is very useful if | |||
the iterator owns an `&mut` slice, for example), or create shared | |||
references and put *those* in the producer. In the case of map, for | |||
example, the parallel iterator owns the `map_op`, and we borrow | |||
references to it which we then put into the `MapProducer` (this means | |||
the `MapProducer` can easily split itself and share those references). | |||
The `with_producer` method can also create resources that are needed | |||
during the parallel execution, since the producer does not have to be | |||
returned. | |||
Unfortunately there is a catch. We can't actually use closures the way | |||
I showed you. To see why, think about the type that `map_producer` | |||
would have to have. If we were going to write the `with_producer` | |||
method using a closure, it would have to look something like this: | |||
```rust | |||
pub trait IndexedParallelIterator: ParallelIterator { | |||
type Producer; | |||
fn with_producer<CB, R>(self, callback: CB) -> R | |||
where CB: FnOnce(Self::Producer) -> R; | |||
... | |||
} | |||
``` | |||
Note that we had to add this associated type `Producer` so that | |||
we could specify the argument of the callback to be `Self::Producer`. | |||
Now, imagine trying to write that `MapProducer` impl using this style: | |||
```rust | |||
impl<I, F> IndexedParallelIterator for Map<I, F> | |||
where I: IndexedParallelIterator, | |||
F: MapOp<I::Item>, | |||
{ | |||
type MapProducer = MapProducer<'f, P::Producer, F>; | |||
// ^^ wait, what is this `'f`? | |||
fn with_producer<CB, R>(self, callback: CB) -> R | |||
where CB: FnOnce(Self::Producer) -> R | |||
{ | |||
let map_op = &self.map_op; | |||
// ^^^^^^ `'f` is (conceptually) the lifetime of this reference, | |||
// so it will be different for each call to `with_producer`! | |||
} | |||
} | |||
``` | |||
This may look familiar to you: it's the same problem that we have | |||
trying to define an `Iterable` trait. Basically, the producer type | |||
needs to include a lifetime (here, `'f`) that refers to the body of | |||
`with_producer` and hence is not in scope at the impl level. | |||
If we had [associated type constructors][1598], we could solve this | |||
problem that way. But there is another solution. We can use a | |||
dedicated callback trait like `ProducerCallback`, instead of `FnOnce`: | |||
[1598]: https://github.com/rust-lang/rfcs/pull/1598 | |||
```rust | |||
pub trait ProducerCallback<T> { | |||
type Output; | |||
fn callback<P>(self, producer: P) -> Self::Output | |||
where P: Producer<Item=T>; | |||
} | |||
``` | |||
Using this trait, the signature of `with_producer()` looks like this: | |||
```rust | |||
fn with_producer<CB: ProducerCallback<Self::Item>>(self, callback: CB) -> CB::Output; | |||
``` | |||
Notice that this signature **never has to name the producer type** -- | |||
there is no associated type `Producer` anymore. This is because the | |||
`callback()` method is generically over **all** producers `P`. | |||
The problem is that now the `||` sugar doesn't work anymore. So we | |||
have to manually create the callback struct, which is a mite tedious. | |||
So our `MapProducer` code looks like this: | |||
```rust | |||
impl<I, F> IndexedParallelIterator for Map<I, F> | |||
where I: IndexedParallelIterator, | |||
F: MapOp<I::Item>, | |||
{ | |||
fn with_producer<CB>(self, callback: CB) -> CB::Output | |||
where CB: ProducerCallback<Self::Item> | |||
{ | |||
return self.base.with_producer(Callback { callback: callback, map_op: self.map_op }); | |||
// ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |||
// Manual version of the closure sugar: create an instance | |||
// of a struct that implements `ProducerCallback`. | |||
// The struct declaration. Each field is something that need to capture from the | |||
// creating scope. | |||
struct Callback<CB, F> { | |||
callback: CB, | |||
map_op: F, | |||
} | |||
// Implement the `ProducerCallback` trait. This is pure boilerplate. | |||
impl<T, F, CB> ProducerCallback<T> for Callback<CB, F> | |||
where F: MapOp<T>, | |||
CB: ProducerCallback<F::Output> | |||
{ | |||
type Output = CB::Output; | |||
fn callback<P>(self, base: P) -> CB::Output | |||
where P: Producer<Item=T> | |||
{ | |||
// The body of the closure is here: | |||
let producer = MapProducer { base: base, | |||
map_op: &self.map_op }; | |||
self.callback.callback(producer) | |||
} | |||
} | |||
} | |||
} | |||
``` | |||
OK, a bit tedious, but it works! |
@@ -0,0 +1,30 @@ | |||
use crate::{ | |||
Consumer, DefaultExecutor, Executor, IndexedConsumer, IndexedExecutor, IndexedParallelIterator, | |||
ParallelIterator, | |||
}; | |||
pub trait Collector: Sized { | |||
type Iterator: ParallelIterator; | |||
type Consumer: Consumer<<Self::Iterator as ParallelIterator>::Item>; | |||
fn exec_with<E>(self, executor: E) -> E::Result | |||
where | |||
E: Executor<Self::Iterator, Self::Consumer>; | |||
fn exec(self) -> <DefaultExecutor as Executor<Self::Iterator, Self::Consumer>>::Result { | |||
self.exec_with(DefaultExecutor::default()) | |||
} | |||
} | |||
pub trait IndexedCollector: Sized { | |||
type Iterator: IndexedParallelIterator; | |||
type Consumer: IndexedConsumer<<Self::Iterator as ParallelIterator>::Item>; | |||
fn exec_with<E>(self, executor: E) -> E::Result | |||
where | |||
E: IndexedExecutor<Self::Iterator, Self::Consumer>; | |||
fn exec(self) -> <DefaultExecutor as Executor<Self::Iterator, Self::Consumer>>::Result { | |||
self.exec_with(DefaultExecutor::default()) | |||
} | |||
} |
@@ -0,0 +1,56 @@ | |||
use super::{Folder, Reducer}; | |||
/// A consumer is effectively a [generalized "fold" operation][fold], | |||
/// and in fact each consumer will eventually be converted into a | |||
/// [`Folder`]. What makes a consumer special is that, like a | |||
/// [`Producer`], it can be **split** into multiple consumers using | |||
/// the `split_off_left` method. When a consumer is split, it produces two | |||
/// consumers, as well as a **reducer**. The two consumers can be fed | |||
/// items independently, and when they are done the reducer is used to | |||
/// combine their two results into one. See [the README][r] for further | |||
/// details. | |||
/// | |||
/// [r]: README.md | |||
/// [fold]: https://doc.rust-lang.org/std/iter/trait.Iterator.html#method.fold | |||
/// [`Folder`]: trait.Folder.html | |||
/// [`Producer`]: trait.Producer.html | |||
pub trait Consumer<Item>: Send + Sized { | |||
/// The type of folder that this consumer can be converted into. | |||
type Folder: Folder<Item, Result = Self::Result>; | |||
/// The type of reducer that is produced if this consumer is split. | |||
type Reducer: Reducer<Self::Result>; | |||
/// The type of result that this consumer will ultimately produce. | |||
type Result: Send; | |||
/// Splits off a "left" consumer and returns it. The `self` | |||
/// consumer should then be used to consume the "right" portion of | |||
/// the data. (The ordering matters for methods like find_first -- | |||
/// values produced by the returned value are given precedence | |||
/// over values produced by `self`.) Once the left and right | |||
/// halves have been fully consumed, you should reduce the results | |||
/// with the result of `to_reducer`. | |||
fn split_off_left(&self) -> (Self, Self::Reducer); | |||
/// Convert the consumer into a folder that can consume items | |||
/// sequentially, eventually producing a final result. | |||
fn into_folder(self) -> Self::Folder; | |||
/// Hint whether this `Consumer` would like to stop processing | |||
/// further items, e.g. if a search has been completed. | |||
fn is_full(&self) -> bool { | |||
false | |||
} | |||
} | |||
/// A stateless consumer can be freely copied. These consumers can be | |||
/// used like regular consumers, but they also support a | |||
/// `split_at` method that does take an index to split. | |||
pub trait IndexedConsumer<Item>: Consumer<Item> { | |||
/// Divide the consumer into two consumers, one processing items | |||
/// `0..index` and one processing items from `index..`. Also | |||
/// produces a reducer that can be used to reduce the results at | |||
/// the end. | |||
fn split_at(self, index: usize) -> (Self, Self, Self::Reducer); | |||
} |
@@ -0,0 +1,21 @@ | |||
use super::{Consumer, IndexedConsumer, IndexedParallelIterator, ParallelIterator}; | |||
pub trait Executor<I, C> | |||
where | |||
I: ParallelIterator, | |||
C: Consumer<I::Item>, | |||
{ | |||
type Result; | |||
fn exec(self, iterator: I, consumer: C) -> Self::Result; | |||
} | |||
pub trait IndexedExecutor<I, C> | |||
where | |||
I: IndexedParallelIterator, | |||
C: IndexedConsumer<I::Item>, | |||
{ | |||
type Result; | |||
fn exec_indexed(self, iterator: I, consumer: C) -> Self::Result; | |||
} |
@@ -0,0 +1,43 @@ | |||
/// The `Folder` trait encapsulates [the standard fold | |||
/// operation][fold]. It can be fed many items using the `consume` | |||
/// method. At the end, once all items have been consumed, it can then | |||
/// be converted (using `complete`) into a final value. | |||
/// | |||
/// [fold]: https://doc.rust-lang.org/std/iter/trait.Iterator.html#method.fold | |||
pub trait Folder<Item>: Sized { | |||
/// The type of result that will ultimately be produced by the folder. | |||
type Result; | |||
/// Consume next item and return new sequential state. | |||
fn consume(self, item: Item) -> Self; | |||
/// Consume items from the iterator until full, and return new sequential state. | |||
/// | |||
/// This method is **optional**. The default simply iterates over | |||
/// `iter`, invoking `consume` and checking after each iteration | |||
/// whether `full` returns false. | |||
/// | |||
/// The main reason to override it is if you can provide a more | |||
/// specialized, efficient implementation. | |||
fn consume_iter<I>(mut self, iter: I) -> Self | |||
where | |||
I: IntoIterator<Item = Item>, | |||
{ | |||
for item in iter { | |||
self = self.consume(item); | |||
if self.is_full() { | |||
break; | |||
} | |||
} | |||
self | |||
} | |||
/// Finish consuming items, produce final result. | |||
fn complete(self) -> Self::Result; | |||
/// Hint whether this `Folder` would like to stop processing | |||
/// further items, e.g. if a search has been completed. | |||
fn is_full(&self) -> bool { | |||
false | |||
} | |||
} |
@@ -0,0 +1,148 @@ | |||
use super::ParallelIterator; | |||
/// `IntoParallelIterator` implements the conversion to a [`ParallelIterator`]. | |||
/// | |||
/// By implementing `IntoParallelIterator` for a type, you define how it will | |||
/// transformed into an iterator. This is a parallel version of the standard | |||
/// library's [`std::iter::IntoIterator`] trait. | |||
/// | |||
/// [`ParallelIterator`]: trait.ParallelIterator.html | |||
/// [`std::iter::IntoIterator`]: https://doc.rust-lang.org/std/iter/trait.IntoIterator.html | |||
pub trait IntoParallelIterator { | |||
/// The parallel iterator type that will be created. | |||
type Iter: ParallelIterator<Item = Self::Item>; | |||
/// The type of item that the parallel iterator will produce. | |||
type Item: Send; | |||
/// Converts `self` into a parallel iterator. | |||
/// | |||
/// # Examples | |||
/// | |||
/// ``` | |||
/// use asparit::*; | |||
/// | |||
/// println!("counting in parallel:"); | |||
/// (0..100).into_par_iter() | |||
/// .for_each(|i| println!("{}", i)); | |||
/// ``` | |||
/// | |||
/// This conversion is often implicit for arguments to methods like [`zip`]. | |||
/// | |||
/// ``` | |||
/// use asparit::*; | |||
/// | |||
/// let v: Vec<_> = (0..5).into_par_iter().zip(5..10).collect(); | |||
/// assert_eq!(v, [(0, 5), (1, 6), (2, 7), (3, 8), (4, 9)]); | |||
/// ``` | |||
/// | |||
/// [`zip`]: trait.IndexedParallelIterator.html#method.zip | |||
fn into_par_iter(self) -> Self::Iter; | |||
} | |||
/// `IntoParallelRefIterator` implements the conversion to a | |||
/// [`ParallelIterator`], providing shared references to the data. | |||
/// | |||
/// This is a parallel version of the `iter()` method | |||
/// defined by various collections. | |||
/// | |||
/// This trait is automatically implemented | |||
/// `for I where &I: IntoParallelIterator`. In most cases, users | |||
/// will want to implement [`IntoParallelIterator`] rather than implement | |||
/// this trait directly. | |||
/// | |||
/// [`ParallelIterator`]: trait.ParallelIterator.html | |||
/// [`IntoParallelIterator`]: trait.IntoParallelIterator.html | |||
pub trait IntoParallelRefIterator<'data> { | |||
/// The type of the parallel iterator that will be returned. | |||
type Iter: ParallelIterator<Item = Self::Item>; | |||
/// The type of item that the parallel iterator will produce. | |||
/// This will typically be an `&'data T` reference type. | |||
type Item: Send + 'data; | |||
/// Converts `self` into a parallel iterator. | |||
/// | |||
/// # Examples | |||
/// | |||
/// ``` | |||
/// use asparit::*; | |||
/// | |||
/// let v: Vec<_> = (0..100).collect(); | |||
/// assert_eq!(v.par_iter().sum::<i32>(), 100 * 99 / 2); | |||
/// | |||
/// // `v.par_iter()` is shorthand for `(&v).into_par_iter()`, | |||
/// // producing the exact same references. | |||
/// assert!(v.par_iter().zip(&v) | |||
/// .all(|(a, b)| std::ptr::eq(a, b))); | |||
/// ``` | |||
fn par_iter(&'data self) -> Self::Iter; | |||
} | |||
/// `IntoParallelRefMutIterator` implements the conversion to a | |||
/// [`ParallelIterator`], providing mutable references to the data. | |||
/// | |||
/// This is a parallel version of the `iter_mut()` method | |||
/// defined by various collections. | |||
/// | |||
/// This trait is automatically implemented | |||
/// `for I where &mut I: IntoParallelIterator`. In most cases, users | |||
/// will want to implement [`IntoParallelIterator`] rather than implement | |||
/// this trait directly. | |||
/// | |||
/// [`ParallelIterator`]: trait.ParallelIterator.html | |||
/// [`IntoParallelIterator`]: trait.IntoParallelIterator.html | |||
pub trait IntoParallelRefMutIterator<'data> { | |||
/// The type of iterator that will be created. | |||
type Iter: ParallelIterator<Item = Self::Item>; | |||
/// The type of item that will be produced; this is typically an | |||
/// `&'data mut T` reference. | |||
type Item: Send + 'data; | |||
/// Creates the parallel iterator from `self`. | |||
/// | |||
/// # Examples | |||
/// | |||
/// ``` | |||
/// use asparit::*; | |||
/// | |||
/// let mut v = vec![0usize; 5]; | |||
/// v.par_iter_mut().enumerate().for_each(|(i, x)| *x = i); | |||
/// assert_eq!(v, [0, 1, 2, 3, 4]); | |||
/// ``` | |||
fn par_iter_mut(&'data mut self) -> Self::Iter; | |||
} | |||
impl<T: ParallelIterator> IntoParallelIterator for T { | |||
type Iter = T; | |||
type Item = T::Item; | |||
fn into_par_iter(self) -> T { | |||
self | |||
} | |||
} | |||
impl<'data, I: 'data + ?Sized> IntoParallelRefIterator<'data> for I | |||
where | |||
&'data I: IntoParallelIterator, | |||
{ | |||
type Iter = <&'data I as IntoParallelIterator>::Iter; | |||
type Item = <&'data I as IntoParallelIterator>::Item; | |||
fn par_iter(&'data self) -> Self::Iter { | |||
self.into_par_iter() | |||
} | |||
} | |||
impl<'data, I: 'data + ?Sized> IntoParallelRefMutIterator<'data> for I | |||
where | |||
&'data mut I: IntoParallelIterator, | |||
{ | |||
type Iter = <&'data mut I as IntoParallelIterator>::Iter; | |||
type Item = <&'data mut I as IntoParallelIterator>::Item; | |||
fn par_iter_mut(&'data mut self) -> Self::Iter { | |||
self.into_par_iter() | |||
} | |||
} |
@@ -0,0 +1,156 @@ | |||
use super::{Consumer, Executor, IndexedConsumer, IndexedProducerCallback, ProducerCallback}; | |||
use crate::inner::for_each::ForEach; | |||
/// Parallel version of the standard iterator trait. | |||
/// | |||
/// The combinators on this trait are available on **all** parallel | |||
/// iterators. Additional methods can be found on the | |||
/// [`IndexedParallelIterator`] trait: those methods are only | |||
/// available for parallel iterators where the number of items is | |||
/// known in advance (so, e.g., after invoking `filter`, those methods | |||
/// become unavailable). | |||
/// | |||
/// For examples of using parallel iterators, see [the docs on the | |||
/// `iter` module][iter]. | |||
/// | |||
/// [iter]: index.html | |||
/// [`IndexedParallelIterator`]: trait.IndexedParallelIterator.html | |||
pub trait ParallelIterator: Sized + Send { | |||
/// The type of item that this parallel iterator produces. | |||
/// For example, if you use the [`for_each`] method, this is the type of | |||
/// item that your closure will be invoked with. | |||
/// | |||
/// [`for_each`]: #method.for_each | |||
type Item: Send; | |||
/// Internal method used to define the behavior of this parallel | |||
/// iterator. You should not need to call this directly. | |||
/// | |||
/// This method causes the iterator `self` to start producing | |||
/// items and to feed them to the consumer `consumer` one by one. | |||
/// It may split the consumer before doing so to create the | |||
/// opportunity to produce in parallel. | |||
/// | |||
/// See the [README] for more details on the internals of parallel | |||
/// iterators. | |||
/// | |||
/// [README]: README.md | |||
fn drive<E, C>(self, executor: E, consumer: C) -> E::Result | |||
where | |||
E: Executor<Self, C>, | |||
C: Consumer<Self::Item>; | |||
/// Internal method used to define the behavior of this parallel | |||
/// iterator. You should not need to call this directly. | |||
/// | |||
/// This method converts the iterator into a producer P and then | |||
/// invokes `callback.callback()` with P. Note that the type of | |||
/// this producer is not defined as part of the API, since | |||
/// `callback` must be defined generically for all producers. This | |||
/// allows the producer type to contain references; it also means | |||
/// that parallel iterators can adjust that type without causing a | |||
/// breaking change. | |||
/// | |||
/// See the [README] for more details on the internals of parallel | |||
/// iterators. | |||
/// | |||
/// [README]: README.md | |||
fn with_producer<CB>(self, callback: CB) -> CB::Output | |||
where | |||
CB: ProducerCallback<Self::Item>; | |||
/// Internal method used to define the behavior of this parallel | |||
/// iterator. You should not need to call this directly. | |||
/// | |||
/// Returns the number of items produced by this iterator, if known | |||
/// statically. This can be used by consumers to trigger special fast | |||
/// paths. Therefore, if `Some(_)` is returned, this iterator must only | |||
/// use the (indexed) `Consumer` methods when driving a consumer, such | |||
/// as `split_at()`. Calling `UnindexedConsumer::split_off_left()` or | |||
/// other `UnindexedConsumer` methods -- or returning an inaccurate | |||
/// value -- may result in panics. | |||
/// | |||
/// This method is currently used to optimize `collect` for want | |||
/// of true Rust specialization; it may be removed when | |||
/// specialization is stable. | |||
fn len_hint_opt(&self) -> Option<usize> { | |||
None | |||
} | |||
/// Executes `OP` on each item produced by the iterator, in parallel. | |||
/// | |||
/// # Examples | |||
/// | |||
/// ``` | |||
/// use asparit::*; | |||
/// | |||
/// (0..100).into_par_iter().for_each(|x| println!("{:?}", x)); | |||
/// ``` | |||
fn for_each<F>(self, operation: F) -> ForEach<Self, F> | |||
where | |||
F: Fn(Self::Item) + Sync + Send, | |||
{ | |||
ForEach::new(self, operation) | |||
} | |||
} | |||
/// An iterator that supports "random access" to its data, meaning | |||
/// that you can split it at arbitrary indices and draw data from | |||
/// those points. | |||
/// | |||
/// **Note:** Not implemented for `u64`, `i64`, `u128`, or `i128` ranges | |||
pub trait IndexedParallelIterator: ParallelIterator { | |||
/// Produces an exact count of how many items this iterator will | |||
/// produce, presuming no panic occurs. | |||
/// | |||
/// # Examples | |||
/// | |||
/// ``` | |||
/// use asparit::*; | |||
/// | |||
/// let par_iter = (0..100).into_par_iter().zip(vec![0; 10]); | |||
/// assert_eq!(par_iter.len(), 10); | |||
/// | |||
/// let vec: Vec<_> = par_iter.collect(); | |||
/// assert_eq!(vec.len(), 10); | |||
/// ``` | |||
fn len_hint(&self) -> usize; | |||
/// Internal method used to define the behavior of this parallel | |||
/// iterator. You should not need to call this directly. | |||
/// | |||
/// This method causes the iterator `self` to start producing | |||
/// items and to feed them to the consumer `consumer` one by one. | |||
/// It may split the consumer before doing so to create the | |||
/// opportunity to produce in parallel. If a split does happen, it | |||
/// will inform the consumer of the index where the split should | |||
/// occur (unlike `ParallelIterator::drive_unindexed()`). | |||
/// | |||
/// See the [README] for more details on the internals of parallel | |||
/// iterators. | |||
/// | |||
/// [README]: README.md | |||
fn drive_indexed<C>(self, consumer: C) -> C::Result | |||
where | |||
C: IndexedConsumer<Self::Item>; | |||
/// Internal method used to define the behavior of this parallel | |||
/// iterator. You should not need to call this directly. | |||
/// | |||
/// This method converts the iterator into a producer P and then | |||
/// invokes `callback.callback()` with P. Note that the type of | |||
/// this producer is not defined as part of the API, since | |||
/// `callback` must be defined generically for all producers. This | |||
/// allows the producer type to contain references; it also means | |||
/// that parallel iterators can adjust that type without causing a | |||
/// breaking change. | |||
/// | |||
/// See the [README] for more details on the internals of parallel | |||
/// iterators. | |||
/// | |||
/// [README]: README.md | |||
fn with_producer_indexed<CB>(self, callback: CB) -> CB::Output | |||
where | |||
CB: IndexedProducerCallback<Self::Item>; | |||
} |
@@ -0,0 +1,17 @@ | |||
mod collector; | |||
mod consumer; | |||
mod executor; | |||
mod folder; | |||
mod into_iter; | |||
mod iterator; | |||
mod producer; | |||
mod reducer; | |||
pub use collector::Collector; | |||
pub use consumer::{Consumer, IndexedConsumer}; | |||
pub use executor::{Executor, IndexedExecutor}; | |||
pub use folder::Folder; | |||
pub use into_iter::{IntoParallelIterator, IntoParallelRefIterator, IntoParallelRefMutIterator}; | |||
pub use iterator::{IndexedParallelIterator, ParallelIterator}; | |||
pub use producer::{IndexedProducer, IndexedProducerCallback, Producer, ProducerCallback}; | |||
pub use reducer::Reducer; |
@@ -0,0 +1,144 @@ | |||
use super::Folder; | |||
/// A variant on `Producer` which does not know its exact length or | |||
/// cannot represent it in a `usize`. These producers act like | |||
/// ordinary producers except that they cannot be told to split at a | |||
/// particular point. Instead, you just ask them to split 'somewhere'. | |||
/// | |||
/// (In principle, `Producer` could extend this trait; however, it | |||
/// does not because to do so would require producers to carry their | |||
/// own length with them.) | |||
pub trait Producer: Send + Sized { | |||
/// The type of item returned by this producer. | |||
type Item; | |||
/// Split midway into a new producer if possible, otherwise return `None`. | |||
fn split(self) -> (Self, Option<Self>); | |||
/// Iterate the producer, feeding each element to `folder`, and | |||
/// stop when the folder is full (or all elements have been consumed). | |||
fn fold_with<F>(self, folder: F) -> F | |||
where | |||
F: Folder<Self::Item>; | |||
} | |||
/// A `Producer` is effectively a "splittable `IntoIterator`". That | |||
/// is, a producer is a value which can be converted into an iterator | |||
/// at any time: at that point, it simply produces items on demand, | |||
/// like any iterator. But what makes a `Producer` special is that, | |||
/// *before* we convert to an iterator, we can also **split** it at a | |||
/// particular point using the `split_at` method. This will yield up | |||
/// two producers, one producing the items before that point, and one | |||
/// producing the items after that point (these two producers can then | |||
/// independently be split further, or be converted into iterators). | |||
/// In Rayon, this splitting is used to divide between threads. | |||
/// See [the `plumbing` README][r] for further details. | |||
/// | |||
/// Note that each producer will always produce a fixed number of | |||
/// items N. However, this number N is not queryable through the API; | |||
/// the consumer is expected to track it. | |||
/// | |||
/// NB. You might expect `Producer` to extend the `IntoIterator` | |||
/// trait. However, [rust-lang/rust#20671][20671] prevents us from | |||
/// declaring the DoubleEndedIterator and ExactSizeIterator | |||
/// constraints on a required IntoIterator trait, so we inline | |||
/// IntoIterator here until that issue is fixed. | |||
/// | |||
/// [r]: https://github.com/rayon-rs/rayon/blob/master/src/iter/plumbing/README.md | |||
/// [20671]: https://github.com/rust-lang/rust/issues/20671 | |||
pub trait IndexedProducer: Send + Sized { | |||
/// The type of item that will be produced by this producer once | |||
/// it is converted into an iterator. | |||
type Item; | |||
/// The type of iterator we will become. | |||
type IntoIter: Iterator<Item = Self::Item> + DoubleEndedIterator + ExactSizeIterator; | |||
/// Convert `self` into an iterator; at this point, no more parallel splits | |||
/// are possible. | |||
fn into_iter(self) -> Self::IntoIter; | |||
/// The minimum number of items that we will process | |||
/// sequentially. Defaults to 1, which means that we will split | |||
/// all the way down to a single item. This can be raised higher | |||
/// using the [`with_min_len`] method, which will force us to | |||
/// create sequential tasks at a larger granularity. Note that | |||
/// Rayon automatically normally attempts to adjust the size of | |||
/// parallel splits to reduce overhead, so this should not be | |||
/// needed. | |||
/// | |||
/// [`with_min_len`]: ../trait.IndexedParallelIterator.html#method.with_min_len | |||
fn min_len(&self) -> usize { | |||
1 | |||
} | |||
/// The maximum number of items that we will process | |||
/// sequentially. Defaults to MAX, which means that we can choose | |||
/// not to split at all. This can be lowered using the | |||
/// [`with_max_len`] method, which will force us to create more | |||
/// parallel tasks. Note that Rayon automatically normally | |||
/// attempts to adjust the size of parallel splits to reduce | |||
/// overhead, so this should not be needed. | |||
/// | |||
/// [`with_max_len`]: ../trait.IndexedParallelIterator.html#method.with_max_len | |||
fn max_len(&self) -> usize { | |||
usize::MAX | |||
} | |||
/// Split into two producers; one produces items `0..index`, the | |||
/// other `index..N`. Index must be less than or equal to `N`. | |||
fn split_at(self, index: usize) -> (Self, Self); | |||
/// Iterate the producer, feeding each element to `folder`, and | |||
/// stop when the folder is full (or all elements have been consumed). | |||
/// | |||
/// The provided implementation is sufficient for most iterables. | |||
fn fold_with<F>(self, folder: F) -> F | |||
where | |||
F: Folder<Self::Item>, | |||
{ | |||
folder.consume_iter(self.into_iter()) | |||
} | |||
} | |||
/// The `ProducerCallback` trait is a kind of generic closure, | |||
/// [analogous to `FnOnce`][FnOnce]. See [the corresponding section in | |||
/// the plumbing README][r] for more details. | |||
/// | |||
/// [r]: https://github.com/rayon-rs/rayon/blob/master/src/iter/plumbing/README.md#producer-callback | |||
/// [FnOnce]: https://doc.rust-lang.org/std/ops/trait.FnOnce.html | |||
pub trait ProducerCallback<T> { | |||
/// The type of value returned by this callback. Analogous to | |||
/// [`Output` from the `FnOnce` trait][Output]. | |||
/// | |||
/// [Output]: https://doc.rust-lang.org/std/ops/trait.FnOnce.html#associatedtype.Output | |||
type Output; | |||
/// Invokes the callback with the given producer as argument. The | |||
/// key point of this trait is that this method is generic over | |||
/// `P`, and hence implementors must be defined for any producer. | |||
fn callback<P>(self, producer: P) -> Self::Output | |||
where | |||
P: Producer<Item = T>; | |||
} | |||
/// The `IndexedProducerCallback` trait is a kind of generic closure, | |||
/// [analogous to `FnOnce`][FnOnce]. See [the corresponding section in | |||
/// the plumbing README][r] for more details. | |||
/// | |||
/// [r]: https://github.com/rayon-rs/rayon/blob/master/src/iter/plumbing/README.md#producer-callback | |||
/// [FnOnce]: https://doc.rust-lang.org/std/ops/trait.FnOnce.html | |||
pub trait IndexedProducerCallback<T> { | |||
/// The type of value returned by this callback. Analogous to | |||
/// [`Output` from the `FnOnce` trait][Output]. | |||
/// | |||
/// [Output]: https://doc.rust-lang.org/std/ops/trait.FnOnce.html#associatedtype.Output | |||
type Output; | |||
/// Invokes the callback with the given producer as argument. The | |||
/// key point of this trait is that this method is generic over | |||
/// `P`, and hence implementors must be defined for any producer. | |||
fn callback<P>(self, producer: P) -> Self::Output | |||
where | |||
P: IndexedProducer<Item = T>; | |||
} |
@@ -0,0 +1,12 @@ | |||
/// The reducer is the final step of a `Consumer` -- after a consumer | |||
/// has been split into two parts, and each of those parts has been | |||
/// fully processed, we are left with two results. The reducer is then | |||
/// used to combine those two results into one. See [the `plumbing` | |||
/// README][r] for further details. | |||
/// | |||
/// [r]: https://github.com/rayon-rs/rayon/blob/master/src/iter/plumbing/README.md | |||
pub trait Reducer<Result> { | |||
/// Reduce two final results into one; this is executed after a | |||
/// split. | |||
fn reduce(self, left: Result, right: Result) -> Result; | |||
} |
@@ -0,0 +1,5 @@ | |||
mod sequential; | |||
pub use sequential::Sequential as SequentialExecutor; | |||
pub type DefaultExecutor = SequentialExecutor; |
@@ -0,0 +1,70 @@ | |||
use crate::core::{ | |||
Consumer, Executor, Folder, IndexedConsumer, IndexedExecutor, IndexedParallelIterator, | |||
IndexedProducer, IndexedProducerCallback, ParallelIterator, Producer, ProducerCallback, | |||
}; | |||
#[derive(Default)] | |||
pub struct Sequential; | |||
struct Callback<C> { | |||
consumer: C, | |||
} | |||
struct IndexedCallback<C> { | |||
consumer: C, | |||
} | |||
impl<I, C> Executor<I, C> for Sequential | |||
where | |||
I: ParallelIterator, | |||
C: Consumer<I::Item>, | |||
{ | |||
type Result = C::Result; | |||
fn exec(self, iterator: I, consumer: C) -> Self::Result { | |||
iterator.with_producer(Callback { consumer }) | |||
} | |||
} | |||
impl<I, C> IndexedExecutor<I, C> for Sequential | |||
where | |||
I: IndexedParallelIterator, | |||
C: IndexedConsumer<I::Item>, | |||
{ | |||
type Result = C::Result; | |||
fn exec_indexed(self, iterator: I, consumer: C) -> Self::Result { | |||
iterator.with_producer_indexed(IndexedCallback { consumer }) | |||
} | |||
} | |||
impl<C, T> ProducerCallback<T> for Callback<C> | |||
where | |||
C: Consumer<T>, | |||
{ | |||
type Output = C::Result; | |||
fn callback<P>(self, producer: P) -> C::Result | |||
where | |||
P: Producer<Item = T>, | |||
{ | |||
if self.consumer.is_full() { | |||
self.consumer.into_folder().complete() | |||
} else { | |||
producer.fold_with(self.consumer.into_folder()).complete() | |||
} | |||
} | |||
} | |||
impl<C, T> IndexedProducerCallback<T> for IndexedCallback<C> | |||
where | |||
C: IndexedConsumer<T>, | |||
{ | |||
type Output = C::Result; | |||
fn callback<P>(self, _producer: P) -> C::Result | |||
where | |||
P: IndexedProducer<Item = T>, | |||
{ | |||
self.consumer.into_folder().complete() | |||
} | |||
} |
@@ -0,0 +1,107 @@ | |||
use crate::{ | |||
core::{Collector, Executor}, | |||
Consumer, Folder, ParallelIterator, | |||
}; | |||
use super::noop::NoOpReducer; | |||
pub struct ForEach<I, F> { | |||
iterator: I, | |||
operation: F, | |||
} | |||
impl<I, F> ForEach<I, F> { | |||
pub fn new(iterator: I, operation: F) -> Self { | |||
Self { | |||
iterator, | |||
operation, | |||
} | |||
} | |||
} | |||
impl<I, F> Collector for ForEach<I, F> | |||
where | |||
I: ParallelIterator, | |||
F: Fn(I::Item) + Sync + Send + Copy, | |||
{ | |||
type Iterator = I; | |||
type Consumer = ForEachConsumer<F>; | |||
fn exec_with<E>(self, executor: E) -> E::Result | |||
where | |||
E: Executor<Self::Iterator, Self::Consumer>, | |||
{ | |||
let iterator = self.iterator; | |||
let operation = self.operation; | |||
let consumer = ForEachConsumer { operation }; | |||
iterator.drive(executor, consumer) | |||
} | |||
} | |||
pub struct ForEachConsumer<F> { | |||
operation: F, | |||
} | |||
impl<F, T> Consumer<T> for ForEachConsumer<F> | |||
where | |||
F: Fn(T) + Sync + Send + Copy, | |||
{ | |||
type Folder = ForEachConsumer<F>; | |||
type Reducer = NoOpReducer; | |||
type Result = (); | |||
fn split_off_left(&self) -> (Self, NoOpReducer) { | |||
( | |||
ForEachConsumer { | |||
operation: self.operation, | |||
}, | |||
NoOpReducer, | |||
) | |||
} | |||
fn into_folder(self) -> Self { | |||
self | |||
} | |||
} | |||
impl<F, T> Folder<T> for ForEachConsumer<F> | |||
where | |||
F: Fn(T) + Sync + Send + Copy, | |||
{ | |||
type Result = (); | |||
fn consume(self, item: T) -> Self { | |||
(self.operation)(item); | |||
self | |||
} | |||
fn consume_iter<I>(self, iter: I) -> Self | |||
where | |||
I: IntoIterator<Item = T>, | |||
{ | |||
iter.into_iter().for_each(self.operation); | |||
self | |||
} | |||
fn complete(self) {} | |||
} | |||
#[cfg(test)] | |||
mod tests { | |||
use super::*; | |||
use crate::*; | |||
#[test] | |||
fn test_for_each() { | |||
(0..10usize) | |||
.into_par_iter() | |||
.for_each(&|j| { | |||
println!("{}", j); | |||
}) | |||
.exec(); | |||
} | |||
} |
@@ -0,0 +1,2 @@ | |||
pub mod for_each; | |||
pub mod noop; |
@@ -0,0 +1,51 @@ | |||
use crate::{Consumer, Folder, IndexedConsumer, Reducer}; | |||
pub struct NoOpConsumer; | |||
impl<T> Consumer<T> for NoOpConsumer { | |||
type Folder = NoOpConsumer; | |||
type Reducer = NoOpReducer; | |||
type Result = (); | |||
fn split_off_left(&self) -> (Self, Self::Reducer) { | |||
(NoOpConsumer, NoOpReducer) | |||
} | |||
fn into_folder(self) -> Self { | |||
self | |||
} | |||
fn is_full(&self) -> bool { | |||
false | |||
} | |||
} | |||
impl<T> Folder<T> for NoOpConsumer { | |||
type Result = (); | |||
fn consume(self, _item: T) -> Self { | |||
self | |||
} | |||
fn consume_iter<I>(self, iter: I) -> Self | |||
where | |||
I: IntoIterator<Item = T>, | |||
{ | |||
iter.into_iter().for_each(drop); | |||
self | |||
} | |||
fn complete(self) {} | |||
} | |||
impl<T> IndexedConsumer<T> for NoOpConsumer { | |||
fn split_at(self, _index: usize) -> (Self, Self, Self::Reducer) { | |||
(NoOpConsumer, NoOpConsumer, NoOpReducer) | |||
} | |||
} | |||
pub struct NoOpReducer; | |||
impl Reducer<()> for NoOpReducer { | |||
fn reduce(self, _left: (), _right: ()) {} | |||
} |
@@ -0,0 +1,11 @@ | |||
mod core; | |||
mod executor; | |||
mod inner; | |||
mod std; | |||
pub use self::core::{ | |||
Consumer, Executor, Folder, IndexedConsumer, IndexedExecutor, IndexedParallelIterator, | |||
IndexedProducer, IntoParallelIterator, IntoParallelRefIterator, IntoParallelRefMutIterator, | |||
ParallelIterator, Producer, ProducerCallback, Reducer, | |||
}; | |||
pub use self::executor::{DefaultExecutor, SequentialExecutor}; |
@@ -0,0 +1 @@ | |||
mod range; |
@@ -0,0 +1,71 @@ | |||
use std::ops::Range; | |||
use crate::{ | |||
Consumer, Executor, Folder, IntoParallelIterator, ParallelIterator, Producer, ProducerCallback, | |||
}; | |||
pub struct Iter { | |||
range: Range<usize>, | |||
} | |||
struct IterProducer { | |||
range: Range<usize>, | |||
} | |||
impl IntoParallelIterator for Range<usize> { | |||
type Iter = Iter; | |||
type Item = usize; | |||
fn into_par_iter(self) -> Self::Iter { | |||
Iter { range: self } | |||
} | |||
} | |||
impl ParallelIterator for Iter { | |||
type Item = usize; | |||
fn drive<E, C>(self, executor: E, consumer: C) -> E::Result | |||
where | |||
E: Executor<Self, C>, | |||
C: Consumer<Self::Item>, | |||
{ | |||
executor.exec(self, consumer) | |||
} | |||
fn len_hint_opt(&self) -> Option<usize> { | |||
Some(self.range.end - self.range.start) | |||
} | |||
fn with_producer<CB>(self, callback: CB) -> CB::Output | |||
where | |||
CB: ProducerCallback<Self::Item>, | |||
{ | |||
callback.callback(IterProducer { range: self.range }) | |||
} | |||
} | |||
impl Producer for IterProducer { | |||
type Item = usize; | |||
fn split(mut self) -> (Self, Option<Self>) { | |||
let index = self.range.len() / 2; | |||
if index > 0 { | |||
let mid = self.range.start.wrapping_add(index); | |||
let right = mid..self.range.end; | |||
self.range.end = mid; | |||
(self, Some(IterProducer { range: right })) | |||
} else { | |||
(self, None) | |||
} | |||
} | |||
fn fold_with<F>(self, folder: F) -> F | |||
where | |||
F: Folder<Self::Item>, | |||
{ | |||
folder.consume_iter(self.range) | |||
} | |||
} |
@@ -5,6 +5,7 @@ authors = ["Bergmann89 <info@bergmann89.de>"] | |||
edition = "2018" | |||
[dependencies] | |||
asparit = "0.1" | |||
async-ecs-derive = "0.1" | |||
env_logger = "0.8" | |||
futures = "0.3" | |||
@@ -13,7 +14,6 @@ hibitset = { version = "0.6", default-features = false } | |||
log = "0.4" | |||
mopa = "0.2" | |||
num_cpus = "1.13" | |||
parallel-iterator = "0.1" | |||
rand = "0.7" | |||
thiserror = "1.0" | |||
tokio = { version = "0.3", features = [ "full", "net", "time", "rt-multi-thread" ] } |
@@ -1,7 +1,7 @@ | |||
use std::cell::UnsafeCell; | |||
use asparit::{Consumer, Executor, Folder, ParallelIterator, Producer}; | |||
use hibitset::BitSetLike; | |||
use parallel_iterator::{Consumer, Executor, Folder, ParallelIterator, Producer}; | |||
use crate::misc::{BitIter, BitProducer, TokioExecutor}; | |||
@@ -11,7 +11,7 @@ pub mod storage; | |||
pub mod system; | |||
pub mod world; | |||
pub use parallel_iterator::ParallelIterator; | |||
pub use asparit::ParallelIterator; | |||
pub use access::{Join, ParJoin, ReadStorage, WriteStorage}; | |||
pub use dispatcher::Dispatcher; | |||
@@ -1,5 +1,5 @@ | |||
use asparit::{Folder, Producer}; | |||
use hibitset::BitSetLike; | |||
use parallel_iterator::{Folder, Producer}; | |||
use crate::misc::bit_average; | |||
@@ -7,9 +7,7 @@ use futures::{ | |||
}; | |||
use tokio::task::{block_in_place, spawn}; | |||
use parallel_iterator::{ | |||
Consumer, Executor, Folder, IndexedConsumer, IndexedProducer, Producer, Reducer, | |||
}; | |||
use asparit::{Consumer, Executor, Folder, IndexedConsumer, IndexedProducer, Producer, Reducer}; | |||
pub struct TokioExecutor; | |||
@@ -1,61 +0,0 @@ | |||
use super::{no_op::NoOpReducer, Consumer, Folder, ParallelIterator}; | |||
pub fn for_each<I, F, T>(iter: I, f: &F) | |||
where | |||
I: ParallelIterator<Item = T>, | |||
F: Fn(T) + Sync, | |||
T: Send, | |||
{ | |||
let consumer = ForEachConsumer { f }; | |||
iter.drive(consumer) | |||
} | |||
struct ForEachConsumer<'f, F> { | |||
f: &'f F, | |||
} | |||
impl<'f, F, T> Consumer<T> for ForEachConsumer<'f, F> | |||
where | |||
F: Fn(T) + Sync, | |||
{ | |||
type Folder = ForEachConsumer<'f, F>; | |||
type Reducer = NoOpReducer; | |||
type Result = (); | |||
fn split_off_left(&self) -> (Self, Self::Reducer) { | |||
(Self { f: self.f }, NoOpReducer) | |||
} | |||
fn into_folder(self) -> Self::Folder { | |||
self | |||
} | |||
} | |||
impl<'f, F, T> Folder<T> for ForEachConsumer<'f, F> | |||
where | |||
F: Fn(T) + Sync, | |||
{ | |||
type Result = (); | |||
fn is_full(&self) -> bool { | |||
false | |||
} | |||
fn complete(self) -> Self::Result {} | |||
fn consume(self, item: T) -> Self { | |||
(self.f)(item); | |||
self | |||
} | |||
fn consume_iter<I>(self, iter: I) -> Self | |||
where | |||
I: IntoIterator<Item = T>, | |||
{ | |||
iter.into_iter().for_each(self.f); | |||
self | |||
} | |||
} |
@@ -1,121 +0,0 @@ | |||
mod for_each; | |||
mod no_op; | |||
pub trait ParallelIterator: Sized + Send { | |||
type Item: Send; | |||
fn for_each<F>(self, f: F) | |||
where | |||
F: Fn(Self::Item) + Sync + Send, | |||
{ | |||
for_each::for_each(self, &f) | |||
} | |||
fn drive<C>(self, consumer: C) -> C::Result | |||
where | |||
C: Consumer<Self::Item>; | |||
fn len_hint_opt(&self) -> Option<usize> { | |||
None | |||
} | |||
} | |||
pub trait IndexedParallelIterator: ParallelIterator { | |||
fn drive_indexed<C>(self, consumer: C) -> C::Result | |||
where | |||
C: IndexedConsumer<Self::Item>; | |||
fn len_hint(&self) -> usize; | |||
} | |||
pub trait Producer: Send + Sized { | |||
type Item; | |||
fn split(self) -> (Self, Option<Self>); | |||
fn fold_with<F>(self, folder: F) -> F | |||
where | |||
F: Folder<Self::Item>; | |||
} | |||
pub trait IndexedProducer: Send + Sized { | |||
type Item; | |||
type IntoIter: Iterator<Item = Self::Item> + DoubleEndedIterator + ExactSizeIterator; | |||
fn into_iter(self) -> Self::IntoIter; | |||
fn min_len(&self) -> usize { | |||
1 | |||
} | |||
fn max_len(&self) -> usize { | |||
usize::MAX | |||
} | |||
fn split_at(self, index: usize) -> (Self, Self); | |||
fn fold_with<F>(self, folder: F) -> F | |||
where | |||
F: Folder<Self::Item>, | |||
{ | |||
folder.consume_iter(self.into_iter()) | |||
} | |||
} | |||
pub trait Consumer<Item>: Send + Sized { | |||
type Folder: Folder<Item, Result = Self::Result>; | |||
type Reducer: Reducer<Self::Result> + Send; | |||
type Result: Send; | |||
fn split_off_left(&self) -> (Self, Self::Reducer); | |||
fn into_folder(self) -> Self::Folder; | |||
fn is_full(&self) -> bool { | |||
false | |||
} | |||
} | |||
pub trait IndexedConsumer<Item>: Consumer<Item> { | |||
fn split_at(self, index: usize) -> (Self, Self, Self::Reducer); | |||
} | |||
pub trait Folder<Item>: Sized { | |||
type Result; | |||
fn is_full(&self) -> bool; | |||
fn complete(self) -> Self::Result; | |||
fn consume(self, item: Item) -> Self; | |||
fn consume_iter<I>(mut self, iter: I) -> Self | |||
where | |||
I: IntoIterator<Item = Item>, | |||
{ | |||
for item in iter { | |||
self = self.consume(item); | |||
if self.is_full() { | |||
break; | |||
} | |||
} | |||
self | |||
} | |||
} | |||
pub trait Reducer<Result> { | |||
fn reduce(self, left: Result, right: Result) -> Result; | |||
} | |||
pub trait Executor { | |||
fn exec<P, C>(self, producer: P, consumer: C) -> C::Result | |||
where | |||
P: Producer, | |||
C: Consumer<P::Item>; | |||
fn exec_indexed<P, C>(self, producer: P, consumer: C) -> C::Result | |||
where | |||
P: IndexedProducer, | |||
C: IndexedConsumer<P::Item>; | |||
} |
@@ -1,7 +0,0 @@ | |||
use super::Reducer; | |||
pub struct NoOpReducer; | |||
impl Reducer<()> for NoOpReducer { | |||
fn reduce(self, _left: (), _right: ()) {} | |||
} |