Skip to content

Overhaul derivatives for vectorisation and user registration - #1384

Merged
bendudson merged 155 commits into
nextfrom
more_templating_difops_TemplateMagic
Dec 4, 2018
Merged

bendudson merged 155 commits into
nextfrom
more_templating_difops_TemplateMagic

Conversation

@d7919

@d7919 d7919 commented Nov 19, 2018

Copy link
Copy Markdown
Member

Overhaul of index derivatives. Designed to provide the following main advantages:

  1. Possible for user registration of derivative methods.
  2. Inner loop of derivative methods vectorise were possible and can be openmp parallelised.
  3. Reduction of duplicate code.
  4. More consistent treatment of staggering in all directions and for both field types.

Also provides the following minor changes:

  1. More robust treatment of enums in boutcore processing.
  2. uppercase routine.
  3. Adds a direction enum class, a staggering enum class and a derivative class (standard, upwind, flux etc.) enum class.
  4. A couple of minor helper routines have been added to mesh to provide methods to request the number of points/guard cells in a particular direction.
  5. Performance examples for measuring the cost of DDX, DDY and DDZ with various supported methods.
  6. Provide an overload for top level derivative methods that takes a std::string instead of a DIFF_METHOD enum. This is required to allow for user registered routines. The std::string methods should be preferred and the DIFF_METHOD overloads may become deprecated.
  7. Provides a templated minus and plus offsetting routine for SpecificInd which provides a simple way to offset in a requested direction with compile time information that works nicely with routines templated on the DIRECTION enum class.
  8. Adds a fourth option to mesh:dd?, currently C2 is the only valid predefined option.

Still to do:

  1. Documentation. (main user documentation added, no developer docs added yet but hopefully source documentation covers this).
  2. More testing. (never ending task! Basic unit testing for derivative store and template combination code added)

Main implementation details:

  1. To achieve vectorisation for the "inner loop" we had to remove the function pointer indirection,
    meaning we have to effectively create an instance of the loop for every possible method and with all possible combinations of staggering, direction etc. This is achieving using some templating routines (in include/bout/template_combinations.hxx) which can do "something" for a cartesian product of options.
  2. The "something" in 1. is to create derivative instances and register them in a singleton derivative store (one for each field type). This store can register routines and can return these at runtime given user options etc.
  3. All the indexDD? routines now just call a single kernel (indexStandardDerivative) that gets the requested method and applies it in the appropriate direction. All the indexVDD? and indexFDD? routines now just call a single kernel (indexFlowDerivative) that gets the requested method and applies it.

The performance examples suggest that most derivative methods now vectorise, giving performance comparable to direct hard-coded nested loop. Whilst this is around a factor 3-4 improvement over the current implementation in next this is not reflected in physics models due to the other costs involved in a full model.

Compilation of the library is likely to be slower due to the template combination expansion that must be done at compile time.

d7919 added 30 commits October 9, 2018 10:13
These are templated on the size of the offset to take and the direction in
which to take the offset.
Can now use type deduction to avoid need to specify what FieldType is
being used.

Provides defaults to enable less typing in common uses
These just return a pointer to the parent field.
This is currently a bit ugly and enforces `nGuard == 1` even for Field2D
which could in principle use more than one guard cell.
in index_derivs.

Makes it clearer that the region is associated with iterating over the
field.

Removes one point of difference between 2D and 3D versions of
index_deriv routines.
Note appropriate type of field for template can be inferred from
function signature and hence we don't need to explicitly specify this at
the call sites
Comment thread include/bout/deriv_store.hxx Outdated

@ZedThree ZedThree left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, a few minor comments!

Comment thread include/bout/deriv_store.hxx Outdated
Comment thread include/bout/mesh.hxx Outdated
Comment thread include/bout/mesh.hxx Outdated
Comment thread include/bout/mesh.hxx Outdated
Comment thread include/bout/region.hxx Outdated
@ZedThree

Copy link
Copy Markdown
Member

Should we namespace some of the new things? The template combo stuff might go in one, though I've not got a name to mind. Possibly bout::utils?

ZedThree
ZedThree previously approved these changes Nov 30, 2018
ZedThree
ZedThree previously approved these changes Nov 30, 2018
@d7919
d7919 requested a review from bendudson November 30, 2018 13:43
Removes them from being mesh members

Rewrite approach for Flux:Split method to be implemented as all other derivatives.
Comment thread include/bout/index_derivs_interface.hxx Outdated
ZedThree
ZedThree previously approved these changes Nov 30, 2018
@bendudson
bendudson merged commit a4f453c into next Dec 4, 2018
@bendudson
bendudson deleted the more_templating_difops_TemplateMagic branch December 4, 2018 08:10
@d7919 d7919 mentioned this pull request Dec 6, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature A PR that adds new capabilities / features performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants