Anonymous function creation is slow, and almost always necessary

Suppose I have a function f(x::Vector, p::SomeData), which I want to compute the Jacobian of. A common way to do this is

p = generate_some_data()
jac = ForwardDiff(x -> f(x, p), x0)

which has very bad performance on every call because we are instantiating a new anonymous function every time, which triggers compilation. A more clever way of doing this is

partially_applied_f = x -> f(x, p)
#repeat the lines below as much as needed
p = generate_some_data()
jac = ForwardDiff(partially_applied_f, x0)

which introduces a weird invisible dependency of the calls of partially_applied_f on the current state of p. Would it be feasible to update method signatures in this package to accept parameters at the call site? Such that the following code is valid:

p = generate_some_data()
jac = ForwardDiff(f, x0, parameter = p)

DifferentialEquations already do something similar with their method calls. This would remove a lot of overhead and imprevisibility in my use cases.

because we are instantiating a new anonymous function every time, which triggers compilation

What do you mean here with "every time"? Only once, right?

I ran the experiment:

julia> using ForwardDiff

julia> f(x, p) = x^p
f (generic function with 1 method)

julia> ForwardDiff.derivative(x -> f(x, 2), 2)
4

julia> @time ForwardDiff.derivative(x -> f(x, 2), 2)
  0.005143 seconds (16.13 k allocations: 800.141 KiB, 80.10% compilation time)
4

julia> @time ForwardDiff.derivative(x -> f(x, 2), 2)
  0.004946 seconds (16.07 k allocations: 796.594 KiB, 98.78% compilation time)
4

julia> @time ForwardDiff.derivative(x -> f(x, 2), 2)
  0.004208 seconds (16.07 k allocations: 797.000 KiB, 98.66% compilation time)
4

which makes a lot of sense, since every time we write x -> f(x, 2). we are actually creating a new anonymous function. The same behavior is found if we re-execute the line defining partially_applied_f, as can be seen by the change in the number in the output (usually something like #ANONYMOUS_FUNCTION_ID (generic function with 1 method)).

Yes, but only if you are doing it in the REPL and retyping it like that. If you have it in a function or store the closure to a variable it is only once.

Well, yes, but then you get this weird behavior where the value of your derivative depends on some current value of the variable that was captured:

using ForwardDiff

f(x, p) = x^p

g = x -> f(x, current_p)

current_p = 2
first_deriv = @time ForwardDiff.derivative(g, 2.0)
println(first_deriv)

current_p = 5
@time ForwardDiff.derivative(g, 2.0)

outputs

0.017199 seconds (19.55 k allocations: 994.824 KiB, 99.64% compilation time)
4.0
  0.000010 seconds (3 allocations: 80 bytes)
80.0

which effectively shows that g was defined once and did not trigger compilation on the second call. However, this dependency on the variable current_p is quite weird and hard to manage. I do not see a way of avoiding both issues at once: the repeated compilation and the clear flow of data to partially applied functions.

A very similar discussion can be found in an old DifferentialEquations issue. I think the solution they arrived at would also be the best fit for ForwardDiff.

You use a let block for that, right?

Inside a function like this there is only one x -> f(x, p), and no cost:

julia> using ForwardDiff, BenchmarkTools

julia> f(x, p) = x^p
f (generic function with 1 method)

julia> function mycalc(xx::Real)
         p = 3
         d = ForwardDiff.derivative(x -> f(x, p), xx)
         d^2
       end
mycalc (generic function with 1 method)

julia> @btime mycalc(1.0)
  0.916 ns (0 allocations: 0 bytes)
9.0

In your example, g is a non-const global which depends on global current_p, this is bad for other reasons.

You use a let block for that, right?

I posted the entire script, written in a file, with no let anywhere.

So, if I understood correctly:

The syntax ForwardDiff.derivative(x -> f(x, p), x0) is bad in the global context, because compilation is triggered everytime.
In the global/script context, you should define a closure before the calls to ForwardDiff. However, defining a function with the syntax g(x) = f(x, p) does not seem to trigger compilation upon redefinition.
The syntax ForwardDiff.derivative(x -> f(x, p), x0) is fine inside functions.

In your example, g is a non-const global which depends on global current_p, this is bad for other reasons.

I understand this is bad, yes, it feels very messy. In your example, compilation must have happened at some point, no? I don't think it is possible for Julia to not compile code.

The behavior of this is not super clear to me. I've had issues with nested anonymous functions passed around many function calls basically triggering compilation at every call to ForwardDiff. So, I guess allowing for extra parameters does not make sense here. Maybe explain this somewhere? I get that it is a direct consequence of Julia's compilation rules but it is not at all obvious.

It can confusing to get used to, but none of this is all that specific to ForwardDiff. Performance-sensitive code should never just be sitting at top-level of a script or at the REPL, it should always be inside a function.

https://docs.julialang.org/en/v1/manual/performance-tips/#Performance-critical-code-should-be-inside-a-function

Yes, everything gets compiled. It's fine to leave things you only call once at top level -- you are implicitly stating that you don't care to save their compiled version for re-use.

Navigation Menu

Repository navigation

Anonymous function creation is slow, and almost always necessary #873

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Anonymous function creation is slow, and almost always necessary #873

Description

Activity

KristofferC commented on Nov 6, 2026

pekpuglia commented on Nov 6, 2026

KristofferC commented on Nov 6, 2026

pekpuglia commented on Nov 6, 2026

pekpuglia commented on Nov 6, 2026

KristofferC commented on Nov 6, 2026

mcabbott commented on Nov 6, 2026

pekpuglia commented on Nov 6, 2026

mcabbott commented on Nov 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions