Performance goals

[digikar99]

Hello, thank you for taking up the maintenance of various libraries as part of lisp-stat!

In my free time, I have been working on numericals built upon polymorphic-functions. But this requires CLTL2 support, and even slightly more than that.

I like the functionality of numerical-utilities as well as lisp-stat systems in general, but do feel that they can use a performance boost wherever possible. In some cases, this could be as simple as inline-ing a function; in other cases, especially if we depend on CFFI libraries, I am not sure builtin optimizations would be sufficient since we might need to dispatch on array element types. I was considering rewriting the relevant parts using polymorphic-functions, but I wondered if I should write it under the system numerical-utilities/polymorphic-functions and let it stay under this repository, or if I should let the lisp-stat systems stay in ANSI CL and make the copy under numericals itself. Let me know what your goals and preferences are!

[snunez1]

Hi, thanks for writing. I had considered a similar approach and think it worth looking at. In my case the data sets didn't justify the effort, as the existing code was 'fast enough', but I certainly wouldn't oppose performance improvements. I think numerical-utilities/polymorphic-functions is the way to go so that the improvements will be incorporated into Lisp-Stat automatically.

Just one comment on CLTL2: I expect that Lisp-Stat will probably always be SBCL specific, so there's no harm in relying on sb-cltl2; we already have that as a dependency.

[digikar99]

Okay, I found some time to take a look at this.

In several cases (eg: sum and product) where methods are specialized on sequence and array, polymorphic-functions cannot be used, because they do not look into the class precedence list to decide dispatching.

In cases where reduce is being used, optimization is still an open question for commander-trashdin (working on lisp-polymorph) and me.

static-dispatch seems to be the best bet, especially on SBCL. Incorporating it is as simple as replacing #:cl with #:static-dispatch-cl in packages.lisp, and adding #:static-dispatch to :depends-on in the asd file.

[snunez1]

Hmmm. I think you're right, static-dispatch seems the best way forward, and it's simple.

[digikar99]

But should static-dispatch be made the default, or should there be a separate system num-utils/static-dispatch with just the packages.lisp file changed?

[snunez1]

I'd suggest starting with a separate system; after a while if no issues are detected, we can fold it into the main system.

[digikar99]

Okay!

Erm, another decision: should both versions be able to coexist in a single lisp image, or should only one of them exist in the image at any given time?

For both of them to coexist, we will also need to rename the packages appropriately. Two ways to achieve this I know of include: (i) Copying all the files and appropriately replacing the in-package lines. This has long term maintenance costs because any change will need to be made in both the files. (ii) Having a compiler-in-package macro that expands into an appropriate in-package and then sym-link all the lisp files in the new system to the old system. This does not have long term maintenance costs, but one requires getting used to the compiler-in-package since at least slime does not play nice with it out of the box.

While, if only one of them needs to be in the image, then we can keep the package names as they are; add in a new packages.lisp, and then symlink to the other existing files without making any changes to them. I find this the simplest, albeit with the cost that not both versions can coexist in a single image.

[snunez1]

Probably only one needs to be in an image at the same time. I can't imagine a situation where both would be needed.