Questions

[Jokeren]

Hi @apd10 ,

After taking a pass over your code, I came up with a few questions below, can you please take a look?

1. If input is MxK, and weight is KxN, as long as K is large, we must allocate large space for the input, right? Then how do we save memory in this case?

This question is important because optimizations are related with the values of M and N. If M is extremely small comparing to N and K (e.g., 4 vs 1024), the implementation will be very different from a large M.

3. What is chunk_size used for?

4. In the cpp implementation, you probably missed the operations on hashed_weight. Do you want me to just go optimize an ordinary matrix multiplication as you implemented in the file? Otherwise, can you please add hashed_weight and hash_func code?

[apd10]

1. The idea is to compress the learnable weights, thus reducing the total model size. Also, multiple weight matrices will share the same compressed weight - thus direct comparison between MxK and KxN is not representative. M while running the code is the batch size which is chosen by user. In most cases it will be small if K is large. Let me lookup some sample configurations from the experiments i am running.
2. chunk_size is not currently used. it is for backward compatibility and i plan to implement it future. Right now, we are using SMLS x SMLS sized chunk_size by default
3. hashed_weight : i see. Thanks for pointing it out. I only implemented matrix multiplication here. ( I actually implemented hashing in the torch-integrated version. Do you think optimizations with and without hashing will be different ? You can see the corresponding hashed version in torch-integrated kernel, the only change is that while reading up the matrix tile , it will use hashed location instead of normal lookup.

[Jokeren]

chunk_size is not currently used. it is for backward compatibility and i plan to implement it future. Right now, we are using SMLS x SMLS sized chunk_size by default

So num_chunks=1? Or num_chunks = (input_dim + chunk_size - 1)/ chunk_size;?

I actually implemented hashing in the torch-integrated version. Do you think optimizations with and without hashing will be different

Yes, it could be very different.