Add NEON fast path for SFP weight decode

[RaviTriv]

Proposed fast path for weight decode on NEON that uses table look up and shift. About the same ops but looks like we can sneak in a few cheaper ones, roughly 1.1x faster on SFP matmuls.

[jan-wassenberg]

Nice work and notable speedup, thanks for adding :) Have you measured on non-AVX3_DL x86 to see if your approach is faster than the generic? If so, we could replace our generic fallback on all platforms, rather than just NEON. SVE is also increasingly available/chosen.

[RaviTriv]

non-AVX3_DL x86

thanks!

good idea, have not measured. taking a look :)

[RaviTriv]

So looks like this is also a good speedup on SVE, roughly 1.1x. The default fallback is still better for AVX2 and AVX3 because x86 doesnt have a per-lane variable shifts for u8.

[jan-wassenberg]

Nice, makes sense SVE2_128 also works well. Would also think SVE2 works, any reason we limit it to just SVE2_128?

[RaviTriv]

Yep, only enabled for SVE2_128 because I was referring to comment indicating that SVE2_128 is the only enabled SVE target right now?

[jan-wassenberg]

True, though this is due to compiler bugs which will hopefully be fixed at some point. I think there is no need to disable/exclude those SVE targets here as well, but let's still land as-is.