New Wasm workloads: Transformers.js ML sentiment analysis and Speech-to-Text

[danleh]

These could be replacements for the tfjs workloads (even though the model file size issue remains). Run via transformersjs-bert-wasm and transformersjs-whisper-wasm.

TODOs: Evaluate startup/model loading performance, take CPU profile, decide whether Whisper task is too long-running, compress model files on-disk for repo size, modify NPM server to handle those?

[netlify]

✅ Deploy Preview for webkit-jetstream-preview ready!

Name	Link
🔨 Latest commit	6b52dad
🔍 Latest deploy log	https://app.netlify.com/projects/webkit-jetstream-preview/deploys/68bfec3b0ebfd90008dcdcb2
😎 Deploy Preview	https://deploy-preview-148--webkit-jetstream-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[eqrion]

I am getting the following error when running the deploy preview in Firefox:

Uncaught (in promise) Error: Unexpected fetch: blob:https://deploy-preview-148--webkit-jetstream-preview.netlify.app/bb376a7e-5209-4208-ac76-defa885759a2
    fetch blob:https://deploy-preview-148--webkit-jetstream-preview.netlify.app/ac76acf2-8e08-42c9-a0b5-087158373d53:23
    getBinary https://deploy-preview-148--webkit-jetstream-preview.netlify.app/?test=transformersjs-whisper-wasm line 688 > injectedScript:38
    init blob:https://deploy-preview-148--webkit-jetstream-preview.netlify.app/ac76acf2-8e08-42c9-a0b5-087158373d53:45
    doRun https://deploy-preview-148--webkit-jetstream-preview.netlify.app/?test=transformersjs-whisper-wasm:57
    <anonymous> https://deploy-preview-148--webkit-jetstream-preview.netlify.app/?test=transformersjs-whisper-wasm:80
[ac76acf2-8e08-42c9-a0b5-087158373d53:23:11](blob:https://deploy-preview-148--webkit-jetstream-preview.netlify.app/ac76acf2-8e08-42c9-a0b5-087158373d53)
    start https://deploy-preview-148--webkit-jetstream-preview.netlify.app/JetStreamDriver.js:257
    AsyncFunctionThrow self-hosted:804
    (Async: async)
    onclick https://deploy-preview-148--webkit-jetstream-preview.netlify.app/JetStreamDriver.js:453
    (Async: EventHandlerNonNull)
    prefetchResources https://deploy-preview-148--webkit-jetstream-preview.netlify.app/JetStreamDriver.js:451
    InterpretGeneratorResume self-hosted:1332
    AsyncFunctionNext self-hosted:800
    (Async: async)
    initialize https://deploy-preview-148--webkit-jetstream-preview.netlify.app/JetStreamDriver.js:405
    initialize https://deploy-preview-148--webkit-jetstream-preview.netlify.app/?test=transformersjs-whisper-wasm:53
    onload https://deploy-preview-148--webkit-jetstream-preview.netlify.app/?test=transformersjs-whisper-wasm:1

[danleh]

I am getting the following error when running the deploy preview in Firefox:

Uncaught (in promise) Error: Unexpected fetch: blob:https://deploy-preview-148--webkit-jetstream-preview.netlify.app/bb376a7e-5209-4208-ac76-defa885759a2
    fetch blob:https://deploy-preview-148--webkit-jetstream-preview.netlify.app/ac76acf2-8e08-42c9-a0b5-087158373d53:23
    getBinary https://deploy-preview-148--webkit-jetstream-preview.netlify.app/?test=transformersjs-whisper-wasm line 688 > injectedScript:38
    init blob:https://deploy-preview-148--webkit-jetstream-preview.netlify.app/ac76acf2-8e08-42c9-a0b5-087158373d53:45
    doRun https://deploy-preview-148--webkit-jetstream-preview.netlify.app/?test=transformersjs-whisper-wasm:57
    <anonymous> https://deploy-preview-148--webkit-jetstream-preview.netlify.app/?test=transformersjs-whisper-wasm:80
[ac76acf2-8e08-42c9-a0b5-087158373d53:23:11](blob:https://deploy-preview-148--webkit-jetstream-preview.netlify.app/ac76acf2-8e08-42c9-a0b5-087158373d53)
    start https://deploy-preview-148--webkit-jetstream-preview.netlify.app/JetStreamDriver.js:257
    AsyncFunctionThrow self-hosted:804
    (Async: async)
    onclick https://deploy-preview-148--webkit-jetstream-preview.netlify.app/JetStreamDriver.js:453
    (Async: EventHandlerNonNull)
    prefetchResources https://deploy-preview-148--webkit-jetstream-preview.netlify.app/JetStreamDriver.js:451
    InterpretGeneratorResume self-hosted:1332
    AsyncFunctionNext self-hosted:800
    (Async: async)
    initialize https://deploy-preview-148--webkit-jetstream-preview.netlify.app/JetStreamDriver.js:405
    initialize https://deploy-preview-148--webkit-jetstream-preview.netlify.app/?test=transformersjs-whisper-wasm:53
    onload https://deploy-preview-148--webkit-jetstream-preview.netlify.app/?test=transformersjs-whisper-wasm:1

My bad, the dynamic import of ONNX runtime didn't work with the blob/preloading in the browser, fixed.

[kmiller68]

Seems like both Bert and Whisper spend a majority of the time in whatever function index 7460 is (I assume some matrix multiplication function). Although, it does look like about 70-75% vs 50-55% in 7460, respectively, so Whisper is doing other things. They also look pretty similar in where they spend their time within that function.

The fact that Whisper does other things (and IIUC, is more popular) makes it somewhat more interesting. On the other hand, it seems like the dominant function in either case is the same, and running faster is a significant benefit.

[danleh]

I didn't look at the Wasm in detail yet, but I think your assumption about function 7460 is correct, at least it contains a bunch of SIMD shuffles, dot products, and adds. WAT excerpt:

...
           local.get $var6
           v128.load offset=16 align=1
           local.get $var8
           v128.load align=1
           local.tee $var73
           v128.const i32x4 0x00000000 0x00000000 0x00000000 0x00000000
           i8x16.shuffle 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3
           local.tee $var78
           i32x4.dot_i16x8_s
           local.get $var71
           i32x4.add
           local.get $var6
           v128.load offset=48 align=1
           local.get $var73
           v128.const i32x4 0x00000000 0x00000000 0x00000000 0x00000000
           i8x16.shuffle 4 5 6 7 4 5 6 7 4 5 6 7 4 5 6 7
           local.tee $var74
           i32x4.dot_i16x8_s
           i32x4.add
           local.get $var6
           v128.load offset=80 align=1
           local.get $var73
           v128.const i32x4 0x00000000 0x00000000 0x00000000 0x00000000
           i8x16.shuffle 8 9 10 11 8 9 10 11 8 9 10 11 8 9 10 11
           local.tee $var75
           i32x4.dot_i16x8_s
           i32x4.add
...

I'll take a more in-depth look tomorrow, i.e., overlap / difference between the Bert and Whisper tasks, and what Whisper does besides the computational kernel.

[danleh]

Regarding the profiles of the Whisper vs. Bert task, it does seem to me that Whisper is substantially "flatter" or more diverse, and subsumes Bert. See:

Whisper bottom-up profile:

Bert bottom-up profile:

So in that light, I'd rather search for ways to make Whisper quicker (although the audio snippet is already pretty short and low number of iterations) and keep that.

In either case, how about merging this since we are generally happy with having this style of workload, and we can still disable and tune in follow-up PRs?

[kmiller68]

LGTM

[kmiller68]

Interestingly, on my M2 Pro Mac I see way more time in 7460 than you're reporting (at least in V8 cli, FF browser, Safari/jsc cli).

Whisper:

Bert:

In either case, how about merging this since we are generally happy with having this style of workload, and we can still disable and tune in follow-up PRs?

Sounds good to me

[danleh]

Thanks, merging! Also interesting difference in the profiles, maybe our x64 backend got more love than arm64, so we should probably look into this in the next weeks/months.