Skip to content

fix: resume audio context #915

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
HardyNLee wants to merge 1 commit into
base: dev
Choose a base branch
from
+99 −59
Open

fix: resume audio context #915

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
118 changes: 63 additions & 55 deletions packages/webgal/src/Core/gameScripts/vocal/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,11 @@ import { getBooleanArgByKey, getNumberArgByKey, getStringArgByKey } from '@/Core
import { IStageState } from '@/store/stageInterface';
import {
audioContextWrapper,
ensureAudioContextReady,
getAudioLevel,
performBlinkAnimation,
performMouthAnimation,
resetMaxAudioLevel,
updateThresholds,
} from '@/Core/gameScripts/vocal/vocalAnimation';
import { match } from '../../util/match';
Expand Down Expand Up @@ -64,7 +66,7 @@ export const playVocal = (sentence: ISentence) => {
return {
arrangePerformPromise: new Promise((resolve) => {
// 播放语音
setTimeout(() => {
setTimeout(async () => {
let VocalControl: any = document.getElementById('currentVocal');
// 设置语音音量
webgalStore.dispatch(setStage({ key: 'vocalVolume', value: volume }));
Expand Down Expand Up @@ -102,64 +104,71 @@ export const playVocal = (sentence: ISentence) => {
stopTimeout: undefined, // 暂时不用，后面会交给自动清除
};
WebGAL.gameplay.performController.arrangeNewPerform(perform, sentence, false);
const finishPerform = () => {
for (const e of WebGAL.gameplay.performController.performList) {
if (e.performName === performInitName) {
isOver = true;
e.stopFunction();
WebGAL.gameplay.performController.unmountPerform(e.performName);
}
}
};

key = key ? key : `fig-${pos}`;
const animationItem = figureAssociatedAnimation.find((tid) => tid.targetId === key);
if (animationItem) {
let maxAudioLevel = 0;
resetMaxAudioLevel();

const foundFigure = freeFigure.find((figure) => figure.key === key);

if (foundFigure) {
pos = foundFigure.basePosition;
}

if (!audioContextWrapper.audioContext) {
let audioContext: AudioContext | null;
audioContext = new AudioContext();
audioContextWrapper.analyser = audioContext.createAnalyser();
audioContextWrapper.analyser.fftSize = 256;
audioContextWrapper.dataArray = new Uint8Array(audioContextWrapper.analyser.frequencyBinCount);
}

if (!audioContextWrapper.analyser) {
audioContextWrapper.analyser = audioContextWrapper.audioContext.createAnalyser();
audioContextWrapper.analyser.fftSize = 256;
}

bufferLength = audioContextWrapper.analyser.frequencyBinCount;
audioContextWrapper.dataArray = new Uint8Array(bufferLength);
let vocalControl = document.getElementById('currentVocal') as HTMLMediaElement;

if (!audioContextWrapper.source || audioContextWrapper.source.mediaElement !== vocalControl) {
if (audioContextWrapper.source) {
audioContextWrapper.source.disconnect();
const isAudioContextReady = await ensureAudioContextReady();
if (isAudioContextReady && audioContextWrapper.audioContext) {
if (!audioContextWrapper.analyser) {
audioContextWrapper.analyser = audioContextWrapper.audioContext.createAnalyser();
audioContextWrapper.analyser.fftSize = 256;
}
audioContextWrapper.source = audioContextWrapper.audioContext.createMediaElementSource(vocalControl);
audioContextWrapper.source.connect(audioContextWrapper.analyser!);
}

audioContextWrapper.analyser.connect(audioContextWrapper.audioContext.destination);
bufferLength = audioContextWrapper.analyser.frequencyBinCount;
audioContextWrapper.dataArray = new Uint8Array(bufferLength);
let vocalControl = document.getElementById('currentVocal') as HTMLMediaElement;

// Lip-snc Animation
audioContextWrapper.audioLevelInterval = setInterval(() => {
const audioLevel = getAudioLevel(
audioContextWrapper.analyser!,
audioContextWrapper.dataArray!,
bufferLength,
);
const { OPEN_THRESHOLD, HALF_OPEN_THRESHOLD } = updateThresholds(audioLevel);
if (!audioContextWrapper.source || audioContextWrapper.source.mediaElement !== vocalControl) {
if (audioContextWrapper.source) {
audioContextWrapper.source.disconnect();
}
audioContextWrapper.source = audioContextWrapper.audioContext.createMediaElementSource(vocalControl);
audioContextWrapper.source.connect(audioContextWrapper.analyser);
}

performMouthAnimation({
audioLevel,
OPEN_THRESHOLD,
HALF_OPEN_THRESHOLD,
currentMouthValue,
lerpSpeed,
key,
animationItem,
pos,
});
}, 50);
audioContextWrapper.analyser.connect(audioContextWrapper.audioContext.destination);
Comment on lines +144 to +147
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

audioContextWrapper.analyser 的类型是 AnalyserNode | undefined。虽然在前面的代码中已经对其进行了初始化，但 TypeScript 编译器可能无法在此处推断出其为非空。为了代码的健壮性和避免潜在的运行时错误，建议在调用 .connect 之前使用非空断言 !。

Suggested change
audioContextWrapper.source.connect(audioContextWrapper.analyser);
}
performMouthAnimation({
audioLevel,
OPEN_THRESHOLD,
HALF_OPEN_THRESHOLD,
currentMouthValue,
lerpSpeed,
key,
animationItem,
pos,
});
}, 50);
audioContextWrapper.analyser.connect(audioContextWrapper.audioContext.destination);
audioContextWrapper.source.connect(audioContextWrapper.analyser!);
}
audioContextWrapper.analyser!.connect(audioContextWrapper.audioContext.destination);


// Lip-sync Animation
audioContextWrapper.audioLevelInterval = setInterval(() => {
const audioLevel = getAudioLevel(
audioContextWrapper.analyser!,
audioContextWrapper.dataArray!,
bufferLength,
);
const { OPEN_THRESHOLD, HALF_OPEN_THRESHOLD } = updateThresholds(audioLevel);

performMouthAnimation({
audioLevel,
OPEN_THRESHOLD,
HALF_OPEN_THRESHOLD,
currentMouthValue,
lerpSpeed,
key,
animationItem,
pos,
});
}, 50);
} else {
logger.warn('AudioContext is not ready, skip lip-sync analyzer for this vocal.');
}

// blinkAnimation
let animationEndTime: number;
Expand All @@ -174,17 +183,16 @@ export const playVocal = (sentence: ISentence) => {
}, 10000);
}

VocalControl?.play();
const playPromise = VocalControl?.play();

VocalControl.onended = () => {
for (const e of WebGAL.gameplay.performController.performList) {
if (e.performName === performInitName) {
isOver = true;
e.stopFunction();
WebGAL.gameplay.performController.unmountPerform(e.performName);
}
}
};
if (playPromise?.catch) {
playPromise.catch((error: unknown) => {
logger.warn('Vocal play was blocked by browser autoplay policy or audio activation state.', error);
finishPerform();
});
}

VocalControl.onended = finishPerform;
}
}, 1);
}),
Expand Down
40 changes: 36 additions & 4 deletions packages/webgal/src/Core/gameScripts/vocal/vocalAnimation.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import { WebGAL } from '@/Core/WebGAL';

interface IAudioContextWrapper {
audioContext: AudioContext;
audioContext: AudioContext | null;
source: MediaElementAudioSourceNode | null;
analyser: AnalyserNode | undefined;
dataArray: Uint8Array | undefined;
Expand All @@ -12,7 +12,7 @@ interface IAudioContextWrapper {

// Initialize the object based on the interface
export const audioContextWrapper: IAudioContextWrapper = {
audioContext: new AudioContext(),
audioContext: null,
source: null,
analyser: undefined,
dataArray: undefined,
Expand All @@ -21,6 +21,34 @@ export const audioContextWrapper: IAudioContextWrapper = {
maxAudioLevel: 0,
};

export const ensureAudioContextReady = async (): Promise<boolean> => {
if (!audioContextWrapper.audioContext) {
const AudioContextCtor =
window.AudioContext ??
(window as unknown as { webkitAudioContext?: typeof AudioContext }).webkitAudioContext;

if (!AudioContextCtor) {
return false;
}

audioContextWrapper.audioContext = new AudioContextCtor();
}

if (audioContextWrapper.audioContext.state === 'suspended') {
try {
await audioContextWrapper.audioContext.resume();
} catch {
return false;
}
}

return audioContextWrapper.audioContext.state === 'running';
};

export const resetMaxAudioLevel = () => {
audioContextWrapper.maxAudioLevel = 0;
};

export const updateThresholds = (audioLevel: number) => {
audioContextWrapper.maxAudioLevel = Math.max(audioLevel, audioContextWrapper.maxAudioLevel);
return {
Expand Down Expand Up @@ -52,8 +80,12 @@ export const performBlinkAnimation = (params: {
};

// Updated getAudioLevel function
export const getAudioLevel = (analyser: AnalyserNode, dataArray: Uint8Array, bufferLength: number): number => {
analyser.getByteFrequencyData(dataArray);
export const getAudioLevel = (
analyser: AnalyserNode,
dataArray: Uint8Array,
bufferLength: number,
): number => {
analyser.getByteFrequencyData(dataArray as any);
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

在这里使用 as any 类型断言是不必要的，并且会降低代码的类型安全性。getAudioLevel 函数的签名已经明确指出 dataArray 的类型是 Uint8Array，这与 analyser.getByteFrequencyData 方法的参数类型相匹配。建议移除 as any。

Suggested change
analyser.getByteFrequencyData(dataArray as any);
analyser.getByteFrequencyData(dataArray);

let sum = 0;
for (let i = 0; i < bufferLength; i++) {
sum += dataArray[i];
Expand Down
Loading