Skip to content

feat: wildlife re-ID pipeline for offline species identification#2

Open
JasonWildMe wants to merge 28 commits intomainfrom
wildme
Open

feat: wildlife re-ID pipeline for offline species identification#2
JasonWildMe wants to merge 28 commits intomainfrom
wildme

Conversation

@JasonWildMe
Copy link

Summary

Adds a complete offline wildlife re-identification pipeline to Off Grid Mobile, enabling field researchers to photograph animals, detect individuals via YOLO object detection, generate MiewID v4 embeddings, and match against pre-built species embedding packs — all without internet connectivity.

Key additions:

  • Wildlife type system (species configs, detector configs, embedding packs, observations)
  • ONNX Runtime inference service (stubs with TODO(P0) for real tensor ops)
  • Detection preprocessing + YOLO postprocessing with NMS
  • MiewID v4 embedding extraction + cosine similarity matching
  • Pack manager for loading/indexing pre-built embedding databases
  • Wildlife Zustand store with AsyncStorage persistence
  • Full navigation structure: Home, Packs, Capture, Detection Results, Match Review, Observations, Sync
  • Merged embedding database builder (pack + local individuals)
  • App startup initialization for packs and wildlife store
  • Integration tests for full pipeline flow and pack loading
  • Stripped unused Off Grid modules (LLM, image gen, voice, tools, chat) to reduce bundle size

Type of Change

  • New feature (non-breaking change that adds functionality)
  • Refactor (code change that neither fixes a bug nor adds a feature)

Screenshots / Screen Recordings

N/A — no device testing yet; screens are structurally complete but require real ONNX inference wiring before visual validation.

Checklist

General

  • My code follows the project's coding style and conventions
  • I have performed a self-review of my code
  • I have added/updated comments where the logic isn't self-evident
  • My changes generate no new warnings or errors

Testing

  • I have tested on Android (physical device or emulator)
  • I have tested on iOS (physical device or simulator)
  • I have tested in light mode and dark mode
  • Existing tests pass locally (npm test) — 408 tests, 29 suites
  • I have added tests that prove my fix is effective or my feature works

React Native Specific

  • No new native module without corresponding platform implementation (Android + iOS)
  • No hardcoded pixel values — uses SPACING / TYPOGRAPHY constants from the theme
  • Styles use useThemedStyles pattern (not inline or static StyleSheet.create)
  • Large lists use FlatList / FlashList (not .map() inside ScrollView)

Performance & Models

  • File paths are resolved correctly on both platforms (no hardcoded / vs \\)
  • Large files (models, assets) are not committed to the repository

Security

  • No secrets, API keys, or credentials are included in the code
  • User input is validated/sanitized where applicable

Related Issues

This implements the wildlife re-ID feasibility plan documented in docs/plans/2026-02-25-wildlife-reid-implementation.md.

Additional Notes

  • ONNX inference methods are currently stubs (marked TODO(P0)) — they return empty tensors. Wiring real image-to-tensor conversion and ONNX session execution is the next step.
  • The sync screen shows a queue UI but sync functionality is not yet implemented (shows an alert).
  • Theme palette now includes statusSuccess, statusWarning, statusError for dark-mode-aware status indicators.
  • DetectorConfig loading falls back to safe YOLO defaults when pack config isn't available.

JasonWildMe and others added 28 commits February 27, 2026 21:13
…nd design doc

Three documents capturing the design for bringing Wildbook's MiewID algorithm
to mobile for offline individual animal re-identification:

- WILDLIFE_REID_FEASIBILITY.md: Platform evaluation and technical research
- EMBEDDING_PACK_FORMAT.md: Detailed spec for the embedding pack zip format
  including Wildbook exporter implementation guide
- plans/2026-02-25-wildlife-reid-design.md: Full approved design covering fork
  strategy, data model, ML pipeline, sync protocol, screens, and MVP scope

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Detailed bite-sized implementation plan covering 6 phases:
- Phase 0: Project setup (ONNX Runtime, type definitions)
- Phase 1: Core services (ONNX inference, embedding matching, pack manager)
- Phase 2: State management (wildlife Zustand store)
- Phase 3: Pipeline orchestration (detect → embed → match → save)
- Phase 4: Navigation and screen scaffolding (8 screens)
- Phase 5: Integration and end-to-end wiring
- Phase 6: Testing, E2E flows, and cleanup

Each task follows TDD with exact file paths, code, test commands, and commit steps.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ch, pack manager)

- OnnxInferenceService: wraps onnxruntime-react-native for model loading,
  detection stubs, and embedding extraction stubs
- EmbeddingMatchService: cosine similarity computation with top-N matching
  against multi-embedding database entries
- PackManager: embedding pack lifecycle (init dirs, load manifest, parse
  binary embeddings, manage individuals, delete packs)
- Add onnxruntime-react-native mock to jest.setup.ts
- Add barrel exports in services/index.ts
- Fix pre-existing test timeouts in ModelDownloadScreen and
  PassphraseSetupScreen (increase per-test timeout to 20s)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The JS/TS section was running `npm test` which chains jest, test:android,
and test:ios. Native tests are already run by their own hook sections
when Swift/Kotlin files are staged. Running them redundantly in the
JS/TS section causes failures when native toolchains are unavailable.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Also fixes flaky test timeouts under WSL2:
- Increase Jest testTimeout from 10s to 60s
- Limit pre-commit test workers to 50% of CPUs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ndividuals

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ky failures

- Add accuracy field to ProcessPhotoParams.gps for consistency with Observation type
- Fix jest.setup.ts setTimeout(10000) overriding jest.config.js testTimeout(60000)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace chat/models navigation with wildlife-specific navigation:
4 tabs (Home, Packs, Observations, Sync) plus root-level stack
screens for Capture, DetectionResults, MatchReview, ObservationDetail,
PackDetails, and Settings. Legacy navigation types preserved for
existing screens until Task 6.4 cleanup.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the PacksScreen placeholder with a full implementation that lists
downloaded embedding packs from the wildlife Zustand store. Each pack card
shows species display name, individual count, formatted export date, and
storage size. Tapping a pack navigates to PackDetails. Empty state shows
a "No Packs Downloaded" message. Includes 17 RNTL tests covering empty
state, card rendering, multiple packs, navigation, and testIDs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the CaptureScreen placeholder with a full directory-based screen
that allows users to take or select a photo, runs the wildlife detection
pipeline, saves the observation to the store, and navigates to the
DetectionResults screen.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace placeholder ObservationsScreen with full implementation featuring
filter bar (All, Pending, Reviewed, Synced), FlatList of observations
with thumbnails, timestamps, detection counts, and review status. Adds
19 RNTL tests covering rendering, filtering, navigation, and empty state.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… observations

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace hardcoded device info with Platform.OS/Version and add
getDeviceLocation helper (returns null for now) so GPS flows through
the pipeline and observation consistently. Tests verify both fields.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When approving a match to an existing local individual, call
addEmbeddingToLocalIndividual to add the detection's embedding and
cropped image URI to the individual's profile, building a richer
representation over time. Pack individuals are unaffected.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ading

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…hat)

Remove all legacy Off Grid AI chat modules that are no longer used by
the wildlife re-identification app. This includes:

- Services: LLM, image generation, voice, whisper, tools, model manager,
  active model service, intent classifier, document/PDF extraction,
  background download, HuggingFace browser, CoreML browser
- Screens: ChatScreen, HomeScreen, ModelsScreen, ModelSettingsScreen,
  DownloadManagerScreen, GalleryScreen, Projects screens, VoiceSettings,
  StorageSettings, DeviceInfo, ModelDownload
- Components: ChatInput, ChatMessage, ModelCard, ModelSelectorModal,
  GenerationSettingsModal, MarkdownText, VoiceRecordButton, DebugSheet,
  ProjectSelectorSheet, ToolPickerSheet, ThinkingIndicator
- Stores: chatStore, projectStore, whisperStore
- Hooks: useVoiceRecording, useWhisperTranscription
- Types: whisper.rn.d.ts, legacy model/chat/image types
- Constants: models.ts, LLM-specific constants
- Utils: coreMLModelUtils, messageContent
- All associated tests (contracts, integration, unit, RNTL)

Updates:
- App.tsx simplified to only initialize pack manager and hardware
- appStore trimmed to theme, onboarding, and device info
- SettingsScreen shows only Security settings
- Navigation types cleaned of legacy routes
- OnboardingScreen navigates to Main after completion
- hardware.ts stripped of model recommendation code
- Barrel exports updated for all modules
- Test helpers and factories stripped of legacy references
- Coverage thresholds adjusted (70/60/70/70) for post-strip baseline

Kept: wildlife pipeline, ONNX inference, embedding match, pack manager,
auth, hardware service, all wildlife screens and tests.

29 test suites, 408 tests passing. TypeScript compiles cleanly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… cleanup)

- Add statusSuccess/Warning/Error to theme palettes for dark mode support
- Replace hardcoded hex colors in BoundingBoxOverlay, SyncScreen, CandidateCard
- Fix critical DetectorConfig empty cast with loadDetectorConfig + safe defaults
- Remove unused gps field from ProcessPhotoParams (kept in Observation)
- Add TODO(P0) comments to ONNX inference stubs
- Type CandidateCard styles prop properly
- Update tests with required mocks for packManager and embeddingDatabaseBuilder

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Register SecuritySettings route in root navigator
  (P1: tapping Security in Settings previously dispatched unhandled nav action)
- Add PackDetailScreen showing pack species, feature class, individual count
- Point PackDetails route to PackDetailScreen instead of PacksScreen
  (P2: tapping a pack previously just pushed another list copy)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant