GardenMate Ecosystem
Custom Vision Transformer AI + Viro AR mobile platform with 12+ modules.
Personalized onboarding and secure multi-role authentication
System Overview & Engineering Longevity
A full-stack mobile ecosystem engineered for React Native and Expo SDK 54, integrating spatial augmented reality, custom deep learning computer vision, and a 4-tier commerce architecture. At its algorithmic core sits a custom fine-tuned Vision Transformer (ViT-B/16) model combined with the PlantNet botanic API in a weighted 60%+40% ensemble decision pipeline. Users can test-place photorealistic 3D botanical models in physical environments using Viro React AR plane detection before purchasing via Stripe, while vendors leverage camera AI to auto-populate marketplace listings with full botanical specifications in seconds.
Key Engineering Highlights
Architected with clean domain separation, strict TypeScript contracts, and automated telemetry logging.
AI Inference Time
FastAPI microservice inference benchmark
ViT + PlantNet Fusion
Weighted confidence decision engine
AR Plane Confidence
Horizontal surface detection stability
Integrated Modules
Comprehensive mobile application ecosystem
End-to-End System Data Flow
How user inputs, computer vision pipelines, and transaction protocols flow across the stack.
Camera Frame Acquisition
User or vendor captures plant leaf or flower via camera; frame is downscaled and normalized into an RGB tensor.
60/40 Dual-Inference Ensemble
Image tensor is evaluated in parallel with PyTorch ViT-B/16 (60% weight) and PlantNet API (40% weight).
Confidence Fusion & Auto-Fill
Softmax probabilities are aggregated; when confidence exceeds 75%, botanical specifications are automatically populated.
Spatial Plane Mapping
Viro React initializes 3D coordinate planes on horizontal ground surfaces to anchor 3D GLTF models.
Stripe Settlement & Commission
Order is finalized via Stripe; 10% platform commission is routed to admin ledger and 90% is settled to vendor wallet.
System Capabilities & 12+ Functional Modules
Hybrid ViT + PlantNet AI Ensemble (60% / 40%)
A dual-inference pipeline combining a fine-tuned PyTorch Vision Transformer (ViT-B/16) with the PlantNet botanical API. The system uses a 60/40 weighted confidence fusion: 60% probability weight is allocated to the local ViT classifier and 40% to the PlantNet API to improve classification robustness across local and less common flora.
AR Spatial Visualizer ("Place Before You Buy")
Utilizes Viro React spatial computing to detect real-world horizontal surfaces (tabletops, floors, patio grounds). Renders 3D plant models with dynamic shadows, 360-degree rotation, scale lock, and surface confidence scoring (e.g. Tabletop · 85%).
Vendor Snap-to-Autolist by Image
Eliminates manual catalog entry for plant nurseries. Merchants snap or upload a plant photo; the AI ensemble identifies the species (e.g. Tomato Plant · 75% confidence) and automatically populates botanical name, care requirements, sun exposure, watering cycle, and suggested price range.
Vendor Autolist by Name
Merchants typing a plant name (e.g. "Golden Barrel Cactus") trigger an automated knowledge lookup that populates scientific taxonomy, recommended soil type, growth rate, and watering intervals without requiring manual research.
Care Engine & Smart Reminders
A centralized plant care management system that generates individualized schedules for watering, fertilizing, repotting, and pruning for each plant in the user garden.
Multi-Vendor Plant Marketplace
A marketplace connecting regional plant nurseries with plant enthusiasts. Features categorized discovery (Indoor, Outdoor, Flowers, Fruits, Deals), search filters, and AR preview badges.
Multi-Item Cart & Stripe Checkout
Shopping cart supporting multi-vendor purchases, regional delivery fee calculations, free delivery threshold progress indicators, and secure Stripe payment processing.
Order Dispute & Photo Proof Portal
A dedicated ticket-based dispute resolution system for live plant deliveries. Customers receiving damaged pots or wilted plants can file a ticket, upload photo evidence, and chat directly with the vendor for resolution.
Executive Admin & Commission Engine
Central administrative portal for platform governance. Monitors collected commissions (10% platform take-rate), pending vendor disbursements, active vendor counts, total users, and open complaint tickets.
Vendor Wallet & Settlement Ledger
Merchant portal displaying revenue analytics, order status progression, weekly payout settlements, and balance withdrawal requests.
Gardening Community & Consultation
Interactive social hub where plant owners share progress pictures, ask botanical questions, exchange tips, and request plant health diagnoses from experienced botanists.
4-Tier Role-Based Access Control
Architectural segregation separating Customer, Vendor, Delivery Rider, and Administrator interfaces with Firebase Auth and scoped Firestore security rules.
Key Engineering Bottlenecks & Solutions
ViT Model Latency on Mobile Devices
Running a full Vision Transformer directly on mobile hardware caused high latency and battery drain. Engineered a hybrid approach where lightweight image preprocessing runs on-device, while inference is dispatched to an asynchronous FastAPI microservice, maintaining response times around 350ms in testing.
AR Plane Jitter on Low-Contrast Surfaces
Standard plane detection struggled with low-contrast textured surfaces like carpets. Implemented a spatial confidence filter that locks coordinate axes only when surface feature tracking confidence exceeds 80%, providing improved plane stability on low-contrast surfaces.
Classification Variance Across Diverse Flora
Standalone models suffered from class imbalance on rare plant varieties. Architected a 60/40 weighted confidence fusion: the fine-tuned PyTorch ViT handles common regional species, while the PlantNet botanical API acts as a knowledge fail-safe.
Vouch Keychain
Cryptographic QR item recovery & real-time telemetry.