Archana Vaidyanathan

Fast-moving AI-native product pods needed usability feedback constantly — but researcher availability couldn't scale to match their pace. Teams would either skip research entirely or run ad hoc evaluations without methodological guardrails, producing findings that were inconsistent, biased, or unusable.

The solution wasn't more researchers. It was building a system that embedded research rigor into the process itself — enabling non-researchers to run structured, consistent usability evaluations independently, in the era of AI innovation.

I architected a custom AI-powered usability evaluation tool built on a proprietary framework of my own usability evaluation methods that were successfully used to inform Risk Review and Privacy products with proven impact on product decisions (see Case Study 2). It is also grounded in 10 UXR Building Blocks — covering Task Clarity, First Impressions, Comprehension, Navigation/Flow, Language/Labels, Expectations, Confidence, Error Recovery, Value Perception, and Suggestions.

The system operates across three integrated tabs:

The most important design decision was building anti-bias guardrails directly into the system — because AI-generated research outputs without human oversight can amplify rather than reduce bias.

Multi-Risk Review was Meta's signature platform bet — expanding Privacy Review to encompass additional and ever expanding risk domains such as Integrity and AI risk simultaneously. The stakes were high: a flawed launch could expose Meta to significant regulatory risk and undermine the credibility of the entire risk review process.

I was embedded as the lead researcher from the earliest design phases through MVP launch — responsible for ensuring that the product actually worked for users from various technical backgrounds, roles, and risk contexts.

I designed and led five rounds of multi-level usability research across the full product development lifecycle — from early concept validation through AB testing of launch-ready designs. Each round was carefully scoped to answer the highest-stakes questions at that moment in the product's evolution. In compliance with NDA, product details and insights are not shared; the usability framework used for this research is presented instead.

• Built a structured evaluation frameworkRather than running ad hoc studies each round, I designed a custom usability evaluation framework that allowed the design team to track quality improvements across rounds — making the research cumulative rather than episodic.
• Surfaced counter-intuitive AI team findingsResearch on how AI teams navigate Risk Review escalation processes revealed friction points the product team hadn't anticipated — shifting the team's strategy.
• Connected findings to regulatory stakesFramed every research insight in terms of risk mitigation — not just user experience — making findings directly actionable for Legal and Policy stakeholders.
• Maintained exec visibility throughoutInsights were cited weekly in executive reviews as evidence of user-surfaced risk mitigation progress — bridging tactical usability findings and strategic organizational confidence.

HOVER's app let users take 8 photos of a home to generate a 3D model with measurements — used by contractors, insurance adjusters, and homeowners. As the first and sole UX Researcher, I joined a product with no prior qualitative research, limited budget, and a fast-moving team that needed to see ROI quickly.

The product was built for pros and revenue. The CEO's stance was clear: "Right now we don't have exponential growth via homeowners." All insights came from NPS scores, app store reviews, and sales-driven feature requests. No external qualitative research existed.

1. Understand the value proposition for non-HOVER users (especially homeowners).

2. Observe first-time user experience and frustrations through the end-to-end flow: installation, sign-up & onboarding, photo capture, and 3D model interaction.

• Users struggled to take photosPoor overall SUS scores, low ratings on photo capture, and low confidence in outcomes. Physical limitations of the home combined with limited in-app guidance created a fundamentally broken photo capture UX. Inaccurate photos = failed jobs = customer churn = loss of revenue.
• 42% of jobs failed due to physical limitationsQuantitative data from the dashboard confirmed what qualitative research surfaced: nearly half of all jobs were failing because users couldn't physically capture the required angles of their homes.
• Homeowners interested but low value propositionHomeowners were genuinely interested in the app, but poor overall UX and confusing 3D model interaction resulted in low perceived value. Material menus were unintuitive, and limited customization options made the tool feel incomplete.

Building a research practice from scratch at a startup taught me to move fast without sacrificing rigor. Key learnings: re-strategize with a research roadmap, invest more time in discovery and secondary research, run field studies across the U.S. for demographic diversity, and mobilize & coach a team to scale research beyond a single researcher.

How do people learn complex scientific concepts when they can experience them rather than just read about them? Science is one of the hardest subjects to teach in a traditional classroom — phenomena like geological events, ocean ecosystems, and atmospheric systems require experiencing, not just memorizing.

Virtual worlds offered a provocative possibility: what if learners could step inside a science simulation and explore it the way they'd explore a video game? This question sat at the heart of my M.A. thesis in Intelligent Technologies at Columbia — bringing together my backgrounds in cognitive science, psychology, HCI, and UX research to study learning through embodied cognition in immersive digital environments.

I designed and conducted a usability study of the Raymaker Field Studies Centre at Bowness Island — a virtual science lab in Second Life spanning six distinct learning zones: Weather Zone, Earthquake Simulation, Active Volcanic Mountain, Underwater Coral Reef, River Rapids, and Space Elevator.

• Positive learning outcomes overallParticipants demonstrated content-specific knowledge gains, particularly in the Coral Reef simulation where 4 out of 5 users showed clear learning about eutrophication, sedimentation, and ocean ecosystems.
• High cognitive load for novice usersUsers with no prior 3D gaming or virtual world experience struggled significantly with navigation in early tasks, creating competing cognitive demands that interfered with learning.
• Progressive learning effectTask difficulty decreased steadily as users gained confidence — suggesting that with sustained exposure, the interface becomes transparent and learning moves to the foreground.
• Experiential simulation drives engagementHands-on simulations generated the highest engagement and emotional responses — users were visibly excited and curious in ways passive information displays did not produce.

Virtual world science labs have genuine educational potential — but only when implemented as part of a long-term curriculum rather than a one-time experience. Students need time to internalize the interface before cognitive load from navigation stops competing with cognitive load from learning.

This study was conducted in 2014. The virtual world it evaluated no longer exists in that form. But the questions it raised are more urgent than ever — now playing out in AR, VR, and AI-powered adaptive learning environments.