Goal Clarity

• Polaris goals changed throughout the project.  Feasibility of achieving the goals was not questioned with the goal changes

• Project was set up with a ‘wave’ approach but team was only able to execute Wave 1 before the goals were changed

• Initiated with best in class consumer experience intending to make up lower performance in lead submissions by higher performance in lead quality

• At launch with leads drop, shifted to focus to leads and then shortly after proposed revenue parity (and possibly profit parity)

• No potential tradeoffs were provided by the business (ex would be willing to accept fewer leads in exchange for higher engagement) = unclear success criteria beyond “parity or better” 

• Too many goals were focused on 

• Tech uplift, redesign, better consumer experience, revenue parity, media uplift

Measurement

• Limited data / unreliable data created large overhead 

• Project started without a dedicated data resource

• Lack of data integrity is an issue across the board (ex leads disconnect in clickstream vs LCS table, leads appearing in one but not the other)

• Low traffic introduced potential noise (high proportion of rental leads) for some segments, invalidating data reliability 

• Risk in making large decisions on limited / poor data

• Action Plan needed to help all teams / universal issue

• Individual metrics are measured differently (ex leads)

• Individual metrics are hard to measure (media revenue)

Goal Alignment within Leadership

• There were conflicting instructions from ELT members / misalignment within ELT 

• Team members heard different information from various ELT members (ex whether certain groups were stakeholders, how to involve stakeholders)

• During the 1.5 years of the project we had 5 variations of a CPO

• Conflicts on various goals were not escalated for ELT alignment (ex telling teams to “not worry about revenue”)

• Sponsoring ELT members were not aligned with other ELT members on success criteria / importance of the project

• Support from ELT decreased over time with loss of sponsoring leaders and presence of new projects (Native #1)

• More involvement may have helped drive alignment and success criteria within ELT

Experiment Approach

• Big Bang approach led to difficulty in attributing success / failures to particular hypotheses within the design 

• Risks entire project instead of winning portions of project 

• Difficult to design with the big bang approach for an experience of this size with only two designers who had minimal historical knowledge 

• Market approach introduced challenges

• Market and audience caveats (ex exclusion zips, no Choice, no non-deeded homes) introduced bias data (data may correlate with nature of Pure markets instead of new page design)

• High investment in Pure markets for the future of RDC 

• Partnership with Optimizely and creative Optimizely setup

• Team pushed Optimizely boundaries to identify best set-up with highest flexibility in feature releases and coordination with other teams’ features

Collaboration

• Collaboration model with stakeholders improved over time

• Initially project was developed in silo which had to be backtracked prior to launch

• Involving stakeholders in regular check-ins and understanding their objectives helped upskill entire team in pursuing page needs

• Could create more scheduled milestone check-ins with ELT / Stakeholders

• Monthly stakeholder updates helped but did not ELT audience

• Resourcing for contributing teams was often side of desk or unavailable (ex Research, component owners on LDP) 

• Squad performance improved after partnership with offshore team Cogent

• Very difficult to collaborate with the Cogent team based in Australia

• Team had little to no overlap of working hours

Development Execution

• Team was able to pivot on changing goals and deliver

• Tremendous success in focusing on key metrics and incrementally improving them 

• Squad ceremonies matured

• Growth in team ownership as functional quad in addressing challenges

• Strong focus on team morale, resilience and ownership

• Under-estimated some work / distractions

• Tech incidents, lead form development, working with team in Australia

• Cost of not having a PM or EM at stages of the project

Design Execution

• Team was able to deliver a new innovative experience

• Massive body of work was created by only two designers and only one for about 4 months — met all expectations for speed and quality of execution

• Foundational design components were contributed to the design system

• Became leaders in cross functional collaboration 

• Created a successful cadence and framework for design teams to collaborate

• Distracted by research and other non-design related work

• Due to the lack of research and data resource the team had to conduct their own user tests and distill the findings

• Native #1 was a distraction

• Director level support was not always available due to the N#1 Big Bet

While Polaris was in its final stages, I was moved over to the company's new priority, Native #1, a visual uplift of our entire native experience.