I asked fellow Autodesk and Portland office colleague Chris Mitchell who manages our very successful Autodesk Inventor pre-release beta programs like the alpha and beta programs for our DLS division a series of 11 questions about the beta. Why 11, because 10 is just so average. My questions are in bold.
Chris can you tell us a bit about your background, role, and how long you have been involved in beta management?
I'll start by going waaaaaaay back; I have a Honours Degree in Mechanical Engineering & a Master Degree in Computer Aided Engineering, both from Nottingham Trent University in England. I am also a Charted Mechanical Engineer through the Institute of Mechanical Engineers. I don't use them much, but I could add BEng (Hons), MEng, CEng, MIMechE after my name. Prior to working at Autodesk, I worked at SDRC/EDS/UGS for 10 years as a consultant on I-DEAS, Metaphase/Teamcenter PLM, & VisMockup. Before that I was a Mechanical Design Engineer at British Steel, designing/analyzing production equipment for the hot rolling process. That's where I first got a real taste of "all things CAD". I've been at Autodesk for 10 years now, initially as a lead Product Designer for Inventor, then the Product Design Manager & more recently a lead member of the Customer Engagement/QA team. I've been involved with "Beta" for all 10 years; initially we hosted the Inventor Design Review Forum (IDRF), which was an avenue to gather direct feedback on design specs from ~ 50 hand-picked Inventor experts. That process was really a pre-cursor to many of the User Experience/Usability/Beta activities which are still performed today. The IDRF gradually morphed to become a Customer-Centric feedback program, & then Beta & more recently the on-going Alpha/Beta program. An integral part of these activities were the face-to-face "Gunslinger" test events, and Autodesk Application Engineer Summits, which our team still leads today. So really I've been involved with "Beta" activities for 10 years, but really the "owner" for the Inventor Beta for last 6 years. I also acted as the overall DLS division Beta owner for the Mechanical Design products in a consulting role for the last 4 years.
Can you please tell us which Autodesk products you manage in your Autodesk DLS division in alpha and beta? As a follow up what the heck does DLS mean for those unaware?
My primary role by a considerable margin is to manage the Inventor product Alpha/Beta's; more recently I've actively contributed to the management of Beta's for all Design, Lifecycle & Simulation Lifecycle & Simulation (DLS), products, (Fusion 360, PLM 360, Factory & Product Design Suite, Alias, HSM, Simulation, Vault), as well as assisting with other Autodesk Beta's such as AutoCAD, Labs, etc.
About how many customer participants are in the Autodesk DLS beta programs?
Currently over 11,500 customers, students, resellers & 3rd party developers have access to the various DLS Alpha/Beta's; the active participation is generally about 20% depending on the specific point in the release cycle.
I know beta season has been evolving from a Winter to Spring event to a year around activity, can you tell us why this change is occurring and the benefit?
For several of the DLS products we have moved to a continuous Alpha/Beta program; instead of waiting until we have testable code in the October timeframe, we now share plans, concepts, & videos well before code is actually written. This enables us to gain early validation to ensure we're on the right track while changes are still relatively cheap. Of course we can't respond to every request but we're certainly listening & more importantly reacting much more readily. Once plans/concepts have been validated then the natural progression through Alpha occurs, (yes, sometimes these are somewhat rough), & then on into more traditional Beta were the focus is not on what/why but more the how & how well with a definite emphasis on quality. At the end of the day we're trying to engage earlier & more frequently than ever before; this results in a much deeper understanding & appreciation of our customers with the ultimate result being a better product. The ongoing Alpha/Beta has definitely increased our workload to a certain degree but it is something that is fully supported throughout the various levels of Inventor leadership.
How important is customer feedback to the DLS product teams and quality of the products?
This is of paramount importance & infinite value; we've obviously seen noticeable improvement in hard metrics such as mean time between failure, & customer satisfaction/NPS but the real value comes from the complete workflows that we now strive to deliver which leads to better functionality & improved competitive advantage. All members of the Inventor product team are strongly encouraged to support the Alpha/Beta process & activities. Without the broad/varied/in-depth feedback from the end users this would not be possible. In many ways the broader & more public nature of the process now ensures that the Inventor team cannot hide from an issue until it's too late; we have to react since we've invited the feedback & now we're "over a barrel" to deliver. The feedback is not the end-all though; through the Beta program we have forged some very strong customer relationships which we leverage for a wide variety of activities. Metrics & more importantly customer quotes/anecdotes a regularly fed into the Inventor leadership team often all the way up to Buzz' (Robert “Buzz” Kross SVP DLS) level. Metrics from the various Beta surveys have traditionally been an important part of the "release readiness" discussions. To me personally, the Alpha/Beta process is the most important thing we do; it's mutually beneficial to everyone, not to mention being extremely enjoyable & gratifying for me.
You have been very thorough in the betas you manage and even providing status to the custom on their reported defects, can you tell us how this works, and why you are doing it?
Yes, we allow customers to log Problem Reports which once validated are automatically pushed directly to our internal defect tracking systems. This expedites much of the testing process using real world use-cases & data & has been invaluable to our QA teams. The process enables all interested parties to be kept abreast of all developments & we openly share all feedback submitted with all members of the projects. Voting & context sensitive search prevents the creation if duplicates too. Once a defect has been investigated & ultimately fixed we use a closed loop notification process to inform & Thank the submitter so that they know when an issue has been addressed. Ultimately this is why many participate in Beta testing, so it's paramount that we provide good notifications when issue are actually resolved. We also proactively support the logging of Enhancement Requests; these used to be captured as part of the Beta system but more recently we have integrated the process to the new IdeaStations which we promote at every opportunity. Much of the process around IdeaStation stemmed from the tools/techniques which we used to use as part of Beta.
Can you explain how you encourage voting on reported issues and discussions during alpha and beta?
All Problem Reports are open to all, so whenever anyone experiences a product issue it's easy for issues to be searched on & anyone can vote on the relative importance & duplicate nature of an issue. We assign weighting values to votes, duplicate matches, comments, views so that we know which issues are most important to the broader community as opposed to investigating each issue singularly. I actively participate in some of the future development/validation of the Centercode Beta platform itself, so that functionality such as this is available to the Beta audience with a view to improving the overall experience. We've tried to make the actual testing of Inventor as easy as possible so that there are no barriers to participation. We leverage cloud hosted or containerized products so that testing can be done alongside normal production versions without "contamination"; this is not a cheap endeavor for us, but is easily justified by the rich feedback we receive. I believe that the Inventor team is unique in that we also proactively support the use of Beta2 onwards for full production use, & guarantee the future migration of all files. At he end of the day this is the best way to get the really important feedback. Additionally we regularly host webinars to help to explain the new Alpha/Beta functionality & provide videos to compliment all major functionality to speed the understanding so that it's easier to provide feedback.
Can you mention a few key enhancements and possible defect catches made as a result of beta feedback for Autodesk Inventor Feedback?
To me some of the most important product changes as a result of Alpha/Beta are not always what was done, but actually certain things which were slated for release but were deemed insufficient. Over the years I've noted more of an acceptance to remove functionality which was not ready, as opposed to prior practice of shipping at all costs, sometimes resulting in "half-baked" workflows. Several years ago as part of a beta for the Inventor ribbon we eventually conceded & shipped 2 sets of icons to alleviate the concern of the Autodesk "Smurf-blue" color scheme, & removed all traces of a new 3D controller driver which "just didn't feel right", (to the shouts of "Power to the People"). Feedback from the Inventor Beta persona surveys was also leveraged in deciding to return Showcase to the Product Design Suite. It's too difficult to note the most important defect since our user base is so varied; what's important to one is irrelevant to another, but let it be noted that during every Inventor Alpha/Beta cycle we will generally exceed 1,000 Problem Reports. All those which are deemed to be "stopship" are addressed, along with the majority of P1's. The Inventor team is currently engaged on 4 week sprints followed by a "Customer Defect Fixing Week" - one of the the primary inputs to what is addressed during the CDW's is the feedback from Alpha/Beta.
Would you say there was an outstanding participant that by their contributions in the Inventor 2016 Beta that deserves a shout out and a public thank you?
I would say that currently there are approximately 50-75 very key members of the Inventor Alpha/Beta who can be relied on at all times to provide the appropriate feedback; I won't name names but they know who they are.
What advice would you have for beta participants and future participants on how to write the most helpful feedback and defect reports that the team can use and take action on?
Firstly just tell us what you're thinking; this can be through Problem Reports or discussion forum topics. However you get the information submitted just ensure you speak up. Despite all the progress we've made we still have a lot of "lurkers" (I know from the system logs), so if you like something or hate something make sure you tell us. We've had occasions in the past where functionality was released & was not quite ready for prime-time & if we'd had more feedback during Alpha/Beta then that may not have been an issue. If you do decide to submit a Problem Report, please ensure to search for duplicates first, vote on others' reports, & then log a new issue. Always provide as much details possible & provide an example workflow & dataset if possible. This can be time-consuming but will ultimately result in improved product quality.
Can you tell us the sign up process to get involved in an Autodesk Inventor beta?
Anyone can apply for the Inventor Alpha/Beta project. Just click on https://bit.ly/InventorBeta - you'll be prompted to complete an Inventor background usage survey & then your application will be reviewed before you're granted access. This is important as it helps us to better understand the user persona of our participants.
Bonus: What is the funniest defect or feedback report you have ever seen during Beta that actually made you shake your head or laugh?
I laugh on a daily basis at some of the feedback we receive; this is generally not laughing at the submitter or feedback itself, but more internally here because it often resolves "disputes" about how important something is or how something should look/work, or why something should (not) be in the product; often we've been so close to it, we couldn't "see the wood for the trees" ! I quietly say Thank-you numerous times every day when a point has been "proven"
Thank you Chris!
We all appreciate your efforts in making sure Autodesk is listening and actively engaged with our customers issues and needs.
-Shaan
