Last week Autodesk hosted some of the Autodesk Expert Elite group members in our sunny San Francisco office. This group of customers are top nominated participants (Expert Elite Nomination Form) in their product assistance to their peers in the Autodesk discussion forums. Scott Sheppard presented to the group on Autodesk Labs and Autodesk Beta and promised to follow up on the topics in blog posts. Scott is posting a blog post on Autodesk Labs (Autodesk Labs versus Beta: Experts Elite Presentation), and I am posting on Autodesk Beta. Beta has long been a passion and responsibility of mine, and is once again a responsibility of mine along with my other Office of the CTO role responsibilities. I was even a Autodesk beta participant for several years as a customer before joining Autodesk.
Autodesk Beta or the Autodesk Feedback Community is pre-release testing community for most every Autodesk product from desktop to mobile and cloud applications. Autodesk Feedback Community is where people can get involved in testing the pre-release products and provide feedback and help us identify the defects before final release. The more people test pre-release betas on different hardware and software configurations as well as test unique workflows to expose issues and feedback, the better the product will be when we ship it.
The Autodesk Feedback Community is also home to some of the future research projects where the product teams can speak confidentially under a non-disclosure with a group of users on possible concepts and ideas to solve user needs and work flows. The members of the community are even used by some teams to recruit for customer visits and in person gunslingers of early versions of software.
Autodesk is rare among large software companies in respect to beta testing engagement and involving it as a vital process across all our products. We could use beta for pre-release marketing activities, but instead we use it to better understand how our products are used, where defects are, and our developers, QA and management personally engage our customers as extended members of our product teams. We want to be as close to the customer as possible to fully understand their needs. After all we are not designing and developing the products for ourselves, but you our customers.
If you want to make a difference in the Autodesk software you use, the Autodesk Feedback Community is the place to be heard & have the potential to affect the software before it is released.
How does one qualify for our beta program?
This is a great question. Basically if you are not prohibited by US banned countries or a competitor looking to see our awesome sauce before we announce it publicly, you are eligible to participate. Each beta is run by the individual product team and some betas can support large numbers of users such as the 30,000 in AutoCAD 2014 but others might be only able to accommodate 200 people so it is up to the team to recruit based on your Feedback Community profile.
Are beta programs available across all products?
Most existing Autodesk products go through a beta cycle in the Autodesk Feedback Community. As I mentioned previously we cover not only the desktop products, but now also cloud and mobile based products.
What is the commitment?
Each betas goals and commitments may vary, but at the core they are all expected to actively download and provide some feedback. If we don’t get the defect report or feedback, we can’t really address the issue in the future version of the software. Those that are accepted into a beta are each expected to be active otherwise they are potentially preventing someone else from being added depending on the number of beta openings for a specific beta. Those that are not active can be dropped from a beta project or not recruited for future betas. The big product betas can be a few months and have a new beta every few weeks and then some betas for extensions and service packs might be a few builds in just 4 weeks. A few betas require that Customer Involvement Program (CIP) data be enabled to collect usage data which is invaluable to the product teams on what features are being tested and problems experienced in what is called MTBF Meantime between Failure which indicates stability. The teams want to have confidence the products have been fully explored and tested and stability is good before they release publicly.
What is the process to sign up?
A person simple goes to http://beta.autodesk.com and signs up creating their profile about them and what they would like to test. In some cases we recruit beta customers using a unique url emailed out, or public links such as the one for Autodesk Fusion 360 and on the page there is a link to get into the beta on the Feedback Community, as well as learn more about the exciting new product.
What does Autodesk do with the feedback collected?
Each beta collects different types of feedback but most commonly defect reports, wish list items, and general feedback and quotes from the participants. All of this feedback is submitted in the Feedback Community and goes into the Autodesk defect system or a database where each submission is reviewed. In the case of defect reports, each is evaluated to see if we can reproduce the issue and perhaps contact the beta participant for more details. If the defect is verified it is rated based on severity, number of occurrences, as-designed, or legacy behavior. The number of people that report an issue or the more severe it is, the more action taken by the QA such as escalation to the developer and in the late stages of beta to an escalation team to review as we don’t want to risk breaking things (regressions) while fixing things in the late stage of beta.
We just finished most of the 2014 product betas over 40 big betas from 3ds Max, Maya, Smoke, Sketchbook Pro, AutoCAD, Revit, Civil 3D, Inventor and many more. Beta season typically starts around August with very small alphas (pre-beta) then big volume beta season in October lasting historically until the Spring.
Risks
Please note beta version programs and files should not to be used without careful consideration that these are unreleased beta products and are not fully supported and may contain software defects.
Rewards
- You get to preview the product before final product is released.
- The delight in discovering bugs that we didn't catch in our in-house testing.
- Enhancing the software for your own use by suggesting possible improvements.
- Your suggestion may add new dimension to our development process!
- We sometimes offer rewards and prizes.
- Most important, it's your chance to contribute to the product's design and features.
Confidentiality
All beta programs are under a non-disclosure agreement and considered confidential which protects both Autodesk and your private and confidential information. During beta no details may be discussed publicly. .
Beta Numbers
Members: Over 65,000 from over 120 countries representing Afghanistan to Zimbabwe. Only 33% are U.S. based.
Beta testing is a critical process to design, building, improving and releasing the best possible products.
I challenge you to get involved and help Autodesk shape the future products.
Cheers,
Shaan
