Written by

Abner Germanow

Date published

February 10, 2025

“The State of [insert category]” research is now a multi-year investment staple for technology vendors serving developers and software company builders. In the early days, quantitative analysis of cloud adoption and developer behaviors was limited to the 2013 RightScale (now Flexera) State of the Cloud survey, Puppet/DORA/SODAR State of DevOps study, and New Relic’s 2009 language adoption behaviors.[1]Today, the firehose of surveys is constant.

When I began collecting studies in Notion database in 2021, I didn’t realize the research volume on building software had skyrocketed. That database now has over 250 studies focused on building software businesses. While some vendor-sponsored studies are laughably biased or suffer from reality distortion fields, most provide helpful insights. I’d even argue overall quality is on the rise.

This post will cover the five methodologies used in vendor-sponsored quantitative research, plus a sixth section on private company valuations and compensation.

Please submit studies or tag me on the socials.

Why do technology vendors invest in research and analysis?

After decades of software development and 15-ish in the cloud, building software companies remains an art crafted inside a hype tornado. Quantitative research aspires to ground customer conversations and to illustrate topics vendors want customers and prospects to prioritize.

Shameless plug: Are you a technical leader of a software business with under 200 engineers and developers? Please take this short survey on a community and collective research effort tailored to advancing your craft outside the hype tornado.

Table of contents and TLDR:

• The disruptor: Behavioral analysis of SaaS platform data — Behavioral data is nifty, even if the demographics are iffy.
• The granddaddy: Form fills — If perception is reality, game on, but when results and analysis feel fishy, drill down.
• Digital water coolers: Peer reviews, evaluations, and tests — Context into the business requirements, resource constraints (or not), and tech stack make all the difference.
• Hot air with one exception: Consultant commentary —The Oscar goes to Thoughtworks Radar.
• Toil on your behalf: LMGTFY meets APIs — Could you gather all the data for your own evaluation? Yes. If someone else packaged it nicely, would you use that instead? Of course.
• Bank on it: Investment deal and compensation data — Know when to hold, raise, and run.
• Help! People to follow and parting thoughts: Send me research. Data nerds. The quality bar continues to rise.

Note: Quadrants, marketscapes, waves, ROI/TCO/TEI, investment research, and market share studies published by analyst firms are not part of this. Perhaps another time.

#1 — The Disruptor: Behavioral Analysis of SaaS Platform Data

Behavioral data is my favorite data source because the data source rarely suffers from ego, faulty targeting, or poor incentives.

Methodology:

Software-as-a-service (SaaS) product managers have a massive advantage over their packaged software brethren in product usage metrics to analyze and monitor customer behaviors. Packaging that data into an analysis of technology, product, or feature usage makes for unique and insightful content.

Strengths:

When a SaaS business reaches sufficient scale, the customer population can be a proxy for broader market behaviors. Even a small sample can provide a reality check to validate or refute buzzy speculation. For example, in 2015, Docker containers were as buzzy as infrastructure technology can get. However, Docker only had anecdotal evidence of how containers were being used. With New Relic’s first container monitoring service and 300+ customers, we had data to ground container buzz in reality.

New Relic’s analysis of container lifetimes shed some light on the question: “Are containers creating new application architectures?” This data opened numerous customer, press, and analyst doors we previously couldn’t access.

Constraints, pitfalls, and harsh realities:

The lack of demographic context is the biggest downside of SaaS behavioral analysis. While I’d like to see companies share or at least reflect on the attributes of their customers generating the usage data, I’m not holding my breath. However, if you are a good-sized customer or prospect, demographics are good questions to ask partly for analysis context but mostly to understand how your company fits into their support and scale capabilities.

Behavior data doesn’t suffer from ego, but even ordinarily level-headed product managers, engineers, and product marketers can find what they want to be true when analyzing their data. The best anecdote to this bias occurs when a vendor shares the data with customers or analysts like Redmonk and then publishes the research with customer commentary to illustrate a customer’s view on the why behind the data. [2]

Another pitfall of behavior data is the assumption users are using the product within the analysts’ Overton window of product usage. For example, the emergence of new application architectures was a big question in the earlier container analysis. One clearly innovative application used 100,000+ containers a day. We approached the customer, who promptly thanked us for pointing out a flaw in their automation system, launching an extra 99,990 containers per day. Whoops.

Finally, a vendor’s Master Service Agreement typically constrains the data they can publish. Customer data is obviously off-limits, but aggregated customer activity data or product function data that helps advance the development of the service is typically inbounds. If you know a vendor has a dataset that oddly isn’t shared in an analysis, MSA limitations might be the answer. [3]

Behavior Data Examples:

Cloud service and technology usage behavior analysis:

Cloud service providers are famously tight-lipped on usage statistics, but monitoring companies have the next best seat to analyzing software team technology adoption behaviors. New Relic was the first to publish platform data on language, cloud, and container trends. Now, DataDog, Dynatrace, and many others regularly share data on usage behaviors of cloud services, Kubernetes, containers, and every step in the software development lifecycle.

Cloud service spending behavior estimates by company:

Intricately analyzes network traffic to estimate cloud service usage and spending for go-to-market teams targeting companies with particular cloud spending profiles. Before being acquired by HG Insights, Intricately published data from their product. After a hiatus, it appears they have restarted that effort.

Cloud service and SaaS spending trends and benchmarks:

FinOps companies are also analyzing their dataset with a different perspective from Intricately/HG Insights and monitoring companies. FinOps companies have more precise spending data than Intricately, but they are in the opposite business of identifying cloud spenders. They can’t and shouldn’t sell the names of their customers. Recently, Vantage has been sharing lots of platform data. They also pull data from the AWS API to present AWS product data in clever ways and have expanded outside of infrastructure into spending on monitoring services.

On the SaaS spending front, Cledara analyzed SaaS buying and renewal habits, and Vertice shows trends in purchases, usage, and cancelations. Slick.

Developer-focused people and vendors may be inclined to skip business SaaS analysis, but the Vertice data shows a spike in dev tool cancelations in the last quarter.

Threat and breach analysis

One of the pioneers of product data analysis is the Verizon Data Breach Investigations study. Now in its 17th year, the study incorporates breach data from multiple vendors and services outside of Verizon. While security hype is often focused on the latest threats (AI), this study grounds the security world in the extent of the problems observed in the wild. 5 stars.

Behavioral Intent Clickstream Data

Finally, one of the newer sources of behavioral data comes from media and education companies. O’Reilly and A Cloud Guru, now Pluralsight, have shared data on their readers’ and students’ content consumption trends. I find this data a nice reality check on the differences between what people search, consume, and post about versus where they spend time learning with training and tutorials.

A special hat tip to Mike Loukides at O’Reilly for three, four? years of analyzing and publishing their reader behaviors and his monthly radar column covering reflections and TLDR news.

For many more examples, check out the Behavior Data Studies view in the survey collection.

[top]

The Perception Granddaddy: Form Fill Surveys

Methodology:

Get people to fill out questionnaires, then analyze the results.

What you get:

While frequently presented as a reflection of reality, form-fill surveys actually test the respondents’ perception of the question, their understanding of terms, what they can recall at the time, and the survey owner's ability to target and convert form-fillers with the desired knowledge and demographics.

Strengths:

If perception is truth, most questionnaire-based surveys provide a lens into what respondents believe to be true and what they believe or desire will happen next.

For those familiar with the DORA “State of DevOps” study, it is, and always has been, an outlier. Combining surveys with detailed assessments and connecting those results with the financial results of the respondent’s business. On top of identifying the habits of high- and low-performing teams, Dr. Nicole Forsgren was the first to have data proving that being better at software translated into a better business. Most vendor-sponsored surveys can’t be nearly that declarative, but some go above and beyond the rest.

Weaknesses:

There are many, but they typically boil down to failures in converting the correct survey targets, poor question choices, or mismatches between a question and the analysis of the responses. Those issues are compounded by respondents answering questions “the right way” instead of reflecting reality. For example, I once fielded a survey to network administrators. Even though 10G Ethernet had only been shipping for a quarter, an impossible 25% of the respondents said they had deployed it! These respondents simply chose the “biggest” option. Ignorance? poor question wording? bad targeting? Probably all of the above.

Red flag - Demographics and sample sizes:

The worst behavior I see regularly in vendor-sponsored surveys is hiding or omitting a survey’s demographics or sample size. Vendors — don’t do this! Anyone who has even read the course description of a stats class will immediately knock down your credibility.

I suspect the primary reason for omitting survey demographics is embarrassment over small samples or believing the demographics disclose too much about a vendor’s customer base.

Small sample sizes

A small sample is insightful if the respondents are well-targeted. For example, Mulesoft used to survey trade show attendees. The event provided a well-filtered pool of respondents. At the same time, the survey gave them a conversation starter and content to validate the challenges of running multiple monitoring tools and the inefficiencies Mulesoft seeks to solve. In this case, a small but high-quality sample validated a common challenge.

What to do when something looks odd:

First, consider the space between what respondents thought they read and what the survey writer thought they asked. The precise wording of questions matters. A common vendor-funded research fault is questions written too far inside a vendor’s reality distortion field. These distortion fields can include terminology and slang used pervasively inside a company but nowhere else.

Next, examine the demographics and motivations of the survey’s respondents. Reaching the right people for the survey is critical. Even then, there are times when respondents exaggerate.

Examples of questionnaire surveys: Two of my recent favorites are Repvue’s quarterly insight into salesperson attainment and ETR’s spending trends research. ETR is a research house, but the company’s visibility into large-enterprise spending trends is solid. Anyone want to start a hedge fund?

Obviously, there are many more. This view of the survey database shows surveys driven by questionnaires.

[top]

#3 Peer Reviews, Evaluations, and Comparisons

Selecting cloud services for your application is complicated. Services have different scaling costs, performance properties, and limitations regularly buried in the stack of unabridged dictionaries known as AWS docs. Fortunately, you probably aren’t the first team to evaluate a product or the differences between two seemingly similar services for your stack. Ideally, someone else has done and published a proper evaluation.

Methodology:

My favorite examples of this genre are teams that combine live tests and spreadsheet calculations in their evaluations and then publish their findings.

Product review sites like Gartner Peer Insights, G2, TrustRadius, and other review aggregators host the largest volume of publicly accessible reviews.

Strengths:

The on-demand nature of cloud services means anyone can evaluate many services by using the service. Behaviors that strike one evaluator as notable are likely noteworthy to others. Additionally, when an engineering team publishes a well-thought-out analysis of a large decision, those posts can help attract talent and give prospects confidence in your product.

Pitfalls:

Your tech stack and business requirements likely differ from the publisher’s requirements. Hopefully, the publisher articulates enough context for you to calibrate the findings.

Another downside of the review sites is visibility into the evaluation process. Was there a structured evaluation process? Or did a VP yolo the product into the company after a few conversations on the sidelines of a U8 soccer game with the vendor’s CEO?

The low entry barrier and limited insight into a reviewer’s business attributes are obvious weaknesses of the review sites. Not as obvious are vendor incentives and site imbalances when vendor #1 sends customers to Peer Insights while vendor #2 sends customers to G2, and other methods of gaming the system.

Examples:

“How we tested service A vs B”

Unfortunately, the SEO allure of attracting engineers making service decisions has generated a flood of posts short on data and long on GPT-ish summaries. In fact, due to the sands of startup mortality and a desire to do other things, I couldn’t find a single example. I know they are out there. I'll update this post when I find them or you send them.

This piece on endorsements and regrets isn’t quite what I’m referring to, but it’s fantastic. Also, I prefer end-user posts, but AWS has 72 articles in an ever-growing database of service comparison explainers.

Review sites

I’m skeptical of the market maps generated within individual product segments of review sites. Reviews can surface red flags, but tend to lean negative or overly positive when a vendor incentivizes customers to write reviews. I’d suggest checking multiple review sites to see what current customers say, but shy away from the summarized rankings. These sites also produce research with surveys and platform data. Good examples can be found in the G2 research hub.

[top]

#4 Consultant/ Subject matter quote collections: Often junk, with one notable exception

If you can’t tell by now, I’m a sucker for “[Hot topic] Survey” or “The State of [Hot Topic]” clickbait. However, when big consulting companies publish those titles, I’ve learned to temper my expectations. What looks like a quantitative survey on the outside is often a collection of thought leadershit quotes on the inside.

Thoughtworks Radar is THE notable exception. 10+ years of applying a structured methodology for assessing the maturity and utility of emerging technologies often provide a solid reality check against market buzz. I recommend reading their FAQ and then checking in at least once or twice a year. Gold star to the Thoughtworks crew. Keep it up!

Methodology:

Query consultants on opinions of a technology, industry segment, or process, ideally using a consistent methodology.

Strengths:

ThoughtWorks owes its kudos to a consistent methodology designed to surface and organize the experiences of its consultants and clientele with technologies and processes as they evolve. They are not afraid to call BS and highlight product limitations.

As for the other firms, if you are considering hiring a consulting firm for a [hot topic] project, reading their quotes may indicate whether they are full of it.

Weaknesses:

There is a simple litmus test for these works—can the reader now ask better questions on a topic? Or does the piece complicate the topic?

Examples:

Good

Bad

I’ll avoid “content-shaming.” You’ll know it when you see pages of quotes from partners about “what they are seeing” with a single chart from a survey they published last year.

[top]

#5 LMGTFY Surveys Meet APIs

Let me google that for you. Yes, you could plug product specs across multiple vendors into a spreadsheet. Or… someone could do that for you. Do these efforts take editorial liberties and distort reality to favor the vendor publishing the table? You bet. Should you make purchasing decisions based on product attributes someone else pulled together? No. But will you? Yes, you will if the data is presented reasonably authentically, and it makes buying this stuff easier.

Speaking of easier, cloud service catalogs are shape-shifting monsters with regionally specific pricing. One company, Cloud Mercado, taps into cloud provider apis to produce a compute and storage catalog that consolidates cloud offerings across providers and regions. LMGTFY on steroids.

[top]

Bonus round you hope to bank on: Investment Deal and Compensation Data

Salaries are nice, but the real money is in equity. Every software business leader needs to understand trends in equity valuations. Before you head off Buck’s, Bluebottle, Copa, or Rosewood to meet investors, you should level up on trends in private company valuations. Likewise, to negotiate your comp plan, arm yourself with data on the types of comp plans and grants your peers are signing.

Fortunately, firms with deal databases and equity management publish analyses of their platform activity and occasional surveys of investors.

Strengths and weaknesses

Comp and valuation data is hard to obtain, can be skewed by outliers, and probably lags actual market conditions by 3 to 6 months. Treat the data as directional, not absolute.

Examples:

I recommend the Carta “State of Startup Compensation” work all the time. They get a gold star for utility, especially since they connected equity grants with salaries. Another recent entrant is the Hiive Private Market Report. Hiive publishes data from private transactions of private company stock.

Finally, this article on security acquisitions and what they mean for employees pulls data from several sources. It is useful for anyone thinking their equity lottery tickets might pay off with an acquisition.

The “Capital and comp” view of the database has those and more examples.

[top]

Help! Resources and parting thoughts

An ask:

Technology leaders of small or medium-sized development teams:

Please take this short survey on a community and collective research effort tailored to advancing your craft outside the hype tornado.

For all of you:

If you see a study you like or publish one related to building and running software, please forward it or tag me. If you found this post or the research collection useful, I’d love to know that too.

A hat tip to everyone working on this research:

The analysts, product marketers, product managers, executives allocating budget, designers, copywriters, and demand gen folks - it’s clear many of you are pouring time and energy into attempting to shine light into the abyss. I might be hallucinating worse than an LLM, but the quality of vendor-sponsored research continues to rise, setting a high bar that benefits us all.

People to follow:

In addition to the ‘Evergreen resources’ in my research database, a couple of people stand out when it comes to data on building software companies.

The NewStack: Keep an eye on everything Lawrence Hecht writes. His piece on “What did data in 2024 tell us?” is a great place to start. Lawrence, Alex, and company not only write about many studies, they also publish data on NewStack reader habits.

Lori Macvitte at F5: Lori publishes The Tech Menagerie, highlighting data published over the last month. As a fellow recovering analyst and part of the team that publishes F5’s annual The State of Application Strategy study, her writing is always insightful.

Matt Harney’s Saasletter: Matt aggregates and publishes all the high quality data to be had on the heartbeat of private and public SaaS company financial health.

Abi Noda’s Engineering Enablement newsletter: Abi regularly reviews new research papers and other studies on developer experience and productivity. He’s also writing a book with Dr. Nicole Forsgren.

Peter Walker at Carta: Follow him on Linkedin and signup for their newsletter. A+ startup data.

Jamin Ball’s Clouded Judgement

Parting thoughts

Will primary research investments continue in a more austere business climate? So far, the answer is mostly yes. The current boom in AI topics kicked off in May 2023 and shows no signs of abating.

Is this database useful? I really don’t know. On one hand, it could be just a weird temple of marketing content, on the other I’ve found it helpful when seeking proof points in my storytelling and product-market-fit work and perhaps others will as well. And yes, I might throw a bunch of these into an LLM and see what happens. Stay tuned.

[top]

About me:

These observations come from the scars of multiple survey studies at IDC, publishing the first analysis of Docker container behavior data at New Relic, and starting a database of vendor-sponsored cloud and developer surveys for a consulting project in 2021. Thanks to the Notion webclipper, I never stopped adding new studies. The database now includes 250+ surveys and reports categorized by topic, sponsor, data source, and other attributes.

I spend most of my time helping AI-powered companies define their ideal customer profiles and then build the research, discovery conversations, and go-to-market strategies to engage prospects. If you or your go-to-market team are struggling to connect the value and promise of your products with prospects and influencers, I may be able to help or at least point you in the right direction.

[top]

Footnotes:

[1] New Relic’s language adoption report debuted in 2009 and ended in 2014, Rightscale’s first State of the Cloud survey was published in 2013, and the first State of DevOps Survey was published in 2014 [Correction - Puppet published a State of DevOps study before 2014 called SODAR]. Those were not the only studies - North Bridge Venture Partners funded a cloud survey between 2010 and 2014 that combined SaaS and cloud infrastructure. Much of this data has been lost to the sands of Internet time and marketing reorgs, but it was groundbreaking then. Finally, yes, IDC, Forrester, and Gartner fielded cloud adoption studies back then, although they often combined SaaS and cloud infrastructure into a single bucket. I’m still bitter about that. [back]

[2] If it’s your job to publish this stuff, I understand customer quotes can be a major pain and threaten to delay your publish date. If you have a customer advisory board, this is a great way to engage them with a “preview,” that hopefully produces a few nuggets to improve your analysis. This can be lightweight; don’t overthink it, but get out of your corporate distortion field if possible. [back]

[3] Vendor protip: Talk to your general counsel as early as possible when embarking on a platform data study. Founders — if you want to publish usage analysis someday, ask the lawyers drafting your master service agreement for language enabling that without letting a study elevate a dispute. You don’t want to give a trigger-happy lawyer the ability to claim the study includes their client’s data as a reason to weasel out of a contract. [back]

AI disclosure:

I use Grammarly to trim my verbose prose. I also asked Anthropic’s Claude for feedback. Meh.