A curated knowledge base of real-world LLMOps implementations, with detailed summaries and technical notes
Year 5 in trends in customer success organizations, staffing, reporting, investments, etc.
Jason Lemkin’s comments on CS reporting to sales are spot on. Failure mode.
Macro analysis on the way brands are discussed and interact on social media networks. 718 million mentions across 347 brands.
AI power demand for north american data centers
What do data and analytics folks think the impact of AI is on data and analytics? Big, needs a new stack, and like all data culture surveys, hopes for the democratization of data usage and analysis.
Very fancy landing page/site. Although, for a survey on data and analytics, this felt pretty light on the data. Nice methodology and demographics slide tho.
Insight into the data center people.
55% of technology purchases involve analysts and in many cases through out the purchase process.
Respondents named case studies as their top 3 influence on vendor selection… Those must be fed to them by the sales teams, because they don’t click on things on their own.
Consumer views toward privacy and security.
A few charts and graphs would have helped. Also, the ROI on this seems to be connected to referrals to various data protection services? Kinda odd. It’s not clear why U.S. News would invest in this research.
Nice example of a VC firm doing research in a space to benefit SaaS builders. In this case vertical saas opportunities and strategies.
Analysis of performance review HR data. Super interesting.
I think they used data from their product, but it’s not really clear where the data from the 23,000 reviews came from. In their 2023 survey they used both review data and survey data.
Startup compensation data. Required reading for startup folks.
DevX from the perspectives of managers and developers able to highlight the differences between their views.
Manager vs doer/front line perception gaps are some of my favorite data points.
Github actions YOLOs software into deployment.
26 privacy regulators from the FTC and similar organizations manually inspected over a thousand sites to document their compliance with privacy and consumer protection regulations.
AI is so hot right now
Downtime and other incidents cost a fortune to diagnose and fix.
This study doesn’t seem to have a name, so I named it. It could also be The State of Customer Impacting Incidents, but that’s a mouthfull.
Cut of F5’s annual application strategy report with an additional 75 respondents responsible for AI technology surveyed with 45% from companies providing digital services as their primary business.
Q1 2024 cloud cost trends.
Always a good read. Gold star.
Searching for AI value… How and are enterprise product dev and IT folks adopting AI tech?
Cool comparison between Oct ‘23 and Feb ‘24, especially on gen AI use cases on expectations vs reality. Also - security is now far less of a concern than “we don’t have the data to generate value.”
Engineering management is under pressure.
Well done
A view into developer tools and productivity inhibitors/drivers.
Customers and prospects are sick of you thinking the world revolves around you.
Pay. The. Product. Marketing. Team. And listen to them.
Wild romp through the digital media business and Google behaviors. Required read for anyone publishing content.
Review of trends in funding and exits in cybersecurity
How 40 AI apps are priced and packaged.
F5’s 10th annual application security study. API security takes center stage this year.
High quality as always. Also check out their 10 years of insights page.
Public tease from ETR on who security folks are interested in talking to at RSA.
Solid survey on developer behaviors and attitudes, especially on AI.
I’m always skeptical of self-reported assessments of how developers spend their time. However, 21% on new code vs 26% (15%+11%) of time on code optimization and maintenance does seem to pass the sniff test.
Sales moral and attainment by company. Insightful AF.
What are 1100 security folks worried about? Ransomeware, a lot.
Year 7 of this survey into research across the AI industry. Solid overview of the state of AI models, training costs, performance, and more.
AI is officially better than humans on several fronts, but wow is it getting expensive to be in front.
What CEOs want out of AI (and customer service)
Do CEOs know where to best apply AI in their company? Maybe.
LOTS of folks want their data deleted. These are the receipts.
2023 threat and vulnerability analysis
Compensation trends and data for technology, product, and design leadership roles. Data is based on candidates placed in 2023 across seed to enterprise companies.
Unclear how many placements were used to generate the data.
The OG of language ranking studies. If you aren’t already paying attention to Redmonk, are you really interested in developers?
Be sure to check out the 2012 to 2024 analysis as well. That’s where the banner image came from.
It’s all fake and you can’t trust anything you see in your web analytics tools.
Spy vs Spy
Private market valuations and trading trends.
63% of the last funding round. That’s the current clearing price of most private companies trading on the private secondary market right now. It’s up from a low of 52%. Say AI one more time baby.
Cool study on the impact of AI in the call center—one of the early and obvious generative AI applications. Although I’ve yet to have a good experience, non-AI call centers hired 89% more agents than AI-powered call centers!
Not vendor sponsored and the full report is paywalled, but I’ve always had lots of respect for the author/analyst and the findings are pretty interesting.
Cloud Development Envirorments - what’s the deal? (Jerry Seinfeld voice)
Salaries plus equity grant data. Solid.
Comparison and analysis of AI models across key metrics including quality, price, performance and speed (throughput & latency), context window & others.
What are telecom providers doing with AI? Or at least what do they aspire to do.
ETR’s spending panel is always interesting. This one looks at DataDog and it’s not great. How well will it align with DD’s actual results? We’ll see.
Rollicking trip through the infrastructure of a startup.
Great fun. Required reading.
2024 venture capital funding stats
Lots of down rounds, bridge funding, and lack of option exercises. A comparison to 2008 or 2001 would be more telling.
What do O’Reilly readers read and train themselves on? It’s all here.
Microservices are down 20% YoY. Long live the monolith?
Findings from 25 developer interviews
Developer demographics and hot topics
A bit heavy on “non-professional” developers in the sample.
How are financial services technology teams using AI? Now in year 4
Crazy weak demographics
A view into Generative AI in the enterprise. Lots of testing, but also some production apps. Folks expect productivity and revenue gains. Very few expect headcount changes.
Vantage platform data on cloud spend. Now includes visibility into DataDog and comparison of service consumption across multiple cloud.
Phenomenal data. The shift in spending on EC2 is an eyebrow raiser.
Analysis of GDPR enforcement actions
Teaser doc for a survey on Developer experience and perceptions for testing and management tools.
Quantification of Gen AI model hallucinations.
Super cool
Spending trends in SaaS based on actual data across 1000+ companies
Very cool trends
Are designers happy? In what areas is collaboration with product teams and others going well, or not?
So many datapoints on SaaS company metrics. Kyle rocks, read his newsletter.
Trends in venture backed investment, valuations, firms, dealflow, etc.
How do we feel about the kubernete in 2023?
Container adoption survey data basck to 2017! Now it’s everyone.
The view from the account rep seats. Indexed. Over time.
Fantastic
Review of industry benchmarks and adoption indicators. Year 6.
What’s hot out on the interwebs among the whippersnappers and everyone else?
Midjourney tops the list with a mere 11 people building the service. Small software teams can have a huge impact.
Analysis of cyber insurance claims data.
How much of your devops process is actually automated? Not much. Also deploying LLM is throwing a wrench in the works.
Comp data from Carta’s customer base. Hiring trends. Matches equity grants and base salaries for the first time.
This is awesome data on multiple fronts. Good work Carta! Constrained option pools and only 28% of options exercised for options expiring in Aug is fascinanting.
20(1)/w=1920,quality=90,fit=scale-down)
How and when do UX researchers use AI?
How do devops and devsecops leaders use and views on using generative AI differ?
Expert romp through the world of AI Agents. A little out of sync with everything else here, but I’ve become a sucker for The State of [insert tech here]
It’s easier to use AI tools than deploy it in a product. Everyone is worried about sensitive data leaking into and out of LLMs.
Not everything is a cloud service. Lots of software is being deployed in customer environments.
The people responsible for managing identity in large companies all say they are hosed. Now they have the data to know they aren’t alone.
Lateral attacks and ransomware are so hot right now. Pretending you are Rose in finance has always been cool. Every bit of writing on this assumes everyone knows what a PAM is. Spell out your TLAs, please.
DevRel roles, responsibilities, comp, and more
Datadog data on where serverless is used, how it’s deployed, and more. Spiffy.
The role of Terraform in building serverless apps is growing and a really big deal in large orgs.
If your design and UX teams can collaborate well with your product team, you’re going to do well.
Consolidation of AI related laws and legislation.
What services do cloud customers use? Great stuff.
They did a study, but where did they put the data? Am I blind? Inquiring minds want to know.
This survey says folks are worried about mucking up their cloud deployments with insecure infrastructure and they hope learning about cloud will result in career advancement.
Happier developers get more done. Larger orgs, larger teams, bad context/observabilioty, and interruptions drag it all down.
5th annual study into UX researcher tools, trends, and behaviors.
Github would like to help you be more productive and everyone else is experiencing good things while using Copilot, so you should too.
Github sourced a bunch of feedback over twitter. Kinda neat.
In it’s third year, the Hashicorp State of the Cloud Survey is consistently well done and insightful. Despite a skew toward the Hashicorp user base, the study is a good portrayal of the state of large cloud users.
One of the OGs of developer surveys. Lots of AI, cloud, language, and developer behaviors across multiple cohorts. Well done as always.
Everyone is using AI tools. Can managers keep priorities updated and in line fast enough?
Okta MFA data over time and segmented by industry and other factors.
Myth-buster: More stringent forms of MFA don’t seem to impact user experience.
Oooh, Databricks customer usage data on AI.
A glimpse into cloud spending patterns via the Vantage customer base. Well done. This is pretty awesome and based on comments from the team that published it, I expect it will continue to get even better.
Trends in growing the people already on your team.
Nice report series on appsec processes, software deployment, and more with an additional AI survey for good measure.
Some of the questions suffer from “the author’s world revolves around Github” and come out a bit odd. A question about using AI in the software development process is really “Do you think AI built into Gitlab tools could be helpful?” but is written as “Will you use AI in the software development process”. Read with care.
Very few, if any companies have this whole remote work thing nailed down. Different jobs, work styles, and meeting cultures are all over the map. Miro tries to put some data around how folks are doing.
What defines an elite software development team? CircleCI takes a crack at it using data from how customers use their platform. Very cool.
The **6th** look at the state of AI by AI developers and researchers, plus a collection of other data relevant to AI.
It used to be ball bearings, but now it’s all APIs.. and constant changes + valuable resources at the other end mean API security is A THING. Kudos to my friends at Salt for using both survey AND data from their platform to provide their insight.
Real-world incident data from the Fire Hydrant platform. Rock-on.
5th annual survey on where enterprises are running their apps, why they are making those hosting decisions, and what’s hard about managing all that.
Multi-cloud is everywhere and it’s a PITA. Everyone is using Kubernetes somewhere and cares about sustainability, but 2/3 don’t know where their data resides.
Potentially a very cool report on software team behaviors thwarted by a complete lack of insight into demographics and/or data sources.
How many people did you survey?!? “Hundreds” WTF.
Additional analysis of the 2022 CNCF survey
Year 12!
This survey is focused on cloud migration and adoption in Hong Kong S.A.R., Malaysia, Singapore, the Philippines, Indonesia, Thailand, Japan and South Korea.
Analysis of how developers use O’Reilly content and learning platform.
I’m a fan of the differences between search trends and content consumption. Great stuff.
Argues cost reduction is the primary driver for K8s.
As the world changes, how does the tech companies use change? Okta sees it all across their customer base and they are kind enough to share it. Pretty awesome. Now in year 7 (I think)
Kandji and Grammerly are killing it. Good for them. If you read this report in Feb, saw that travel was taking off, and bought airline stocks, you would be up huge. Stay on your toes.
This is a “high-level overview” survey, not a “fill out this form” survey.. that said, TeleGeography has long been the source of awesome subterranean cables. They just have a solid view of global bandwidth trends that can help inform decisions for anyone making global investments.
Where the network goes, everything else follows.
How are cloud strategies changing due to the economic climate? Google’s quarterly survey of 1900 tech and business leaders asks that question.
A glimpse into cloud spending patterns via the Vantage customer base. Well done. This is pretty awesome and based on comments from the team that published it, I expect it will continue to get even better.
Mulesoft with a survey of IT folk making the (correct) arguement that APIs are critical to transforming the massive pile of enterprise applications you have into a better user and developer experience.
Very cool study on kubernetes usage using Dynatrace product data. (Apologies to the Dynatrace graphics folks for the fugly header)
Rust focused derivative of the JetBrains 2022 Developer Ecosystem study
ETR does some cool spending trend surveys among a nicely selected panel.
2/3 of workloads are still on prem. This feels like a better number to me than the 80% certain entities have been throwing around lately.
If you want to understand the state of equity grants and compensation, this report and. the Carta reports are both super informative.
Ooh this one is cool. Incidents - most of what you thought you knew is probably wrong. Root cause analysis? Not meaningful. MTTR? Not a great way to measure distributed systems because averages.
Everything except for DNS. It was DNS
Migration and modernization - what, where, why, and what’s next.
The state of Open Source software
Some cool data on cloud security posture management, data sources, BUT… how many people did they survey? Also, I’m now completely confused by what Vectra does.
No N!!! WTF?
What skills do developers have and what’s in demand? Measured by the Hackerank platform.
On one hand, this survey grates my gears because Adam Larson and I did the very first version of this before New Relic made decisions about infrastructure monitoring. On the other, this has turned into a pretty amazing piece of work that charts the growth of one of the fastest technology adoptions ever. It’s fantastic they copied our first study and kept it going all these years.
Nice mix of SRE pains in survey data and quotes.
Summer of 22, SREs are skeptical of AI.
Everything you wanted to know about Jamstack community demographics. Really good methodology overview.
Cool survey on developer productivity and retention that argues Low Code can increase developer happiness.
Datadog’s first published look into security data on AWS. IAM is hard.
The Frankencloud is here, and enterprise companies don’t have the skills to manage the monster.
Damn it, this isn’t an actual survey, it’s Accenture’s take on Cloud = greener computing. I’m going to leave it up tho becuase I think they are right and they do share some data on ESG business benefits. I might delete this tho.
Mostly a discussion of Chronosphere with a dash of data from 451 Research’s Voice of the Enterprise: Cloud Native, Observability 2022 survey. I really like the “what we thought we were going to get vs what we actually got” question format.
Not a survey, but I like the Chronosphere team and that question format, so I included it.
“On behalf of the Go team, thank you to the 5,752 people who told us about their experience working with new features introduced in Go 1.18, including generics, security tooling, and workspaces. You’ve helped us better understand how developers are discovering and using this functionality, and as this article will discuss, provided useful insights for additional improvements.”
Cool to see security take a large role in this survey.
[Read the post on Silicon Angle] Where are the gaps between the high flying private company valuations of 2021 and company performance? ETR takes a look and whooee is this interesting.
The discussion of churn for Rubrick and Cohesity sounds like actual churn vs survey results. This is a nit, but $1 says there are slacks and emails grumbling about that paragraph at both companies. When you throw a vendor’s business under the bus like this, some caveats and color on little sample size and bias is super helpful.
Do you know the Warren Buffet quote about a receding tide showing who is and is not wearing pants? ETR does a nice job of indicating what the private company beach might look like at low tide. Kudos to Dave Vellante for this write-up and perhaps turning ETR data snapshots into an ongoing series? (Yes, please)
Do actual prospects really fill out a form to read about cloud security maturity models? With barely a hors d'oeuvres worth of survey findings? I wouldn’t, but I’m not the target. That said, kudos to Ermetic for year two of a survey, I expect this report is a nice door opener for their SDR team vs driving form fills. Hopefully a blog with more color is coming soon.
Is multicloud really a thing? What makes multicloud or for that matter cloud work for companies? And what holds them back?
If Hashi’s survey is right, then either the market has changed (likely) or my old take of “very few choose multicloud, rather, it is foisted upon them” is WRONG. In fact, those who choose multicloud a few years ago, often did so for reliability and resistance. Surveys often suffer from aspirational answers and it does feel like there is a bit of that in this survey - like 27% of respondents said they build their own multicloud infrastructure tools from scratch. Um, maybe if you consider a YAML file an automation tool, but no. That’s a minor nit in what is otherwise a solid piece of work.
Is multi-cloud management a thing? App Portability? Cost controls? Tool sprawl? Nice presentation, no demographics. “US and UK Decision Makers”
The granddaddy of developer surveys. Required reading every year.
Good view into the state of zero trust - TLDR: it’s lagging on assuming breaches will occur with optimistic and/or overly reliant on monitoring.
If you are professionally paranoid, what’s keeping you up at night? This survey reveals those topics with data since 2015. Good fun, ignore at your peril.
“Despite concerns about ransomware and attacks targeting cloud services and supply chain, phishing remains the No. 1 threat for organizations.” ← Since the dawn of infosec a version of “get the basics right” has been true.
Bessemer does a nice job of tracking and analyzing SaaS metrics across public and private markets - this report summarizes and illustrates trends in that dataset. It’s mostly SaaS apps vs my tendency to focus on infrastructure, but I like their stuff, so here you go.
Private valuations are still disconnected from reality.
The 15th!!! granddaddy of security incident surveys. Required reading for security folk. Cool data cuts by industry. And written by folks not afraid of riffs like, “if the slanted areas of two (or more) bars overlap, you can’t really say one is bigger than the other without angering the math gods.”
Cloud survey for tech marketeers. Some good decision making and brand affiliation stuff like only 40% of responents feel their primary cloud is a “strategic partner”
In year 11, the Flexera, ne Rightscale report is the GrandDaddy of “multicloud is a thing” “how can we model cloud spend by company size” and “Azure might be bigger than we thought” - goes a bit harder into “cloud waste is out of hand” - if you aren’t reading this thing are you even into cloud research?
I’m oddly happy they included more body than face in their depiction of “cloud guy” this year over last year. (also, scroll down to the bottom for stupid martech tricks.
Trends in early stage financing metrics among the top VC firms.
Aside from demonstrating the size and scale of Okta’s footprint, the company has a pole position on insight into the apps and services corporate folk use to get their jobs done. Do they have a venture fund? Seems prudent.
Kudos to O’Reilly for instrumenting, analyzing, and sharing findings on the behavior of their customer base. Attempting to understand technology trends based on content consumption is like sex and teenagers - the talk vs doing ratio is big, but in this case, laudable.
Ok, I’m going to let you in on secret. Survey data is crap. Seriously, take everything you come across in this database with doubt rivaling the certainty a 2012 EC2 instance in US-East1 would be super stable. What’s better? Actual behavior data. And while AWS laughed at me when I asked for serverless usage data, Intricately monitors enough network traffic to make cloud spend estimates and know whose workloads are in which cloud. Super cool and very powerful.
The 80/20 rule is in full effect when it comes to 80% of the customers give you 20% or less of the revenue. Startups are cool and fun, but the money is at the adult table.
You don’t have enough observability and you don’t like what you do have so… how about take a call from our SDR?
Who uses Kubernetes and why?
Oooh - snazzy layout with cross tabs you can play with. I’m in love. Someone give the designers at Snyk a gold star and a pile of stock.
In case you either thought you were crazy and alone while everyone else clouds along nicely or somehow didn’t know identity and access management in the cloud is a complete pain, now you have the data to demonstrate it is a pain and no one is doing it well.
The landing page… if you are a white guy in cloud in 2021, you have a beard. (I’m guilty too, but it’s not my landing page)
Required reading on developer productivity
The 2019/2020 survey results aren’t accessible from the main site, but can be accessed via the pdf link below
People are the expensive and hard and most important element of a software business. While cloud has absorbed the undifferentiated heavy lifting of owning and operating a datacenter, operating cloud infrastructure is no walk in the park.
Rollup of multiple data sources on outages, adoption of security protocols, and more. Constantly updated.
All the Puppet sponsored “State if DevOps” reports in one place. Note that starting in 2019ish DORA/Google and Puppet State of DevOps studies split after DORA was acquired by Google.
Very cool way to compare and contrast EC2 instance types and ballpark costs. Also availible for RDS, Elasticache, Redshift, and Opensearch
Ranked list of total spending on each AWS service across Vantage’s user base for the trailing 30 days. Very cool.
Internet traffic trends.
Regularly updated performance and pricing benchmarks along with consumption trends across European and global cloud providers. Also, check out their consolidated and global compute and storage catalog - very cool.
Pitchbook analysis of venture capital and investment data in the Pitchbook dataset.
Datadog publishes a ton of the usage data on customer behaviors. It’s awesome and also painful as I kicked this off at New Relic to great success before Datadog was Datadog, but then we snatched defeat from the jaws of victory.
DZone has been publishing high-quality research on software development and infrastructure technologies for some time. You can find it all at this link.
This index tracks the size of public, private and national HPC clusters, as well as the utilisation of various AI chips in AI research papers.
Always sharing wonderful developer survey research.
Analysis of cultural and industry trends via crunching numbers across multiple social media platforms.
Always interesting
Twice a year insight into the maturity and value of technologies Thoughtworks consultants come across in their work. Rarely boring, often a decent reality check, almost always something you didn’t even know about.
Developer focused research house
G2 research hub. Lots of good data on B2B buyer behaviors and attitudes
B2B tech buyer attitudes and expectations
Lots of data on privacy laws
Lots of data and reports on funding, acquisitions, and deal flow across many software markets. Covers both private and public companies. Strong international coverage.
Landing page for all of Redmonk’s data analysis research.
All of Google’s research publications. Warning, lots of volume. 700+ publications in 2023
Regular summaries and explanations of academic and vendor research on marketing science and human psychology.
Open source question bank for surveys.
