At zomato we used statsd exporter for envoy metrics initially and later when we had the usecase to report application metrics a couple of years ago, we added a core module in our monolith php codebase to send stats to the statsd exporter sidecar that we are already using for envoy metrics.
our elasticsearch cluster(part of our elk stack) is a pretty standard cluster hot/cold cluster.
Thanos is great to query from multiple prometheus instances, and after setting up needs almost very little maintenance, but is still not horizontally scalable atleast not out of the box. Although the querier and store components are scalable, a significant amount of efforts are spent on the setting up individual prometheus instances to scrap the targets. this means setting up prometheus instances everytime a major metrics component is released in the infrastructure like cadvisor, node exporter, envoy or application level proxies like redis-proxy(predixy), mysql-proxy(proxysql) etc,..
Often applications need to install it’s dependencies through a package manager, and most langauge don’t use system package manager to do this job. ex: python uses pip, php uses composer, ruby uses gem. and all of these package managers install packages for a project based on a package list file. and it is a common pattern to copy our code into docker container in this fashion.