Facebook’s vs Twitter’s Approach to Real-Time Analytics - 10/19/2013

Facebook’s vs Twitter’s Approach to Real-Time Analytics

Last year, Twitter and Facebook have released new versions of their real-time analytics systems.

In both cases, the motivation was relatively similar — they wanted to provide their customers with better insights on the performance and effectiveness of their marketing activities. Facebook’s measurement includes “likes” and “comments” to monitor interactions. For Twitter, the measurement is based on the effectiveness of a given tweet – typically called “Reach” – basically a measure of the number of followers that were exposed to the tweet. Beyond the initial exposure, you often want to measure the number of clicks on that tweet, which indicate the number of users who saw the tweet and also looked into its content.

Facebook’s vs Twitter’s Approach to real-time Analytics 

Facebook Real-Time Analytics Architecture – Logging-Centric Approach:

Relies on Apache Hadoop framework for real-time and batch (map/reduce) processing. Using the same underlying system simplifies the maintenance of that system.

• Limited real-time processing — the logging-centric approach basically delegates most of the heavy lifting to the backend system. Performing even a fairly simple correlation beyond simple counting isn’t a trivial task.

• real-time is often measured in tens of seconds. In many analytics system, this order of magnitude is often more than enough to express a real-time view of what is going on in your system.

• It is suitable for simple processing. Because of the logging nature of the Facebook architecture, most of the heavy lifting of processing cannot be done in real-time and is often pushed into the backend system.

• Low parallelization — Hadoop systems do not give you ways to ensure ordering and consistency based on the data. Because of that, Facebook came up with their Puma service that collects and inputs data into a centralized service, thus making it easier to processes events in order.

• Facebook collects user click streams from your Facebook wall through an Ajax listener which, then, sends those events back into the Facebook data centers. The info is stored on Hadoop File System via Scribe and collected by PTail.

• Puma aggregates logs in-memory, batching them in windows of 1.5 seconds and stores the information in Hbase.

• The Facebook approach puts a huge limit as to the volume of events that the system can handle and have significant implications over the utilization of the overall system.

Twitter Real-Time Analytics Architecture – Event-Driven Approach:

• Unlike Facebook, Twitter uses Hadoop for batch processing and Storm for real-time processing. Storm was designed to perform fairly complex aggregation of the data that comes through the stream as it flows into the system, before it is sent back to the batch system for further analysis.

• Real-time can be measured in milliseconds. While having second or millisecond latency is not crucial to the end user — it does have a significant effect on the overall processing time and the level of analysis that we can produce and push through the system. As many of those analyses involve thousands of operations to get to the actual result.

• It is suitable for complex processing. With Storm, it is possible to perform a large range of complex aggregation while the data flows through the system. This has a significant impact on the complexity of the processing. A good example is calculating trending words. With the event-driven approach, we can assume that we have the current state and just make the change to that state to update the list of trending words. In contrast, a batch system will have to read the entire set of words, re-calculate, and re-order the words for every update. This is why those operations are often done in long batches.

• Extremely parallel – Asynchronous events are, by definition, easier to parallelize. Storm was designed for extreme parallelization. Ultimately, it determines the speed level of utilization that we can get per machine in our system. Looking at the bigger picture, this quite substantially adds to the cost of our system and to our ability to perform complex analyses.

Final Words

Quite often, we get caught in the technical details of these discussions and lose sight of what this all really means.

If all you are looking for is to collect data streams and simply update counters, then both approaches would work. The main difference between the two is felt in the level and complexity of processing that you would like to process in real-time. If you want to continuously update a different form of sorted lists or indexes, you’ll find that doing so in an event-driven approach, as is the case of Twitter, can be exponentially faster and more efficient than the logging-centric approach. To put some numbers behind that, Twitter reported that calculating the reach without Storm took 2 hours whereas Storm could do the same in less than a second.

Such a difference in speed and utilization have a direct correlation with the business bottom line, as it determines the level and depth of intelligence that it can run against its data. It also determines the cost of running the analytics systems and, in some cases, the availability of those systems. When the processing is slower there would be larger number of scenarios that could saturate the system.

(source: Nati Shalom’s blog)

Wix - 11/11/2014

Nifty Architecture Tricks From Wix – Building A Publishing Platform At Scale

Wix operates websites in the long tale. As a HTML5 based WYSIWYG web publishing platform, they have created over 54 million websites, most of which receive under 100 page views per day. So traditional caching strategies don’t apply, yet it only takes four web servers to handle all the traffic. That takes some smart work.

Aviran Mordo, Head of Back-End Engineering at Wix, has described their solution in an excellent talk: Wix Architecture at Scale. What they’ve developed is in the best tradition of scaling is specialization. They’ve carefully analyzed their system and figured out how to meet their aggressive high availability and high performance goals in some most interesting ways.

Wix uses multiple datacenters and clouds. Something I haven’t seen before is that they replicate data to multiple datacenters, to Google Compute Engine, and to Amazon. And they have fallback strategies between them in case of failure.

Wix doesn’t use transactions. Instead, all data is immutable and they use a simple eventual consistency strategy that perfectly matches their use case.

Wix doesn’t cache (as in a big caching layer). Instead, they pay great attention to optimizing the rendering path so that every page displays in under 100ms.

Wix started small, with a monolithic architecture, and has consciously moved to a service architectureusing a very deliberate process for identifying services that can help anyone thinking about the same move.

This is not your traditional LAMP stack or native cloud anything. Wix is a little different and there’s something here you can learn from. Let’s see how they do it…

Stats

• 54+ million websites, 1 million new websites per month.
• 800+ terabytes of static data, 1.5 terabytes of new files per day
• 3 data centers + 2 clouds (Google, Amazon)
• 300 servers
• 700 million HTTP requests per day
• 600 people total, 200 people in R&D
• About 50 services.
• 4 public servers are needed to serve 45 million websites

Platform

• MySQL
• Google and Amazon clouds
• CDN
• Chef

Evolution

• Simple initial monolithic architecture. Started with one app server. That’s the simplest way to get started. Make quick changes and deploy. It gets you to a particular point.
  • Tomcat, Hibernate, custom web framework
  • Used stateful logins.
  • Disregarded any notion of performance and scaling.
• Fast forward two years.
  • Still one monolithic server that did everything.
  • At a certain scale of developers and customers it held them back.
  • Problems with dependencies between features. Changes in one place caused deployment of the whole system. Failure in unrelated areas caused system wide downtime.
• Time to break the system apart.
  • Went with a services approach, but it’s not that easy. How are you going to break functionality apart and into services?
  • Looked at what users are doing in the system and identified three main parts: edit websites, view sites created by Wix, serving media.
  • Editing web sites includes data validation of data from the server, security and authentication, data consistency, and lots of data modification requests.
  • Once finished with the web site users will view it. There are 10x more viewers than editors. So the concerns are now:
    • high availability. HA is the most important feature because it’s the user’s business.
    • high performance
    • high traffic volume
    • the long tail. There are a lot of websites, but they are very small. Every site gets maybe 10 or 100 page views a day. The long tail make caching not the go to scalability strategy. Caching becomes very inefficient.
  • Media serving is the next big service. Includes HTML, javascript, css, images. Needed a way to serve files the 800TB of data under a high volume of requests. The win is static content is highly cacheable.
  • The new system looks like a networking layer that sits below three segment services: editor segment (anything that edits data), media segment (handles static files, read-only), public segment (first place a file is viewed, read-only).

Guidelines For How To Build Services

• Each service has its own database and only one service can write to a database.
• Access to a database is only through service APIs. This supports a separation of concerns and hiding the data model from other services.
• For performance reasons read-only access is granted to other services, but only one service can write. (yes, this contradicts what was said before)
• Services are stateless. This makes horizontal scaling easy. Just add more servers.
• No transactions. With the exception of billing/financial transactions, all other services do not use transactions. The idea is to increase database performance by removing transaction overhead. This makes you think about how the data is modeled to have logical transactions, avoiding inconsistent states, without using database transactions.
• When designing a new service caching is not part of the architecture. First, make a service as performant as possible, then deploy to production, see how it performs, only then, if there are performance issues, and you can’t optimize the code (or other layers), only then add caching.

Editor Segment

• Editor server must handle lots of files.
• Data stored as immutable JSON pages (~2.5 million per day) in MySQL.
• MySQL is a great key-value store. Key is based on a hash function of the file so the key is immutable. Accessing MySQL by primary key is very fast and efficient.
• Scalability is about tradeoffs. What tradeoffs are we going to make? Didn’t want to use NoSQL because they sacrifice consistency and most developers do not know how to deal with that. So stick with MySQL.
• Active database. Found after a site has been built only 6% were still being updated. Given this then these active sites can be stored in one database that is really fast and relatively small in terms of storage (2TB).
• Archive database. All the stale site data, for sites that are infrequently accessed, is moved over into another database that is relatively slow, but has huge amounts of storage. After three months data is pushed to this database is accesses are low. (one could argue this is an implicit caching strategy).
• Gives a lot of breathing room to grow. The large archive database is slow, but it doesn’t matter because the data isn’t used that often. On first access the data comes from the archive database, but then it is moved to the active database so later accesses are fast.

High Availability For Editor Segment

• With a lot of data it’s hard to provide high availability for everything. So look at the critical path, which for a website is the content of the website. If a widget has problems most of the website will still work. Invested a lot in protecting the critical path.
• Protect against database crashes. Want to recover quickly. Replicate databases and failover to the secondary database.
• Protect against data corruption and data poisoning.  Doesn’t have to be malicious, a bug is enough to spoil the barrel. All data is immutable. Revisions are stored for everything. Worst case  if corruption can’t be fixed is to revert to version where the data was fine.
• Protect against unavailability. A website has to work all the time. This drove an investment inreplicating data across different geographical locations and multiple clouds. This makes the system very resilient.
  • Clicking save on a website editing session sends a JSON file to the editor server.
  • The server sends the page to the active MySQL server which is replicated to another datacenter.
  • After the page is saved to locally, an asynchronous process is kicked upload the data to a static grid, which is the Media Segment.
  • After data is uploaded to the static grid, a notification is sent to a archive service running on the Google Compute Engine. The archive goes to the grid, downloads a page, and stores a copy on the Google cloud.
  • Then a notification is sent back to the editor saying the page was saved to GCE.
  • Another copy is saved to Amazon from GCE.
  • One the final notification is received it means there are three copies of the current revision of data: one in the database, the static grid, and on GCE.
  • For the current revision there are three copies. For old revision there two revisions (static grid, GCE).
  • The process is self-healing. If there’s a failure the next time a user updates their website everything that wasn’t uploaded will be uploaded again.
  • Orphan files are garbage collected.

Modeling Data With No Database Transactions

• Don’t want a situation where a user edit two pages and only one page is saved in the database, which is an inconsistent state.
• Take all the JSON files and stick them in the database one after the other. When all the files are saved another save command is issued which contains a manifest of all the IDs (which is hash of the content which is the file name on the static server) of the saved pages that were uploaded to the static servers.

Media Segment

• Stores lots of files. 800TB of user media files, 3M files uploaded daily, and 500M metadata records.
• Images are modified. They are resized for different devices and sharpened. Watermarks can be inserted and there’s also audio format conversion.
• Built an eventually consistent distributed file system that is multi datacenter aware with automatic fallback across DCs. This is before Amazon.
• A pain to run. 32 servers, doubling the number every 9 months.
• Plan to push stuff to the cloud to help scale.
• Vendor lock-in is a myth. It’s all APIs. Just change the implementation and you can move to different clouds in weeks.
• What really locks you down is data. Moving 800TB of data to a different cloud is really hard.
• They broke Google Compute Engine when they moved all their data into GCE. They reached the limits of the Google cloud. After some changes by Google it now works.
• Files are immutable so the are highly cacheable.
• Image requests first go to a CDN. If the image isn’t in the CDN the request goes to their primary datacenter in Austin. If the image isn’t in Austin the request then goes to Google Cloud. If it’s not in Google cloud it goes to a datacenter in Tampa.

Public Segment

• Resolve URLs (45 million of them), dispatch to the appropriate renderer, and then render into HTML, sitemap XML, or robots TXT, etc.
• Public SLA is that response time is < 100ms at peak traffic. Websites have to be available, but also fast. Remember, no caching.
• When a user clicks publish after editing a page, the manifest, which contains references to pages, are pushed to Public. The routing table is also published.
• Minimize out-of-service hops. Requires 1 database call to resolve the route. 1 RPC call to dispatch the request to the renderer. 1 database call to get the site manifest.
• Lookup tables are cached in memory and are updated every 5 minutes.
• Data is not stored in the same format as it is for the editor. It is stored in a denormalized format, optimized for read by primary key. Everything that is needed is returned in a single request.
• Minimize business logic. The data is denormalized and precalculated. When you handle large scale every operation, every millisecond you add, it’s times 45 million, so every operation that happens on the public server has to be justified.
• Page rendering.
  • The html returned by the public server is bootstrap html. It’s a shell with JavaScript imports and JSON data with references to site manifest and dynamic data.
  • Rendering is offloaded to the client. Laptops and mobile devices are very fast and can handle the rendering.
  • JSON was chosen because it’s easy to parse and compressible.
  • It’s easier to fix bugs on the client. Just redeploy new client code. When rendering is done on the server the html will be cached, so fixing a bug requires re-rendering millions of websites again.

High Availability For Public Segment

• Goal is to be always available, but stuff happens.
• On a good day: a browser makes a request, the request goes to a datacenter, through a load balancer, goes to a public server, resolves the route, goes to the renderer, the html goes back to the browser, and the browser runs the javascript. The javascript fetches all media files and the JSON data and renders a very beautiful web site. The browser then make a request to the Archive service. The Archive service replays the request in the same way the browser does and stores the data in a cache.
• On a bad day a datacenter is lost, which did happen. All the UPSs died and the datacenter was down. The DNS was changed and then all the requests went to the secondary datacenter.
• On a bad day Public is lost. This happened once when a load balancer got half of a configuration so all the Public servers were gone. Or a bad version can be deployed that starts returning errors. Custom code in the load balancer handles this problem by routing to the Archive service to fetch the cached if the Public servers are not available. This approach meant customers were not affected when Public went down, even though the system was reverberating with alarms at the time.
• On a bad day the Internet sucks. The browser makes a request, goes to the datacenter, goes to the load balancer, gets the html back. Now the JavaScript code has to fetch all the pages and JSON data. It goes to the CDN, it goes to the static grid and fetches all the JSON files to render the site. In these processes Internet problems can prevent files from being returned. Code in JavaScript says if you can’t get to the primary location, try and get it from the archive service, if that fails try the editor database.

Lessons Learned

• Identify your critical path and concerns. Think through how your product works. Develop usage scenarios. Focus your efforts on these as they give the biggest bang for the buck.
• Go multi-datacenter and multi-cloud. Build redundancy on the critical path (for availability).
• De-normalize data and Minimize out-of-process hops (for performance). Precaluclate and do everything possible to minimize network chatter.
• Take advantage of client’s CPU power. It saves on your server count and it’s also easier to fix bugs in the client.
• Start small, get it done, then figure out where to go next. Wix did what they needed to do to get their product working. Then they methodically moved to a sophisticated services architecture.
• The long tail requires a different approach. Rather than cache everything Wix chose to optimize the heck out of the render path and keep data in both an active and archive databases.

• Go immutable. Immutability has far reaching consequences for an architecture. It affects everything from the client through the back-end. It’s an elegant solution to a lot of problems.
• Vendor lock-in is a myth. It’s all APIs. Just change the implementation and you can move to different clouds in weeks.
• What really locks you down is data. Moving lots of data to a different cloud is really hard.

( Via HighScalability.com )

StackOverflow - 1/15/2015

StackExchange created a very cool performance dashboard that looks to be updated from real system metrics.

StackOverflow Update: 560M Pageviews A Month, 25 Servers, And It’s All About Performance

The folks at Stack Overflow remain incredibly open about what they are doing and why. So it’s time for another update. What has Stack Overflow been up to?

The network of sites that make up StackExchange, which includes StackOverflow, is now ranked 54th for traffic in the world; they have 110 sites and are growing at a rate of 3 or 4 a month; 4 million users; 40 million answers; and 560 million pageviews a month.

This is with just 25 servers. For everything. That’s high availability, load balancing, caching, databases, searching, and utility functions. All with a relative handful of employees. Now that’s quality engineering.

This update is based on The architecture of StackOverflow (video) by Marco Cecconi and What it takes to run Stack Overflow (post) by Nick Craver. In addition, I’ve merged in comments from various sources. No doubt some of the details are out of date as I meant to write this article long ago, but it should still be representative.

Stack Overflow still uses Microsoft products. Microsoft infrastructure works and is cheap enough, so there’s no compelling reason to change. Yet SO is pragmatic. They use Linux where it makes sense. There’s no purity push to make everything Linux or keep everything Microsoft. That wouldn’t be efficient.

Stack Overflow still uses a scale-up strategy. No clouds in site. With their SQL Servers loaded with 384 GB of RAM and 2TB of SSD, AWS would cost a fortune. The cloud would also slow them down, making it harder to optimize and troubleshoot system issues. Plus, SO doesn’t need a horizontal scaling strategy. Large peak loads, where scaling out makes sense, hasn’t  been a problem because they’ve been quite successful at sizing their system correctly.

So it appears Jeff Atwood’s quote: “Hardware is Cheap, Programmers are Expensive”, still seems to be living lore at the company.

Marco Ceccon in his talk says when talking about architecture you need to answer this question first: what kind of problem is being solved?

First the easy part. What does StackExchange do? It takes topics, creates communities around them, and creates awesome question and answer sites.

The second part relates to scale. As we’ll see next StackExchange is growing quite fast and handles a lot of traffic. How does it do that? Let’s take a look and see….

Stats

• StackExchange network has 110 sites growing at a rate of 3 or 4 a month.
• 4 million users
• 8 million questions
• 40 million answers
• As a network #54 site for traffic in the world
• 100% year over year growth
• 560 million pageviews a month
• Peak is more like 2600-3000 requests/sec on most weekdays. Programming, being a profession, means weekdays are significantly busier than weekends.
• 25 servers
• 2 TB of SQL data all stored on SSDs
• Each web server has 2x 320GB SSDs in a RAID 1.
• Each ElasticSearch box has 300 GB also using SSDs.
• Stack Overflow has a 40:60 read-write ratio.
• DB servers average 10% CPU utilization
• 11 web servers, using IIS
• 2 load balancers, 1 active, using HAProxy
• 4 active database nodes, using MS SQL
• 3 application servers implementing the tag engine, anything searching by tag hits
• 3 machines doing search with ElasticSearch
• 2 machines for distributed cache and messaging using Redis
• 2 Networks (each a Nexus 5596 + Fabric Extenders)
• 2 Cisco 5525-X ASAs (think Firewall)
• 2 Cisco 3945 Routers
• 2 read-only SQL Servers for used mainly for the Stack Exchange API
• VMs also perform functions like deployments, domain controllers, monitoring, ops database for sysadmin goodies, etc.

Platform

• ElasticSearch
• Redis
• HAProxy
• MS SQL
• Opserver
• TeamCity
• Jil – Fast .NET JSON Serializer, built on Sigil
• Dapper – a micro ORM.

UI

• The UI has message inbox that is sent a message when you get a new badge, receive a message, significant event, etc. Done using WebSockets and is powered by redis.
• Search box is powered by ElasticSearch using a REST interface.
• With so many questions on SO it was impossible to just show the newest questions, they would change too fast, a question every second. Developed an algorithm to look at your pattern of behaviour and show you which questions you would have the most interest in. It’s uses complicated queries based on tags, which is why a specialized Tag Engine was developed.
• Server side templating is used to generate pages.

Servers

• The 25 servers are not doing much, that is the CPU load is low. It’s calculated SO could run on only 5 servers.
• The database server is at 10%, except when it bursts while performing a backups.
• How so low? The databases servers have 384GB of RAM and the web servers are at 10%-15% CPU usage.
• Scale-up is still working. Other scale-out sites with a similar number of pageviews tend to run on 100, 200, up to 300 servers.
• Simple system. Built on .Net. Have only 9 projects, others systems have 100s. Reason to have so few projects is is so compilation is lightning fast, which requires planning at the beginning. Compilation takes 10 seconds on a single computer.
• 110K lines of code. A small number given what it does.
• This minimalist approach comes with some problems. One problem is not many tests. Tests aren’t needed because there’s a great community. Meta.stackoverflow is a discussion site for the community and where bugs are reported. Meta.stackoverflow is also a beta site for new software. If users find any problems with it they report the bugs that they’ve found, sometimes with solution/patches.
• Windows 2012 is used in New York but are upgrading to 2012 R2  (Oregon is already on it). For Linux systems it’s Centos 6.4.
• Load is really almost all over 9 servers, because 10 and 11 are only for meta.stackexchange.com,meta.stackoverflow.com, and the development tier. Those servers also run around 10-20% CPU which means we have quite a bit of headroom available.

SSDs

• Intel 330 as the default (web tier, etc.)
• Intel 520 for mid tier writes like Elastic Search
• Intel 710 & S3700 for the database tier. S3700 is simply the successor to the high endurance 710 series.
• Exclusively RAID 1 or RAID 10 (10 being any arrays with 4+ drives). Failures have not been a problem, even with hundreds of intel 2.5″ SSDs in production, a single one hasn’t failed yet. One or more spare parts are kept for each model, but multiple drive failure hasn’t been a concern.
• ElasticSearch performs much better on SSDs, given SO writes/re-indexes very frequently.
• SSD changes the use of search. Lucene.net couldn’t handle SO’s concurrent workloads due to locking issues, so they moved to ElasticSearch. It turns out locks around the binary readers really aren’t necessary in an all SSD environment.
• The only scale-up problems so far is SSD space on the SQL boxes due to the growth pattern of reliability vs. space in the non-consumer space, that isdrives that have capacitors for power loss and such.

High Availability

• The main datacenter is in New York and the backup datacenter is in Oregon.
• Redis has 2 slaves, SQL has 2 replicas, tag engine has 3 nodes, elastic has 3 nodes – any other service has high availability as well (and exists in both data centers).
• Not everything is slaved between data centers (very temporary cache data that’s not needed to eat bandwidth by syncing, etc.) but the big items are, so there is still a shared cache in case of a hard down in the active data center. A start without a cache is possible, but it isn’t very graceful.
• Nginx was used for SSL, but a transition has been made to using HAProxy to terminate SSL.
• Total HTTP traffic sent is only about 77% of the total traffic sent. This is because replication is happening to the secondary data center in Oregon as well as other VPN traffic. The majority of this traffic is the data replication to SQL replicas and redis slaves in Oregon.

Databasing

• MS SQL Server.
• Stack Exchange has one database per-site, so Stack Overflow gets one, Super User gets one, Server Fault gets one, and so on. The schema for these is the same. This approach of having different database is effectively a form of partitioning and horizontal scaling.
• In the primary data center (New York) there is usually 1 master and 1 read-only replica in each cluster. There’s also 1 read-only replica (async) in the DR data center (Oregon). When running in Oregon then the primary is there and both of the New York replicas are read-only and async.
• There are a few wrinkles. There is one “network wide” database which has things like login credentials, and aggregated data (mostly exposed through stackexchange.com user profiles, or APIs).
• Careers Stack Overflow, stackexchange.com, and Area 51 all have their own unique database schema.
• All the schema changes are applied to all site databases at the same time. They need to be backwards compatible so, for example, if you need to rename a column – a worst case scenario – it’s a multiple steps process: add a new column, add code which works with both columns, back fill the new column, change code so it works with the new column only, remove the old column.
• Partitioning is not required. Indexing takes care of everything and the data just is not large enough. If something warrants a filtered indexes, why not make it way more efficient? Indexing only on DeletionDate = Null and such is a common pattern, others are specific FK types from enums.
• Votes are in 1 table per item, for example 1 table for post votes, 1 table for comment votes. Most pages we render real-time, caching only for anonymous users. Given that, there’s no cache to update, it’s just a re-query.
• Scores are denormalized, so querying is often needed. It’s all IDs and dates, the post votes table just has 56,454,478 rows currently. Most queries are just a few milliseconds due to indexing.
• The Tag Engine is entirely self-contained, which means not having to depend on an external service for very, very core functionality. It’s a huge in-memory struct array structure that is optimized for SO use cases and precomputed results for heavily hit combinations. It’s a simple windows service running on a few boxes working in a redundant team. CPU is about 2-5% almost always. Three boxes are not needed for load, just redundancy. If all those do fail at once, the local web servers will load the tag engine in memory and keep on going.
• On Dapper’s lack of a compiler checking queries compared to traditional ORM. The compiler is checking against what you told it the database looks like. This can help with lots of things, but still has the fundamental disconnect problem you’ll get at runtime. A huge problem with the tradeoff is the generated SQL is nasty, and finding the original code it came from is often non-trivial. Lack of ability to hint queries, control parameterization, etc. is also a big issue when trying to optimize queries. For example. literal replacement was added to Dapper to help with query parameterization which allows the use of things like filtered indexes. Dapper also intercepts the SQL calls to dapper and add add exactly where it came from. It saves so much time tracking things down.
  
  

Coding

• The process:
  • Most programmers work remotely. Programmers code in their own batcave.
  • Compilation is very fast.
  • Then the few test that they have are run.
  • Once compiled, code is moved to a development staging server.
  • New features are hidden via feature switches.
  • Runs on same hardware as the rest of the sites.
  • It’s then moved to Meta.stackoverflow for testing. 1000 users per day use the site, so its a good test.
  • If it passes it goes live on the network and is tested by the larger community.
• Heavy usage of static classes and methods, for simplicity and better performance.
• Code is simple because the complicated bits are packaged in a library and open sourced and maintained. The number of .Net projects stays low because community shared parts of the code are used.
• Developers get two or three monitors. Screens are important, they help you be productive.

Caching

• Cache all the things.
• 5 levels of caches.
• 1st: is the network level cache: caching in the browser, CDN, and proxies.
• 2nd: given for free by the .Net framework and is called the HttpRuntime.Cache. An in-memory, per server cache.
• 3rd: Redis. Distributed in-memory key-value store. Share cache elements across different servers that serve the same site. If StackOverflow has 9 servers then all servers will be able to find the same cached items.
• 4th: SQL Server Cache. The entire database is cached in-memory. The entire thing.
• 5th: SSD. Usually only hit when the SQL server cache is warming up.
• For example, every help page is cached. Code to access a page is very terse:
  • Static methods and static classes re used. Really bad from an OOP perspective, but really fast and really friendly towards terse code. All code is directly addressed.
  • Caching is handled by a library layer of Redis and Dapper, a micro ORM.
• To get around garbage collection problems, only one copy of a class used in templates are created and kept in a cache. Everything is measured, including GC operation, from statistics it is known that layers of indirection increase GC pressure to the point of noticeable slowness.
• CDN hits vary, since the  query string hash is based on file content, it’s only re-fetched on a build. It’s typically 30-50 million hits a day for 300 to 600 GB of bandwidth.
• A CDN is not used for CPU or I/O load, but to help users find answers faster.

Deploying

• Want to deploy 5 times a day. Don’t build grand gigantic things and then put then live. Important because:
  • Can measure performance directly.
  • Forced to build the smallest thing that can possibly work.
• TeamCity builds then copies to each web tier via a powershell script. The steps for each server are:
  • Tell HAProxy to take the server out of rotation via a POST
  • Delay to let IIS finish current requests (~5 sec)
  • Stop the website (via the same PSSession for all the following)
  • Robocopy files
  • Start the website
  • Re-enable in HAProxy via another POST
• Almost everything is deployed via puppet or DSC, so upgrading usually consist of just nuking the RAID array and installing from a PXE boot. it’s very fast and you know it’s done right/repeatable.

Teaming

• Teams:
  • SRE (System Reliability Engineering): – 5 people
  • Core Dev (Q&A site) : ~6-7 people
  • Core Dev Mobile: 6 people
  • Careers team that does development solely for the SO Careers product: 7 people
• Devops and developer teams are really close-knit.
• There’s a lot of movement between teams.
• Most employees work remotely.
• Offices are mostly sales, Denver and London exclusively so.
• All else equal, it is slightly prefered to have people in NYC, because the in-person time is a plus for the casual interaction that happens in between “getting things done”. But the set up makes it possible to do real work and official team collaboration works almost entirely online.
• They’ve learned that the in-person benefit is more than outweighed by how much you get from being able to hire the best talent that loves the product anywhere, not just the ones willing to live in the city you happen to be in.
• The most common reason for someone going remote is starting a family. New York’s great, but spacious it is not.
• Offices are in Manhattan and a lot of talent is there. The data center needs to not be a crazy distance away since it is always being improved. There’s also a slightly faster connection to many backbones in the NYC location – though we’re talking only a few milliseconds (if that) of difference there.
• Making an awesome team: Love geeks. Early Microsoft, for example, was full of geeks and they conquered the world.
• Hire from Stack Overflow community. They looks for a passion for coding, a passion for helping others, and a passion for communicating.

Budgeting

• Budgets are pretty much project based. Money is only spent as infrastructure is added for new projects. The web servers that have such low utilization are the same ones purchased 3 years ago when the data center was built.

Testing

• Move fast and break things. Push it live.
• Major changes are tested by pushing them. Development has an equally powerful SQL server and it runs on the same web tier, so performance testing isn’t so bad.
• Very few tests. Stack Overflow doesn’t use many unit tests because of their active community and heavy usage of static code.
• Infrastructure changes. There’s 2 of everything, so there’s a backup with the old configuration whenever possible, with a quick failback mechanism. For example, keepalived does failback quickly between load balancers.
• Redundant systems fail over pretty often just to do regular maintenance. SQL backups are tested by having a dedicated server just for restoring them, constantly (that’s a free license – do it). Plan to start full data center failovers every 2 months or so – the secondary data center is read-only at all other times.
• Unit tests, integration tests and UI tests run on every push. All the tests must succeed before a production build run is even possible. So there’s some mixed messages going on about testing.
• The things that obviously should have tests have tests. That means most of the things that touch money on the Careers product, and easily unit-testable features on the Core end (things with known inputs, e.g. flagging, our new top bar, etc), for most other things we just do a functionality test by hand and push it to our incubating site (formerly meta.stackoverflow, now meta.stackexchange).

Monitoring / Logging

• Now considering using http://logstash.net/ for log management. Currently a dedicated service inserts the syslog UDP traffic into a SQL database. Web pages add headers for the timings on the way out which are captured with HAProxy and are included in the syslog traffic.
• Opserver and Realog. are how many metrics are surfaced. Realog is a logging display system built by Kyle Brandt and Matt Jibson in Go
• Logging is from the HAProxy load balancer via syslog instead of via IIS. This is a lot more versatile than IIS logs.

Clouding

• Hardware is cheaper than developers and efficient code. You are only as fast as your slowest bottleneck and all the current cloud solutions have fundamental performance or capacity limits.
• Could you build SO well if building for the cloud from day one? Mostl likely. Could you consistency render all your pages performing several up to date queries and cache fetches across that cloud network you don’t control and getting sub 50ms render times? That’s another matter. Unless you’re talking about substantially higher cost (at least 3-4x), the answer is no – it’s still more economical for SO to host in their own servers.

Performance As A Feature

• StackOverflow puts a heavy emphasis on performance. The goal for the main page  is to load in less than 50ms, but can be as low as 28ms.
• Programmers are fanatic about reducing page load times and improving the user experience.
• Timings for every single request to the network are recorded. With these kind of metrics you can make decisions on where to improve your system.
• The primary reason their servers run at such low utilization is efficient code. Web servers average between 5-15% CPU, 15.5 GB of RAM used and 20-40 Mb/s network traffic.  The SQL servers average around 5-10% CPU, 365 GB of RAM used, and 100-200 Mb/s of network traffic. This has three major benefits: general room to grow before and upgrade is necessary; headroom to stay online for when things go crazy (bad query, bad code, attacks, whatever it may be); and the ability to clock back on power if needed.

Lessons Learned

• Why use Redis if you use MS products? gabeech: It’s not about OS evangelism. We run things on the platform they run best on. Period. C# runs best on a windows machine, we use IIS. Redis runs best on a *nix machine we use *nix.
• Overkill as a strategy. Nick Craver on why their network is over provisioned: Is 20 Gb massive overkill? You bet your ass it is, the active SQL servers average around 100-200 Mb out of that 20 Gb pipe.  However, things like backups, rebuilds, etc. can completely saturate it due to how much memory and SSD storage is present, so it does serve a purpose.
• SSDs Rock. The database nodes all use SSD and the average write time is 0 milliseconds.
• Know your read/write workload.
• Keeping things very efficient means new machines are not needed often. Only when a new project comes along that needs different hardware for some reason is new hardware added. Typically memory is added, but other than that efficient code and low utilization means it doesn’t need replacing. So typically talking about adding a) SSDs for more space, or b) new hardware for new projects.
• Don’t be afraid to specialize. SO uses complicated queries based on tags, which is why a specialized Tag Engine was developed.
• Do only what needs to be done. Tests weren’t necessary because an active community did the acceptance testing for them. Add projects only when required. Add a line of code only when necessary. You Aint Gone Need It really works.
• Reinvention is OK. Typical advice is don’t reinvent the wheel, you’ll just make it worse, by making it square, for example. At SO they don’t worry about making a “Square Wheel”. If developers can write something more lightweight than an already developed alternative, then go for it.
• Go down to the bare metal. Go into the IL (assembly language of .Net). Some coding is in IL, not C#. Look at SQL query plans. Take memory dumps of the web servers to see what is actually going on. Discovered, for example, a split call generated 2GB of garbage.
• No bureaucracy. There’s always some tools your team needs. For example, an editor, the most recent version of Visual Studio, etc. Just make it happen without a lot of process getting in the way.
• Garbage collection driven programming. SO goes to great lengths to reduce garbage collection costs, skipping practices like TDD, avoiding layers of abstraction, and using static methods. While extreme, the result is highly performing code. When you’re doing hundreds of millions of objects in a short window, you can actually measure pauses in the app domain while GC runs. These have a pretty decent impact on request performance.
• The cost of inefficient code can be higher than you think.  Efficient code stretches hardware further, reduces power usage, makes code easier for programmers to understand.

( via HighScalability.com )