[Node.js] Web Architecture Design

1) Traditional Web Application Processing Model - Multi-Threaded Request-Response

Any Web Application developed without NodeJS typically follows the "Multi-Threaded Request-Response model" (or just "Request/Response Model"). This differs from the NodeJS way in that it uses multiple threads to handle concurrent client requests.

Under the hood, server waits in infinite loop and creates one thread per client request. If the server gets many client requests that require long blocking I/O operations, then more threads become 'busy'. As a result, the remaining clients requests have to wait longer due to core-thread resources.

1.1) Detailed Flow

  • Ahead of time:
    • Web Server is in infinite loop and waiting for Client Incoming Requests.
    • Web Server internally maintains a Limited Thread pool to provide services to Client Requests.
    • Once threads free up in thread pool, server pick up those threads and assign them to remaining client requests.
  1. Clients send request to Web Server
  2. Web Server receives request.
  3. Web Server picks a thread from thread pool to process Client Request, including blocking I/O operations.
  4. This thread finishes processing and sends a response back to Client.

1.2) Drawbacks

  • When concurrent client requests increase, it uses more threads, which then eat up more memory (highly dependent on memory)
  • Under high traffic, Client's requests need to wait for available threads to process their requests.
  • Wastes time in processing blocking IO tasks.
  • Cannot handle as many concurrent client requests as NodeJS can.

2) NodeJS Architecture - Single-Threaded Event Loop

NodeJS platform does not follow Request/Response Multi-Threaded stateless model. Instead, it follows Single Threaded with Event Loop. The main reason is it scales well as the # of concurrent client requests increase. 

2.1) Detailed Flow

  • Ahead of time:
    • NodeJS Web Server internally has a infinite loop, called "single-threaded event loop". This loop checks any Client Requests placed in the Event Queue. 
    • NodeJS Web Server internally maintains a limited thread pool, via libuv, to process blocking operations.
  1. Clients send request to Web Server
  2. If client request is processed with no blocking operations, then process everything, prepare response and send back to client.
    • Note that asynchronous I/O operations are non-blocking, such as DB requests.
  3. If client request is processed with blocking operations, eg. any CPU-intensive tasks or any I/O operations that are not asynchronous, then it will assign available threads from thread pool to process  it, then send response to client when it is processed.

3) Comparisons

3.1) Concurrent Client Requests/Connections

As for the traditional multi-threaded request-response model, if we assume that each thread takes up 2MB of memory, running on a system with 8GB of RAM puts us at a theoretical maximum of 4000 concurrent connections, plus the cost of context-switching between threads. 

But for NodeJS' single-threaded event loop model, NodeJS achieves scalability of over 1M concurrent connections and over 600K concurrent websocket connections (based on http://blog.caustik.com/2012/08/19/node-js-w1m-concurrent-connections and https://blog.jayway.com/2015/04/13/600k-concurrent-websocket-connections-on-aws-using-node-js/)

3.2) Scaling with hardware

During high traffic, single-threaded NodeJS may not be able to fully utilize all of the cores, whereas the traditional model scales with cores and RAM - as long as you have enough memory and cores, you shouldn't face scalability problems.

3.3) Code Maintenance

Multi-Threads are not easy to maintain. However, a single-threaded event loop is not easy as well because everything relies on 1 thread (the single-threaded event loop). Heavy computations can choke up the event loop and unhandled errors can effectively crash the entire program.

3.4) CPU-intensive tasks

NodeJS' model is not suitable for CPU-intensive tasks, because it is single-threaded. CPU-intensive tasks does not have to complex algorithms. Sorting, JSON parsing or traversing an array can be considered CPU-intensive on medium-sized data structures; especially when you are using only 1 thread to handle thousands of client requests.

4) Hybrid

In short, by using NgInx, Apache2 or any load balancing mechanism, you can spawn multiple processes of the Web Server and distribute it across them.

Resources

Comments

Post a Comment

Popular posts from this blog

[Redis] Redis Cluster vs Redis Sentinel

1) In-Depth Guides to Redis Cluster and Sentinel For more in-depth explanation for Redis Cluster or Redis Sentinel, please read my other posts: Sentinel:  https://notafraidofwong.blogspot.com/2019/05/redis-redis-sentinel.html Cluster:  https://notafraidofwong.blogspot.com/2019/05/redis-redis-cluster.html 2) Deciding between Redis Sentinel and Redis Cluster Similarities : Both solutions provide high availability for your system. 2.1) Redis Cluster 2.1.1) Pros Shards data across multiple nodes Has replication support Has built-in failover of master 2.1.2) Cons Not every library supports it May not be as robust (yet) as Standalone Redis or Sentinel Setup and maintenance is more complicated 2.2) Redis Sentinel 2.2.1) Pros Automatically selects new master in case of failure Easy to setup, (seems) easy to maintain 2.2.2) Cons No data sharding; master might be overutilized. Another distributed system to maintain. 3) So how do I decide? In
Read more

[Unit Testing] Test Doubles (Stubs, Mocks....etc)

Image
In unit testing, isolation is key. The goal of unit testing is to test individual components, and not an entire system . The class/object/function you are testing is System Under Test (SUT) , and the other components of the system are Collaborators or Dependencies. Test Double  is a generic term for any kind of 'pretend' object used in place of a real object for testing purposes.  There are several types of Testing Doubles. We will start with: Fake Stub Mock Command Queries Mocks vs Stubs And then: Dummy Spies Spies vs Mocks I) Basic Testing Doubles a) Fake Fake are objects that have working implementations, but not same as production one. Usually they take some shortcut and have simplified version of production code. Note that Fake is a generic term - that can point to anything; usually mock and stubs. An example of this is an in-memory implementation of a database. This fake implementation will not access the actual database, but will u
Read more

[Node.js] Pending HTTP requests lead to unresponsive nodeJS

This is a recap conclusion to the potential problems of a NodeJS program I worked on previously. The program simulates thousands of clients navigating a website by sending thousands of http requests to a given API . The idea is to load test a particular API to make sure they can handle high traffic. Because we are simulating clients, it's important to keep track of the state of the simulated client as well. The problem with this program is it is buggy and becomes unresponsive from time to time (the only solution is to restart it). The problem was fixed by re-writing the entire program in Java, but it was regretful that I never had enough time to find the exact reason the nodeJS version of it failed. The following are potential causes I suspect: 1. Exceptions The NodeJS program was not written very neatly and stacked callbacks on top of callbacks on top of callbacks. This made it very difficult to keep track of exceptions and unexpected http responses are prone to happen. So
Read more