[Optimization] Parallel Computing, Concurrent Computing and Threading

Parallel Computing =/= Concurrent Computing. What is threading?

Parallel computing is a type of computation in which many calculations or the execution of processes are carried out simultaneously.

Large problems can often be divided into smaller ones, which can then be solved at the same time.

There is no taking turns, they are advanced at the same time. Naturally this is not possible with single-core CPU, but multiple-core architecture is required instead.

It can be said that if computation is parallel it is also concurrent (but not necessarily the other way around) - since parallel computation also fulfills the definition of concurrent computation.





Concurrent computing is a form of computing in which several computations are executed during overlapping time periods—concurrently—instead of sequentially (one completing before the next starts).

This is not exactly same as Parallel computing. Concurrent operation means that two computations can both make progress and advance regardless of the other. If there are two threads, for example, then both make progress independently. The second computation doesn't need to wait for the first one to complete before it can be advanced.

In single-core setup it is possible that the computations take turns in execution. The CPU might run few instructions from the other thread, then suspend it and switch over to the second and run few steps from it and so on.

Summary

Concurrent = Two queues and one coffee machine.
Parallel = Two queues and two coffee machines.

Threading vs Parallelism

Threading is usually referred to having multiple processes working at the same time on a single CPU (well actually not - you think they do but they switch very fast between them).

Parallelism is having multiple processes working at the same time on multiple CPU's.

You can have multi-threading on a single core machine, but you can only have parallelism on a multi core machine. Thus, threading is a specific implementation of parallelism. 

In general:
  • If we think CPU as a company and threads as its workers then, it help us to understand threading and parallelism more easily.
  • Like a company have many workers, the CPU also have many threads.
  • Also there may be more than one company and therefore there may be more than one CPU's.
  • Therefore when workers(threads) work in a company(CPU), it is called threading.
  • And when two or more companies(CPU) work independently or together, it is called parallelism.

More Reading:

Good read on threading and parallism performance analysis using Ruby (and JRuby - since Ruby doesn't have parallism) -->  https://hackernoon.com/need-faster-code-try-multithreading-5dc30c83837c

Resource:

  • https://stackoverflow.com/questions/806499/threading-vs-parallelism-how-do-they-differ
  • https://stackoverflow.com/questions/1897993/what-is-the-difference-between-concurrent-programming-and-parallel-programming
  • https://en.wikipedia.org/wiki/Parallel_computing
  • https://en.wikipedia.org/wiki/Concurrent_computing
  • https://bytearcher.com/articles/parallel-vs-concurrent/

Comments

Post a Comment

Popular posts from this blog

[Redis] Redis Cluster vs Redis Sentinel

1) In-Depth Guides to Redis Cluster and Sentinel For more in-depth explanation for Redis Cluster or Redis Sentinel, please read my other posts: Sentinel:  https://notafraidofwong.blogspot.com/2019/05/redis-redis-sentinel.html Cluster:  https://notafraidofwong.blogspot.com/2019/05/redis-redis-cluster.html 2) Deciding between Redis Sentinel and Redis Cluster Similarities : Both solutions provide high availability for your system. 2.1) Redis Cluster 2.1.1) Pros Shards data across multiple nodes Has replication support Has built-in failover of master 2.1.2) Cons Not every library supports it May not be as robust (yet) as Standalone Redis or Sentinel Setup and maintenance is more complicated 2.2) Redis Sentinel 2.2.1) Pros Automatically selects new master in case of failure Easy to setup, (seems) easy to maintain 2.2.2) Cons No data sharding; master might be overutilized. Another distributed system to maintain. 3) So how do I decide? In
Read more

[Unit Testing] Test Doubles (Stubs, Mocks....etc)

Image
In unit testing, isolation is key. The goal of unit testing is to test individual components, and not an entire system . The class/object/function you are testing is System Under Test (SUT) , and the other components of the system are Collaborators or Dependencies. Test Double  is a generic term for any kind of 'pretend' object used in place of a real object for testing purposes.  There are several types of Testing Doubles. We will start with: Fake Stub Mock Command Queries Mocks vs Stubs And then: Dummy Spies Spies vs Mocks I) Basic Testing Doubles a) Fake Fake are objects that have working implementations, but not same as production one. Usually they take some shortcut and have simplified version of production code. Note that Fake is a generic term - that can point to anything; usually mock and stubs. An example of this is an in-memory implementation of a database. This fake implementation will not access the actual database, but will u
Read more

[Node.js] Pending HTTP requests lead to unresponsive nodeJS

This is a recap conclusion to the potential problems of a NodeJS program I worked on previously. The program simulates thousands of clients navigating a website by sending thousands of http requests to a given API . The idea is to load test a particular API to make sure they can handle high traffic. Because we are simulating clients, it's important to keep track of the state of the simulated client as well. The problem with this program is it is buggy and becomes unresponsive from time to time (the only solution is to restart it). The problem was fixed by re-writing the entire program in Java, but it was regretful that I never had enough time to find the exact reason the nodeJS version of it failed. The following are potential causes I suspect: 1. Exceptions The NodeJS program was not written very neatly and stacked callbacks on top of callbacks on top of callbacks. This made it very difficult to keep track of exceptions and unexpected http responses are prone to happen. So
Read more