[100 Days of ML Code] OneHotEncoder

One Hot Encoding is process by which categorical variables are converted into a form that could be provided to ML algorithms to do a better job in prediction.

Normally, we will have a dataset of:
╔════════════╦════════╗ 
║ CompanyName║ Price  ║
╠════════════╣════════║ 
║ VW         ║ 20000  ║
║ Acura      ║ 10011  ║
║ Honda      ║ 50000  ║
║ Honda      ║ 10000  ║
╚════════════╩════════╝



Many ML algorithms cannot work with label data (as a string) directly. Therefore, we need to convert these labels into some numeric value:






╔════════════╦═════════════════╦════════╗ 
║ CompanyName Categoricalvalue ║ Price  ║
╠════════════╬═════════════════╣════════║ 
║ VW         ╬      1          ║ 20000  ║
║ Acura      ╬      2          ║ 10011  ║
║ Honda      ╬      3          ║ 50000  ║
║ Honda      ╬      3          ║ 10000  ║
╚════════════╩═════════════════╩════════╝



Now, 'categoricalValue' is a numerical value that represents the companyName.



However, the problem with the above table is it assumes that higher the categorical value, the better the category is; and we know that that is not correct.



Therefore, we use OneHotEncoding, which makes our table look like this:


╔════╦══════╦══════╦════════╦
║ VW ║ Acura║ Honda║ Price  ║
╠════╬══════╬══════╬════════╬
║ 1  ╬ 0    ╬ 0    ║ 20000  ║
║ 0  ╬ 1    ╬ 0    ║ 10011  ║
║ 0  ╬ 0    ╬ 1    ║ 50000  ║
║ 0  ╬ 0    ╬ 1    ║ 10000  ║
╚════╩══════╩══════╩════════╝



One hot encoder performs "binarization" of the category.

The CategoricalValue is removed and replaced with 3 columns because we have 3 different label (vw, acura, honda).

You can think of each column as is_vw, is_acura, is_honda, relatively.



While this table looks overly-complicated, it works very well with most ML algorithms.

The binary variables (VW, Acura, Honda) are known as "Dummy Variables"







Resources

https://www.quora.com/What-is-one-hot-encoding-and-when-is-it-used-in-data-science

https://hackernoon.com/what-is-one-hot-encoding-why-and-when-do-you-have-to-use-it-e3c6186d008f

https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/







































Comments











Post a Comment























Popular posts from this blog









[Redis] Redis Cluster vs Redis Sentinel





















 1) In-Depth Guides to Redis Cluster and Sentinel   For more in-depth explanation for Redis Cluster or Redis Sentinel, please read my other posts:    Sentinel:  https://notafraidofwong.blogspot.com/2019/05/redis-redis-sentinel.html   Cluster:  https://notafraidofwong.blogspot.com/2019/05/redis-redis-cluster.html   2) Deciding between Redis Sentinel and Redis Cluster   Similarities : Both solutions provide high availability for your system.    2.1) Redis Cluster   2.1.1) Pros    Shards data across multiple nodes  Has replication support  Has built-in failover of master     2.1.2) Cons    Not every library supports it  May not be as robust (yet) as Standalone Redis or Sentinel  Setup and maintenance is more complicated     2.2) Redis Sentinel   2.2.1) Pros    Automatically selects new master in case of failure  Easy to setup, (seems) easy to maintain     2.2.2) Cons    No data sharding; master might be overutilized.  Another distributed system to maintain.     3) So how do I decide?   In








Read more










[Unit Testing] Test Doubles (Stubs, Mocks....etc)




















Image







In unit testing, isolation is key. The goal of unit testing is to test individual components, and not an entire system .   The class/object/function you are testing is System Under Test (SUT) , and the other components of the system are Collaborators or Dependencies.     Test Double  is a generic term for any kind of 'pretend' object used in place of a real object for testing purposes.      There are several types of Testing Doubles. We will start with:    Fake  Stub  Mock  Command Queries  Mocks vs Stubs    And then:    Dummy  Spies  Spies vs Mocks       I) Basic Testing Doubles    a) Fake   Fake are objects that have working implementations, but not same as production one. Usually they take some shortcut and have simplified version of production code.  Note that Fake is a generic term  - that can point to anything; usually mock and stubs.   An example of this is an in-memory implementation of a database. This fake implementation will not access the actual database, but will u








Read more










[Node.js] Pending HTTP requests lead to unresponsive nodeJS





















This is a recap conclusion to the potential problems of a NodeJS program I worked on previously.   The program simulates thousands of clients navigating a website by sending thousands of http requests to a given API . The idea is to load test a particular API to make sure they can handle high traffic. Because we are simulating clients, it's important to keep track of the state of the simulated client as well. The problem with this program is it is buggy and becomes unresponsive from time to time (the only solution is to restart it). The problem was fixed by re-writing the entire program in Java, but it was regretful that I never had enough time to find the exact reason the nodeJS version of it failed.   The following are potential causes I suspect:    1. Exceptions  The NodeJS program was not written very neatly and stacked callbacks on top of callbacks on top of callbacks. This made it very difficult to keep track of exceptions and unexpected http responses are prone to happen. So








Read more