Posts

[Node.js] Pending HTTP requests lead to unresponsive nodeJS

This is a recap conclusion to the potential problems of a NodeJS program I worked on previously. The program simulates thousands of clients navigating a website by sending thousands of http requests to a given API . The idea is to load test a particular API to make sure they can handle high traffic. Because we are simulating clients, it's important to keep track of the state of the simulated client as well. The problem with this program is it is buggy and becomes unresponsive from time to time (the only solution is to restart it). The problem was fixed by re-writing the entire program in Java, but it was regretful that I never had enough time to find the exact reason the nodeJS version of it failed. The following are potential causes I suspect: 1. Exceptions The NodeJS program was not written very neatly and stacked callbacks on top of callbacks on top of callbacks. This made it very difficult to keep track of exceptions and unexpected http responses are prone to happen. So

[AI Talk Summary] Adapting and Explaining Deep Learning for Autonomous Systems - Trevor Darrell

Problem definition The talk was given by Prof. Trevor Darrell and he starts by comparing Machine Learning models to the mind of a human being. He questions why ImageNet can perform very well in static images, but poorly in videos even though a video is just a sequence of images? Possible reasons to this question is dataset bias, which prevents the model to adapt to different environment (such as lower resolution than trained), alterations in the image (such as motion blur) and so forth. Prof. Trevor Darrell is concerned that machine learning models are trained only to do specific tasks and he goes on to talk about his vision of “Beyond Supervised AI”. There are three themes: Adaptation How can we build models that can work across domains (or a change in environment)? Exploration How can we teach a model to explore instead of providing it very specific rewards/goals? Explanation Can we design models to tell us why they think the way they do? Algorithm, Results and Discu

[Cloud Computing] Kafka

What is the paper trying to do? The paper is trying to introduce a novel messaging system for log processing, called Kafka. Kafka is a combination of the benefits of traditional log aggregator and messaging systems. Kafka is useful in processing huge volumes of log data streams, such as a messaging system. It is able to achieve higher throughput than other messaging systems because it focuses on log processing applications. There is a also integration for distributed support and can scale out. What do you think is the contribution of the paper? I think the main contribution of Kafka is the decisions the developer made to implement Kafka. For example, to make the system efficient, Kafka has a very simple storage layout, where each partition of a topic corresponds to a logical log. Another interesting decision is by making the consumer keep track of how much each consumer has consumed, as opposed to traditional methods where it is done by the broker. This reduces a lot of complexity

[Cloud Computing] Supporting Security Sentitive Tenants in a Bare Metal Cloud Review

What is the paper trying to do? The paper, “Supporting Security Sentitive Tenants in a Bare Metal Cloud”, is trying to present Bolted, an architecture for a bare metal cloud. Bolded is special in that it satisfies the needs of both security sensitive and insensitive tenants; as security sensitive tenants can control their own security and insensitive tenants can use default security. What do you think is the contribution of the paper? What are its major strengths? Allows security sensitive tenants to control their security (assuming physical security and availability is not an issue). Does not impose overhead on security insensitive tenants and does not employ extra cost on flexibility and operational efficiency of the provider. Eliminate the need to trust the provider to disk scrub via disk-less provisioning. “remote attestation” allow tenants to inspect source code used to generate the firmware it runs on. Performance Evaluation Can rapidly set up secure servers with com

[Cloud Computing] Dynamo (a highly available key-value storage system)

What is the paper trying to do? This paper is trying to “present the design and implementation of Dynamo, a highly available key-value storage system”, that is used in Amazon’s core services. By sacrificing consistency under certain failure scenarios, Dynamo is able to reach an “always-on” experience in terms of availability. This allows it to be successful in handling server failures, data center failures and network partitions. Additionally, Dynamo is incrementally scalable and can scale up and down while it is up and running. Dynamo uses a combination of technologies, including extensive use of object versioning, application-assisted conflict resolution, data partitioning, replication via consistent hashing. Additionally, during updates, quorum-like technique and a decentralized replica synchronization protocol is used to maintain consistency amongst replicas. What do you think is the contribution of the paper? The main contribution of this paper is “the evaluation of how differ

[Cloud Computing] Spark: Cluster Computing with Working Sets

What is the paper trying to do? What do you think is the contribution of the paper? The paper, “Spark: Cluster Computing with Working Sets”, is trying to explain Spark, a new framework that can retain the scalability and fault tolerance similar to MapReduce, and at the same time, support applications that reuse a working set of data across multiple parallel operations. This type of application is not as effective in MapReduce because MapReduce only works well with acyclic data flow graphs. However, it works well in Spark. Spark achieves this by using an abstraction called “resilient distributed datasets (RDD)”. It is a “read-only collection of objects partitioned across a set of machines that can be rebuilt if a partition is lost”. Although Spark is still in prototype, the authors demonstrate that Spark can outperform “Hadoop by 10x in iterative machine learning workloads and can be used interactively to scan a 39GB dataset with sub-second latency”. What are its major strengths?

[Distributed File System] MapReduce: simplified data processing on large clusters Overview

This is a paper summary of the paper, "MapReduce: simplified data processing on large clusters". What is the paper trying to do? Inspired by the map and reduce primitives in functional languages, the paper introduces a new abstraction whereby map and reduce operations allows the program to parallelize large computations easily. In short, it allows the anyone to execute programs with parallelization, fault-tolerance, data distribution and load balancing without bothering with the mess that usually comes with it. What do you think is the contribution of the paper? The major contribution of this paper is developing an interface that distributes large-scale computations using MapReduce. This allows it to achieve ‘automatic parallelization’. Another big contribution is implementing this interface on large clusters of commodify PCs. What are its major strengths? Model is easy-to-use. Because it hides away the details of parallelization, fault-tolerance, locality optimiza