Monday, November 10, 2008
Talk Review: A Pictorial Introduction to Separation Logic
In this talk, the speaker talks about a new logic that unifies the concepts of pointers, concurrency, weak memory and communication. The speaker starts by defining four degrees of separation between two program traces: 1) they are disjoint traces 2) one is sequentially after another 3) both of the traces can be executed in parallel 4) both the traces are mutually exclusive - they cannot happen together.
It is then seen that 4) => 3) => 2) => 1) - that is, the relation 3) holds whenever 4) holds, 2) holds whenever 3) holds and so on.
"Hoare triples" are then introduced. These are related to the concept that post-condition are true when precondition and invariant of a program are true. The triples can be composed in interesting manners.
Though I did not understand much of what follows because of these results, through google search it seems that these results are useful in proving program correctness. The next day Dr. Hoare presented a topic related to program correctness in another conference.
Presentation slides are here (only about half could be covered in his talk). Further reading keywords: Galois Connections (pdf, pdf), Category Theory, Event Algebra.
Saturday, November 08, 2008
Wednesday, October 29, 2008
My Faves for Tuesday, October 28, 2008
PLINQ is a query execution engine that accepts any LINQ-to-Objects or LINQ-to-XML query and automatically utilizes multiple processors or cores for execution when they are ...
Microsoft Solver Foundation is a brand new framework and managed code runtime for mathematical programming, modeling, and optimization. It is designed to run on NETfx 3.5+ and is completely implemented in C#. Solver Foundation is useable from all CLS-compliant languages, including F#, IronPython, VB.NET and C++. It is also available via a visual, Add-in Designer for Excel 2007 users. Solver Foundation provides solvers and services to a broad community of users: from Excel users and analysts to programmers working on business critical scheduling, configuration, risk management, and planning solutions. It provides services for model checking, parallel solving and workload scheduling, model interchange, and declarative data binding via LINQ and other NETFx technologies. As an open framework designed for third party extensibility, it exposes facilities for users to plug-in their own solvers while still leveraging all of the modeling services and capabilities of Solver Foundation.
Sunday, October 26, 2008
My Faves for Saturday, October 25, 2008
Computer scientists at Stanford University have developed an artificial intelligence system that lets robotic helicopters teach themselves to fly difficult stunts by watching other helicopters
Saturday, October 25, 2008
Course on Introduction to Statistics - ST5201
In this course, we learnt the techniques of fitting parameters into observed data, and testing hypotheses (method of moments, maximum likelihood estimate and bootstrap methods). Distributions related to normal distribution play an important role in estimating parameters and their variance from actual value.
Our textbook: Mathematical Statistics and Data Analysis (Statistics) by John A. Rice. The book is very well-written and understandable. Technical aspects are "smoothed out" and made accessible and useful to more general audience. Cramster has several solutions to problems from the book. Here is the errata list from the book.
Other Resources
Another course on statistics is here. Read the comments on hypothesis testing here. Lectures on Decision Theory and Bayesian Inference.
SPSS and SAS are widely used statistical packages. Another free statistical tool is the GNU-S, also know as R. An R-tutorial is here.
Thursday, October 23, 2008
My Faves for Wednesday, October 22, 2008
Lecture Series on Artificial Intelligence by Prof. P. Dasgupta, Department of Computer Science & Engineering, I.I.T,kharagpur.
Wednesday, October 22, 2008
Course on The Art of Doing Research - CS6281
We also read and discussed several papers related to artificial intelligence - first in context of solving one problem of motion planning, and then several individual topics. The aim of reading the papers was not the technical aspects, but the relationship between ideas, what is new in the paper, what is good and bad about the papers and so on. For example, in the research done in motion planning, we read papers on:
- Devising framework that simplifies the problem.
- Uniform sampling for computational efficiency and approximation
- More intelligent sampling
- Computational efficiency of collision detection
- Different approach of sampling: Sensing and utility
- Another approach of sampling: lazy sampling
This gave an overview of how researchers till now approached the problems, and what novel was contributed by the later papers (when many aspects of the problem were already solved).
Sunday, October 19, 2008
Talk Review: Internet Pioneer Vinton Cerf
1. India and China will have very large internet population and economies by 2035. Fresh water and energy would be more precious than today.
2. Collaboration on the internet creates many new opportunities - like early discovery of epidemics using medical queries made by people on the internet. Real time data will be much more useful and available.
3. People will travel a lot less for office and virtual interactions will be much more real.
4. Much more kinds of devices and individual contributions are enabled by the internet. Devices will be more intelligent.
5. Much more efficient and managable computers will be used in cloud computing. Processes can have unique identifiers which enables them to transfer from one computer to another.
6. Work on interplanetary internet using new protocols is in its initial phase.
Friday, October 17, 2008
My Faves for Thursday, October 16, 2008
Course - Sublinear Algorithms - Only a small portion of the input data is read in such algorithms.
[tags: algorithms, palgos, resources]
Wednesday, August 13, 2008
Presentation Review: How the Computational Perspective is Transforming the Sciences
In this talk, Papadimitriou speaks about how computational aspects of sciences have influenced the sciences and are leading to several important results. He touches upon several sciences: quantum physics, statistical mechanics, biological systems, game theory, social networks, economics and brain as he explains how they are deeply interacting with computer science.
Quantum computers can be used to solve problems which otherwise take an exponential amount of time. However, the existing quantum computer experiments work for small number of particles (~6), which translates to quantum algorithms succeeding only for small inputs. As of now, it is unclear whether quantum computers can be built for large inputs. This is a question of testing the theory of Quantum Physics. As Umesh Vazirani says "Quantum computation is as much about testing Quantum Physics as it is about building powerful computers".
In statistical mechanics, when parameters of local interactions evolve, macroscopic properties
change dramatically, and we have a phase transition (melting of solid into liquid is an example). In computer science, certain randomized algorithms are known to converge exponentially faster when the parameters are in the right range. A deep fact is that two phenomena are indeed the same. The same phenomena also occur/used in coding theory and artificial intelligence, as this reference explains.
Then Papadimitriou moves on to explore the relation between biological systems and computer science. One example he shows is a 2006 paper "An optimal brain can be composed of conflicting agents" in which the authors find that "under natural physiological limitations, an optimal decision-making system can involve ‘selfish’ agents that are in conflict with one another, even though the system is designed for a single purpose." This result is on common ground between artificial intelligence multi-agent systems and theory of evolution in biology.
The next biology example is related to Charles Darwin's statement "To think that the eye could evolve by natural selection seems, I freely confess, absurd to the highest degree” considering that such an evolution might be highly improbable. Leslie Valiant in a recent paper "Evolvability" suggests a theory to "explain quantitatively which mechanisms can evolve in realistic population sizes within realistic time periods, and which are too complex. If evolution merely performed a random search it would require exponential time, much too long to explain the complexity of existing biological structures." Natural selection has been used for decades in solving large search problems in artificial intelligence and computer science.
The third biology example involves population selection processes used in search in artificial intelligence and computer science (also on the lines of Darwin's theory of selection). Simulated annealing and genetic algorithms are widely used search methods. In these methods search is carried out by a population of search "threads" - and these threads evolve through a selection process in each "generation" of threads. Simulated annealing is akin to asexual reproduction. Genetic algorithms are akin to sexual reproduction. Simulated annealing works so much better than genetic algorithms in solving optimization problems. Why is that? As per a recent paper "Genetic Modularity" by Livnat and Papadimitriou (yet to be published), sex favors variance over optimization. This also means sex favors creation of new species.
Internet is a large system that was never designed and its study is akin to the study of natural processes and social science. In game theory, Nash and other proved the existence of equilibria, which are predictions of behavior. Computing equilibria can be used to predict the stock market, for example. A recent result (paper here) says that computing a Nash equilibrium is an intractable problem.
In social networks, recent results (paper) show that "it is a small world". That is, most people are within a few "hops" or connections from you. This result follows from maths developed for study of internet viz how many routers does a packet have to traverse to reach its destination.
Link to the original presentation.
Tuesday, July 29, 2008
Challenges in Parallel Computing
Parallel Programming is not constrained to just the problem of selection of whether to code using threads, message passing or some other tool. Having said that, I would tend to apply the 80-20 rule to deduce that most junior programmers and most high level applications would need to consider only this issue. But in general, all the members of the software team must participate in parallelization and must consider the overall picture containing a plethora of issues:
- Understanding the hardware: An understanding of the parallel computer architecture is necessary for efficient mapping and distribution of computational tasks. A simplified classification of parallel architectures is UMA/NUMA and distributed systems. A typical application may have to run on a combination of these architectures.
- Mapping and distribution on to the hardware: Mapping and distribution of both computational tasks on processors and of data onto memory elements must be considered. The whole application must be divided into components and subcomponents and then these components and subcomponents distributed on the hardware. The distribution may be static or dynamic.
- Concurrency Implementation using Threading and Shared Memory/Message Passing/something else: Different kinds of concurrency implementation may be useful for different applications and different levels of software and hardware. A small fine grained software component running within a single die may be implemented using threading and shared memory for high performance, while message passing may be used in cases performance is not an issue or the computations are run on physically distributed computers.
- Infrastructure level: A library of concurrent data structures having a well defined interface. This library obviously has to be based on the kind of concurrency implementation.
- ... still to do (I am sure there are more things)
- This is an important issue since it is a new way of programming and the whole team and hierarchy must be concerned.
- There is no kill-all-birds gun. All parallelization tools and methods are different shades, and useful in different situations.
- It would be useful to simplify the implementation as much as possible. For example simpler to use tools (simpler to debug, implement and reason about) must be preferred in case performance is not important.
Monday, April 28, 2008
Course on Real Time Systems - EE5903
The following major topics were covered:
- Process Management, Synchronization, Concurrency (handout from Operating System Principles by Bic and Shaw)
- Distributed RTS, Deadlock Management, Distributed Deadlocks (papers on HARTS, Token Passing for mutual exclusion)
- RTS scheduling (paper on Rate Monotonic Scheduling by Lui and Layland), RT Operating System Kernel (paper on Spring Architecture)
- RTS system development diagramming - DARTS, UML statechart (paper on Software Design by Gomaa)
Some ideas for future work:
- Development of a flexible-load distributed game by continuing the work done in project.
- Multithreading the Box2D library - this is an open source physics library we used in our project.
Course on Knowledge Based Systems - CS4244
We encountered the following systems: STRIPS (planner), MYCIN (medical diagnosis rule based system), EMYCIN (empty mycin), R1/XCON (configuration of computer systems), CENTAUR (explanation system), MOLGEN (genetic experiment planner).
Book we followed: Engrossing book by Peter Jackson: "Introduction to Expert Systems"
We did a project on "Weekend Itinerary Planning". Here is the report. The planner had the following major components: itinerary generation, event selection, travel selection and user interface.
From this course, I generated two ideas for "good things to have":
- An Itinerary Planner for Weekend Travel (not just a toy project but on WWW);
- A Rule Based Library in C++/Java based on CLIPS engine - with one use in Computer Gaming.
Tuesday, February 19, 2008
Courses I did
* Advanced Computer Architecture This course taught Flynn's taxonomy, speedup, pipelining, caching and bus. In the project got some experience of coding with Cell Broadband.
* Knowledge Based Systems This course covered Rule Based Systems, declarative programming, representing uncertainty, knowledge representation, knowledge acquisition, design for explanation, and basics of blackboard architectures, case based reasoning and truth maintenance. We did project on Weekend Itinerary Planning. More ...
* Real Time Systems Diagramming, Real Time Scheduling, Synchronization, Deadlock Management and Distributed Real Time Systems. More ...
* Introduction to Statistics Probability, Random Variables, Expected Values, Distribution related to normal distribution, limit theorems, estimation of parameters, hypothesis testing. More ...
* The Art of Doing Research Making good presentations, relations between research papers, writing research plan, writing research paper well. More ...
Saturday, January 19, 2008
An Essential Note on Parallel Programming
See Essay by Dave Patterson: Parallel or Bust: Computing at a Crossroads for more. Follow up with View from Berkeley. A news article: Intel and Microsoft donate $20 Million to Two Universities for Multicore Research. Here is a presentation by David Patterson on What Future Apps May Be, How Research and Industry Can Work Together on Multicore Research and What is Possible.
My Research
Right now I am working in a central field of Aritificial Intelligence called Automated Planning. Specifically, I am looking for ways to scale Planning on modern computers. In our lab, we also apply our Planning technology to story planning in games. I was previously a software developer and in the future want to apply AI as an entrepreneur. Only time can tell what I will do afterwards.
A Simple Overview of Artificial Intelligence
Artificial Intelligence is applied not only to robotics, but also corporate decision-making, building large-scale systems, operations research, medicine, computer games and many other fields. If mathematics is the queen of sciences, AI must be the queen of computer sciences. More on this soon ...My Team's Research on Automated Planning
Automated Planning studies how an intelligent agent can combine actions in order to reach specified goals. An intelligent agent with ability to plan can "think ahead". Automated Planning systems have been used where human involvement is difficult or partial (like space missions, elderly assistance), or problems are quite complex (like composition of web-services) or little time is available before execution (like computer gaming, emergency evacuation). While previously most researchers studied planning for simplified toy problems, nowadays many researchers study how planning can be used in the real world.My Research
Modern computers are different from computers in the 20th century. They implement a large degree of parallelism; the parallelism is increasing every year. Software systems must be designed and implemented differently to run on these parallel computers. We study how real-time AI Planning can be implemented to effectively utilize the resources of these parallel computers.Some Ideas on Entrepreurship
AI is much more in the books and labs than in the real-world applications. Decision Support, Case-Based Reasoning and Expert Systems have been widely used previously. However, planning and scheduling are largely unexplored areas till now.Some Ideas to Stumble Upon Further in the Future
Agent-Based Simulations looks quite interesting. Large scale simulations of human/human-like agent communities can be carried out for purposes of hypothesis testing and prediction.Sunday, January 13, 2008
What is Planning?
Planning is the reasoning side of acting. It is a deliberation process that chooses and organizes actions by anticipating their expected outcomes. This deliberation aims at achieving as best as possible some prestated objectives. Automated Planning is an area of Artificial Intelligence (AI) that studies this deliberation process computationally.As opposed to an ordinary control program, a Planning System or Planner would be more flexible and able to take advantage of opportunities in a dynamic environment. Usage of a Planner may also decrease cost in some situations due to lesser involvement of humans.
Here is an example scenario of planning:
When a robot Wall-e is asked to fetch a drink from the local store, it must internally generate a plan to reach the store, buy the drink and then return back home. While going out, when Wall-e identifies that the door of the room is closed, it must update the plan to include an action to open the door and close it on the way out.
Some real life applications of planning:
- NASA used Autonomous Remote Agent system based on planning and scheduling techniques on board Deep Space mission. The Remote Agent was able to generate plans and execute high-level commands like “during the next period, take pictures of the following asteroids”.
- Robot motion planning, and goal-directed planning
- Process planning is the task of preparing detailed operating instructions for transforming an engineering design into a final product.
- Autominder is a Planning, Monitoring and Reminding Assistive Agent for the Elderly.
- Emergency evacuation planning
- Simulated environments like training and games use goal directed planners.
Our Work
At Interactive Intelligence Lab we are building a widely applicable real time, dynamic and efficient planning system. The planning system will be first applied to Automated Story Generation in Computer Gaming.See Also: AAAI Page on Planning and Scheduling
Thursday, November 22, 2007
List of Libraries
- >Intel Threading Building Blocks
- [download][tut from intel][Licensed under GPLv2 with the runtime exception]
Sunday, November 18, 2007
Sunday, October 07, 2007
Links to External Resource Links
Parallel Computing
> Parallel Computing Research Groups by Jonathan Hardwick
> Parallel Processing at IEEE Computer Society
AI
> General AI by Alexander Nareyek
> AI on the web by Russell and Norvig
Planning
> AAAI Page on Planning and Scheduling
> Planning Systems and Testbeds by Rob St. Amant
> Planet List of Planners and Schedulers
Algorithms
> Algorithms and Implementations by NIST
Saturday, October 06, 2007
Artificial Intelligence Courses
> Intelligent Information Agents for the Web by Matthias Klusch
Courses in AI Planning
> Planning and Learning by Subbarao Kambhampati
- covers a lot of material from Automated Planning: Theory and Practice. Yochan research group.
- > Automated Planning by Dan Weld
- Context of application domains ranging from NASA testbeds to the semantic web. Temporal Graphplan planner
- > AI Planning by Jim Blythe, Jose-Luis Ambite, and Yolanda Gil.
- Interactive Knowledge Capture research group.
- > Planning in Stochastic, Dynamic, Metric & Incomplete Worlds by Subbarao Kambhampati
> Agents Architectures and Multi-Agent Systems by Prof. Marie desJardins
Thursday, September 27, 2007
Herb Sutter's Talk on Machine Architecture
Machine Architecture: Things Your Programming Language Never Told YouUnfortunately, the video is not the best quality and you would need the PDF slides of the presentation - available at Sutter's blog.
Know Your Enemy: Latency
or rather,
LATENCY
To summarize, the talk describes how important latency is in computation today - that an overwhelming fraction of effort, money and space of computing resources is dedicated to reducing latency. For example, 99% of transistors in the modern CPU are used to "hide" or reduce latency, and only 1% for computation. Much of the programming effort today also has to be dedicated to reduce latency. Indeed, latency is the root of nearly all complexity in hardware.
Latency Lags Bandwidth
Reduction in latency has not kept pace with increase in bandwidth. For example, in 20 years "CPU bandwidth" or computation speed has grown 2,250X but latency has decreased only 21X. This means the contribution of latency is always growing. Memory latency is now 210 cycles, instead of 1.4 cycles in 1980 - worse off 150X. Therefore, L1, L2 caches and registers have become much more important.
Our Programming Model Must Graduate
Our programming model must graduate from a obsolete CPU-Memory-Disk to include:
- CPU: CPUs, Registers, L1 Cache - Instruction (2 cycles) and Data (3 cycles), L2 Cache (14 cycles), Store Buffers
- Memory: DRAMs (200 cycles), Flash (3M cycles) and other nodes
- Disk: Flash, Hard Disk including compressed files (15M cycles), Network
Some numbers would be indicative of the scale of importance of changing the computation model - see figure here.
Little's Law says Concurrency = Bandwidth x Latency and is useful in applications in many areas of science. Therefore, we have to add concurrency everywhere to cope up with latency. This may even affect the correctness of my code, not just performance. Compilers have to attempt to go all the way in optimizing the code, that it breaks some implicit (incorrect) assumptions programmers often make.
In summary, to save the correctness of your code, Sutter says the following:
* Use locks to protect the shared variables.
* Avoid lock-free code since it is too difficult.
* Avoid fences or memory barriers.
There is a scenario when taking conditional locks may not be advisable since the compiler would rearrange the code so that some code might be executed without the condition. So pessimistically take locks even if updates are conditional and you believe you do not need a lock in a particular condition.
To exploit locality, the memory allocation, and memory access pattern become important.
Saturday, September 15, 2007
First Encounter with Intel's Threading Building Blocks
We are now seeing the initial stages of the next major change in software development, as over the next few years the software industry brings concurrency pervasively into mainstream software development, just as it has done in the past for objects, garbage collection, generics, and other technologies.
The next speaker in the webinar series is James Reinders, on the topic "Steps to Parallelism NOW". He is also the author of "Intel® Threading Building Blocks: Outfitting C++ for Multi-core Processor Parallelism". Chapter 1 of the book is available for download here (excerpt below in this article).
Intel® Threading Building Blocks
The Intel site says "Intel Threading Building Blocks" library has ready to use parallel algorithms, cross-platform support, task based parallelism, highly concurrent containers and generic algorithms. More on this here.
The library is open source and released under GPLv2 with the runtime exception - which means, it is as good as using gcc libstdc++ in your application, and I think (and please correct me if I'm wrong) commercial applications can use the library without restrictions and without releasing code as open source.
Some related notes on Threading Building Blocks:
Why Too Many Threads Hurts Performance, and What to do About It
This article discusses the impact of having too many threads.
Partitioning a fixed amount of work among too many threads gives each thread too little work that the overhead of starting and terminating threads swamps the useful work.Since the most efficient number of compute threads depends upon the hardware, the division of program into threads is not a good way to do multithreaded programming. It is often better to do formulate the program into logical tasks, which are much more lightweight than logical threads. Task based programming also allows the distribution of tasks to processors unfairly to gain efficiency while giving up fairness. Task based programming is a lot easier since programmer is not forced to think about the low level parallelism. Scalability is easier to achieve using task based approach.
The article gives an example of summing a tree. Under breadth-first execution traversal, parallelism is maximized but so is memory consumption because the number of tasks doubles at each level of execution - there are many more tasks than hardware parallelism supports. Using depth-first traversal reduces both parallelism and memory consumption. The queue for keeping tasks is a shared resource and may become a bottleneck due to communication overheads.
A modern method to solve these problems is to keep a separate queue for each thread, and "work depth-first, steal breadth-first". This means memory consumption is reduced, while there are just enough tasks to keep the hardware busy. This technique was used by MIT's Cilk language project.
Excerpts from Chapter 1 of "Intel Threading Building Blocks" by James Reinders
When considering it for C programs, OpenMP has been referred to as “excellent for Fortran-style code written in C.” That is not an unreasonable description of OpenMP since it focuses on loop structures and C code. OpenMP offers nothing specific for C++. Looking to address weaknesses in OpenMP, Threading Building Blocks is designed for C++, and thus to provide the simplest possible solutions for the types of programs written in C++.
Sunday, September 02, 2007
Hi!
[Please Note: This blog is no longer updated. Please visit www.cattywumpus.in for updated articles.]
Amit Kumar
PhD Student
Interactive Intelligence Lab
Electrical and Computer Engineering
National University of Singapore
email: amit_kumar at nus dot edu.sg
research | courses | resources | personal page | home
Welcome to my page. I intend to post most of my research-related work here. All the entries on this site are always under construction.