A simple optimization example using multivariate gaussian learning model

This is an example showing how to use Mateda 2.0 for finding the maximum probability of a multivariate normal distribution model. The multivariate normal distribution function in matlab is used to serve as an objective function, in which  the following parameters are used.

Number of variables: 2
means: [0 0]
Variance, co-variance matrix (Sigma): [.25 .3; .3 1]

For definition of the multivariate normal distribution function, see here.

As a result, the fitness function to be maximized is written as a matlab function as follows:

function [Z] = MVNPDF_( vector )
%MVNPDF_ Summary of this function goes here
%   Detailed explanation goes here
global mu %means
global  Sigma %variance-co-variance matrix
Z=mvnpdf(vector, mu, Sigma); %Get the probability given vector of x and y
return

Above function is saved in a file called _MVNPDF.m.

Next, the following is matlab code for finding the maximum value of the above function using EDA.

% Finding maximum of a multivariate normal distribution
PopSize = 100; %if too small say 10, it may not return good results 
n = 2; %Number of variables, x and y only 
Card(1,:) = -3*ones(1,n); %this is starting range (-3) of the variables
Card(2,:) = 3*ones(1,n); %this is ending range (3) of the variables.
cache  = [1,1,1,1,1]; %Turn on all the result logging 
% Use Multivariate Gaussian Learning model rather than 
% Univariate Gaussian Learning model (LearnGaussianUnivModel) 
edaparams{1}={'learning_method','LearnGaussianFullModel',{}};
% Of course, have to use the same class of sampling method
edaparams{2} = {'sampling_method','SampleGaussianFullModel',{PopSize,1}};
% Typical elitism replacement method
edaparams{3}={'replacement_method','elitism',{1,'fitness_ordering'}};
% Select promising solutions based on proportion of fitness
edaparams{4}={'selection_method','prop_selection',{}};
%Define mu and Sigma for setting up the fitness function F
global mu;
mu = [0 0]; % define means for F
global  Sigma;
Sigma = [.25 .3; .3 1]; % define sigma for F
F = 'MVNPDF_'; % Set F to the file name of the fitness function.
[AllStat,Cache]=RunEDA(PopSize,n,F,Card,cache,edaparams) 

The following is the results:

PopSize: 100
Number of variables: 2
Function: MVNPDF_
seeding_pop_method:  RandomInit
Params: 
sampling_method:  SampleGaussianFullModel
    'Params: '    [100]    [1]
repairing_method:  none
Params: 
local_opt_method:  none
Params: 
replacement_method:  elitism
    'Params: '    [1]    'fitness_ordering'
selection_method:  prop_selection
    'Params: '
learning_method:  LearnGaussianFullModel
    'Params: '
statistics_method:  simple_pop_statistics
    'Params: '    'fitness_ordering'
verbose_method:  simple_verbose
Params: 
stop_cond_method:  max_gen
    'Params: '    [50]
....

There were 50 generations and the following are the best individuals for each of the 50 generations.

[-0.105761617623641,0.0457401246596891]
[-0.105761617623641,0.0457401246596891]
[-0.105761617623641,0.0457401246596891]
[-0.0218213711430172,-0.0568160229471882]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[0.00104128400186346,-0.00379801083192315]
[-0.000545786613022278,0.000430316063058732]
[-0.000545786613022278,0.000430316063058732]
[-0.000545786613022278,0.000430316063058732]
[-0.000545786613022278,0.000430316063058732]
[-0.000545786613022278,0.000430316063058732]
[-0.000545786613022278,0.000430316063058732]
[-0.000545786613022278,0.000430316063058732]

[-0.000545786613022278,0.000430316063058732]

The optimal values are [-0.000545786613022278,0.000430316063058732]. Put it back to mvnpdf, e.g.mvnpdf([0 0],mu,Sigma), it returns the optimal value of Z.

Although it can find the optimal value, it is interesting to see the model it learnt. To check the learnt model, check Cache{3,:} and this returns 50  [1x2 double]    [2x2 double]. Just check the last one and the following vectors/matrix can be identified.

mu'
[0.0351083399147978,-0.0201286199194872]

Sigma'
[0.00247712813037407,-0.000591245499891970;

-0.000591245499891970,0.00391133037686946]

The mu' is closed to the mu [0 0]. However, values in Sigma [.25 .3; 
.3 1] are very different than that in Sigma'.

The reason is that this problem is to find the optimal value led by the fitness function. Sigma' and Sigma are referring to different notions. Sigma' provides how accurate the mu' is, see sigma is closed to zero. Instead, Sigma is used to define the distribution model for fitness function only.

ENDS

Discrete UMDA oneMax algorithm using Mateda2.0

This blog is about how to run a discrete UMDA for solving oneMax problem using Mateda2.0

The following is the matlab code.

PopSize = 10;  # no. of population
n = 3; # no. of ones, or length of the string of the oneMax problem
cache  = [1,1,1,1,1]; # matrix to record 1th one.Entire population, 2nd one. Selected population, 3rd one. Probabilistic model, 4th one. Fitness values of the entire population and 5th one. Fitness values of the selected population. Change the value from one to zero for turning off the logging.

Card = 2*ones(1,n); #vector of dimension of the discrete variables, or a vector showing number of possible values each node can hold, in this case, each node can hold either '0' or '1'

MaxGen=5; # Maximum number of generation to stop the algorithm
MaxVal=n; # Maximum value of the fitness function to stop the algorithm

stop_cond_params={MaxGen,MaxVal}; #Define the stopping criteria with the above two settings.

NumbVar=n; #Number of variables 
Cliques=CreateMarkovModel(NumbVar,0); #This creates a markov chain model with each variable/node depending on how many previous variables (i.e. specify by '0' in this case meaning all variables are independent.

edaparams{1}={'learning_method','LearnFDA',{Cliques}}; #define LearnFDA as the learning method, which computes the marginal frequencies for each variable.
edaparams{2} = {'sampling_method','SampleFDA',{PopSize}}; #define sampling method
edaparams{3} = {'stop_cond_method','maxgen_maxval',stop_cond_params}; #define stopping conditions
F = 'sum'; % Onemax function;
[AllStat,Cache]=RunEDA(PopSize,n,F,Card,cache,edaparams) #Run the EDA using above settings.

Below is the result:

General Settings
PopSize: 10
Number of variables: 3
Function: sum
seeding_pop_method:  RandomInit
Params: 
sampling_method:  SampleFDA
    'Params: '    [10]
repairing_method:  none
Params: 
local_opt_method:  none
Params: 
replacement_method:  best_elitism
Params: fitness_ordering
selection_method:  truncation_selection
    'Params: '    [0.5000]    'fitness_ordering'
learning_method:  LearnFDA
    'Params: '    [3x3 double]
statistics_method:  simple_pop_statistics
    'Params: '    'fitness_ordering'
verbose_method:  simple_verbose
Params: 
stop_cond_method:  maxgen_maxval
    'Params: '    [5]    [3]

*****************  Generation 1 ********************** 

Max objective values: 2 
Sum of max objective values: 2 
Mean objective values: 1.400000e+00 
Sum of mean objective values: 1.400000e+00 
Median objective values: 1 
Sum of median objective values: 1 
Min objective values: 1 
Sum of min objective values: 1 
Variance of the objective values: 2.666667e-01 
Best individual: 0 1 1 
Number of different individuals: 5 
Max values of the variables: 1 1 1 
Mean values of the variables: 2.000000e-01 7.000000e-01 5.000000e-01 
Median values of the variables: 0 1 5.000000e-01 
Min values of the variables: 0 0 0 
Variance of the variables: 1.777778e-01 2.333333e-01 2.777778e-01 
number of evaluations: 10 
Time: sampling 0, repairing 0, evaluation 0, local_opt 0, replacement 0, selection 0, learning 0, and total 1.560010e-02 

*****************  Generation 2 ********************** 

Max objective values: 3 
Sum of max objective values: 3 
Mean objective values: 2 
Sum of mean objective values: 2 
Median objective values: 2 
Sum of median objective values: 2 
Min objective values: 1 
Sum of min objective values: 1 
Variance of the objective values: 2.222222e-01 
Best individual: 1 1 1 <<<< best solution found then stop iteration
Number of different individuals: 5 
Max values of the variables: 1 1 1 
Mean values of the variables: 3.000000e-01 8.000000e-01 9.000000e-01 
Median values of the variables: 0 1 1 
Min values of the variables: 0 0 0 
Variance of the variables: 2.333333e-01 1.777778e-01 1.000000e-01 
number of evaluations: 20 
Time: sampling 3.120020e-02, repairing 0, evaluation 0, local_opt 0, replacement 0, selection 0, learning 0, and total 3.120020e-02 

We should see the following output matrices 
AllStat = 
    [5x1 double]    [1x3 double]    [5]    [5x3 double]    [10]    [1x8 double]
    [5x1 double]    [1x3 double]    [5]    [5x3 double]    [20]    [1x8 double]

AllStat stores information about population in each generation. For each generation, it provides six groups of information and we take the above case as an example such that:
1st generation
[2;1.40;1;1;0.267] [0,1,1] 5 5x3 double 10 [0,0,0,0,0,0,0,0.0156]
2nd generation
[3;2;2;1;0.22] [1,1,1] 5 5x3 double 20 [0.0312002000000007,0,0,0,0,0,0,0.0312]

There are two rows because two generations of population were computed only. For each row, it contains the following (Take the 1st row as an example):
1st item: [2;1.40;1;1;0.267] are max,mean,median,min and variance of the fitness function in that population
2nd item: [0,1,1] provides the best individual in the population
3rd item: 5 is the no.of different individuals in the population
4th item: 5x3 double provides a matrix of 5 rows x no.of variables (i.e.3). The five values in each rows provide max,mean,median,min and variance of each variables. Values found in the 1st generation:
(max)1 1 1
(mean)  0.20 0.70 0.50
(median)0 1 0.50
(min)   0 0 0

(var)   0.17 0.23 0.278
5th item: 10, this is no. of function evaluations until this generation. 
6th item: [0,0,0,0,0,0,0,0.0156], time for each of the steps in seconds:sampling, repairing, evaluation, local optimization, replacement, selection, learning and total. 

Cache = 
    [10x3 double]    [10x3 double]
    [ 5x3 double]    [ 5x3 double]
    { 1x2 cell  }    { 1x2 cell  }
    [10x1 double]    [10x1 double]
    [ 5x1 double]    [ 5x1 double]

The following is result of the cache:
10x3 double       10x3 double           (Entire population)
5x3 double                 5x3 double            (Selected population)
1x2 cell                 1x2 cell              (Probabilistic model)
[2;1;2;1;1;1;1;2;2;1] [3;2;2;2;2;2;2;2;2;1] (Fitness values of the population)

[2;2;2;2;1]         [3;2;2;2;2]           (Fitness values of selected pop)

Regarding the probabilistic model, I have the following:
1st generation

[0,1,1;0,1,2;0,1,3] This is the structure of the clique. This indicates three independent variables.
1x3 cell 

[0.714285714285714,0.285714285714286] This indicates probability of '0' and '1'
[0.285714285714286,0.714285714285714] ditto
[0.285714285714286,0.714285714285714] ditto

It is interesting to see the probability of getting '0' is higher than that of '1' in the 1st variable. Supposingly, '1' should get the higher probability. This situation is reversed meaning the probability of getting '1' for all variables becomes high when the number of generation is high, say 5.

Nevertheless, once the probabilistic model indicating bias to '1', it is not necessary gain higher probability, higher fitness value can easily be generated.

ENDS.

Parameter file for ECJ

The following is an example of parameter file for running ECJ.

#No. of breeding threads, in general, it is set it to match the
#no. of cores or
#processors running on the computer
breedthreads = 1

#No. of evaluation threads
evalthreads = 1

#Setting the seed to ensure ECJ to produce the exact same 
#results.
seed.0 = 4357

# This setup the evolution state, which is created by ec.Evolve 
# for performing evolutionary process.
# However, prior to create the ec.EvolutionState, ec.Evolve will 
# create the following objects beforehand.
# i . Parameter Database, which holds all parameters 
# that ec.EvolutionState uses to build and process
# ii. Output, e.g. out.stat
# iii. Checkpointing Facility
# iv. No. of Threads, and random number generators, 
# one per thread
# v. No. of Jobs
state = ec.simple.SimpleEvolutionState

# Load class for population holding the individuals in the evolutionary process pop = ec.Population

# Load class for Initializer to create the initial population
# ec.simple.SimpleInitializer is the only known class
init = ec.simple.SimpleInitializer

# Finisher for clean up job at the very end of the run
# ec.simple.SimpleFinisher is the only known class
finish = ec.simple.SimpleFinisher

# Breeder for creating new population based on existing 
# population using mutation and/or crossover
# There are four types of breeder in ECJ including:
# DEBreeder, MuCommaLambdaBreeder, PSOBreeder, SimpleBreeder
breed = ec.simple.SimpleBreeder

# For assigning qualifier to each member of the Population
# There are four types of evaluator in ECJ including:
# CompetitiveEvaluator, MultiPopCoevolutionaryEvaluator,
# SimpleEvaluator
eval = ec.simple.SimpleEvaluator

# For statistics generation
# ChartableStatistics, RandomRestarts, SimpleShortStatistics,
# SimpleStatistics, TarpeianStatistics
stat = ec.simple.SimpleStatistics

# For importing / exporting individuals from/to the population
# InterPopulationExchange, IslandExchange, SimpleExchanger
exch = ec.simple.SimpleExchanger

# Parameters for the EvolutionState
generations = 1000

# whether or not to quit ECJ when it finds an ideal individual. 
# otherwise, it will continue until it runs out of generation.
quit-on-run-complete = false

# checkpointing is a facility to save state of process to a file
checkpoint = false
checkpoint-prefix = ec
# modulo tells by how many generation it will create one 
# checkpoint file
checkpoint-modulo = 1

# Output file
stat.file = $out.stat

# No. of population pop.subpops = 1
# Class of the population subpop.0, which is a repository of 
# individuals
pop.subpop.0 = ec.Subpopulation

# Size of the subpop.0 pop.subpop.0.size = 50
# This tells ECJ not to regenerate individual if it is 
# duplicate. 
pop.subpop.0.duplicate-retries = 0
# Species of the individual 
pop.subpop.0.species = ec.vector.BitVectorSpecies

# Define the fitness model and ec.simple.SimpleFitness defines 
# fitness values from 0.0 inclusive to infinity exclusive, 
# where 0.0 is the worst fitness
# Further, it can define an ideal fitness to any value 
# other than 0.
# There are three types of fitness in 
# ECJ KozaFitness, MultiObjectiveFitness, SimpleFitness 
pop.subpop.0.species.fitness = ec.simple.SimpleFitness

# Define representation
# The type of individual in the species
pop.subpop.0.species.ind = ec.vector.BitVectorIndividual

# Parameters of the species 
pop.subpop.0.species.genome-size = 200
pop.subpop.0.species.crossover-type = one
pop.subpop.0.species.crossover-prob = 0.3
pop.subpop.0.species.mutation-prob = 0.01

# Define pipeline for creating new individuals
# Top node is a vector mutation pipe 
pop.subpop.0.species.pipe = ec.vector.breed.VectorMutationPipeline
# Lower node is a vector crossover pipe
pop.subpop.0.species.pipe.source.0 = ec.vector.breed.VectorCrossoverPipeline
# Define two sources of individual for the crossover using tournament selection
pop.subpop.0.species.pipe.source.0.source.0 = ec.select.TournamentSelection
pop.subpop.0.species.pipe.source.0.source.1 = same

# set tournament size to 2 for all tournament cases
select.tournament.size = 2

# set the fitness function
eval.problem = ec.app.tutorial1.MaxOnes

Modify the fitness equation of Tutorial 1 of ECJ

Okay, this time I switch to ECJ, which is a comprehensive library for evolutionary computation as I have surfed the web again for an easier platform than those C++ libraries.  I've tried the Tutorial 1 that is simple and easy to follow. As you may know, evolutionary process relies on a fitness equation to find optimal value.

In tutorial 1, the optimal solution is defined as the maximum  number of "1s" or "trues" in a chromosome / genome. To do this, it defines the  fitness value using the following codes in the MaxOnes.java file.
int sum=0; for(int x=0; xsum += (ind2.genome[x] ? 1 : 0);

The best genome will be 11111 when the size of genome is 5. Alternatively, I want the best genome be 10101. To do this, I replace the above code with the followings:

int sum=0; 

 for(int x=0; x

 if ((ind2.genome[x]==true) && ((x+1)%2!=0)) sum++;
else if ((ind2.genome[x]==true) && ((x+1)%2!=0)) sum++;

This increases the fitness value by 1 if the gene value is true at odd position or if the gene value is false at even position. The result is an alternating 1s and 0s string. Next, I will try to understand the entire evolutionary process in ECJ and how to define a representation and evolving operators.

Compiling Evolving Objects on Windows 7

Recently, I am studying the use of genetic algorithm for my research studies. After a quick search for C++ API on the Internet, I can identify GALib and Evolving Objects (EO), which seems to be a good start to study how it works.

Setting up the GALib is simple that requires downloading the code and add it to a IDE. And I can run the code without much problem. GALib is also well-documented. However, it does not support Estimation of Distribution Algorithm (EDA) which EO does. For EO, it takes me two days to figure out how to compile and run the build-in examples. Simply because I don't know CMake and EO requires it for building the program, while I cannot find any useful information on the Internet for setting up the EO on Windows platform. That's why I write this blog to record the procedure to setting up EO on Windows.

I am using Windows 7 Home and NetBeans 7.3.1 (on Java SE 1.6). To start, I have installed the following compiler and Make Tool.

• Cygwin 4.7.3 (includes gcc, g++ and GNU assembler, GNU Make and GNU Debugger)
• CMake Tool 2.8.4

The following is step-by-step procedure and I also recommend to read the FAQ in the EO website before start.

1. Use the GIT tools in NetBeans to clone the code using the following link.

git://eodev.git.sourceforge.net/gitroot/eodev/eodev

2. Select EO1.3.1 only from the list of remote branches. Click Next and enter d:\ as parent directory and specify "eodev" as clone name. Then, click Finish and wait until the source code populated in the d:\eodev directory.

3. After completion of the cloning, Netbeans will ask for creating a new project. Then, choose create C/C++ project using existing sources, specify folder containing the sources, tool collection and configuration model you prefer. Choose custom mode and click Next.

4. Then you will see the following build tools configuration.The CMakeLists.txt is important that tells how to generate the Makefile for later compiling. Just leave all the items unchanged and click Next. 

5. Specify the working directory and build/clean command, then click Next.

6. Specify the source directory and click Next

7. Specify automatic configuration for the code assistance settings

8. Finally, specify the project name and click Finish.

9. A new project is added to the projects explorer and Netbeans will try to build the software. However, it will return the following error messages telling it cannot identify the C and C++ compilers. As you may see all the backslashes are missing in the location name.

10. To build the code, it requires two files:CMakeLists.txt and Makefiles. CMakeLists.txt is located under the EO directory, i.e. d:\eodev\eo. Try to modify the following lines in the file and save it.

• Uncomment the following line

#cmake_minimum_required(VERSION 2.8)

• Add the following lines under ADD_SUBDIRECTORY(tutorial) so that all the lesson codes are included during the build:

ADD_SUBDIRECTORY(tutorial/lesson1)

ADD_SUBDIRECTORY(tutorial/lesson2)

ADD_SUBDIRECTORY(tutorial/lesson3)

ADD_SUBDIRECTORY(tutorial/lesson4)

ADD_SUBDIRECTORY(tutorial/lesson5)

ADD_SUBDIRECTORY(tutorial/lession6)

11. Start the CMake GUI, specify d:/eodev/eo as the location of source code and location to build the binaries. Clear the cache by clicking File>Delete Cache.

12. Click "Configure" and a window will be displayed asking for generator, select "Unix Makefiles" and select "Specify Native Compilers". Enter C:/cygwin/bin/g++.exe for C++ and C:/cygwin/bin/gcc.exe for C.

13. Click Advanced to see all the variables. In particular, specify the followings

CMAKE_BUILD_TYPE = debug

CMAKE_CXX_COMPILER = C:/cygwin/bin/g++.exe

CMAKE_CXX_COMPILER_WITH_PATH = C:/cygwin/bin/g++.exe

CMAKE_C_COMPILER = C:/cygwin/bin/gcc.exe

CMAKE_C_COMPILER_WITH_PATH = C:/cygwin/bin/gcc.exe

CMAKE_RC_COMPILER= c:/cygwin/bin/windres.exe
CMAKE_RC_COMPILER_ENV_VAR = c:/cygwin/bin/windres.exe
CMAKE_CXX_FLAGS =
CMAKE_CXX_FLAGS_RELWITHDEBINFO= -O2 -g
CMAKE_CXX_FLAGS_RELEASE=-O2
CMAKE_CXX_FLAGS_DEBUG=-O0 -g

Then click "Configure". If everything okay, "Configuring done" will be displayed. Further click "Generate" to create the makefile.

14. Back to Netbeans, click "Clean and Build" to start the building process.

15. During the building process, the following errors are encountered.

D:/eodev/eo/tutorial/Lesson2/FirstBitEA.cpp:128:40: error: no matching function for call to ‘eoPropCombinedQuadOp >::add(eoNPtsBitXover >&, const double&, bool)’
D:/eodev/eo/tutorial/Lesson2/FirstBitEA.cpp:128:40: note: candidate is:
In file included from D:/eodev/eo/src/eo:63:0,
                 from D:/eodev/eo/tutorial/Lesson2/FirstBitEA.cpp:18:
D:/eodev/eo/src/eoProportionalCombinedOp.h:202:16: note: void eoPropCombinedQuadOp::add(eoQuadOp&, double) [with EOT = eoBit]

D:/eodev/eo/src/eoProportionalCombinedOp.h:202:16: note:   candidate expects 2 arguments, 3 provided

This refers to line 128 of FirstBitEA.cpp that the eoPropCombinedQuadOp.add function accepts two arguments only, but 3 are found. Just remove the third argument. Similar error can be found in FirstRealEA.cpp line 125 and other files.

16. This completes the settings and you can start to explore the example codes.

Configuring PyScripter for Quantum GIS (qgis) on Windows7

Recently, I am studying QGIS if it is easy for customization and whether it can be used as a testing platform for my research studies. In fact, I has been using OpenJump for over 1 year and it works great especially for its capability adding plugin for new functionality. However, after reading some blogs and comments from the Internet, it seems many people are using QGIS for development and many comprehensive and updated tutorials and resources (in fact, tutorials found are more or less the same) can be found when compared with OpenJump. This triggered me to have a in-depth study in QGIS. QGIS is developed by C++. I do think for serious GIS development, C++ is the only solution. Besides C++, QGIS also supports PYTHON for building plugin. As revealed from the Internet, PYTHON is comparatively easier than C++ due to its readable syntax. Frankly, I think JAVA is better. I think the event and action model, i.e. Connect, SIGNAL and SLOT, is not easy to follow.

The reason I write this blog is to record the way I setup PyScripter for QGIS as it has taken me part of my Christmas holiday to figure out how to do it. To do this, you should know the following:
1. QGIS is using QT4 for UI design;
2. QGIS has its own library for Python called PyQGIS;

The goal is to setup PyScripter so that it can provide auto-complete function for programming PyQGIS. Without auto-complete, you can still program it but it would be much easier to figure out typo and syntax errors in the program with auto-complete.

The following is the setup procedure (cover both QGIS and PyScripter):
1. Download and install QGIS and OSGeo4W from here. (Then, you will have a OSGeo4W directory, QGIS and all the required libraries.)
2. Download and install PyScripter from here.
3. Read this to setup batch file for startup PyScripter. The batch file I am using is as follows (Last statement is the key):
@echo off
SET OSGEO4W_ROOT=C:\OSGeo4W
call "%OSGEO4W_ROOT%"\bin\o4w_env.bat
call "%OSGEO4W_ROOT%"\bin\gdal16.bat
@echo offapps\qgis\python\qgis
path %PATH%;"C:\Program Files\Quantum GIS Wroclaw\bin";%OSGEO4W_ROOT%\apps\qgis\python\qgis;
Start "PyScripter" /B "C:\pyscripter\pyscripter.exe" --python25 --pythondllpath=C:\OSGeo4W\bin

4. Edit the python_init.py of the PyScripter and make sure the following statements are there.
from PyQt4 import QtCore, QtGui
from qgis.core import *
from qgis.gui import *

5. In the menu, select Tools>Options>IDE Options. Add PyQt4, qgis in the special package.

6. Add the following paths in the system PATH and a new system variable "PYTHONPATH" in the Windows environment. Without the following PATH, PyScripter cannot find the qgis libraries, e.g. qgis_core.dll
System Path: %PATH%;C:\OSGeo4W\apps\qgis\bin;C:\OSGeo4W\bin;
PYTHONPATH: C:\OSGeo4W\apps\qgis\python;C:\OSGeo4W\apps\Python25;

After all, the PyScripter should start without any error. However, it seems it still cannot retrieve some functions e.g. QMessageBox.

A quote from Napoleon Bonaparte

Never ascribe to conspriracy that which is adequately explained by incompetence. Incompetence is an easy scapegoat.

唔好當一樣可以用蠢解釋嘅嘢做陰謀. 蠢只是替罪的羔羊.

重有:

A practical observation on the risks of stupidity (蠢) was made by the German General Kurt von Hammerstein-Equord in Truppenführung, 1933: "I divide my officers into four classes; the clever, the lazy, the industrious, and the stupid. Each officer possesses at least two of these qualities. Those who are clever and industrious are fitted for the highest staff appointments. Use can be made of those who are stupid and lazy. The man who is clever and lazy however is for the very highest command; he has the temperament and nerves to deal with all situations. But whoever is stupid and industrious is a menace and must be removed immediately!"

德國將軍 Kurt von Hammerstein-Equord 響 Truppenführung（1933年）留意到蠢嘅實際風險。渠話：「我分我啲軍官做四類；精、懶、勤、蠢。 呢啲質素每一軍官都有至少兩種。又精又勤嗰啲適合做最高參謀（staff appointments）。又蠢又懶嘅可用得到。 而又精又懶嘅就啱做最高指揮；渠有應付任何情況嘅性格同膽(temperament and nerves)。但又蠢又勤嗰啲，係種威脅，要即刻整走！