Genetic programming is a powerful technique for automatic program synthesis.  Essentially, genetic programming is a parallel search technique
modelled after natural evolution that automatically synthesizes computer programs to solve a given problem.   Genetic programming has proven
quite effective in recent years, producing solutions to a variety of different problems in different fields, including machine learning, mechanical engineering, analog circuit design, molecular biology, pattern recognition, theorem proving, and cellular automata.

Genetic programming is a technique that seems to be able to solve harder problems given more computational resources.  Thus, parallelization seems like a natural step.  Luckily, genetic programming is quite parallelizable, almost embarassingly so.  This web page describes the successful parallel implementation of genetic programming on a network of processing nodes that use the transputer architecture.  With this approach, genetic programming can be applied to very difficult real world problems. This approach is illustrated by a comparison of the computational effort required to solve a benchmark problem.  Because of the decoupled character of genetic programming, our approach achieved a nearly linear speed up from parallelization.  In addition, for the best choice of parameters tested, the use of subpopulations delivered a super linear speed-up in terms of the ability of the algorithm to solve the problem.  In other words, not only does parallelization speed up the algorithm, but it makes the algorithm work better.  Several examples are also presented where the parallel genetic programming system evolved solutions that are competitive with human performance.

Note:  Several sections and figures of this web page are from (Andre and Koza, 1996).  The new work is in the overview, the key idea, the interesting research issues,
and the discussion.
 

GP is a technique that, in many ways, is an extension of the kind of search performed by game playing programs such as Deep Blue.  However, instead of performing only brute force search, GP searches more "intelligently", using a population of programs to search a problem space in parallel. Operations such as crossover combine parts from different good programs, hopefully creating better programs.  Over the past few years, it has seemed that the main factor limiting GP is the size of the population.  Given large enough populations, even very difficult problems can be attempted.  By using parallelization, it is not unreasonable to think that GP can today be used for very difficult real world problems that typically require large expenditures of human resources, such as analog circuit design, data mining, scientific discovery in molecular biology, and many other types of problems.

Genetic Programming

Genetic programming (Koza, 1992) is a variant of the genetic algorithm in which the genetic population consists of computer programs rather than of fixed length bitstrings. The initial population of programs consists of randomly generated programs represented as parse trees that are composed of the available simple programmatic building blocks.  Genetic programming then breeds these programs using the Darwinian principle of survival of the fittest and the crossover operation, which is similar to sexual recombination in nature.  Tackett (1994) provides analysis that genetic programming can be viewed as a method of stochastic beam search utilizing the crossover operation. 

Genetic programming is an extension of the genetic algorithm in which the population consists of computer programs (compositions of simple programmatic ingredients), such as those shown in Fig. 1. As described in John Koza's seminal work (1992), genetic programming is a method to breed populations of computer programs.   Genetic programming consists of the following steps:

(1) Generate an initial population of random compositions of the functions and terminals of the problem (i.e., computer programs).
(2) Iteratively perform the following substeps until the termination criterion has been satisfied:
(A) Execute each program in the population and assign it a fitness value using the fitness measure.
(B) Create a new population of computer programs by applying the following operations.  The operations are applied to computer program(s) chosen from the population with a probability based on fitness.
(i) Darwinian Reproduction: Reproduce an existing program by copying it into the new population.
(ii) Crossover: Create two new computer programs from two existing programs by genetically recombining randomly chosen parts of two existing programs using the crossover operation (described below) applied at a randomly chosen crossover point within each program.
(iii) Mutation: Create one new computer program from one existing program by mutating a randomly chosen part of the program.
(3) The program that is identified by the method of result designation is designated as the result for the run (e.g., the best-so-far individual).  This result may be a solution (or an approximate solution) to the problem.

An example of the crossover operation is shown in Fig. 1.  The computer programs in Fig. 1a,b are the parents for the crossover operation, and were chosen probabilistically based on their fitness on the particular Boolean function regression problem being solved.  The crossover operation exchanges a subtree from one parent with a subtree from the other parent to create the two offspring.  The heavy lines in Fig. 1 indicate the subtrees that were exchanged to create the two children shown in Fig. 1c,d.

 The structure of the programs expressed in Fig. 1 is simple, but by no means indicates that the structures evolved by genetic programming must be simple.  Genetic programming can evolve programs utilizing iteration and subroutine calls, as discussed in Koza (1994).  Automatically Defined Functions (ADFs) are subroutines that are co-evolved with the main program, and have been found to increase the power of genetic programming (Koza, 1994).  For more information on genetic programming, see Koza (1992, 1994), Kinnear (1994), Angeline and Kinnear, 1996, the proceedings of the annual Genetic Programming conference, John Koza's web site, and the Genetic Programming Notebook.

Why the Transputer

Time and memory demands of genetic algorithms

The genetic operations (such as reproduction, crossover, and mutation) employed in the genetic algorithm and other methods of evolutionary computation can be rapidly performed and do not consume large amounts of computer time.  Thus, for most practical problems, the dominant factor with respect to computer time is the evaluation of the fitness of each individual in the population.  Fortunately, these time-consuming fitness evaluations are totally decoupled from one another and can be performed independently for each individual in the population.

The dominant factor with respect to memory is the storage of the population of individuals (since the population typically involves large numbers of individuals for non-trivial problems).  Storage of the population does not usually constitute a major burden on computer memory for the genetic algorithm operating on fixed-length character strings; however, memory usage is an important consideration when the individuals in the population are large program trees of varying sizes and shapes (as is the case in genetic programming).

The island model of parallelization

In a coarse-grained or medium-grained parallel computer, subpopulations are situated at the separate processing nodes.  This approach is called the distributed genetic algorithm  or the island model for parallelization.  The subpopulations are often called demes .
When a run of the distributed genetic algorithm begins, each processing node locally creates its own initial random subpopulation.  Each processing node then measures the fitness of all individuals in its local subpopulation.  Individuals from the local subpopulation are then probabilistically selected based on fitness to participate in genetic operations (such as reproduction, crossover, and perhaps mutation).  The offspring resulting from the genetic operations are then measured for fitness and this iterative process continues.  Because the time-consuming fitness measurement is performed independently for each individual and independently at each processing node, this approach delivers an overall increase in performance that is nearly linear with the number of independent processing nodes.

Upon completion of a generation (involving a designated number of genetic operations), a certain number of the individuals in each subpopulation are probabilistically selected for emigration to each adjacent node within the toroidal topology of processing nodes.  When the immigrants arrive at their destination, a like number of individuals are selected for deletion from the destination processor.  The amount of migration may be small (thereby potentially creating distinct subspecies within the overall population) or large (thereby approximating, for all practical purposes, a panmictic population, which is a population where the individual to be selected to participate in a genetic operation can be from anywhere in the population).  The inter-processor communication requirements of these migrations are low because only a small number of individuals migrate from one subpopulation to another and because the migration occurs infrequently -- only after completion of the entire generation.  The distributed genetic algorithm is well suited to loosely coupled, low bandwidth, parallel computation.

Design considerations for parallel genetic programming

The largest of the programs evolved in Genetic Programming [Koza 1992] and Genetic Programming II [Koza 1994] contained only a few hundred points (i.e., the number of functions and terminals actually appearing in the work-performing bodies of the various branches of the overall program). The parallel implementation of genetic programming described herein was specifically designed for problems whose solutions require evolved programs that are considerably larger than those described in the two books and whose fitness evaluations are considerably more time-consuming.  For design purposes, we hypothesized multi-branch programs that would contain up to 2,500 points, where each point can be stored in one byte of memory.  In this representation, each point represents either a terminal or a function (with a known arity).  Thus, a population of 10,000 2,500-point programs can be accommodated in 25 megabytes of storage with this one-byte representation.  We further hypothesized, for purposes of design, a measurement of fitness requiring one tenth of a second of computer time for each 2,500-point individual in the population.  With this assumption, the evaluation of 10,000 such individuals would require about 15 minutes and 50 such generations could be run in about a half day, and 100 such generations would occupy about a day. Our design criteria were thus centered on a parallel system with a one-second fitness evaluation per program, 10,000 2,500-point programs per processing node, and one-day runs consisting of 100 generations.  Our selection criteria was the overall price-to-performance ratio.
"Performance" was measured (or estimated) for the system executing our application involving genetic programming, not generic benchmarks (although the results were usually similar).  "Price" includes not only the obvious costs of the computing machinery and vendor-supplied software, but also includes our estimate of the likely cost, in both time and money, required for the initial implementation of the approach, the ongoing management and maintenance of the system, and the programming effort needed to implement the expected succession of new problems that we would approach with genetic programming.

Qualitative evaluation of several parallel systems

Using the above design and selection criteria and our price-to-performance ratio as a guide, we examined serial supercomputers, parallel supercomputers, networks of single-board computers, workstation clusters, special-purpose microprocessors, and transputers.

Since we had previously been using a serial computer, an extremely fast, well priced serial supercomputer would optimally satisfy our criteria as to the cost of initial implementation, ongoing management and maintenance, and programming of new problems.  Current extremely fast serial supercomputers (e.g., Cray machines) attain their very high speeds by vectorization or pipelining.  We believed (and several investigators have verified) that these machines would not prove to deliver particularly good performance on genetic programming applications because of the disorderly sequence of conditionals and other operations in the program trees of the individuals in the population.  The Intel i860 microprocessor was also avoided for the same reason.   Although a serial solution would have been ideal as far as the simplicity of implementation, we believe that greater gains in computing speed for doing experimental work on large problems in the field of genetic programming will come, in the next decade or two, from parallelization.

The fine-grained SIMD ("Single Instruction, Multiple Data") supercomputers (such as Thinking Machine's CM-2 and MasPar's machines) are largely inappropriate for genetic programming because the individual programs in the population are generally of different sizes and shapes and contain numerous conditional branches.  Although it may, in fact, be possible to efficiently use a fine-grained SIMD machine for an application that seemingly requires a MIMD machine (Dietz 1992), doing so would require substantial expertise outside our existing capabilities, as well as extensive customizing of the parallel code for each new problem.

The traditional coarse-grained and medium-grained MIMD ("Multiple Instruction, Multiple Data") supercomputers (such as Thinking Machine's CM-5, Intel's Paragon, the NCUBE machines, and the Kendall Research machines) do not appear to be a cost-effective way to do genetic programming because they are designed to deliver a larger ratio of inter-processor communication bandwidth relative to processor speed than is optimal for genetic programming.  The price of each processing node of a typical supercomputer was in the low to middle five-figure range while the microprocessor at each node was usually equivalent to that of a single ordinary workstation.  In some cases (e.g., Thinking Machine's CM-5), processing could be accelerated by additional SIMD capability at each node, but most genetic programming applications cannot take advantage of this capability.  When one buys any of these coarse-grained and medium-grained MIMD machines, one is primarily buying high bandwidth inter-processor communication, rather than raw computational speed.

Given the high expense and inappropriate features of many of the existing parallel supercomputers and serial supercomputers, we looked into the possibility of simpler parallel systems built out of a network of processor boards or workstations. Printer controller boards are inexpensive, have substantial on-board memory, have on-board Ethernet connectivity, and have very high processing capabilities (often being more powerful than the computer they serve).  However, these devices are typically programmed once (at considerable effort) for an unchanging application and there is insufficient high-level software for writing and debugging new software for the constant stream of new problems to which we apply genetic programming.

We also considered the idea of networks of workstations or networks of single-board computers (each containing an Ethernet adapter); however, at the time of our initial implementation (1993), these approaches presented difficulties involving the ongoing management and maintenance of the independent processes on many autonomous machines.  Even for a network of single-board computers, the tools and operating systems used in these systems were not as sophisticated as those in the system we finally selected.  Now, of course,tools for networks of workstations are considerably more advanced, and the next system built will be of that type.

Transputers

One important reason for considering the transputer (designed by INMOS, a division of SGS-Thomson Microelectronics) was that it was designed to support parallel computation involving multiple inter-communicating processes on each processor and inter-processor communication.  The transputer was engineered so that processes could be easily loaded onto all the processing nodes of a network, so that the processes on each node can be easily started up, so that messages can be easily sent and received between two processes on the same or different processing nodes, so that messages can be easily sent and received between one processing node and a central supervisory program, so that messages can be easily sent and received by a host computer from the central supervisory program, and so that the processes on each node could be easily stopped.  Moreover, debugging tools exist. As it happens, there are a considerable number of successful applications of networks of transputers in the parallel implementation of genetic algorithms operating on fixed length character strings.

The other key fact was that the transputer architecture represented a good compromise for us in terms of communication ease, communication speed, and computatinal power, especially in the PowerPC system described below.
 
 

Technical Details

We have implemented two versions of the parallel GP system.  The first used transputers for both communication and computation.  The second used PowerPC processors at each node for computation, but still used the transputer for communication.  The next subsection describes the system
with reference to our first system, as our basic architecture remained the same.  Following it, the PowerPC system is briefly described.

Our initial system (1993)

A network of 66 TRAMs and one Pentium type computer are arranged in the overall system as follows:  the host computer consisting of a keyboard, a video display monitor, and a large disk memory, a transputer running the debugger, a transputer containing the central supervisory process (the Boss process), and the 64 transputers of the network, each running a Monitor process, a Breeder process, an Exporter process, and an Importer process.

The physical system

The physical system was purchased from Transtech of Ithaca, NY.  A PC computer (running Windows 3.11) is the host and acts as the file server for the overall system.  Two TRAMs are physically housed on a B008 expansion board within the host computer and serve as the central supervisory process for running genetic programming (the Boss Node) and a node to run the INMOS debugger (the Debugger node).  The remaining 64 TRAMs (the processing nodes) are physically housed on 8 boards in a VME box.  The 64 processing nodes are arranged in a toroidal network in which 62 of the 64 processing nodes are physically connected to four neighbors (in the N, E, W, and S directions) in the network.  Two of the 64 nodes of the network are exceptional in that they are physically connected to the Boss Node and only three other processing nodes.  The Boss node is physically linked to only two transputers of the network.

Figure 2.  An overview of the system.    The boxes denote the various computers, including the host Pentium computer, the Debugger Node, the Boss Node, and the network of processing nodes.  For simplicity, only nine of the 64 processing nodes are shown.  The ovals denote the various major files on the host computer, including the input parameter file for each run and two output files (one containing the results of the run and one containing information needed by a monitoring process).   The rounded boxes denote the input-output devices of the host computer.  The diamond denotes the Visual Basic (VB) monitoring process that displays information about the run on the video display monitor of the host computer.  Heavy unbroken lines are used to show the physical linkage between the various elements of the system.  Heavy broken lines show the lines of virtual communication between the Boss node and the processing nodes.  Light unbroken lines show the virtual channels connecting each of the processing nodes with their neighbors.
 
 
The communication between processes on different transputers is by means of one-way, point-to-point, unbuffered, channels.  The channels are laid out along the physical links using a virtual router provided in the INMOS Toolkit.  Figure 2 shows the elements of the system.

Intercommunicating processes

Transputers are programmed using inter-communicating processes connected by channels.  The Host computer runs two processes.  The Host process on the Host computer receives input from the keyboard, reads an input file containing the parameters for controlling the run, writes the two output files, and communicates with the boss.  The second process on the Host computer is the monitoring process.

The TRAM with the Debugger runs only one process, the INMOS-supplied Debugger process.  The TRAM with the Boss runs only one process, the Boss.  Each of the 64 TRAMs in the toroidal network concurrently runs the following four processes:  the Importer, the Exporter, the Breeder, and the Monitor (Figure 3).  The primary process on each processing node is the Breeder process, which runs the main operations of genetic programming.  The other three processes permit asynchronous communication and serve to avoid deadlocks.

Figure 3. The four processes on each of the 64 processing nodes.  The Breeder process performs the bulk of the computation in genetic programming, and the three other processes facilitate communication with the Boss and the four neighboring nodes.

The Boss process

The Boss process is responsible for communicating with the Host process, for sending initial start messages to each of the processors in the network, for tabulating information sent up from each processor at the end of each generation, for sending information to the monitoring process on the host, and for handling error conditions.

At the beginning of a run of genetic programming, the Boss process initializes various data structures, creates the set of fitness cases either functionally or by obtaining information from a file on the Host, creates a different random seed for each processor, pings each processor to be sure it is ready, and reads in a set of parameters that controls genetic programming from a file on the Host.  Then, the Boss sends a Start-Up message to each Monitor process (which will in turn send it along to the Breeder process).  This message contains:  the control parameters for creating the initial random subpopulation, including the size of the subpopulation, the method of generation and the maximum size for the initial random individuals, the common, network-wide table of random constants for the problem (if any), the control parameters specifying the number of each genetic operation to be performed for each generation, and a node-specific seed for the randomizer.

After sending the Start-up message, the Boss enters a loop where it handles the various messages that each Monitor sends until an error condition occurs, a solution is found, or all processors have either crashed or completed a specified number of generations.

The Monitor process

The Monitor process of each processing node is continually awaiting messages from both the Boss process of the Boss node as well as from the Breeder process of its processing node.  Upon receipt of a Start-Up message from the Boss, the Monitor process passes this message along to the Breeder process on its node.  The Monitor process also passes the following messages from the Breeder process of its node along to the Boss:

End-of-Generation:  The end-of-generation message contains the best-of-generation individual for the current subpopulation, statistics about that individual, and some general statistics.
Eureka:  The eureka message announces that the processing node has just created an individual in its subpopulation that satisfies the success criterion of the problem and contains the just-created best-of-run individual and various statistics about it.
Trace:  The trace message announces that the Breeder process has reached certain milestones in its code (e.g., received its start-up message, completed creation of the initial random subpopulation for the node).
Error:  The error message announces that the Breeder process has encountered certain anticipatable error conditions.

The Breeder process

After the Breeder process of a processing node receives the Start-up message, it creates the initial random subpopulation of individuals for the node.  Then, in the main generational loop, the Breeder process of a processing node iteratively performs the following steps:
1. Evaluates the fitness of every individual in the subpopulation.
2. Selects, probabilistically, based on fitness, a small number of individuals to be emigrants (except on generation 0) and sends them to a buffer in the
Exporter process.
3. Assimilates the immigrants currently waiting in the buffers of the Importer process (except on generation 0).
4. Creates an end-of-generation report for the subpopulation.
5. Performs the genetic operations on the subpopulation.
The breeder process runs until one individual is created that satisfies the success predicate of the problem or until it is stopped by the Boss process.

Asynchronous operation

For most problems the amount of computer time required to evaluate individuals in genetic programming usually varies considerably among subpopulations.  The presence of just one or a few time-consuming programs in a particular subpopulation can dramatically affect the amount of computer time required to run one generation.  Any attempt to synchronize the activities of the algorithm at the various processing nodes would require slowing every processing node to the speed of the slowest.  Therefore, each processing node operates asynchronously with respect to all other processing nodes.  After a few generations, the various processing nodes of the system will typically be working on different generations.  This variation arises from numerous factors, including the different sizes of the individual programs, the mix of faster and slower primitive functions within the programs, and the number, nature, and content of the function-defining branches of the overall program.

The asynchrony of the generations on nearby processors requires that the exporting and importing of migrating programs take place in a manner that does not require that the breeder  ever wait for a neighboring process to finish a generation.  To allow the breeder nearly uninterrupted computing time, the Exporter and Importer processes were created to handle the communication.  The Monitor process acts in a similar fashion for communication with the Boss process.  The use of multiple processes is also important in preventing dead-locks.

The Exporter process

The Exporter process periodically interacts with the Breeder process of its processing node as part of the Breeder's main generational loop for each generation (except generation 0).  At that time, the Breeder sends four boatloads of emigrants to a buffer of the Exporter process.  The Exporter process then sends one boatload of emigrants to the Importer process of each of the four neighboring processing nodes of the network.

The Importer process

The purpose of the Importer is to store incoming boatloads of emigrants until the Breeder is ready to incorporate them into the subpopulation.  When a boatload of immigrants arrives via any one of the four channels connecting the Importer process to its four neighboring Exporter processes, the Importer consumes the immigrants from that channel and places these immigrants into the buffer associated with that channel (occasionally overwriting previously arrived, but not yet assimilated, immigrants in that buffer).  When the Breeder process is ready to assimilate immigrants, it calls for the contents of the Importer's buffers.  If all four buffers are full, the four boatloads of immigrants replace the emigrants that were just dispatched by the Breeder process to the Exporter process of the node.  If fewer than four buffers are full, the new immigrants replace as many of the just-dispatched emigrants as possible.

More processing power -- (our 1995 system)

The transputer, although a moderately fast processor in its own time, is not particularly powerful by today's standards.  Several companies have capitalized on the success of the transputer architecture for communication and multiprocessing in a parallel environment by creating hybrid systems that combine a more powerful microprocessor (such as the DEC Alpha or the Motorola PowerPC) with the transputer architecture and the INMOS toolkit.  These companies use the transputer microprocessor solely for communication.

One such company, Parsytec of Aachen, West Germany, has produced several commercial parallel processing systems that integrate the PowerPC chip with the transputer architecture.  The basis of their systems are TRAM-like components that combine a PowerPC chip with a transputer, some memory, and some communication hardware.  Their system utilizes the INMOS toolkit in its entirety along with several special add-on libraries and tools to handle the additional processor.  Their system delivers a increase in performance over the transputer-only system, but requires little change in the architecture, because the INMOS toolkit eliminates most of the vexatious details.

 Although our first implementation of parallel genetic programming had served us well for two years and was still functioning well in spring 1995, we needed greater computational power.  Thus, we moved to a PowerPC based system.  Our second implementation of parallel genetic programming has almost the same architecture as our first implementation.  It consists of a network of 64 PowerPC TRAMs (containing both a PowerPC 601 processor, a transputer T805 processor, and 32 megabyte of RAM), one transputer-only TRAM with 16 MB of RAM as the Boss, and a PC 586/90 type computer acting as host.  Each of the 64 pairs of PowerPC and transputer nodes acts as a processing node in the network.  The debugger node has been eliminated.

A good example of the ease of use of INMOS' parallel toolkit is the fact that the entire port of our parallel implementation to the new Parsytec system took less than a week.  This week included the system setup, learning several tools to analyze the PowerPC nodes in the network, and the porting, compiling, and debugging of all the code in the parallel genetic programming kernel and our benchmark problems.  In less than a week, we were able to run the 5-parity problem on our PowerPC network and compare it to the performance of the transputer system.

On several runs of the 5-parity problem, we found that the Parsytec PowerPC and transputer system outperformed the transputer system by a factor of more than 22.  However, the cost of each node in the PowerPC and transputer system cost only approximately five times more than a node in the first implementation.  Our second implementation thus provides a significantly better price-to-performance ratio over our first system.
 
 

The problem of symbolic regression of the Boolean even-5-parity function will be used to illustrate the operation of the parallel system and to compare the computational effort associated with different migration rates between the processing nodes.

The Boolean even-5-parity function takes five Boolean arguments and returns T if an even number of its arguments are T, but otherwise returns NIL.  The terminal set, Tadf, for this problem is {D0, D1, D2, D3, D4}.  The function set, Fadf, is {AND, OR, NAND, NOR}.  The standardized fitness of a program measures the sum, over the 32 fitness cases consisting of all combinations of the five Boolean inputs, of the Hamming-distance errors between the result produced by the program and the correct Boolean value of the even-5-parity function.  For a full description of this problem, see [6].
Numerous runs of this problem with a total population size of 32,000 were made using ten different approaches.  This particular population size (which is much smaller than the population size that can be supported on the parallel system) was chosen because it was possible for us to run this size of population on a single serial computer (a Pentium) with panmictic selection and thereby compare serial versus parallel processing using a common population size of 32,000.  The first approach tested runs on a single processor with a panmictic population of size M = 32,000; the second approach tested runs on the parallel system with D = 64 demes, a population size of Q = 500 per deme, and a migration rate of B = 12% (in each of four directions on each generation of each deme); the third through tenth approaches tested run on the parallel system with D = 64 demes, a population size of Q = 500 per deme, and a migration rate of B = 8%, 6%, 5%, 4%, 3%, 2%, 1%, and 0%, respectively.

A probabilistic algorithm may or may not produce a successful result on a given run.  One way to measure the performance of a probabilistic algorithm is to compute the computational effort required to solve the problem with a certain specified probability (say, 99%) (see (Koza, Andre, Bennett, and Keane, 1998).  Table 1 shows the computational effort for the ten approaches to solving this problem.
 
 Table 1.

There have been several recent examples of problems (from the fields of cellular automata, molecular biology, and circuit design) in which genetic programming, running on our parallel PowerPC system, evolved a computer program that produced results that were slightly better than, or as good as, human performance for the same problem.

For example, various human-written algorithms have appeared in the past two decades for the majority classification task for one-dimensional two-state cellular automata.  Parallel genetic programming with automatically defined functions has evolved a rule for this task with an accuracy of 82.326% [Andre, Bennett, and Koza, 1996].  This level of accuracy exceeds that of the original human-written Gacs-Kurdyumov-Levin (GKL) rule, all other known subsequent human-written rules, and all other known rules produced by automated approaches for this problem.

A second example involves the transmembrane segment identification problem where the goal is to classify a given protein segment (i.e., a subsequence of amino acid residues from a protein sequence) as being a transmembrane domain or non-transmembrane area of the protein (without using biochemical knowledge concerning hydrophobicity typically used by human-written algorithms for this task).  Four different versions of parallel genetic programming have been applied to this problem.  The performance of each of the four versions evolved using genetic programming is slightly superior to that of algorithms written by knowledgeable human investigators.

A third example illustrates how automated parallel methods may prove to be useful in discovering biologically meaningful information hidden in the rapidly growing databases of DNA sequences and protein sequences.  Parallel genetic programming successfully evolved motifs for detecting the D-E-A-D box family of proteins and for detecting the manganese superoxide dismutase family.  Both motifs were evolved without prespecifying their length, and both detect the two families either as well as, or slightly better than, the comparable human-written motifs found in the PROSITE database created by an international committee of experts on molecular biology.

Finally, our parallel system has been used to evolve both the topology and component values of electrical circuits to solve a variety of engineering problems, including low and high pass filters, multiple band pass filters, woofer and tweeter filters, simple amplifiers, high-gain, low-distortion amplifiers, temperature sensing circuits, and computational circuits (such as a cube-root circuit).  Additionally, parallel genetic programming was able to evolve a solution for an asymmetric band-pass filter problem that was presented as a difficult-to-solve filter problem in an analog circuit journal.

At present, Professor Koza and his colleagues are building a new machine, this one built out of a network of workstations connected with high speed ethernet.  Each workstation will be a 533 Mhz Alpha with 64 Mb of memory.  The plan is to build a network of 1000 workstations.  Calculations indicate that network bandwidth using ethernet will not be a problem for even 10,000 nodes.  The system (already at 6 nodes, and will be at 100 by Feburary, and 1000 by June) represents a leap of several orders of magnitude over the PC system. 

Although it may at first seem that GP is simply an embarassingly parallel application, there are several areas where I think interesting research possibilities still exist.
 

Dave's web site
John Koza's web site
 

PDPTA paper
Parallel Tech Report (1994)