Emergent Models: Machine Learning from Cellular Automata

Short intro

In collaboration with my startup BrightStarLabs and Wolfram Institute, I theorized a new kind of Machine Learning model that uses Cellular Automata (or any other dynamical system) as a general mean of modeling, substituting Neural Networks.

Note: This work includes preliminary analyses and unverified hypotheses that are still under active investigation. Readers are advised to interpret these findings accordingly.

We believe that this could be a possible road to AGI.
Instead of relying on the universal approximation theorem (of neural networks) we rely on the Turing Completeness of some cellular automatas as a way to model any possible algorithm.
We show theoretically that such models have “Universal Meta-Learning capabilities” and can develop inductive biases, because they can self-modify their state, that encode the algorithm they model (it's like if a neural network could learn to modify its own weights, without needing an external gradient descent optimizer).

More info will be coming soon, we'll apply for funding and release a full paper in the next months.

We made an early version of our paper, still working on it.
We're searching for people interested in revising it and helping us see how “readable” it is.

Full PDF paper: https://github.com/BoccheseGiacomo/Emergent-Models

Feel free to comment the following work and the PDF.

Abstract

This proposal aims to introduce the concept of Emergent Models (EMs): dynamical systems inspired by Cellular Automata and Turing Machines that can be used as Machine Learning (ML) models instead of neural networks. By encoding complexity in a large state space and recursively applying a fixed update rule until a halting condition is met, these models aim to emulate modeling and learning behaviors in a fundamentally different way from neural networks and other traditional machine learning models. Current ML models, especially neural networks, usually tend to capture surface patterns rather than achieve true generalization. This shortfall may arise from their reliance on a one-shot, complex, and highly parametric transition function. In contrast, biological systems and physical processes evolve by applying many times a simple transition function on a large state space, much like a Turing machine. EMs, which are inspired by Turing Completeness and use simple local rules applied iteratively over a large state space, offer a highly expressive yet efficient alternative to neural networks. This approach likely reduces overfitting and enhances generalization, making EMs a promising pathway toward artificial general intelligence (AGI). Furthermore, due to the inherent flexibility of their architecture, they could present advanced meta-learning capabilities. To leverage all this flexibility, EMs are trained with black box algorithms, like with genetic and Bayesian optimization. We propose to implement and compare EMs on a toy reinforcement learning (RL) task, driving a dummy car in a 2D environment, to evaluate expressivity, stability, learning speed, spontaneous emergence of meta-learning abilities, and searching for the best configurations to allow the model to work and learn efficiently. All findings will then be formalized into a research paper, and we also plan to release a GitHub repository to reproduce our work.

Introduction

Traditional ML models, such as Neural Networks, encode learned complexity within their trainable parameters, forming complex transition functions. Furthermore, multilayer perceptron (MLP) neural networks generally operate in a one-shot manner: they predict an output by applying a very complex closed-form parametric function in a single time for each input. 

Recurrent neural networks and Transformers for autoregressive tasks operate in a slightly different way since they allow loops where information can be processed taking into account past states. However, even if they are somewhat stateful, most complexity is not derived from the state space (which is low dimensional, e.g. hidden state in neural networks or context in transformers) but from the transition function. 

Moreover, even autoregressive ML systems operate by applying the transition function only once per input, which increases the complexity of correctly modeling the relation. Additionally, since the model complexity for each prediction step is fixed, it poses limitations in reasoning tasks, where computational depth should be adaptable. 

In contrast, EMs employ a fundamentally different strategy for processing information and making predictions. Rather than applying a fixed and highly complex transition function in a single step, EMs iteratively build their esti mates by updating internal representations over multiple computational cycles. This iterative process allows EMs to dynamically allocate computation time depending on the complexity of the input and the task at hand, to reduce the parameter count needed, and to improve generalization.

Summarized formalism

Here is included a short mathematical summary of how EMs work from a formal perspective. Full text on PDF.