Emergent Models: a general modeling framework as an alternative to Neural Networks

Authors & Affiliations

Giacomo Bocchese [1,2]
Nicola Giacobbo [2]

[1] Wolfram Institute for Computational Foundations of Science
[2] BrightStar Labs

1. Overview

This project aims to verify and test the concept of Emergent Models (EMs), which employ Cellular Automata and analogous dynamical systems as a universal modeling substrate in place of Neural Networks.

It extends the first version of our paper, in which we formulated EMs and their theoretical capabilities: https://doi.org/10.55277/ResearchHub.70e8enig.

The compelling need for a new class of Machine Learning (ML) models that solve key issues of Neural Networks (poor generalization, need for differentiable loss, high number of trainable parameters, unability to self-modify their parameters), drove us to develop EMs, which align with our vision of exploiting complex systems physics to build the next generation of ML models.

By leveraging the Turing completeness [1] of carefully chosen Cellular Automata [2,3] (rather than the universal‐approximation theorem of Neural Networks), EMs are theoretically capable of emulating any algorithm and use this ability to build world models.

Making use of Computational Universality (Turing Completeness), we hypothesize that these models have a much higher tendency to generalize and model the deep rules behind training data, if compared to NNs.

Theoretical analyses show that EMs have “Universal Meta-Learning capabilities” and can develop inductive biases, because they can self-modify their state, which encodes the algorithm they model (it's as if a neural network could modify its own weights, without needing an external gradient descent optimizer).

The intention of this pre-registration is to finance:

A rigorous formal verification of our deductions
Empirical testing of Emergent Models, where generalization and meta-learning abilities are accurately checked
Publishing a complete paper with our findings
Releasing a minimal GitHub repository to allow anyone to train and infer emergent models for reinforcement learning tasks

The project described in this proposal follows Open Research principles and has an estimated duration of 6 months.

2. Introduction

Emergent models are dynamical systems inspired by Cellular Automata [3,4] and Turing Machines [5], and can be used as general modeling means instead of Neural Networks [6].

By encoding information on a large state space and recursively applying a fixed update rule until a halting condition is met, these systems exhibit modeling and learning behaviors in a fundamentally different way from neural networks and other machine learning models.

Traditional ML models, such as Neural Networks, encode learned behaviours within their trainable parameters, forming complex transition functions. Furthermore, multilayer perceptron (MLP) [6] neural networks generally operate in a one-shot manner: they predict an output by applying a very complex closed-form parametric function a single time for each input.
In contrast, biological systems and physical processes evolve by applying many times a simple transition function on a large state space, much like a Turing machine.

We hypothesize that reliance on a one-shot, complex and highly parametric transition function is what prevents Neural Networks from achieving data-efficient generalization, and we propose a more natural, physics-inspired modeling approach that employs simple local transition functions applied recursively over a large state space.

This approach likely reduces overfitting and enhances generalization, making EMs a promising pathway toward artificial general intelligence (AGI).

Furthermore, due to the inherent flexibility of their architecture, we hypothesize they possess advanced meta-learning [7] capabilities and the tendency to develop strong inductive biases [8] that drastically improve data efficiency in training and adaptation to new scenarios.

To leverage all this flexibility, EMs are trained with black box algorithms, like genetic algorithm [9] and Bayesian optimization [10] , until the emergence of self-learning behaviours that should drastically speed up the training process from the inside.

Figure: Hypothesized learning curves for Emergent Models vs Neural Networks

Full PDF with rigorous description of EMs: https://github.com/BoccheseGiacomo/Emergent-Models

Hypotheses

H1: Emergent Models present a higher tendency to generalization on out-of-distribution data than Neural Networks

H2: Emergent Models are more parameter-efficient in modeling complex behaviours (need fewer parameters to achieve the same representation complexity)

H3: During black-box training, Emergent Models spontaneously develop Meta Learning strategies and Inductive Biases

H4: Emergent Models are more suitable than Neural Networks for Reinforcement Learning tasks, due to better generalization, higher robustness and adaptation tendency

3. Methods

Our research methodology focuses on:

formalization: constructing precise mathematical definitions and intuitive explanations to ground Emergent Models
hypothesizing: proving existence conditions that guarantee any target emergent property lies within the model’s expressive power
testing: empirically validating those emergent behaviors by training on benchmark tasks and measuring the computational complexity required to reach outcomes
reporting: packaging our deductions and experimental results into clear, reproducible documentation and open-source code so that others can understand, reproduce, and extend our work

Current state of the project

Currently, the Emergent Models project has already achieved several key milestones. First, we have completed the formalization of the architecture, defining its core components and establishing the existence conditions that guarantee the mentioned emergent properties. This has resulted in the publication of a preprint paper, which is subject to ongoing improvement.

We have also implemented two practical frameworks: the Convolutional Turing Machine (CTM) [11], which is inspired by Lenia Cellular Automata [12], and designed for control and reinforcement‐learning tasks; and the Game of Life Language Model (GOL-LM) [13], which leverages the Turing-completeness of Conway’s Game of Life [14,2] for language modeling and logical inference. Currently, only the inference pipeline is operational, while the training routines are yet to be implemented.

Next steps

Several critical tasks remain to fully validate the effectiveness of EMs. We will begin with an external verification of our assumptions: an independent review from complex systems mathematicians to ensure formal consistency and eliminate biases. At the same time, we will undertake the training of emergent models using black-box optimization algorithms to assess their learning capabilities and emergent properties. We will also conduct a study of computational complexity and training efficiency, measuring convergence times. Finally, we will devote special attention to the stability of the model’s behaviour.

To minimize computational cost and maintain a lean compute profile, we’ll focus on a small set of clear, representative tasks with limited combinatorial complexity and brief training runs. This ensures our experiments stay tractable and cost-effective.

The following milestones outline the future research roadmap included in this pre-registration:

Verification: Verify mathematical definitions with the support of external experts, ensuring EMs are well-defined at each point. Develop more intuitive explanations to facilitate the understanding.
Training and testing in RL Tasks: Implement EMs in a simple reinforcement learning scenario, such as navigating a dummy car in a 2D environment with LIDAR input. Train the model with black-box optimization algorithms on such tasks.
Training and testing in logic/arithmetic tasks: copy, binary addition, binary multiplication, etc.
Studying emergent properties and training efficiency. Evaluation criteria will include:
- Convergence speed;
- Stability;
- Emergence of meta-learning and higher-order behaviors;
- Comparative analysis against neural networks regarding performance, learning speed, efficiency, generalization capability, stability, robustness, and emergent behaviors;
Publishing: Publish a detailed paper on Emergent Models with rigorous definitions and experimental findings. Additionally, release an open-source repository to facilitate building and training EMs.

2-D Dummy Car Environment RL

This environment represents a simplified 2-dimensional racing scenario for reinforcement learning. A car moves along a predefined track and can perform actions such as steering, accelerating, and braking. The primary objective is to complete the track in the shortest possible time without exiting the track boundaries or colliding with obstacles.

The EM serves as the policy model that decides the car’s actions based on sensory inputs (black visual rays). The interaction follows standard reinforcement learning principles.

Observation Variables (Model Inputs)

Speed: Current speed of the car.
LIDAR Sensors: Distances from the car to nearby obstacles (visual rays shown in black).

Action Variables (Model Outputs)

Acceleration: Controls the car’s speed (accelerate/brake).
Steering: Direction control (left/right).

Reward Structure

The reward encourages efficient and safe driving behavior by accounting for:

Maintaining the car on track (penalties for going off-track).
Total distance traveled along the intended track direction.
Average speed maintained along the correct path.
Computational efficiency (penalty proportional to the computation time before halting, to encourage faster reasoning).

Logic/Arithmetic tasks

Generally, novel ML models are tested on their ability to solve basic logic and arithmetic tasks, to evaluate their practical expressivity and efficiency.
These tasks include:

Modeling logic gates (and, or, xor, etc)
Binary copy task (transfer a binary sequence from input to output)
Binary addition
Binary multiplication

Open Research License

This proposal’s theoretical studies and experimental tests adhere to Open Research principles. All materials produced in this work will be open source and freely available; the only requirement for their use is proper citation of our work.

4. Budget and duration

We plan to conduct this research over six months.
Because this work is primarily knowledge-intensive rather than compute-intensive, most of the budget covers researcher time and external mathematical review. We estimate a total budget of $10,000, allocated as follows:

Compute: $500 (Covers HPC server time, for simulations that cannot be run on a local machine)
External review/consultancy: $1,500 (Engages mathematicians and complex-systems experts to validate our proofs and provide critical feedback on theoretical claims)
Researchers’ compensation: $7,000 (Reflects the six-month effort of our core team, validating the mathematical framework, implementing and testing EMs, drafting the final report and codebase)
Miscellaneous: $1,000

5. References

[1] Wikipedia contributors. Turing completeness — Wikipedia, The Free Encyclopedia.https://en.wikipedia.org/w/index.php?title=Turing_completeness&oldid=1279841424 [Online; accessed 5-April-2025]. 2025.

[2] Paul Rendell. Turing Machine Universality of the Game of Life. 1st ed. Emergence, Complexity and Computation. Springer Cham, 2015, pp. XV, 177. ISBN: 978-3-319-19841-5. DOI: https://doi.org/https://doi.org/10.1007/978-3-319-19842-2.

[3] Eric W. Weisstein. Universal Cellular Automaton. MathWorld–A Wolfram Web Resource. URL: https://mathworld.wolfram.com/UniversalCellularAutomaton.html (visited on 04/05/2025).

[4] Stephen Wolfram. Cellular Automata and Complexity: Collected Papers. https://www.stephenwolfram.com/publications/cellular-automata-complexity/. Accessed on: [accessed 5-April-2025].

[5] John E. Hopcroft, Rajeev Motwani, and Jeffrey D. Ullman. Introduction to Automata Theory, Languages, and Computation. Third. Boston: Pearson/Addison Wesley, 2007. ISBN: 0321455363 9780321455369 0321462254 97803214622510321455371978032145537603214761749780321476173. URL:http://infolab.stanford.edu/~ullman/ialc.html

[6] Christopher M. Bishop. Neural Networks for Pattern Recognition. Oxford University Press, 1995.

[7] Huimin Peng. “A comprehensive overview and survey of recent advances in meta-learning”. In: arXiv preprint arXiv:2004.11149 (2020).

[8] Tom M Mitchell. “The need for biases in learning generalizations”.

[9] Tanweer Alametal. “Genetic Algorithm: Reviews, Implementations, and Applications”. In: International Journal of Engineering Pedagogy (iJEP) (2020). arXiv: 2007.12673 [cs.NE]. URL: https://arxiv.org/pdf/2007.12673 (visited on 04/05/2025).

[10] Jasper Snoek et al. “Scalable Bayesian optimization using deep neural networks”. In: International conference on machine learning. PMLR. 2015, pp. 2171–2180.

[11] G. Bocchese, “Convolutional Turing Machine: studying Meta-Learning Emergence from Cellular Automata,” GitHub repository, 2023. [Online]. Available: https://github.com/BoccheseGiacomo/ConvolutionalTuringMachine

[12] Bert Wang-Chak Chan. “Lenia-biology of artificial life”. In: arXiv preprint arXiv:1812.05433 (2018).

[13] G. Bocchese, “Gol-LM: A Language Model based on Game Of Life Cellular Automata with potential meta learning capabilities,” GitHub repository, 2024. [Online]. Available: https://github.com/BoccheseGiacomo/gol_lm

[14] Martin Gardner. “Mathematical Games: The Fantastic Combinations of John Conway’s New Solitaire Game “Life”. In: Scientific American 223 (Oct. 1970), pp. 120–123. DOI: 10.1038/scientificamerican1070 120. URL: http://dx.doi.org/10.1038/scientificamerican1070-120 (visited on 04/05/2025).

Emergent Models: a general modeling framework as an alternative to Neural Networks

Emergent Models: a general modeling framework as an alternative to Neural Networks

Topics

DOI