Data issue

[yujg1992]

where to load my personal data?

[AwardOfSky]

Hello @yujg1992.
Could you be more specific with which kind of data you want to load?

If you have a custom objective function that you want to define as the target of your evolutionary process, you can:

1 - Define the function as an expression:
Say that your target is the function: f(x, y) = 5x + 6y,
you simply define the expression as the string and pass it to the engine:

exp = "add(mult(scalar(5.0), x), mult(scalar(6.0), y))"
engine = Engine(target = exp,
		... # other parameters,
		)
engine.run()

2 - same thing if you have a tensor/array, except you should convert it first to a tensorflow tensor first

optimal_solution = tf.convert_to_tensor(F) # where F is an array of arbitrary dimensionality (define with numpy, raw python, ...)
engine = Engine(target = optimal_solution ,
		... # other parameters,
		)
engine.run()

If you have a custom function to the function set or a custom terminal to the set you can also do so (please refer to the readme or to issue #1 if you have more troubles).

If you want to specify a parameter of the evolutionary run, then you can do so by defining them when the engine object is called (refer to the "GP parameters" section of the readme for more information).
If you want to load the parameters to the engine as a configuration file, that feature is still not implemented, although we plan on doing that in the near future.

Hope this helps you.

[yujg1992]

hello @AwardOfSky , thank you for your reply. Tha data I use is an array, with N features and 1 target. I would like to generate some new features using your program, just like Symbolic Transformer in gplearn. Where should I put my feature into your code? Or should I just put all the data into target like you said in example 2?

[AwardOfSky]

Hello @yujg1992. I confess that I am not entirely familiarized with the Symbolic Transformer of gplearn. Aside from the target which you can provide using example 2 of my comment above, you can provide a specific population of individuals by providing a file: engine = Engine(read_init_pop_from_file = 'pop.txt' , ...) (search for the "Custom Initial population" section of the readme for more information).

[NQevxvEtg]

Hi @AwardOfSky,

Thanks for creating this awesome library.
Having a similar issue with running custom data.
For example I have the following dataframe, is it possible to solve for target by using all columns except for target?

import pandas as pd


data = {'target': [], 'col2': [], 'col3': [], 'col4': [], 'col5': []}


data['target'] = [1, 2, 3, 4, 5]
data['col2'] = [6, 7, 8, 9, 10]
data['col3'] = [11, 12, 13, 14, 15]
data['col4'] = [16, 17, 18, 19, 20]
data['col5'] = [21, 22, 23, 24, 25]


df = pd.DataFrame(data)
print(df)

# example of solution for target 
# "add(mult(col2, col3), sin(mult(sub(col2, scalar(col3)), sub(col3, scalar(col5)))))"

[AwardOfSky]

Hi @NQevxvEtg ,

Fisrt and foremost, sorry for the delay in my response.
If I understood correctly, you want to add the columns as custom terminals. This is indeed possible to do, you need to get the terminal set for the initial constructor and then add/remove the variables you want, like so:

# initialize engine 
engine = Engine(...)

# Get terminal set from initial constructor
ts = engine.get_terminal_set()

# add and remove
ts.add_to_set("col2", np.asarray(data['col2']), engref=engine)
ts.add_to_set("col3", np.asarray(data['col3']), engref=engine)
ts.add_to_set("col4", np.asarray(data['col4']), engref=engine)
ts.add_to_set("col5", np.asarray(data['col5']), engref=engine)

# you can remove index variables x if you wish
ts.remove_from_set("x")

Below I provide a working example with the indexing variable ("x" which is added automatically) removed from the terminal set (this way the only terminals are the cols).

PS, IMPORTANT: Please be sure to use the new "engine.py" from the tensorgp folder as I needed to bring some things upto date!

Example code:

import numpy as np
import pandas as pd
from tensorgp.engine_git import *


data = {'target': [], 'col2': [], 'col3': [], 'col4': [], 'col5': []}

# Standard RMSE fitness function
def calc_fit(**kwargs):
    # read parameters
    population = kwargs.get('population')
    generation = kwargs.get('generation')
    tensors = kwargs.get('tensors')
    f_path = kwargs.get('f_path')
    _stf = kwargs.get('stf')
    target = kwargs.get('target')

    fn = f_path + "gen_" + str(generation).zfill(5)
    fitness = []
    times = []
    best_ind = 0

    # set objective function according to min/max
    fit = 0
    condition = lambda: (fit < max_fit)  # minimizing
    max_fit = float('inf')

    for i in range(len(tensors)):

        start_ind = time.time()
        print(target)
        fit = tf_rmse(tensors[i], target).numpy()
        if condition():
            max_fit = fit
            best_ind = i

        times.append((time.time() - start_ind) * 1000.0)
        population[i]['fitness'] = fit

    return population, best_ind

# Custom data
data['target'] = [1, 2, 3, 4, 5]
data['col2'] = [6, 7, 8, 9, 10]
data['col3'] = [11, 12, 13, 14, 15]
data['col4'] = [16, 17, 18, 19, 20]
data['col5'] = [21, 22, 23, 24, 25]
df = pd.DataFrame(data)
print(df)

# engine initialization
engine = Engine(target_dims=[5, 1],
                seed=2023,
                do_final_transform = True,
                show_graphics=False,
                final_transform = [0, 2],
                operators={'add', 'mult', 'sub', 'div', 'cos', 'sin', 'tan', 'abs', 'sign', 'pow'},
                fitness_func=calc_fit,
                target=np.asarray(data['target']),
                stop_value=2,
                population_size=5,
                effective_dims=1)

# Get terminal set from initial constructor
ts = engine.get_terminal_set()

# add and remove
ts.add_to_set("col2", np.asarray(data['col2']), engref=engine)
ts.add_to_set("col3", np.asarray(data['col3']), engref=engine)
ts.add_to_set("col4", np.asarray(data['col4']), engref=engine)
ts.add_to_set("col5", np.asarray(data['col5']), engref=engine)
print(ts.set['x'])
# you can remove index variables x if you wish
ts.remove_from_set("x")

print(ts.set['col2'])
print(ts.set['col3'])
print(ts.set['col4'])
print(data['target'])

# run engine
engine.run()


# example of solution for target
# "add(mult(col2, col3), sin(mult(sub(col2, scalar(col3)), sub(col3, scalar(col5)))))"

[NQevxvEtg]

Hi @AwardOfSky,

Thank you so much for such a thorough explanation. The code works great!

[mangomqy]

hello @AwardOfSky , thank you for your reply. Tha data I use is an array, with N features and 1 target. I would like to generate some new features using your program, just like Symbolic Transformer in gplearn. Where should I put my feature into your code? Or should I just put all the data into target like you said in example 2?

Hello @AwardOfSky , I also confused with how to use personal data. For example, I need to get a typical symbolic regression y = X0^2 - X1^2 + X1 -1. I generate train_data and test data. How do I train this train_data and anticipate the test_data

`
X_train = rng.uniform(-1, 1, 100).reshape(50, 2)
y_train = X_train[:, 0]**2 - X_train[:, 1]**2 + X_train[:, 1] - 1

X_test = rng.uniform(-1, 1, 100).reshape(50, 2)
y_test = X_test[:, 0]**2 - X_test[:, 1]**2 + X_test[:, 1] - 1`