Utilising Genetic algorithm to overcome different size datasets in model

Question

SO I realise the question I am asking here is large and complex. A potential solution to variences in sizes of In all of my searching through statistical forums and posts I haven't come across a scientifically sound method of taking into account the type of data that I am encountering, but I have thought up a (novel?) potential solutions to account perfectly (in my mind) for large and small datasets within the same model. The proposed method involves using a genetic algorithm to alter two numbers defining a relationship between the size of the dataset making up an `implied strike` rate and the percentage of the `implied strike` to be used, with the target of the model to maximise the homology of the number `1` in two columns of the following csv. (ultra simplified but hopefully demonstrates the principle) Example data ``` Date,PupilName,Unique class,Achieved rank,x,y,x/y,Average xy 12/12/2012,PupilName1,UniqueClass1,1,3000,9610,0.312174818,0.08527 12/12/2012,PupilName2,UniqueClass1,2,…

Accepted Answer

The problem you are facing sounds to me like "Bias Variance Dilemna" from a general point of view. In a nutshell, a more precise model favours variance (sensitivity to change in a single training set), a more general model favours bias (model works for many training sets)

May I suggest not to focus on GA but look at Instance Base Learning and advanced regression techniques. The Andrew moore page at CMU is a good entry point.

And particularly those slides.

[EDIT]

After a second reading, here is my second understanding:

- You have a set of example data with two related attributes X and Y.

- You do not want X/Y to dominate when Y is small, (considered as less representative).

- As a consequence you want to "weigth" the examples with a adapted value adjusted_xy .

You want adjusted_xy to be related to a third attribute R (rank). Related such as,per class, adjusted_xy is sorted like R.

To do so you suggest to put it as an optimization problem, searching for PARAMS of a given function F(X,Y,PARAMS)= adjusted_xy .

- With the constraint that D=Distance( achieved rank for this class, rank of adjusted_xy for this class ) is minimal.

Your question, at least for me, is in the field of attribute selection/attribute adaptation. (I guess the data set will later be used for supervised learning ).

One problem that I see in your approach (if well understood) is that, at the end, rank will be highly related to adjusted_xy which will bring therefore no interesting supplementary information.

Once this said, I think you surely know how GA works . You have to

- define the content of the chromosome : this appears to be your alpha parameter.

- define an appropriate fitness function

The fitness function for one individual can be a sum of distances over all examples of the dataset.

As you are dealing with real values , other metaheuristics such as Evolution Strategies (ES) or Simulated Anealing may be more adapted than GA.

As solving optimization problems is cpu intensive, you might eventually consider C or Java instead of Python. (as fitness at least will be interpreted and thus cost a lot).

Alternatively I would look at using Y as a weight to some supervised learning algorithm (if supervised learning is the target).

Utilising Genetic algorithm to overcome different size datasets in model

Solution

Problem