# What Should I do with NaN Values when Transforming Categorial Data to Numerical Data?

C

#### Connor

##### Guest
Connor Asks: What Should I do with NaN Values when Transforming Categorial Data to Numerical Data?
I'm currently working on a dataset with several categorial features, which I need to transform into numerical features through one hot encoding.

However, some of the features have NaN values in them. I understand that, for numerical features, these should be replaced by the feature's mean value to avoid any issues, but I'm unsure of what to do for categorical features.

Will I lose any information if I don't include NaN as a category? What's the typical approach to this situation?

SolveForum.com may not be responsible for the answers or solutions given to any question asked by the users. All Answers or responses are user generated answers and we do not have proof of its validity or correctness. Please vote for the answer that helped you in order to help others find out which is the most helpful answer. Questions labeled as solved may be solved or may not be solved depending on the type of question and the date posted for some posts may be scheduled to be deleted periodically. Do not hesitate to share your thoughts here to help others.

#### How to best train a CNN with longitude/latitude output

Joshua Lin Asks: How to best train a CNN with longitude/latitude output
Problem

I am new to CNNs and I'm starting off with a geolocation problem where the input is an image and the output should be a longitude value and a latitude value.

I am unsure of the best way to implement this and I'm having problems finding examples online to help. The most similar ones I can find use a grid/tile approach w/ probabilities rather than having the output be long/lat values directly. I would prefer to stick to this approach if possible.

Current Approach

I'm working with TensorFlow and currently have it set up so that the last layer is Dense(2).

I am using MSE as the loss function, although I think that's probably not ideal. I'm guessing I should use geodesic distance (Haversine formula).

SolveForum.com may not be responsible for the answers or solutions given to any question asked by the users. All Answers or responses are user generated answers and we do not have proof of its validity or correctness. Please vote for the answer that helped you in order to help others find out which is the most helpful answer. Questions labeled as solved may be solved or may not be solved depending on the type of question and the date posted for some posts may be scheduled to be deleted periodically. Do not hesitate to share your thoughts here to help others.

#### Is there a point in hyperparameter tuning for Random Forests?

BoS_88 Asks: Is there a point in hyperparameter tuning for Random Forests?
I have a binary classification task with substantial class imbalance (99% negative - 1% positive). I want to developed a Random Forest model to make prediction, and after establishing a baseline (with default parameters), I proceed to hyperparameter tuning with scikit-learn's GridSearchCV.

After setting some parameters (e.g. max_depth, min_samples_split, etc.), I noticed that the best parameters, once GridSearch was done, are highest max parameters (max_depth) and the smallest min parameters (min_samples_split, min_samples_leaf). In other words, GridSearchCV favored the combination of parameters that fits most closely to the training set, i.e. overfitting it. I always thought that cross-validation would protect from this scenario.

Therefore, my question is 'What is the point of GridSearch if the outcome is overfitting?' Have I misunderstood its purpose?

My code:

Code:
rf = RandomForestClassifier(random_state=random_state)

param_grid = {
'n_estimators': [100, 200],
'criterion': ['entropy', 'gini'],
'max_depth': [5, 10, 20],
'min_samples_split': [5, 10],
'min_samples_leaf': [5, 10],
'max_features': ['sqrt'],
'bootstrap': [True],
'class_weight': ['balanced']
}

rf_grid = GridSearchCV(estimator=rf,
param_grid=param_grid,
scoring=scoring_metric,
cv=5,
verbose=False,
n_jobs=-1)

best_rf_grid = rf_grid.fit(X_train, y_train)


SolveForum.com may not be responsible for the answers or solutions given to any question asked by the users. All Answers or responses are user generated answers and we do not have proof of its validity or correctness. Please vote for the answer that helped you in order to help others find out which is the most helpful answer. Questions labeled as solved may be solved or may not be solved depending on the type of question and the date posted for some posts may be scheduled to be deleted periodically. Do not hesitate to share your thoughts here to help others.

#### ValueError: cannot reshape array of size 36276416 into shape (96,227,227,1)

Josh Asks: ValueError: cannot reshape array of size 36276416 into shape (96,227,227,1)
I am running my LeNet code with LFW, but when I run it, I am getting the following error message:

Here is the code that it is getting the error

Code:
# Import the packages
from keras.preprocessing.image import ImageDataGenerator

# Image Data Augmentation
train_generator = ImageDataGenerator(rotation_range=2, horizontal_flip=True, zoom_range=.1)
val_generator = ImageDataGenerator(rotation_range=2, horizontal_flip=True, zoom_range=.1)
test_generator = ImageDataGenerator(rotation_range=2, horizontal_flip=True, zoom_range=.1)

# Fitting the augmentation defined above to the data
train_generator.fit(xtrain)
val_generator.fit(x_val)
test_generator.fit(xtest)

# Construct the image generator for data augmentation
aug = ImageDataGenerator(width_shift_range=0.1, height_shift_range=0.1,
horizontal_flip=True, fill_mode="nearest")

Code:
# Fitting the augmentation defined above to the data
train_generator.fit(xtrain.reshape(96, 227, 227, 1))
val_generator.fit(x_val.reshape(96, 227, 227, 1))
test_generator.fit(xtest.reshape(96, 227, 227, 1))

but then got this:

Code:
Traceback (most recent call last):
File "C:\Users\JoshG\PycharmProjects\LeNet\LeNet.py", line 134, in <module>
train_generator.fit(xtrain.reshape(96, 227, 227, 1))
ValueError: cannot reshape array of size 36276416 into shape (96,227,227,1)

I have added the full code for more help on how to combat this issue. What is the fix for something like this?

UPDATE: I then made the following changes to:

Code:
train_generator.fit(xtrain.reshape(-1, 227, 227))
val_generator.fit(x_val.reshape(-1, 227, 227))
test_generator.fit(xtest.reshape(-1, 227, 227))

to indicate that the value shall be computed automatically, but then when I run it. I get this error message:

Code:
Traceback (most recent call last):
File "C:\Users\JoshG\PycharmProjects\LeNet\LeNet.py", line 135, in <module>
train_generator.fit(xtrain.reshape(-1, 227, 227))
File "C:\Users\JoshG\AppData\Local\Programs\Python\Python39\lib\site-packages\keras_preprocessing\image\image_data_generator.py", line 935, in fit
raise ValueError('Input to .fit() should have rank 4. '
ValueError: Input to .fit() should have rank 4. Got array with shape: (704, 227, 227)`

SolveForum.com may not be responsible for the answers or solutions given to any question asked by the users. All Answers or responses are user generated answers and we do not have proof of its validity or correctness. Please vote for the answer that helped you in order to help others find out which is the most helpful answer. Questions labeled as solved may be solved or may not be solved depending on the type of question and the date posted for some posts may be scheduled to be deleted periodically. Do not hesitate to share your thoughts here to help others.

#### What is the positional encoding in the transformer model?

Peyman Asks: What is the positional encoding in the transformer model?
I'm trying to read and understand the paper Attention is all you need and in it, there is a picture:

I don't know what positional encoding is. by listening to some youtube videos I've found out that it is an embedding having both meaning and position of a word in it and has something to do with $sin(x)$ or $cos(x)$

but I couldn't understand what exactly it is and how exactly it is doing that. so I'm here for some help. thanks in advance.

SolveForum.com may not be responsible for the answers or solutions given to any question asked by the users. All Answers or responses are user generated answers and we do not have proof of its validity or correctness. Please vote for the answer that helped you in order to help others find out which is the most helpful answer. Questions labeled as solved may be solved or may not be solved depending on the type of question and the date posted for some posts may be scheduled to be deleted periodically. Do not hesitate to share your thoughts here to help others.

#### Sorting a collection of tuples using merge rearrangements

7H3ju Asks: Sorting a collection of tuples using merge rearrangements
Given a collection of tuples $X=\{(x_1,y_1),\dots,(x_n,y_n)\}$, where elements $x_i, y_i \in R_{\geq 0}$ are non-negative real values. The collection $X$ is sorted if $x_i \leq x_{i+1}$ and $y_i \leq y_{i+1}$ for all $i \in [n-1]$. Sorting $X$ is not always possible for instance if the given input has two tuples $(x_i,y_i), (x_j,y_j)$ such that $x_i > x_j$ and $y_i < y_j$ for some $i \neq j$. So we want to merge tuples in $X$ so that the resulting collection is sorted and the merge operation $\phi(i,j,k)$ is defined as $$\phi(i,j,k) := \Big\{\text{assign}~X[k] \gets \big\{(x_k,y_k) = \big(\frac{x_i+x_j}{2}, \frac{y_i+y_j}{2}\big)\big\} ~\text{and delete}~(x_i,y_i), (x_j,y_j)~\text{from collection}~X \big\}.$$

The problem always has an obvious solution with $(n-1)$ merge operations i.e, merging everything to a single tuple is always feasible. But we would like to find the minimum number of merge operations required to sort the collection.

Even though we suspect that finding the minimum number of merge operations is NP-hard, we do not have a hardness proof to support the claim. The problem looks like something which might have been already studied in the literature. If you are aware of any related or similar problems please guide us to relevant results. Any pointers or clues for hardness or algorithmic results are helpful.

Example: Given $X=\{(1,4),(2,2),(3,2),(4,1)\}$ with two merge operations i.e, $\phi(1,4,3)$ followed by $\phi(1,2,1)$ we can obtain $\{(2.5,2),(2.5,2.5)\}$, which is sorted.

$$\{(1,4),(2,2),(3,2),(4,1)\} \xrightarrow[]{\phi(1,4,3)} \{(2,2),(3,2),(2.5,2.5)\}$$ $$\{(2,2),(3,2),(2.5,2.5)\} \xrightarrow[]{\phi(1,2,1)} \{(2.5,2),(2.5,2.5)\}$$

Note: If the merge operation is restricted to operations of the form $\phi(i,i+1,i)$, then the minimum number of operations can be found in polynomial time using a dynamic programming algorithm.

SolveForum.com may not be responsible for the answers or solutions given to any question asked by the users. All Answers or responses are user generated answers and we do not have proof of its validity or correctness. Please vote for the answer that helped you in order to help others find out which is the most helpful answer. Questions labeled as solved may be solved or may not be solved depending on the type of question and the date posted for some posts may be scheduled to be deleted periodically. Do not hesitate to share your thoughts here to help others.

#### AVL-tree insertion complexity proof

Byteq Asks: AVL-tree insertion complexity proof
I tried to figure out the proof of insertion operation in AVL-tree is O(log n), but I do not know how. I also tried to find it somewhere on the Internet, but I could not find any good results. Do you guys have any ideas how do we proof that?

SolveForum.com may not be responsible for the answers or solutions given to any question asked by the users. All Answers or responses are user generated answers and we do not have proof of its validity or correctness. Please vote for the answer that helped you in order to help others find out which is the most helpful answer. Questions labeled as solved may be solved or may not be solved depending on the type of question and the date posted for some posts may be scheduled to be deleted periodically. Do not hesitate to share your thoughts here to help others.

#### [Solved] Which coordinate system should be used with qgis2web (QGIS to Leaflet)?

• Chewcata
• Geography
• Replies: 0
Chewcata Asks: Which coordinate system should be used with qgis2web (QGIS to Leaflet)?
For my project in University I want to create an interactive map. My Prof told me, Leaflet only uses WGS84, but he wasn't sure. When using the qgis2web plugin, should I have WGS84 as project coordinate system or is any other system also fine? If no, which WGS84 coordinate system should I use (e.g. EPSG:4326; EPSG:4979).

SolveForum.com may not be responsible for the answers or solutions given to any question asked by the users. All Answers or responses are user generated answers and we do not have proof of its validity or correctness. Please vote for the answer that helped you in order to help others find out which is the most helpful answer. Questions labeled as solved may be solved or may not be solved depending on the type of question and the date posted for some posts may be scheduled to be deleted periodically. Do not hesitate to share your response here to help other visitors like you. Thank you, solveforum.