Dimensionality Reduction in DataRobot Using t-SNE

Horizontal

t-SNE (t-Distributed Stochastic Neighbor Embedding) is a powerful technique for dimensionality reduction that can effectively visualize high-dimensional data in a lower-dimensional space.

Build with Free Trial

Dimensionality reduction can improve machine learning results by reducing computational complexity of the algorithms, preventing overfitting, and focusing on the most relevant features in the dataset. Note that this technique should only be used when the number of features is low.

Import libraries

In [ ]:
import datarobot as dr
import pandas as pd
import seaborn as sns
from sklearn.manifold import TSNE

Connect to DataRobot

Instructions for obtaining your endpoint and token are located in the DataRobot API documentation here.

In [3]:

# either directly pass in your endpoint/token, use a config file, or connect using DataRobot notebooks
dr.Client()
Out [3]:

<datarobot.rest.RESTClientObject at 0x7f5f10312280>

Get dataset

This example uses data on the movement of a double pendulum which has already been loaded into DataRobot for this example, but can be found here.

In [40]:

# replace the dataset ID with your own data
ds_id = "62fbcdf583b30f0ef972dc31"

# get dataset from DataRobot
ds = dr.Dataset.get(ds_id)
df = ds.get_as_dataframe()
display(df)

Out[40]:

tx1x2v1v2a1a2
00.0000002.363.14-0.0100-0.01000-9.246.53
10.0008622.363.14-0.0180-0.00437-9.246.53
20.0017202.363.14-0.02590.00126-9.246.53
30.0025902.363.14-0.03390.00689-9.246.53
40.0034502.363.14-0.04180.01250-9.246.53
24249.970000-14.70-22.401.14001.820006.94-3.84
24259.980000-14.70-22.301.20001.790007.04-3.64
24269.980000-14.70-22.301.25001.760007.12-3.42
24279.990000-14.70-22.301.31001.730007.20-3.19
242810.000000-14.70-22.301.37001.700007.28-2.95
2429 rows × 7 columns

Reduce the number of features in the dataset

In [ ]:

# features to exclude from reduction
# can be target columns or ID columns or other
exclude_cols = ["t", "a2"]

model = TSNE(learning_rate=100, random_state=42)
transformed = model.fit_transform(df.drop(exclude_cols, axis=1))
In [25]:

transformed
Out [25]:

array([[  2.542573 , -80.301025 ],
       [  2.5057044, -80.29103  ],
       [  2.869162 , -80.113396 ],
       ...,
       [  9.5524645,  74.92201  ],
       [  9.630235 ,  74.90384  ],
       [  9.827253 ,  74.67084  ]], dtype=float32)

Create new dataframe with reduced columns and previously excluded columns

In [39]:

# get the tsne dataset
reduced_df = pd.DataFrame(transformed, columns=["tsne_x", "tsne_y"])

# join in target and time columns from original dataset
reduced_df = pd.concat([reduced_df, df[exclude_cols]], axis=1)

display(reduced_df)

Out[39]:

tsne_xtsne_yta2
02.542573-80.3010250.0000006.53
12.505704-80.2910310.0008626.53
22.869162-80.1133960.0017206.53
32.899721-80.0681080.0025906.53
42.924986-80.0203320.0034506.53
24249.65827174.4330379.970000-3.84
24259.41713574.9999929.980000-3.64
24269.55246474.9220129.980000-3.42
24279.63023574.9038399.990000-3.19
24289.82725374.67083710.000000-2.95
2429 rows × 4 columns

Upload back to DataRobot

In [42]:


ds = dr.Dataset.create_from_in_memory_data(
    data_frame=reduced_df, fname=f"{ds.name}.csv"
)
ds.modify(name=f"{ds.name} t-SNE Reduced")
ds
     
Out [42]:

Dataset(name='Double Pendulum.csv.csv t-SNE Reduced', id='65a970bc040d9a438cdfb9de')
     
Get Started with Free Trial

Experience new features and capabilities previously only available in our full AI Platform product.

Get Started with Dimensionality Reduction

Explore more Industry Agnostic AI Accelerators

Explore more AI Accelerators