AWS sagemaker

Why AWS SageMaker:

Imagine a scenario, where you run a website of house listings, your website acts as a platform where tenants search for the apartments available for rents and Owners would upload their house in listings.

And to make your website more unique, you could help the Owner could actually upload their listing in the website and if they had an option of how much rent they could expect in that particular area, and with particular amenities, let's take common use case as we know if owner has their house in city centre, they could expect more rent and we accept this with ease as it is common info we had that city centre rents were higher.

We had information prebuilt here and we could have an estimate around, city centre rents in Hyderabad would be 10,000INR , if it is Bengaluru,its around 13000INR and if its Vizag, its around 8000INR, all the data we have is something we had heard or seen somehwere, and if someone asks, if we often give a rough estimate and this rough estimate helps us plan things accordingly. If your an owner- the rent you could expect, Tenant- the rent you would pay.

AWS sage-maker works the similar way, it takes data, trains data and deploys model, let us understand it more clearly with a real-time project to make sense to the statements being specified here and how AWS sage-maker makes difference in lot dynamic businesses.

let us use similar business case and deploy a project in AWS sagemaker to understand how it helps in real time scenarios

Before we begin, also understand a simple concept of how AWS sagemaker works, it takes data from AWS S3, turn up an instance required to train Data, prepares model for it which we can use, uploads trained data and model into s3 bucket back, and turns off the instance.

 

Also Create notebook instance in this particular regions only where AWS sagemaker is available.

Search for AWS sagemaker AI

Navigate to Notebooks > Create Notebook Instance

also select role and also based on it

And here IAM role is important, you can create your Own role or use default role

the default role has access to s3 buckets as we need

now, leaving everything as default, click on "Create Notebook instance"

it takes few mins for our notebook instance to spin up

once instance is up, click on Open Jupyter (this is similar to Jupyter notebook)

click on New > Conda_python3

and now rename it to anything apart from untitled to avoid confusions

Now double click on it, it opens on separate browser, select conda_python3 as kernel

Now let us generate a random data, if you want to train existing data which you have, you can skip this step and upload your data directly to S3 Bucket

import pandas as pd

data = {

    "Area": [

        "City Centre",

        "City Centre",

        "City Centre",

        "City Centre",

        "City Centre"

    ],



    "Size_sqft": [

        450,

        600,

        700,

        850,

        1000

    ],



    "Bedrooms": [

        1,

        1,

        2,

        2,

        3

    ],



    "Bathrooms": [

        1,

        1,

        2,

        2,

        2

    ],



    "Furnished": [

        "Yes",

        "Yes",

        "No",

        "Yes",

        "Yes"

    ],



    "BillsIncluded": [

        "Yes",

        "No",

        "No",

        "Yes",

        "Yes"

    ],



    "Balcony": [

        "No",

        "Yes",

        "Yes",

        "Yes",

        "Yes"

    ],



    "DistanceToMetro_min": [

        2,

        5,

        4,

        3,

        1

    ],



    "MonthlyRent": [

        1100,

        1300,

        1450,

        1750,

        2200

    ]

}

df = pd.DataFrame(data)
df

click on ">" play button to execute the cell

once executed, you get the data in tabular format.

Now, lets save this generated file into CSV file so that we can upload to s3 bucket

df.to_csv("city_centre_rent.csv", index=False)

you will be able to find the generated CSV file in the same path 

Points to remember if you are using Jupyter notebook for first time.

Each code placed is placed inside the cell, and play button is run, once the code gets into next block, it means your code executed without errors, also if there are any files generated, you can always find it in home tab. always remember to rename untitled files as jupyter notebook opens each file in different tabs, naming files can avoid potential confusions.

Now we would use sagemaker here, because of the role we attached to instance, it would be able to create s3 buckets for us and store data 

import sagemaker

session = sagemaker.Session()

SAGEMAKER_SUPPRESS_V2_WARNING=1

bucket = session.default_bucket()

print(bucket)

Now the above command creates bucket for us and we can see the bucket name.

As the bucket is created, let us know upload the Data that we generated to this s3 bucket and verify the path 

s3_path = session.upload_data(
    path="city_centre_rent.csv",
    bucket=bucket,
    key_prefix="rent-demo"
)

print(s3_path)





Note: you can verify it in AWS s3 bucket and it would be the same

Now, lets, get into trainining our model, we will use scikit-learn for it
    
!pip install scikit-learn




In the below command, we are actually having X as questions and Y as answers, X taking information about everything except 

  



MontlyRent and Y taking only Monthlyrent

this train and test helps our model to check on the accuracy, for example we have values around
5 building and now 4th building value is taken as test, Model uses values of 1,2,3 and 5, trains the data and tries to predict rent of 4th building, once it predicted than compares with actual rent value of 4th building

Also to understand, since model understands values in numbers,we are converting words into 
numbers here using enconder

for suppose, we have City Centre as only area here, so it takes it as 0, sameway
if there are other values like City outskirts, it takes it values as 1, Subway area as 2

Also yes or no questions as 1 and 0
in the below command, we are encoding furnishes "yes" as 1. furnished "No" as 0
same goes to BillsIncluded and Balcony

 
from sklearn.preprocessing import LabelEncoder

encoder = LabelEncoder()

df["Area"] = encoder.fit_transform(df["Area"])
Copy

binary_columns = [
    "Furnished",
    "BillsIncluded",
    "Balcony"
]

for column in binary_columns:
    df[column] = encoder.fit_transform(df[column])


X = df.drop("MonthlyRent", axis=1)

y = df["MonthlyRent"]


from sklearn.model_selection import train_test_split


X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)


   



Now, let's do the training of Model, we will use RandomForestRegressor method, think of it something
like you are asking 100 real estate agents rent in a particular area, have all the values and average it.

from sklearn.ensemble import RandomForestRegressor

model = RandomForestRegressor(
    n_estimators=100,
    random_state=42
)

model.fit(X_train, y_train)





Now we were able to train our model with values given and let us also now test the values, as we 
have placed questions, in X, we will use this model to check predictions.

    
predictions = model.predict(X_test)

print(predictions)

and the above X_test values are random which our model picks up and this is only a test to see
if our model is working as per requirement or not.

since these are predicted models, let us search error in predictions that our model can make
and below, we use mean absolute error.



    
from sklearn.metrics import mean_absolute_error

mae = mean_absolute_error(y_test, predictions)

print(mae)


here, the value 81.5 means, our model has an error of 81.5, for suppose actual rent is 1300,
our model predicts value to be 1381.5 or 1218.5, since these are rents, 100 INR difference 
would not make much and we can proceed on deploying model, but in case high precision
results are required, we train our model with more data to reduce this error.

let us now dump our model
    
import joblib

joblib.dump(model, "rent_model.pkl")











Now since our model is ready, let us upload our model to s3 Bucket.


    
session.upload_data(
    path="rent_model.pkl",
    bucket=bucket,
    key_prefix="trained-model"
)





Now we have our trained Model ready, you can call this model from any other notebook instance
or use this model as API, lets say a button called "Predict Rent" in your website and if user inputs
all the data and then clicks on Predict rent, it would should answer like

the apporoximate rate on this area could be around 1300 euros.

let us test our model by giving few values that we expect user would input.

we will import model first and then have the data as below and lets check the answer



import joblib

model = joblib.load("rent_model.pkl")



import pandas as pd

new_apartment = pd.DataFrame({
    "Area": [0],
    "Size_sqft": [750],
    "Bedrooms": [2],
    "Bathrooms": [2],
    "Furnished": [1],
    "BillsIncluded": [1],
    "Balcony": [1],
    "DistanceToMetro_min": [3]
})


prediction = model.predict(new_apartment)

print(prediction)
Because our model does not understand strings,  and here furnishes 1 means yes, and Area 0
is City Centre





you can also do a manual mapping to have a clear idea on values

command:




import joblib
import pandas as pd

model = joblib.load("rent_model.pkl")

# Manual mapping
area_map = {
    "City Centre": 0
}

yes_no_map = {
    "No": 0,
    "Yes": 1
}

new_apartment = pd.DataFrame({
    "Area": [area_map["City Centre"]],
    "Size_sqft": [750],
    "Bedrooms": [2],
    "Bathrooms": [2],
    "Furnished": [yes_no_map["Yes"]],
    "BillsIncluded": [yes_no_map["Yes"]],
    "Balcony": [yes_no_map["Yes"]],
    "DistanceToMetro_min": [3]
})

prediction = model.predict(new_apartment)

print(prediction)



import joblib
import pandas as pd

model = joblib.load("rent_model.pkl")

# Manual mapping
area_map = {
    "City Centre": 0
}

yes_no_map = {
    "No": 0,
    "Yes": 1
}

new_apartment = pd.DataFrame({
    "Area": [area_map["City Centre"]],
    "Size_sqft": [750],
    "Bedrooms": [2],
    "Bathrooms": [2],
    "Furnished": [yes_no_map["Yes"]],
    "BillsIncluded": [yes_no_map["Yes"]],
    "Balcony": [yes_no_map["Yes"]],
    "DistanceToMetro_min": [3]
})

prediction = model.predict(new_apartment)

print(prediction)





Additional Steps if Resource Quota is available
And now, if you have data that keeps incoming and you want to train data dynamically everytime
for example, your webiste now provides a data in a new city Rajahmundry

you can always add the new data into CSV file or at particular path and now AWS sagemaker
would help you train and deploy model with simple commands.

create a new file and name it as train.py




Now paste the below file there, this is the same training that we did earlier
    
import os
import argparse
import joblib
import pandas as pd

from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error


# ======================================================
# Step 1: Read SageMaker environment variables
# ======================================================

parser = argparse.ArgumentParser()

parser.add_argument(
    "--model-dir",
    type=str,
    default=os.environ["SM_MODEL_DIR"]
)

parser.add_argument(
    "--train",
    type=str,
    default=os.environ["SM_CHANNEL_TRAIN"]
)

args = parser.parse_args()


# ======================================================
# Step 2: Load training dataset
# ======================================================

train_file = os.path.join(
    args.train,
    "city_centre_rent.csv"
)

print(f"Reading training data from: {train_file}")

df = pd.read_csv(train_file)

print(df.head())


# ======================================================
# Step 3: Encode categorical columns
# ======================================================

encoder = LabelEncoder()

df["Area"] = encoder.fit_transform(df["Area"])

binary_columns = [
    "Furnished",
    "BillsIncluded",
    "Balcony"
]

for column in binary_columns:
    df[column] = encoder.fit_transform(df[column])


# ======================================================
# Step 4: Separate Features and Target
# ======================================================

X = df.drop("MonthlyRent", axis=1)

y = df["MonthlyRent"]


# ======================================================
# Step 5: Split the dataset
# ======================================================

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)


# ======================================================
# Step 6: Train the model
# ======================================================

model = RandomForestRegressor(
    n_estimators=100,
    random_state=42
)

model.fit(X_train, y_train)

print("Model training completed.")


# ======================================================
# Step 7: Evaluate the model
# ======================================================

predictions = model.predict(X_test)

mae = mean_absolute_error(
    y_test,
    predictions
)

print(f"Mean Absolute Error: {mae}")


# ======================================================
# Step 8: Save the trained model
# ======================================================

model_path = os.path.join(
    args.model_dir,
    "model.joblib"
)

joblib.dump(
    model,
    model_path
)

print(f"Model saved to: {model_path}")



save the python file





Get s3 path and role

    

import sagemaker

session = sagemaker.Session()

bucket = session.default_bucket()

print(bucket)

s3_path = session.upload_data(
    path="city_centre_rent.csv",
    bucket=bucket,
    key_prefix="rent-demo"
)

print(s3_path)

    
import sagemaker

session = sagemaker.Session()

role = sagemaker.get_execution_role()

print(role)










the next command, in case if you have resource quota available, it turns up instances and adds
up additional expenses for a demo project.

the additional steps are only to show how AWS sagemaker works in case of huge data that
needs to be trained dynamically

and based on instance_type selected, costs inccur and instance type has GPU required
    
import sagemaker
from sagemaker.sklearn.estimator import SKLearn

session = sagemaker.Session()
role = sagemaker.get_execution_role()

estimator = SKLearn(
    entry_point="train.py",
    role=role,
    framework_version="1.2-1",
    py_version="py3",
    instance_count=1,
    instance_type="ml.m5.large"
)

estimator.fit(
    {"train": s3_path},
    wait=True,
    logs=True
)




you can search instance available for you in your region in simplest way from UI, by navigating to training models,










this concludes blog here.