Mohammed Asif

Learn Microsoft Excel

2020-06-11T00:00:00-04:00

1. Introduction

Microsoft Excel is a powerful spreadsheet program that is part of the Microsoft Office suite. It’s widely used for various tasks such as data entry, analysis, calculation, and visualization. Here’s a brief introduction to some of its key features:

1.1 Data Entry and Formatting

Excel provides a grid of cells for entering data, making it easy to organize information in rows and columns. You can format cells to change the appearance of data, apply styles, and use conditional formatting to highlight specific values.

1.2 Formulas and Functions

Excel supports a vast array of formulas and functions for performing calculations. Formulas can range from basic arithmetic operations like addition and subtraction to complex calculations involving statistical, mathematical, and financial functions.

1.3 Data Cleaning and Manipulation

Excel is instrumental for cleaning and manipulating data. Its sorting and filtering capabilities allow users to organize data efficiently. Text functions like CONCATENATE, LEFT, RIGHT, and LEN help clean and transform text-based data.

1.4 Charts and Graphs

Creating charts and graphs in Excel is a key feature for visualizing data trends. Users can choose from various chart types such as bar charts, pie charts, line charts, and scatter plots. This visual representation enhances data analysis and presentation.

1.5 Data Analysis

Excel includes tools like PivotTables that simplify data analysis and summarization. Users can quickly generate insights by arranging and summarizing large datasets. The Analysis ToolPak adds advanced statistical and engineering analysis capabilities.

1.6 Dashboards and Visualization

Excel is widely used for creating interactive dashboards. Users can combine charts, graphs, and other visual elements to provide a comprehensive overview of complex datasets. Sparklines offer in-cell mini-charts for quick trend analysis.

1.7 Collaboration

Excel supports collaboration features, allowing multiple users to work on the same workbook simultaneously. Users can track changes, add comments, and share workbooks seamlessly, enhancing team productivity.

1.8 Automation with Macros

Excel’s macro feature enables users to automate repetitive tasks by recording and playing back sequences of actions. This boosts efficiency and reduces the time spent on routine operations.

1.9 Importing and Exporting Data:

Excel facilitates the import of data from external sources, including databases, text files, and online services. The ability to export data to various formats enhances compatibility with other software applications.

1.10 Conditional Formatting

Conditional formatting is a powerful feature that allows users to highlight cells based on specific conditions. This visual enhancement makes it easier to identify patterns, trends, or outliers in the data.

Whether you’re an Excel beginner or an advanced user, these features collectively make Excel a versatile and indispensable tool for various tasks, from basic calculations to complex data analysis and reporting.

References

[1] Excel practice

Visual Recognition for Vietnamese Foods

2020-05-26T00:00:00-04:00

Imagine that you are a world traveller and are traveling to a country famous for its street foods. Walking in a night market street full of food trucks with many delicious-looking options, you have no idea what these foods are and whether they contain any ingredient you are allergic to. You want to ask the local but you don’t know the language. You wish that you have an app on your phone that allows you to take a picture of that food you want to have and will return all the information you need to know about it.

It’s one simple application of computer vision that can make our life better, besides many other applications being implemented in autonomous driving or cancer detection.

Today we are going to build a world-class image classifier using the fastai library to classify 11 popular Vietnamese dishes. The fastai library is built on top of PyTorch and allows us to quickly and easily build the latest neural networks and train our models to receive state-of-the-art results.

Before we begin, let’s load packages that we are going to use in this project.

%reload_ext autoreload
%autoreload 2
%matplotlib inline

import os
import numpy as np
import urllib.request
from fastai.vision import *
from fastai.metrics import error_rate
from PIL import Image, ImageFile
ImageFile.LOAD_TRUNCATED_IMAGES = True

Looking at the data

I have built an image dataset of popular Vietnamese dishes using the Bing Image Search API by following PyImageSearch’s tutorial. We can directly download this dataset from Google Drive.

%%time
# Download and unzip
if not os.path.exists("data"):
    !wget -O "dataset.zip" "https://www.googleapis.com/drive/v3/files/13GD8pcwHJPiAPbPtm6KeC20Qw1zm9xdy?alt=media&key=AIzaSyCmo6sAQ37OK8DK4wnT94PoLx5lx-7VTDE"
    !unzip dataset.zip
    !rm dataset.zip

Each class is stored in a seperate folder in data and we can use ImageDataBunch.from_folder to quickly load our dataset. In addition, we resize our images to 224 x 224 pixel and use get_transforms to flip, rotate, zoom, warp, adjust lighting our original images (which is called data augmentation, a strategy that enables practitioners to significantly increase the diversity of data available for training models, without actually collecting new data). We also normalize the images using statistics from ImageNet dataset.

path = Path("./data/")
np.random.seed(42)
data = ImageDataBunch.from_folder(path, train='.', ds_tfms=get_transforms(),
                                  valid_pct=0.2, bs=64, size=224).normalize(imagenet_stats)

The dataset has more than 6k images and we use 20% of them as a validation set.

len(data.train_ds), len(data.valid_ds)

(4959, 1239)

Let’s look at some samples:

data.show_batch(rows=3, figsize=(7, 6))

The dataset has a total of 11 classes. They are:

print("Number of classes: ", len(data.classes))
print("Classes: ", data.classes)

Number of classes:  11
Classes:  ['banh-mi', 'banh-xeo', 'bubble-tea', 'bun-bo-hue', 'bun-bo-nam-bo', 'bun-cha', 'bun-dau-mam-tom', 'che', 'hu-tieu', 'pho', 'spring-rolls']

Training: ResNet-50

Now we will finetune a ResNet-50 model on our customized dataset. ResNet is from the paper Deep Residual Learning for Image Recognition and is the best default model for computer vision. This ResNet-34 model is trained on ImageNet with 1000 classes, so first we need to initialize a new head for the model to be adapted to the number of classes in our dataset. The cnn_learner method will do this for us (Read documentation).

Then we will train the model in two stages:

first we freeze the body weights and only train the head,
then we unfreeze the layers of the backbone and fine-tune the whole model.

Stage 1: Finetune the top layers only

learn = cnn_learner(data, models.resnet50, metrics=error_rate, pretrained=True)

We will train the model for 20 epoches following the 1cycle policy. The number of epoches we need to train depends on how different the dataset is from ImageNet. Basically, we train the model until the validation loss stops decrease. The default learning rate here is 3e-3.

learn.fit_one_cycle(20)

epoch	train_loss	valid_loss	error_rate	time
0	1.356109	0.657256	0.206618	01:31
1	0.870719	0.560793	0.167070	01:27
2	0.687549	0.526645	0.167877	01:32
3	0.621711	0.444908	0.143664	01:27
4	0.493342	0.397156	0.126715	01:28
5	0.435479	0.381771	0.123487	01:27
6	0.373893	0.389900	0.121065	01:28
7	0.325564	0.371386	0.112994	01:28
8	0.295842	0.349679	0.106538	01:29
9	0.271255	0.348150	0.098467	01:28
10	0.233799	0.317944	0.091203	01:28
11	0.205257	0.306772	0.086360	01:27
12	0.171645	0.310830	0.084746	01:28
13	0.159442	0.293081	0.078289	01:27
14	0.137362	0.295972	0.083939	01:29
15	0.112108	0.289382	0.084746	01:26
16	0.093431	0.279121	0.083939	01:30
17	0.094814	0.281516	0.079903	01:29
18	0.083980	0.277226	0.073446	01:33
19	0.090189	0.274643	0.081517	01:28

learn.save('resnes50-stage-1')

Stage 2: Unfreeze and finetune the entire networks

At this stage we will unfreeze the whole model and finetune it with smaller learning rates. The code below will help us find the learning rate for this stage.

learn.load('resnes50-stage-1')
learn.lr_find()
learn.recorder.plot()

When finetuning the model at state 2, we will use different learning rates for different layers. The top layers will be update at greater rates than the bottom layers. As a rule of thumb, we use learning rates between (a, b), in which:

a is taken from the LR Finder above where the loss starts to decrease for a while,
b is 5 to 10 times smaller than the default rate we used in stage 1.

In this case, the learning rate we will use is from 3e-6 to 3e-4 We continnue to train the model until the validation loss stops decreasing.

learn.unfreeze()
learn.fit_one_cycle(8, max_lr=slice(3e-6, 3e-4))

epoch	train_loss	valid_loss	error_rate	time
0	0.077156	0.277789	0.079903	01:30
1	0.089096	0.269736	0.075061	01:30
2	0.092252	0.282279	0.079096	01:29
3	0.074603	0.265167	0.067797	01:31
4	0.071408	0.278237	0.074253	01:33
5	0.047503	0.250248	0.065375	01:29
6	0.037788	0.249852	0.070218	01:30
7	0.035603	0.251429	0.066990	01:29

After two stages, the error rate on the validation set is about 6% and the accuracy for our dataset with 11 classes is 94%. That’s a pretty accurate model. It’s a good practice to save the current stage of the model in case we want to train it for more epoches.

learn.save('resnes50-stage-2')

Evaluation

Now let’s look on the model’s predictions. Specifically, we will look at examples where the model makes wrong predictions. Below are 9 examples with the top losses, meaning on these examples the model gives the actual class a very low score.

learn.load('resnes50-stage-2')
interp = ClassificationInterpretation.from_learner(learn)
losses, idxs = interp.top_losses()
interp.plot_top_losses(9, figsize=(15,11))

The confusion matrix below will us a big picture of which pair of dishes make our model confused. The number of wrong predictions are quite low.

interp.plot_confusion_matrix(figsize=(12,12), dpi=60)

interp.most_confused(min_val=3)

[('pho', 'hu-tieu', 11),
 ('bun-bo-nam-bo', 'bun-cha', 5),
 ('hu-tieu', 'bun-bo-nam-bo', 5),
 ('pho', 'bun-bo-hue', 5),
 ('bun-bo-hue', 'pho', 4),
 ('hu-tieu', 'bun-cha', 4),
 ('bun-bo-hue', 'hu-tieu', 3),
 ('bun-cha', 'bun-bo-nam-bo', 3),
 ('hu-tieu', 'bun-bo-hue', 3),
 ('hu-tieu', 'pho', 3)]

Production

After being satisfied with our model, we can save it to disk and use it to classify new data. We can also deploy it into a magical app that I described at the beginning of the notebook.

# Export model
learn.export()

For inference, we also want to run the model on CPU instead of GPU.

defaults.device = torch.device('cpu')

The functions below will download an image from an URL and use the model to make prediction for that image.

def open_image_url(url):
    """Download image from URL, return fastai image and PIL image."""
    urllib.request.urlretrieve(url, "./img/test.jpg")
    return open_image("./img/test.jpg"), Image.open("./img/test.jpg")

def predict(url):
    """Make prediction for image from url, show image and predicted probability."""
    img, pil_img = open_image_url(url)
    pred_class, pred_idx, outputs = learn.predict(img)
    print("Predicted Class: ", pred_class)
    print(f"Probability: {outputs[pred_idx].numpy() * 100:.2f}%")
    
    plt.figure(figsize=(15, 4))
    # Show image
    plt.subplot(1, 2, 1)
    plt.imshow(pil_img)
    plt.axis('off')
    plt.title('Image')
    
    # Plot Probabilities
    plt.subplot(1, 2, 2)
    pd.Series(outputs, data.classes).sort_values().plot(kind='barh')
    plt.xlabel("Class")
    plt.title("Probability");

# Load model
learn = load_learner(path)

# Make prediction from URL
url = str(input("URL: "))
predict(url)

URL:  https://i.pinimg.com/originals/9b/63/d7/9b63d76a44be020e03eeaec0a1134e95.jpg


Predicted Class:  bun-bo-hue
Probability: 99.99%

I have also built a web app for this model. Check it out below:

Conclusion

We have walked through the whole pipeline to finetune a ResNet-50 model on our customized dataset, evaluate its strengths and weaknesses and deploy the model for production. With the fastai library, we can quickly achieve state-of-the-art results with very neat and clean codes.

In this project, we built a classifier for only 11 popular Vietnamese dishes, but we can easily scale up the model by collecting more data for hundreds or thounsands of classes following this PyImageSearch’s tutorial.

I have seen many interesting ideas using computer vision, such as Plantix, a plant doctor app which can tell what diseases your plants might have from a single photo and how you should take care of them. And yes, with some software engineering skills, some data and an idea, you can build your own impactful computer vision application. I look forwards to seeing these applications make the world a better place in all directions.

Reference

Create a Minimalism GitHub Page for Your Data Science Portfolio in 30 Minutes

2020-01-13T00:00:00-05:00

In the early days of my journey in data science a year ago, I spent most of my time reading articles on Towards Data Science to create my own Data Science roadmap. The opinions are different in the knowledge one needs to acquire to become a Data Scientist and how to get there, but there is one thing in common: at a point in that journey, one should have a portfolio where she can host her Data Science projects.

I created my first portfolio about 6 months after I wrote my first line of Python, when I completed the Python for Data Science and Machine Learning Bootcamp on Udemy, to host simple projects I had done during the course. Since then, building and maintaining my portforlio is one of my favorite things to do. I enjoy organizing my ideas, writing them down, explaining things and make them neat.

Having a portforlio encourages me to seriously document any projects I work on. For job search, I usually bring an iPad with my portfolio opened to career events and interviews so when I share about my projects, I can guide interviewers through my codes and visualizations. It is a very efficient way to build impression and maintain the conversations.

In this tutorial, we will step-by-step learn how to build a simple but powerful GitHub page to host your Data Science projects. The whole process will take about 30 minutes. Let’s get started!

Step 1: Create a GitHub Account

First, we need to sign up a GitHub account at https://github.com/.

After signing up, we will log in and move to Step 2.

Step 2: Create a Repository Named `user-name.github.io`

After all steps in this tutorial are completed, our GitHub page can be accessed at https://user-name.github.io. In this step, we will create a repository named user-name.github.io where user-name is the user name we use to log into GitHub. My user name is ktran3-simon so I will create a repository name ktran3-simon.github.io.

To create a new repository, we click on the + sign at the top right of the screen, next to our profile picture, and select New repository.

We fill the repository name with user-name.github.io, select Public and then click Create repository.

Next, we will upload the theme to this repository. The theme we will use is the Jekyll Minimal theme. This GitHub repository has a more concise version of the theme.

To download the theme, we go to https://github.com/evanca/quick-portfolio, click Clone or download and select Download ZIP.

Now, let’s open our newly created repository, which is still empty. We will click uploading an existing file. After downloading the theme, we unzip the file and upload these files into our repository. After the uploading is complete, we click Commit changes.

Now, by going to user-name.github.io, we can already see our website! In the next step, we will go through some instructions to customize our portfolio.

(Optional) A faster way to complete this step is to simply click the Fork button to copy the entire repository to our GitHub account and then change the repository’s name to user-name.github.io. However I think the above explanation is friendlier for first-time GitHub users.

Step 3: Customize Our Portfolio

Our GitHub page has a two-column layout. On the left is our profile picture and some description, and on the right is the main page where we present our projects. I really like this design because of its simplicity yet efficiency.

To customize the sidebar (the left part), we will edit the file _config.yml in our repository. We can also add Google Analytics ID to track and analyze traffic to our page.

To customize the main page (the right part), where we display our projects, we will need to edit index.md. This file is written in Markdown. If you frequently work with Jupyter Notebook, you must be very familiar with this language. Markdown is very easy to use. Here is a helpful Markdown Cheatsheet that I often refer to.

(Optional) More Customizations

As we customize the sidebar, we will see that we cannot edit the last two lines by editting _config.yml:

To remove them, we need go to the original repository of the Jekyll Minimal theme, and copy the content of default.html in _layouts. Then we create _layouts/default.html in our repository by clicking Creating new file and typing _layouts/default.html, paste the copied content here and commit.

Now we can remove lines 29-31 in _layouts/default.html to remove View My GitHub Profile,

<p class="view"><a href="https://github.com/datacrunchcentral">View My GitHub Profile</a></p>

and line 50 to remove Hosted on GitHub pages - Theme by orderedlist.

<p><small>Hosted on GitHub Pages &mdash; Theme by <a href="https://github.com/orderedlist">orderedlist</a></small></p>

Step 4: Upload Our Projects

After customizing the design of our GitHub page, we can start adding projects to the page by editing index.md. There are several ways to do that, including:

Link to our GitHub repositories,
Link to our Jupyter Notebooks,
Write blog posts in Markdown.

My favorite way to add projects is creating a folder in the repository to save the html files of Jupyter Notebooks, and add the link https://user-name.github.io/folder-name/file-name.html to my main page. Alternatively, we can insert Google Colab links so that viewers can run our projects directly. Google Colab is basically the same as Jupyter Notebook with GPU supported by Google, which helps increase the speed of running our cells, especially in Deep Learning projects. Here is a nice Google Colab tutorial. If you are an R user, you can use R markdown in R Studio to render your projects and export them in the html format.

We can also write blog posts in Markdown and upload them to our repository. You can always refer to the file sample_page.md as an example.

Tips and Tricks

Badges with Shields.io

In official repositories on GitHub, we usually see authors use badges to show the status of their project. For example:

I really like to use these badges to embed links with call for actions, such as:

You can go to https://shields.io/ to create your own badges. Basically, we just need to create links in a specific format and use them as image links.

Format:

https://img.shields.io/badge/label-message-color?logo=logo_name

where:

color could be brightgreen, green, yellowgreen, yellow, orange, red, blue, lightgrey or any HEX, RBG color codes
logo: the list of logo of popular brands and their brand color codes can be found at simple-icons
Visit shields.io to learn more

Examples:

https://img.shields.io/badge/Spotify-My_Musics-1ED760?logo=Spotify will give us:

https://img.shields.io/badge/PyTorch-Run_in_Colab-EE4C2C?logo=PyTorch will give us:

To insert links with badges we created, we only need to type [![](link-to-our-badge)](link-to-our-project).

More Themes

There are several other themes that we can utilize to be more creative with our portfolio. To use them, we can simply Fork the repository to our account and change its name to user-name.github.io.

Cayman: repo - preview
Minimal Mistake: repo - preview. I really like this portfolio where the author uses this theme.

A preview of the Minimal Mistake Theme

Content of Your Portfolio

Last but also the most important thing I want to say in this tutorial, the reason I like a minimalism theme is that it took me minimal time on designing works; thus I can spend more time on the content of my projects. Ultimately, the purpose of building a Data Science portfolio is to present our Data Science project, rather than to show our web-design skills. Below are some articles that I found super helpful when I started building my first portfolio.

How to Build a Data Science Portfolio in Towards Data Science
Data Science Portfolios That Will Get You the Job in Dataquest
Building a Data Science Portfolio That Stands Out in Springboard Blog
5 Data Science Projects That Will Get You Hired in 2018 in Medium

Feel free to visit my portfolio to see how I write my Data Science projects. For example, this is a detailed notebook I wrote after completing a Kaggle competition, in which I went through all important steps of a Data Science project, including Exploratory Data Analysis, Data Cleaning, Feature Engineering, Modeling and Evaluation. Now I still often revisit this notebook to copy the cross-validation codes to reuse. I found I learn the most by reading notebooks on Kaggle and writing my own projects.

I also made some changes in my portfolio compared to the original version, such as making the sidebar narrower and the main page wider. You can fork my repo (https://github.com/chriskhanhtran/minimal-portfolio) and change the codes in the diretory _sass/jekyll-theme-minimal.scss as you like, including changing width, font size or image size. However, be careful when you do so because it might mess your page up. If so, just recover the settings by copying from the theme’s original repo.

change max-width in .wrapper to change the width of the entire page
change max-width in section to change the width of the main page
change width in header to change the width of the side bar

My GitHub Page

Last Words

Having completed your minimalism portfolio, you now can remove or modify these files in your repository:

LICENSE
README.md: you can modify it to the description of your page.
sample_page.md: you can remove or change it to a blog post.
pdf/sample_presentation.pdf

You can also visit the original tutorial with more tips such as:

How to create thumbnails for your project
How to create a round profile picture

For me, I use Photoshop and Powerpoint to create pictures used in my GitHub pages.

Thank you so much for staying with me to this point of my first tutorial. Don’t hesitate to reach out to me if you’ve got any questions. Please connect with me on LinkedIn and share with me your Data Science portfolio.