After a significant pause of this chapter, it's finally revived. So if you are located in close geographic proximity or are interested to remotely follow the events, join the official Meetup Group.

Reflecting on the exhilarating experience of spearheading the community's revival during the first event post-hiatus! It was an honor to contribute to reigniting the community spirit. Our exploration into the realms of interoperability and operationalizing ML models with ONNX was truly rewarding. The added excitement of delving into the seamless deployment capabilities offered by ONNX Runtime made this journey unforgettable. A heartfelt thanks to everyone who joined the discussion, and let's look forward to many more insightful conversations on the horizon!

Synopsis of my talk

Challenges: Taming the Complexity

First off, let's acknowledge the challenges we face in the ever-expanding universe of deep learning. Multiple frameworks like TensorFlow and PyTorch, coupled with varied training accelerators (think T4 GPUs and VPUs), make the landscape intricate. This is where ONNX steps in, initiated by tech giants AWS, Microsoft, and Meta. It acts as a unifying force, providing an abstraction layer over frameworks and hardware, reducing the complexity of integrating these different components.

Design Principles: Flexibility and Standardization

ONNX isn't just another acronym; it's a set of design principles that support both deep learning and traditional machine learning. These principles aim to be flexible enough to keep up with the rapid advances in AI while providing a standardized, cross-platform representation for serialization. Imagine it as the Common Language Runtime for programming languages, minimizing the number of moving parts in the deep learning landscape.

Understanding the ONNX Specification

So, what's under the hood of ONNX? The ONNX specification reveals a structured format where each computational graph is a directed acyclic graph. Nodes represent inputs, outputs, and operators, with metadata documenting crucial details about the model and its production environment. Data types, including tensor types and non-tensor types in ONNX-ML, showcase the framework's versatility.

Operators: The Essence of ONNX

Now, let's talk about operators – the heart and soul of ONNX. These are defined by name, domain, and version, and they encapsulate the essence of various operations. Take, for instance, the Relu and Abs operators, which I'll be diving into during the presentation. These examples demonstrate input-output relationships and type constraints, showcasing the elegance and power of ONNX.

Practical Demos: Bridging Theory and Application

Enough theory; let's get hands-on! In Demo 1, we'll walk through defining a PyTorch model, training it, exporting it to ONNX, and visualizing the resulting graph. The practical application comes to life as we tackle a real-world problem – training a classifier for the MNIST dataset using a convolutional neural network in PyTorch.

ONNX Runtime: Bridging the Gap

What about deployment? Enter ONNX Runtime, the bridge between the ONNX file format and deploying the graph on different hardware. Microsoft takes the lead here, maintaining ONNX Runtime as a distinct project from the ONNX specification governed by the Linux Foundation AI.

Interoperability: ONNX in Action

But ONNX isn't limited to theoretical discussions. We'll explore interoperability, touching on the ONNX Model Zoo, Azure Cognitive Services, and methods to convert existing models or train from scratch. Practical tools like Netron and VisualDL will be your companions in understanding and visualizing ONNX models.

Demo 2: ONNX in the Real World

The finale? Demo 2, where we take the model from Demo 1, integrate ONNX Runtime in JavaScript, and build an inference REST API with Express.js. This real-world application underscores ONNX's cross-language applicability, bringing everything full circle from theory to practice.

Join the ONNX Adventure

I'm beyond excited to share this ONNX adventure with you. Whether you're a seasoned deep learning enthusiast or just dipping your toes into the data science world, this journey promises to offer insights, practical knowledge, and a newfound appreciation for the unifying force that ONNX brings to the table.

In the realm of software development, the pursuit of observability and data-driven decision-making has become a fundamental aspect of building and maintaining robust microservices architectures. Drawing inspiration from this ethos, I sought to apply these principles to a rather unexpected domain - the care and cultivation of my cherished Bonsai tree.

Similar to the complex systems we engineer, the art of growing a Bonsai tree involves a multitude of parameters that must be carefully balanced for optimal results. However, fine-tuning these variables can be a time-consuming process. Fortunately, certain rules of thumb exist within this horticultural art form that provide valuable guidance in our quest for nurturing these miniature living masterpieces.

Be sure not to water your tree if the soil is still wet, but don't let the tree dry out either.

As a beginner, use your fingers at about one centimeter deep, (0.4") to check the soil moisture. If it's slightly dry, go ahead and water your tree.

followed by:

Avoid watering all of your trees on a daily routine, until you know exactly what you are doing.

This sounds not too exact (for me at least) given my objective is to have a good-looking tree on my desk with healthy and green leaves

Requirements

To get started you'll need:

• Arduino-like Board

• Soil-Moisture Sensor

• Wi-Fi module (optional)

In this project, NodeMCU V2 ESP8266 paired with Soil Moisture Hygrometer Detection Humidity Sensor was used.

Diagram

The schematics are pretty straight-forward and as follows:

I've settled on analogous mode for the sensor.

Code

The prerequisites of this step are to connect your development board to a serial monitor and decide on a Baud Rate (in this example 9600). The source code is available on Aleksandar1932/overkill-bonsai.

The next step is to write a lib that provides an API to the soil moisture sensor.

Stating by taking defining a few constants in lib/Constants /Constants.cpp:

#define sensorPower 0
#define sensorPin A0

#define WET_THRESHOLD 500     // Define max value we consider soil 'wet'
#define DRY_TREHSHOLD 750     // Define min value we consider soil 'dry'
#define MEASURE_INTERVAL 1000 // Define how often we check soil moisture (milliseconds)

Next is the lib/Moisture /Moisture.cpp:

#include <Arduino.h>
#include <Constants.h>

int readSensor()
{
    digitalWrite(sensorPower, HIGH);
    delay(10);
    int val = analogRead(sensorPin);
    digitalWrite(sensorPower, LOW);
    return val;
}

void setupSoilMoistureSensor()
{
  pinMode(sensorPower, OUTPUT);
  digitalWrite(sensorPower, LOW);
}

void logMoisture(int moisture)
{
  Serial.print("Analog Output: ");
  Serial.println(moisture);

  // Determine status of our soil
  if (moisture < WET_THRESHOLD)
  {
    Serial.println("Status: Soil is too wet");
  }
  else if (moisture >= WET_THRESHOLD && moisture < DRY_TREHSHOLD)
  {
    Serial.println("Status: Soil moisture is perfect");
  }
  else
  {
    Serial.println("Status: Soil is too dry - time to water!");
  }
}

At this point, the core API is defined, and the reset is implementing the presentation layer that will allow us to interact with the sensor. For this I've used ESP8266WebServer for the web server and ESP8266WiFiMulti for Wi-Fi connectivity.

The web server implements two handlers

• on / will return json response containing the soil-moisture reading alongside the status (wet, perfect and dry) determined by the thresholds.

• on /display will return an HTML response containing the Moisture and some "prettier" UI.

The presentation layer alongside rest of the logic is as follows:

#include <Arduino.h>
#include <ESP8266WiFi.h>
#include <WiFiClient.h>
#include <ESP8266WebServer.h>
#include <ESP8266WiFiMulti.h>
#include <ESP8266mDNS.h>
#include <Moisture.h>
#include <Constants.h>

ESP8266WebServer server(80);
ESP8266WiFiMulti wifiMulti;

void handleMeasurement();
void handleDisplayHTML();
void connectToWifi();
void handleDisplayPrettyHTML();

void setup()
{
  Serial.begin(9600);
  connectToWifi();
  setupSoilMoistureSensor();
  server.on("/", handleMeasurement);
  server.on("/display", handleDisplayHTML);
  server.begin();
}

void loop()
{
  int moisture = readSensor();
  logMoisture(moisture);
  delay(MEASURE_INTERVAL);
  server.handleClient();
}

void handleMeasurement()
{
  int moisture = readSensor();
  server.send(200, "application/json", "{\"moisture\": " + String(moisture) + ", \"status\": \"" + (moisture < WET_THRESHOLD ? "wet" : (moisture >= WET_THRESHOLD && moisture < DRY_TREHSHOLD ? "perfect" : "dry")) + "\"}");
}

void handleDisplayHTML()
{
  server.send(200, "text/html", "<html><head><title>ESP8266 Soil Moisture Sensor</title></head><body><h1>Aleksandar's Bonsai</h1><p>Moisture: " + String(readSensor()) + "</p></body></html>");
}

void connectToWifi()
{
  wifiMulti.addAP(getenv("WIFI_SSID"), getenv("WIFI_PASSWORD"));
  Serial.println("Connecting ...");

  while (wifiMulti.run() != WL_CONNECTED)
  {
    delay(250);
    Serial.print('.');
  }
  Serial.println('\n');
  Serial.print("Connected to ");
  Serial.println(WiFi.SSID());
  Serial.print("IP address:\t");
  Serial.println(WiFi.localIP());

  if (MDNS.begin(getenv("MDNS_HOSTNAME")))
  {
    Serial.println("mDNS responder started");
  }
  else
  {
    Serial.println("Error setting up MDNS responder!");
  }
}

Additionally, I've used mDNS for development purposes.

Determining optimal thresholds

To determine the thresholds I've used this chart as a reference, but the freedom is yours to fine-tune to achieve optimal results (whatever your objective is).

Final thoughts

The integrated solution turned out to look as:

This implementation serves as an initial foundation for an IoT project, with future plans to expand its capabilities. I intend to enhance the system by integrating a water pump and canister, enabling automated watering when the soil moisture threshold is low. Moreover, I plan to incorporate a feedback loop using a photo-sensor directed at the leaves, providing insights into the tree's performance and enabling the determination of an optimal threshold.

In addition to these advancements, I aim to integrate the board into the Tuya platform, replacing the existing presentation layer. This integration will seamlessly incorporate the Bonsai care system into my smart-home ecosystem, enhancing its accessibility and integration with other connected devices.

Historically speaking, people have always craved to discover patterns within their environments. The simplest form of pattern recognition can be identified in the earliest days of mankind. Centuries later, starting in the third industrial revolution data began to manifest in a digital form, and so recognizing patterns within these digital data.

Today, we are witnesses of machine learning models embedded in nearly every application we use, starting from entertainment games and going all the way up to the most industry-grade applications. Academia and Industry have both joint forces and put enormous amounts of resources into research and development of state of the art models (SOTA), many of them available open-source. Those models combined with transfer learning and a vast amount of open data, allow one to develop machine learning applications within an extremely short time frame after the initial idea is defined. Those ideas framed into applications spawn new startups at a rate never seen before.

Every model (from failed to SOTA) starts in a research environment, commonly in a notebook. As the project starts being more serious, utility libraries are generalized for further reuse, scripts for training, deployment, evaluating, and visualizing results are written, logging at several levels embodied, and when put into production, performance monitoring setup being built. Tasks such as retraining and prevention of performance degradation are often part of those monitoring setups, which analyze input drift or evaluate through ground truth data.

After the research phase, much of the focus and effort is pure software engineering. Historically, processes and technologies, commonly recognized in the industry have revolutionized and modernized software development. They appear in different flavors of agile methodologies, review and release processes, version control, conventions, CI/CD... Big players in the industry frequently share their experiences about the adoption of proces through various forms of media. Such example being Why Google Stores Billions of Lines of Code in a Single Repository?.

Data Scientists should focus on exploring and developing models. All the other stuff should be uniform and standardized. Developing models should be put beyond Jupyter Notebooks, and custom scripts.

Data should be versioned, results visualized, training monitored, real-time reports available for all stakeholders. All and all, a step forward should be given towards the explainability and interoperability of results. This can be done (and it is currently done) from scratch within every organization and team, and it works perfectly. Data is versioned following conventions and rules, scripts for model evaluation and reports are developed, monitoring dashboards and reports for stakeholders are drafted. Now, let's shift scope and imagine if every software company developed its version control system and container technology - the industry would be a total mess.

MLOps

What is it?

A formal definition states, MLOps is a set of practices that aims to deploy and maintain machine learning models in production reliably and efficiently.

Who does it?

In software development, there are many different roles, each of them carrying responsibilities and perceiving the same products from different aspects. In a data science project the following ones can be differentiated:

• Business Analyst or Domain Expert
• Data Analyst
• Software Engineers and Developers
• Data and ML Architects
• Data Engineers
• MLOps Engineers
• Optimization engineers
• Data Scientists

What are the common tasks?

The first attempt to identify the problems in current ML applications workflow was done by a group of researchers at Google in their paper Hidden Technical Debt in Machine Learning Systems.

This table summarizes some of the core MLOps principles. Full table and text are available at ml-ops.org.

Where are we know and where should we seek to?

A clear set of tasks and rules cannot be defined due to the nature of software development in general but from my point of view, there are several things worth paying attention to.

Since a great percentage of ML applications codebases is not ML code, good practices for writing clean code should be put wherever possible. Efforts into drawing abstractions should be put, and common practices should evolve into patterns. Teams and organizations should be encouraged to share their experiences and knowledge obtained during research and development. Data Versioning, experiment management and reproducibility, sanity checks of data, pipelines, dependencies, and performance monitoring should be embodied in every ML project, regardless of its size, industry, or impact.

But, those processes and tasks wouldn't define and evolve by themselves, but indeed the participants in the process should start practicing them.

Starting from Data Scientists, more effort should be put into practice clean code, design patterns, and knowledge sharing. Educators (formal or informal) should focus more on creating resources that cover the topics discussed in this post. We are witnesses of Computer Science graduates with extraordinary Data Science skills, but still writing messy code and lacking fundamental knowledge about software engineering in general. There are extreme cases in which CS graduates are unfamiliar with version control systems and HTTP protocol. Because after all:

Every Data Scientist should be an engineer first. And every ML model is just another microservice within the boundaries of its system.

Where should you start from

In my opinion, every data scientist should have a concrete and solid knowledge of the fundamental concepts discussed in this post. I strongly recommend reading some books, starting with Introducing MLOps: How to Scale Machine Learning in the Enterprise, Practical MLOps: Operationalizing Machine Learning Models and related textbooks.

Exploring tools like Weights & Biases, Data Version Control, Tensor Board and others will give you a great start.

In machine learning, hyperparameter optimization or tuning is the problem of choosing a set of optimal hyperparameters for a learning algorithm. A hyperparameter is a parameter whose value is used to control the learning process. By contrast, the values of other parameters (typically node weights) are learned.

Strategies

Several strategies can be used for performing optimization. The most simple one is manual tuning. One such example is using the Elbow Method for determining the number of clusters in k-nearest neighbors algorithm. On the other hand complex models, have dozens of hyperparameters, and combined with the fact that some of them are continuous, the size of the search space explodes, so the manual effort. To tackle this issue, several other "smarter" approaches exist. Some of them are:

• Grid search
• Random search
• Bayesian optimization
• Gradient-based optimization
• Evolutionary optimization
• Population-based

To get more familiar with how these approaches work and what are the differences between them, I encourage you to read these articles 7 Hyperparameter Optimization Techniques Every Data Scientist Should Know, How To Make Deep Learning Models That Don’t Suck or Hyperparameter Optimization Approaches .

Tooling

Several frameworks provide implementations of the approaches mentioned above. In this tutorial, we are going to explore Weights & Biases - Sweeps, (WANDB for short).

Setup

For this tutorial, we are going to build a classifier for the Heart Disease UCI dataset. We will use RandomForestClassifier from sklearn to predict the presence of heart disease.

This tutorial does not focus on data pre-processing, so we'll dive straight into splitting the data into train and test data, and train the model once with default values for hyperparameters.

import pandas as pd
from sklearn.model_selection import train_test_split

df = pd.read_csv('data\heart.csv')

X = df.drop(['target'], axis=1)
y = df['target']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

Train the model

rfc = RandomForestClassifier()
rfc.fit(X_train, y_train)

and that's it, now we have successfully trained a random forest classifier with default values for its hyperparameters. This classifier has the following hyperparameters:

• bootstrap
• max_depth
• max_features
• min_samples_leaf
• min_samples_split
• n_estimators

Now let's get into setting up the optimization:

Optimization

Step 1: Define the training script

For more details, see the docs.

This script should serve as the main entry point for optimization. It performs one training and evaluation of the model with values for the hyperparameters injected from outside (through wandb.config).

It gets the configuration from the outside and performs training and evaluation of the model with fixed values for all of the hyperparameters. The name could be arbitrary, and for this example is train.py.

import wandb
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split

WANDB_PROJECT_NAME = "hyperparameter-optimization"

with wandb.init(project=WANDB_PROJECT_NAME):
    df = pd.read_csv('data\heart.csv')
    X = df.drop(['target'], axis=1)
    y = df['target']

    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

    config = wandb.config
    rfc = RandomForestClassifier(
        bootstrap=config.bootstrap,
        max_depth = config.max_depth,
        max_features = config.max_features,
        min_samples_leaf = config.min_samples_leaf,
        min_samples_split = config.min_samples_split,
        n_estimators = config.n_estimators,
    )

    rfc.fit(X_train, y_train)
    y_pred = rfc.predict(X_test)

    wandb.log({'accuracy': accuracy_score(y_test, y_pred)})

Step 2: Define the optimization strategy and configuration

To determine the optimization strategy, i.e. which approach to be used for optimization, what values or ranges should be tried for every hyper-parameter, and what objective to be optimized, a configuration file sweep.yml needs to be defined.

This file contains some additional configuration regarding the python path, training script path (from Step 1), to get more familiar see the docs.

For this example the sweep.yml file is.

program: train.py
method: bayes
project: hyperparameter-optimization
command:
- ${env} 
- ~/envs/hyperopt/bin/python
- ${program}
- ${args}

metric:
  name: accuracy
  goal: maximize
parameters:
  bootstrap:
    values: [True, False]
  max_depth:
    values: [2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, None]
  max_features:
    values: ['auto', 'sqrt']
  min_samples_leaf:
    values: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
  min_samples_split:
    values: [2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
  n_estimators:
    values: [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 200, 300, 500]

Interpretation is that the training script is located at train.py, Bayesian optimization is going to be used and for bootstrap the values True and `False is going to be tried, for max_depth [2, 3, 4, .... ] and similarly for all of the hyperparameters. The objective is maximizing accuracy. So in other words, we want to assign values to the hyperparameters such that the accuracy is maximized.

Step 3: Initializing and Running the optimization

To start optimization, open a shell (with your favorite terminal emulator).

1. Activate the python virtual environment (for UNIX source ~/envs/hyperopt/bin/python, for other see the official Python guide.)

2. Initialize the sweep

   wandb sweep .\sweep.yml
   

   Expected Output:

3. Run the sweep

   wandb agent aleksandar1932/hyperparameter-optimization/r3s5xf4d
   

   Expected Output:

And that's it, now the running sweep can be observed at WANDB.

Step 4: Monitoring

Go to the sweep URL, from your shell output. For this example, the output is available here, and bellow.

As the model is trained for different combinations for the hyperparameters, the results are updated in real-time. We can wait for the given approach to find the combination of values that when used for training, provide the best model, or stop the optimization either by terminating the running shell or through WANDB.

Conclusion

This sweep, performed 79 runs, and the best model scored 0.9016 accuracy on a randomly sampled test set for that particular run. (In train.py train_test_split is done for every run)

So, it can be concluded that the best RandomForestClassifier should be instantiated with the following hyperparameters.

    model = RandomForestClassifier(
        bootstrap=False,
        max_depth = 4,
        max_features = 'sqrt',
        min_samples_leaf = 3,
        min_samples_split = 4,
        n_estimators = 200,
    )

The code from this tutorial is available on GitHub.

Recently, at the international convention MIPRO 2021, I presented the full research paper titled "Leveraging Smartphones for Distributed Global Navigation Satellite System Post Processing". The research span for 6 months and I was the main author of the paper working together with my appreciated co-authors.

Brief walkthrough of the paper

After the conference proceedings, a published version of the paper will be available, so I would give you a brief walkthrough and introduction.

The main starting point of the research was the final course project for Parallel and Distributed Computing taught by Vladimir Zdraveski at FCSE. The starting point was Statista's analysis on the daily usage of smartphones which is 13% of the time. The next step was to find an applicable use case since we didn't want this research to solve an imagined problem.

The above-mentioned process defined our research hypothesis:

Can we take advantage to use smartphones while they are idle for GNSS data post-processing?

To prove the hypothesis we started architecting a system, so at the core we have:

• Clients - the producers of raw data,
• Post-processing provider - the entity which receives the raw data produced by the clients. Responsible for orchestrating the post-processing, and managing the pool of workers,
• Smartphone workers - managed devices that process batches of the raw data.

After implementing a proof-of-concept, we used it to show that the post-processing of GNSS data, can be significantly improved. In this case we have an improvement of 5317% compared with traditional post-processing providers, and 1595% improvement compared to in-house data post-processing.

Taking a step back

This blog post is not primary intended to showcase the research (the paper serves that purpose), but indeed to emphasize the word, term, process called research.

In my example, this started as a course project in which I put all of my focus, available knowledge and resources at the time being and resulted in one of my greatest achievements. This principle does not only apply to academia, instead it applies to every aspect of our life.

Tireless work, perseverance and getting into the smallest and most delicate detail or edge case are necessary to achieve top results in any field regardless of whether and what kind of recognition will be received for it. That way of working will make us leaders, not leaders in a company, organization, community, but leaders on our lives and our skillset.

How CodeChem fits into this story?

At the beginning of the research I had never heard of CodeChem, but during the time of my research I participated in Open Day 2021 and shortly after I started my internship. Alongside the learning process involved and the number of new concepts I was introduced, which rendered a never growing list, one of the things that fascinated me was the amount of research and effort for deep understanding put by the team for every new challenge that came along the way. Regardless of the project or the task, enormous thinking process and research is done by every member of the team, starting from a task estimated with couple of story points to a complex system being developed by several teams. That's why I am honored to be part of CodeChem.