Enter Iconify, a solution that streamlines the use of icons in web projects. Iconify offers a vast library of icons from various collections, simplifying their integration into projects. It enables dynamic generation of SVG icons, eliminating the need to download and store them manually. Through the use of APIs or client-side libraries, Iconify fetches and renders SVG icons on-demand based on the icons' identifiers.

While icons can be added to HTML through CSS or directly as SVG, using inline SVG provides greater flexibility. This method allows for complex interactions and styling, making icons more accessible and adaptable to the application's state, thereby improving user experience.

This article will show you how to set up Iconify along with TailwindCSS so that you can easily manage your icons. It assumed that you have set the tailwindcss node package and that you use this to generate your CSS file.

The Goal

If you are using the CSS method, you can easily use Iconify as follows.

<span class="icon-[mdi-light--home]"></span>
<span class="icon-[flowbite--archive-solid"></span>

To achieve this, install the @iconify/tailwind package, then open, import from iconify/tailwind, and add it to the list of plugins. Example:

const { addDynamicIconSelectors } = require('@iconify/tailwind');

/** @type {import('tailwindcss').Config} */
module.exports = {
   content: ['./src/*.html'],
   plugins: [
       require('@iconify-json/flowbite'),
       require('@iconify-json/mdi'),
       addDynamicIconSelectors(),
   ],
};

However, in this article, we will use the inline method. Essentially, what we want to achieve is the following:

<svg xmlns="http://www.w3.org/2000/svg"
     class="w-4 w-4 me-2"
     fill="currentColor"
     viewBox="0 0 20 20">
  {% include "icons/icon-[mdi-light--home]" %}
</svg>

Or, if you use Django Slippers like me, you can define a component.

{# icons.html #}
{% var size=size|match:"extra-small: h-3 w-3 , small: h-3 w-3, base: h-3.5 w-3.5 , large: h-4 w-4, extra-large: h-5 w-5"|default:"h-3.5 w-3.5" %}
{% var color=color|default:"" %}
{% var fill_color=fill_color|default:"currentColor" %}
{% var viewbox=viewbox|default:"0 0 24 24" %}
{% var extra_class=extra_class|default:"" %}

{% var icon_path="icons/"|add:name %}

<svg xmlns="http://www.w3.org/2000/svg"
     class="{{ size }} {{ color }} {{ extra_class }}"
     fill="{{ fill_color }}"
     viewBox="{{ viewbox }}">
  {% include icon_path %}
</svg>

And use it in your templates as follows.

{% icon name="icon-[flowbite--adjustments-horizontal-outline]"  %}
{% icon name="icon-[flowbite--face-grin-stars-outline]"  %}
{% icon name="icon-[flowbite--archive-solid]" size="small" %}
{% icon name="icon-[flowbite--annotation-solid]" %}

Install Necessary Packages

Locate your tailwind.config.js file and write the following functions.

import { readFileSync, mkdir, writeFileSync } from 'fs';
import { getIconData, matchIconName } from '@iconify/utils';

const TARGET = 'reux/templates/icons';
const CACHE = Object.create(null);


function loadIconSet(prefix) {
  let main = require.resolve(`@iconify-json/${prefix}/icons.json`);
  if (!main) {
    return;
  }

  if (CACHE[main]) {
    return CACHE[main];
  }

  try {
    const result = JSON.parse(readFileSync(main, 'utf8'));
    CACHE[main] = result;
    return result;
  }
  catch { }
}


function addDynamicIconSelectors(options) {
  const prefix = 'icon';
  const outDir = `${TARGET}`;

  mkdir(outDir, { recursive: true }, (err) => { if (err) throw err; });

  return (({ matchComponents }) => {
    matchComponents({
      [prefix]: (icon) => {
        // Get icon identifier
        const nameParts = icon.split(/--|\:/);

        if (nameParts.length !== 2) { 
            throw new Error(`Invalid icon name: "${icon}"`); 
        }

        // Get icon set prefix and icon name
        const [iconSetPrefix, name] = nameParts;
        if (!(iconSetPrefix.match(matchIconName) && name.match(matchIconName))) {
          throw new Error(`Invalid icon name: "${icon}"`);
        }

        // Load icon set
        const iconSet = loadIconSet(iconSetPrefix);
        if (!iconSet) {
          throw new Error(
              `Cannot load icon set for "${prefix}".
               Install "@iconify-json/${prefix}" as dev dependency?`
          );
        }

        // Get icon data
        const data =  getIconData(iconSet, name);

        // Save data to file
        writeFileSync(`${__dirname}/${outDir}/${prefix}-[${icon}]`, 
                      data['body']);
        return;
     }
    });
  })
}


/** @type {import('tailwindcss').Config} */
module.exports = {
  content: [
      './reux/templates/**/*.html',
  ],
  theme: {
    extend: {},
  },
  plugins: [
      require('flowbite/plugin'),
      require('@iconify-json/flowbite'),
      require('@iconify-json/mdi'),
      addDynamicIconSelectors(),
  ],
}

The addDynamicIconSelectors will scan the template files, and when it finds any mention of the text that matches our icon identifier (e.g., icon-[flowbite--annotation-solid]), it will read the icon set, get the SVG path data, and save it to the file, which we can then include using Django Template Language.

Conclusion

Incorporating inline SVG icons into our web application with Iconify and Tailwind CSS offers a powerful way to enhance the project's visual appeal and user interaction. By following the steps outlined above, you can easily manage and use a vast library of icons, providing a more flexible and accessible experience.

Requirements

• facebook_business. This is the official Facebook Python SDK. See its GitHub page.

• requests-oauthlib. Python OAuth library that we can use to authenticate to Facebook. For more info, see Requests-OAuthlib.

Before We Begin

To use Facebook API, we need to have App ID, App Secret and Access Token. The Access Token is something that you need to generate. But the App ID and App Secret are something you get from the Facebook app dashboard.

That means we need to create an app before we can start using the API. You can create an app from the Facebook developer website. Here is a short guide on how to create a new app.

Once you have created an application, you will have access to the App ID and the App Secret from the App Settings > Basic menu.

Access Token

To get an access token, we can use the requests_oauthlib

from requests_oauthlib.compliance_fixes import facebook_compliance_fix

my_app_id = "<app id>"
my_app_secret = "<app secret>"

authorization_base_url = "https://www.facebook.com/v18.0/dialog/oauth"
token_url = "https://graph.facebook.com/v18.0/oauth/access_token" 
redirect_uri = "https://localhost/" 

facebook = OAuth2Session(
    my_app_id,
    redirect_uri=redirect_uri,
    scope="ads_management,ads_read"
)
facebook = facebook_compliance_fix(facebook)

# Print the authorization URL. Access the URL using your browser.
authorization_url, state = facebook.authorization_url(authorization_base_url)
print(f"Please go here and authorize: {authorization_url}")

# Wait for the user to input the redirect URL
redirect_response = input("Paste the full redirect URL here:")

# Get the token
token = facebook.fetch_token(token_url, client_secret=my_app_secret,
                              authorization_response=redirect_response)

When you run the code above in a terminal, it will print out an authorization_url. And wait for you to paste the redirect URL. You then need to use your browser to open the link. Facebook will ask for your consent, and if everything goes well, you will be redirected to a blank page. Don't be alarmed. All you need to do now is copy the whole URL from your browser location bar and paste it into the terminal that is waiting. Once you are done, we can finally get the access token.

The token variable will have something like:

{'access_token': '<access_token>',
 'token_type': 'bearer',
 'expires_in': 5132946,
 'expires_at': 1703429997.4087522}

Now that we have all the needed items, we can continue playing around with the API.

Authorization and Authentication

Before making any API calls, we need to invoke the FacebookAdsApi.init. This method sets up a default FacebooksAdsApi object to be used.

from facebook_business.api import FacebookAdsApi

my_app_id = "<app_id>"
my_app_secret = "<app_secret>"
my_access_token = "<app_token>"

FacebookAdsApi.init(my_app_id, my_app_secret, my_access_token)

Getting Accounts

The first API we will try is getting the account. The simplest way is to use the User class with the fbid="me" parameters.

from facebook_business.adobjects.user import User

me = User(fbid="me")
my_accounts = list(me.get_ad_accounts(
    fields=["account_id", "id", "name"])
)

The my_accounts will have something like the following.

[<AdAccount> {
     "account_id": "<account id>",
     "id": "act_<account id>",
     "name": "Account 1"
 },
 <AdAccount> {
     "account_id": "<account id>",
     "id": "act_<account id>",
     "name": "Account 2"
 }]

Great. We just made our first API call.

A brief note: Although the method above is functional, it has a minor issue. It retrieves all accounts linked to your Facebook account. If you possess a lot of accounts, managing them could become challenging.

The alternative to this approach is to use a Business account. On Facebook, you can create a Business and establish an app associated with that business. This method makes it easier to manage, even if you have a lot of ad accounts. You can do the following to obtain accounts associated with a specific business.

from facebook_business.adobjects.application import Application
from facebook_business.api import FacebookAdsApi

FacebookAdsApi.init(business_id, my_app_secret, my_access_token)

app = Application(fbid=business_id)
my_accounts = list(app.get_authorized_ad_accounts(
    fields=["id", "account_id", "name"])
)

Getting Campaigns

Now that you get the account list. How do we get campaigns for all those accounts? Use the AdAccount class and invoke the get_campaigns method.

from facebook_business.adobjects.adaccount import AdAccount

campaign_list = []
for account in my_accounts:
    my_account = AdAccount(account["id"])
    campaigns = my_account.get_campaigns(
        fields=["id", "account_id", "name"]
    )
    campaign_list.extend(campaigns)

The campaign_list will look something like:

[<Campaign> {
    "account_id": "<account_id>",
    "id": "<campaign_id>",
    "name": "<campaign_name>"
}, ...]

Getting Campaigns Insight

To get campaign insight, we can run get_insights with the level set to the campaign. This will get the campaign cost in bulk. The alternative is to get the campaign list for the account, loop through each campaign, and fetch the insight data. This will deplete your API rate limit. But of course, we must be aware of the amount of data. We will be using yesterday as the date_preset in the example below, so this might not be a problem. But if the data gets big and causes time out, you need to switch to the async version (the SDK provides this but requires a bit more handling).

from facebook_business.adobjects.adaccount import AdAccount

insight_list = []
for account in my_accounts:
    my_account = AdAccount(account["id"])
    insight = my_account.get_insights(
        fields=["account_id", "account_name", 
                "campaign_id", "campaign_name",
                "date_start", "date_stop", "spend"],
        params={"date_preset":"yesterday ",   
                "level":"campaign"})
    insight_list.extend(insight)

For the above example, I only get the spend data. There are other data you can get, like impression, cpc, cpm, etc.

When we run the code above, the insight_list will look something like the following.

[[<AdsInsights> {
     "account_id": "<account id>",
     "account_name": "<account name>",
     "campaign_id": "<campaign id>",
     "campaign_name": "<campaign name>",
     "date_start": "<yesterday date>",
     "date_stop": "<today date>",
     "spend": "<the cost>"
 }, ...]

Summary

The Facebook Marketing API offers a powerful and efficient way to manage your ad campaigns programmatically. By using Python and the official SDK, you can easily authenticate, retrieve accounts, fetch campaign data, and gain insights into your campaign costs. With this foundational knowledge, you can now start integrating the API into your marketing strategies and harness its full potential to optimize your advertising efforts on Facebook.

The article you are about to read was originally published on GlassFlow Blog. GlassFlow provides serverless and production-ready streaming data pipelines setup for the data team.

Venturing into the world of data pipelines may feel daunting, especially when juggling several advanced technologies. Work with us as we create an analytics dashboard tailored for clickstream data, leveraging Python's Streamline, Apache Kafka, and Apache Pinot. Even if you're new to data pipelines, this comprehensive guide is here to help.

Overview - Build your first clickstream dashboard

By the end of this project, we will have built a simple dashboard that shows a funnel and Sankey chart. There are four components that we need to work on. First, we need to set up Apache Kafka and Pinot. Then, we will create some data to stream to our Kafka. Finally, we need to create a dashboard that visualizes the data we get from Pinot.

Tools I Used

• Apache Kafka. Apache Kafka is a distributed streaming platform that is used for building real-time data pipelines and streaming apps.

• Apache Pinot. Apache Pinot is an open-source, distributed data system designed for real-time analytics and suited for time series databases, OLAP queries, and real-time monitoring systems.

• Streamlit. Streamlit is an open-source Python library that allows developers to create dashboards featuring visualizations, widgets, and more.

• Plotly. Plotly is an open-source graphing library. We will use this to create the funnel and Sankey chart.

Clickstream Data

Clickstream data is the information collected when a user uses a web browser. Clickstream analytics is the process of tracking, analyzing, and reporting data on the pages a user visits and user behavior while on a webpage. This data can reveal much about user behavior, including how they found the product or service and their actions while using it.

The data for the project looks like this

To make it easy to process, we turn the timestamp (When They Visited column) into order. Essentially, we sort the timestamp and assign an integer from smallest to largest.

In the real world, the data is sent as the user navigates through the website. But in this project, we will read the data from a CSV file and send its content row by row. The data looks like the following.

user_id,web_page,order,location,visit
1,home,1,Colorado,1
1,login,2,Colorado,1
1,shop,3,Colorado,1
1,cart,4,Colorado,1
1,checkout,5,Colorado,1
2,home,1,California,1
2,login,2,California,1
2,shop,3,California,1
2,cart,4,California,1
2,checkout,5,California,1

You can also use Python random library or any mock tools (for example, Mockaroo) to generate this kind of data.

Setting Up Kafka and Pinot

Next, we need to set up Kafka and Pinot. However, manually setting up such complex systems can be daunting, even for experienced developers. You have to deal with complex dependencies, for example, using Zookeeper to manage distributed systems. Installing and configuring these dependencies can be time-consuming. You have to pay attention to many configurations. You need to make sure all components and their dependencies are correct. And so on.

Since the focus of this article is to give an end-to-end review of the whole system, let’s not get bogged down with a manual setup. Let’s use Docker-Compose. A side note: Docker-Compose is not meant for a production environment. It is primarily used in development because it is easy to manage multi-container applications like this project.

Docker-Compose offers a simplified approach. It will help streamline the setup process. I will provide two docker-compose files—one for Apple Silicon and the other for Intel-based machines.

Docker-Compose File for Apple Silicon

version: '3.7'
services:
  zookeeper:
    image: zookeeper:latest
    container_name: "zookeeper-clickstream"
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
  kafka:
    image: wurstmeister/kafka:latest
    restart: unless-stopped
    container_name: "kafka-clickstream"
    ports:
      - "9092:9092"
    expose:
      - "9093"
    depends_on:
      - zookeeper
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-clickstream:2181/kafka
      KAFKA_BROKER_ID: 0
      KAFKA_ADVERTISED_HOST_NAME: kafka-clickstream
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-clickstream:9093,OUTSIDE://localhost:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,OUTSIDE:PLAINTEXT
  pinot-controller:
    image: apachepinot/pinot:0.12.0-arm64
    command: "StartController -zkAddress zookeeper-clickstream:2181"
    container_name: "pinot-controller-clickstream"
    volumes:
      - ./config:/config
      - ./data:/data
    restart: unless-stopped
    ports:
      - "9000:9000"
    depends_on:
      - zookeeper
  pinot-broker:
    image: apachepinot/pinot:0.12.0-arm64
    command: "StartBroker -zkAddress zookeeper-clickstream:2181"
    restart: unless-stopped
    container_name: "pinot-broker-clickstream"
    volumes:
      - ./config:/config
    ports:
      - "8099:8099"
    depends_on:
      - pinot-controller
  pinot-server:
    image: apachepinot/pinot:0.12.0-arm64
    command: "StartServer -zkAddress zookeeper-clickstream:2181"
    restart: unless-stopped
    container_name: "pinot-server-clickstream"
    volumes:
      - ./config:/config
    ports:
      - "8098:8098"
      - "8097:8097"
    depends_on:
      - pinot-broker

Docker-Compose File for x86

version: '3.7'
services:
  zookeeper:
    image: zookeeper:3.5.6
    container_name: "zookeeper-clickstream"
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
  kafka:
    image: wurstmeister/kafka:latest
    restart: unless-stopped
    container_name: "kafka-clickstream"
    ports:
      - "9092:9092"
    expose:
      - "9093"
    depends_on:
      - zookeeper
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-clickstream:2181/kafka
      KAFKA_BROKER_ID: 0
      KAFKA_ADVERTISED_HOST_NAME: kafka-clickstream
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-clickstream:9093,OUTSIDE://localhost:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,OUTSIDE:PLAINTEXT
  pinot-controller:
    image: apachepinot/pinot:0.12.0
    command: "StartController -zkAddress zookeeper-clickstream:2181 -dataDir /data"
    container_name: "pinot-controller-clickstream"
    volumes:
      - ./config:/config
      - ./data:/data
    restart: unless-stopped
    ports:
      - "9000:9000"
    depends_on:
      - zookeeper
  pinot-broker:
    image: apachepinot/pinot:0.12.0
    command: "StartBroker -zkAddress zookeeper-clickstream:2181"
    restart: unless-stopped
    container_name: "pinot-broker-clickstream"
    volumes:
      - ./config:/config
    ports:
      - "8099:8099"      
    depends_on:
      - pinot-controller
  pinot-server:
    image: apachepinot/pinot:0.12.0
    command: "StartServer -zkAddress zookeeper-clickstream:2181"
    restart: unless-stopped
    container_name: "pinot-server-clickstream"
    volumes:
      - ./config:/config
    ports:
      - "8098:8098"
    depends_on:
      - pinot-broker

Spin Up Kafka and Pinot

Using the above docker-compose file, we can now spin both Kafka and Pinot up using the following command:

# For x86 machine
$ docker-compose up

# For Apple silicon   
$ docker-compose -f docker-compose-m1.yml up

Creating Topic

Before sending a bunch of data to Kafka, we must create a “topic.” A topic in Kafka is a category or name to which records are published. Think of it like a channel on a TV. If you want to watch a particular type of show or get specific news updates, you tune into a specific channel. Similarly, in Kafka, producers send records to topics, and consumers read records from topics.

The following command will create a topic named “clickstream_events” in Kafka.

$ docker exec -it kafka-clickstream kafka-topics.sh \\
  --bootstrap-server localhost:9092 \\
  --partitions 5 \\
  --topic clickstream_events \\
  --create

Add Pinot Table

Before Pinot can work, it must know how to save the data and connect to Kafka. To allow that, we must first define a schema.json and a table.json. Then, we need to tell Pinot to create the table following our schema and definition.

The schema will follow the data definition from the previous section. There will be user_id, web_page, order, location, and visit. We also add a timestamp column named ts. This timestamp column is vital in Pinot because it determines the time boundary and manages the life cycle of data stored in Pinot segments.

schema.json

{
    "schemaName": "clickstream",
    "dimensionFieldSpecs": [
      {
        "name": "user_id",
        "dataType": "INT"
      },
      {
        "name": "web_page",
        "dataType": "STRING"
      },
      {
        "name": "order",
        "dataType": "INT"
      },
      {
        "name": "location",
        "dataType": "STRING"
      },
      {
        "name": "visit",
        "dataType": "INT"
      }
    ],
    "dateTimeFieldSpecs": [
      {
        "name": "ts",
        "dataType": "TIMESTAMP",
        "format": "1:MILLISECONDS:EPOCH",
        "granularity": "1:MILLISECONDS"
      }
    ]
  }

To connect Pinot with Kafka, we need to create a table and define the connection configuration. This is so Pinot knows where to connect, what topic to read from, and other settings.

The complete table.json content is as follows. Notice the streamConfigs section where we define the stream type (Kafka), the broker (kafka-clickstream:9093), and the topic name (clickstream-events). This is how Pinot gets the data From Kafka.

table.json

{
    "tableName": "clickstream",
    "tableType": "REALTIME",
    "segmentsConfig": {
      "timeColumnName": "ts",
      "schemaName": "clickstream",
      "replicasPerPartition": "1"
    },
    "tenants": {},
    "tableIndexConfig": {
      "streamConfigs": {
        "streamType": "kafka",
        "stream.kafka.topic.name": "clickstream-events",
        "stream.kafka.broker.list": "kafka-clickstream:9093",
        "stream.kafka.consumer.type": "lowlevel",
        "stream.kafka.consumer.prop.auto.offset.reset": "smallest",
        "stream.kafka.consumer.factory.class.name": 
          "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
        "stream.kafka.decoder.class.name": 
          "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
          "realtime.segment.flush.threshold.rows": "1000",
          "realtime.segment.flush.threshold.time": "24h",
          "realtime.segment.flush.segment.size": "100M"
      }
    },
    "metadata": {},
    "ingestionConfig": {
      "transformConfigs": [
        {
          "columnName": "user_id",
          "transformFunction": "JSONPATH(meta, '$.user_id')"
        },
        {
          "columnName": "web_page",
          "transformFunction": "JSONPATH(meta, '$.web_page')"
        },
        {
          "columnName": "order",
          "transformFunction": "JSONPATH(meta, '$.order')"
        },
        {
          "columnName": "location",
          "transformFunction": "JSONPATH(meta, '$.location')"
        },
        {
          "columnName": "visit",
          "transformFunction": "JSONPATH(meta, '$.visit')"
        },
        {
            "columnName": "ts",
            "transformFunction": "\\"timestamp\\" * 1000"
        }
      ]
    }
  }

Now that we have defined our schema and table configuration, we only need to run the following command.

$ docker exec -it pinot-controller-wiki bin/pinot-admin.sh AddTable \\
  -tableConfigFile /config/table.json \\
  -schemaFile /config/schema.json \\
  -exec

Sending Data

Once we have set up Kafka and Pinot, it is time to send some data. As mentioned before, we will read the funnel data from a CSV and send every row to Kafka. Giving some random sleep between each transmission.

Our code that sends this data to Kafka is as follows.

import json
import time
import random
import csv

from confluent_kafka import Producer

def acked(err, msg):
    if err is not None:
        print(f"Failed to deliver message: {msg.value()}: {err.str()}")

producer = Producer({'bootstrap.servers': 'localhost:9092'})

with open('funnel_steps.csv', newline='') as csvfile:
    reader = csv.DictReader(csvfile)

    for row in reader:
        print(f'Sending payload: {row}')
        # Send to Kafka
        payload = json.dumps(row)
        producer.produce(topic='clickstream-events', key=str(row['user_id']),
                         value=payload, callback=acked)

        # Random sleep
        sleep_time = random.randint(1, 4)
        time.sleep(sleep_time)

Building the Dashboard

Now that the data is sent to Kafka and Pinot, we can display our dashboard. I use this Streamlit code to show a funnel and user flow data.

import streamlit as st
import pandas as pd
import plotly.graph_objects as go
from streamlit_autorefresh import st_autorefresh
import pinotdb

def get_funnel_figure(df):
    trace = go.Funnel(
        x=df.agg('sum', numeric_only=1).values,
        y=['home', 'login', 'cart', 'shop', 'help', 'error',
           'checkout', 'OLD_CHECKOUT']
    )

    layout = go.Layout(margin={"l": 180, "r": 0, "t": 30, "b": 0, "pad": 0},
                       funnelmode="stack",
                       showlegend=False,
                       hovermode='closest',
                       title='',
                       legend=dict(orientation="v",
                                   bgcolor='#E2E2E2',
                                   xanchor='left',
                                   font=dict(size=12)))
    fig = go.Figure(trace, layout)
    fig.update_layout(title_text="Funnel", font_size=10)
    return fig

def get_sankey_figure(df):
    # Process the data to capture transitions
    all_transitions = []

    for path in df['web_page']:
        steps = path.split(',')
        transitions = list(zip(steps[:-1], steps[1:]))
        all_transitions.extend(transitions)

    transition_df = pd.DataFrame(all_transitions, columns=['source', 'target'])
    trans_count = (transition_df.groupby(['source', 'target'])
                   .size()
                   .reset_index(name='value')
                   .sort_values('value', ascending=False))

    # Create unique labels for the nodes
    unique_labels = pd.concat([trans_count['source'],
                               trans_count['target']]).unique()

    # Map the source and target strings to numeric values
    trans_count['source'] = trans_count['source'].map(
        {label: idx for idx, label in enumerate(unique_labels)})
    trans_count['target'] = trans_count['target'].map(
        {label: idx for idx, label in enumerate(unique_labels)})

    # Create the Sankey diagram
    fig = go.Figure(go.Sankey(
        node=dict(pad=15, thickness=15,
                  line=dict(color="black", width=0.5),
                  label=unique_labels),
        link=dict(arrowlen=15,
                  source=trans_count['source'],
                  target=trans_count['target'],
                  value=trans_count['value'])
    ))

    fig.update_layout(title_text="User Flow", font_size=10)
    return fig

def get_connection():
    conn = pinotdb.connect(host='localhost', port=9000,
                           path='/sql', scheme='http')
    return conn

def get_funnel_data(conn):
    query = """SELECT
                SUM(case when web_page='home' then 1 else 0 end) as home,
                SUM(case when web_page='login' then 1 else 0 end) as login,
                SUM(case when web_page='cart' then 1 else 0 end) as cart,
                SUM(case when web_page='shop' then 1 else 0 end) as shop,
                SUM(case when web_page='help' then 1 else 0 end) as help,
                SUM(case when web_page='error' then 1 else 0 end) as error,
                SUM(case when web_page='checkout' then 1 else 0 end) as checkout,
                SUM(case when web_page='OLD_CHECKOUT' then 1 else 0 end) as OLD_CHECKOUT,
                location,
                user_id
                FROM clickstream
                GROUP BY location, user_id
                LIMIT 200
            """
    df = pd.read_sql_query(query, conn)
    return df

def get_sankey_data(conn):
    df = pd.read_sql_query('SELECT * FROM clickstream LIMIT 200', conn)
    df = (df.groupby(['location', 'user_id'])['web_page']
                    .apply(lambda x: ','.join(x))
                  .reset_index())
    return df

conn = get_connection()

# update every 30 seconds
st_autorefresh(interval=30 * 1000, key="dataframerefresh")

# Funnel Chart
funnel_data = get_funnel_data(conn)
funnel_fig = get_funnel_figure(funnel_data)
st.plotly_chart(funnel_fig, use_container_width=True)

# Sankey Chart
sankey_data = get_sankey_data(conn)
sankey_fig = get_sankey_figure(sankey_data)
st.plotly_chart(sankey_fig, use_container_width=True)

Result

Once everything is set up correctly, we can see the result.

Conclusion

After building your first clickstream dashboard you might think embarking on the data pipeline journey can be intimidating. With some tools that are available right now, such as Docker, Kafka, Pinot, and Streamlit, we can make this complex task manageable. Whether you're analyzing user behaviors, optimizing sales funnels, or just experimenting, these tools offer a scalable and efficient way to handle and visualize data.

Now, you might be wondering why we should go through all this trouble. Why not just batch process the data every day or every half a day? Why use streaming when you can use batching?

Modern applications need to act fast. Often up to the millisecond fast. With the sheer volume of data being produced at every moment, it is critical to act fast on the data before it becomes stale or obsolete. This continuous data processing offers advantages that can transform the way businesses run.

As businesses and technologies grow, the paradigms of data handling and processing are shifting. Being equipped with the right tools and understanding the significance of real-time processing could be the determining factor in staying ahead in the competitive landscape.

There are two types of byte order: big-endian and little-endian. For example, the network byte order is big-endian, with the most significant byte first, so a 16-bit integer with the value 1 would be the two hex bytes 00 01. However, the most common processors (x86/AMD64, ARM, RISC-V) are little-endian, with the least significant byte first - that same 1 would be 01 00.

When transmitting binary data over a network using sockets, it is essential to ensure that both the sender and receiver use the same byte order. The transmitted data may be corrupted or unreadable if the byte order is inconsistent. To ensure that both the sender and receiver are using the same byte order, a common byte order convention must be established.

We can use the struct module in Python to simplify handling binary data. The struct module converts between Python values and C structs represented as Python bytes objects.

The module provides functions for packing and unpacking binary data in a specific format. The first character of the format string can be used to indicate the byte order according to the following table:

If the first character is not of these, @ is assumed.

After the byte order indicator, we need to put in format characters. Some of the popular format characters are:

Here is an implementation example in Python that demonstrates how to pack and unpack binary data using the struct module:

```python
import socket
import struct

HOST = "localhost"
PORT = 5000


def serve():
    # Create a TCP/IP socket
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.bind((HOST, PORT))
    sock.listen(1)
    print("Server listening on {}:{}".format(HOST, PORT))

    # Wait for a connection
    conn, addr = sock.accept()
    print("Connected by ", addr)

    # Receive the data
    data = conn.recv(1024)
    print("Received: {!r}".format(data))

    # Unpack the binary data
    unpacked_data = struct.unpack("!Hf", data)
    print("Unpacked data: ", unpacked_data)

    conn.close()
    sock.close()


def client():
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.connect((HOST, PORT))

    # Pack the binary data
    data = struct.pack("!Hf", 123, 3.14)

    # Send the data
    sock.sendall(data)

    sock.close()
```

In this example, the server listens for incoming connections on localhost on port 5000. When a client connects, the server receives binary data from the client using the recv method of the socket. The server then unpacks the binary data using the struct.unpack function.

The client packs binary data in network byte order using the struct.pack function and sends it to the server using the sendall method of the socket.

Conclusion

In conclusion, handling binary data in socket programming requires attention to byte order, as different systems use different byte orders. To ensure compatibility, it is recommended to use network byte order, which is defined as big-endian byte order. The struct module in Python provides functions for packing and unpacking binary data in a specific format, making it easy to handle binary data in socket programming.

Fortunately, there are several methods you can use to handle message boundaries in socket programming. In this article, I will explore some of the most common methods and provide examples for each one.

Method 1: Fixed-length messages

One of the simplest ways to handle message boundaries is to send messages of a fixed length. This means that the sender and receiver agree on the length of each message, and the receiver knows exactly how many bytes to read in order to receive a complete message.

Here is an example of how you can send and receive fixed-length messages:

import socket

HOST = "localhost"
PORT = 12345
MSG_LEN = 10


def serve():
    # Initialize Socket
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.bind((HOST, PORT))
    sock.listen()
    print("Server listening on {}:{}".format(HOST, PORT))

    # Accept connection from outside
    conn, addr = sock.accept()
    print("Connected by", addr)

    # Receive the message data
    chunks = []
    bytes_recd = 0

    # recv only tell you how many bytes they handled.
    # So you need to keep recv until the message length is
    # correct.
    while bytes_recd < MSG_LEN:
        chunk = conn.recv(min(MSG_LEN - bytes_recd, 2048))
        if not chunk:
            raise RuntimeError("ERROR")
        chunks.append(chunk)
        bytes_recd += len(chunk)

    data = b"".join(chunks)

    # Print the message 
    message = data.decode("utf-8").strip()
    print("Received message: ", message)

    # Close connection and socket
    conn.close()
    sock.close()    


def client():
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.connect((HOST, PORT))

    message = input("Enter a message: ")

    # Truncate or pad the message to fit the fixed length
    message = message[:MSG_LEN].ljust(MSG_LEN)

    # Send the message data
    sock.sendall(message.encode("utf-8"))

    sock.close()

In this example, the server is set up to listen for incoming connections and receive messages of exactly 10 bytes. If the client sends a message with less than 10 bytes, we will pad the message with space. For this simplified version, if the message is larger than MSG_LEN, we simply truncate the message.

This method is pretty easy to understand and implement. However, the downside of this method is how inefficient it is when dealing with lots of small messages. Plus you still have to deal with data not fitting into one message.

Method 2: Delimiter-Separated Messages

Another common method for handling message boundaries is to use a delimiter to separate messages. This means that the sender and receiver agree on a special character or sequence of characters that will mark the end of each message. The only thing you need to pay attention to is when you allow multiple messages to be sent back to back. You may end up reading the start of the following message. You will need to hold onto it until it is needed.

Here is an example of how you can send and receive delimiter-separated messages using Python:

import socket

HOST = "localhost"
PORT = 12346
DELIMITER = b"\r\n"
BUFFER_SIZE = 4096


class Buffer(object):
    def __init__(self, sock):
        self.sock = sock
        self.buffer = b""

    def get_line(self):
        while b"\r\n" not in self.buffer:
            data = self.sock.recv(BUFFER_SIZE)
            if not data: # socket is closed
                return None
            self.buffer += data
        line, sep, self.buffer = self.buffer.partition(b"\r\n")
        return line.decode()


def serve():
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.bind((HOST, PORT))
    sock.listen()
    print(f"Server listening on {HOST}:{PORT}")

    conn, addr = sock.accept()
    print("Connected by", addr)

    buff = Buffer(conn)
    while True: 
        line = buff.get_line()
        if line is None:
            break
        print("Received message: ", line)

    conn.close()
    sock.close()


def client():
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.connect((HOST, PORT))

    messages = [b"Line One\r\n",
                b"Line ",
                b"Two\r\nLine",
                b" Three\r\n"]

    # Send the message data
    for message in messages:
       sock.sendall(message)

    sock.close()

In this example, the server is set up to listen for incoming connections and receive messages that are delimited by \r\n characters. The client sends four messages, and the server splits the incoming data into separate messages based on the delimiter.

Method 3: Message Length Header

A more advanced method for handling message boundaries is to include a message length header in each message. This means that the sender and receiver agree on a fixed number of bytes that will indicate the length of the message that follows.

In this method, we will use a message length header, which is a fixed-length field that indicates the length of the message to follow. The sender first sends the length of the message and then the actual message. The receiver reads the message length, reads the corresponding number of bytes from the socket, and then processes the message.

Here is an example of how you can send and receive messages with a length header using Python:

import socket

HOST = "localhost"
PORT = 12345
BUFFER_SIZE = 1024


def serve():
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.bind((HOST, PORT))
    sock.listen()
    print("Server listening on {}:{}".format(HOST, PORT))

    conn, addr = sock.accept()
    print("Connected by", addr)

    while True:
        # Receive the length
        header = conn.recv(4)
        if not header:
            break

        # Parse the header
        msg_len = int.from_bytes(header[0:4], byteorder="big")

        # Receive the message data
        chunks = []
        bytes_recd = 0
        while bytes_recd < msg_len:
            chunk = conn.recv(min(msg_len - bytes_recd,
                                  BUFFER_SIZE))
            if not chunk:
                raise RuntimeError("ERROR")
            chunks.append(chunk)
            bytes_recd += len(chunk)

        data = b"".join(chunks)

        # Print the message
        message = data.decode("utf-8").strip()
        print("Received message:", message)

    conn.close()
    sock.close()

def client():
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.connect((HOST, PORT))

    message = input('Enter a message: ')

    # Create the header
    msg_len = len(message.encode('utf-8'))
    header = msg_len.to_bytes(4, byteorder='big')

    # Send the header and message data
    sock.sendall(header + message.encode('utf-8'))

    sock.close()

In this example, the server is set up to listen for incoming connections and receive messages with a length header. The client sends two messages, each preceded by a 4-byte length header. The server first reads the message length header, unpacks it using the struct module, and then reads the corresponding number of bytes from the socket to get the message.

Warning

The above example has two recv, the first is to get the length, and the second one is in a loop to get the rest. But, since we are working with a network buffer, the first recv might not get all 4 characters in one recv. If you choose to ignore this, in high network loads your code will break. To be bulletproof, you need two recv loops, to first determine the length, and the second part to get the data.

Conclusion

Handling message boundaries in socket programming using Python can be challenging, but there are several methods you can use to solve this problem. Fixed-length messages, delimiter-separated messages, and message-length headers are some of the most common methods, each with its own advantages and disadvantages. By understanding these methods and how to implement them in Python, you can write robust and reliable network applications.

The need for a table that supports multiple actions by the user is becoming increasingly important, especially in today's fast-paced digital environment. Users expect to be able to perform actions on multiple rows in a single click, providing a more efficient and user-friendly experience. By implementing a table that supports multiple actions by the user, developers can improve their applications' overall usability and satisfaction, making it a necessary feature for modern web development.

In this article, we will discuss how to implement a table that supports bulk actions by the user using the combination of the django-tables2, htmx, and AlpineJS libraries. The django-tables2 package is a powerful library that allows developers to create and manage tables in Django projects with minimal effort. It provides a simple and flexible way to define the columns, data, and behavior of the table. The htmx and Alpine.js packages are lightweight JavaScript libraries that allow developers to perform rich interactions and dynamic updates on web pages without the need for extensive JavasScript code.

Together, these libraries provide a powerful solution for building tables without compromising performance, scalability, and maintainability.

The Requirements

Before we start, let’s write down what we want to build. The following are the rough specification

1. The table shall support pagination and sorting when the user clicks on the column header.

2. The table shall have a search query form that allows the user to filter the table. The pagination and sort shall be ignored when the user performs a search.

3. The table shall allow the user to select the row(s) and perform a specific action on those selected row(s).

   • Provide a checkbox column. The user shall be able to select individual rows by clicking on the checkbox of each row.

   • When the user clicks on the checkbox column on the table header, all the checkboxes in the rows on this page shall be selected.

   • The user shall be able to select multiple rows using the combination of the shift key and a mouse click on each row.

4. The table shall be rendered in a way that provides a seamless interaction.

   • No page refresh on any table actions

   • A progress bar to indicate the process of fetching data on all activities that requires access to the server (pagination, sorting, filtering, etc.)

   • Provide a highlight for rows that are recently updated.

The Packages Used For This Project

To build this table, we will need to use the following packages:

1. htmx. htmx gives access to using AJAX directly in HTML, using attributes. It is simple and, for a package this small, quite powerful.

2. django-tables2. This Django app lets you define tables like you define Django models. It can automatically generate a table based on a Django model. It supports pagination, column-based table sorting, custom column functionality via subclassing, and many other features.

3. django-filter. I use this package for the filtering functionality. It has APIs similar to Django's ModelForm and works well with django-tables2.

4. django-htmx. For htmx to work, Django view needs to tell which request is made using htmx and which is not. It has a middleware that adds htmx attribute to a request object.

5. Alpine-js. This project requires a more complex behavior, htmx alone will not be enough. Specifically, the table needs to make sorting, pagination, and filtering work together nicely. This is where Alpine-js comes in. This small javascript package allows me to store data and trigger action based on changes happening to a variable.

The Model

First, let's define a model to work with.

from django.db import models  


class Product(models.Model):  
    class Status(models.IntegerChoices):  
        ACTIVE = 1, "Active"  
        INACTIVE = 2, "Inactive"  
        ARCHIVED = 3, "Archived"  

    name = models.CharField(max_length=255)  
    category = models.CharField(max_length=255)  
    price = models.DecimalField(max_digits=10, decimal_places=2)  
    cost = models.DecimalField(max_digits=10, decimal_places=2)  
    status = models.PositiveSmallIntegerField(choices=Status.choices)  

    class Meta:
        ordering = ("pk",)  

    def __str__(self):  
        return self.name

The Table

Next, based on our model above, we are going to define the table. For this project, we need to:

• Add a checkbox column. We do this by using tables.CheckBoxColumn(accessor="pk", orderable=False).

• To make the checkbox column appear as the first column, we will utilize the sequence parameter on the table meta's class.

• We also need to add a class to our rows to indicate when rows are updated. We use this class to give some animation when a user updates rows. To achieve this, we will use the row_attrs parameter on the table meta's class. To get the information of the updated rows, we will add a new parameter for the table __init__ method named selection.

• Also, we will allow a user to select multiple rows using Shift Key and a mouse click. To achieve this, we are going to define an attribute attrs={td__input:{ "@click": "checkRange"}}. The AlpineJS package will handle the rest.

import django_tables2 as tables  

from .models import Product

def rows_higlighter(**kwargs):  
    # Add highlight class to rows 
    # when the product is recently updated.
    # Recently updated rows are in the table
    # selection parameter.  
    selected_rows = kwargs["table"].selected_rows  
    if selected_rows and kwargs["record"].pk in selected_rows:  
        return "highlight-me"  
    return ""


class ProductHTMxBulkActionTable(tables.Table):  
    # Add a checkbox column to the table.  
    selection = tables.CheckBoxColumn(accessor="pk", orderable=False,   
                                      attrs={  
                                        "td__input": {  
                                            "@click": "checkRange"  
                                        }  
                                      })  
    # Status is not orderable
    status = tables.Column(accessor="status", orderable=False)

    class Meta:  
        model = Product  
        template_name = "tables/bootstrap_htmx_bulkaction.html"
        show_header = False

        # This will put the checkbox column first.  
        sequence = ("selection", "...")  

        # This will add the highlight class to the rows  
        # when the product is recently updated.
        row_attrs = {  
            "class": rows_higlighter  
        }  

        # Additional class for easier styling.  
        attrs = {"class": "table checkcolumn-table"}  

    def __init__(self, selected_rows=None, *args, **kwargs):  
        super().__init__(*args, **kwargs)  

        # The selection parameter is a list of product ids  
        # that are recently updated.
        self.selected_rows = selected_rows 
        return

The Filter

The table supports a single query form so that the user can filter the table. We define the filter below.

from decimal import Decimal  

from django.db.models import Q  
from django.forms import TextInput  
import django_filters  

from .models import Product

# Custom widget that uses search input type
class SearchInput(TextInput):  
    input_type = "search"  


class ProductUniversalFilter(django_filters.FilterSet):  
    query = django_filters.CharFilter(  
        method="universal_search",  
        label="",  
        widget=SearchInput(attrs={"placeholder": "Search..."}),  
    )  

    class Meta:  
        model = Product  
        fields = ["query"]  

    def universal_search(self, queryset, name, value):  
        if value.replace(".", "", 1).isdigit():  
            value = Decimal(value)  
            return Product.objects.filter(Q(price=value) | Q(cost=value))  

        return Product.objects.filter(  
            Q(name__icontains=value) | Q(category__icontains=value)  
        )

I made a simple custom widget to change the input_type to search. The main basic differences come in the way browsers handle them. The first thing to note is that some browsers show a cross icon that can be clicked on to remove the search term instantly if desired.

The Views

Now that we have our model, table, and filter, we can start implementing our view. First, we need to implement a view that renders the table.

from django_tables2 import SingleTableMixin  
from django_filters.views import FilterView  

from .models import Product  
from .tables import ProductHTMxBulkActionTable
from .filters import ProductUniversalFilter


class ProductHTMxBulkActionView(SingleTableMixin, FilterView):  
    table_class = ProductHTMxBulkActionTable  
    queryset = Product.objects.all()  
    filterset_class = ProductUniversalFilter  
    paginate_by = 10  

    def get_template_names(self):  
        if self.request.htmx:  
            template_name = "products/product_table_partial.html"  
        else:  
            template_name = "products/product_table_bulkaction.html"  

        return template_name  

    def get_table_kwargs(self):  
        # Get the list of recently updated products.  
        # Pass the list to the table kwargs.
        kwargs = super().get_table_kwargs()  
        selected_rows = self.request.GET.get("selection", None)  
        if selected_rows:  
            selected_rows = [int(_) for _ in selected_rows.split(",")]  
            kwargs["selected_rows"] = selected_rows  

        return kwargs

In the get_template_names, we render different templates based on whether or not the request is made via htmx. Also, since we need to pass the selected rows to our table constructor, we need to implement the get_table_kwargs.

Next, we need a view to handle the bulk update request.

from django.http import HttpResponseRedirect 
from django.utils.http import urlencode  
from django.urls import reverse_lazy  


def reverse_querystring(view, urlconf=None, args=None, kwargs=None,
                        current_app=None, query_kwargs=None):  
    '''Custom reverse to handle query strings.  
    Usage:
        reverse('app.views.my_view', kwargs={'pk': 123}, 
                 query_kwargs={'search': 'Bob'})
    '''
    base_url = reverse_lazy(view, urlconf=urlconf, args=args, 
                            kwargs=kwargs, current_app=current_app)  
    if query_kwargs:  
        return '{}?{}'.format(base_url, urlencode(query_kwargs))  
    return base_url


def response_updateview(request):  
    if request.method == "POST" and request.htmx:  
        # Get the selected products  
        selected_products = request.POST.getlist("selection")  

        # Check if the activate/deactivate button is pressed  
        if request.htmx.trigger_name == "activate":  
            # Activate the selected products              
            Product.objects
                   .filter(pk__in=selected_products)
                   .update(status=Product.Status.ACTIVE)  
        elif request.htmx.trigger_name == "deactivate":  
            # Deactivate the selected products  
            Product.objects
                   .filter(pk__in=selected_products)
                   .update(status=Product.Status.INACTIVE)  

        # Get the page number  
        page = request.POST.get("page", 1)  
        page = int(page)  

        # Get the sort by column  
        sort_by = request.POST.get("sort", None)

        # Get the query  
        query = request.POST.get("query", "")  

        # Get selection  
        selection = ",".join(selected_products)  

    # Redirect to table view 
    return HttpResponseRedirect(  
        reverse_querystring("products:products_htmx_bulkaction",   
                            query_kwargs={"page": page, "sort": sort_by,  
                                          "query": query,
                                          "selection": selection}))

In this view, we first check what action is triggered (activate/deactivate) and update our data. Finally, we redirect to our table view, including all the needed parameters like pagination, sort, query/filter, and the selected row.

The Templates

There are three templates in play here:

1. A template to render the whole page. Where I generate the page with an HTML header and body and the table.

2. A template to render only the table. When we perform some action like sorting or filtering, we only need to render the partial page.

3. Custom table template to override the default. I need to remove the table definition. And add some Alpine-js attributes for the pagination to work.

Template to Render the Entire Page

For this project, I put the table header and query form on this page. This template has a lot of functionality.

1. Bulk activate or de-active products. To achieve this, we need to include all information like pagination, sorting, and filtering. We accomplish this using hx-include="#bulk-actions, #id_query".

2. For searching, we included a simple search form. There are two ways to trigger the search functionality. The first is when the user types something in the search box. We delay the trigger for 500ms, and the trigger only happens when the content is changed. The second trigger is when the user clicks on the cross icon on the search box. We achieve this by using hx-trigger="keyup changed delay:500ms, search". Also, when we search, we will remove the pagination and sorting information. To achieve this, we first need to define x-data and trigger the custom clear-pagination-and-sort event before every request. To accomplish this functionality, we use x-on:htmx:before-request="$dispatch('clear-pagination-and-sort'). The custom event is defined later via the @clear-pagination-and-sort.window functionality.

3. Pagination and sorting. I define two extra hidden input fields. The sort and page input. When a user performs an action, be it sorting, jumping between pages, or filtering, I will submit all three pieces of information back to the server.

4. We also added functionality in the check box column, where the whole rows in that page got selected upon clicking. We achieved this using @click="toggleSelection()". Inside the toggleSelection we simply checked all rows on that page.

5. To support multiple rows selection when the user uses Shift Key and mouse click, I added the checkRange function at the bottom.

{% extends "products/base.html" %}

{% load render_table from django_tables2 %}
{% load i18n %}
{% load django_tables2 %}

{% block bulkaction_table %}active{% endblock %}

{% block table_main %}
<h1>Product Table</h1>

<div class="table-top-container">

  {# Bulk actions and search bar #}
  <div class="row justify-content-between">
    <div class="col-4">
      <div>
          <button id="activate" type="submit" name="activate"
                  class="btn btn-secondary"
                  hx-post="{% url 'tables:products_htmx_bulkaction_update' %}"
                  hx-target=".table-container"
                  hx-indicator=".progress"
                  hx-include="#bulk-actions, #id_query">
          Activate
        </button>
        <button id="deactivate" type="submit" name="deactivate"
                class="btn btn-secondary"
                hx-post="{% url 'tables:products_htmx_bulkaction_update' %}"
                hx-target=".table-container"
                hx-indicator=".progress"
                hx-include="#bulk-actions, #id_query">
        Deactivate
        </button>
      </div>
    </div>
    <div class="col-4">
      <div class="form-inline">
        <div class="d-flex justify-content-end">
          <input type="search" name="query" placeholder="Search..."
                 class="searchinput form-control" id="id_query"
                 hx-trigger="keyup changed delay:500ms, search"
                 hx-get="{% url 'tables:products_htmx_bulkaction' %}"
                 hx-indicator=".progress"
                 hx-target=".table-container"
                 x-data
                 x-on:htmx:before-request="$dispatch('clear-pagination-and-sort')">
        </div>
      </div>
    </div>
  </div>

  {# Table header #}
  <form id="bulk-actions"
        hx-get="{% url 'tables:products_htmx_bulkaction' %}"
        hx-target=".table-container"
        hx-trigger="sort-initiated, pagination-initiated"
        hx-swap="outerHTML"
        hx-include="#id_query"
        hx-indicator=".progress"
        x-data="{ sort_by: '',
                  page_by: 1,
                  select_all: false,
                  last_checked: false }"
        @clear-pagination-and-sort.window="page_by = 1; sort_by = ''"
        x-on:htmx:after-swap="select_all = false">

    {% csrf_token %}

    {# Hidden input to store pagination page and column to sort by #}
    <input type="hidden" name="sort" x-ref="sort_input" x-model="sort_by"
           x-init="$watch('sort_by',
                           () => $refs.sort_input.dispatchEvent(
                                    new Event('sort-initiated',
                                              { bubbles: true })))">

    <input type="hidden" name="page" x-ref="paginate_input" x-model="page_by"
           x-init="$watch('page_by',
                           () => $refs.paginate_input.dispatchEvent(
                                    new Event('pagination-initiated',
                                              { bubbles: true })))">

    <table class="table checkcolumn-table header">
      <thead {{ table.attrs.thead.as_html }}>
        <tr>
          {% for column in table.columns %}
            {% if column.name == 'selection' %}
            <th {{ column.attrs.th.as_html }}
                x-data="{ toggleSelection(event) {
                  select_all = !select_all;
                  let checkboxes = document.getElementsByName('selection');
                  [...checkboxes].map((el) => {
                    el.checked = select_all;
                  })
                }
              }"
              @click="toggleSelection()"
              style="cursor: pointer;">
              <input type="checkbox" x-model="select_all">
            </th>
            {% else %}
              {% if column.orderable %}
                <th {{ column.attrs.th.as_html }}
                    x-data="{ col_name: '{{ column.order_by_alias }}',
                    toggleSort(event) {
                      this.col_name = this.col_name.startsWith('-') ? this.col_name.substring(1) : ('-' + this.col_name);
                      sort_by = this.col_name;
                    }
                  }"
                  @click="toggleSort()"
                  :class="sort_by !== '' ? (sort_by === col_name ? (sort_by.startsWith('-') ? 'desc' : 'asc') : '') : ''"
                  style="cursor: pointer;">
                  {{ column.header }}
                </th>
              {% else %}
                <th {{ column.attrs.th.as_html }}>{{ column.header }}</th>
              {% endif %}
            {% endif %}
          {% endfor %}
        </tr>
      </thead>
    </table>

    {# Progress indicator #}
    <div class="progress">
      <div class="indeterminate"></div>
    </div>

    {# Content table #}
    {% render_table table %}
  </form>
</div>
{% endblock %}

{% block footer %}
<script>
    // Set the checkbox to be checked from the start 
    // to end when the user presses the shift key.
  function checkRange(event) {
    let checkboxes = document.getElementsByName('selection');
    let inBetween =  false;
    if( event.shiftKey && event.target.checked ) {
      checkboxes.forEach( checkbox => {
        if( checkbox === event.target || checkbox === last_checked ) {
          inBetween = !inBetween;
        }
        if( inBetween ) {
          checkbox.checked = true;
        }
      });
    }
    last_checked = event.target;
  }
</script>
{% endblock %}

Template to Render Table

Here we only render the table content, no HTML header, and no HTML body. Just the content.

{# templates/products/product_table_partial.html #}
{% load render_table from django_tables2 %}

{% render_table table %}

Template to Override the Default Table

This template extends the default table from django-tables2. It seamlessly updates the page_by variable in x-data when a user triggers any pagination action. This, in turn, updates the value of the page input, which is linked via the x-model. And it starts a custom event called pagination-initiated, which tells htmx to send the complete form back to the server, including all sorting, pagination, and filtering information. It's like magic!

{% extends "django_tables2/bootstrap4.html" %}  

{% load django_tables2 %}  
{% load i18n %}  

{% block pagination.previous %}  
<li class="previous page-item" role="button">  
    <div @click="page_by = {{table.page.previous_page_number}}"  
         class="page-link">  
        <span aria-hidden="true">&laquo;</span>  
        {% trans 'previous' %}  
    </div>  
</li>  
{% endblock pagination.previous %}  

{% block pagination.range %}  
{% for p in table.page|table_page_range:table.paginator %}  
<li class="page-item{% if table.page.number == p %} active{% endif %}" role="button">  
    <div class="page-link" {% if p != '...' %}@click="page_by={{p}}"{% endif %}>  
        {{ p }}  
    </div>  
</li>  
{% endfor %}  
{% endblock pagination.range %}  

{% block pagination.next %}  
<li class="next page-item user-select" role="button">  
    <div @click="page_by = {{table.page.next_page_number}}" class="page-link">  
        {% trans 'next' %}  
        <span aria-hidden="true">&raquo;</span>  
    </div></li>  
{% endblock pagination.next %}

Style

Column Width

/* Column Width */
.checkcolumn-table td:nth-child(1),  
.checkcolumn-table th:nth-child(1) {  
    width: 1%;  
}  

.checkcolumn-table td:nth-child(2),  
.checkcolumn-table th:nth-child(2) {  
    width: 8%;  
}  

.checkcolumn-table td:nth-child(3),  
.checkcolumn-table th:nth-child(3) {  
    width: 34%;  
}  

.checkcolumn-table td:nth-child(4),  
.checkcolumn-table th:nth-child(4) {  
    width: 32%;  
}  

.checkcolumn-table td:nth-child(5),  
.checkcolumn-table td:nth-child(6),
.checkcolumn-table td:nth-child(7),  
.checkcolumn-table th:nth-child(5),  
.checkcolumn-table th:nth-child(6),  
.checkcolumn-table th:nth-child(7) {  
    width: 8%;  
}

Pagination

/* Pagination */
ul.pagination {  
  justify-content: end !important;  
}

Sorting

/* Sorting */
th.asc:after {  
    content: '\0000a0\0025b2';  
    float: right;  
    width: 10%;  
}  

th.desc:after {  
    content: '\0000a0\0025bc';  
    float: right;  
    width: 10%;  
}

Highlight Rows on Update

/* Rows highlight */
.highlight-me {  
  background-color: white;  
  animation-name: blink;  
  animation-duration: 2s;  
  transition-timing-function: ease-in;  
  transition: 0.2s;  
} 

@keyframes blink {  
  0% { background-color: orange; color: white;}  
  50% { background-color: orange; color: white; }  
  51% { background-color: white; }  
  100% { background-color: white; }  
}

Progress Bar

/* Progress bar */  
.checkcolumn-table.header {  
    margin-bottom: 0;  
}

.progress {  
    height: 4px;  
    width: 100%;  
    border-radius: 2px;  
    background-clip: padding-box;  
    overflow: hidden;  
    position: relative;  
    opacity: 0;  
}  

.htmx-request .progress,
.htmx-request.progress {  
    opacity: 1;  
}  

.progress .indeterminate {  
    background-color: blue;  
}  

.progress .indeterminate:after,  
.progress .indeterminate:before {  
    content: '';  
    position: absolute;  
    background-color: inherit;  
    top: 0;  
    left: 0;  
    bottom: 0;  
    will-change: left, right;  
}  

.progress .indeterminate:before {  
    animation: indeterminate 2.1s cubic-bezier(0.65, 0.815, 0.735, 0.395) infinite;  
}  

.progress .indeterminate:after {  
    animation: indeterminate-short 2.1s cubic-bezier(0.165, 0.84, 0.44, 1) infinite;  
}

@keyframes indeterminate {  
    0% { left: -35%; right: 100%; }  
    60% { left: 100%; right: -90%; }  
    100% {  left: 100%;  right: -90%; }  
}  

@keyframes indeterminate-short {  
    0% { left: -200%; right: 100%; }  
    60% { left: 107%; right: -8%; }  
    100% { left: 107%; right: -8%; }  
}

Summary

In this article, we have covered the process of building a table that supports bulk actions by the user using the combination of django-tables2, htmx, and AlpineJS. We have gone through the implementation of pagination and sorting, the search query form, and the checkbox column. We also covered how to enhance the user experience by improving the table's design and layout, adding responsive design, and providing feedback to the user.

We have seen how django-tables2 is a powerful library that allows developers to create and manage tables in Django projects with minimal effort. The htmx allows developers to perform rich interactions and dynamic updates on web pages without the need for extensive JavaScript code, and Alpine.js will enable developers to add interactive functionality to their web pages with minimal code. Together, these libraries provide a powerful solution for building tables that support bulk actions by the user.

I tried out miniconda as an alternative to anaconda. To do that, I need to uninstall my anaconda.

$ conda install anaconda-clean
$ anaconda-clean --yes
$ rm -rf ~/opt/anaconda3

Then I just installed miniconda for my Apple Silicon. I use the graphical installer.

This project is inspired by a case study made by Fantasy.co. You can still find the project description at the Fantasy.co website.

The Packages Used For This Project

• Mapbox GL JS. Mapbox GL JS is a JavaScript library for vector maps on the Web.
• turf. Turf is an advanced geospatial javascript tool that is simple to use and runs very fast.

File setup

For this simple project, we only need to have three files. An index.html, a style.css and a map.js. The content of index.html and style.css are detailed here. I will talk about the content of map.js in more detail later.

index.html

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width">
  <title>Animated </title>
  <link href="https://api.mapbox.com/mapbox-gl-js/v2.10.0/mapbox-gl.css" rel="stylesheet">
  <script src="https://api.mapbox.com/mapbox-gl-js/v2.10.0/mapbox-gl.js"></script>
  <script src="https://unpkg.com/@turf/turf@6/turf.min.js"></script>
  <link rel="stylesheet" href="style.css">
</head>
<body>
    <div id='map'></div>        
    <script src="map.js"></script>
</body>
</html>

style.css

#map {
    width: 600px;
    height: 600px;
}

Now that we have the index.html and style.css let's move on to the juicy stuff.

Making Semi-Circle Path

First, let's make the semi-circle path. There are many ways to make a path between two points. But, in this article, I created a number of new coordinates between the start and end points. Creating more points is also helpful since we will animate a marker using these new coordinates later. More points will generate a smoother animation.

The idea of the process is as follows:

1. Get the midpoint between origin and destination. Also, get the bearing and radius of the semi-circle.
2. Translate that midpoint based on the bearing (modified by some angle) and the radius.
3. Repeat this process multiple times. In this article, I make 1000 additional coordinates between the origin and destination. More coordinates mean smoother lines and smoother animation.

By the end of this process, we will get a list of new coordinates.

And the function to generate this is as follows.

// Generate arc from start point to the end point.
// The segments parameter tell us how many
// new point should we generate.
function generateArc(start, end, segments) {
    // Get the mid point between start and end
    let midPoint = turf.midpoint(start, end);

    // Get the bearing 
    let bearing = turf.bearing(end, start);

    // Get half of the distance, because we 
    // start from the midpoint.
    let distance = turf.distance(start, end) / 2;

    // Add the start coordinate
    let arc = [start.geometry.coordinates];

    // We devide 180 angle by segments, and for each angle
    // we transform the mid point by distance and an angle.
    for (let angle = 0; angle < 180; angle += (180/ (segments))) {
        let rotatedPoint = turf.transformTranslate(midPoint,
                                                   distance,
                                                   bearing - angle);
        arc.push(rotatedPoint.geometry.coordinates);
    }

    // Add the last coordinate
    arc.push(end.geometry.coordinates);

    return arc
}

Animate a Marker

Once we created a bunch of new coordinates. We can start moving a point or a marker between these coordinates.

So for every coordinates pair we created from the previous step, we will:

1. rotate our marker using turf.bearing .
2. move the marker using mapbox.getSource().setData()

In the reference, the animation slows down halfway. So I tried using some ease-out functions from https://easings.net/, but the result was not quite what I wanted. So I decided to simply slow down the animation when the marker reached halfway.

function animate(map, counter, route, point, source_id, segments) {

    const start = 
        route.features[0].geometry.coordinates[
            counter >= segments ? counter - 1 : counter
        ];
    const end =
        route.features[0].geometry.coordinates[
            counter >= segments ? counter : counter + 1
        ];

    if (!start || !end) return;

    // Update point geometry to a new position 
    // based on counter denoting the index to access the arc.
    point.features[0].geometry.coordinates =
        route.features[0].geometry.coordinates[counter];

    // Calculate the bearing to ensure the icon is 
    // rotated to match the route arc.
    point.features[0].properties.bearing = turf.bearing(
        turf.point(start),
        turf.point(end)
    );

    // Update the source with this new data
    map.getSource('point-' + source_id).setData(point);

    // Request the next frame of animation 
    if (counter < (segments/2)) {
        // Before halfway, just run the animation 
        // as normal.
        requestAnimationFrame(function() {
            animate(map, counter, route, point, source_id, segments);
        });
    } else {
        // Slow down the animation after half way.
        let fps = 20
        setTimeout(() => {
            requestAnimationFrame(function() {
                animate(map, counter, route, point, source_id, segments);
            });
          }, 1000 / fps);
    }
    counter += 1;
}

Zoom Level

Another thing I like to do is to set a proper zoom level. For this, I utilize the fitBounds functionality. To correctly calculate the bounding box, I should probably add the rotated midpoint. But here, I only use the original city coordinates to get the bounding box. And added some padding when I invoked the fitBounds().

let bbox = turf.bbox(turf.points(cityCoordinates));
map.fitBounds(bbox, {
    linear: true,
    easing: function(x){
        return x * x * x;
    },
    padding: {top: 100, bottom:100, left: 100, right: 300}
});

All together now

With all of the above, we can write our map.js as follow.

map.js

// Set origin, destination, and route
//

// Cities
const cities = {
    "JKT": [106.8227, -6.208763],
    "SG": [103.8198, 1.3521],
    "KL": [101.693207, 3.140853]
}

var cityCoordinates = Object.keys(cities).map(function(key){
    return cities[key];
});

// Route
const routes = [
    [cities["JKT"], cities["SG"]],
    [cities["SG"], cities["KL"]],
];

// Map setup
//
mapboxgl.accessToken = '<USE YOUR TOKEN>';

// Get the center coordinate based on the city coordinates
let center = turf.center(
                turf.points(cityCoordinates)
             ).geometry.coordinates;

// Setup map
var map = new mapboxgl.Map({
    container: 'map',
    style: 'mapbox://styles/mapbox/dark-v10',
    center: center,
    zoom: 5
});

// Add navigation control
map.addControl(new mapboxgl.NavigationControl());

// Load the map
//
map.on('load', () => {

    // Set zoom level.
    let bbox = turf.bbox(turf.points(cityCoordinates));
    map.fitBounds(bbox, {
        linear: true,
        easing: function(x){
            return x * x * x;
        },
        padding: {top: 100, bottom:100, left: 100, right: 300}
    });

    // Number of segment to use in the arc.
    // More segment means a smoothe arc.
    const segments = 1000;

    for (let i = 0; i < routes.length; i++) {
        // Route
        //
        let route = routes[i];

        // Create the line segments based on the route
        let arc = generateArc(turf.point(route[0]), 
                              turf.point(route[1]),
                              segments);

        // Turn the object into feature collection
        let mapRoute = turf.featureCollection([turf.lineString(arc)]);

        // Put the information on to the map
        map.addSource("route-" + i, {
            "type": "geojson",
            "data": mapRoute
        });

        map.addLayer({
            "id": "route-layer-" + i,
            "type": "line",
            "source": "route-" + i,
            "layout": {
                "line-join": "round",
                "line-cap": "round"
            },
            "paint": {
            "line-color": "#fff",
            "line-width": 1
            }
        });

        // Point
        //

        let mapPoint = turf.featureCollection([turf.point(routes[i][0])]);
        map.addSource('point-' + i, {
            'type': 'geojson',
            'data': mapPoint
        });

        map.addLayer({
            'id': 'point-layer-' + i,
            'source': 'point-' + i,
            'type': 'symbol',
            'layout': {
                'icon-image': 'airport-15',
                'icon-size': 1,
                'icon-rotate': ['get', 'bearing'],
                'icon-rotation-alignment': 'map',
                'icon-allow-overlap': true,
                'icon-ignore-placement': true
            }
        });

        // Animate
        animate(map, 0, mapRoute, mapPoint, i, segments);
    }
});

function generateArc(start, end, segments) {
    // See the above
    // ...
}

function animate(map, counter, route, point, source_id, segments) {
    // See the above
    // ...
}

And when you open the index.html in your browser, you will get the following.

Conclusion

In this article, I show how to create a semi-circle arc between two points using MapBox. I also demonstrated how you could move a point or a marker along these routes. I hope you can learn something from this. Thank you for reading!

The Python socket module provides a straightforward interface to the Berkeley sockets API. But it uses Python’s object-oriented style. The socket() function returns a socket object whose methods implement the various socket system calls.

The primary socket API functions are:

• socket()

• .bind()

• .listen()

• .accept()

• .connect()

• .send()

• .recv()

• .close()

Server socket vs client socket

Sockets can mean subtly different things depending on context. There is a difference between a client socket and a server socket.

A client socket is an endpoint of a conversation. It usually is only used for one exchange or a small set of sequential exchanges, and that is it.

A server socket on the other hand is more like a switchboard operator. It doesn't send or receive any data. It only produces client sockets. Each client socket is created in response to some other client socket doing connect() to the server. Once a client socket is created, the two clients are free to communicate with each other.

An example of this distinction can be found on your browser. The web browser you operate uses client sockets exclusively. The web server that handles requests from the client uses both server sockets and client sockets.

Echo client and server

Below is the minimum example code. A server that echoes all data it receives back (servicing only one client) and a client that sends data.

Echo Server

import socket

QUEUE_SIZE = 5
BUFFER_SIZE = 4096

def serve(host, port):
    # Create a server socket
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    # Bind socket to host and port
    sock.bind((host, port))

    # Officially become a server socket
    sock.listen(QUEUE_SIZE)

    # Accept connection from outside
    client_socket, addr = sock.accept()

    while True:
        data = client_socket.recv(BUFFER_SIZE)
        if not data:
            break
        client_socket.sendall(data)

    # Close the client socket
    client_socket.close()    

    # Close the server socket
    sock.close()

A server must perform the sequence socket(), bind(), listen(), accept(). The server does not send(), sendall() or recv() on the socket it is listening on but on the new socket returned by accept().

The socket() function takes two parameters: address family and socket types. The address family is usually IPv4 (socket.AF_INET) or IPv6 (socket.AF_INET6). Socket types are usually streaming sockets (socket.SOCK_STREAM) or datagram sockets (socket.SOCK_DGRAM).

The value passed to bind() depends on the socket's address family. For socket.AF_INET (IPv4), it expects a (host, port) tuple. host can be a hostname, IP address, or an empty string. The IP address 127.0.0.1 is the standard IPv4 address for the loopback interface, so only processes on the host will be able to connect to the server. If you pass an empty string, the server will accept connections on all available IPv4 interfaces.

Socket listen() can take a parameter called backlog. When a backlog is specified, it tells the system to allow a number of unaccepted connections before refusing new connections. A good number for a backlog is 5 (the typical max). If you write your code correctly, five should be plenty.

Echo Client

import socket

BUFFER_SIZE = 4096


def connect(host, port, message):
    # Create a client socket
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    # Connect to a server
    sock.connect((host, port))

    # Send message to the server
    sock.sendall(bytes(message, "utf-8"))

    # Capture any information back 
    # from the server
    data = sock.recv(BUFFER_SIZE)

    # Close the connection
    sock.close()

Notes on recv() and send()

send and recv operate on network buffers. They do NOT necessarily handle all the bytes you hand them (or expect from them). They return when the network buffers have been filled (send) or emptied (recv). They then tell you how many bytes they handled. It is your responsibility to call them again until your message has been completely dealt with.

The recv() method receive data from the socket. It returns the number of bytes received, which may be less than the size of the data. It requires the bufsize parameters to be specified. It is the maximum amount of data to be received at once. As for the value of the bufsize, the documentation recommended the following:

For best match with hardware and network realities, the value of bufsize should be a relatively small power of 2, for example, 4096.

Typically we set this to 4096 or 8192.

The other side has closed the connection when a recv returns 0 bytes. You will NOT receive any more data from this connection.

The send() method also sends the number of bytes sent, which may be less than the size of the data passed in. To ensure you send all the data, use the sendall() method instead.

Conclusion

As we have seen, sockets are the fundamental building blocks of network communications. By learning how to use them in Python, we can easily add this powerful tool to our arsenal. With a little practice, you'll be able to build robust client-server applications that can communicate across networks. Thanks for reading!

Sockets are simply an endpoint for communication, similar to how we use ports on our phones for sending and receiving data across the network. They are used nearly everywhere, but as a technology, they often need to be understood.

Brief History

Before we start, let's take a brief look at the short history of the socket.

• 1971. The term socket appeared in the publication of RPC 147 when it was used in the ARPANET.
• 1983. Most modern implementations of sockets are based on Berkeley sockets.
• 1989. UC Berkeley released versions of the network library free from the licensing constraints of AT&T copyright-protected Unix.
• 1991. Windows define its network API called WinSock.

Socket Types

There are several types of sockets, but only two appear to be useful. They are stream sockets and datagram sockets.

Stream sockets are reliable two-way connected communication streams. For instance, your web browsers use HTTP, which uses stream sockets to get pages. They are reliable because they use the Transmission Control Protocol (TCP) protocol. TCP makes sure data arrives sequentially and error-free.

Datagram sockets use the User Datagram Protocol (UDP). If you use datagram sockets to send data, it may or may not arrive. It may come out of order. If it arrives, though, the data within the packet will be error-free. Datagram sockets are typically used in a system where you can ignore the dropped packets—applications like games, audio, or video, for instance.

Socket Abstraction

The socket was invented in Berkeley as part of the BSD flavor of Unix. And it has become the standard way to communicate over the internet. Today all machines, be it Windows, Linux, or Mac, support the so-called Berkeley Socket API.

The primary Berkeley socket API are:

• socket() creates a new socket. An endpoint for communication and returns a file descriptor.
• bind() associates a socket with an address, i.e., an IP address and a port number. Bind is typically used on the server side.
• listen() is used on the server side and causes a bound TCP socket to be ready for incoming connections.
• accept() is used on the server side. It creates a new socket for each connection and removes the connection from the listening queue.
• connect() establishes a direct communication link to a specific remote host. It is used on the client side and assigns a free local port number to a socket.
• send(), recv(), sendto(), and recvfrom() are used for sending and receiving data.
• close() causes the system to release resources allocated to a socket. In the case of TCP, the connection is terminated.

TCP Socket Flow

You will need more than simply knowing the socket abstraction above to help you with how to actually use it in coding. You need to know in what order to call the socket API. Let's take a look at the sequence of API calls and data flow for TCP:

For the server to be able to accept a connection, it needs to call socket(), bind(), listen() and accept() consecutively. Once we've set up the server, the client calls connect() to establish a connection to the server and initiate the three-way handshake. Socket uses send and recv to exchange data between the client and server. When done, the client and server close their respective sockets.

Conclusion

In this post, we've taken a high-level look at what sockets are and how they work. We briefly covered the history of sockets and their place in computer networking. They start at Berkeley but become the standard of communication between systems over a network. In a future article, I will talk about socket programming in Python. Thanks for reading!

I am looking into making a data visualization dashboard for an analytics app. The library I am currently looking into is Bokeh. For aesthetic reasons, I would like a smooth area chart instead of a janky-looking graph. In C3 (a D3-based chart library), this function is available through the area-spline graph type, but in Bokeh you have to do it manually. The key to having a smooth graph is through the interpolation operation. For this, I use the pchip_interpolate from the wonderful scipy library.

def get_ts_area_plot(x, y, plot):
    date_range = x.astype('int64')
    date_space = np.linspace(date_range.min(),
                             date_range.max(),
                             len(x) * 10)
    value_smooth = pchip_interpolate(date_range, 
                                     y,
                                    date_space)

    fill_color = "#1f77b4"
    area_opacity = 0.2

    # Line chart
    plot.line(x=pd.to_datetime(date_space),
              y=value_smooth,
              color=fill_color)

    # Area chart
    plot.varea(x=pd.to_datetime(date_space),
               y1=0, y2=value_smooth,
               alpha=area_opacity,
               fill_color=fill_color)

    # Dot 
    plot.circle(x=x, y=y, size=5, color=fill_color)

    return plot

I stumbled upon a book titled Designerly Ways of Knowing. The book starts by saying that there are these “two cultures” in general education: science and humanities. But there is a third neglected culture which is design. While sciences deal with the natural world and humanities deal with human experience. Design deals with the artificial world. While we value objectivity, rationality, and neutrality in the sciences, we value subjectivity and imagination in the humanities. In design, however, we value practicality, ingenuity, and empathy. I have not finished reading the whole document, but it is pretty interesting, and I intend to finish this book.

I started incorporating some Okinawan diet components into my daily meal to stay healthy. The Okinawan diet has produced one of the world’s longest-lived populations. The most common ingredients in their diet are purple sweet potato, bitter gourd, tofu, turmeric, and green tea . According to one source, about 60% of all calories came from the purple sweet potato. The purple potato is high in B vitamins and potassium. It also has a higher concentration of the antioxidant anthocyanin (from purple pigment) than blueberries. Bitter gourd has potent compounds that control blood sugar. Out of all the so-called “blue zones,” Okinawa is the closest to where I live, and I can easily find the ingredients.

However, in Four Thousand Weeks, Oliver Burkeman presents a different approach to productivity and time management. Burkeman argues that you can never have perfect control over your time. That it is delusional to think that you can get everything done. And the reason that managing time feels like such a struggle is that we are constantly trying to master it.

In this post, I summarize some of Burkeman's approaches to time management that help me look at time management differently. I also outline a set of practices I hope to use in my daily life when dealing with productivity and time management issues.

How to Approach Time Management

Too Much to Do

Do you ever feel like you have too much to do and not enough time to do it? You are not alone. There are just so many things. Works to submit, movies to watch, places to visit, etc. For most people, this can be frustrating and overwhelming. But you need to accept this: There will always be things to do. More than you can ever finish them.

Tough Choices

Now that you know that there are just too many things to do. You need to understand that you can not avoid tough choices. You are limited. Your time is limited. You simply can not get everything done. You have to choose. Think about what matters most to you. Then you make your decisions.

World does not run at your preferred speed

Things progress just the way they are. No matter how hard you might wish they were not. You need to work with this fact. Only then can you really make an impact on your situation. Problems often arise when you try to rush things or postpone things. You can not push the pace or slow it down.

You can not predict the future

You can't know in advance if an experience will turn out to be painless and well. Often people delay things because they worry about what will happen in the future. Or rush things in the hope of "getting there" quicker. But you can not tell what will happen in the future. So instead of imagining things too much, make a decision. Do the work or don't do the work. Just stop fidgeting over things you can't control.

It does not matter

Stop overvaluing your existence. This will only lead to an unrealistic definition of what it would mean to be successful. You don't have to have impressive accomplishments. From a cosmic viewpoint, when it is all over, it will not have counted for very much anyway. Don't undervalue things because you think they are not "significant" enough. Find meaning in your daily mundane life.

Daily Practice

Embrace Finitude

Embrace the truth. You have limited time and limited control over time. Only by accepting this reality can you get to accomplish more of what matters and feel more fulfilled about it.

Be Present

Learn to stop focusing exclusively on where you are headed at the expense of focusing on where you are. Do not treat everything you do as the groundwork for something else. You are already living in the moment. You do not need to modify your relationship to the present moment in time. Just know that you are here now.

Rest

Modern people are so used to working so much that rest is difficult to do. Even in their spare time, people often do stuff related to their work. But spare time is not time to "invest" in your personal growth. Accept that your days are not progressing toward a perfect future of invulnerable happiness, where things are calmer, better, and more fulfilling. Where you can finally rest. That future may never actually exist. So learn to rest for the sake of rest now.

Be Patience

Things are the way there are, no matter how hard you wish they were not. Your only hope of having any real influence over the World is to work with that fact. You want to move things faster, but know that moving faster can be counterproductive. Working too hastily will lead to more errors, which you will have to go back to correct.

You will feel less anxious when you finally face the truth that you can not dictate how fast things go. So, be willing to stop, to stay where you are. Engage in the journey instead of badgering reality to hurry up.

End note

We can not change how time works, but we can change our attitude towards it. Instead of trying to control every minute of a day, accept that you are limited. There are simply too many things you need to do. Learn to make tough decisions. Be patience. Be here now. And when it is time to rest, well, rest.

To become better at your craft, developing a strong set of supporting skills is crucial. In this article, I point out several skills to help you become a better developer.

While this list is not exhaustive, it will give you a good starting point as you continue your journey to become a better developer. I encourage you to take the time to learn and hone these skills; they will serve you well in your career.

The Shell

A shell is a powerful tool that can make your life easier. The shell is an environment that lets you interact with your computer using text commands. You can do amazing things with a few simple commands with the shell.

Many people treat the shell as a black box. They use it but don't understand it. And that's a shame. If you master the shell, you can take control and make your computer work for you.

There are two broad categories of shell commands you need to master. The first one is using the shell for basic operation. Operations like listing, copying, and deleting files and directories. The other category is the knowledge of the shell environment. You need to know about disks and filesystems and how to handle processes and schedule jobs. You also need to understand how to manage users and some network operations.

IDEs and Editors

As a developer, you spend hours in front of a text editor daily. So it is wise to invest in the skills that can help you work faster and more efficiently.

There are many IDEs and Editors out there. Each IDE and editor has its own strengths and weaknesses, so it's important to know which one will work best for the task at hand.

Personally, I use two IDEs and two editors. My primary IDE is PyCharm. When I need to do a quick code-build-debug cycle, I use Visual Studio Code (yes, I consider Visual Studio Code as IDE). For editing text, I use Sublime Text and Vim. So these are tools that I try to master.

In the end, finding an editor or IDE that makes you productive is essential. You need to find something you are comfortable using and stick with it. Once you have found your go-to editor or IDE, spend time learning all its features. With a bit of practice, you will be able to work faster and more efficiently than ever before.

Version Control System

Developers who want to be proficient in their craft should learn how to use a version control system. Version control is the practice of tracking and managing changes to software code. Version control systems are software tools that help software teams manage changes to source code over time.

In the programming world, git has become the go-to source code management system. It's used by developers for everything from tracking individual file changes to managing entire project histories. While git can seem intimidating at first, it's not that difficult to understand. With a bit of practice, you can get relatively productive. With all the powerful features, it's easy to see why it's so popular.

Debugging and Profiling

As a developer, you will bump into problems. This is why learning about debuggers and profilers is essential for developers. These tools help you troubleshoot issues with your code and identify performance bottlenecks.

There are several tools that help you to do this. One of the most well-known debugger is of course, gdb. It is widely available on many Unix-like systems. And it works for many programming languages, including C, C++, and Go. In Python world, you can use pdb for debugging and cProfile for profiling.

Of course, you can also use IDE for debugging. PyCharm and Visual Studio Code, for instance, are excellent for debugging. Both also allow you to optimize your code using profilers.

Debuggers and profilers are essential tools for any developer, regardless of experience level. By learning how to use them effectively, you can speed up your development process, identify and fix problems quickly, and improve the overall quality of your code.

Regular Expression

If you're a developer, there's a good chance you've heard of regular expressions before. They can make your life a lot easier, and it's worth taking the time to learn how to use them.

A regular expression or regex is a sequence of characters that specifies a search pattern in a text. With regex, you can quickly and easily find and fix errors in large chunks of text, or extract specific information from a document. So if you're looking to become a more efficient developer, learning regex is essential.

Shell Scripting

Many developers and programmers avoid learning shell because they think it is too difficult or complex. Me included. However, shell scripting is a handy skill to have, and it can be learned relatively easily.

Shell scripting is a programming language that lets you control your computer's operations by running scripts. Scripts are short programs that can be written in a text editor and executed from the command line.

Shell scripting is a powerful tool that can make your job as a developer easier. It can help you automate tedious tasks and make your work faster and easier.

Endnote

These skills are not specific to coding but rather general computer operation and maintenance. As you can see, there is a lot that goes into being a developer than just writing code. If you want to be the best developer you can be, it's important that you take the time to learn these essential skills. Have you worked on developing any of these skills? Are there any other essential digital skills you think every developer should know? Let me know in the comments below!

To learn anything, we need to encode information into our brains. But what does that actually mean though? And how can encoding help us succeed in learning? In this blog post, I'll explore the different ways encoding can help us learn and retain information. But first, let's talk about how our memory is stored.

How Memory is Stored

The human mind is complex. The most influential model on how we actually store information treats our mind as an information-processing system similar to computers. We get input from the environment, process it, and output decisions.

The stages of information processing according to this model consists of three parts:

1. Sensory memory. Any new input occurs in this memory. This is where we interact with our environment. It is a temporary register of all the information your senses are taking in.
2. Working memory. It is just whatever you are thinking about right at this moment. It is also called short-term memory. Working memory can hold about seven plus or minus two pieces of information at a time, so about five to nine.
3. Long-term memory is the final stage in the information processing model. We can store a lot of information in long-term memory.

So what does encoding have to do with all of this?

What is Encoding

Encoding is about moving information. Moving information from a temporary store into your working memory. And moving that into the permanent store in your long-term memory. It is the process in which data is processed and categorized for storage and retrieval. It is how information from sensory input is changed into a form so it can be stored in the brain. It transforms internal thoughts and external events into working memory and long-term memory.

Encoding is vital because if you encode information well, it is much easier to retrieve it later. Okay, great. But how do I use this knowledge to remember information better?

Encoding Strategies

There is a lot of encoding strategy. However, all successful encoding requires you to tie any new information with old information you already have in your head. This article outlines the most essential strategies. Remember, to be successful, use what you already know to help you encode the new information.

Chunking

Chunking is grouping information into groups or categories. For example, if I ask you to remember a shopping list that contains: mangos, oranges, grapes, bread, rice, beef, almonds, flour, eggs, and butter. It will be difficult for you to remember them all as is. But if you group them into categories, it will be much easier to remember. For instance, you can group mangos, oranges, and grapes into the fruit category. Bread and rice into carbs. Beef and almonds into proteins. Flour, eggs, and butter into the baking item's category. This is much easier to remember.

You can change the category in whatever way that makes sense to you. It is definitely okay to chunk those items differently. As long as they make sense to you.

Mnemonic Devices

Mnemonic devices are a set of tools to help you remember stuff. There are many mnemonic devices. The first thing you learn in school is probably the use of acronyms. Acronyms make remembering stuff super easy. For example, I am not from the US, but I remember the US Great Lakes because the acronym is so easy. HOMES: Huron, Ontario, Michigan, Erie, Superior.

Other effective devices involve memory palace, peg system, visual alphabet, etc. You can read the Memory Craft by Lynne Kelly to learn more about memory techniques. Or you can visit the artofmemory.com wiki website. They have an extensive list of mnemonic devices.

Self-referencing

Self-referencing is to look at a piece of information and think about how you relate to that information personally. You can use it to imagine the information. For instance, if you are learning about the history of a Chinese emperor, you can imagine being the emperor himself. Live the experience, and you will remember more.

Another typical way to use this technique is to prepare the material as if you are going to teach them. Writing a blog post elaborating on the topic also works.

Spacing

Spacing is not precisely an encoding technique. It is a meta-strategy. While other methods point to what you should do during the learning process. Spacing is a strategy for how you should structure your learning session.

The point of this technique is to space your study session apart. Learning for one hour every day is much better than learning five hours in a single day. Spacing your study apart has been proven to help you learn more information and retain it longer.

Endnote

Converting information into a code that your brain can easily remember makes learning faster and more efficient. So why not give it a try? The next time you're studying for an exam or trying to learn a new skill, use some encoding techniques to help get the job done. You may be surprised at just how effective they can be!

This article will show how I build a table with a multi-column filter. The table I am going to implement looks like this:

As a side note, this article builds on the previous article, where I create a simple table using Django and htmx; I won't revisit how htmx work. Now, let's get started.

The Packages Used For This Project

1. htmx. htmx gives access to using AJAX directly in HTML, using attributes. It is simple and, for a package this small, quite powerful.
2. django-tables2. This Django app lets you define tables like you define Django models. It can automatically generate a table based on a Django model. It supports pagination, column-based table sorting, custom column functionality via subclassing, and many other features.
3. django-filter. I use this package for the filtering functionality. It has APIs similar to Django's ModelForm and works well with django-tables2.
4. django-htmx. For htmx to work, Django view needs to tell which request is made using htmx and which is not. It has a middleware that adds htmx attribute to a request object.
5. Alpine-js. This project requires a more complex behavior, htmx alone will not be enough. Specifically, the table needs to make sorting, pagination, and filtering work together nicely. This is where Alpine-js comes in. This small javascript package allows me to store data and trigger action based on changes happening to a variable.

The Model

The model I use for this project is as follows:

# products/models.py
from django.db import models

class Product(models.Model):
    class Status(models.IntegerChoices):
        ACTIVE = 1, "Active"
        INACTIVE = 2, "Inactive"
        ARCHIVED = 3, "Archived"

    name = models.CharField(max_length=255)
    category = models.CharField(max_length=255)
    price = models.DecimalField(max_digits=10, decimal_places=2)
    cost = models.DecimalField(max_digits=10, decimal_places=2)
    status = models.PositiveSmallIntegerField(choices=Status.choices)

    def __str__(self):
        return self.name

The Table

The table is straightforward to set up. The show_header is set to False because I put the table header, which includes columns name, in a separate template. I will discuss this further in the template section below.

# products/tables.py
import django_tables2 as tables
from products.models import Product

class ProductHTMxMultiColumnTable(tables.Table):
    class Meta:
        model = Product
        show_header = False
        template_name = "tables/bootstrap_col_filter.html"

The Filter

The filter I use is as follows:

# products/filters.py
from decimal import Decimal

from django.db.models import Q
from django.forms import TextInput
import django_filters

from products.models import Product


class ProductFilter(django_filters.FilterSet):
    id = django_filters.NumberFilter(label="")
    name = django_filters.CharFilter(label="", lookup_expr="istartswith")
    category = django_filters.CharFilter(label="", lookup_expr="istartswith")
    price = django_filters.NumberFilter(label="", method="filter_decimal")
    cost = django_filters.NumberFilter(label="", method="filter_decimal")
    status = django_filters.ChoiceFilter(label="", choices=Product.Status.choices)

    class Meta:
        model = Product
        fields = ["id", "name", "category", "price", "cost", "status"]

    def filter_decimal(self, queryset, name, value):
        # For price and cost, filter based on
        # the following property:
        # value <= result < floor(value) + 1

        lower_bound = "__".join([name, "gte"])
        upper_bound = "__".join([name, "lt"])

        upper_value = math.floor(value) + Decimal(1)

        return queryset.filter(**{lower_bound: value,
                                  upper_bound: upper_value})

The View

The view will send out a full page when the request is not made by htmx and send partial results when the request is made by htmx.

# products/views.py
from django.core.paginator import Paginator, EmptyPage

from django_tables2 import SingleTableMixin
from django_filters.views import FilterView

from products.models import Product
from products.tables import ProductHTMxMultiColumnTable
from products.filters import ProductFilter


class CustomPaginator(Paginator):
    def validate_number(self, number):
        try:
            return super().validate_number(number)
        except EmptyPage:
            if int(number) > 1:
                # return the last page
                return self.num_pages
            elif int(number) < 1:
                # return the first page
                return 1
            else:
                raise


class ProductHTMxMultiColumTableView(SingleTableMixin, FilterView):
    table_class = ProductHTMxMultiColumnTable
    queryset = Product.objects.all()
    filterset_class = ProductFilter
    paginate_by = 10
    paginator_class = CustomPaginator

    def get_template_names(self):
        if self.request.htmx:
            template_name = "products/product_table_partial.html"
        else:
            template_name = "products/product_table_col_filter.html"

        return template_name

I also add a CustomPaginator to handle an edge case in pagination. Specifically, this is to handle a case where for instance:

1. A user selects a valid page 8
2. The user then uses the filter
3. Because the result of the filter is less than 8 pages long, Django then raises 404.

Here I can choose to use a different template when Django raises 404. But I am not sure if it is desirable. The template will not have any pagination information. I think this will confuse users since they would not know that the empty page results from an invalid page and filtering, rather than just filtering.

So, when a user filter and the page is invalid, I simply return either the last page or the first page of the filtering result.

The Template

There are three templates in play here:

1. A template to render the whole page. Where I generate the page with HTML header and body and the table.
2. A template to render only the table. When we perform some action like sorting or filtering, we only need to render the partial page.
3. Custom table template to override the default. I need to remove the table definition. And add some Alpine-js attributes for the pagination to work.

Template to Render the Entire Page

For this project, I put the table header and filter form on this page. Here I define two extra hidden input fields. The sort and page input. When a user performs an action, be it sorting, jumping between pages, or filtering, I will submit all three pieces of information back to the server.

To make this happens, I need to use Alpine-js. Here is the summary of how it works:

1. When a user sorts a table by clicking on a column header, the value of the sort_by that I define in the x-data gets updated via toggle function. This also changes the sort input value because this is linked through the use of x-model. To toggle between the ascending or descending arrow, I use :class in the column header.
2. I use x-init to watch if the input value for sort has changed or not. When the value changes, I send out a custom event called sort-initiated.
3. Since I have specified that custom event in the hx-trigger attribute, when this custom event is made, htmx will submit the whole form, which contains information on the filter, the sorting, and the pagination.

{# templates/products/product_table_col_filter #}
{% extends "base.html" %}

{% load render_table from django_tables2 %}
{% load i18n %}
{% load django_tables2 %}
{% load crispy_forms_tags %}

{% block multi_htmx_table %}active{% endblock %}

{% block main %}
<h1>Product table</h1>

<div id="table-container" class="table-container">

    <form class="form-inline"
          hx-get="{% url 'tables:products_htmx_multicol' %}"
          hx-target=".table-body-container"
          hx-trigger="input, select, sort-initiated, pagination-initiated"
          hx-swap="outerHTML"
          hx-indicator=".progress"
          x-data="{ sort_by: '', page_by: 1 }">

        <input type="hidden" name="sort" x-ref="sort_input" x-model="sort_by"
                x-init="$watch('sort_by',
                        () => $refs.sort_input.dispatchEvent(
                                new Event('sort-initiated', { bubbles: true })))">

        <input type="hidden" name="page" x-ref="paginate_input" x-model="page_by"
               x-init="$watch('page_by',
                        () => $refs.paginate_input.dispatchEvent(
                                new Event('pagination-initiated', { bubbles: true })))" >

        <table {% render_attrs table.attrs class="table multi-col-header" %}>
            <thead {{ table.attrs.thead.as_html }}>
                <tr>
                    {% for column in table.columns %}
                    <th {{ column.attrs.th.as_html }}
                        x-data="{ col_name: '{{ column.order_by_alias }}',
                                  toggle(event) {
                                    this.col_name = this.col_name.startsWith('-') ? this.col_name.substring(1) : ('-' + this.col_name);
                                    sort_by = this.col_name;}}"
                        @click="toggle()"
                        :class="sort_by !== '' ? (sort_by === col_name ? (sort_by.startsWith('-') ? 'desc' : 'asc') : '') : ''"
                        style="cursor: pointer;">
                        {{ column.header }}
                    </th>
                    {% endfor %}
                </tr>
                <tr>
                    {% for field in filter.form %}
                    <td>{{ field|as_crispy_field }}</td>
                    {% endfor %}
                </tr>
            </thead>
        </table>

        <div class="progress">
            <div class="indeterminate"></div>
        </div>

        {% render_table table %}
    </form>
</div>
{% endblock %}

Template to Render Table

Here we only render the table content, no HTML header, no HTML body. Just the content.

{# templates/products/product_table_partial.html #}
{% load render_table from django_tables2 %}

{% render_table table %}

Template to Override the Default Table

This template overrides the default table from django-tables2. On this template, I render the table body and pagination information. The pagination works like sorting. When a user triggers any pagination action, the page_by variable in x-data gets updated, which changes the value of page input because it is linked via x-model. Which creates a custom event pagination-initiated. Which then tells htmx to send the complete form with all sorting, pagination, and filtering information back to the server.

{# templates/tables/bootstrap_col_filter.html #}
{% load django_tables2 %}
{% load i18n %}

{% block table-wrapper %}
    <div class="table-body-container">
        {% block table %}
            <table {% render_attrs table.attrs class="table" %}>
                {% block table.thead %}{% endblock table.thead %}

                {% block table.tbody %}
                    <tbody id="body-target" {{ table.attrs.tbody.as_html }}>
                    {% for row in table.paginated_rows %}
                        {% block table.tbody.row %}
                            <tr {{ row.attrs.as_html }}>
                                {% for column, cell in row.items %}
                                    <td {{ column.attrs.td.as_html }}>{% if column.localize == None %}{{ cell }}{% else %}{% if column.localize %}{{ cell|localize }}{% else %}{{ cell|unlocalize }}{% endif %}{% endif %}</td>
                                {% endfor %}
                            </tr>
                        {% endblock table.tbody.row %}
                    {% empty %}
                        {% if table.empty_text %}
                            {% block table.tbody.empty_text %}
                                <tr><td colspan="{{ table.columns|length }}">{{ table.empty_text }}</td></tr>
                            {% endblock table.tbody.empty_text %}
                        {% endif %}
                    {% endfor %}
                    </tbody>
                {% endblock table.tbody %}
                {% block table.tfoot %}
                    {% if table.has_footer %}
                        <tfoot {{ table.attrs.tfoot.as_html }}>
                        <tr>
                            {% for column in table.columns %}
                                <td {{ column.attrs.tf.as_html }}>{{ column.footer }}</td>
                            {% endfor %}
                        </tr>
                        </tfoot>
                    {% endif %}
                {% endblock table.tfoot %}
            </table>
        {% endblock table %}

        {% block pagination %}
            {% if table.page and table.paginator.num_pages > 1 %}
                <nav aria-label="Table navigation">
            <ul class="pagination justify-content-end">
                {% if table.page.has_previous %}
                {% block pagination.previous %}
                <li class="previous page-item" role="button">
                    <div @click="page_by = {{table.page.previous_page_number}}"
                         class="page-link">
                        <span aria-hidden="true">&laquo;</span>
                        {% trans 'previous' %}
                    </div>
                </li>
                {% endblock pagination.previous %}
                {% endif %}
                {% if table.page.has_previous or table.page.has_next %}
                {% block pagination.range %}
                {% for p in table.page|table_page_range:table.paginator %}
                <li class="page-item{% if table.page.number == p %} active{% endif %}" role="button">
                    <div class="page-link" {% if p != '...' %}@click="page_by={{p}}"{% endif %}>
                        {{ p }}
                    </div>
                </li>
                {% endfor %}
                {% endblock pagination.range %}
                {% endif %}
                {% if table.page.has_next %}
                {% block pagination.next %}
                <li class="next page-item user-select" role="button">
                    <div @click="page_by = {{table.page.next_page_number}}" class="page-link">
                        {% trans 'next' %}
                        <span aria-hidden="true">&raquo;</span>
                    </div>
                </li>
                {% endblock pagination.next %}
                {% endif %}
            </ul>
        </nav>
            {% endif %}
        {% endblock pagination %}
    </div>
{% endblock table-wrapper %}

The Result

This is what it looks like in the end.

Sorting	Pagination
Filtering	Progress Bar
All actions

Summary

In this article, I explore how to use Django, htmx to build a slightly more complex table. For a more complex behavior, htmx alone is not enough. This is where I use Alpine-js. With Alpine-js I built a table with a multi-column filtering feature. It is pretty straightforward to make. I hope you find this article helpful.

In the future, I will explore how to build an even more complex table with features like complex filters, column selection, and saved filters.

In his book, "The Psychology of Money," Morgan Housel writes about a set of behavior you can use to achieve financial success. He argues that financial success is not hard science. It is a soft skill. Doing well with money has little to do with how smart you are and a lot to do with your behavior. This article summarizes the most helpful tips on how you should think about money and some helpful tips on managing your money effectively.

How to think about money

Humility and Compassion

Handling money is hard. We all have wildly different views on how to handle money. The world is complex. Luck and risk are hard to identify. Know that when making decisions regarding money, sometime you will be correct, sometime you will be wrong. Find humility when things are going right. And do not beat yourself up when things are going wrong. Be compassionate when judging yourself and others. Things are never as good or as bad as it looks.

Control Over Time

Ultimately, we all want to be happy. One of the most potent ways that affect your happiness is having control over your time. To be able to do what you want, when you want, with who you want, and for as long as you wish will significantly boost your happiness. Instead of chasing money, you should see money as a means to gain control over your time.

Reasonable > Rational

You should aim to be mostly reasonable rather than trying to be coldly rational. Reasonable is more realistic. It may be rational to take more risk on a certain decision. And some people are fine taking a higher risk if it means they are earning the highest returns. But others will only get a good rest if they invest more conservatively. To each their own. Learn to manage your money in a way that helps you sleep at night.

Compounding Works

Time is your friend. Time is the most powerful force in investing. It makes little things grow big, and big mistakes fade away. It can not neutralize luck and risk, but it pushes results closer to what you deserve. Consider this: $81.5 billion of Warren Buffett's $84.5 billion net worth came after his 65th birthday.

Play Your Own Game

Define the game you are playing and ensure your actions are not influenced by people playing a different game. A short-term trader has a different mindset than a long-term investor. Once you know which game you play, define the cost of success and pay it.

Respect the Mess

A lot of things can and will go wrong. You will make mistakes. And it is okay to make mistakes. Believe that you can be wrong half the time and still make a fortune because a small minority of things account for most outcomes. There is no single right answer when it comes to handling money. There is just the answer that works for you. Smart, informed, and reasonable people can disagree in finance because we all have different goals and desires.

Daily Practice

Save

Saving money is the path to wealth. Wealth is created by suppressing what you could buy today to have more stuff or options in the future. No matter how much you earn, you will never build wealth unless you can limit how much fun you can have with your money today.

Be Nicer and Less Flashy

No one is impressed with your possessions as much as you are. Nobody cares about your sports car or your watch. What you want is to be respected and admired by others. But using money to buy fancy things does not help. You gain respect and admiration through kindness and humility.

Survival Mindset

Getting money is one thing. Keeping it is another. There are a million ways to get wealthy, but there is only one way to stay wealthy: some combination of frugality and paranoia. The world is full of surprises, and the future is impossible to predict or define. Having a survival mentality is essential with money. Having a survival mentality means more than big returns; you aim to be financially unbreakable. It means you have a plan, but also prepare when things are not going according to plan. It means you have hope for the future but are paranoid about what will prevent you from getting to that future.

Conclusion

These are lessons we can carry with us as we go about our lives, making minor tweaks to how we think and behave around money. Remember, money is just one piece of the puzzle – do not let it become your entire life.

In her book "How To Decide," Annie Duke presents a framework that helps you make decisions. This article is a summary of the decision-making framework detailed in her book.

For a piece of background information, Annie Duke is a former professional poker player and author in cognitive-behavioral decision science and decision education. Duke's total lifetime live tournament winnings of $4,270,548 place her at fourth overall in the women's all-time live tournament winnings.

Things to Understand

Decision Multiverse

There are many possible futures. When you make a decision, you see the future possibilities like the branches of a tree, each branch representing how things could unfold. To be a better decision-maker, you need to learn to put possible outcomes (good or bad) in perspective.

Willingness to guess

Most people are reluctant to estimate the likelihood of something happening in the future. ("That's speculative." "I don't know enough." "I'd just be guessing.”). Even though your information is usually imperfect, you know something about most things, enough to make an educated guess.

The willingness to guess is essential to improving decisions. If you don't make yourself guess, you'll be less likely to ask, "What do I know?" and "What don't I know?"

Things to Avoid

Outcome Bias

Outcome bias is a mental shortcut in which we use the quality of an outcome to figure out the quality of a decision. It is the tendency to look at whether a result was good or bad to figure out whether a decision was good or bad.

You should not judge if a decision is good or bad based on the result. There is only a loose relationship between the quality of the decision and the quality of the outcome. When we decide in real life, there is an element of luck. Luck intervenes between your decision and the actual outcome. You can make a good decision and have a bad result due to luck. You can also make a super dumb decision that has a good result.

Hindsight Bias

Hindsight bias is the tendency to believe that an outcome is predictable or inevitable after it occurs. To get better at making decisions, you need to learn from your choices and their outcomes, but hindsight bias distorts the way you process outcomes.

Once you know how a decision turns out, you can experience memory creep, where the stuff that reveals itself after the fact creeps into your memory of what you knew or was knowable before the decision.

Decision-Making Framework

The Three Ps

Preference

There is a greater liking for one alternative over another for every outcome you identified. This liking or preference is individual to you and depends on your goals and values. How much you prefer a particular result relative to other possibilities will naturally be different from another person's preference for the same outcome relative to other possibilities. That doesn't make either of you wrong. It just means that you are different people with particular likes and dislikes.

Payoffs

For any set of outcomes, there will be stuff you could gain (upside potential) and stuff you could lose (downside potential). These gains or losses are called payoffs. Payoffs can be measured in anything you value, like money, time, happiness, health, etc. They will drive your preference because you will prefer gains over losses. When you are figuring out whether a decision is good or bad, you are comparing the upside vs. downside. Does the upside potential compensate for the risk of the downside potential?

Probabilities

Probabilities express how likely something is to occur. To figure out if a decision is good or bad, you need to know the likelihood of each possible outcome.

Without information about the probability of any possibility unfolding, you can beat yourself up over a bad result because you can not see that the result was improbable to happen in the first place.

Six steps to better decision-making

1. Identify the reasonable set of possible outcomes
2. Identify your preference using the payoff for each outcome-- to what degree do you like or dislike each outcome, given your values
3. Estimate the likelihood of each outcome unfolding
4. Assess the relative likelihood of outcomes you like and dislike for the option under consideration
5. Repeat steps 1-4 for other options under consideration.
6. Compare the options to one another

How to Spend Decision-Making Time Wisely

The average person spends 250–275 hours per year deciding what to eat, watch, and wear. We take too much time when we make decisions. And many of those decisions we take are routine inconsequential decisions.

When making any decisions, we are faced with a trade-off. Increasing accuracy costs time. Saving time costs accuracy.

The key to balancing this trade-off is to figure out the penalty for not getting the decision exactly right. When you are making decisions consider these four scenarios.

Happiness Test

If the outcome of your decision, good or bad, has no effect on your happiness in a year, you have passed the happiness test, and you can speed up.

Freerolling

When you face a situation where the upside is positive, and the downside is limited, go fast. The faster you decide to seize the opportunity, the quicker you realize the one-sided upside potential of the decision.

Sheep in wolf's clothing

When you are considering two options that are close, then the decision is easy. Whichever one you choose, you can't be wrong since the difference between the two is so tiny.

Quit early, quit often

The lower the cost to quit, the faster you can go because it is easier to unwind the decision and choose a different option, including options you may have rejected in the past.

Decisions with a low cost to quit, known as two-way-door decisions, also provide you with low-cost opportunities to make experimental decisions to gather information and learn about your values and preferences for future decisions.

When facing a decision with a high or prohibitive cost of changing your mind, try decision stacking, making two-way-door decisions ahead of the one-way-door decision. You can also defray opportunity costs by exercising multiple options in parallel.

Here is a simple flow chart on how to manage the time-accuracy trade-off.

Knowing When Your Decision Process Is Finished

Because you can not be sure of the outcome of your decision, you will make most decisions while still uncertain. To know if it is worthwhile to have additional time to decide, ask yourself, "Is there additional information (available at a reasonable cost) that would change your mind?" If yes, find it. If no, decide and move on.

It turns out that there are underlying patterns for most of these problems. Finding the right technique to solve an algorithm problem can be the difference between a correct solution and spending hours staring at a problem.

I have summarized some patterns that help me solve algorithmic problems in a short study note. There are 14 techniques overall. Each method is explained with examples that illustrate how it can be used to solve specific types of problems. You can learn how to apply these techniques to solving problems in order to produce correct solutions more efficiently.

I also provide just a summary of the coding techniques without the code examples for quick reference.

With a bit of practice, you will be able to apply these patterns to any algorithm problem and start solving problems. Pick up a copy of coding techniques today!

But before we talk about memory techniques, let's talk about the need for memorization.

Why memorize?

There is a lot of debate on whether or not you should memorize things. Some people swear by it, while others say that there are better ways to learn and remember information. Here are some reasons why you should memorize:

1. To reason, you need facts. When you commit data to memory, you can play with information in your head. The more information you have in your memory, the more you can use it to reason. Memorizing facts gives you a bank of material you can use to form or test a hypothesis.
2. Foundation for higher-order thinking. Only with a factual base that you believe can you effectively use your higher levels of thinking. Memorizing makes information readily available for learning deeper and making connections to new material. You can not be creative or innovative with information you do not have.
3. Organise your knowledge. To memorize any information, you need to first organize it into little chunks that flow in a logical order. In turn, it gives you a structure and context in which to put any new information. Memorizing a set of facts means that new information can be inserted in an orderly way, sandwiched, or used to enhance other facts in an organized framework.
4. It stays with you. You can now look up anything thanks to technology, but that consumes time and working memory. By committing facts to memory, you are much more efficient and can work around the working memory limitations. Even if you don't understand it the first time, memorizing information allows you to return to it. And research has shown that the retrieval practice will significantly increase learning. Having to pull the information out and make meaning is how learning occurs.

Keep in mind that memorization is different from rote learning. Rote learning is learning by repetition without any regard for understanding. It is not the primary goal of learning, but it is a critical component.

Memory techniques and learning

When you first learn any new information, it is put in your working memory. Working memory is very good at receiving data from our long-term memory and manipulating it. Unfortunately, your working memory can only hold between five to seven items.

To make learning effective, you need to move the information you have learned to long-term memory. Encoding is the process of transferring information from working memory to long-term memory. If you encode correctly, you will have an easier time remembering the information in the future.

Memory techniques or mnemonic devices are among the best techniques to help you encode information. Other encoding techniques are chunking, self-referencing, and spacing.

Effective memory techniques

There are many different types of memory techniques, and people have used them for centuries to help remember everything from essential facts to the order of cards in a deck. In this article, I will focus on the three most effective memory techniques you can use when you learn.

First of all, the basis of all memory techniques is story, imagination, and characters. To remember better, you need to be able to make up stories, imagination, and characters in your mind. The more vivid and wild, and unusual you make your images association, the more likely you can remember information. Mundane information will not excite your brain, but wild stories and wild imagination will.

Other memory techniques rely on your ability to create stories, images, and characters. Luckily everyone can make these things up. And the more often you do this, the better you get at creating this.

Now that you understand the basics of memory techniques let us move on to the more advanced technique: memory palace, visual alphabet, and Dominic system.

Memory palace

Nearly everyone has heard of the phrase "memory palace." Still, not many people know what it is or how to use it. A memory palace is a technique used to improve your memory by associating objects with specific locations. You attach knowledge to the locations by imagining events there. It can be used for short-term and long-term memories and is the most effective technique available.

To "build" a memory palace, you need to choose a location. Potential memory palaces can be your homes, schools, workplaces, parks, neighborhoods, libraries, malls, gyms, etc. You can turn any location you are familiar with into a memory palace. But to be effective, you should choose locations carefully. According to an ancient Roman textbook, "Rhetorica ad Helenium," you should choose locations that are far away from distractions. The locations should be well lit. They should be pretty distinct from each other. Locations should be moderate in size and have a moderate distance between them.

Once you have a location you want to turn into a memory palace, you can plan out the whole route. For example, gate, front porch, front door, shoe rack, bathroom, living room, etc. When you have a list that you want to memorize, take one or two items and place a mental image of them in each locus of your memory palace. Exaggerate the visualization of the information and have them interact with the location. Even better if you can turn the images alive with your senses.

Visual alphabet

When it comes to memorizing a list, there is no better technique than the visual alphabet. This method uses images to represent each letter of the alphabet. By creating a mental picture associated with each letter, it becomes much easier to remember items on a list. For example, the letter A might be represented by an image of an apple, B could be a banana, and so on. Here is the visual alphabet that I have created and used.

Most of the time, I use this technique in conjunction with other memory techniques. For instance, when I want to memorize a piece of information that contain sub-components. I use the memory palace to store the main points. If there are sub-components, I will just use this technique to create a list and place it on a locus.

Dominic system

The Dominic system is a memory technique invented by Dominic O'Brien. It is a system for memorizing long sequences of numbers by first converting them into pairs of letters. And then associating those letters with easier-to-remember people and actions.

The digit-letter association used by the Dominic system are :

For instance, 04 is "OD," which becomes Oscar De La Hoya punching a sandbag. The number 62 is "SB" and could be Sandra Bullock driving a bus.

To use this system, I have made visualization for numbers from 00 - 99. To help me remember this better, I dump them into my Anki and review the numbers every day. Here is an example of my first 20 visualizations.

Long numbers are chunked in 4s. The first 2 digits are converted into a person, and the second 2 digits become an action. To remember the year 1619, I imagine Arnold Schwarzenegger riding a motorcycle.

To remember a long number like 17191902, for example, you need to chunk them into 2763 and 6339. Then convert them into AGAN ANOB. Using my own list, I can create a story of Ariana Grande blowing up a mine with dynamite and getting cut in half by Alfred Nobel, who wields a lightsaber.

Tips for using memory techniques effectively

Memory techniques can be a great way to boost your ability to remember information. However, using them effectively can be tricky. Here are some tips for getting the most out of your memory techniques, according to Alex Mullen, a memory athlete and a medical student from mullenmemory.com.

1. Do not memorize everything using memory techniques. Memory techniques are just tools. If you can intuitively remember a concept, you don't need to utilize memory techniques. Use your critical thinking about the material. Think about what is worth memorizing and what is not.
2. Do not encode everything using images. Maintain an optimal mix of understanding/intuition coupled with memory techniques as a supplement
3. Use memory palace to structure material. The memory palace is incredibly good at helping you organize and sequence information.
4. Use spaced repetition software to maintain your mnemonics. Images can get faded away in your memory. Spaced repetition software like Anki can help you manage your memory. Spaced repetition is a critical component of building lasting retention.

Summary

Memory techniques are an effective way to learn and remember facts. The three techniques I have shared should help you get started with improving your memory. Have you tried any of these methods before? What has been your experience?

What is personal content curation

Content curation is the process of sifting through the vast amount of information available and organizing them in a meaningful way for an audience. Content curation is not about mindlessly collecting links; it is about putting them into a context with organization, annotation, and presentation. A content curator continually seeks, makes sense of, and shares the best and most relevant content on a particular topic online to share with their community.

Personal content curation is no different from content curation. The work involves sifting, sorting, and arranging. The main difference is that you are the primary audience, which means that the value of the content lies in its usefulness and application to you, first and foremost.

Why you should curate

We live in an age where there is simply too much information. Even the most minor niches like growing bonsai or home hydroponics gardens have excessive information. This is why curating content plays an essential role. Here are three reasons why you should curate content.

1. Tackle information overload. There is so much information available to us today. Social networks, websites, emails, and other digital sources make data that is now measured in exabytes, equal to a quintillion bytes. The problem is not the amount of information per se but our consumption. We are indulging too much in the easy access we get to the information. Content curation can help in this regard because mindful consumption of information is at the heart of content curation practice.
2. Build knowledge. Information is not equal to knowledge. Just because you have access to information does not mean you are more knowledgeable and skillful. To create understanding, you must know how to sift through the vast amount of information and judge the value, organize, and connect that information. The amount of available information leaves many unable to organize their thoughts and ideas. As a result, their work is merely information without knowledge. Content curation can be an essential tool for you to start building knowledge. The process of organizing resources forces you to construct connections between concepts and identify areas of synthesis, encouraging critical thinking and rhetorical engagement with the information.
3. Develop your tastes. Content curation teaches you to distinguish what is excellent from merely good. It teaches you to see what is missing or not working. It helps you uncover what you like or dislike, what you think is worth your time, and what is not. Why is this important? Content curation helps us form our tastes, and our tastes influence our work.

What to curate

If you are unsure what to curate, you can start by asking yourself some questions. In "Show your work!" Austin Kleon, the author, suggests that you ask the following questions to kick off your curating journey:

Where do you get inspirations? What sorts of things do you fill your head with? What do you read? Do you subscribe to anything? What sites do you visit on the Internet? What music do you listen to? What movies do you see? Do you look at art? What do you collect? What's inside your scrapbook? What do you pin to the corkboard above your desk? What do you stick on your refrigerator? Who's done work that you admire? Who do you steal ideas from? Do you have any heroes? Who do you follow online? Who are the practitioners you look up to in your field?

How to start curating

It can be tough to know where to start when it comes to curating content. But do not worry, I am here to help! Here are five ways to get started:

1. Create a repository. Create a place where you can put your collections. This can be a Pinterest board or a Raindrop collection. You can start by collecting a small set of valuable sources and personal insights for yourself.
2. Filter your sources. Pull from a consistent set of high-quality sources to save you time. Be intentional. Good content is everywhere, and you do not want to be overwhelmed. Pick a handful of great blogs or newsletters and start from there. Do not capture unless the content is of high quality.
3. Add your personality. Think about why the content is relevant or interesting to you and take some notes. Always add your interpretation or commentary.
4. Organize your judgment. Take important critical takeaways from the content and turn them into an outline. Compare it with other content. Whenever you choose one piece of content over another, you refine your judgment about what is essential and what is not. Curating requires and improves your judgment skill.
5. Learn and fail in public. Consider sharing your curation with the public. When you share with others, you might get comments and complaints that test your understanding and surface doubts and weak points. This will lead you to an even deeper understanding of your collection.

Personal content curation tools

There are numerous tools to help you curate content. From the supply side, there are a lot of websites (example: Medium, Hackernews, etc.), news aggregators (example: Google news), and countless newsletters (example: Hackernews, Python Weekly, PyCoder, etc.) for you to curate content. Remember to start from a small set of high-quality sources and build up from that.

Once you have read them, you want to store some of them. For this, at the moment, I am using the following platform:

1. Raindrop. I am currently using Raindrop most of the time. It is a bookmark app with a lot of features. You can save a bookmark, move it into a collection, give it proper tags, etc. It supports multiple viewing modes. You can display your bookmark in list format, card formats, headlights format, and mood board. It is not perfect by any means; for instance, you can only save a bookmark to a single collection. You can work around this issue by playing around with the URL a bit. But it is a bit annoying having to do that.
2. Notion. I use Notion to create lists, tables, and databases. At the moment, Notion is mostly for storing programming problems or security hacking writeups. But I am also experimenting with using GitHub for this. So I might stop using Notion in the future.
3. Pinterest. Pinterest probably is the most well-known curating platform. It is effortless to use, and there is a lot of great content on the platform. The downside, it can turn into a rabbit hole. For this reason, I only use it for several types of content. But it is by far the easiest to start and use.

I should also mention Feedly. It is a news aggregator application. It compiles news feeds from many online sources, and you can customize this, of course. You can even subscribe to a newsletter via Feedly, so any email from the newsletter gets into Feedly, keeping your mail inbox a bit more tidy. But it also has a curation feature called Board, similar to Pinterest board or Raindrop collections. This idea of having a news feeder and a curation platform in a single place is excellent. So I am looking into this app at the moment.

Tips for content curation

Here are some tips for staying organized and efficient when curating your own content.

1. Select a theme. A theme is a unifying idea. You should decide on a theme when you make a repository or a collection. For example, suppose you are learning about Algorithms. In that case, you can have a narrow collection such as "Top 5 resources for learning Algorithms" or a broad theme like "Algorithms." It is easier to find content that will fit a broad theme. But a narrower theme makes your collection more cohesive.
2. Know your space. Know the parameters. Is it a Pinterest board or a RainDrop collection? Both can have an unlimited number of pins or bookmarks. Think about what it means if you put a lot of items, say a hundred items vs. ten items, in your collection. A hundred items may sound significant, but it might make sense for some collections. A "Top 100 resources for learning algorithm" might be too significant to be useful. But a "A 100 of most influential persons in history" collection, a hundred makes a lot of sense. There is no right or wrong answer to the number of items you can have in your collection. You decide the number based on the theme you chose.
3. Provide information. Curating is like writing an essay -- you want to present your work in a certain way. On Pinterest, you can add a description to any board. Unfortunately, you can not do it on Raindrop. On both platforms, you can manually arrange the position of the curated content. In Raindrop, there are more options; you can use lists, cards, headlines, and mood boards. So you can set up the collections a bit better. For instance, you can set up a collection in chronological order.
4. Selecting work. You should start by choosing a lot of content. But, select only a handful of content that best fits your theme and is of high quality. This is where you do the work of comparing, connecting, and making sense of the information. For instance, there is a lot of book on Algorithm out there. Which one should you choose? Some books are rigorous with math, and some are not. Some start with Graph, and some are not. There is also a matter of personal preference. Maybe you like the way the author writes. It makes more sense to you. And so on.
5. Write a note. Add standard labels like name, title, and other basic information. This should be automatic if you are using Raindrop and Pinterest. Both Pinterest and Raindrop also allow you to add a description to a pin or a bookmark. Always write down a description. Even a short commentary is better than no note at all. You should also write additional available information or comments if available. For example, why it fits in the collections.
6. Think about anchor pieces. Every collection has certain elements that stand out. Think about which content stands out. And what makes that content an anchor piece. When you share your collections with the public, think about how these anchor pieces can direct your audience's attention. For instance, my anchor piece in my top learning algorithm collection is the "Algorithm Design Manual" book.

How to use curated content in different areas of your life

Curating content does not have to stop in the digital world. The concept of curating, which is selecting and organizing items in a meaningful way, can also be applied to other areas of your life. Here are some examples of how to use curated content in different areas of your life:

1. Content creation. Curating content is one of the best ways to start creating content. The act of curating the works of others can help you in creating your own work. Examples of things you can make as a curator include curated news feeds, guides, comparison tables, diagrams, learning curriculums, etc. Maybe you can be a part of the new curator economy.
2. Fashion. In the fashion world, there is the term capsule wardrobe. A capsule wardrobe is a small collection of garments designed to be worn together, harmonizing in color and line. If you think about it, a capsule wardrobe is essentially a curation. Instead of having a lot of pieces of clothing, you can start curating a handful of pieces of clothing items. A collection that suits you better saves you money and that you can wear for any occasion.
3. Books. You probably have heard about the "100 books to read before you die" collection. If you are an avid reader, maybe you should start your own collections. And you do not have to stop at books. You can make collections of movies, wine, art, etc.

Summary

Content curation can be a great way to tackle information overload, build your knowledge, and develop your taste. Although it takes time to develop the habit, personal content curation can boost your productivity and knowledge development. It allows you to focus on learning what is important to you while building a valuable reference library along the way. By using the tips I shared in this post, I hope you feel inspired and motivated to start curating content for your own use. Remember, the key is to find what works best for you and to stick with it. The more personal your approach, the better. What are some of your favorite methods for finding and organizing information?