{
    "version": "https:\/\/jsonfeed.org\/version\/1",
    "title": "LEFT JOIN: blog on analytics, visualisation & data science",
    "home_page_url": "https:\/\/en.leftjoin.ru\/",
    "feed_url": "https:\/\/en.leftjoin.ru\/rss\/",
    "icon": "https:\/\/en.leftjoin.ru\/user\/userpic@2x.jpg",
    "author": {
        "name": "Nikolay Valiotti",
        "url": "https:\/\/en.leftjoin.ru\/",
        "avatar": "https:\/\/en.leftjoin.ru\/user\/userpic@2x.jpg"
    },
    "items": [
        {
            "id": "72",
            "url": "https:\/\/en.leftjoin.ru\/all\/google-app-script\/",
            "title": "How to: Google App Script",
            "content_html": "<p>Today’s post will shed some light on how you can use one of the very handy basic analytics tools – Google Sheets – even more effectively.<br \/>\nOften, analysts need to work a lot with tables and create selections for various conditions. In working with tables, the problem usually lies in the fact that you have to process and organize large amounts of data stored on different sheets. For example, you keep track of customers on one sheet of a spreadsheet, uploading data from third-party services, and you would like to add or change the data on another sheet. Let’s see how easy and simple we suggest to solve this problem.<br \/>\n<img src=\"https:\/\/leftjoin.ru\/pictures\/1-2.jpg\" border=\"0\" width=\"100%\" height=\"100%\"><\/a><\/p>\n<h2>Script editor<\/h2>\n<p>If you have a Google account and data sheets uploaded to Google Sheets, then you can create a script for this table. Select the table in which you want to automate the transfer of information from one sheet to another, open it and select the “Script Editor” item from the “Tools” menu. The browser will redirect you to the Apps Script page, where you can create and edit scripts for the table.<\/p>\n<p><img src=\"https:\/\/leftjoin.ru\/pictures\/--2021-11-10--16.20.29.png\" border=\"0\" width=\"100%\" height=\"100%\"><\/a><\/p>\n<h2>Automating line transfer to another sheet<\/h2>\n<p>So, our script should automatically perform the task of line transfer to another sheet. To do this, we create one more column in the table, in which it will be possible to check the box to transfer the line or remove it to cancel this action. Let’s see how exactly this is done.<br \/>\nWhen writing a script, you can use a trigger function that fires when a certain condition is met. You can read more about trigger functions <a href=\"https:\/\/developers.google.com\/apps-script\/guides\/triggers?hl=en#onedite\">in the documentation<\/a>. In our script, we use the OnEdit(e) function, which is triggered when a table cell is edited (toggle the checkbox on and off). We will call this function at the end of the script, but for now, we will write everything that should happen in the function.<\/p>\n<pre class=\"e2-text-code\"><code>function myScript(e) {   \r\n  \/\/ We set the following condition for the function: you only need to react to clicking the checkmark in the eighth column on the &quot;Лиды-воронка&quot; sheet.\r\n  if (e.source.getActiveSheet().getName() == &quot;Лиды- воронка&quot; &amp;&amp; e.range.getColumn() == 8)\r\n  {\r\n    \/\/ Saving the objects of the source sheet and the destination sheet\r\n    destSheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName('test');\r\n    sourceSheet = e.source.getActiveSheet();\r\n    \/\/ We clear the destination sheet, starting from the second line, since we have a title in the table.\r\n    destSheet.getRange(2, 1, destSheet.getLastRow(), destSheet.getLastColumn()).clearContent();\r\n    \/\/ We go through all the cells with check marks, looking for those cells in which the check marks are put down.\r\n    range = sourceSheet.getRange(1, 1, sourceSheet.getLastRow(), sourceSheet.getLastColumn());       \r\n    for (i = 2; i &lt;= range.getNumRows(); i++) \r\n    {      \r\n      \/\/ We get all the checkboxes.\r\n      if (range.getCell(i,8).getValue())\r\n      {        \r\n        \/\/ If the checkbox is filled, the current line is transferred to a new sheet.\r\n        currentRow = sourceSheet.getRange(i, 1, i, sourceSheet.getLastColumn());           \r\n        destSheet.appendRow(currentRow.getValues()[0]);\r\n      }      \r\n    }    \r\n  }\r\n\r\n\/\/ Then, we call a trigger function that will call our script every time we edit a cell.\r\nfunction onEdit(e) {\r\n  myScript(e)\r\n}<\/code><\/pre><p>As a result, if a check mark in a column is selected, then the corresponding row will be transferred to another sheet. Also, it is important to remember that when you remove a check mark, the line is removed from another sheet.<br \/>\nIn our example, we got the following two tables:<\/p>\n<p><img src=\"https:\/\/leftjoin.ru\/files\/gas.gif\" border=\"0\" width=\"100%\" height=\"100%\"><\/a><\/p>\n<h2>Conclusions<\/h2>\n<p>Today we showed you an example of a script that will simplify, and maybe even improve, the routine work with tables. Of course, many more processes can be automated with Google Sheets scripts. You can read <a href=\"https:\/\/developers.google.com\/apps-script\/overview?hl=ru\">about other features<\/a> yourself, and we’ll cover some of them in detail in future posts!<\/p>\n",
            "date_published": "2021-11-10T16:43:25+03:00",
            "date_modified": "2021-11-10T16:48:39+03:00",
            "_date_published_rfc2822": "Wed, 10 Nov 2021 16:43:25 +0300",
            "_rss_guid_is_permalink": "false",
            "_rss_guid": "72",
            "_e2_data": {
                "is_favourite": false,
                "links_required": [
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css"
                ],
                "og_images": []
            }
        },
        {
            "id": "71",
            "url": "https:\/\/en.leftjoin.ru\/all\/mean-vs-median\/",
            "title": "Mean VS median: how to choose a target metric?",
            "content_html": "<p>In today’s article, we would like to highlight a simple, but important topic – how to choose a simple metric to evaluate a particular dataset. Everyone has been familiar with the arithmetic mean for a long time, almost every student knows very well that you should sum up all the available values, divide by their number and get the average value. However, school knowledge does not include any alternative options, of which, in fact, there are many in statistics – almost for every occasion. In solving research and marketing problems, people often take mean as a target. Is this legitimate or is there a better option? Let’s figure it out.<\/p>\n<p>To begin with, it is worth remembering the definitions of the two metrics that we will talk about today.<br \/>\nMean is the most popular statistic used to calculate a data center. What is the median? Median is a value that splits data, sorted in order of increasing values, into two equal parts. This means that the median shows the central value in the sample if the number of cases is odd and the arithmetic mean of the two values ​​if the number of cases in the sample is even.<\/p>\n<h2>Research tasks<\/h2>\n<p>So, the estimation of the sample mean is often important in many research questions. For instance, specialists studying demography often study changes in the number of regions in Russia in order to track the dynamics and reflect it in reports. Let’s try to calculate the average size of the Russian city, as well as the median, and then compare the results.<br \/>\nFirst, you need to find and load data by connecting the pandas library for this.<\/p>\n<pre class=\"e2-text-code\"><code>import pandas as pd\r\ncity = pd.read_csv('city.csv', sep = ';')<\/code><\/pre><p>Then, you need to calculate the mean and median of the sample.<\/p>\n<pre class=\"e2-text-code\"><code>mean_pop = round (city.population_2020.mean (), 0)\r\nmedian_pop = round (city.population_2020.median (), 0)<\/code><\/pre><p>The values, of course, are different, since the distribution of observations in the sample is different from the normal one. In order to understand whether they are very different, let’s build a distribution graph and display the mean and median.<\/p>\n<pre class=\"e2-text-code\"><code>import matplotlib.pyplot as plt\r\nimport seaborn as sns\r\n\r\nsns.set_palette('rainbow')\r\nfig = plt.figure(figsize = (20, 15))\r\nax = fig.add_subplot(1, 1, 1)\r\ng = sns.histplot(data = city, x= 'population_2020', alpha=0.6, bins = 100, ax=ax)\r\n\r\ng.axvline(mean_pop, linewidth=2, color='r', alpha=0.9, linestyle='--', label = 'Mean = {:,.0f}'.format(mean_pop).replace(',', ' '))\r\ng.axvline(median_pop, linewidth=2, color='darkgreen', alpha=0.9, linestyle='--', label = 'Median = {:,.0f}'.format(median_pop).replace(',', ' '))\r\n\r\nplt.ticklabel_format(axis='x', style='plain')\r\nplt.xlabel(&quot;Population&quot;, fontsize=20)\r\nplt.ylabel(&quot;Number of cities&quot;, fontsize=20)\r\nplt.title(&quot;Distribution of population of russian cities&quot;, fontsize=20)\r\nplt.legend(fontsize=&quot;xx-large&quot;)\r\nplt.show()<\/code><\/pre><div class=\"e2-text-picture\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/first.jpg\" width=\"1440\" height=\"1080\" alt=\"\" \/>\n<\/div>\n<p>Also, on this data it is worth building a boxplot for more accurate visualization with the main distribution quantiles, median, mean and outliers.<\/p>\n<pre class=\"e2-text-code\"><code>fig = plt.figure(figsize = (10, 10))\r\nsns.set_theme(style=&quot;whitegrid&quot;)\r\nsns.set_palette(palette=&quot;pastel&quot;)\r\n\r\nsns.boxplot(y = city['population_2020'], showfliers = False)\r\n\r\nplt.scatter(0, 550100, marker='*', s=100, color = 'black', label = 'Outlier')\r\nplt.scatter(0, 560200, marker='*', s=100, color = 'black')\r\nplt.scatter(0, 570300, marker='*', s=100, color = 'black')\r\nplt.scatter(0, mean_pop, marker='o', s=100, color = 'red', edgecolors = 'black', label = 'Mean')\r\nplt.legend()\r\n\r\nplt.ylabel(&quot;Population&quot;, fontsize=15)\r\nplt.ticklabel_format(axis='y', style='plain')\r\nplt.title(&quot;Boxplot of population of russian cities&quot;, fontsize=15)\r\nplt.show()<\/code><\/pre><div class=\"e2-text-picture\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/second.jpg\" width=\"720\" height=\"720\" alt=\"\" \/>\n<\/div>\n<p>It follows from the graphs that the median is significantly less than the average, and it is also clear that this is a consequence of the presence of outliers which are Moscow and St. Petersburg. Since the arithmetic mean is an extremely sensitive metric to outliers, it is not worth relying on conclusions about the mean if they are present in the sample. An increase or decrease in the population of Moscow can greatly change the average population in Russia, but this will not affect the real regional trend.<br \/>\nUsing the arithmetic mean, we say that the number of a typical (average) city in the Russian Federation is 268 thousand people. However, this misleads us, since the mean value is significantly higher than the median solely due to the size of the population of Moscow and St. Petersburg. In fact, the number of a typical Russian city is significantly less (2 times less!) and is approximately 104 thousand citizens.<\/p>\n<h2>Marketing tasks<\/h2>\n<p>In a business tasks, the difference between the mean and the median is also important, as using the wrong metric can seriously affect the results of the advertising campaign or make it difficult to achieve the goal. Let’s take a look at a real example of the difficulties an entrepreneur can face in retail if he chooses the wrong target metric.<br \/>\nTo begin with, as in the previous example, let’s load a dataset about supermarket purchases. Let’s select the dataset columns necessary for analysis and rename them to simplify the code in the future. Since this data is not as well prepared as the previous ones, it is necessary to group all purchased items by receipts. In this case, it is necessary to group by two variables: by the customer’s id and by the date of purchase (the date and time is determined by the moment of closing the bill, therefore, all purchases within one bill coincide by date variable). Then, let’s name the resulting column “total_bill”, that is, the check amount and calculate the average and median.<\/p>\n<pre class=\"e2-text-code\"><code>df = pd.read_excel ('invoice_data.xlsx')\r\ndf.columns = ['user', 'total_price', 'date']\r\ngroupped_df = pd.DataFrame (df.groupby (['user', 'date']). total_price.sum ())\r\ngroupped_df.columns = ['total_bill']\r\nmean_bill = groupped_df.total_bill.mean ()\r\nmedian_bill = groupped_df.total_bill.median ()<\/code><\/pre><p>Now, as in the previous example, you need to plot the distribution of customer checks and boxplot, and also display the median and mean on each of them.<\/p>\n<pre class=\"e2-text-code\"><code>sns.set_palette('rainbow')\r\nfig = plt.figure(figsize = (20, 15))\r\nax = fig.add_subplot(1, 1, 1)\r\nsns.histplot(groupped_df, x = 'total_bill', binwidth=200, alpha=0.6, ax=ax)\r\nplt.xlabel(&quot;Purchases&quot;, fontsize=20)\r\nplt.ylabel(&quot;Total bill&quot;, fontsize=20)\r\nplt.title(&quot;Distribution of total bills&quot;, fontsize=20)\r\nplt.axvline(mean_bill, linewidth=2, color='r', alpha=1, linestyle='--', label = 'Mean = {:.0f}'.format(mean_bill))\r\nplt.axvline(median_bill, linewidth=2, color='darkgreen', alpha=1, linestyle='--', label = 'Median = {:.0f}'.format(median_bill))\r\nplt.legend(fontsize=&quot;xx-large&quot;)\r\nplt.show()<\/code><\/pre><div class=\"e2-text-picture\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/third.jpg\" width=\"1440\" height=\"1080\" alt=\"\" \/>\n<\/div>\n<pre class=\"e2-text-code\"><code>fig = plt.figure(figsize = (10, 10))\r\nsns.set_theme(style=&quot;whitegrid&quot;)\r\nsns.set_palette(palette=&quot;pastel&quot;)\r\n\r\nsns.boxplot(y = groupped_df['total_bill'], showfliers = False)\r\n\r\nplt.scatter(0, 1800, marker='*', s=100, color = 'black', label = 'Outlier')\r\nplt.scatter(0, 1850, marker='*', s=100, color = 'black')\r\nplt.scatter(0, 1900, marker='*', s=100, color = 'black')\r\nplt.scatter(0, mean_bill, marker='o', s=100, color = 'red', edgecolors = 'black', label = 'Mean')\r\nplt.legend()\r\n\r\nplt.ticklabel_format(axis='y', style='plain')\r\nplt.ylabel(&quot;Total bill&quot;, fontsize=15)\r\nplt.title(&quot;Boxplot of total bills&quot;, fontsize=15)\r\nplt.show()<\/code><\/pre><div class=\"e2-text-picture\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/forth.jpg\" width=\"720\" height=\"720\" alt=\"\" \/>\n<\/div>\n<p>The graphs show that the distribution is different from normal, which means that the median and mean are not equal. The median value is smaller than the average by about 220 rubles.<br \/>\nNow, imagine that marketers have a task to increase the average buyer’s bill. A marketer may decide that since the average check is 601 rubles, the following promotion can be offered: “All buyers who make a purchase over 600 rubles, will get 20% discount on any good for 100 rubles.” In general, it is a reasonable offer, however, in reality, the average check is lower – 378 rubles. Thus, the majority of buyers will not be interested in the offer, since their purchase usually does not reach the proposed threshold. This means, that they will not take advantage of the offer and will not receive a discount, and the company will not be able to achieve its goal and increase the profit. The point is that the initial assumptions were wrong.<\/p>\n<h2>Conclusions<\/h2>\n<p>As you already understood, the mean often shows a more pleasant result, both for business and for research tasks, because it is always nicer to imagine the situation with the average check or the demographic situation in the country better than it really is. However, one must always remember about the shortcomings of mean in order to be able to correctly choose the appropriate analogue for assessing a particular situation.<\/p>\n",
            "date_published": "2021-10-27T14:20:56+03:00",
            "date_modified": "2021-10-27T14:20:52+03:00",
            "image": "https:\/\/en.leftjoin.ru\/pictures\/first.jpg",
            "_date_published_rfc2822": "Wed, 27 Oct 2021 14:20:56 +0300",
            "_rss_guid_is_permalink": "false",
            "_rss_guid": "71",
            "_e2_data": {
                "is_favourite": false,
                "links_required": [
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css"
                ],
                "og_images": [
                    "https:\/\/en.leftjoin.ru\/pictures\/first.jpg",
                    "https:\/\/en.leftjoin.ru\/pictures\/second.jpg",
                    "https:\/\/en.leftjoin.ru\/pictures\/third.jpg",
                    "https:\/\/en.leftjoin.ru\/pictures\/forth.jpg"
                ]
            }
        },
        {
            "id": "70",
            "url": "https:\/\/en.leftjoin.ru\/all\/graph-of-telegram-channels-related-to-analytics\/",
            "title": "Graph of telegram channels related to analytics",
            "content_html": "<p><a href=\"https:\/\/leftjoin.ru\/files\/analytics-graph.html\" style=\"text-decoration:none; border-line\"> <img src = \"https:\/\/leftjoin.ru\/pictures\/graph.png \"border =\" 0 \"width =\" 100% \"height =\" 150% \"> <\/a><\/p>\n<p>The authors of various Telegram-blogs often publish a selection of their favorite channels so as to share their professional choice with their audience. The idea, of course, is not new, but I decided not just to make a rating of interesting analytical telegram blogs, but to solve this problem analytically.<\/p>\n<p>As part of the current course of my studies, I am learning many modern approaches to data analysis and visualization. At the very beginning of the course, there was a warm-up exercise: object-oriented programming in Python for collecting and iterative building a graph with <a href=\"https:\/\/www.themoviedb.org\/documentation\/api\">TMDB API<\/a>. Usually, this method is used to construct a connection graph of actors, where the connection is a role in the same film. However, I decided that I could apply it to another problem: building a graph of connections for the analytic community.<\/p>\n<p>Since my time has been particularly limited recently, and I have already completed a similar task for the course, I decided to transfer this knowledge to someone else who is interested in analytics. Fortunately, at that moment, a candidate for the vacancy of a junior data analyst, Andrey, texted me in direct messages. He is now in the process of comprehending all the intricacies of analytics, so we agreed on an internship, in which Andrey parsed data from telegram channels.<\/p>\n<p>Andrey’s main task was to collect all texts from the Internet analyst’s telegram channel, select the channels, which Aleksey Nikushin linked, collect texts from these telegram channels and links on these channels. “Link” means any mention of the channel: through @, through a link or repost. As a result of parsing, Andrey got two files: nodes and edges.<br \/>\nNow I will present to you <a href=\"https:\/\/leftjoin.ru\/files\/analytics-graph.html\">the graph that I got based on this data<\/a> and comment on the results.<\/p>\n<p>I would like to take this opportunity to express my compliments to the karpov.courses team, as Andrey has excellent knowledge of the Python language!<\/p>\n<p>As a result, the top 10 channels in terms of degree (number of connections) looks like this:<\/p>\n<ol start=\"1\">\n<li><a href=\"https:\/\/t.me\/internetanalytics\">Интернет-аналитика<\/a><\/li>\n<li><a href=\"https:\/\/t.me\/revealthedata\">Reveal The Data<\/a><\/li>\n<li><a href=\"https:\/\/t.me\/rockyourdata\">Инжиниринг Данных<\/a><\/li>\n<li><a href=\"https:\/\/t.me\/data_events\">Data Events<\/a><\/li>\n<li><a href=\"https:\/\/t.me\/datalytx\">Datalytics<\/a><\/li>\n<li><a href=\"https:\/\/t.me\/chartomojka\">Чартомойка<\/a><\/li>\n<li><a href=\"https:\/\/t.me\/leftjoin\">LEFT JOIN<\/a><\/li>\n<li><a href=\"https:\/\/t.me\/epicgrowth_chat\">Epic Growth<\/a><\/li>\n<li><a href=\"https:\/\/t.me\/rtdlinks\">RTD: ссылки и репосты<\/a><\/li>\n<li><a href=\"https:\/\/t.me\/dashboardets\">Дашбордец<\/a><\/li>\n<\/ol>\n<p>In my opinion, it turned out extremely exciting and visually interesting, and Andrey is a great fellow! By the way, he also started his own channel <a href=\"https:\/\/t.me\/eto_analytica\">”Это разве аналитика?”<\/a>, where analytics news is published.<\/p>\n<p>Looking ahead, this problem has a continuation. With the help of the Markov chain, we modeled where the user ends up if he iteratively navigates through all the mentions in the channels. It turned out unexpectedly, but we will tell about it next time!<\/p>\n",
            "date_published": "2021-10-20T14:03:07+03:00",
            "date_modified": "2021-10-20T14:02:51+03:00",
            "_date_published_rfc2822": "Wed, 20 Oct 2021 14:03:07 +0300",
            "_rss_guid_is_permalink": "false",
            "_rss_guid": "70",
            "_e2_data": {
                "is_favourite": false,
                "links_required": [],
                "og_images": []
            }
        },
        {
            "id": "69",
            "url": "https:\/\/en.leftjoin.ru\/all\/bubble-charts-basics-area-vs-radius\/",
            "title": "Bubble charts basics: area vs radius",
            "content_html": "<p>Data visualization is a skill used in any industry where data is present, because tables are only good for storing information. When there is a need to present data, or rather certain conclusions derived from them, the data must be presented on graphs of a suitable type. So, here you are faced with two tasks: the first is to choose the right type of graph, the second is to display the results in a plausible way. Today we will tell you about one mistake that designers sometimes make when visualizing data on bubble-charts and how this mistake can be avoided.<\/p>\n<h2>The crux of building a bubble-chart<\/h2>\n<p>Firstly, let us tell you a bit of boring theory before we start analyzing the data. Bubble-chart is a convenient way to show three numerical variables without building a 3D model. The usual X and Y axes indicate the values ​​of two parameters, and the third is shown by the size of the circle that corresponds to each observation. This is what makes it possible to avoid the need to build a complex 3D chart, that is, anyone who sees a bubble-chart will be able to draw conclusions about the data much faster.<\/p>\n<h2>A mistake that a designer, but not a data analyst, can make<\/h2>\n<p>With the metrics that are displayed on the axes of the graph, no questions arise. This is the usual way of visualizing data, but with the data shown by the size of the circles there is some difficulty: how to correctly and accurately display changes in the values ​​of a variable, if the control is not a point on the axis, but the size of this point?<br \/>\nThe fact is that when building such a graph without using analytical tools, for example, in a graphics editor, the author can draw circles, taking the radius of the circle as its size. At first glance, everything seems to be absolutely correct – the larger the value of the variable, the larger the radius of the circle. However, in this case, the area of ​​the circle will increase not as a linear, but as a power function, because S = π × r2. For instance, the figure below shows that if you double the radius of a circle, the area will quadruple.<\/p>\n<p><details><br \/>\n<summary> <span style = \"color: # 7ea9b8\"> Draw a circle in Matplotlib <\/span> <\/summary><\/p>\n<pre class=\"e2-text-code\"><code>fig = plt.figure (figsize = (10, 10))\r\nax = fig.add_subplot (1, 1, 1)\r\ns = 4 * 10e3\r\n\r\n\r\nax.scatter (100, 100, s = s, c = 'r')\r\nax.scatter (100, 100, s = s \/ 4, c = 'b')\r\nax.scatter (100, 100, s = 10, c = 'g')\r\nplt.axvline (99, c = 'black')\r\nplt.axvline (101, c = 'black')\r\nplt.axvline (98, c = 'black')\r\nplt.axvline (102, c = 'black')\r\n\r\n\r\nax.set_xticks (np.arange (95, 106, 1))\r\nax.grid (alpha = 1)\r\n\r\nplt.show ()<\/code><\/pre><p><\/details><\/p>\n<div class=\"e2-text-picture\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/example.png\" width=\"720\" height=\"720\" alt=\"\" \/>\n<\/div>\n<p>This means that the graph will look implausible, because the dimensions will not reflect the real change in the variable, and the viewer pays attention and compares exactly the area of ​​the circles on the graph.<\/p>\n<h2>How to build such a graph correctly?<\/h2>\n<p>Fortunately, if you build bubble-charts using Python libraries (Matplotlib and Seaborn), then the size of the circle will be determined by the area, which is absolutely correct from in terms of visualization.<br \/>\nNow, using the example of real data found on Kaggle, we will show how to build a bubble-chart. The data contains the following variables: country, population size, percentage of literate population. For the chart to be readable, let’s take a subsample of the top 10 countries after sorting all the data in order of increasing GDP.<\/p>\n<p>First, let’s load all the necessary libraries:<\/p>\n<pre class=\"e2-text-code\"><code>import pandas as pd\r\nimport matplotlib.pyplot as plt\r\nimport seaborn as sns<\/code><\/pre><p>Then, load the data, clear it from all rows with missing values ​​and transform the population of countries to millions of people:<\/p>\n<pre class=\"e2-text-code\"><code>data = pd.read_csv ('countries of the world.csv', sep = ',')\r\ndata = data.dropna ()\r\ndata = data.sort_values ​​(by = 'Population', ascending = False)\r\ndata = data.head (10)\r\ndata ['Population'] = data ['Population']. apply (lambda x: x \/ 1000000)<\/code><\/pre><p>Now that all the preparations are complete, you can build a bubble-chart:<\/p>\n<pre class=\"e2-text-code\"><code>sns.set (style = &quot;darkgrid&quot;)\r\nfig, ax = plt.subplots (figsize = (10, 10))\r\ng = sns.scatterplot (data = data, x = &quot;Literacy (%)&quot;, y = &quot;GDP ($ per capita)&quot;, size = &quot;Population&quot;, sizes = (10,1500), alpha = 0.5)\r\nplt.xlabel (&quot;Literacy (Percentage of literate citizens)&quot;)\r\nplt.ylabel (&quot;GDP per Capita&quot;)\r\nplt.title ('Chart with bubbles as area', fontdict = {'fontsize': 'x-large'})\r\n\r\ndef label_point (x, y, val, ax):\r\n    a = pd.concat ({'x': x, 'y': y, 'val': val}, axis = 1)\r\n    for i, point in a.iterrows ():\r\n        ax.text (point ['x'], point ['y'] + 500, str (point ['val']))\r\n\r\nlabel_point (data ['Literacy (%)'], data ['GDP ($ per capita)'], data ['Country'], plt.gca ())\r\n\r\nax.legend (loc = 'upper left', fontsize = 'medium', title = 'Population (in mln)', title_fontsize = 'large', labelspacing = 1)\r\n\r\nplt.show ()<\/code><\/pre><div class=\"e2-text-picture\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/_32.png\" width=\"720\" height=\"720\" alt=\"\" \/>\n<\/div>\n<p>This graph displays three metrics in an understandable way: the level of GDP per capita on the Y axis, the percentage of the literate population on the X axis, and the population – by the area of ​​the circle.<\/p>\n<p>We recommend using size of the circle to show one of the variables, if there is a need to show 3 or more variables on one chart.<\/p>\n",
            "date_published": "2021-10-14T17:58:24+03:00",
            "date_modified": "2021-10-14T17:58:05+03:00",
            "image": "https:\/\/en.leftjoin.ru\/pictures\/example.png",
            "_date_published_rfc2822": "Thu, 14 Oct 2021 17:58:24 +0300",
            "_rss_guid_is_permalink": "false",
            "_rss_guid": "69",
            "_e2_data": {
                "is_favourite": false,
                "links_required": [
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css"
                ],
                "og_images": [
                    "https:\/\/en.leftjoin.ru\/pictures\/example.png",
                    "https:\/\/en.leftjoin.ru\/pictures\/_32.png"
                ]
            }
        },
        {
            "id": "68",
            "url": "https:\/\/en.leftjoin.ru\/all\/export-jupyter-notebook-to-pdf\/",
            "title": "How and why should you export reports from Jupyter Notebook to PDF",
            "content_html": "<p>If you are a data analyst and you need to present a report to a client, if you are looking for a job and do not know how to draw up a test task in such a way that people will pay attention to you, if you have a lot of educational projects related to data analytics and visualization, this post will be very, very useful to you.<br \/>\nLooking at someone else’s code in a Jupyter Notebook can be problematic, because the result is often lost between lines of code with data preparation, importing the necessary libraries and a series of attempts to implement the idea. That is why a method such as exporting results to a PDF file in LaTeX format is a great option for final visualization. It will save time and look presentable. In scientific circles, articles and reports are very often formatted using LaTeX, since it has a number of advantages:<\/p>\n<ul>\n<li>Math equations and formulas look neater.<\/li>\n<li>The bibliography is automatically generated based on all references used in the document.<\/li>\n<li>The author can focus on the content (not on the appearance of the document), since the layout of the text and other data is set automatically by specifying the necessary parameters in the code.<\/li>\n<\/ul>\n<p>Today we will talk in detail about how to export such beautiful reports from Jupyter Notebook to PDF using LaTeX.<\/p>\n<p><a href=\"https:\/\/leftjoin.ru\/files\/sample_LaTeX_report.pdf\" style=\"text-decoration:none; border\"> <img src = \"https:\/\/leftjoin.ru\/pictures\/sample_latex_report.png \"border =\" 0 \"width =\" 150% \"height =\" 150% \"> <\/a><\/p>\n<h2>Installing LaTeX<\/h2>\n<p>The most important point in generating a report from a Jupyter Notebook in Python is exporting it to the final file. The main library you need to install is – <a href=\"https:\/\/nbconvert.readthedocs.io\/en\/latest\/install.html\">nbconvert<\/a> – which converts your notebook into any convenient document format: pdf (as in our case), html, latex, etc. This library needs not only to be installed, but some preinstalling of several other packages as well: Pandoc, TeX, and Chromium. According to the link to the library, the whole process is described in detail for each software, so we will not dwell on it here.<br \/>\nOnce you have completed all the preliminary steps, you need to install and import the library into your Jupyter Notebook.<\/p>\n<pre class=\"e2-text-code\"><code>! pip install nbconvert\r\nimport nbconvert<\/code><\/pre><h2>Export tables to Markdown format<\/h2>\n<p>Usually, tables look a bit odd in reports, as they can be difficult to read quickly, but sometimes it is still necessary to add a small table to the final document. In order for the table to look neat, you need to save it in Markdown format. This can be done manually, but if there is a lot of data in the table, it is better to come up with a more convenient method. We suggest using the following simple pandas_df_to_markdown_table () function, which converts any dataframe to a markdown-table. Note: after the conversion, indices disappear, therefore, if they are important (as in our example), it is worth saving them into a variable in the first column of the dataframe.<\/p>\n<pre class=\"e2-text-code\"><code>data_g = px.data.gapminder ()\r\nsummary = round (data_g.describe (), 2)\r\nsummary.insert (0, 'metric', summary.index)\r\n\r\n# Function to convert dataframe to Markdown Table\r\ndef pandas_df_to_markdown_table (df):\r\n    from IPython.display import Markdown, display\r\n    fmt = ['---' for i in range (len (df.columns))]\r\n    df_fmt = pd.DataFrame ([fmt], columns = df.columns)\r\n    df_formatted = pd.concat ([df_fmt, df])\r\n    display (Markdown (df_formatted.to_csv (sep = &quot;|&quot;, index = False)))\r\n\r\npandas_df_to_markdown_table (summary)<\/code><\/pre><h2>Export image to report<\/h2>\n<p>In this example, we will build a bubble-chart, the construction method of which was described <a href=\"https:\/\/leftjoin.ru\/all\/principy-postroeniya-bubble-charts-ploschad-vs-radius\/\">in a recent post<\/a>. Previously we used the Seaborn library, which shown that the display of data with the size of circles on the graph is correct. The same graphs can be created using the Plotly library.<br \/>\nIn order to display the plot in the final report, you also need to complete an additional step. The point is that plt.show () will not help to display the graph when exporting. Therefore, you need to save the graph in the working directory, and then, using the iPython.display library, display it using the Image () function.<\/p>\n<pre class=\"e2-text-code\"><code>from IPython.display import Image\r\nimport plotly.express as px\r\nfig = px.scatter (data_g.query (&quot;year == 2007&quot;), x = &quot;gdpPercap&quot;, y = &quot;lifeExp&quot;,\r\n                 size = &quot;pop&quot;, color = &quot;continent&quot;,\r\n                 log_x = True, size_max = 70)\r\nfig.write_image ('figure_1.jpg')\r\nImage (data = 'figure_1.jpg', width = 1000)<\/code><\/pre><h2>Formation and export of the report<\/h2>\n<p>When all stages of data analysis are completed, the report can be exported. If you need headings or text in the report, then write them in the cells of the notebook, changing the format from Code to Markdown. For export, you can use the terminal, running the second line there without an exclamation mark, or you can run the code written below in the cell of the Jupiter Notebook. We advise you not to load the report with code and use TemplateExporter.exclude_input = True parameter so that the cells with the code are not exported. Also, when you run this cell in your notebook, the code produces a standard output, and you need to write %% capture at the beginning of the cell not to export it.<\/p>\n<pre class=\"e2-text-code\"><code>%% capture\r\n! jupyter nbconvert --to pdf --TemplateExporter.exclude_input = True ~ \/ Desktop \/ VALIOTTI \/ Reports \/ Sample \\ LaTeX \\ Report.ipynb\r\n! open ~ \/ Desktop \/ VALIOTTI \/ Reports \/ Sample \\ LaTeX \\ Report.pdf<\/code><\/pre><p>If you did everything correctly and methodically, then you will end up with a report <a href=\"https:\/\/leftjoin.ru\/files\/sample_LaTeX_report.pdf\">similar to this one<\/a>!<br \/>\nPresent your data nicely :)<\/p>\n",
            "date_published": "2021-10-11T15:46:40+03:00",
            "date_modified": "2021-10-11T15:57:02+03:00",
            "_date_published_rfc2822": "Mon, 11 Oct 2021 15:46:40 +0300",
            "_rss_guid_is_permalink": "false",
            "_rss_guid": "68",
            "_e2_data": {
                "is_favourite": false,
                "links_required": [
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css"
                ],
                "og_images": []
            }
        },
        {
            "id": "67",
            "url": "https:\/\/en.leftjoin.ru\/all\/newborn\/",
            "title": "Dashboard for the first 8 months of a child’s life",
            "content_html": "<p><a href=\"https:\/\/en.leftjoin.ru\/tableau\/newborn.html\" style=\"text-decoration:none; border:0\"><img src=\"https:\/\/en.leftjoin.ru\/pictures\/newborn-1.png-4.jpg\" border=\"0\" width=\"100%\" height=\"150%\"><\/a><\/p>\n<p>In December 2020, I became a dad, which means that our family life with my wife has changed drastically. Of course, I am sharing this news with you for a reason, but in the context of the data that we will study and research today. They are very personal for me, and therefore have some special magic and value. Today I want to show how dramatically the life of a family is changing by the example of my own analysis of the life data of the first 8 months of a baby.<\/p>\n<h2>Data collection<\/h2>\n<p>Initial data: tracking the main elements of caring for a baby in the first 8 months: sleep, nursing, changing a diaper. The data was collected using <a href=\"https:\/\/nighp.com\/babytracker\/\">BabyTracker app<\/a>.<br \/>\nMy wife is a great fellow, because during the first 7 months she carefully and regularly monitored all the important points. She forgot to turn off the timer for nursing the baby at night only a couple of times, but I quickly saw noticeable outliers in the data, and the dataset was cleared of them.<br \/>\nInitially, I had several data visualization ideas in my head, and I tried to immediately implement them into the projected dashboard. I wanted to show the baby’s sleep intervals in the form of a vertical Gantt chart, but the night’s sleep went through the midnight (0:00), and it was completely incomprehensible how this could be corrected in Tableau. After a number of unsuccessful independent attempts to find a solution to this problem, I decided to consult with <a href=\"https:\/\/t.me\/revealthedata\">Roman Bunin<\/a>. Unfortunately, we came to the conclusion together that there is no way to solve this. Then I had to write a little Python code that splits such time intervals and adds new lines to the dataset.<br \/>\nHowever, while we were texting, Roma sent an example identical to my idea! This example claims that a woman collected data on her child’s sleep and wakefulness in the first year of child’s life, and then wrote the code with which it turned out to be embroidered <a href=\"https:\/\/youtu.be\/2R3dXARPH10?t=1723\">towel with pattern datavis baby sleep<\/a>. For me, this was surprising, since it turned out that this way of visualization is the main method that allows you to show how difficult life and sleep of parents is in the first months of the birth of a child.<br \/>\nIn my <a href=\"https:\/\/en.leftjoin.ru\/tableau\/newborn.html\">dashboard on Tableau Public<\/a> I got three semantic blocks and several “KPIs” about which I would like to tell you in detail and share the basic everyday wisdom. At the top of the dashboard, you can see the key averages of the daytime and nighttime sleep hours, nursing hours, frequency of nursing, and the number of diaper changes in the first three months. I have allocated exactly three months, because I think this is the most difficult period, because significant changes that require serious adaptation are taking place in your life .<\/p>\n<h2>Dream<\/h2>\n<p>The left diagram – called “Towel” – illustrates the baby’s sleeping periods. In this diagram, it is important to pay attention to white gaps, especially at night. These are the hours when the baby is awake, which means that the parents are also awake. Look at how the chart changes, especially in the early months, when we gave up the habit of going to bed at 1 or 2 in the morning and fell asleep earlier. Roughly speaking, in the first three months (until March 2021), the child could fall asleep at 2 or 3 in the morning, but we were lucky that our child’s night sleep was quite long.<br \/>\nThe right graph clearly illustrates how the baby’s day and night sleep length changes over time, and the boxplots show the distribution of the hours of daytime and nighttime sleep. The graph confirms the conclusion: “This is temporary and will definitely get better soon!”<\/p>\n<h2>Nursing<\/h2>\n<p>From the left diagram, you can see how the number and duration of nursing change. This number is gradually decreasing, and the duration of nursing periods is shortened. Since mid-July, we have changed the way we track nursing periods, so they are not valid for this analysis.<br \/>\nFrom my point of view, the findings are a great opportunity for couples planning a pregnancy, not to create illusions about the opportunity to work or do any other business in the first months after giving birth. Pay attention to the frequency and duration of nursing, all this time the parent is completely busy with the child. However, do not be overly alarmed: over time, the number of nursing periods will decrease.<\/p>\n<h2>Diaper change<\/h2>\n<p>The left graph is the highlight of this dashboard. As you can imagine, this is a map of the most fun moments – changing a diaper. The stars represent the moments of the day when you need to change the diaper, and the light gray color below shows the number of changes per day. The graph on the right shows diaper changes counted by part of the day. In general, the diagram does not show any interesting dependencies, however, it prepares you for the fact that this process is frequent, regular and happens at any time of the day.<\/p>\n<h2>Conclusions<\/h2>\n<p>It seems to me that the use of real personal data and such visualization is sometimes much more revealing than a lot of videos or books about what this period will be like. That is why I decided to share my findings and observations with you here. The main conclusion that I wanted you to draw from the dataviz: children are great! ❤️<\/p>\n",
            "date_published": "2021-09-29T15:52:15+03:00",
            "date_modified": "2021-09-29T15:46:04+03:00",
            "image": "https:\/\/en.leftjoin.ru\/pictures\/newborn-1.png-4.jpg",
            "_date_published_rfc2822": "Wed, 29 Sep 2021 15:52:15 +0300",
            "_rss_guid_is_permalink": "false",
            "_rss_guid": "67",
            "_e2_data": {
                "is_favourite": false,
                "links_required": [],
                "og_images": [
                    "https:\/\/en.leftjoin.ru\/pictures\/newborn-1.png-4.jpg"
                ]
            }
        },
        {
            "id": "66",
            "url": "https:\/\/en.leftjoin.ru\/all\/python-and-lyrics-of-zemfiras-new-album-capturing-the-spirit-of\/",
            "title": "Python and lyrics of Zemfira’s new album: capturing the spirit of her songs",
            "content_html": "<div class=\"e2-text-picture\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/header.jpg\" width=\"600\" height=\"600\" alt=\"\" \/>\n<\/div>\n<p>Zemfira’s latest studio album, Borderline, was released in February, 8 years after the previous one. For this album, various people cooperated with her, including her relatives – the riff for the song “Таблетки” was written by her nephew from London. The album turned out to be diverse: for instance, the song “Остин” is dedicated to the main character of the Homescapes game by the Russian studio Playrix (by the way, check out the latest <a href=\"https:\/\/youtu.be\/SOx8afEUTnE\">Business Secrets with the Bukhman brothers<\/a>, they also mention it there). Zemfira liked the game a lot, thus, she contacted Playrix to create this song. Also, the song “Крым” was written as a soundtrack to a new film by Zemfira’s colleague Renata Litvinova.<\/p>\n<p class=\"note\">Listen new album in <a href=\"https:\/\/music.apple.com\/ru\/album\/бордерлайн\/1554865105\">Apple Music<\/a> \/ <a href=\"https:\/\/music.yandex.ru\/album\/14052981\">Яндекс.Музыке<\/a> \/ <a href=\"https:\/\/open.spotify.com\/album\/6khBsXmKA1FKjYVCIBy9kt\">Spotify<\/a><\/p>\n<p>Nevertheless, the spirit of the whole album is rather gloomy – the songs often repeat the words ‘боль’, ‘ад’, ‘бесишь’ and other synonyms. We decided to conduct an exploratory analysis of her album, and then, using the Word2Vec model and a cosine measure, look at the semantic closeness of the songs and calculate the general mood of the album.<\/p>\n<p>For those who are bored with reading about data preparation and analysis steps, you can <a href=\"https:\/\/leftjoin.ru\/all\/borderline-text-analysis\/#result\">go directly to the results<\/a>.<\/p>\n<h2>Data preparation<\/h2>\n<p>For starters, we write a data processing script. The purpose of the script is to collect a united csv-table from a set of text files, each of which contains a song. At the same time, we have to get rid of all punctuation marks and unnecessary words as we need to focus only on significant content.<\/p>\n<pre class=\"e2-text-code\"><code>import pandas as pd\r\nimport re\r\nimport string\r\nimport pymorphy2\r\nfrom nltk.corpus import stopwords<\/code><\/pre><p>Then we create a morphological analyzer and expand the list of everything that needs to be discarded:<\/p>\n<pre class=\"e2-text-code\"><code>morph = pymorphy2.MorphAnalyzer()\r\nstopwords_list = stopwords.words('russian')\r\nstopwords_list.extend(['куплет', 'это', 'я', 'мы', 'ты', 'припев', 'аутро', 'предприпев', 'lyrics', '1', '2', '3', 'то'])\r\nstring.punctuation += '—'<\/code><\/pre><p>The names of the songs are given in English, so we have to create a dictionary for translation into Russian and a dictionary, from which we will later make a table:<\/p>\n<pre class=\"e2-text-code\"><code>result_dict = dict()\r\n\r\nsongs_dict = {\r\n    'snow':'снег идёт',\r\n    'crimea':'крым',\r\n    'mother':'мама',\r\n    'ostin':'остин',\r\n    'abuse':'абьюз',\r\n    'wait_for_me':'жди меня',\r\n    'tom':'том',\r\n    'come_on':'камон',\r\n    'coat':'пальто',\r\n    'this_summer':'этим летом',\r\n    'ok':'ок',\r\n    'pills':'таблетки'\r\n}<\/code><\/pre><p>Let’s define several necessary functions. The first one reads the entire song from the file and removes line breaks, the second clears the text from unnecessary characters and words, and the third one converts the words to normal form, using the pymorphy2 morphological analyzer. The pymorphy2 module does not always handle ambiguity well – additional processing is required for the words ‘ад’ and ‘рай’.<\/p>\n<pre class=\"e2-text-code\"><code>def read_song(filename):\r\n    f = open(f'{filename}.txt', 'r').read()\r\n    f = f.replace('\\n', ' ')\r\n    return f\r\n\r\ndef clean_string(text):\r\n    text = re.split(' |:|\\.|\\(|\\)|,|&quot;|;|\/|\\n|\\t|-|\\?|\\[|\\]|!', text)\r\n    text = ' '.join([word for word in text if word not in string.punctuation])\r\n    text = text.lower()\r\n    text = ' '.join([word for word in text.split() if word not in stopwords_list])\r\n    return text\r\n\r\ndef string_to_normal_form(string):\r\n    string_lst = string.split()\r\n    for i in range(len(string_lst)):\r\n        string_lst[i] = morph.parse(string_lst[i])[0].normal_form\r\n        if (string_lst[i] == 'аду'):\r\n            string_lst[i] = 'ад'\r\n        if (string_lst[i] == 'рая'):\r\n            string_lst[i] = 'рай'\r\n    string = ' '.join(string_lst)\r\n    return string<\/code><\/pre><p>After all this preparation, we can get back to the data and process each song and read the file with the corresponding name:<\/p>\n<pre class=\"e2-text-code\"><code>name_list = []\r\ntext_list = []\r\nfor song, name in songs_dict.items():\r\n    text = string_to_normal_form(clean_string(read_song(song)))\r\n    name_list.append(name)\r\n    text_list.append(text)<\/code><\/pre><p>Then we combine everything into a DataFrame and save it as a csv-file.<\/p>\n<pre class=\"e2-text-code\"><code>df = pd.DataFrame()\r\ndf['name'] = name_list\r\ndf['text'] = text_list\r\ndf['time'] = [290, 220, 187, 270, 330, 196, 207, 188, 269, 189, 245, 244]\r\ndf.to_csv('borderline.csv', index=False)<\/code><\/pre><p>Result:<\/p>\n<div class=\"e2-text-picture\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/2-table.png\" width=\"477\" height=\"365\" alt=\"\" \/>\n<\/div>\n<h2>Word cloud for the whole album<\/h2>\n<p>To begin with the analysis, we have to construct a word cloud, because it can display the most common words found in these songs. We import the required libraries, read the csv-file and set the configurations:<\/p>\n<pre class=\"e2-text-code\"><code>import nltk\r\nfrom wordcloud import WordCloud\r\nimport pandas as pd\r\nimport matplotlib.pyplot as plt\r\nfrom nltk import word_tokenize, ngrams\r\n\r\n%matplotlib inline\r\nnltk.download('punkt')\r\ndf = pd.read_csv('borderline.csv')<\/code><\/pre><p>Now we create a new figure, set the design parameters and, using the word cloud library, display words with the size directly proportional to the frequency of the word. We additionally indicate the name of the song above the corresponding graph.<\/p>\n<pre class=\"e2-text-code\"><code>fig = plt.figure()\r\nfig.patch.set_facecolor('white')\r\nplt.subplots_adjust(wspace=0.3, hspace=0.2)\r\ni = 1\r\nfor name, text in zip(df.name, df.text):\r\n    tokens = word_tokenize(text)\r\n    text_raw = &quot; &quot;.join(tokens)\r\n    wordcloud = WordCloud(colormap='PuBu', background_color='white', contour_width=10).generate(text_raw)\r\n    plt.subplot(4, 3, i, label=name,frame_on=True)\r\n    plt.tick_params(labelsize=10)\r\n    plt.imshow(wordcloud)\r\n    plt.axis(&quot;off&quot;)\r\n    plt.title(name,fontdict={'fontsize':7,'color':'grey'},y=0.93)\r\n    plt.tick_params(labelsize=10)\r\n    i += 1<\/code><\/pre><div class=\"e2-text-picture\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/3-wordcloud.jpg\" width=\"2560\" height=\"1707\" alt=\"\" \/>\n<\/div>\n<h2>EDA of the lyrics<\/h2>\n<p>Let us move to the next part and analyze the lyrics. To do this, we have to import special libraries to deal with data and visualization:<\/p>\n<pre class=\"e2-text-code\"><code>import plotly.graph_objects as go\r\nimport plotly.figure_factory as ff\r\nfrom scipy import spatial\r\nimport collections\r\nimport pymorphy2\r\nimport gensim\r\n\r\nmorph = pymorphy2.MorphAnalyzer()<\/code><\/pre><p>Firstly, we should count the overall number of words in each song, the number of unique words, and their percentage:<\/p>\n<pre class=\"e2-text-code\"><code>songs = []\r\ntotal = []\r\nuniq = []\r\npercent = []\r\n\r\nfor song, text in zip(df.name, df.text):\r\n    songs.append(song)\r\n    total.append(len(text.split()))\r\n    uniq.append(len(set(text.split())))\r\n    percent.append(round(len(set(text.split())) \/ len(text.split()), 2) * 100)<\/code><\/pre><p>All this information should be written in a DataFrame and additionally we want to count the number of words per minute for each song:<\/p>\n<pre class=\"e2-text-code\"><code>df_words = pd.DataFrame()\r\ndf_words['song'] = songs\r\ndf_words['total words'] = total\r\ndf_words['uniq words'] = uniq\r\ndf_words['percent'] = percent\r\ndf_words['time'] = df['time']\r\ndf_words['words per minute'] = round(total \/ (df['time'] \/\/ 60))\r\ndf_words = df_words[::-1]<\/code><\/pre><div class=\"e2-text-picture\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/4-table.png\" width=\"480\" height=\"369\" alt=\"\" \/>\n<\/div>\n<p>It would be great to visualize the data, so let us build two bar charts: one for the number of words in the song, and the other one for the number of words per minute.<\/p>\n<pre class=\"e2-text-code\"><code>colors_1 = ['rgba(101,181,205,255)'] * 12\r\ncolors_2 = ['rgba(62,142,231,255)'] * 12\r\n\r\nfig = go.Figure(data=[\r\n    go.Bar(name='📝 Total number of words,\r\n           text=df_words['total words'],\r\n           textposition='auto',\r\n           x=df_words.song,\r\n           y=df_words['total words'],\r\n           marker_color=colors_1,\r\n           marker=dict(line=dict(width=0)),),\r\n    go.Bar(name='🌀 Unique words',\r\n           text=df_words['uniq words'].astype(str) + '&lt;br&gt;'+ df_words.percent.astype(int).astype(str) + '%' ,\r\n           textposition='inside',\r\n           x=df_words.song,\r\n           y=df_words['uniq words'],\r\n           textfont_color='white',\r\n           marker_color=colors_2,\r\n           marker=dict(line=dict(width=0)),),\r\n])\r\n\r\nfig.update_layout(barmode='group')\r\n\r\nfig.update_layout(\r\n    title = \r\n        {'text':'&lt;b&gt;The ratio of the number of unique words to the total&lt;\/b&gt;&lt;br&gt;&lt;span style=&quot;color:#666666&quot;&gt;&lt;\/span&gt;'},\r\n    showlegend = True,\r\n    height=650,\r\n    font={\r\n        'family':'Open Sans, light',\r\n        'color':'black',\r\n        'size':14\r\n    },\r\n    plot_bgcolor='rgba(0,0,0,0)',\r\n)\r\nfig.update_layout(legend=dict(\r\n    yanchor=&quot;top&quot;,\r\n    xanchor=&quot;right&quot;,\r\n))\r\n\r\nfig.show()<\/code><\/pre><iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https:\/\/plotly.com\/~Elisejj\/96.embed?showlink=false\" height=\"650\" width=\"100%\"><\/iframe>\n<pre class=\"e2-text-code\"><code>colors_1 = ['rgba(101,181,205,255)'] * 12\r\ncolors_2 = ['rgba(238,85,59,255)'] * 12\r\n\r\nfig = go.Figure(data=[\r\n    go.Bar(name='⏱️ Track length, min.',\r\n           text=round(df_words['time'] \/ 60, 1),\r\n           textposition='auto',\r\n           x=df_words.song,\r\n           y=-df_words['time'] \/\/ 60,\r\n           marker_color=colors_1,\r\n           marker=dict(line=dict(width=0)),\r\n          ),\r\n    go.Bar(name='🔄 Words per minute',\r\n           text=df_words['words per minute'],\r\n           textposition='auto',\r\n           x=df_words.song,\r\n           y=df_words['words per minute'],\r\n           marker_color=colors_2,\r\n           textfont_color='white',\r\n           marker=dict(line=dict(width=0)),\r\n          ),\r\n])\r\n\r\nfig.update_layout(barmode='overlay')\r\n\r\nfig.update_layout(\r\n    title = \r\n        {'text':'&lt;b&gt;Track length and words per minute&lt;\/b&gt;&lt;br&gt;&lt;span style=&quot;color:#666666&quot;&gt;&lt;\/span&gt;'},\r\n    showlegend = True,\r\n    height=650,\r\n    font={\r\n        'family':'Open Sans, light',\r\n        'color':'black',\r\n        'size':14\r\n    },\r\n    plot_bgcolor='rgba(0,0,0,0)'\r\n)\r\n\r\n\r\nfig.show()<\/code><\/pre><iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https:\/\/plotly.com\/~Elisejj\/98.embed?showlink=false\" height=\"650\" width=\"100%\"><\/iframe>\n<h2>Working with Word2Vec model<\/h2>\n<p>Using the gensim module, load the model pointing to a binary file:<\/p>\n<pre class=\"e2-text-code\"><code>model = gensim.models.KeyedVectors.load_word2vec_format('model.bin', binary=True)<\/code><\/pre><p class=\"note\">Для материала мы использовали готовую обученную на Национальном Корпусе Русского Языка модель от сообщества <a href=\"https:\/\/rusvectores.org\/ru\/models\/\">RusVectōrēs<\/a><\/p>\n<p>The Word2Vec model is based on neural networks and allows you to represent words in the form of vectors, taking into account the semantic component. It means that if we take two words – for instance, “mom” and “dad”, then represent them as two vectors and calculate the cosine, the values ​​will be close to 1. Similarly, two words that have nothing in common in their meaning have a cosine measure close to 0.<\/p>\n<p>Now we will define the get_vector function: it will take a list of words, recognize a part of speech for each word, and then receive and summarize vectors, so that we can find vectors even for whole sentences and texts.<\/p>\n<pre class=\"e2-text-code\"><code>def get_vector(word_list):\r\n    vector = 0\r\n    for word in word_list:\r\n        pos = morph.parse(word)[0].tag.POS\r\n        if pos == 'INFN':\r\n            pos = 'VERB'\r\n        if pos in ['ADJF', 'PRCL', 'ADVB', 'NPRO']:\r\n            pos = 'NOUN'\r\n        if word and pos:\r\n            try:\r\n                word_pos = word + '_' + pos\r\n                this_vector = model.word_vec(word_pos)\r\n                vector += this_vector\r\n            except KeyError:\r\n                continue\r\n    return vector<\/code><\/pre><p>For each song, find a vector and select the corresponding column in the DataFrame:<\/p>\n<pre class=\"e2-text-code\"><code>vec_list = []\r\nfor word in df['text']:\r\n    vec_list.append(get_vector(word.split()))\r\ndf['vector'] = vec_list<\/code><\/pre><p>So, now we should compare these vectors with one another, calculating their cosine proximity. Those songs with a cosine metric higher than 0.5 will be saved separately – this way we will get the closest pairs of songs. We will write the information about the comparison of vectors into the two-dimensional list result.<\/p>\n<pre class=\"e2-text-code\"><code>similar = dict()\r\nresult = []\r\nfor song_1, vector_1 in zip(df.name, df.vector):\r\n    sub_list = []\r\n    for song_2, vector_2 in zip(df.name.iloc[::-1], df.vector.iloc[::-1]):\r\n        res = 1 - spatial.distance.cosine(vector_1, vector_2)\r\n        if res &gt; 0.5 and song_1 != song_2 and (song_1 + ' \/ ' + song_2 not in similar.keys() and song_2 + ' \/ ' + song_1 not in similar.keys()):\r\n            similar[song_1 + ' \/ ' + song_2] = round(res, 2)\r\n        sub_list.append(round(res, 2))\r\n    result.append(sub_list)<\/code><\/pre><p>Moreover, we can construct the same bar chart:<\/p>\n<pre class=\"e2-text-code\"><code>df_top_sim = pd.DataFrame()\r\ndf_top_sim['name'] = list(similar.keys())\r\ndf_top_sim['value'] = list(similar.values())\r\ndf_top_sim.sort_values(by='value', ascending=False)<\/code><\/pre><p>И построим такой же bar chart:<\/p>\n<pre class=\"e2-text-code\"><code>colors = ['rgba(101,181,205,255)'] * 5\r\n\r\nfig = go.Figure([go.Bar(x=df_top_sim['name'],\r\n                        y=df_top_sim['value'],\r\n                        marker_color=colors,\r\n                        width=[0.4,0.4,0.4,0.4,0.4],\r\n                        text=df_top_sim['value'],\r\n                        textfont_color='white',\r\n                        textposition='auto')])\r\n\r\nfig.update_layout(\r\n    title = \r\n        {'text':'&lt;b&gt;Топ-5 closest songs&lt;\/b&gt;&lt;br&gt;&lt;span style=&quot;color:#666666&quot;&gt;&lt;\/span&gt;'},\r\n    showlegend = False,\r\n    height=650,\r\n    font={\r\n        'family':'Open Sans, light',\r\n        'color':'black',\r\n        'size':14\r\n    },\r\n    plot_bgcolor='rgba(0,0,0,0)',\r\n    xaxis={'categoryorder':'total descending'}\r\n)\r\n\r\nfig.show()<\/code><\/pre><iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https:\/\/plotly.com\/~Elisejj\/100.embed?showlink=false\" height=\"650\" width=\"100%\"><\/iframe>\n<p>Given the vector of each song, let us calculate the vector of the entire album – add the vectors of the songs. Then, for such a vector, using the model, we get the words that are the closest in spirit and meaning.<\/p>\n<pre class=\"e2-text-code\"><code>def get_word_from_tlist(lst):\r\n    for word in lst:\r\n        word = word[0].split('_')[0]\r\n        print(word, end=' ')\r\n\r\nvec_sum = 0\r\nfor vec in df.vector:\r\n    vec_sum += vec\r\nsim_word = model.similar_by_vector(vec_sum)\r\nget_word_from_tlist(sim_word)<\/code><\/pre><p><span style=\"color: '#65b5cd'; font-size: 1.2em\"><b>небо тоска тьма пламень плакать горе печаль сердце солнце мрак<\/b><\/span><\/p>\n<p>This is probably the key result and the description of Zemfira’s album in just 10 words.<\/p>\n<p>Finally, we build a general heat map, each cell of which is the result of comparing the texts of two tracks with a cosine measure.<\/p>\n<pre class=\"e2-text-code\"><code>colorscale=[[0.0, &quot;rgba(255,255,255,255)&quot;],\r\n            [0.1, &quot;rgba(229,232,237,255)&quot;],\r\n            [0.2, &quot;rgba(216,222,232,255)&quot;],\r\n            [0.3, &quot;rgba(205,214,228,255)&quot;],\r\n            [0.4, &quot;rgba(182,195,218,255)&quot;],\r\n            [0.5, &quot;rgba(159,178,209,255)&quot;],\r\n            [0.6, &quot;rgba(137,161,200,255)&quot;],\r\n            [0.7, &quot;rgba(107,137,188,255)&quot;],\r\n            [0.8, &quot;rgba(96,129,184,255)&quot;],\r\n            [1.0, &quot;rgba(76,114,176,255)&quot;]]\r\n\r\nfont_colors = ['black']\r\nx = list(df.name.iloc[::-1])\r\ny = list(df.name)\r\nfig = ff.create_annotated_heatmap(result, x=x, y=y, colorscale=colorscale, font_colors=font_colors)\r\nfig.show()<\/code><\/pre><iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https:\/\/plotly.com\/~Elisejj\/82.embed?showlink=false\" height=\"650\" width=\"100%\"><\/iframe>\n<h2><a name=\"result\">Results and data interpretation<\/a><\/h2>\n<p>To give valuable conclusions, we would like to take another look at everything we got. First of all, let us focus on the word cloud. It is easy to see that the words ‘боль’, ‘невозможно’, ‘сорваться’, ‘растерзаны’, ‘сложно’, ‘терпеть’, ‘любить’ have a very decent size, because such words are often found throughout the entire lyrics:<\/p>\n<p>Давайте ещё раз посмотрим на всё, что у нас получилось — начнём с облака слов. Нетрудно заметить, что у слов «боль», «невозможно», «сорваться», «растерзаны», «сложно», «терпеть», «любить» размер весьма приличный — всё потому, что такие слова встречаются часто на протяжении всего текста песен:<\/p>\n<div class=\"e2-text-picture\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/9-wordcloud.jpg\" width=\"2560\" height=\"1707\" alt=\"\" \/>\n<\/div>\n<p>The song “Крым” turned out to be one of the most diverse songs – it contains 74% of unique words. Also, the song “Снег идет” contains very few words, so the majority, which is 82%, are unique. The largest song on the album in terms of amount of words is the track “Таблетки” – there are about 150 words in total.<\/p>\n<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https:\/\/plotly.com\/~Elisejj\/96.embed?showlink=false\" height=\"650\" width=\"100%\"><\/iframe>\n<p>As it was shown on the last chart, the most dynamic track is “Таблетки”, as much as 37 words per minute – nearly one word for every two seconds – and the longest track is “Абьюз”, and according to the previous chart, it also has the lowest percentage of unique words – 46%.<\/p>\n<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https:\/\/plotly.com\/~Elisejj\/98.embed?showlink=false\" height=\"650\" width=\"100%\"><\/iframe>\n<p>Top 5 most semantically similar text pairs:<\/p>\n<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https:\/\/plotly.com\/~Elisejj\/100.embed?showlink=false\" height=\"650\" width=\"100%\"><\/iframe>\n<p>We also got the vector of the entire album and found the closest words. Just take a look at them – ‘тьма’, ‘тоска’, ‘плакать’, ‘горе’, ‘печаль’, ‘сердце’ – this is the list of words that characterizes Zemfira’s lyrics!<\/p>\n<p><span style=\"color: '#65b5cd'; font-size: 1.2em\"><b>небо тоска тьма пламень плакать горе печаль сердце солнце мрак<\/b><\/span><\/p>\n<p>The final result is a heat map. From the visualization, it is noticeable that almost all songs are quite similar to each other – the cosine measure for many pairs exceeds the value of 0.4.<\/p>\n<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https:\/\/plotly.com\/~Elisejj\/82.embed?showlink=false\" height=\"650\" width=\"100%\"><\/iframe>\n<h2>Conclusions<\/h2>\n<p>In the material, we carried out an EDA of the entire text of the new album and, using the pre-trained Word2Vec model, we proved the hypothesis – most of the “Borderline” songs are permeated with rather dark lyrics. However, this is normal, because we love Zemfira precisely for her sincerity and straightforwardness.<\/p>\n",
            "date_published": "2021-09-07T13:46:21+03:00",
            "date_modified": "2021-09-07T13:46:08+03:00",
            "image": "https:\/\/en.leftjoin.ru\/pictures\/header.jpg",
            "_date_published_rfc2822": "Tue, 07 Sep 2021 13:46:21 +0300",
            "_rss_guid_is_permalink": "false",
            "_rss_guid": "66",
            "_e2_data": {
                "is_favourite": false,
                "links_required": [
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css"
                ],
                "og_images": [
                    "https:\/\/en.leftjoin.ru\/pictures\/header.jpg",
                    "https:\/\/en.leftjoin.ru\/pictures\/2-table.png",
                    "https:\/\/en.leftjoin.ru\/pictures\/3-wordcloud.jpg",
                    "https:\/\/en.leftjoin.ru\/pictures\/4-table.png",
                    "https:\/\/en.leftjoin.ru\/pictures\/9-wordcloud.jpg"
                ]
            }
        },
        {
            "id": "65",
            "url": "https:\/\/en.leftjoin.ru\/all\/altinity-clickhouse-training-101\/",
            "title": "Clickhouse Training 101 by Altinity",
            "content_html": "<p>Just recently I have completed a <a href=\"https:\/\/altinity.com\/clickhouse-training\/?utm_source=leftjoin\">Clickhouse Training by Altinity (101 Series Training). <\/a> For those who are just getting to know Clickhouse, Altinity offers free basic training: <a href=\"https:\/\/altinity.com\/data-warehouse-basics\/?utm_source=leftjoin\">Data Warehouse Basics<\/a>. I recommend starting with it if you are planning to dive into learning.<\/p>\n<div class=\"e2-text-picture\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/altinity-clickhouse-developer-300px.png\" width=\"300\" height=\"300\" alt=\"\" \/>\n<div class=\"e2-text-caption\">Certification by Altinity<\/div>\n<\/div>\n<p>I would like to share my experience on the training as well as the <a href=\"https:\/\/capable-stream-f18.notion.site\/Clickhouse-Training-101-by-Altinity-notes-120f1b6467f44a30956d6d7ffeff7b08\">training notes<\/a>.<br \/>\nThe training costs $500 and lasts 4 days for 2 hours. It is carried out in the evenings Moscow time (starting from 19:00 GMT +3).<\/p>\n<h2>Session # 1.<\/h2>\n<p>The first day mostly revises everything covered in Data Warehouse Basics, but it has several new ideas on how to get useful information on queries from system tables.<\/p>\n<p>For example, this query will show which commands are running and their status.<\/p>\n<pre class=\"e2-text-code\"><code>SELECT command, is_done\r\nFROM system.mutations\r\nWHERE table = 'ontime'<\/code><\/pre><p>Besides, for me it was useful to learn about column compression with the use of codecs:<\/p>\n<pre class=\"e2-text-code\"><code>ALTER TABLE ontime\r\n MODIFY COLUMN TailNum LowCardinality(String) CODEC(ZSTD(1))<\/code><\/pre><div class=\"e2-text-picture\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/--2021-08-01--11.53.59.png\" width=\"1732\" height=\"1048\" alt=\"\" \/>\n<\/div>\n<p>For those who are just starting with Clickhouse, the first day will be super useful as it will help in understanding table engines and syntax for their creation, partitions, inserting data (for example directly from S3).<\/p>\n<pre class=\"e2-text-code\"><code>INSERT INTO sdata\r\nSELECT * FROM s3(\r\n 'https:\/\/s3.us-east-1.amazonaws.com\/d1-altinity\/data\/sdata*.csv.gz',\r\n 'aws_access_key_id',\r\n 'aws_secret_access_key',\r\n 'Parquet',\r\n 'DevId Int32, Type String, MDate Date, MDatetime\r\nDateTime, Value Float64')<\/code><\/pre><h2>Session # 2.<\/h2>\n<p>I found the second day the most intense and useful as within this session Robert from Altinity talks about aggregate functions and materialized views ( <a href=\"https:\/\/www.notion.so\/Session-2-35af1ed8d2c54c6fa7fcbea3c9385810#f36adc3df7d74deebedcb3c04e019661\">detailed scheme for the creation of materialized views<\/a> ) in Clickhouse in more detail.<\/p>\n<div class=\"e2-text-picture\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/--2021-08-02--18.11.45.png\" width=\"877\" height=\"495\" alt=\"\" \/>\n<\/div>\n<p>It was super useful for me to learn about index types in Clickhouse.<\/p>\n<div class=\"e2-text-picture\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/--2021-08-02--18.20.35.png\" width=\"874\" height=\"472\" alt=\"\" \/>\n<\/div>\n<h2>Session # 3.<\/h2>\n<p>During the third day, colleagues share their knowledge on how to work with Kafka and JSON objects stored in the tables.<br \/>\nIt was interesting to find out that working with arrays in Clickhouse is very similar to arrays in Python:<\/p>\n<pre class=\"e2-text-code\"><code>WITH [1, 2, 4] AS array\r\nSELECT\r\n array[1] AS First,\r\n array[2] AS Second,\r\n array[3] AS Third,\r\n array[-1] AS Last,\r\n length(array) AS Length<\/code><\/pre><p>When working with arrays, there is a great feature called ARRAY JOIN which “unrolls” arrays to rows.<\/p>\n<div class=\"e2-text-picture\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/--2021-08-02--19.14.28.png\" width=\"812\" height=\"518\" alt=\"\" \/>\n<\/div>\n<p>Clickhouse allows you to efficiently interact with JSON objects stored in a table:<\/p>\n<pre class=\"e2-text-code\"><code>-- Get a JSON string value\r\nSELECT JSONExtractString(row, 'request') AS request\r\nFROM log_row LIMIT 3\r\n-- Get a JSON numeric value\r\nSELECT JSONExtractInt(row, 'status') AS status\r\nFROM log_row LIMIT 3<\/code><\/pre><p>This piece of code is an example of how to extract the elements of the JSON array “request” and “status” separately.<\/p>\n<pre class=\"e2-text-code\"><code>ALTER TABLE log_row\r\n ADD COLUMN\r\nstatus Int16 DEFAULT\r\n JSONExtractInt(row, 'status')\r\nALTER TABLE log_row\r\nUPDATE status = status WHERE 1 = 1<\/code><\/pre><h2>Session # 4.<\/h2>\n<p>The most difficult topic from my point of view was saved for the last day – <a href=\"https:\/\/www.notion.so\/Session-4-f2aa33b6fe434a4e8542f0f64f9439bc#3a3038e94dbf4b47a10284dc1dc226ec\">building sharding and replication patterns <\/a> and building queries on distributed Clickhouse servers.<\/p>\n<div class=\"e2-text-picture\">\n<div class=\"fotorama\" data-width=\"938\" data-ratio=\"1.8073217726397\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/--2021-08-02--19.43.22.png\" width=\"938\" height=\"519\" alt=\"\" \/>\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/--2021-08-02--19.47.52.png\" width=\"934\" height=\"520\" alt=\"\" \/>\n<\/div>\n<\/div>\n<p>Special respect to Altinity for an excellent collection of labs during the training.<\/p>\n<p><b>Links<\/b>:<\/p>\n<ul>\n<li><a href=\"https:\/\/capable-stream-f18.notion.site\/Clickhouse-Training-101-by-Altinity-notes-120f1b6467f44a30956d6d7ffeff7b08\">Notes in Notion<\/a><\/li>\n<li><a href=\"https:\/\/altinity.com\/clickhouse-training\/?utm_source=leftjoin\">ClickHouse 101 Training by Altinity<\/a><\/li>\n<\/ul>\n",
            "date_published": "2021-08-09T08:41:15+03:00",
            "date_modified": "2021-08-09T08:49:34+03:00",
            "image": "https:\/\/en.leftjoin.ru\/pictures\/altinity-clickhouse-developer-300px.png",
            "_date_published_rfc2822": "Mon, 09 Aug 2021 08:41:15 +0300",
            "_rss_guid_is_permalink": "false",
            "_rss_guid": "65",
            "_e2_data": {
                "is_favourite": false,
                "links_required": [
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/fotorama\/fotorama.css",
                    "system\/library\/fotorama\/fotorama.js"
                ],
                "og_images": [
                    "https:\/\/en.leftjoin.ru\/pictures\/altinity-clickhouse-developer-300px.png",
                    "https:\/\/en.leftjoin.ru\/pictures\/--2021-08-01--11.53.59.png",
                    "https:\/\/en.leftjoin.ru\/pictures\/--2021-08-02--18.11.45.png",
                    "https:\/\/en.leftjoin.ru\/pictures\/--2021-08-02--18.20.35.png",
                    "https:\/\/en.leftjoin.ru\/pictures\/--2021-08-02--19.14.28.png",
                    "https:\/\/en.leftjoin.ru\/pictures\/--2021-08-02--19.43.22.png",
                    "https:\/\/en.leftjoin.ru\/pictures\/--2021-08-02--19.47.52.png"
                ]
            }
        },
        {
            "id": "64",
            "url": "https:\/\/en.leftjoin.ru\/all\/data-normalization-with-sql\/",
            "title": "Data normalization with SQL",
            "content_html": "<p>According to GIGO (garbage in, garbage out) principle, errors in input data lead to erroneous analysis results. The results of our work directly depend on the quality of data preparation.<\/p>\n<p>For instance, when we need to prepare data to use in ML algorithms (like k-NN, k-means, logistic regression, etc.), features in the original dataset may vary in scale like the age and height of a person. This may lead to the incorrect performance of the algorithm. Thus, such data needs to be rescaled first.<\/p>\n<p>In this tutorial, we will consider the ways to scale the data using SQL query: min-max normalization, min-max normalization for an arbitrary range, and z-score normalization. For each of these methods we have prepared two SQL query options: one using a SELECT subquery and another using a window function OVER().<\/p>\n<p>We will work with the simple table <b>students<\/b> that contains the data on the height of the students:<\/p>\n<div class=\"e2-text-table\">\n<table cellpadding=\"0\" cellspacing=\"0\" border=\"0\">\n<tr>\n<td>name<\/td>\n<td>height<\/td>\n<\/tr>\n<tr>\n<td>Ivan<\/td>\n<td style=\"text-align: right\">174<\/td>\n<\/tr>\n<tr>\n<td>Peter<\/td>\n<td style=\"text-align: right\">181<\/td>\n<\/tr>\n<tr>\n<td>Dan<\/td>\n<td style=\"text-align: right\">199<\/td>\n<\/tr>\n<tr>\n<td>Kate<\/td>\n<td style=\"text-align: right\">158<\/td>\n<\/tr>\n<tr>\n<td>Mike<\/td>\n<td style=\"text-align: right\">179<\/td>\n<\/tr>\n<tr>\n<td>Silvia<\/td>\n<td style=\"text-align: right\">165<\/td>\n<\/tr>\n<tr>\n<td>Giulia<\/td>\n<td style=\"text-align: right\">152<\/td>\n<\/tr>\n<tr>\n<td>Robert<\/td>\n<td style=\"text-align: right\">188<\/td>\n<\/tr>\n<tr>\n<td>Steven<\/td>\n<td style=\"text-align: right\">177<\/td>\n<\/tr>\n<tr>\n<td>Sophia<\/td>\n<td style=\"text-align: right\">165<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<h2>Min-max rescaling<\/h2>\n<p>Min-max scaling approach scales the data using the fixed range from 0 to 1. In this case, all the data is on the same scale which will exclude the impact of outliers on the conclusions.<\/p>\n<p>The formula for the min-max scaling is given as:<\/p>\n<p>We multiply the numerator by 1.0 in order to get a floating point number at the end.<\/p>\n<p>SQL-query with a subquery:<\/p>\n<pre class=\"e2-text-code\"><code>SELECT height, \r\n       1.0 * (height-t1.min_height)\/(t1.max_height - t1.min_height) AS scaled_minmax\r\n  FROM students, \r\n      (SELECT min(height) as min_height, \r\n              max(height) as max_height \r\n         FROM students\r\n      ) as t1;<\/code><\/pre><p>SQL-query with a window function:<\/p>\n<pre class=\"e2-text-code\"><code>SELECT height, \r\n       (height - MIN(height) OVER ()) * 1.0 \/ (MAX(height) OVER () - MIN(height) OVER ()) AS scaled_minmax\r\n  FROM students;<\/code><\/pre><p>As a result, we get the values in [0, 1] range where 0 is the height of the shortest student and 1 is the height of the tallest one.<\/p>\n<div class=\"e2-text-table\">\n<table cellpadding=\"0\" cellspacing=\"0\" border=\"0\">\n<tr>\n<td>name<\/td>\n<td>height<\/td>\n<td>scaled_minmax<\/td>\n<\/tr>\n<tr>\n<td>Ivan<\/td>\n<td style=\"text-align: right\">174<\/td>\n<td>0.46809<\/td>\n<\/tr>\n<tr>\n<td>Peter<\/td>\n<td style=\"text-align: right\">181<\/td>\n<td>0.61702<\/td>\n<\/tr>\n<tr>\n<td>Dan<\/td>\n<td style=\"text-align: right\">199<\/td>\n<td>1<\/td>\n<\/tr>\n<tr>\n<td>Kate<\/td>\n<td style=\"text-align: right\">158<\/td>\n<td>0.12766<\/td>\n<\/tr>\n<tr>\n<td>Mike<\/td>\n<td style=\"text-align: right\">179<\/td>\n<td>0.57447<\/td>\n<\/tr>\n<tr>\n<td>Silvia<\/td>\n<td style=\"text-align: right\">165<\/td>\n<td>0.2766<\/td>\n<\/tr>\n<tr>\n<td>Giulia<\/td>\n<td style=\"text-align: right\">152<\/td>\n<td>0<\/td>\n<\/tr>\n<tr>\n<td>Robert<\/td>\n<td style=\"text-align: right\">188<\/td>\n<td>0.76596<\/td>\n<\/tr>\n<tr>\n<td>Steven<\/td>\n<td style=\"text-align: right\">177<\/td>\n<td>0.53191<\/td>\n<\/tr>\n<tr>\n<td>Sophia<\/td>\n<td style=\"text-align: right\">165<\/td>\n<td>0.2766<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<h2>Rescaling within a given range<\/h2>\n<p>This is an option of min-max normalization between an arbitrary set of values. When it comes to data scaling, the values do not always need to be in the range between 0 and 1. In these cases, the following formula is applied.<\/p>\n<p>This allows us to scale the data to an arbitrary scale. In our example, let’s assume a=10.0 and b=20.0.<\/p>\n<p>SQL-query with subquery:<\/p>\n<pre class=\"e2-text-code\"><code>SELECT height, \r\n       ((height - min_height) * (20.0 - 10.0) \/ (max_height - min_height)) + 10 AS scaled_ab\r\n  FROM students,\r\n      (SELECT MAX(height) as max_height, \r\n              MIN(height) as min_height\r\n         FROM students  \r\n      ) t1;<\/code><\/pre><p>SQL-query with a window function:<\/p>\n<pre class=\"e2-text-code\"><code>SELECT height, \r\n       ((height - MIN(height) OVER() ) * (20.0 - 10.0) \/ (MAX(height) OVER() - MIN(height) OVER())) + 10.0 AS scaled_ab\r\n  FROM students;<\/code><\/pre><p>We receive similar results as before, but with data spread between 10 and 20.<\/p>\n<div class=\"e2-text-table\">\n<table cellpadding=\"0\" cellspacing=\"0\" border=\"0\">\n<tr>\n<td>name<\/td>\n<td>height<\/td>\n<td>scaled_ab<\/td>\n<\/tr>\n<tr>\n<td>Ivan<\/td>\n<td style=\"text-align: right\">174<\/td>\n<td>14.68085<\/td>\n<\/tr>\n<tr>\n<td>Peter<\/td>\n<td style=\"text-align: right\">181<\/td>\n<td>16.17021<\/td>\n<\/tr>\n<tr>\n<td>Dan<\/td>\n<td style=\"text-align: right\">199<\/td>\n<td>20<\/td>\n<\/tr>\n<tr>\n<td>Kate<\/td>\n<td style=\"text-align: right\">158<\/td>\n<td>11.2766<\/td>\n<\/tr>\n<tr>\n<td>Mike<\/td>\n<td style=\"text-align: right\">179<\/td>\n<td>15.74468<\/td>\n<\/tr>\n<tr>\n<td>Silvia<\/td>\n<td style=\"text-align: right\">165<\/td>\n<td>12.76596<\/td>\n<\/tr>\n<tr>\n<td>Giulia<\/td>\n<td style=\"text-align: right\">152<\/td>\n<td>10<\/td>\n<\/tr>\n<tr>\n<td>Robert<\/td>\n<td style=\"text-align: right\">188<\/td>\n<td>17.65957<\/td>\n<\/tr>\n<tr>\n<td>Steven<\/td>\n<td style=\"text-align: right\">177<\/td>\n<td>15.31915<\/td>\n<\/tr>\n<tr>\n<td>Sophia<\/td>\n<td style=\"text-align: right\">165<\/td>\n<td>12.76596<\/td>\n<\/tr>\n<\/table>\n<\/div>\n<h2>Z-score normalization<\/h2>\n<p>Using Z-score normalization, the data will be scaled so that it has the properties of a standard normal distribution where the mean (μ) is equal to 0 and the standard deviation (σ) to 1.<\/p>\n<p>Z-score is calculated using the formula:<\/p>\n<p>SQL-query with a subquery:<\/p>\n<pre class=\"e2-text-code\"><code>SELECT height, \r\n       (height - t1.mean) * 1.0 \/ t1.sigma AS zscore\r\n  FROM students,\r\n      (SELECT AVG(height) AS mean, \r\n              STDDEV(height) AS sigma\r\n         FROM students\r\n        ) t1;<\/code><\/pre><p>SQL-query with a window function:<\/p>\n<pre class=\"e2-text-code\"><code>SELECT height, \r\n       (height - AVG(height) OVER()) * 1.0 \/ STDDEV(height) OVER() AS z-score\r\n  FROM students;<\/code><\/pre><p>As a result, we can easily notice the outliers that exceed the standard deviation.<\/p>\n<div class=\"e2-text-table\">\n<table cellpadding=\"0\" cellspacing=\"0\" border=\"0\">\n<tr>\n<td>name<\/td>\n<td>height<\/td>\n<td>zscore<\/td>\n<\/tr>\n<tr>\n<td>Ivan<\/td>\n<td style=\"text-align: right\">174<\/td>\n<td>0.01488<\/td>\n<\/tr>\n<tr>\n<td>Peter<\/td>\n<td style=\"text-align: right\">181<\/td>\n<td>0.53582<\/td>\n<\/tr>\n<tr>\n<td>Dan<\/td>\n<td style=\"text-align: right\">199<\/td>\n<td>1.87538<\/td>\n<\/tr>\n<tr>\n<td>Kate<\/td>\n<td style=\"text-align: right\">158<\/td>\n<td>-1.17583<\/td>\n<\/tr>\n<tr>\n<td>Mike<\/td>\n<td style=\"text-align: right\">179<\/td>\n<td>0.38698<\/td>\n<\/tr>\n<tr>\n<td>Silvia<\/td>\n<td style=\"text-align: right\">165<\/td>\n<td>-0.65489<\/td>\n<\/tr>\n<tr>\n<td>Giulia<\/td>\n<td style=\"text-align: right\">152<\/td>\n<td>-1.62235<\/td>\n<\/tr>\n<tr>\n<td>Robert<\/td>\n<td style=\"text-align: right\">188<\/td>\n<td>1.05676<\/td>\n<\/tr>\n<tr>\n<td>Steven<\/td>\n<td style=\"text-align: right\">177<\/td>\n<td>0.23814<\/td>\n<\/tr>\n<tr>\n<td>Sophia<\/td>\n<td style=\"text-align: right\">165<\/td>\n<td>-0.65489<\/td>\n<\/tr>\n<\/table>\n<\/div>\n",
            "date_published": "2021-05-21T16:58:48+03:00",
            "date_modified": "2021-05-21T16:58:37+03:00",
            "_date_published_rfc2822": "Fri, 21 May 2021 16:58:48 +0300",
            "_rss_guid_is_permalink": "false",
            "_rss_guid": "64",
            "_e2_data": {
                "is_favourite": false,
                "links_required": [
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css"
                ],
                "og_images": []
            }
        },
        {
            "id": "63",
            "url": "https:\/\/en.leftjoin.ru\/all\/sap-analytics-cloud-dashboard-overview\/",
            "title": "SAP Analytics Cloud Dashboard Overview",
            "content_html": "<p>Our last guide on BI tools was dedicated to <a href=\"https:\/\/en.leftjoin.ru\/all\/qliksense-dashboard-overview\/\">QlikSense<\/a> and today we will have a look at SAP Analytics Cloud. The dashboard has been prepared for us by the head BI consultant at SAPRUN <a href=\"https:\/\/t.me\/avsalynin\">Alexey Salynin<\/a>.<\/p>\n<p>In this overview, we will touch the following topics: creating a new data source in SAP, adding filters, making a Tree map,<br \/>\nsetting up a preview on mobile devices, working with the Smart Insight assistant, multi language feature.<\/p>\n<p>This dashboard has the same structure as our previous dashboards. At the top, we can see the KPI cards and a date filter. Below there is a heatmap that shows top provinces and a chart that shows profit dynamics on the right. In the second part, we can see the products and customers analysis.<\/p>\n<div class=\"e2-text-picture\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/image_2021-05-17_15-47-44.png\" width=\"1911\" height=\"855\" alt=\"\" \/>\n<\/div>\n<h2>Creating a new data source in SAP<\/h2>\n<p>To add a data source in SAP, we can either work with models or datasets. Models allow us to work with several data sources at the same time and carry out planning in order to solve complex problems. In our case, it’s a simple Excel file, so we used a normal dataset. As we have chosen only the orders sheet. Here is how the dataset looks:<\/p>\n<div class=\"e2-text-picture\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/--2021-05-17-175652.png\" width=\"1447\" height=\"751\" alt=\"\" \/>\n<\/div>\n<p>Some of the columns were created, such as products hierarchy, geo hierarchy and some additional calculations.<\/p>\n<h2>Adding a filter<\/h2>\n<p>There are a number of ways to add a filter starting from default filtration methods to input fields to a timeline. If you click the edit button, you can see the filter format.<\/p>\n<div class=\"e2-text-picture\">\n<div class=\"fotorama\" data-width=\"532\" data-ratio=\"2.4403669724771\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/image_2021-05-17_15-48-14.png\" width=\"532\" height=\"218\" alt=\"\" \/>\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/image_2021-05-17_15-48-32.png\" width=\"500\" height=\"342\" alt=\"\" \/>\n<\/div>\n<\/div>\n<p>A tree map shows the top provinces by sales. Each province contains information on variation similarly to the KPI cards. The map allows filtering by provinces.<\/p>\n<div class=\"e2-text-picture\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/Picture3-1.png\" width=\"624\" height=\"336\" alt=\"\" \/>\n<\/div>\n<p>On the right of the tree map there is a profit dynamics chart. The chart has a dynamic axis which allows choosing the time period in time series charts. Time series graphs in SAP can show a forecast. The forecast is done automatically using built-in algorithms. In our case, triple exponential smoothing was chosen. When hovering over the forecasted point, we can see upper and lower confidence bounds.<\/p>\n<div class=\"e2-text-picture\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/Picture4.png\" width=\"624\" height=\"300\" alt=\"\" \/>\n<\/div>\n<p>In the next part of the dashboard we can see the products and customers analysis section.<\/p>\n<div class=\"e2-text-picture\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/image_2021-05-17_15-47-18.png\" width=\"1833\" height=\"652\" alt=\"\" \/>\n<\/div>\n<p>Here you can see that products are displayed in a list without any hierarchy. To activate the hierarchy we need to change the dimension properties and set the hierarchy.<\/p>\n<div class=\"e2-text-picture\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/Picture5.png\" width=\"624\" height=\"277\" alt=\"\" \/>\n<\/div>\n<p>By default, the first level is displayed. We can change the level and see the products by category. Moreover, the bars are filters for the table on the right.<\/p>\n<div class=\"e2-text-picture\">\n<div class=\"fotorama\" data-width=\"838\" data-ratio=\"1.7944325481799\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/image_2021-05-17_15-47-03.png\" width=\"838\" height=\"467\" alt=\"\" \/>\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/image_2021-05-17_15-46-48.png\" width=\"990\" height=\"568\" alt=\"\" \/>\n<\/div>\n<\/div>\n<h2>Smart Insight<\/h2>\n<p>One of the interesting features in SAP is smart insight. SAP automatically analyses the value and builds charts to provide more insights. For example, if I want to see what contributed to high level of furniture sales, I can click on smart insights and see the following results:<\/p>\n<div class=\"e2-text-picture\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/image_2021-05-17_15-50-55.png\" width=\"571\" height=\"841\" alt=\"\" \/>\n<\/div>\n<p>The feature is very handy as the insights are not simple graphs, but they are connected to each other and show a story. The insights allow us to interact with the data and explore it deeper. We can later add these insights into our visualization. This feature can also be used on a new dataset and empty dashboard to explore the data without building anything.<\/p>\n<p>The whole dashboard was built in a responsive view mode so it will be automatically adapted to different screens. Some visualizations like tables are not adapted for the mobile screen, so we can hide them for mobile screens.<\/p>\n<p>We can save the dashboard in the catalog and add it to favourites. Unfortunately, the functionality of the system doesn’t allow posting dashboards on public sources. But we hope that with new releases the feature will appear.<\/p>\n<p>Last but not least, the dashboard has been created in Russian and when we needed to translate it into English SAP multi language feature came very handy. We can choose the target language and translate all the text. Then the dashboard will be displayed in a language based on the user’s location.<\/p>\n<div class=\"e2-text-picture\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/image_2021-05-17_14-49-02.png\" width=\"1822\" height=\"857\" alt=\"\" \/>\n<\/div>\n<p>We have evaluated the dashboard together with Alexey (his scores in brackets) and received the following results:<br \/>\nMeets the tasks – 9.8 (10)<br \/>\nLearning curve  – 6.5 (7)<br \/>\nTool functionality – 9.8 (9)<br \/>\nEase of use – 8.2 (8)<br \/>\nCompliance with the layout – 9.3 (10)<br \/>\nVisual evaluation – 8.6 (9)<\/p>\n<p>Overall: 8.71 out of 10. Have a look at the final results:<\/p>\n<div class=\"e2-text-picture\">\n<img src=\"https:\/\/en.leftjoin.ru\/pictures\/superstore_sales_sac_en@2x.png\" width=\"956\" height=\"1040\" alt=\"\" \/>\n<\/div>\n",
            "date_published": "2021-05-20T10:18:45+03:00",
            "date_modified": "2021-05-17T16:26:40+03:00",
            "image": "https:\/\/en.leftjoin.ru\/pictures\/superstore_sales_sac_en.png",
            "_date_published_rfc2822": "Thu, 20 May 2021 10:18:45 +0300",
            "_rss_guid_is_permalink": "false",
            "_rss_guid": "63",
            "_e2_data": {
                "is_favourite": false,
                "links_required": [
                    "system\/library\/fotorama\/fotorama.css",
                    "system\/library\/fotorama\/fotorama.js",
                    "system\/library\/fotorama\/fotorama.css",
                    "system\/library\/fotorama\/fotorama.js"
                ],
                "og_images": [
                    "https:\/\/en.leftjoin.ru\/pictures\/superstore_sales_sac_en.png",
                    "https:\/\/en.leftjoin.ru\/pictures\/image_2021-05-17_15-47-44.png",
                    "https:\/\/en.leftjoin.ru\/pictures\/--2021-05-17-175652.png",
                    "https:\/\/en.leftjoin.ru\/pictures\/image_2021-05-17_15-48-14.png",
                    "https:\/\/en.leftjoin.ru\/pictures\/image_2021-05-17_15-48-32.png",
                    "https:\/\/en.leftjoin.ru\/pictures\/Picture3-1.png",
                    "https:\/\/en.leftjoin.ru\/pictures\/Picture4.png",
                    "https:\/\/en.leftjoin.ru\/pictures\/image_2021-05-17_15-47-18.png",
                    "https:\/\/en.leftjoin.ru\/pictures\/Picture5.png",
                    "https:\/\/en.leftjoin.ru\/pictures\/image_2021-05-17_15-47-03.png",
                    "https:\/\/en.leftjoin.ru\/pictures\/image_2021-05-17_15-46-48.png",
                    "https:\/\/en.leftjoin.ru\/pictures\/image_2021-05-17_15-50-55.png",
                    "https:\/\/en.leftjoin.ru\/pictures\/image_2021-05-17_14-49-02.png",
                    "https:\/\/en.leftjoin.ru\/pictures\/superstore_sales_sac_en@2x.png"
                ]
            }
        }
    ],
    "_e2_version": 3386,
    "_e2_ua_string": "E2 (v3386; Aegea)"
}