{ "version": "https:\/\/jsonfeed.org\/version\/1", "title": "LEFT JOIN: blog on analytics, visualisation & data science, posts tagged: analysis", "home_page_url": "https:\/\/en.leftjoin.ru\/tags\/analysis\/", "feed_url": "https:\/\/en.leftjoin.ru\/tags\/analysis\/json\/", "icon": "https:\/\/en.leftjoin.ru\/user\/userpic@2x.jpg", "author": { "name": "Nikolay Valiotti", "url": "https:\/\/en.leftjoin.ru\/", "avatar": "https:\/\/en.leftjoin.ru\/user\/userpic@2x.jpg" }, "items": [ { "id": "71", "url": "https:\/\/en.leftjoin.ru\/all\/mean-vs-median\/", "title": "Mean VS median: how to choose a target metric?", "content_html": "

In today’s article, we would like to highlight a simple, but important topic – how to choose a simple metric to evaluate a particular dataset. Everyone has been familiar with the arithmetic mean for a long time, almost every student knows very well that you should sum up all the available values, divide by their number and get the average value. However, school knowledge does not include any alternative options, of which, in fact, there are many in statistics – almost for every occasion. In solving research and marketing problems, people often take mean as a target. Is this legitimate or is there a better option? Let’s figure it out.<\/p>\n

To begin with, it is worth remembering the definitions of the two metrics that we will talk about today.
\nMean is the most popular statistic used to calculate a data center. What is the median? Median is a value that splits data, sorted in order of increasing values, into two equal parts. This means that the median shows the central value in the sample if the number of cases is odd and the arithmetic mean of the two values ​​if the number of cases in the sample is even.<\/p>\n

Research tasks<\/h2>\n

So, the estimation of the sample mean is often important in many research questions. For instance, specialists studying demography often study changes in the number of regions in Russia in order to track the dynamics and reflect it in reports. Let’s try to calculate the average size of the Russian city, as well as the median, and then compare the results.
\nFirst, you need to find and load data by connecting the pandas library for this.<\/p>\n

import pandas as pd\r\ncity = pd.read_csv('city.csv', sep = ';')<\/code><\/pre>
Then, you need to calculate the mean and median of the sample.<\/p>\n
mean_pop = round (city.population_2020.mean (), 0)\r\nmedian_pop = round (city.population_2020.median (), 0)<\/code><\/pre>
The values, of course, are different, since the distribution of observations in the sample is different from the normal one. In order to understand whether they are very different, let’s build a distribution graph and display the mean and median.<\/p>\n
import matplotlib.pyplot as plt\r\nimport seaborn as sns\r\n\r\nsns.set_palette('rainbow')\r\nfig = plt.figure(figsize = (20, 15))\r\nax = fig.add_subplot(1, 1, 1)\r\ng = sns.histplot(data = city, x= 'population_2020', alpha=0.6, bins = 100, ax=ax)\r\n\r\ng.axvline(mean_pop, linewidth=2, color='r', alpha=0.9, linestyle='--', label = 'Mean = {:,.0f}'.format(mean_pop).replace(',', ' '))\r\ng.axvline(median_pop, linewidth=2, color='darkgreen', alpha=0.9, linestyle='--', label = 'Median = {:,.0f}'.format(median_pop).replace(',', ' '))\r\n\r\nplt.ticklabel_format(axis='x', style='plain')\r\nplt.xlabel("Population", fontsize=20)\r\nplt.ylabel("Number of cities", fontsize=20)\r\nplt.title("Distribution of population of russian cities", fontsize=20)\r\nplt.legend(fontsize="xx-large")\r\nplt.show()<\/code><\/pre>
\n\"\"\n<\/div>\n
Also, on this data it is worth building a boxplot for more accurate visualization with the main distribution quantiles, median, mean and outliers.<\/p>\n
fig = plt.figure(figsize = (10, 10))\r\nsns.set_theme(style="whitegrid")\r\nsns.set_palette(palette="pastel")\r\n\r\nsns.boxplot(y = city['population_2020'], showfliers = False)\r\n\r\nplt.scatter(0, 550100, marker='*', s=100, color = 'black', label = 'Outlier')\r\nplt.scatter(0, 560200, marker='*', s=100, color = 'black')\r\nplt.scatter(0, 570300, marker='*', s=100, color = 'black')\r\nplt.scatter(0, mean_pop, marker='o', s=100, color = 'red', edgecolors = 'black', label = 'Mean')\r\nplt.legend()\r\n\r\nplt.ylabel("Population", fontsize=15)\r\nplt.ticklabel_format(axis='y', style='plain')\r\nplt.title("Boxplot of population of russian cities", fontsize=15)\r\nplt.show()<\/code><\/pre>
\n\"\"\n<\/div>\n
It follows from the graphs that the median is significantly less than the average, and it is also clear that this is a consequence of the presence of outliers which are Moscow and St. Petersburg. Since the arithmetic mean is an extremely sensitive metric to outliers, it is not worth relying on conclusions about the mean if they are present in the sample. An increase or decrease in the population of Moscow can greatly change the average population in Russia, but this will not affect the real regional trend.
\nUsing the arithmetic mean, we say that the number of a typical (average) city in the Russian Federation is 268 thousand people. However, this misleads us, since the mean value is significantly higher than the median solely due to the size of the population of Moscow and St. Petersburg. In fact, the number of a typical Russian city is significantly less (2 times less!) and is approximately 104 thousand citizens.<\/p>\n
Marketing tasks<\/h2>\n
In a business tasks, the difference between the mean and the median is also important, as using the wrong metric can seriously affect the results of the advertising campaign or make it difficult to achieve the goal. Let’s take a look at a real example of the difficulties an entrepreneur can face in retail if he chooses the wrong target metric.
\nTo begin with, as in the previous example, let’s load a dataset about supermarket purchases. Let’s select the dataset columns necessary for analysis and rename them to simplify the code in the future. Since this data is not as well prepared as the previous ones, it is necessary to group all purchased items by receipts. In this case, it is necessary to group by two variables: by the customer’s id and by the date of purchase (the date and time is determined by the moment of closing the bill, therefore, all purchases within one bill coincide by date variable). Then, let’s name the resulting column “total_bill”, that is, the check amount and calculate the average and median.<\/p>\n
df = pd.read_excel ('invoice_data.xlsx')\r\ndf.columns = ['user', 'total_price', 'date']\r\ngroupped_df = pd.DataFrame (df.groupby (['user', 'date']). total_price.sum ())\r\ngroupped_df.columns = ['total_bill']\r\nmean_bill = groupped_df.total_bill.mean ()\r\nmedian_bill = groupped_df.total_bill.median ()<\/code><\/pre>
Now, as in the previous example, you need to plot the distribution of customer checks and boxplot, and also display the median and mean on each of them.<\/p>\n
sns.set_palette('rainbow')\r\nfig = plt.figure(figsize = (20, 15))\r\nax = fig.add_subplot(1, 1, 1)\r\nsns.histplot(groupped_df, x = 'total_bill', binwidth=200, alpha=0.6, ax=ax)\r\nplt.xlabel("Purchases", fontsize=20)\r\nplt.ylabel("Total bill", fontsize=20)\r\nplt.title("Distribution of total bills", fontsize=20)\r\nplt.axvline(mean_bill, linewidth=2, color='r', alpha=1, linestyle='--', label = 'Mean = {:.0f}'.format(mean_bill))\r\nplt.axvline(median_bill, linewidth=2, color='darkgreen', alpha=1, linestyle='--', label = 'Median = {:.0f}'.format(median_bill))\r\nplt.legend(fontsize="xx-large")\r\nplt.show()<\/code><\/pre>
\n\"\"\n<\/div>\n
fig = plt.figure(figsize = (10, 10))\r\nsns.set_theme(style="whitegrid")\r\nsns.set_palette(palette="pastel")\r\n\r\nsns.boxplot(y = groupped_df['total_bill'], showfliers = False)\r\n\r\nplt.scatter(0, 1800, marker='*', s=100, color = 'black', label = 'Outlier')\r\nplt.scatter(0, 1850, marker='*', s=100, color = 'black')\r\nplt.scatter(0, 1900, marker='*', s=100, color = 'black')\r\nplt.scatter(0, mean_bill, marker='o', s=100, color = 'red', edgecolors = 'black', label = 'Mean')\r\nplt.legend()\r\n\r\nplt.ticklabel_format(axis='y', style='plain')\r\nplt.ylabel("Total bill", fontsize=15)\r\nplt.title("Boxplot of total bills", fontsize=15)\r\nplt.show()<\/code><\/pre>
\n\"\"\n<\/div>\n
The graphs show that the distribution is different from normal, which means that the median and mean are not equal. The median value is smaller than the average by about 220 rubles.
\nNow, imagine that marketers have a task to increase the average buyer’s bill. A marketer may decide that since the average check is 601 rubles, the following promotion can be offered: “All buyers who make a purchase over 600 rubles, will get 20% discount on any good for 100 rubles.” In general, it is a reasonable offer, however, in reality, the average check is lower – 378 rubles. Thus, the majority of buyers will not be interested in the offer, since their purchase usually does not reach the proposed threshold. This means, that they will not take advantage of the offer and will not receive a discount, and the company will not be able to achieve its goal and increase the profit. The point is that the initial assumptions were wrong.<\/p>\n
Conclusions<\/h2>\n
As you already understood, the mean often shows a more pleasant result, both for business and for research tasks, because it is always nicer to imagine the situation with the average check or the demographic situation in the country better than it really is. However, one must always remember about the shortcomings of mean in order to be able to correctly choose the appropriate analogue for assessing a particular situation.<\/p>\n",
            "date_published": "2021-10-27T14:20:56+03:00",
            "date_modified": "2021-10-27T14:20:52+03:00",
            "image": "https:\/\/en.leftjoin.ru\/pictures\/first.jpg",
            "_date_published_rfc2822": "Wed, 27 Oct 2021 14:20:56 +0300",
            "_rss_guid_is_permalink": "false",
            "_rss_guid": "71",
            "_e2_data": {
                "is_favourite": false,
                "links_required": [
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css"
                ],
                "og_images": [
                    "https:\/\/en.leftjoin.ru\/pictures\/first.jpg",
                    "https:\/\/en.leftjoin.ru\/pictures\/second.jpg",
                    "https:\/\/en.leftjoin.ru\/pictures\/third.jpg",
                    "https:\/\/en.leftjoin.ru\/pictures\/forth.jpg"
                ]
            }
        },
        {
            "id": "66",
            "url": "https:\/\/en.leftjoin.ru\/all\/python-and-lyrics-of-zemfiras-new-album-capturing-the-spirit-of\/",
            "title": "Python and lyrics of Zemfira’s new album: capturing the spirit of her songs",
            "content_html": "
\n\"\"\n<\/div>\n
Zemfira’s latest studio album, Borderline, was released in February, 8 years after the previous one. For this album, various people cooperated with her, including her relatives – the riff for the song “Таблетки” was written by her nephew from London. The album turned out to be diverse: for instance, the song “Остин” is dedicated to the main character of the Homescapes game by the Russian studio Playrix (by the way, check out the latest Business Secrets with the Bukhman brothers<\/a>, they also mention it there). Zemfira liked the game a lot, thus, she contacted Playrix to create this song. Also, the song “Крым” was written as a soundtrack to a new film by Zemfira’s colleague Renata Litvinova.<\/p>\n
Listen new album in Apple Music<\/a> \/ Яндекс.Музыке<\/a> \/ Spotify<\/a><\/p>\n
Nevertheless, the spirit of the whole album is rather gloomy – the songs often repeat the words ‘боль’, ‘ад’, ‘бесишь’ and other synonyms. We decided to conduct an exploratory analysis of her album, and then, using the Word2Vec model and a cosine measure, look at the semantic closeness of the songs and calculate the general mood of the album.<\/p>\n
For those who are bored with reading about data preparation and analysis steps, you can go directly to the results<\/a>.<\/p>\n
Data preparation<\/h2>\n
For starters, we write a data processing script. The purpose of the script is to collect a united csv-table from a set of text files, each of which contains a song. At the same time, we have to get rid of all punctuation marks and unnecessary words as we need to focus only on significant content.<\/p>\n
import pandas as pd\r\nimport re\r\nimport string\r\nimport pymorphy2\r\nfrom nltk.corpus import stopwords<\/code><\/pre>
Then we create a morphological analyzer and expand the list of everything that needs to be discarded:<\/p>\n
morph = pymorphy2.MorphAnalyzer()\r\nstopwords_list = stopwords.words('russian')\r\nstopwords_list.extend(['куплет', 'это', 'я', 'мы', 'ты', 'припев', 'аутро', 'предприпев', 'lyrics', '1', '2', '3', 'то'])\r\nstring.punctuation += '—'<\/code><\/pre>
The names of the songs are given in English, so we have to create a dictionary for translation into Russian and a dictionary, from which we will later make a table:<\/p>\n
result_dict = dict()\r\n\r\nsongs_dict = {\r\n    'snow':'снег идёт',\r\n    'crimea':'крым',\r\n    'mother':'мама',\r\n    'ostin':'остин',\r\n    'abuse':'абьюз',\r\n    'wait_for_me':'жди меня',\r\n    'tom':'том',\r\n    'come_on':'камон',\r\n    'coat':'пальто',\r\n    'this_summer':'этим летом',\r\n    'ok':'ок',\r\n    'pills':'таблетки'\r\n}<\/code><\/pre>
Let’s define several necessary functions. The first one reads the entire song from the file and removes line breaks, the second clears the text from unnecessary characters and words, and the third one converts the words to normal form, using the pymorphy2 morphological analyzer. The pymorphy2 module does not always handle ambiguity well – additional processing is required for the words ‘ад’ and ‘рай’.<\/p>\n
def read_song(filename):\r\n    f = open(f'{filename}.txt', 'r').read()\r\n    f = f.replace('\\n', ' ')\r\n    return f\r\n\r\ndef clean_string(text):\r\n    text = re.split(' |:|\\.|\\(|\\)|,|"|;|\/|\\n|\\t|-|\\?|\\[|\\]|!', text)\r\n    text = ' '.join([word for word in text if word not in string.punctuation])\r\n    text = text.lower()\r\n    text = ' '.join([word for word in text.split() if word not in stopwords_list])\r\n    return text\r\n\r\ndef string_to_normal_form(string):\r\n    string_lst = string.split()\r\n    for i in range(len(string_lst)):\r\n        string_lst[i] = morph.parse(string_lst[i])[0].normal_form\r\n        if (string_lst[i] == 'аду'):\r\n            string_lst[i] = 'ад'\r\n        if (string_lst[i] == 'рая'):\r\n            string_lst[i] = 'рай'\r\n    string = ' '.join(string_lst)\r\n    return string<\/code><\/pre>
After all this preparation, we can get back to the data and process each song and read the file with the corresponding name:<\/p>\n
name_list = []\r\ntext_list = []\r\nfor song, name in songs_dict.items():\r\n    text = string_to_normal_form(clean_string(read_song(song)))\r\n    name_list.append(name)\r\n    text_list.append(text)<\/code><\/pre>
Then we combine everything into a DataFrame and save it as a csv-file.<\/p>\n
df = pd.DataFrame()\r\ndf['name'] = name_list\r\ndf['text'] = text_list\r\ndf['time'] = [290, 220, 187, 270, 330, 196, 207, 188, 269, 189, 245, 244]\r\ndf.to_csv('borderline.csv', index=False)<\/code><\/pre>
Result:<\/p>\n
\n\"\"\n<\/div>\n
Word cloud for the whole album<\/h2>\n
To begin with the analysis, we have to construct a word cloud, because it can display the most common words found in these songs. We import the required libraries, read the csv-file and set the configurations:<\/p>\n
import nltk\r\nfrom wordcloud import WordCloud\r\nimport pandas as pd\r\nimport matplotlib.pyplot as plt\r\nfrom nltk import word_tokenize, ngrams\r\n\r\n%matplotlib inline\r\nnltk.download('punkt')\r\ndf = pd.read_csv('borderline.csv')<\/code><\/pre>
Now we create a new figure, set the design parameters and, using the word cloud library, display words with the size directly proportional to the frequency of the word. We additionally indicate the name of the song above the corresponding graph.<\/p>\n
fig = plt.figure()\r\nfig.patch.set_facecolor('white')\r\nplt.subplots_adjust(wspace=0.3, hspace=0.2)\r\ni = 1\r\nfor name, text in zip(df.name, df.text):\r\n    tokens = word_tokenize(text)\r\n    text_raw = " ".join(tokens)\r\n    wordcloud = WordCloud(colormap='PuBu', background_color='white', contour_width=10).generate(text_raw)\r\n    plt.subplot(4, 3, i, label=name,frame_on=True)\r\n    plt.tick_params(labelsize=10)\r\n    plt.imshow(wordcloud)\r\n    plt.axis("off")\r\n    plt.title(name,fontdict={'fontsize':7,'color':'grey'},y=0.93)\r\n    plt.tick_params(labelsize=10)\r\n    i += 1<\/code><\/pre>
\n\"\"\n<\/div>\n
EDA of the lyrics<\/h2>\n
Let us move to the next part and analyze the lyrics. To do this, we have to import special libraries to deal with data and visualization:<\/p>\n
import plotly.graph_objects as go\r\nimport plotly.figure_factory as ff\r\nfrom scipy import spatial\r\nimport collections\r\nimport pymorphy2\r\nimport gensim\r\n\r\nmorph = pymorphy2.MorphAnalyzer()<\/code><\/pre>
Firstly, we should count the overall number of words in each song, the number of unique words, and their percentage:<\/p>\n
songs = []\r\ntotal = []\r\nuniq = []\r\npercent = []\r\n\r\nfor song, text in zip(df.name, df.text):\r\n    songs.append(song)\r\n    total.append(len(text.split()))\r\n    uniq.append(len(set(text.split())))\r\n    percent.append(round(len(set(text.split())) \/ len(text.split()), 2) * 100)<\/code><\/pre>
All this information should be written in a DataFrame and additionally we want to count the number of words per minute for each song:<\/p>\n
df_words = pd.DataFrame()\r\ndf_words['song'] = songs\r\ndf_words['total words'] = total\r\ndf_words['uniq words'] = uniq\r\ndf_words['percent'] = percent\r\ndf_words['time'] = df['time']\r\ndf_words['words per minute'] = round(total \/ (df['time'] \/\/ 60))\r\ndf_words = df_words[::-1]<\/code><\/pre>
\n\"\"\n<\/div>\n
It would be great to visualize the data, so let us build two bar charts: one for the number of words in the song, and the other one for the number of words per minute.<\/p>\n
colors_1 = ['rgba(101,181,205,255)'] * 12\r\ncolors_2 = ['rgba(62,142,231,255)'] * 12\r\n\r\nfig = go.Figure(data=[\r\n    go.Bar(name='📝 Total number of words,\r\n           text=df_words['total words'],\r\n           textposition='auto',\r\n           x=df_words.song,\r\n           y=df_words['total words'],\r\n           marker_color=colors_1,\r\n           marker=dict(line=dict(width=0)),),\r\n    go.Bar(name='🌀 Unique words',\r\n           text=df_words['uniq words'].astype(str) + '<br>'+ df_words.percent.astype(int).astype(str) + '%' ,\r\n           textposition='inside',\r\n           x=df_words.song,\r\n           y=df_words['uniq words'],\r\n           textfont_color='white',\r\n           marker_color=colors_2,\r\n           marker=dict(line=dict(width=0)),),\r\n])\r\n\r\nfig.update_layout(barmode='group')\r\n\r\nfig.update_layout(\r\n    title = \r\n        {'text':'<b>The ratio of the number of unique words to the total<\/b><br><span style="color:#666666"><\/span>'},\r\n    showlegend = True,\r\n    height=650,\r\n    font={\r\n        'family':'Open Sans, light',\r\n        'color':'black',\r\n        'size':14\r\n    },\r\n    plot_bgcolor='rgba(0,0,0,0)',\r\n)\r\nfig.update_layout(legend=dict(\r\n    yanchor="top",\r\n    xanchor="right",\r\n))\r\n\r\nfig.show()<\/code><\/pre>