{ "version": "https:\/\/jsonfeed.org\/version\/1", "title": "LEFT JOIN: blog on analytics, visualisation & data science, posts tagged: Analytics engineering", "home_page_url": "https:\/\/en.leftjoin.ru\/tags\/analytics-engineering\/", "feed_url": "https:\/\/en.leftjoin.ru\/tags\/analytics-engineering\/json\/", "icon": "https:\/\/en.leftjoin.ru\/user\/userpic@2x.jpg", "author": { "name": "Nikolay Valiotti", "url": "https:\/\/en.leftjoin.ru\/", "avatar": "https:\/\/en.leftjoin.ru\/user\/userpic@2x.jpg" }, "items": [ { "id": "66", "url": "https:\/\/en.leftjoin.ru\/all\/python-and-lyrics-of-zemfiras-new-album-capturing-the-spirit-of\/", "title": "Python and lyrics of Zemfira’s new album: capturing the spirit of her songs", "content_html": "
\n\"\"\n<\/div>\n

Zemfira’s latest studio album, Borderline, was released in February, 8 years after the previous one. For this album, various people cooperated with her, including her relatives – the riff for the song “Таблетки” was written by her nephew from London. The album turned out to be diverse: for instance, the song “Остин” is dedicated to the main character of the Homescapes game by the Russian studio Playrix (by the way, check out the latest Business Secrets with the Bukhman brothers<\/a>, they also mention it there). Zemfira liked the game a lot, thus, she contacted Playrix to create this song. Also, the song “Крым” was written as a soundtrack to a new film by Zemfira’s colleague Renata Litvinova.<\/p>\n

Listen new album in Apple Music<\/a> \/ Яндекс.Музыке<\/a> \/ Spotify<\/a><\/p>\n

Nevertheless, the spirit of the whole album is rather gloomy – the songs often repeat the words ‘боль’, ‘ад’, ‘бесишь’ and other synonyms. We decided to conduct an exploratory analysis of her album, and then, using the Word2Vec model and a cosine measure, look at the semantic closeness of the songs and calculate the general mood of the album.<\/p>\n

For those who are bored with reading about data preparation and analysis steps, you can go directly to the results<\/a>.<\/p>\n

Data preparation<\/h2>\n

For starters, we write a data processing script. The purpose of the script is to collect a united csv-table from a set of text files, each of which contains a song. At the same time, we have to get rid of all punctuation marks and unnecessary words as we need to focus only on significant content.<\/p>\n

import pandas as pd\r\nimport re\r\nimport string\r\nimport pymorphy2\r\nfrom nltk.corpus import stopwords<\/code><\/pre>
Then we create a morphological analyzer and expand the list of everything that needs to be discarded:<\/p>\n
morph = pymorphy2.MorphAnalyzer()\r\nstopwords_list = stopwords.words('russian')\r\nstopwords_list.extend(['куплет', 'это', 'я', 'мы', 'ты', 'припев', 'аутро', 'предприпев', 'lyrics', '1', '2', '3', 'то'])\r\nstring.punctuation += '—'<\/code><\/pre>
The names of the songs are given in English, so we have to create a dictionary for translation into Russian and a dictionary, from which we will later make a table:<\/p>\n
result_dict = dict()\r\n\r\nsongs_dict = {\r\n    'snow':'снег идёт',\r\n    'crimea':'крым',\r\n    'mother':'мама',\r\n    'ostin':'остин',\r\n    'abuse':'абьюз',\r\n    'wait_for_me':'жди меня',\r\n    'tom':'том',\r\n    'come_on':'камон',\r\n    'coat':'пальто',\r\n    'this_summer':'этим летом',\r\n    'ok':'ок',\r\n    'pills':'таблетки'\r\n}<\/code><\/pre>
Let’s define several necessary functions. The first one reads the entire song from the file and removes line breaks, the second clears the text from unnecessary characters and words, and the third one converts the words to normal form, using the pymorphy2 morphological analyzer. The pymorphy2 module does not always handle ambiguity well – additional processing is required for the words ‘ад’ and ‘рай’.<\/p>\n
def read_song(filename):\r\n    f = open(f'{filename}.txt', 'r').read()\r\n    f = f.replace('\\n', ' ')\r\n    return f\r\n\r\ndef clean_string(text):\r\n    text = re.split(' |:|\\.|\\(|\\)|,|"|;|\/|\\n|\\t|-|\\?|\\[|\\]|!', text)\r\n    text = ' '.join([word for word in text if word not in string.punctuation])\r\n    text = text.lower()\r\n    text = ' '.join([word for word in text.split() if word not in stopwords_list])\r\n    return text\r\n\r\ndef string_to_normal_form(string):\r\n    string_lst = string.split()\r\n    for i in range(len(string_lst)):\r\n        string_lst[i] = morph.parse(string_lst[i])[0].normal_form\r\n        if (string_lst[i] == 'аду'):\r\n            string_lst[i] = 'ад'\r\n        if (string_lst[i] == 'рая'):\r\n            string_lst[i] = 'рай'\r\n    string = ' '.join(string_lst)\r\n    return string<\/code><\/pre>
After all this preparation, we can get back to the data and process each song and read the file with the corresponding name:<\/p>\n
name_list = []\r\ntext_list = []\r\nfor song, name in songs_dict.items():\r\n    text = string_to_normal_form(clean_string(read_song(song)))\r\n    name_list.append(name)\r\n    text_list.append(text)<\/code><\/pre>
Then we combine everything into a DataFrame and save it as a csv-file.<\/p>\n
df = pd.DataFrame()\r\ndf['name'] = name_list\r\ndf['text'] = text_list\r\ndf['time'] = [290, 220, 187, 270, 330, 196, 207, 188, 269, 189, 245, 244]\r\ndf.to_csv('borderline.csv', index=False)<\/code><\/pre>
Result:<\/p>\n
\n\"\"\n<\/div>\n
Word cloud for the whole album<\/h2>\n
To begin with the analysis, we have to construct a word cloud, because it can display the most common words found in these songs. We import the required libraries, read the csv-file and set the configurations:<\/p>\n
import nltk\r\nfrom wordcloud import WordCloud\r\nimport pandas as pd\r\nimport matplotlib.pyplot as plt\r\nfrom nltk import word_tokenize, ngrams\r\n\r\n%matplotlib inline\r\nnltk.download('punkt')\r\ndf = pd.read_csv('borderline.csv')<\/code><\/pre>
Now we create a new figure, set the design parameters and, using the word cloud library, display words with the size directly proportional to the frequency of the word. We additionally indicate the name of the song above the corresponding graph.<\/p>\n
fig = plt.figure()\r\nfig.patch.set_facecolor('white')\r\nplt.subplots_adjust(wspace=0.3, hspace=0.2)\r\ni = 1\r\nfor name, text in zip(df.name, df.text):\r\n    tokens = word_tokenize(text)\r\n    text_raw = " ".join(tokens)\r\n    wordcloud = WordCloud(colormap='PuBu', background_color='white', contour_width=10).generate(text_raw)\r\n    plt.subplot(4, 3, i, label=name,frame_on=True)\r\n    plt.tick_params(labelsize=10)\r\n    plt.imshow(wordcloud)\r\n    plt.axis("off")\r\n    plt.title(name,fontdict={'fontsize':7,'color':'grey'},y=0.93)\r\n    plt.tick_params(labelsize=10)\r\n    i += 1<\/code><\/pre>
\n\"\"\n<\/div>\n
EDA of the lyrics<\/h2>\n
Let us move to the next part and analyze the lyrics. To do this, we have to import special libraries to deal with data and visualization:<\/p>\n
import plotly.graph_objects as go\r\nimport plotly.figure_factory as ff\r\nfrom scipy import spatial\r\nimport collections\r\nimport pymorphy2\r\nimport gensim\r\n\r\nmorph = pymorphy2.MorphAnalyzer()<\/code><\/pre>
Firstly, we should count the overall number of words in each song, the number of unique words, and their percentage:<\/p>\n
songs = []\r\ntotal = []\r\nuniq = []\r\npercent = []\r\n\r\nfor song, text in zip(df.name, df.text):\r\n    songs.append(song)\r\n    total.append(len(text.split()))\r\n    uniq.append(len(set(text.split())))\r\n    percent.append(round(len(set(text.split())) \/ len(text.split()), 2) * 100)<\/code><\/pre>
All this information should be written in a DataFrame and additionally we want to count the number of words per minute for each song:<\/p>\n
df_words = pd.DataFrame()\r\ndf_words['song'] = songs\r\ndf_words['total words'] = total\r\ndf_words['uniq words'] = uniq\r\ndf_words['percent'] = percent\r\ndf_words['time'] = df['time']\r\ndf_words['words per minute'] = round(total \/ (df['time'] \/\/ 60))\r\ndf_words = df_words[::-1]<\/code><\/pre>
\n\"\"\n<\/div>\n
It would be great to visualize the data, so let us build two bar charts: one for the number of words in the song, and the other one for the number of words per minute.<\/p>\n
colors_1 = ['rgba(101,181,205,255)'] * 12\r\ncolors_2 = ['rgba(62,142,231,255)'] * 12\r\n\r\nfig = go.Figure(data=[\r\n    go.Bar(name='📝 Total number of words,\r\n           text=df_words['total words'],\r\n           textposition='auto',\r\n           x=df_words.song,\r\n           y=df_words['total words'],\r\n           marker_color=colors_1,\r\n           marker=dict(line=dict(width=0)),),\r\n    go.Bar(name='🌀 Unique words',\r\n           text=df_words['uniq words'].astype(str) + '<br>'+ df_words.percent.astype(int).astype(str) + '%' ,\r\n           textposition='inside',\r\n           x=df_words.song,\r\n           y=df_words['uniq words'],\r\n           textfont_color='white',\r\n           marker_color=colors_2,\r\n           marker=dict(line=dict(width=0)),),\r\n])\r\n\r\nfig.update_layout(barmode='group')\r\n\r\nfig.update_layout(\r\n    title = \r\n        {'text':'<b>The ratio of the number of unique words to the total<\/b><br><span style="color:#666666"><\/span>'},\r\n    showlegend = True,\r\n    height=650,\r\n    font={\r\n        'family':'Open Sans, light',\r\n        'color':'black',\r\n        'size':14\r\n    },\r\n    plot_bgcolor='rgba(0,0,0,0)',\r\n)\r\nfig.update_layout(legend=dict(\r\n    yanchor="top",\r\n    xanchor="right",\r\n))\r\n\r\nfig.show()<\/code><\/pre>