{ "version": "https:\/\/jsonfeed.org\/version\/1", "title": "LEFT JOIN: blog on analytics, visualisation & data science, posts tagged: web-crawling", "home_page_url": "https:\/\/en.leftjoin.ru\/tags\/web-crawling\/", "feed_url": "https:\/\/en.leftjoin.ru\/tags\/web-crawling\/json\/", "icon": "https:\/\/en.leftjoin.ru\/user\/userpic@2x.jpg", "author": { "name": "Nikolay Valiotti", "url": "https:\/\/en.leftjoin.ru\/", "avatar": "https:\/\/en.leftjoin.ru\/user\/userpic@2x.jpg" }, "items": [ { "id": "6", "url": "https:\/\/en.leftjoin.ru\/all\/collecting-receipts-with-python-p1\/", "title": "Collecting data from hypermarket receipts on Python", "content_html": "

Recently, once again buying products in a hypermarket, I recalled that, according to the Russian Federal Act FZ-54, any trade operator, that issues a receipt, is obliged to send the data thereof to the Tax Service.<\/p>\n

\n\"\"\n
Receipt from “Lenta” hypermarket. The QR-code of our interest is circled.<\/div>\n<\/div>\n

So, what does it mean for us, data analysts? It means that we can know ourselves and our needs better, and also acquire interesting data on own purchases.<\/p>\n

Let’s try to assemble a small prototype of an app that will allow to make a dynamic of our purchases within the framework of blog posts’ series. So, we’ll start from the fact, that each receipt has a QR-code, and if you identify it, you’ll receive the following line:<\/p>\n

\n

t=20190320T2303&s=5803.00&fn=9251440300007971&i=141637&fp=4087570038&n=1<\/i><\/p>\n<\/blockquote>\n

This line comprises:<\/p>\n

\n

t<\/i> – timestamp, the time when you made a purchase
\ns<\/i> – sum of the receipt
\nfn<\/i> – code number of fss, will be needed further in a request to API
\ni<\/i> – receipt number, will be needed further in a request to API
\nfp<\/i> – fiscalsign parameter, will be needed further in a request to API<\/p>\n<\/blockquote>\n

Within the solution of the first step, we will parse the receipt data and collect it in pandas<\/i> dataframe, using Python modules.<\/p>\n

We will use API<\/a>, that provides data on the receipt from the Tax Service website.<\/p>\n

Initially, we will receive authentication data:<\/p>\n

import requests\r\nyour_phone = '+7XXXYYYZZZZ' #you need to state your phone number, SMS with password will arrive thereon\r\nr = requests.post('https:\/\/proverkacheka.nalog.ru:9999\/v1\/mobile\/users\/signup', json = {"email":"email@email.com","name":"USERNAME","phone":your_phone})<\/code><\/pre>
As a result of performing POST request we receive a password in SMS to the indicated phone number. Further on, we will be using it in a variable pwd<\/b><\/i><\/p>\n
Now we’ll parse our line with values from QR-code:<\/p>\n
import re\r\nqr_string='t=20190320T2303&s=5803.00&fn=9251440300007971&i=141637&fp=4087570038&n=1'\r\nt=re.findall(r't=(\\w+)', qr_string)[0]\r\ns=re.findall(r's=(\\w+)', qr_string)[0]\r\nfn=re.findall(r'fn=(\\w+)', qr_string)[0]\r\ni=re.findall(r'i=(\\w+)', qr_string)[0]\r\nfp=re.findall(r'fp=(\\w+)', qr_string)[0]<\/code><\/pre>
We’ll use the variables obtained in order to extract the data.
\nOne Habr post<\/a> pretty thoroughly examines status of errors at formation of API request, therefore I won’t repeat this information.<\/p>\n
In the beginning, we need to verify the presence of data on this receipt, so we form a GET request.<\/p>\n
headers = {'Device-Id':'', 'Device-OS':''}\r\npayload = {'fiscalSign': fp, 'date': t,'sum':s}\r\ncheck_request=requests.get('https:\/\/proverkacheka.nalog.ru:9999\/v1\/ofds\/*\/inns\/*\/fss\/'+fn+'\/operations\/1\/tickets\/'+i,params=payload, headers=headers,auth=(your_phone, pwd))\r\nprint(check_request.status_code)<\/code><\/pre>
In the request one needs to indicate headers, at least empty ones. In my case, GET request returns error 406, thus I get that such receipt is found (why GET request returns 406 remains a mystery to me, so I will be glad to receive some clues in comments). If not indicating sum or date, GET request returns error 400 – bad request.<\/p>\n
Let’s move on to the most interesting part, receiving data of the receipt:<\/p>\n
request_info=requests.get('https:\/\/proverkacheka.nalog.ru:9999\/v1\/inns\/*\/kkts\/*\/fss\/'+fn+'\/tickets\/'+i+'?fiscalSign='+fp+'&sendToEmail=no',headers=headers,auth=(your_phone, pwd))\r\nprint(request_info.status_code)\r\nproducts=request_info.json()<\/code><\/pre>
We should receive code 200 (successful execution of the request), and in the variable products<\/i> – everything, that applies to our receipt.<\/p>\n
In order to further work with this data, let’s use pandas<\/i> and transform everything in dataframe.<\/p>\n
import pandas as pd\r\nfrom datetime import datetime\r\nmy_products=pd.DataFrame(products['document']['receipt']['items'])\r\nmy_products['price']=my_products['price']\/100\r\nmy_products['sum']=my_products['sum']\/100\r\ndatetime_check = datetime.strptime(t, '%Y%m%dT%H%M') #((https:\/\/docs.python.org\/3\/library\/datetime.html#strftime-and-strptime-behavior formate the date))\r\nmy_products['date']=datetime_check\r\nmy_products.set_index('date',inplace=True)<\/code><\/pre>
Now we have working pandas.dataframe with receipts, visually it looks as follows:<\/p>\n
\n\"\"\n
“Header” of receipt data<\/div>\n<\/div>\n
You can construct a bar chart of purchases or observe everything as a box plot:<\/p>\n
import matplotlib.pyplot as plt\r\n%matplotlib inline\r\nmy_products['sum'].plot(kind='hist', bins=20)\r\nplt.show()\r\nmy_products['sum'].plot(kind='box')\r\nplt.show()<\/code><\/pre>
\n\"\"\n
boxplot_cheques.png<\/div>\n<\/div>\n
In conclusion, we will simply get descriptive statistics as text, using a command .describe()<\/i>:<\/p>\n
my_products.describe()<\/code><\/pre>
\n\"\"\n<\/div>\n
It’s convenient to write down data as .csv file, so that the next time you can amend the statistics:<\/p>\n
with open('hyper_receipts.csv', 'a') as f:\r\n             my_products.to_csv(f, header=True)<\/code><\/pre>",
            "date_published": "2019-03-22T17:41:37+03:00",
            "date_modified": "2020-01-27T11:33:13+03:00",
            "image": "https:\/\/en.leftjoin.ru\/pictures\/lenta-receipt@2x.jpg",
            "_date_published_rfc2822": "Fri, 22 Mar 2019 17:41:37 +0300",
            "_rss_guid_is_permalink": "false",
            "_rss_guid": "6",
            "_e2_data": {
                "is_favourite": false,
                "links_required": [
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css",
                    "system\/library\/highlight\/highlight.js",
                    "system\/library\/highlight\/highlight.css"
                ],
                "og_images": [
                    "https:\/\/en.leftjoin.ru\/pictures\/lenta-receipt@2x.jpg",
                    "https:\/\/en.leftjoin.ru\/pictures\/2019-03-22_17-14-33@2x.png",
                    "https:\/\/en.leftjoin.ru\/pictures\/hist_cheques.png",
                    "https:\/\/en.leftjoin.ru\/pictures\/2019-03-22_17-27-06@2x.png"
                ]
            }
        }
    ],
    "_e2_version": 3386,
    "_e2_ua_string": "E2 (v3386; Aegea)"
}