Shulou(Shulou.com)06/03 Report--

Preface

Although the data of financial applications such as flush is good enough, there is still an impulse to customize the data, and the data will not be much better than the former, but it is good to customize according to your own ideas.

target

Get the data through the free data interface, update the historical transaction data incrementally every day, and then do visualization and data analysis through Kibana.

In fact, it is also a good choice to do visualization through visualization frameworks such as echarts, but the upfront cost is too high. In addition, pandas+matplotlib is enough to meet most of the needs, but the sense of interaction is too weak, so it is necessary to use a visual application. Here we choose kibana, and its competitor is Grafana.

This goal should last forever. You can get the code through the following link.

Https://github.com/youerning/stock_playground

Environment configuration

Python3 (Anaconda installation recommended)

Install related dependencies:

Pip install-r requirement.txt

Configure eleasticsearch, kibana environment (docker is recommended)

Elasticsearch, Logstash, Kibana 7.2.0

Data source

There are many ways to obtain data, charge or free, as amateur natural choice is free, here choose tushare.pro, but in fact, tushare will have some restrictions, such as the frequency of obtaining data is limited, and the interface is also limited, requiring a lot of points. If you are interested in signing up for this, please register through my recommended link so that I can share more about data visualization and share the data I downloaded.

Https://tushare.pro/register?reg=277890

It is worth noting that there are almost unlimited free versions of tushare. But pro version data is more complete, in order to avoid later maintenance costs, so choose pro version.

In fact, there are other free ways to obtain data, you can try it yourself.

PytdxfooltraderQUANTAXIS acquires data

Configure your own token

Import tushare as tsts.set_token (") pro = ts.pro_api (")

For access to Token, please refer to the link.

Https://tushare.pro/document/1?doc_id=39

Try to get the data manually

Take the whole history of a certain day df = pro.daily (trade_date='20190725') df.head () ts_code trade_date open high low close pre_close change pct_chg vol amount value0 000032.SZ 20190725 9.49 9.60 9.47 9.56 9.49 0.7376 12658.35 12075.625 8906.9810001 000060.SZ 20190725 4.39 4.40 4.35 4.364.39-0.6834 129331.65 56462.292-38586.3303532 000078.SZ 20190725 3.37 3.38 3.38 3.37 0.2967 76681.00 25795.633 7653.5643113 000090.SZ 20190725 5.66 5.66 5.66 5.61 5.64-0.03-0.5319 105582.72 59215.389 -31496.6654094 000166.SZ 20190725 4.97 4.98 4.93 4.96 4.97-0.2012 268122.48 132793.120-26717.975744 get the daily market data of a stock data = ts.pro_bar (ts_code= "601668.SH" Adj='qfq' Start_date= "20120101") data.head () ts_code trade_date open high low close pre_close change pct_chg vol amount0 601668.SH 20190726 6.01 6.06 5.98 6.04-0.17 696833.16 419634.5471 601668.SH 20190725 6.05 6.07 6.04 0.00 543074.55 327829 .3802 601668.SH 20190724 6.09 6.11 6.02 6.050.010.17 788228.12 477542.6093 601668.SH 20190723 5.93 6.075.95.94 0.11 1.85 1077243.46 650250.0214 601668.SH 201907226.035.92 5.946.000.06 -1.00 811369.73 485732.343

Data acquisition naturally needs to be automated, but due to interface limitations, the following issues need to be considered.

The stock list determines whether the interface limit is exceeded, and if so, pauses for a period of time

Key code part

Def save_data (code, start_date, fp): print ("download stock (% s) daily line data to% s"% (code, fp)) try: data = ts.pro_bar (ts_code=code, adj='qfq') Start_date=start_date) # return None if data is None: time.sleep (10) return pass_set.add (code) except Exception: time.sleep (10) print ("stock:% s download failed"% code) return if len (data) = 0: pass_set when the call limit is exceeded .add (code) return try: data.trade_date = pd.to_datetime (data.trade_date) data = data.sort_values ("trade_date") if path.exists (fp): data.to_csv (fp) Mode= "a", header=False, index=False) else: data.to_csv (fp, index=False) except Exception: print ("Stock:% s save failed"% code)

You can refer to the save_data.py of my GitHub warehouse and download the data automatically through the following command

Python save_data.py

The starting time of the configuration in the code is 2012-01-01. Classes that need to be changed will be changed by themselves. It is worth noting that you need to configure a config.json in the same level directory, as follows

{"token": ""}

With your own token.

Configure elasticsearch, kibana

Docker is used here for configuration.

# pull image docker pull sebp/elk:720# to start docker environment docker run-p 5601 docker run 5601-p 9200 docker run 9200-p 504415 5044-v / home/elasticsearch/:/var/lib/elasticsearch-itd sebp/elk:720 dump data

Upload the data to elasticsearch for data analysis

Configure settings.py

# change ip:port to your own elasticsearch address, such as 192.168.56.102:9200config ["es_host"] = ["ip:port"]

Run the code

# upload stock data python cmd.py dump# upload Shanghai Stock Index data python cmd.py dump_index visualization

It takes some time to configure kibana. Fortunately, most configurations of kibana now support import and export, so you can import directly through the export.ndjson file of my warehouse.

Effect display

Due to the limited interface and limited access to stock factors, when I have more points, I will add more dashboard and visualization.

Postscript

Hope to complete their own series of articles on building a trading system from scratch, and then lead to the road to freedom of working hours and places.

Don't ask for absolute wealth and freedom, wish time and place are free ^ _ ^

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.