Longer Vision Technology

Github Blog

0%

Posted on Edited on In

With the most inspiring speech by Arnold Schwarzenegger, 2019 arrived… I’ve got something in my mind as well…

Schwarzenegger's Speech

Everybody has a dream/goal. NEVER EVER blow it out… Here comes the preface of my PhD’s thesis… It’s been 10 years already… This is NOT ONLY my attitude towards science, BUT ALSO towards those so-called professors(叫兽) and specialists(砖家).

Preface of My PhD Thesis

So, today, I’m NOT going to repeat any cliches, BUT happily encouraged by Arnold Schwarzenegger’s speech. We will definitely have a fruitful 2019. Let’s have some pizza.

Pizza

Posted on Edited on In

Merry Christmas everybody. Let’s take a look at the Christmas Lights around Lafarge Lake in Coquitlam FIRST.

Lafarge Lake Christmas Lights 01 Lafarge Lake Christmas Lights 02
Lafarge Lake Christmas Lights 01 Lafarge Lake Christmas Lights 02
Lafarge Lake Christmas Lights 03 Lafarge Lake Christmas Lights 04
Lafarge Lake Christmas Lights 03 Lafarge Lake Christmas Lights 04
Lafarge Lake Christmas Lights 05 Lafarge Lake Christmas Lights 06
Lafarge Lake Christmas Lights 05 Lafarge Lake Christmas Lights 06

Now, it’s the time to test PyDicom.

Get Started

Copy and paste that piece of code from Medical Image Analysis with Deep Learning  with trivial modifications as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
import cv2
import numpy as np
import matplotlib.pyplot as plt
import pydicom as pdicom
import os
import glob
import pandas as pd
import scipy.ndimage
from skimage import measure, morphology
from mpl_toolkits.mplot3d.art3d import Poly3DCollection


# Prepare dicom images
INPUT_FOLDER = './'
patients = os.listdir(INPUT_FOLDER)
patients.sort()


# Collect all dicom images
lstFilesDCM = []   # create an empty list
def load_scan2(path):
    for dirName, subdirList, fileList in os.walk(path):
        for filename in fileList:
            if ".dcm" in filename.lower():  
                lstFilesDCM.append(os.path.join(dirName, filename))
    return lstFilesDCM

first_patient = load_scan2(INPUT_FOLDER)


# Get ref file
print (lstFilesDCM[0])
RefDs = pdicom.read_file(lstFilesDCM[0])

# Load dimensions based on the number of rows, columns and slices (along z axis)
ConstPixelDims = (int(RefDs.Rows), int(RefDs.Columns), len(lstFilesDCM))

# Load spacing values (in mm)
ConstPixelSpacing = (float(RefDs.PixelSpacing[0]), float(RefDs.PixelSpacing[1]), float(RefDs.SliceThickness))


x = np.arange(0.0, (ConstPixelDims[0]+1)*ConstPixelSpacing[0], ConstPixelSpacing[0])
y = np.arange(0.0, (ConstPixelDims[1]+1)*ConstPixelSpacing[1], ConstPixelSpacing[1])
z = np.arange(0.0, (ConstPixelDims[2]+1)*ConstPixelSpacing[2], ConstPixelSpacing[2])


# The array is sized based on 'ConstPixelDims'

ArrayDicom = np.zeros(ConstPixelDims, dtype=RefDs.pixel_array.dtype)

# loop through all the DICOM files
for filenameDCM in lstFilesDCM:
    # read the file
    ds = pdicom.read_file(filenameDCM)
    # store the raw image data
    ArrayDicom[:, :, lstFilesDCM.index(filenameDCM)] = ds.pixel_array


plt.figure(dpi=1600)
plt.axes().set_aspect('equal', 'datalim')
plt.set_cmap(plt.gray())
plt.pcolormesh(y, x, np.flipud(ArrayDicom[:, :, 80]))
plt.show()

Display DICOM

I did a MRI scan in 2015, in SIAT, for the purpose of research. The scan resolution is NOT very high. Anyway, a bunch of DICOM images can be viewed as follows:

DICOM images

MRI - me

Posted on Edited on In

Merry Christmas buddy!

Christmas Lights in Surrey

It’s been quite a while without writing anything. Today, we are going to introduce PyFlux for time seriers analysis. Cannonical models are to be directly adopted from PyFlux Documents and tested in this blog.

Get Started

Copy and paste that piece of code from PyFlux Documents with trivial modifications as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
import pandas as pd
import numpy as np
from pandas_datareader.data import DataReader
from datetime import datetime
import pyflux as pf
import matplotlib.pyplot as plt


a = DataReader('JPM',  'yahoo', datetime(2006,6,1), datetime(2016,6,1))
a_returns = pd.DataFrame(np.diff(a['Adj Close'].values))
a_returns.index = a.index.values[1:a.index.values.shape[0]]
a_returns.columns = ["JPM Returns"]

print(a_returns.head())

plt.figure(figsize=(15, 5))
plt.ylabel("Returns")
plt.plot(a_returns)
plt.show()

pf.acf_plot(a_returns.values.T[0])
pf.acf_plot(np.square(a_returns.values.T[0]))

After running the above piece of code, we’ll get the following time series loaded:

1
2
3
4
5
6
            JPM Returns
2006-06-02     0.167995
2006-06-05    -0.613506
2006-06-06    -0.430914
2006-06-07    -0.094942
2006-06-08     0.073048

and the following resultant figures:

Time Series

Autocorrelation

Squared Autocorrelation

Cannonical Models

ARIMA

We can load and test ARIMA model using the following piece of code:

1
2
3
4
5
6
7
8
9
10
my_model = pf.ARIMA(data=a_returns, ar=4, ma=4, family=pf.Normal())
print(my_model.latent_variables)

result = my_model.fit("MLE")
result.summary()

my_model.plot_z(figsize=(15,5))
my_model.plot_fit(figsize=(15,10))
my_model.plot_predict_is(h=50, figsize=(15,5))
my_model.plot_predict(h=20,past_values=20,figsize=(15,5))

ARIMA’s latent variables are printed as:

1
2
3
4
5
6
7
8
9
10
11
12
Index    Latent Variable           Prior           Prior Hyperparameters     V.I. Dist  Transform
======== ========================= =============== ========================= ========== ==========
0        Constant                  Normal          mu0: 0, sigma0: 3         Normal     None
1        AR(1)                     Normal          mu0: 0, sigma0: 0.5       Normal     None
2        AR(2)                     Normal          mu0: 0, sigma0: 0.5       Normal     None
3        AR(3)                     Normal          mu0: 0, sigma0: 0.5       Normal     None
4        AR(4)                     Normal          mu0: 0, sigma0: 0.5       Normal     None
5        MA(1)                     Normal          mu0: 0, sigma0: 0.5       Normal     None
6        MA(2)                     Normal          mu0: 0, sigma0: 0.5       Normal     None
7        MA(3)                     Normal          mu0: 0, sigma0: 0.5       Normal     None
8        MA(4)                     Normal          mu0: 0, sigma0: 0.5       Normal     None
9        Normal Scale              Flat            n/a (non-informative)     Normal     exp

And the fitting result is summarized as:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Normal ARIMA(4,0,4)
======================================================= ==================================================
Dependent Variable: JPM Returns                         Method: MLE
Start Date: 2006-06-08 00:00:00                         Log Likelihood: -3106.9508
End Date: 2016-06-02 00:00:00                           AIC: 6233.9016
Number of observations: 2514                            BIC: 6292.1979
==========================================================================================================
Latent Variable                          Estimate   Std Error  z        P>|z|    95% C.I.
======================================== ========== ========== ======== ======== =========================
Constant                                 0.0126     0.0136     0.9273   0.3538   (-0.0141 | 0.0394)
AR(1)                                    0.0596     0.198      0.3008   0.7635   (-0.3284 | 0.4475)
AR(2)                                    -0.3591    0.1858     -1.9331  0.0532   (-0.7231 | 0.005)
AR(3)                                    -0.2205    0.2687     -0.8208  0.4118   (-0.7471 | 0.3061)
AR(4)                                    0.5272     0.1373     3.8406   0.0001   (0.2581 | 0.7962)
MA(1)                                    -0.1667    0.1995     -0.8354  0.4035   (-0.5578 | 0.2244)
MA(2)                                    0.3234     0.202      1.6008   0.1094   (-0.0726 | 0.7193)
MA(3)                                    0.1603     0.2396     0.6692   0.5033   (-0.3093 | 0.63)
MA(4)                                    -0.5817    0.1618     -3.5942  0.0003   (-0.8989 | -0.2645)
Normal Scale                             0.8331
==========================================================================================================

And 4 images are to be produced as:

ARIMA Latent Variable Plot

ARIMA Model vs. Data

ARIMA Forcast vs. Data

ARIMA Forcast

GARCH

We can load and test GARCH model using the following piece of code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
my_model = pf.GARCH(p=1, q=1, data=a_returns)
print(my_model.latent_variables)

my_model.adjust_prior(1, pf.TruncatedNormal(0.01, 0.5, lower=0.0, upper=1.0))
my_model.adjust_prior(2, pf.TruncatedNormal(0.97, 0.5, lower=0.0, upper=1.0))

result = my_model.fit('M-H', nsims=20000)
result.summary()

my_model.plot_z([1,2])
my_model.plot_fit(figsize=(15,5))
my_model.plot_sample(nsims=10, figsize=(15,7))

from scipy.stats import kurtosis
my_model.plot_ppc(T=kurtosis)
my_model.plot_predict(h=30, figsize=(15,5))

GARCH’s latent variables are printed as:

1
2
3
4
5
6
Index    Latent Variable           Prior           Prior Hyperparameters     V.I. Dist  Transform
======== ========================= =============== ========================= ========== ==========
0        Vol Constant              Normal          mu0: 0, sigma0: 3         Normal     exp
1        q(1)                      Normal          mu0: 0, sigma0: 0.5       Normal     logit
2        p(1)                      Normal          mu0: 0, sigma0: 0.5       Normal     logit
3        Returns Constant          Normal          mu0: 0, sigma0: 3         Normal     None

And the fitting result is summarized as:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
~/.local/lib/python3.6/site-packages/numdifftools/limits.py:126: UserWarning: All-NaN slice encountered
  warnings.warn(str(msg))
Acceptance rate of Metropolis-Hastings is 0.000125
Acceptance rate of Metropolis-Hastings is 0.00075
Acceptance rate of Metropolis-Hastings is 0.105525
Acceptance rate of Metropolis-Hastings is 0.13335
Acceptance rate of Metropolis-Hastings is 0.1907
Acceptance rate of Metropolis-Hastings is 0.232
Acceptance rate of Metropolis-Hastings is 0.299

Tuning complete! Now sampling.
Acceptance rate of Metropolis-Hastings is 0.36655
GARCH(1,1)
======================================================= ==================================================
Dependent Variable: JPM Returns                         Method: Metropolis Hastings
Start Date: 2006-06-05 00:00:00                         Unnormalized Log Posterior: -2671.5492
End Date: 2016-06-02 00:00:00                           AIC: 5351.717896880396
Number of observations: 2517                            BIC: 5375.041188860938
==========================================================================================================
Latent Variable                          Median             Mean               95% Credibility Interval
======================================== ================== ================== =========================
Vol Constant                             0.0059             0.0057             (0.004 | 0.0076)
q(1)                                     0.0721             0.0728             (0.0556 | 0.0921)
p(1)                                     0.9202             0.9199             (0.9013 | 0.9373)
Returns Constant                         0.0311             0.0311             (0.0109 | 0.0511)
==========================================================================================================

And 5 images are to be produced as:

GARCH Latent Variable Plot

GARCH Volatility

GARCH Volatility Sampling

GARCH Posterior Predictive

GARCH Forcast

GAS

VAR

Posted on Edited on In

This blog is just to solve some bugs while using MySQL as a server. Assuming you’ve already installed mysql-server, mysql-server-5.7, and mysql-server-core-5.7.

1. Errors

1.1 Failed to start mysql.service: Unit mysql.service not found.

1
2
3
4
5
6
7
➜  ~ sudo service mysqld start
[sudo] password for jiapei:
Failed to start mysqld.service: Unit mysqld.service not found.
➜  ~ /etc/init.d/mysql stop
[ ok ] Stopping mysql (via systemctl): mysql.service.
➜  ~ /etc/init.d/mysql start
[ ok ] Starting mysql (via systemctl): mysql.service.

Conclusion: use /etc/init.d/mysql start instead of service mysqld start.

1.2 ERROR 1045 (28000): Access denied for user ‘root‘@’localhost’ (using password: YES)

1
2
3
➜  ~ mysql -u root -p
Enter password: 
ERROR 1698 (28000): Access denied for user 'root'@'localhost'

We FIRST list all existing MySQL processes and kill them all.

1
2
3
➜  ~ ps aux | grep mysql
...
➜  ~ sudo kill PID

Util now, there should be NO MySQL process running.

1.3 ERROR 2002 (HY000): Can’t connect to local MySQL server through socket ‘/var/run/mysqld/mysqld.sock’

Since there is NO MySQL process running, of course we cannot connect to MySQL server.

1
2
3
4
5
6
7
➜  ~ mysql -u root -p
Enter password: 
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
➜  ~ /etc/init.d/mysql start
[....] Starting mysql (via systemctl): mysql.serviceJob for mysql.service failed because the control process exited with error code.
See "systemctl status mysql.service" and "journalctl -xe" for details.
 failed!

Then, we will have to have these 3 packages reinstalled: mysql-server, mysql-server-5.7, and mysql-server-core-5.7.

1
2
3
4
➜  ~ sudo apt remove mysql-server mysql-server-5.7 mysql-server-core-5.7
...
➜  ~ sudo apt install mysql-server mysql-server-5.7 mysql-server-core-5.7
...

/var/run/mysqld/mysqld.sock is NOW back, and MySQL seems to run automatically.

1
2
3
4
5
➜  ~ ls -ls /var/run/mysqld/mysqld.sock
0 srwxrwxrwx 1 mysql mysql 0 Nov 30 03:08 /var/run/mysqld/mysqld.sock
➜  ~ ps aux | grep mysql
mysql    13785  0.1  0.3 1323300 172680 ?      Sl   03:08   0:00 /usr/sbin/mysqld --daemonize --pid-file=/run/mysqld/mysqld.pid
jiapei   14453  0.0  0.0  21556  2560 pts/0    R+   03:16   0:00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn mysql

Then, use chown and chmod to set up the owners and permissions suitably as follows.

1
2
3
4
5
6
7
8
9
10
11
12
13
➜  run pwd
/var/run
➜  run sudo chown mysql:mysql -R mysql
➜  run sudo chmod 755 -R mysql
➜  run ls -lsd mysql
0 drwxr-xr-x 2 mysql mysql 100 Nov 30 02:38 mysqld
➜  run cd ../lib
➜  lib pwd
/var/lib
➜  lib sudo chown mysql:mysql -R mysql
➜  lib sudo chmod 755 -R mysql
➜  lib ls -lsd mysql
4 drwxr-xr-x 6 mysql mysql 4096 Nov 30 02:38 mysql

1.4 ERROR 1698 (28000): Access denied for user ‘root‘@’localhost’

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
➜  ~ mysql -u root -p
Enter password: 
ERROR 1698 (28000): Access denied for user 'root'@'localhost'
➜  ~ sudo mysql -u root
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.24-0ubuntu0.18.04.1 (Ubuntu)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> USE lvrsql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> CREATE USER 'YOUR_SYSTEM_USER'@'localhost' IDENTIFIED BY 'YOUR_PASSWORD';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'YOUR_SYSTEM_USER'@'localhost';
mysql> FLUSH PRIVILEGES;
mysql> exit;

➜  ~ service mysql restart
➜  ~

1.5 [ERROR] InnoDB: Unable to lock ./ibdata1 (Additional)

If you meet the above ERROR message, what you can do is to restart MySQL by:

1
2
3
➜  ~ /etc/init.d/mysql restart
[ ok ] Restarting mysql (via systemctl): mysql.service.
➜  ~

2. Fundamental Commands in MySQL

Posted on Edited on In

Finally, I’m talking about Geographic Information System, namely, GIS. Remote Sensing and Information Engineer is my bachelor’s major in Wuhan Technical University of Surveying and Mapping, which is NOW a part of Wuhan University. When I was in the university, I NEVER thought I was in such an amazing department, studying such an amazing major.

Alright… Let’s start today’s blog. We’re going to play OpenStreetMap for some fun.

1. ArcGIS with OpenStreetMap

We FIRST test out how ArcGIS displays OpenStreetMap.

1.1 ArcGIS 3.26

A snippet of sample code is provided for ArcGIS Version 3.26, cited as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
<!DOCTYPE html>
<html> 
  <head> 
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <meta name="viewport" content="initial-scale=1, maximum-scale=1,user-scalable=no" />
    <title>OpenStreetMap</title> 
    <link rel="stylesheet" href="https://js.arcgis.com/3.26/esri/css/esri.css">
    <style>
      html, body, #esri-map-container {
        padding: 0px;
        margin: 0px;
        height: 100%;
      }
    </style>
    <script src="https://js.arcgis.com/3.26/"></script>
    <script>
      require([
        "esri/map",
        "esri/layers/OpenStreetMapLayer",
        "dojo/domReady!"
      ], function (Map, OpenStreetMapLayer){

        var map, openStreetMapLayer;

        map = new Map("esri-map-container", {
          center: [-89.924, 30.036],
          zoom: 12
        });
        openStreetMapLayer = new OpenStreetMapLayer();
        map.addLayer(openStreetMapLayer);
      });
    </script>
  </head> 
  
  <body>
    <div id="esri-map-container"></div>
  </body> 
</html>

Please click on the following image and have a try.

ArcGIS 3.26

1.2 ArcGIS LATEST

A snippet of sample code is provided for ArcGIS LATEST Version 4.9, cited as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="initial-scale=1,maximum-scale=1,user-scalable=no">
  <title>OpenStreetMapLayer - 4.9</title>

  <style>
    html,
    body,
    #viewDiv {
      padding: 0;
      margin: 0;
      height: 100%;
      width: 100%;
    }
  </style>

  <link rel="stylesheet" href="https://js.arcgis.com/4.9/esri/css/main.css">
  <script src="https://js.arcgis.com/4.9/"></script>

  <script>
    require([
        "esri/layers/OpenStreetMapLayer",
        "esri/Map",
        "esri/views/SceneView"
      ],
      function(
        OpenStreetMapLayer,
        Map,
        SceneView
      ) {

        var map = new Map({
          ground: "world-elevation"
        });

        var view = new SceneView({
          map: map,
          container: "viewDiv"
        });

        var osmLayer = new OpenStreetMapLayer();
        map.add(osmLayer);
      });
  </script>
</head>

<body>
  <div id="viewDiv"></div>
</body>

</html>

Please click on the following image and have a try.

ArcGIS 4.9

ArcGIS is like the CROWN of GIS. Its SDK/API seems NOT fully open source but commercial. Therefore, I’d love to breifly test out some other open source.

2. WebGLEarth

Several WebGLEarth examples are given at http://examples.webglearth.com/, and we just try out the very FIRST example Satellite. The code is cited as:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
<!DOCTYPE HTML>
<html>
  <head>
    <script src="http://www.webglearth.com/v2/api.js"></script>
    <script>
      function initialize() {
        var options = {atmosphere: true, center: [0, 0], zoom: 0};
        var earth = new WE.map('earth_div', options);
        WE.tileLayer('http://tileserver.maptiler.com/nasa/{z}/{x}/{y}.jpg', {
          minZoom: 0,
          maxZoom: 5,
          attribution: 'NASA'
        }).addTo(earth);
      }
    </script>
    <style>
      html, body{padding: 0; margin: 0;}
      #earth_div{top: 0; right: 0; bottom: 0; left: 0;
                 background-color: #000; position: absolute !important;}
      </style>
      <title>WebGL Earth API: Satellite</title>
    </head>
    <body onload="initialize()">
      <div id="earth_div"></div>
  </body>
</html>

Please click on the following image and have a try.

LeafletJS

3. OpenGlobus Based On OpenStreetMap

“OpenGlobus is a javascript library designed to display interactive 3d maps and planets with map tiles, imagery and vector data, markers and 3d objects. It uses the WebGL technology, open source and completely free.” (Cited from OpenGlobus Github). Let’s just take the very FIRST example Og - examples: Attribution as our example. Code is cited as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
<html>

<head>
    <title>Attribution Example</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link rel="stylesheet" href="http://www.openglobus.org/og.css" type="text/css" />
    <script src="http://www.openglobus.org/og.js"></script>
</head>

<body>
    <div id="globus" style="width:100%;height:100%"></div>
    <button id="btnOSM" style="position: absolute; left:0; margin:10px;">Set attribution</button>
    <script>

        document.getElementById("btnOSM").onclick = function () {
            states.setAttribution("Hello, WMS default USA population states example!");
        };

        let osm = new og.layer.XYZ("OpenStreetMap", {
            specular: [0.0003, 0.00012, 0.00001],
            shininess: 20,
            diffuse: [0.89, 0.9, 0.83],
            isBaseLayer: true,
            url: "http://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png",
            visibility: true,
            attribution: 'Data @ OpenStreetMap contributors, ODbL'
        });

        let states = new og.layer.WMS("USA Population", {
            extent: [[-128, 24], [-66, 49]],
            visibility: true,
            isBaseLayer: false,
            url: "http://openglobus.org/geoserver/",
            layers: "topp:states",
            opacity: 1.0,
            attribution: 'Hi!',
            transparentColor: [1.0, 1.0, 1.0]
        });

        let globus = new og.Globe({
            "target": "globus",
            "name": "Earth",
            "terrain": new og.terrain.GlobusTerrain(),
            "layers": [osm, states]
        });

        globus.planet.flyExtent(states.getExtent());
    </script>
</body>

</html>

Please click on the following image and have a try.

OpenGlobus

4. LeafletJS Based On OpenStreetMap

Several LeafletJS examples are given at https://leafletjs.com/examples.html. Here, we took Leaflet on Mobile as an example (code copied).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
<!DOCTYPE html>
<html>
<head>

	<title>Mobile tutorial - Leaflet</title>

	<meta charset="utf-8" />
	<meta name="viewport" content="width=device-width, initial-scale=1.0">

	<link rel="shortcut icon" type="image/x-icon" href="docs/images/favicon.ico" />

    <link rel="stylesheet" href="https://unpkg.com/leaflet@1.3.4/dist/leaflet.css" integrity="sha512-puBpdR0798OZvTTbP4A8Ix/l+A4dHDD0DGqYW6RQ+9jxkRFclaxxQb/SJAWZfWAkuyeQUytO7+7N4QKrDh+drA==" crossorigin=""/>
    <script src="https://unpkg.com/leaflet@1.3.4/dist/leaflet.js" integrity="sha512-nMMmRyTVoLYqjP9hrbed9S+FzjZHW5gY1TWCHA5ckwXZBadntCNs8kEqAWdrb9O7rxbCaA4lKTIWjDXZxflOcA==" crossorigin=""></script>


	<style>
		html, body {
			height: 100%;
			margin: 0;
		}
		#map {
			width: 600px;
			height: 400px;
		}
	</style>

	<style>body { padding: 0; margin: 0; } #map { height: 100%; width: 100vw; }</style>
</head>
<body>

<div id='map'></div>

<script>
	var map = L.map('map').fitWorld();

	L.tileLayer('https://api.tiles.mapbox.com/v4/{id}/{z}/{x}/{y}.png?access_token=pk.eyJ1IjoibWFwYm94IiwiYSI6ImNpejY4NXVycTA2emYycXBndHRqcmZ3N3gifQ.rJcFIG214AriISLbB6B5aw', {
		maxZoom: 18,
		attribution: 'Map data &copy; <a href="https://www.openstreetmap.org/">OpenStreetMap</a> contributors, ' +
			'<a href="https://creativecommons.org/licenses/by-sa/2.0/">CC-BY-SA</a>, ' +
			'Imagery © <a href="https://www.mapbox.com/">Mapbox</a>',
		id: 'mapbox.streets'
	}).addTo(map);

	function onLocationFound(e) {
		var radius = e.accuracy / 2;

		L.marker(e.latlng).addTo(map)
			.bindPopup("You are within " + radius + " meters from this point").openPopup();

		L.circle(e.latlng, radius).addTo(map);
	}

	function onLocationError(e) {
		alert(e.message);
	}

	map.on('locationfound', onLocationFound);
	map.on('locationerror', onLocationError);

	map.locate({setView: true, maxZoom: 16});
</script>

</body>
</html>

Please click on the following image and have a try.

LeafletJS

5. Mapbox GL JS Based on OpenStreetMap

From the above LeafletJS example, it’s clearly show that LeafletJS is based on Mapbox, which is further based on OpenStreetMap. “Mapbox is a Live Location Platform.” (Cited from https://www.mapbox.com/).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
<!DOCTYPE html>
<html>
<head>
    <meta charset='utf-8' />
    <title>Display a map</title>
    <meta name='viewport' content='initial-scale=1,maximum-scale=1,user-scalable=no' />
    <script src='https://api.tiles.mapbox.com/mapbox-gl-js/v0.51.0/mapbox-gl.js'></script>
    <link href='https://api.tiles.mapbox.com/mapbox-gl-js/v0.51.0/mapbox-gl.css' rel='stylesheet' />
    <style>
        body { margin:0; padding:0; }
        #map { position:absolute; top:0; bottom:0; width:100%; }
    </style>
</head>
<body>

<div id='map'></div>
<script>
mapboxgl.accessToken = 'pk.eyJ1IjoibG9uZ2VydmlzaW9uIiwiYSI6ImNqb2ZpN2Z5ZjA0bnMzd3FoY29oeTZwNW0ifQ.th9s23R3Njgxewe9rDdXzA';
var map = new mapboxgl.Map({
    container: 'map', // container id
    style: 'mapbox://styles/mapbox/streets-v9', // stylesheet location
    center: [-74.50, 40], // starting position [lng, lat]
    zoom: 9 // starting zoom
});
</script>

</body>
</html>

Please click on the following image and have a try.

Mapbox GL

6. OSMBuildings Based On OpenStreetMap

OSMBuildings is “a 3D building geometry viewer based on OpenStreetMap data.” (Cited from https://github.com/OSMBuildings/OSMBuildings). Unfortunately, from OSMBuildings Registration, it is NOT free. However, its demonstrations are amazing - please refer to https://osmbuildings.org/developer/.

OSMBuildings Examples

7. Tiles Upon OpenStreetMap

OpenStreetMap may be covered by different tiles in order to show different maps. For instance, Google map vs Google satellite map. Let’s take a look at Home Page of WebGLEarth, you’ll notice MapTiler at the left bottom corner. Here, we summarize 3 widely used tiles in the following:

Generally, there are the following types of common tiles:

  • OpenStreetMap vector tiles
  • Contour lines vector tiles
  • Hillshading raster tiles
  • Landcover vector tiles
  • Satellite raster tiles
  • Terrain - quantized mesh tiles
  • etc.

8. OpenLayers

Besides tiles, there might be several layers (including one layer of tile) upon OpenStreetMap, here comes OpenLayers: “OpenLayers makes it easy to put a dynamic map in any web page. It can display map tiles, vector data and markers loaded from any source.” (cited from https://openlayers.org/)

A simple example of is in OpenLayers QuickStart. The code is cited as:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
<!doctype html>
<html lang="en">
  <head>
    <link rel="stylesheet" href="https://cdn.rawgit.com/openlayers/openlayers.github.io/master/en/v5.3.0/css/ol.css" type="text/css">
    <style>
      .map {
        height: 400px;
        width: 100%;
      }
    </style>
    <script src="https://cdn.rawgit.com/openlayers/openlayers.github.io/master/en/v5.3.0/build/ol.js"></script>
    <title>OpenLayers example</title>
  </head>
  <body>
    <h2>My Map</h2>
    <div id="map" class="map"></div>
    <script type="text/javascript">
      var map = new ol.Map({
        target: 'map',
        layers: [
          new ol.layer.Tile({
            source: new ol.source.OSM()
          })
        ],
        view: new ol.View({
          center: ol.proj.fromLonLat([37.41, 8.82]),
          zoom: 4
        })
      });
    </script>
  </body>
</html>

Please click on the following image and have a try.

OpenLayers Quick Start

9. CesiumJS

There is a complete example from Cesium Tutorial, which is cited directly as:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <script src="https://cesium.com/downloads/cesiumjs/releases/1.68/Build/Cesium/Cesium.js"></script>
  <link href="https://cesium.com/downloads/cesiumjs/releases/1.68/Build/Cesium/Widgets/widgets.css" rel="stylesheet">
</head>
<body>
  <div id="cesiumContainer" style="width: 700px; height:400px"></div>
  <script>
    Cesium.Ion.defaultAccessToken = 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiI1ZTcwZTRmMS1jNmZkLTQ1NzctYTU3ZC03Mzc1Y2U2YWU5OTMiLCJpZCI6MjY1MzUsInNjb3BlcyI6WyJhc3IiLCJnYyJdLCJpYXQiOjE1ODc5MDM0MDB9.SAx2MQ2F3mUmH2InfoHWJvUoQBhq1CvGwo_2Z8oNC7Y';
    var viewer = new Cesium.Viewer('cesiumContainer', {
        terrainProvider: Cesium.createWorldTerrain()
    });

    var tileset = viewer.scene.primitives.add(
        new Cesium.Cesium3DTileset({
            url: Cesium.IonResource.fromAssetId(your_asset_id)
        })
    );
    viewer.zoomTo(tileset);
  </script>
</body>
</html>

You are welcome to click on the following image and have a try.

A Completed Example of CesiumJS

10. Esri

As the LAST demo of this blog, let’s try something interesting: A Live Demo - Wind-JS.

You are of course welcome to click on the following image and have a try:

Esri Wind of the World

11. MapQuest

MapQuest provides Open Search (Nominatim) API, which relies solely on OpenStreetMap data.

Geocoding based on OpenStreetMap is exactly for ME.

😍

Several Geocoding services are provided as follows:

Please take a look at the bottom of my other website https://longervision.cc/.

Posted on Edited on In

Yesterday, we briefly introduced Quantopian and how to ingest Quandl WIKI Bundle data. Today, we are going to briefly summarize the financial data source and how to retrieve Google and Yahoo finance data using pandas_datareader.

1. Google Finance Data and Yahoo Finance Data

State-of-the-art finance data source are partly summarized at Financial Hacker. Our recommended solution is to retrieve stock data by using a SINGLE framework: pandas_datareader. Yahoo Finance Data and Google Finance Data can be easily retrieved by This Learn Data Science Tutorial.

Source Request Link Stock Exchange Symbol (INDEX) Stock Symbol (DATA)
Google Finance https://www.google.ca/finance?q=INDEX:DATA INDEX, ex:
INDEXDJX: Dow Jones
NASDAQ
INDEXNASDAQ
NYSE: NYSE COMPOSITE (DJ)
INDEXSP: S&P 500		 DATA, ex:
APPL
Yahoo Finance https://finance.yahoo.com/quote/DATA?p=DATA Totally, Yahoo provides the following indexes:
%5EGSPC: S&P 500
%5EDJI: Dow 30
%5EIXIC: NASDAQ Composite
%5ERUT: Russell 2000
CL=F: Crude Oil Dec 18
GC=F: Gold Dec 18
SI=F: Silver Dec 18
%5ETNX: CBOE Interest Rate 10 Year T No
%5EVIX: CBOE Volatility Index
BTC-USD: Bitcoin USD
%5EFTSE: FTSE 100
%5EN225: Nikkei 225		 DATA ex:
APPL

2. Use pandas_datareader to Retrieve Finance Data

A comparatively old blog Python for Finance, Part I: Yahoo & Google Finance API, pandas, and matplotlib has already given a fabulous summary. However, with the development of Python, code for finance data loading needs to be modified a bit. For 4 different data resource:

2.1 Google Finance

1
2
3
4
5
6
7
# Define the instruments to download. We would like to see Apple, Microsoft and the S&P500 index.
tickers = ['AAPL', 'MSFT', '^GSPC']

# We would like all available data from 01/01/2017 until 12/31/2017.
start_date = '2017-01-01'
end_date = '2017-12-31'
df = data.DataReader(tickers, data_source='google', start=start_date, end=end_date)
Exception has occurred: pandas_datareader.exceptions.ImmediateDeprecationError Google finance has been immediately deprecated due to large breaks in the API without the introduction of a stable replacement. Pull Requests to re-enable these data connectors are welcome. See https://github.com/pydata/pandas-datareader/issues File "....../test.py", line 16, in df = data.DataReader(tickers, data_source='google', start=start_date, end=end_date)

From pandas-datareader issues 487, we can tell Google Finance API is currently down. Therefore, let’s ignore Google Finance for now.

On the other hand, from the following 3 finance data resource, the following printed tables and generated pictures demonstrate that stock data can be successfully loaded.

2.2 Yahoo Finance

1
2
3
4
5
6
7
# Define the instruments to download. We would like to see Apple, Microsoft and the S&P500 index.
tickers = ['AAPL', 'MSFT', '^GSPC']

# We would like all available data from 01/01/2017 until 12/31/2017.
start_date = '2017-01-01'
end_date = '2017-12-31'
df = data.DataReader(tickers, data_source='yahoo', start=start_date, end=end_date)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
Attributes        High                                 Low             \
Symbols           AAPL       MSFT        ^GSPC        AAPL       MSFT
Date
2017-01-03  116.330002  62.840000  2263.879883  114.760002  62.130001
2017-01-04  116.510002  62.750000  2272.820068  115.750000  62.119999
2017-01-05  116.860001  62.660000  2271.500000  115.809998  62.029999
2017-01-06  118.160004  63.150002  2282.100098  116.470001  62.040001
2017-01-09  119.430000  63.080002  2275.489990  117.940002  62.540001
2017-01-10  119.379997  63.070000  2279.270020  118.300003  62.279999
2017-01-11  119.930000  63.230000  2275.320068  118.599998  62.430000
2017-01-12  119.300003  63.400002  2271.780029  118.209999  61.950001
2017-01-13  119.620003  62.869999  2278.679932  118.809998  62.349998

Attributes                     Open                               Close  \
Symbols           ^GSPC        AAPL       MSFT        ^GSPC        AAPL
Date
2017-01-03  2245.129883  115.800003  62.790001  2251.570068  116.150002
2017-01-04  2261.600098  115.849998  62.480000  2261.600098  116.019997
2017-01-05  2260.449951  115.919998  62.189999  2268.179932  116.610001
2017-01-06  2264.060059  116.779999  62.299999  2271.139893  117.910004
2017-01-09  2268.899902  117.949997  62.759998  2273.590088  118.989998
2017-01-10  2265.270020  118.769997  62.730000  2269.719971  119.110001
2017-01-11  2260.830078  118.739998  62.610001  2268.600098  119.750000
2017-01-12  2254.250000  118.900002  63.060001  2271.139893  119.250000
2017-01-13  2271.510010  119.110001  62.619999  2272.739990  119.040001

Attributes                              Volume                            \
Symbols          MSFT        ^GSPC        AAPL        MSFT         ^GSPC
Date
2017-01-03  62.580002  2257.830078  28781900.0  20694100.0  3.770530e+09
2017-01-04  62.299999  2270.750000  21118100.0  21340000.0  3.764890e+09
2017-01-05  62.299999  2269.000000  22193600.0  24876000.0  3.761820e+09
2017-01-06  62.840000  2276.979980  31751900.0  19922900.0  3.339890e+09
2017-01-09  62.639999  2268.899902  33561900.0  20256600.0  3.217610e+09
2017-01-10  62.619999  2268.899902  24462100.0  18593000.0  3.638790e+09
2017-01-11  63.189999  2275.320068  27588600.0  21517300.0  3.620410e+09
2017-01-12  62.610001  2270.439941  27086200.0  20968200.0  3.462130e+09
2017-01-13  62.700001  2274.639893  26111900.0  19422300.0  3.081270e+09

Attributes   Adj Close
Symbols           AAPL       MSFT        ^GSPC
Date
2017-01-03  112.620964  60.431488  2257.830078
2017-01-04  112.494911  60.161095  2270.750000
2017-01-05  113.066986  60.161095  2269.000000
2017-01-06  114.327484  60.682560  2276.979980
2017-01-09  115.374664  60.489429  2268.899902
2017-01-10  115.491020  60.470112  2268.899902
2017-01-11  116.111588  61.020542  2275.320068
2017-01-12  115.626762  60.460461  2270.439941
2017-01-13  115.423149  60.547371  2274.639893
AAPL Closing Price MSFT Closing Price S&P 500 Closing Price
AAPL Closing Price MSFT Closing Price S&P 500 Closing Price

2.3 IEX

1
2
3
4
5
6
7
# Define the instruments to download. We would like to see Apple, Microsoft and the S&P500 index.
tickers = ['AAPL', 'MSFT', 'SPY']

# We would like all available data from 01/01/2017 until 12/31/2017.
start_date = '2017-01-01'
end_date = '2017-12-31'
df = data.DataReader(tickers, data_source="iex", start=start_date, end=end_date)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
Attributes      open                         high                     \
Symbols         AAPL     MSFT       SPY      AAPL     MSFT       SPY   
date                                                                   
2017-01-03  112.6732  60.6343  217.9106  113.1889  60.6825  218.6756   
2017-01-04  112.7219  60.3349  218.4722  113.3641  60.5956  219.5664   
2017-01-05  112.7900  60.0549  219.1016  113.7087  60.5087  219.4018   
2017-01-06  113.6268  60.1611  219.3534  114.9695  60.9819  220.5347   
2017-01-09  114.7652  60.6053  219.7213  116.2052  60.9143  219.8764   
2017-01-10  115.5630  60.5763  219.3050  116.1566  60.9046  220.2442   
2017-01-11  115.5338  60.4604  219.1888  116.6917  61.0591  219.9053   
2017-01-12  115.6847  60.8950  219.3243  116.0787  61.2233  219.5664   
2017-01-13  115.8939  60.4701  219.5470  116.3901  60.7067  220.1958   

Attributes       low                        close                     \
Symbols         AAPL     MSFT       SPY      AAPL     MSFT       SPY   
date                                                                   
2017-01-03  111.6613  59.9921  216.7909  113.0138  60.4315  218.1043   
2017-01-04  112.6246  59.9873  218.4625  112.8873  60.1611  219.4018   
2017-01-05  112.6830  59.9003  218.3366  113.4614  60.1611  219.2275   
2017-01-06  113.3251  59.9100  218.7433  114.7263  60.6825  220.0118   
2017-01-09  114.7554  60.3928  219.2433  115.7771  60.4894  219.2856   
2017-01-10  115.1057  60.1418  218.8499  115.8939  60.4701  219.2856   
2017-01-11  115.3976  60.2866  218.4432  116.5166  61.0205  219.9053   
2017-01-12  115.0182  59.8231  217.8283  116.0301  60.4604  219.3534   
2017-01-13  115.6020  60.2094  219.5083  115.8257  60.5473  219.8569   

Attributes    volume                      
Symbols         AAPL      MSFT       SPY  
date                                      
2017-01-03  28781865  20694101  91366522  
2017-01-04  21118116  21339969  78744433  
2017-01-05  22193587  24875968  78379012  
2017-01-06  31751900  19922919  71559922  
2017-01-09  33561948  20382730  46939676  
2017-01-10  24462051  18593004  63771939  
2017-01-11  27588593  21517335  74650016  
2017-01-12  27086220  20968223  72113181  
2017-01-13  26111948  19422310  62717865

As shown, for IEX, the retrieved data is in reversed order, and there is a hidden ERROR about the time so far. Let’s just ignore it at this stage.

2.4 Quandl

Note: for Quandl,

  • you may have to load ONE data at a time
  • you may have to subscribe and pay to obtain S&P 500 data? Anyway, please refer to Quandl MULTPL and multpl.com
1
2
3
4
5
# We would like all available data from 01/01/2017 until 12/31/2017.
start_date = '2017-01-01'
end_date = '2017-12-31'
df = data.DataReader('AAPL', data_source="quandl", start=start_date, end=end_date, access_key="your-quandl-api-key")
# df = data.DataReader('MSFT', data_source="quandl", start=start_date, end=end_date, access_key="your-quandl-api-key")

AAPL

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
              Open     High      Low   Close      Volume  ExDividend  \
Date
2017-12-29  170.52  170.590  169.220  169.23  25643711.0         0.0
2017-12-28  171.00  171.850  170.480  171.08  15997739.0         0.0
2017-12-27  170.10  170.780  169.710  170.60  21672062.0         0.0
2017-12-26  170.80  171.470  169.679  170.57  32968167.0         0.0
2017-12-22  174.68  175.424  174.500  175.01  16052615.0         0.0
2017-12-21  174.17  176.020  174.100  175.01  20356826.0         0.0
2017-12-20  174.87  175.420  173.250  174.35  23000392.0         0.0
2017-12-19  175.03  175.390  174.090  174.54  27078872.0         0.0
2017-12-18  174.88  177.200  174.860  176.42  28831533.0         0.0

            SplitRatio  AdjOpen  AdjHigh   AdjLow  AdjClose   AdjVolume
Date
2017-12-29         1.0   170.52  170.590  169.220    169.23  25643711.0
2017-12-28         1.0   171.00  171.850  170.480    171.08  15997739.0
2017-12-27         1.0   170.10  170.780  169.710    170.60  21672062.0
2017-12-26         1.0   170.80  171.470  169.679    170.57  32968167.0
2017-12-22         1.0   174.68  175.424  174.500    175.01  16052615.0
2017-12-21         1.0   174.17  176.020  174.100    175.01  20356826.0
2017-12-20         1.0   174.87  175.420  173.250    174.35  23000392.0
2017-12-19         1.0   175.03  175.390  174.090    174.54  27078872.0
2017-12-18         1.0   174.88  177.200  174.860    176.42  28831533.0

MSFT

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
             Open     High     Low  Close      Volume  ExDividend  SplitRatio  \
Date                                                                            
2017-12-29  85.63  86.0500  85.500  85.54  18162779.0         0.0         1.0   
2017-12-28  85.90  85.9300  85.550  85.72   9872795.0         0.0         1.0   
2017-12-27  85.65  85.9800  85.215  85.71  13000828.0         0.0         1.0   
2017-12-26  85.31  85.5346  85.030  85.40   9737412.0         0.0         1.0   
2017-12-22  85.40  85.6300  84.920  85.51  14033977.0         0.0         1.0   
2017-12-21  86.05  86.1000  85.400  85.50  16638402.0         0.0         1.0   
2017-12-20  86.20  86.3000  84.710  85.52  23425009.0         0.0         1.0   
2017-12-19  86.35  86.3500  85.270  85.83  23241979.0         0.0         1.0   
2017-12-18  87.12  87.4999  86.230  86.38  21551076.0         0.0         1.0   

            AdjOpen  AdjHigh  AdjLow  AdjClose   AdjVolume  
Date                                                        
2017-12-29    85.63  86.0500  85.500     85.54  18162779.0  
2017-12-28    85.90  85.9300  85.550     85.72   9872795.0  
2017-12-27    85.65  85.9800  85.215     85.71  13000828.0  
2017-12-26    85.31  85.5346  85.030     85.40   9737412.0  
2017-12-22    85.40  85.6300  84.920     85.51  14033977.0  
2017-12-21    86.05  86.1000  85.400     85.50  16638402.0  
2017-12-20    86.20  86.3000  84.710     85.52  23425009.0  
2017-12-19    86.35  86.3500  85.270     85.83  23241979.0  
2017-12-18    87.12  87.4999  86.230     86.38  21551076.0

| AAPL Closing Price | MSFT Closing Price |
|:———-:|:———-:|:———-:|
| AAPL Closing Price | MSFT Closing Price |

3. Additional Python Tools

3.1 fix-yahoo-finance

fix-yahoo-finance.

3.2 googlefinance.client

googlefinance.client

Posted on Edited on In

Today, let’s have some fun of AI in finance. We are going to use a tool named Quantopian. As far as I know, functional programming language Haskell has been widely used in quantitative analysis. That’s possibly why a dependent package of Quantopian, namely pandoc, requires The Haskell Tool Stack for its installation. Therefore, Haskell and pandoc need to be installed before Quantopian installation.

1. Preparations

1.1 Haskell Installation

Please refer to https://docs.haskellstack.org/en/stable/install_and_upgrade/:

1
➜  ~ curl -sSL https://get.haskellstack.org/ | sh

1.2 pandoc Installation

Please refer to https://pandoc.org/installing.html:

1
2
3
4
➜  ~ git clone https://github.com/jgm/pandoc
➜  ~ cd pandoc
➜  ~ stack setup
➜  ~ stack install

1.3 Zipline Installation with pip (ERRORS)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
➜  ~ pip install -U zipline --user
Collecting zipline
  Using cached https://files.pythonhosted.org/packages/be/59/8c5802a7897c1095fdc409fb557f04df8f75c37174e80d2ba58c8d8a6488/zipline-1.3.0.tar.gz
Requirement already satisfied, skipping upgrade: pip>=7.1.0 in ~/.local/lib/python3.6/site-packages (from zipline) (18.1)
Requirement already satisfied, skipping upgrade: setuptools>18.0 in ~/.local/lib/python3.6/site-packages (from zipline) (40.5.0)
Collecting Logbook>=0.12.5 (from zipline)
  Using cached https://files.pythonhosted.org/packages/74/fc/3e7557ed1ef1bd4e3ee189fc670416abfc7192b550e8d3c1d858a63f41ab/Logbook-1.4.1.tar.gz
Requirement already satisfied, skipping upgrade: pytz>=2016.4 in ~/.local/lib/python3.6/site-packages (from zipline) (2018.7)
Requirement already satisfied, skipping upgrade: numpy>=1.11.1 in ~/.local/lib/python3.6/site-packages (from zipline) (1.15.4)
Requirement already satisfied, skipping upgrade: requests-file>=1.4.1 in ~/.local/lib/python3.6/site-packages (from zipline) (1.4.3)
Requirement already satisfied, skipping upgrade: scipy>=0.17.1 in ~/.local/lib/python3.6/site-packages (from zipline) (1.1.0)
Collecting pandas<=0.22,>=0.18.1 (from zipline)
  Using cached https://files.pythonhosted.org/packages/da/c6/0936bc5814b429fddb5d6252566fe73a3e40372e6ceaf87de3dec1326f28/pandas-0.22.0-cp36-cp36m-manylinux1_x86_64.whl
Collecting pandas-datareader>=0.2.1 (from zipline)
  Using cached https://files.pythonhosted.org/packages/cc/5c/ea5b6dcfd0f55c5fb1e37fb45335ec01cceca199b8a79339137f5ed269e0/pandas_datareader-0.7.0-py2.py3-none-any.whl
Requirement already satisfied, skipping upgrade: patsy>=0.4.0 in ~/.local/lib/python3.6/site-packages (from zipline) (0.5.1)
Collecting statsmodels>=0.6.1 (from zipline)
  Using cached https://files.pythonhosted.org/packages/85/d1/69ee7e757f657e7f527cbf500ec2d295396e5bcec873cf4eb68962c41024/statsmodels-0.9.0-cp36-cp36m-manylinux1_x86_64.whl
Requirement already satisfied, skipping upgrade: python-dateutil>=2.4.2 in ~/.local/lib/python3.6/site-packages (from zipline) (2.7.5)
Requirement already satisfied, skipping upgrade: six>=1.10.0 in ~/.local/lib/python3.6/site-packages (from zipline) (1.11.0)
Requirement already satisfied, skipping upgrade: requests>=2.9.1 in ~/.local/lib/python3.6/site-packages (from zipline) (2.20.0)
Requirement already satisfied, skipping upgrade: Cython>=0.25.2 in ~/.local/lib/python3.6/site-packages (from zipline) (0.29)
Collecting cyordereddict>=0.2.2 (from zipline)
  Using cached https://files.pythonhosted.org/packages/d1/1a/364cbfd927be1b743c7f0a985a7f1f7e8a51469619f9fefe4ee9240ba210/cyordereddict-1.0.0.tar.gz
Collecting bottleneck>=1.0.0 (from zipline)
  Using cached https://files.pythonhosted.org/packages/05/ae/cedf5323f398ab4e4ff92d6c431a3e1c6a186f9b41ab3e8258dff786a290/Bottleneck-1.2.1.tar.gz
Requirement already satisfied, skipping upgrade: contextlib2>=0.4.0 in ~/.local/lib/python3.6/site-packages/contextlib2-0.5.5-py3.6.egg (from zipline) (0.5.5)
Requirement already satisfied, skipping upgrade: decorator>=4.0.0 in ~/.local/lib/python3.6/site-packages (from zipline) (4.3.0)
Collecting networkx<2.0,>=1.9.1 (from zipline)
  Using cached https://files.pythonhosted.org/packages/d3/2c/e473e54afc9fae58dfa97066ef6709a7e35a1dd1c28c5a3842989322be00/networkx-1.11-py2.py3-none-any.whl
Collecting numexpr>=2.6.1 (from zipline)
  Using cached https://files.pythonhosted.org/packages/db/ea/efd9e16283637eb5b6c0042b6cc3521f1b9a5b47767ac463c88bbd37670c/numexpr-2.6.8-cp36-cp36m-manylinux1_x86_64.whl
Collecting bcolz<1,>=0.12.1 (from zipline)
  Using cached https://files.pythonhosted.org/packages/6c/8b/1ffa01f872cac36173c5eb95b58c01040d8d25f1b242c48577f4104cd3ab/bcolz-0.12.1.tar.gz
Requirement already satisfied, skipping upgrade: click>=4.0.0 in ~/.local/lib/python3.6/site-packages (from zipline) (7.0)
Requirement already satisfied, skipping upgrade: toolz>=0.8.2 in ~/.local/lib/python3.6/site-packages (from zipline) (0.9.0)
Requirement already satisfied, skipping upgrade: multipledispatch>=0.4.8 in ~/.local/lib/python3.6/site-packages (from zipline) (0.6.0)
Requirement already satisfied, skipping upgrade: MarkupSafe>=0.23 in ~/.local/lib/python3.6/site-packages (from zipline) (1.1.0)
Requirement already satisfied, skipping upgrade: Mako>=1.0.1 in /usr/lib/python3/dist-packages (from zipline) (1.0.7)
Collecting sqlalchemy>=1.0.8 (from zipline)
  Using cached https://files.pythonhosted.org/packages/1e/98/4dba86354d271344e25fa01dc38a0bf0e0ba6407ad2d5e8426496a95b568/SQLAlchemy-1.2.13.tar.gz
Collecting alembic>=0.7.7 (from zipline)
  Using cached https://files.pythonhosted.org/packages/c0/f3/e60af9a36ae3b8cafabc7e0834d8df6a2965b3feecf27b9b11352dc05dd4/alembic-1.0.2.tar.gz
Collecting sortedcontainers>=1.4.4 (from zipline)
  Using cached https://files.pythonhosted.org/packages/be/e3/a065de5fdd5849450a8a16a52a96c8db5f498f245e7eda06cc6725d04b80/sortedcontainers-2.0.5-py2.py3-none-any.whl
Collecting intervaltree>=2.1.0 (from zipline)
  Using cached https://files.pythonhosted.org/packages/ca/c1/450d109b70fa58ca9d77972b02f69222412f9175ccf99fdeaf167be9583c/intervaltree-2.1.0.tar.gz
Collecting lru-dict>=1.1.4 (from zipline)
  Using cached https://files.pythonhosted.org/packages/00/a5/32ed6e10246cd341ca8cc205acea5d208e4053f48a4dced2b1b31d45ba3f/lru-dict-1.1.6.tar.gz
Collecting empyrical>=0.5.0 (from zipline)
  Using cached https://files.pythonhosted.org/packages/7b/55/a01b05162b764830dbbac868462f44cd847a5b6523a01ca9f955721819da/empyrical-0.5.0.tar.gz
Collecting tables>=3.3.0 (from zipline)
  Using cached https://files.pythonhosted.org/packages/d7/1b/21f4c7f296b718575c17ef25e61c05742a283c45077b4c8d5a190b3e0b59/tables-3.4.4-cp36-cp36m-manylinux1_x86_64.whl
Collecting trading-calendars>=1.0.1 (from zipline)
  Using cached https://files.pythonhosted.org/packages/6e/e9/e9e2aeadacab58580ed9493e6c252f051cd8ee9d7efbe49d2f041371c45d/trading_calendars-1.5.1.tar.gz
Requirement already satisfied, skipping upgrade: wrapt in ~/.local/lib/python3.6/site-packages (from pandas-datareader>=0.2.1->zipline) (1.10.11)
Requirement already satisfied, skipping upgrade: lxml in ~/.local/lib/python3.6/site-packages (from pandas-datareader>=0.2.1->zipline) (4.2.5)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in ~/.local/lib/python3.6/site-packages (from requests>=2.9.1->zipline) (2018.10.15)
Requirement already satisfied, skipping upgrade: idna<2.8,>=2.5 in ~/.local/lib/python3.6/site-packages (from requests>=2.9.1->zipline) (2.7)
Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in /usr/lib/python3/dist-packages (from requests>=2.9.1->zipline) (3.0.4)
Requirement already satisfied, skipping upgrade: urllib3<1.25,>=1.21.1 in ~/.local/lib/python3.6/site-packages (from requests>=2.9.1->zipline) (1.24.1)
Collecting python-editor>=0.3 (from alembic>=0.7.7->zipline)
  Using cached https://files.pythonhosted.org/packages/65/1e/adf6e000ea5dc909aa420352d6ba37f16434c8a3c2fa030445411a1ed545/python-editor-1.0.3.tar.gz
Building wheels for collected packages: zipline, Logbook, cyordereddict, bottleneck, bcolz, sqlalchemy, alembic, intervaltree, lru-dict, empyrical, trading-calendars, python-editor
  Running setup.py bdist_wheel for zipline ... done
  Stored in directory: ~/.cache/pip/wheels/a4/d6/67/f303ab028b004bf8e00c05b5b04fba83d8ec238b6547becdb7
  Running setup.py bdist_wheel for Logbook ... done
  Stored in directory: ~/.cache/pip/wheels/06/13/e9/88e9e8184d89671ffc754dc80f5eb01dabd72071bdb802c5d1
  Running setup.py bdist_wheel for cyordereddict ... done
  Stored in directory: ~/.cache/pip/wheels/0b/9d/8b/5bf3e22c1edd59b50f11bb19dec9dfcfe5a479fc7ace02b61f
  Running setup.py bdist_wheel for bottleneck ... done
  Stored in directory: ~/.cache/pip/wheels/f2/bf/ec/e0f39aa27001525ad455139ee57ec7d0776fe074dfd78c97e4
  Running setup.py bdist_wheel for bcolz ... done
  Stored in directory: ~/.cache/pip/wheels/c5/cc/1b/2cf1f88959af5d7f4d449b7fc6c9452d0ecbd86fd61a9ee376
  Running setup.py bdist_wheel for sqlalchemy ... done
  Stored in directory: ~/.cache/pip/wheels/ba/2e/47/b194b03e6aeaa658f521dfba094a42f1d37f4ee4018b9e607e
  Running setup.py bdist_wheel for alembic ... done
  Stored in directory: ~/.cache/pip/wheels/95/f3/c2/22155963496514f044049f5c96e2757b9838da4a5cae34e76e
  Running setup.py bdist_wheel for intervaltree ... done
  Stored in directory: ~/.cache/pip/wheels/6b/cf/b0/f7ef2d0f504d26f3e9e70c2369e5725591ccfaf67d528fcbc5
  Running setup.py bdist_wheel for lru-dict ... done
  Stored in directory: ~/.cache/pip/wheels/b7/ef/06/fbdd555907a7d438fb33e4c8675f771ff1cf41917284c51ebf
  Running setup.py bdist_wheel for empyrical ... done
  Stored in directory: ~/.cache/pip/wheels/83/14/73/34fb27552601518d28bd0813d75124be76d94ab29152c69112
  Running setup.py bdist_wheel for trading-calendars ... done
  Stored in directory: ~/.cache/pip/wheels/13/0f/d5/517a7a51d9998c20483d777bf2e35e71ec17445509a9ac4cc1
  Running setup.py bdist_wheel for python-editor ... done
  Stored in directory: ~/.cache/pip/wheels/36/e0/98/ba386b125a00ea9dd52e2c16aa2ec0adbbd639b84bfe2e001d
Successfully built zipline Logbook cyordereddict bottleneck bcolz sqlalchemy alembic intervaltree lru-dict empyrical trading-calendars python-editor
pyro-ppl 0.2.1+fbaa779 has requirement networkx>=2.0.0, but you'll have networkx 1.11 which is incompatible.
pyro-ppl 0.2.1+fbaa779 has requirement torch==0.4.0, but you'll have torch 1.0.0a0+6c8d47f which is incompatible.
Installing collected packages: Logbook, pandas, pandas-datareader, statsmodels, cyordereddict, bottleneck, networkx, numexpr, bcolz, sqlalchemy, python-editor, alembic, sortedcontainers, intervaltree, lru-dict, empyrical, tables, trading-calendars, zipline
  Found existing installation: pandas 0.23.4
    Uninstalling pandas-0.23.4:
      Successfully uninstalled pandas-0.23.4
  Found existing installation: networkx 2.2
    Uninstalling networkx-2.2:
      Successfully uninstalled networkx-2.2
Successfully installed Logbook-1.4.1 alembic-1.0.2 bcolz-0.12.1 bottleneck-1.2.1 cyordereddict-1.0.0 empyrical-0.5.0 intervaltree-2.1.0 lru-dict-1.1.6 networkx-1.11 numexpr-2.6.8 pandas-0.22.0 pandas-datareader-0.7.0 python-editor-1.0.3 sortedcontainers-2.0.5 sqlalchemy-1.2.13 statsmodels-0.9.0 tables-3.4.4 trading-calendars-1.5.1 zipline-1.3.0
➜  ~

You’ll possibly meet these ERROR messages:

  • pyro-ppl 0.2.1+fbaa779 has requirement networkx>=2.0.0, but you’ll have networkx 1.11 which is incompatible.
  • pyro-ppl 0.2.1+fbaa779 has requirement torch==0.4.0, but you’ll have torch 1.0.0a0+6c8d47f which is incompatible.

In such a case, pyro is to be manually installed.

1.4 Manually Install pyro

Please refer to https://github.com/uber/pyro:

1
2
3
4
➜  uber git clone https://github.com/uber/pyro.git
➜  cd pyro
➜  pyro git:(dev) python setup.py build
➜  pyro git:(dev) python setup.py install --user

1.5 Install Zipline Again

Please refer to http://www.zipline.io/install.html:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
➜  ~ pip install -U zipline --user
Collecting zipline
Requirement already satisfied, skipping upgrade: numexpr>=2.6.1 in ./.local/lib/python3.6/site-packages (from zipline) (2.6.8)
Requirement already satisfied, skipping upgrade: cyordereddict>=0.2.2 in ./.local/lib/python3.6/site-packages (from zipline) (1.0.0)
Requirement already satisfied, skipping upgrade: tables>=3.3.0 in ./.local/lib/python3.6/site-packages (from zipline) (3.4.4)
Requirement already satisfied, skipping upgrade: pytz>=2016.4 in ./.local/lib/python3.6/site-packages (from zipline) (2018.7)
Requirement already satisfied, skipping upgrade: scipy>=0.17.1 in ./.local/lib/python3.6/site-packages (from zipline) (1.1.0)
Collecting bcolz<1,>=0.12.1 (from zipline)
Requirement already satisfied, skipping upgrade: patsy>=0.4.0 in ./.local/lib/python3.6/site-packages (from zipline) (0.5.1)
Requirement already satisfied, skipping upgrade: empyrical>=0.5.0 in ./.local/lib/python3.6/site-packages (from zipline) (0.5.0)
Requirement already satisfied, skipping upgrade: setuptools>18.0 in ./.local/lib/python3.6/site-packages (from zipline) (40.5.0)
Requirement already satisfied, skipping upgrade: requests-file>=1.4.1 in ./.local/lib/python3.6/site-packages (from zipline) (1.4.3)
Requirement already satisfied, skipping upgrade: requests>=2.9.1 in ./.local/lib/python3.6/site-packages (from zipline) (2.20.0)
Requirement already satisfied, skipping upgrade: intervaltree>=2.1.0 in ./.local/lib/python3.6/site-packages (from zipline) (2.1.0)
Requirement already satisfied, skipping upgrade: MarkupSafe>=0.23 in ./.local/lib/python3.6/site-packages (from zipline) (1.1.0)
Requirement already satisfied, skipping upgrade: contextlib2>=0.4.0 in ./.local/lib/python3.6/site-packages/contextlib2-0.5.5-py3.6.egg (from zipline) (0.5.5)
Requirement already satisfied, skipping upgrade: sqlalchemy>=1.0.8 in ./.local/lib/python3.6/site-packages (from zipline) (1.2.13)
Requirement already satisfied, skipping upgrade: toolz>=0.8.2 in ./.local/lib/python3.6/site-packages (from zipline) (0.9.0)
Requirement already satisfied, skipping upgrade: six>=1.10.0 in ./.local/lib/python3.6/site-packages (from zipline) (1.11.0)
Requirement already satisfied, skipping upgrade: pandas-datareader>=0.2.1 in ./.local/lib/python3.6/site-packages (from zipline) (0.7.0)
Requirement already satisfied, skipping upgrade: numpy>=1.11.1 in ./.local/lib/python3.6/site-packages (from zipline) (1.15.4)
Collecting pandas<=0.22,>=0.18.1 (from zipline)
  Using cached https://files.pythonhosted.org/packages/da/c6/0936bc5814b429fddb5d6252566fe73a3e40372e6ceaf87de3dec1326f28/pandas-0.22.0-cp36-cp36m-manylinux1_x86_64.whl
Requirement already satisfied, skipping upgrade: click>=4.0.0 in ./.local/lib/python3.6/site-packages (from zipline) (7.0)
Requirement already satisfied, skipping upgrade: Mako>=1.0.1 in /usr/lib/python3/dist-packages (from zipline) (1.0.7)
Requirement already satisfied, skipping upgrade: statsmodels>=0.6.1 in ./.local/lib/python3.6/site-packages (from zipline) (0.9.0)
Requirement already satisfied, skipping upgrade: bottleneck>=1.0.0 in ./.local/lib/python3.6/site-packages (from zipline) (1.2.1)
Requirement already satisfied, skipping upgrade: Logbook>=0.12.5 in ./.local/lib/python3.6/site-packages (from zipline) (1.4.1)
Collecting networkx<2.0,>=1.9.1 (from zipline)
  Using cached https://files.pythonhosted.org/packages/d3/2c/e473e54afc9fae58dfa97066ef6709a7e35a1dd1c28c5a3842989322be00/networkx-1.11-py2.py3-none-any.whl
Requirement already satisfied, skipping upgrade: alembic>=0.7.7 in ./.local/lib/python3.6/site-packages (from zipline) (1.0.2)
Requirement already satisfied, skipping upgrade: sortedcontainers>=1.4.4 in ./.local/lib/python3.6/site-packages (from zipline) (2.0.5)
Requirement already satisfied, skipping upgrade: lru-dict>=1.1.4 in ./.local/lib/python3.6/site-packages (from zipline) (1.1.6)
Requirement already satisfied, skipping upgrade: multipledispatch>=0.4.8 in ./.local/lib/python3.6/site-packages (from zipline) (0.6.0)
Requirement already satisfied, skipping upgrade: pip>=7.1.0 in ./.local/lib/python3.6/site-packages (from zipline) (18.1)
Requirement already satisfied, skipping upgrade: decorator>=4.0.0 in ./.local/lib/python3.6/site-packages (from zipline) (4.3.0)
Requirement already satisfied, skipping upgrade: Cython>=0.25.2 in ./.local/lib/python3.6/site-packages (from zipline) (0.29)
Requirement already satisfied, skipping upgrade: python-dateutil>=2.4.2 in ./.local/lib/python3.6/site-packages (from zipline) (2.7.5)
Requirement already satisfied, skipping upgrade: trading-calendars>=1.0.1 in ./.local/lib/python3.6/site-packages (from zipline) (1.5.1)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in ./.local/lib/python3.6/site-packages (from requests>=2.9.1->zipline) (2018.10.15)
Requirement already satisfied, skipping upgrade: urllib3<1.25,>=1.21.1 in ./.local/lib/python3.6/site-packages (from requests>=2.9.1->zipline) (1.24.1)
Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in /usr/lib/python3/dist-packages (from requests>=2.9.1->zipline) (3.0.4)
Requirement already satisfied, skipping upgrade: idna<2.8,>=2.5 in ./.local/lib/python3.6/site-packages (from requests>=2.9.1->zipline) (2.7)
Requirement already satisfied, skipping upgrade: wrapt in ./.local/lib/python3.6/site-packages (from pandas-datareader>=0.2.1->zipline) (1.10.11)
Requirement already satisfied, skipping upgrade: lxml in ./.local/lib/python3.6/site-packages (from pandas-datareader>=0.2.1->zipline) (4.2.5)
Requirement already satisfied, skipping upgrade: python-editor>=0.3 in ./.local/lib/python3.6/site-packages (from alembic>=0.7.7->zipline) (1.0.3)
pyro-ppl 0.2.1+913618a has requirement networkx>=2.0.0, but you'll have networkx 1.11 which is incompatible.
Installing collected packages: bcolz, pandas, networkx, zipline
  Found existing installation: bcolz 1.2.1
    Uninstalling bcolz-1.2.1:
      Successfully uninstalled bcolz-1.2.1
  Found existing installation: pandas 0.23.4
    Uninstalling pandas-0.23.4:
      Successfully uninstalled pandas-0.23.4
  Found existing installation: networkx 2.2
    Uninstalling networkx-2.2:
      Successfully uninstalled networkx-2.2
  Found existing installation: zipline 1.0.1+1816.gb7f454b7.dirty
    Uninstalling zipline-1.0.1+1816.gb7f454b7.dirty:
      Successfully uninstalled zipline-1.0.1+1816.gb7f454b7.dirty
Successfully installed bcolz-0.12.1 networkx-1.11 pandas-0.22.0 zipline-1.3.0
➜  ~ pip show networkx
Name: networkx
Version: 1.11
Summary: Python package for creating and manipulating graphs and networks
Home-page: http://networkx.github.io/
Author: NetworkX Developers
Author-email: networkx-discuss@googlegroups.com
License: BSD
Location: /home/jiapei/.local/lib/python3.6/site-packages
Requires: decorator
Required-by: zipline, scikit-image, pyro-ppl
➜  ~ pip show pyro-ppl
Name: pyro-ppl
Version: 0.2.1+913618a
Summary: A Python library for probabilistic modeling and inference
Home-page: http://pyro.ai
Author: Uber AI Labs
Author-email: pyro@uber.com
License: MIT License
Location: /home/jiapei/.local/lib/python3.6/site-packages/pyro_ppl-0.2.1+913618a-py3.6.egg
Requires: contextlib2, graphviz, networkx, numpy, opt-einsum, six, torch, tqdm
Required-by:
➜  ~

1.6 Python Package Compatibility

Let’s take a look at the compatibility of python packages.

1
2
3
4
5
6
➜  ~ pip list --outdated
Package    Version   Latest Type 
---------- --------- ------ -----
bcolz      0.12.1    1.2.1  sdist
networkx   1.11      2.2    sdist
pandas     0.22.0    0.23.4 wheel

Clearly, Zipline is NOT compatible with the above 3 LATEST python packages.

2. Register Quandl and Ingest Data

As known, Quantopian is a Python collections of various quantitative analysis algorithms. However, where can we obtain the stock data for our testing? Either the real-time data, or the historical data will do.

We can of course use the real-time Yahoo Finance Data. But here, we are going to test on some historical data downloaded from Quandl, which is JUST the data adopted in Quantopian’s Beginner’s Tutorial.

After you’ve successfully registered Quandl and obtained an API key, the following command will help you to ingest Quandl WIKI Bundle data easily, please refer to http://www.zipline.io/bundles.html#ingesting-data:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
➜  quantopian QUANDL_API_KEY=<yourkey> zipline ingest -b quandl
[2018-11-07 05:18:37.895463] INFO: zipline.data.bundles.quandl: Downloading WIKI metadata.
Downloading WIKI Prices table from Quandl  [####################################]  100%          
[2018-11-07 05:18:54.369791] INFO: zipline.data.bundles.quandl: Parsing raw data.
[2018-11-07 05:19:24.118879] INFO: zipline.data.bundles.quandl: Generating asset metadata.
Merging daily equity files:  [-----------------------------#------]  1731~/.local/lib/python3.6/site-packages/zipline/data/us_equity_pricing.py:417: UserWarning: Ignoring 1 values because they are out of bounds for uint32:             open  high   low  close        volume  ex_dividend  split_ratio
2011-04-11  1.79  1.84  1.55    1.7  6.674913e+09          0.0          1.0
  winsorise_uint32(raw_data, invalid_data_behavior, 'volume', *OHLC)
Merging daily equity files:  [####################################]      
[2018-11-07 05:21:34.367628] INFO: zipline.data.bundles.quandl: Parsing split data.
[2018-11-07 05:21:34.572459] INFO: zipline.data.bundles.quandl: Parsing dividend data.
~/.local/lib/python3.6/site-packages/zipline/data/us_equity_pricing.py:931: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  if not issubdtype(actual, expected):
➜  quantopian

3. First Try

Details can be easily found at http://www.zipline.io/beginner-tutorial.html. Here, we ONLY test the example ./zipline/examples/buyapple.py, which can also be found on Github.

Following the Beginner’s Tutorial, by executing the following command,

1
➜  examples git:(master) ✗ zipline run -f ./buyapple.py --start 2016-1-1 --end 2018-1-1 -o buyapple_out.pickle

we should be able to obtain the resultant figure as follows:

Quantopian Example - Buy Apple

Posted on Edited on In ,

October 31, 2018 is Halloween… ^_^ Happy Halloween everyone…

Happy Halloween

And, Happy Birthday To Myself. Such an old NOBODY… DON’T ask my age. Anyway, enjoy life…

Happy Birthday

Today, we are testing out some pose estimation open source, based on various open source AI frameworks, including:

1. OpenPose

Body Tracking Body Foot
Tracking Body ONLY Body and Foot
Body Hand Body Face
Body and Hand Body and Face

The related paper can be found at Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields.

2. tf-pose-estimation

Please refer to Posture Recognition by A Single 2D Webcam

3. pose-tensorflow

Single Person Multiple Persons
Single Person Multiple Persons

4. Facebook Detectron

5. Facebook DetectAndTrack

6. Facebook DensePose

6.1 COCO Visualize

Original Image Collected Points Subplots
Original Image Collected Points Subplots

6.2 RCNN Texture Transfer

Texture Mesh Synth UV
Texture Mesh Synth UV
IUV Texture Transfer Original Image
IUV Mesh Original Image

6.3 RCNN Visualize

IUV Mesh Contour
IUV Mesh Contour

7. Facebook maskrcnn-benchmark

Test on maskrcnn-benchmark

Posted on Edited on In ,

Hi, everyone… I’ve been so busy and haven’t written anything for a bit. My previous boss at Lenovo is now in Vancouver for annual executive meeting. She brought me a Huawei HiSilicon Hikey 970.

Several things before testing:

Now, let’s start today’s journey.

PART A: Introduction

1. Layout of Hikey 970

Hikey 970 looks like (cited from Hikey 970 ):

HiSilicon Hikey 970

2. Connection of Hikey 970

HiSilicon Hikey 970

Please pay attention to that pink rectangle, where 4 TINY switches are located and will be adjusted while flashing.

PART B: Install Lebuntu onto Hikey 970

1. Download Lebuntu for Hikey 970

We FIRST go visiting the website http://www.lemaker.org/product-hikey970-download-84.html and click Download icon, a file named Hikey 970 Lebuntu.rar will be automatically downloaded. After the extraction, a folder named hikey970-lebuntu-16.04 will be generated.

2. Preparations for Making Full Use of Flash Memony On Hikey 970

By integrating the following 2 contents:

we should be able to successfully install Lebuntu, and make FULL use of the whole flash memory on board.

2.1 Mount Boot Image

Make sure the current path is hikey970-lebuntu-16.04, then:

1
2
sudo mkdir /mnt/boot-hikey970
sudo mount -o loop boot-hikey970.uefi.img /mnt/boot-hikey970

2.2 Edit grub.cfg

1
sudo vim /mnt/boot-hikey970/boot/grub/grub.cfg
  • sdd12 -> sdd15
  • gpt12 -> gpt15

2.3 Edit flash scripts

There are 3 flash scripts:

  • flash-all-binaries.sh
  • flash-minimum-binaries.sh
  • binaries/recovery-flash.sh

Change all cases in the above 3 files:

  • sudo fastboot flash system leubuntu-16.04.img -> sudo fastboot flash userdata leubuntu-16.04.img.

3 Flash Lebuntu onto Hikey 970

3.1 Adjust the Switches

  • Switch 1 : ON
  • Switch 2 : ON
  • Switch 3 : ON
  • Switch 4 : OFF

3.2 Connect Type C USB cable and Power Adapter

  • Connect Type C USB cable between the host computer and Hikey 970.
  • Connect power adapter.
  • Make sure Hikey 970 is configured as ttyUSB0.
1
$ ls -ls /dev/ttyUSB0

3.3 System Installation

1
./binaries/recovery-flash.sh

3.4 Unplug Power Supply then Re-adjust the Switches

Unplug the power adapter, and then re-adjust the switches:

  • Switch 1 : ON
  • Switch 2 : OFF
  • Switch 3 : OFF
  • Switch 4 : OFF

3.5 Connect HDMI Cable then Power Adapter

  • Connect HDMI cable.
  • Connect the Power Adapter.

NOTE: Plug the HDMI cable before powering up. Lebuntu may NOT boot up without plugging in the HDMI.

3.6 Boot up

You may have to re-plug in the power adapter multiple times, in order to boot up Lebuntu. Username and password are respectively:

  • Username: shunya
  • Password: shunya

Expand the Partition

In order to make full use of the partition, we need to resize the file system as:

1
shunya@hikey970:~$ sudo resize2fs /dev/sdd15

PART C: Waiting for HuaWei HiAI

Longer Vision Technology is waiting for HuaWei Developer account activation:

HuaWei Developer Account Activation

I may talk about Lebian with Pre-built Tensorflow some time later.

So tired these days… Nice dream…

Posted on Edited on In

I’ve been looking for an IDE for debugging various microcontroller boards, such as OpenMV and PyBoard, etc. There are some fundamental requirements:

  • Open source
  • Python debugger, MicroPython support
  • Any GUI is fine, but Qt preferred

Therefore, after a careful search, I found uPyCraft, which seems to be designed by a Chinese team? Because I found the Chinese version of uPyCraft documentation as well.

However, it seems Linux support for uPyCraft is NOT very good. My working environment:

1
2
3
4
5
6
7
8
9
10
11
12
13
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:        18.04
Codename:       bionic
$ uname -a
Linux jiapei-GT72-6QE 4.18.7-041807-generic #201809090930 SMP Sun Sep 9 09:33:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ python --version
Python 3.6.5
$ qmake --version
QMake version 3.1
Using Qt version 5.11.1 in /opt/Qt/Current/gcc_64/lib

PART A: Test Built uPyCraft

1. Try the Executable

In fact, there is one executable Linux version for uPyCraft. However, to run it after downloading, I obtained the following ERROR messages:

1
2
3
4
5
6
$ ./uPyCraft_linux_V1.0 
Traceback (most recent call last):
  File "uPyCraft.py", line 2, in <module>
  File "/usr/local/lib/python3.5/dist-packages/PyInstaller/loader/pyimod03_importers.py", line 714, in load_module
ImportError: /tmp/_MEIoq8Xi8/libz.so.1: version `ZLIB_1.2.9' not found (required by /usr/lib/x86_64-linux-gnu/libpng16.so.16)
[324] Failed to execute script uPyCraft

2. Try source code

We can also clone uPyCraft source code and build it from scratch. However, clearly, current uPyCraft relies on PyQt4 instead of PyQt5.

Therefore, I’ve got to modify uPyCraft a bit to have this uPyCraft_PyQt5.

PART B: Build uPyCraft with PyQt5

Before testing uPyCraft_PyQt5, you need to make sure

**What’s more important: Please uninstall enum34 but keep the original enum. Please refer to Stack Overflow for more details.

1. Check out and build my modified code of uPyCraft_PyQt5

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
$ git clone git@github.com:jiapei100/uPyCraft_PyQt5.git
$ cd uPyCraft_PyQt5
$ pyinstaller -F uPyCraft.py
47 INFO: PyInstaller: 3.4
47 INFO: Python: 3.6.5
47 INFO: Platform: Linux-4.18.7-041807-generic-x86_64-with-Ubuntu-18.04-bionic
48 INFO: wrote ....../uPyCraft_PyQt5/uPyCraft.spec
50 INFO: UPX is not available.
51 INFO: Extending PYTHONPATH with paths
['....../uPyCraft_PyQt5',
 '....../uPyCraft_PyQt5']
52 INFO: checking Analysis
54 INFO: Building because inputs changed
54 INFO: Initializing module dependency graph...
57 INFO: Initializing module graph hooks...
59 INFO: Analyzing base_library.zip ...
2747 INFO: running Analysis Analysis-00.toc
2776 INFO: Caching module hooks...
2779 INFO: Analyzing ....../uPyCraft_PyQt5/uPyCraft.py
3619 INFO: Loading module hooks...
3619 INFO: Loading module hook "hook-PyQt5.QtGui.py"...
3749 INFO: Loading module hook "hook-pydoc.py"...
3750 INFO: Loading module hook "hook-PyQt5.QtWidgets.py"...
3832 INFO: Loading module hook "hook-xml.py"...
3914 INFO: Loading module hook "hook-PyQt5.QtCore.py"...
3933 INFO: Loading module hook "hook-PyQt5.QtPrintSupport.py"...
4057 INFO: Loading module hook "hook-encodings.py"...
4126 INFO: Loading module hook "hook-PyQt5.py"...
4170 WARNING: Hidden import "sip" not found!
4179 INFO: Looking for ctypes DLLs
4274 WARNING: library setupapi required via ctypes not found
4299 WARNING: library Advapi32 required via ctypes not found
4367 INFO: Analyzing run-time hooks ...
4370 INFO: Including run-time hook 'pyi_rth_qt5.py'
4376 INFO: Looking for dynamic libraries
8225 INFO: Looking for eggs
8225 INFO: Python library not in binary dependencies. Doing additional searching...
8292 INFO: Using Python library /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
8301 INFO: Warnings written to ....../uPyCraft_PyQt5/build/uPyCraft/warn-uPyCraft.txt
8339 INFO: Graph cross-reference written to ....../uPyCraft_PyQt5/build/uPyCraft/xref-uPyCraft.html
8352 INFO: checking PYZ
8354 INFO: Building because name changed
8354 INFO: Building PYZ (ZlibArchive) ....../uPyCraft_PyQt5/build/uPyCraft/PYZ-00.pyz
8762 INFO: Building PYZ (ZlibArchive) ....../uPyCraft_PyQt5/build/uPyCraft/PYZ-00.pyz completed successfully.
8765 INFO: checking PKG
8767 INFO: Building because name changed
8767 INFO: Building PKG (CArchive) PKG-00.pkg
33163 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
33168 INFO: Bootloader ~/.local/lib/python3.6/site-packages/PyInstaller/bootloader/Linux-64bit/run
33168 INFO: checking EXE
33170 INFO: Building because name changed
33170 INFO: Building EXE from EXE-00.toc
33175 INFO: Appending archive to ELF section in EXE ....../uPyCraft_PyQt5/dist/uPyCraft
33753 INFO: Building EXE from EXE-00.toc completed successfully.

2. Run and test uPyCraft_PyQt5

To run uPyCraft_PyQt5

1
2
$ cd dist
$ ./uPyCraft

uPyCraft

So far, 6 boards are particularly supported:

For the sake of our requirements, openmv is going to be supported in particular.