Longer Vision Technology

Github Blog

Posted on Edited on In

This blog is just to solve some bugs while using MySQL as a server. Assuming you've already installed mysql-server, mysql-server-5.7, and mysql-server-core-5.7.

1. Errors

1.1 Failed to start mysql.service: Unit mysql.service not found.

1
2
3
4
5
6
7
➜  ~ sudo service mysqld start
[sudo] password for jiapei:
Failed to start mysqld.service: Unit mysqld.service not found.
➜  ~ /etc/init.d/mysql stop
[ ok ] Stopping mysql (via systemctl): mysql.service.
➜  ~ /etc/init.d/mysql start
[ ok ] Starting mysql (via systemctl): mysql.service.

Conclusion: use /etc/init.d/mysql start instead of service mysqld start.

1.2 ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

1
2
3
➜  ~ mysql -u root -p
Enter password: 
ERROR 1698 (28000): Access denied for user 'root'@'localhost'

We FIRST list all existing MySQL processes and kill them all.

1
2
3
➜  ~ ps aux | grep mysql
...
➜  ~ sudo kill PID

Util now, there should be NO MySQL process running.

1.3 ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock'

Since there is NO MySQL process running, of course we cannot connect to MySQL server.

1
2
3
4
5
6
7
➜  ~ mysql -u root -p
Enter password: 
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
➜  ~ /etc/init.d/mysql start
[....] Starting mysql (via systemctl): mysql.serviceJob for mysql.service failed because the control process exited with error code.
See "systemctl status mysql.service" and "journalctl -xe" for details.
 failed!

Then, we will have to have these 3 packages reinstalled: mysql-server, mysql-server-5.7, and mysql-server-core-5.7.

1
2
3
4
➜  ~ sudo apt remove mysql-server mysql-server-5.7 mysql-server-core-5.7
...
➜  ~ sudo apt install mysql-server mysql-server-5.7 mysql-server-core-5.7
...

/var/run/mysqld/mysqld.sock is NOW back, and MySQL seems to run automatically.

1
2
3
4
5
➜  ~ ls -ls /var/run/mysqld/mysqld.sock
0 srwxrwxrwx 1 mysql mysql 0 Nov 30 03:08 /var/run/mysqld/mysqld.sock
➜  ~ ps aux | grep mysql
mysql    13785  0.1  0.3 1323300 172680 ?      Sl   03:08   0:00 /usr/sbin/mysqld --daemonize --pid-file=/run/mysqld/mysqld.pid
jiapei   14453  0.0  0.0  21556  2560 pts/0    R+   03:16   0:00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn mysql

Then, use chown and chmod to set up the owners and permissions suitably as follows.

1
2
3
4
5
6
7
8
9
10
11
12
13
➜  run pwd
/var/run
➜  run sudo chown mysql:mysql -R mysql
➜  run sudo chmod 755 -R mysql
➜  run ls -lsd mysql
0 drwxr-xr-x 2 mysql mysql 100 Nov 30 02:38 mysqld
➜  run cd ../lib
➜  lib pwd
/var/lib
➜  lib sudo chown mysql:mysql -R mysql
➜  lib sudo chmod 755 -R mysql
➜  lib ls -lsd mysql
4 drwxr-xr-x 6 mysql mysql 4096 Nov 30 02:38 mysql

1.4 ERROR 1698 (28000): Access denied for user 'root'@'localhost'

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
➜  ~ mysql -u root -p
Enter password: 
ERROR 1698 (28000): Access denied for user 'root'@'localhost'
➜  ~ sudo mysql -u root
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.24-0ubuntu0.18.04.1 (Ubuntu)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> USE lvrsql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> CREATE USER 'YOUR_SYSTEM_USER'@'localhost' IDENTIFIED BY 'YOUR_PASSWORD';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'YOUR_SYSTEM_USER'@'localhost';
mysql> FLUSH PRIVILEGES;
mysql> exit;

➜  ~ service mysql restart
➜  ~

1.5 [ERROR] InnoDB: Unable to lock ./ibdata1 (Additional)

If you meet the above ERROR message, what you can do is to restart MySQL by:

1
2
3
➜  ~ /etc/init.d/mysql restart
[ ok ] Restarting mysql (via systemctl): mysql.service.
➜  ~

2. Fundamental Commands in MySQL

Posted on Edited on In

Finally, I'm talking about Geographic Information System, namely, GIS. Remote Sensing and Information Engineer is my bachelor's major in Wuhan Technical University of Surveying and Mapping, which is NOW a part of Wuhan University. When I was in the university, I NEVER thought I was in such an amazing department, studying such an amazing major.

Alright... Let's start today's blog. We're going to play OpenStreetMap for some fun.

1. ArcGIS with OpenStreetMap

We FIRST test out how ArcGIS displays OpenStreetMap.

1.1 ArcGIS 3.26

A snippet of sample code is provided for ArcGIS Version 3.26, cited as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
<!DOCTYPE html>
<html> 
  <head> 
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <meta name="viewport" content="initial-scale=1, maximum-scale=1,user-scalable=no" />
    <title>OpenStreetMap</title> 
    <link rel="stylesheet" href="https://js.arcgis.com/3.26/esri/css/esri.css">
    <style>
      html, body, #esri-map-container {
        padding: 0px;
        margin: 0px;
        height: 100%;
      }
    </style>
    <script src="https://js.arcgis.com/3.26/"></script>
    <script>
      require([
        "esri/map",
        "esri/layers/OpenStreetMapLayer",
        "dojo/domReady!"
      ], function (Map, OpenStreetMapLayer){

        var map, openStreetMapLayer;

        map = new Map("esri-map-container", {
          center: [-89.924, 30.036],
          zoom: 12
        });
        openStreetMapLayer = new OpenStreetMapLayer();
        map.addLayer(openStreetMapLayer);
      });
    </script>
  </head> 
  
  <body>
    <div id="esri-map-container"></div>
  </body> 
</html>

Please click on the following image and have a try.

ArcGIS 3.26

1.2 ArcGIS LATEST

A snippet of sample code is provided for ArcGIS LATEST Version 4.9, cited as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="initial-scale=1,maximum-scale=1,user-scalable=no">
  <title>OpenStreetMapLayer - 4.9</title>

  <style>
    html,
    body,
    #viewDiv {
      padding: 0;
      margin: 0;
      height: 100%;
      width: 100%;
    }
  </style>

  <link rel="stylesheet" href="https://js.arcgis.com/4.9/esri/css/main.css">
  <script src="https://js.arcgis.com/4.9/"></script>

  <script>
    require([
        "esri/layers/OpenStreetMapLayer",
        "esri/Map",
        "esri/views/SceneView"
      ],
      function(
        OpenStreetMapLayer,
        Map,
        SceneView
      ) {

        var map = new Map({
          ground: "world-elevation"
        });

        var view = new SceneView({
          map: map,
          container: "viewDiv"
        });

        var osmLayer = new OpenStreetMapLayer();
        map.add(osmLayer);
      });
  </script>
</head>

<body>
  <div id="viewDiv"></div>
</body>

</html>

Please click on the following image and have a try.

ArcGIS 4.9

ArcGIS is like the CROWN of GIS. Its SDK/API seems NOT fully open source but commercial. Therefore, I'd love to breifly test out some other open source.

2. WebGLEarth

Several WebGLEarth examples are given at http://examples.webglearth.com/, and we just try out the very FIRST example Satellite. The code is cited as:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
<!DOCTYPE HTML>
<html>
  <head>
    <script src="http://www.webglearth.com/v2/api.js"></script>
    <script>
      function initialize() {
        var options = {atmosphere: true, center: [0, 0], zoom: 0};
        var earth = new WE.map('earth_div', options);
        WE.tileLayer('http://tileserver.maptiler.com/nasa/{z}/{x}/{y}.jpg', {
          minZoom: 0,
          maxZoom: 5,
          attribution: 'NASA'
        }).addTo(earth);
      }
    </script>
    <style>
      html, body{padding: 0; margin: 0;}
      #earth_div{top: 0; right: 0; bottom: 0; left: 0;
                 background-color: #000; position: absolute !important;}
      </style>
      <title>WebGL Earth API: Satellite</title>
    </head>
    <body onload="initialize()">
      <div id="earth_div"></div>
  </body>
</html>

Please click on the following image and have a try.

LeafletJS

3. OpenGlobus Based On OpenStreetMap

"OpenGlobus is a javascript library designed to display interactive 3d maps and planets with map tiles, imagery and vector data, markers and 3d objects. It uses the WebGL technology, open source and completely free." (Cited from OpenGlobus Github). Let's just take the very FIRST example Og - examples: Attribution as our example. Code is cited as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
<html>

<head>
    <title>Attribution Example</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link rel="stylesheet" href="http://www.openglobus.org/og.css" type="text/css" />
    <script src="http://www.openglobus.org/og.js"></script>
</head>

<body>
    <div id="globus" style="width:100%;height:100%"></div>
    <button id="btnOSM" style="position: absolute; left:0; margin:10px;">Set attribution</button>
    <script>

        document.getElementById("btnOSM").onclick = function () {
            states.setAttribution("Hello, WMS default USA population states example!");
        };

        let osm = new og.layer.XYZ("OpenStreetMap", {
            specular: [0.0003, 0.00012, 0.00001],
            shininess: 20,
            diffuse: [0.89, 0.9, 0.83],
            isBaseLayer: true,
            url: "http://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png",
            visibility: true,
            attribution: 'Data @ OpenStreetMap contributors, ODbL'
        });

        let states = new og.layer.WMS("USA Population", {
            extent: [[-128, 24], [-66, 49]],
            visibility: true,
            isBaseLayer: false,
            url: "http://openglobus.org/geoserver/",
            layers: "topp:states",
            opacity: 1.0,
            attribution: 'Hi!',
            transparentColor: [1.0, 1.0, 1.0]
        });

        let globus = new og.Globe({
            "target": "globus",
            "name": "Earth",
            "terrain": new og.terrain.GlobusTerrain(),
            "layers": [osm, states]
        });

        globus.planet.flyExtent(states.getExtent());
    </script>
</body>

</html>

Please click on the following image and have a try.

OpenGlobus

4. LeafletJS Based On OpenStreetMap

Several LeafletJS examples are given at https://leafletjs.com/examples.html. Here, we took Leaflet on Mobile as an example (code copied).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
<!DOCTYPE html>
<html>
<head>

	<title>Mobile tutorial - Leaflet</title>

	<meta charset="utf-8" />
	<meta name="viewport" content="width=device-width, initial-scale=1.0">

	<link rel="shortcut icon" type="image/x-icon" href="docs/images/favicon.ico" />

    <link rel="stylesheet" href="https://unpkg.com/leaflet@1.3.4/dist/leaflet.css" integrity="sha512-puBpdR0798OZvTTbP4A8Ix/l+A4dHDD0DGqYW6RQ+9jxkRFclaxxQb/SJAWZfWAkuyeQUytO7+7N4QKrDh+drA==" crossorigin=""/>
    <script src="https://unpkg.com/leaflet@1.3.4/dist/leaflet.js" integrity="sha512-nMMmRyTVoLYqjP9hrbed9S+FzjZHW5gY1TWCHA5ckwXZBadntCNs8kEqAWdrb9O7rxbCaA4lKTIWjDXZxflOcA==" crossorigin=""></script>


	<style>
		html, body {
			height: 100%;
			margin: 0;
		}
		#map {
			width: 600px;
			height: 400px;
		}
	</style>

	<style>body { padding: 0; margin: 0; } #map { height: 100%; width: 100vw; }</style>
</head>
<body>

<div id='map'></div>

<script>
	var map = L.map('map').fitWorld();

	L.tileLayer('https://api.tiles.mapbox.com/v4/{id}/{z}/{x}/{y}.png?access_token=pk.eyJ1IjoibWFwYm94IiwiYSI6ImNpejY4NXVycTA2emYycXBndHRqcmZ3N3gifQ.rJcFIG214AriISLbB6B5aw', {
		maxZoom: 18,
		attribution: 'Map data &copy; <a href="https://www.openstreetmap.org/">OpenStreetMap</a> contributors, ' +
			'<a href="https://creativecommons.org/licenses/by-sa/2.0/">CC-BY-SA</a>, ' +
			'Imagery © <a href="https://www.mapbox.com/">Mapbox</a>',
		id: 'mapbox.streets'
	}).addTo(map);

	function onLocationFound(e) {
		var radius = e.accuracy / 2;

		L.marker(e.latlng).addTo(map)
			.bindPopup("You are within " + radius + " meters from this point").openPopup();

		L.circle(e.latlng, radius).addTo(map);
	}

	function onLocationError(e) {
		alert(e.message);
	}

	map.on('locationfound', onLocationFound);
	map.on('locationerror', onLocationError);

	map.locate({setView: true, maxZoom: 16});
</script>

</body>
</html>

Please click on the following image and have a try.

LeafletJS

5. Mapbox GL JS Based on OpenStreetMap

From the above LeafletJS example, it's clearly show that LeafletJS is based on Mapbox, which is further based on OpenStreetMap. "Mapbox is a Live Location Platform." (Cited from https://www.mapbox.com/).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
<!DOCTYPE html>
<html>
<head>
    <meta charset='utf-8' />
    <title>Display a map</title>
    <meta name='viewport' content='initial-scale=1,maximum-scale=1,user-scalable=no' />
    <script src='https://api.tiles.mapbox.com/mapbox-gl-js/v0.51.0/mapbox-gl.js'></script>
    <link href='https://api.tiles.mapbox.com/mapbox-gl-js/v0.51.0/mapbox-gl.css' rel='stylesheet' />
    <style>
        body { margin:0; padding:0; }
        #map { position:absolute; top:0; bottom:0; width:100%; }
    </style>
</head>
<body>

<div id='map'></div>
<script>
mapboxgl.accessToken = 'pk.eyJ1IjoibG9uZ2VydmlzaW9uIiwiYSI6ImNqb2ZpN2Z5ZjA0bnMzd3FoY29oeTZwNW0ifQ.th9s23R3Njgxewe9rDdXzA';
var map = new mapboxgl.Map({
    container: 'map', // container id
    style: 'mapbox://styles/mapbox/streets-v9', // stylesheet location
    center: [-74.50, 40], // starting position [lng, lat]
    zoom: 9 // starting zoom
});
</script>

</body>
</html>

Please click on the following image and have a try.

Mapbox GL

6. OSMBuildings Based On OpenStreetMap

OSMBuildings is "a 3D building geometry viewer based on OpenStreetMap data." (Cited from https://github.com/OSMBuildings/OSMBuildings). Unfortunately, from OSMBuildings Registration, it is NOT free. However, its demonstrations are amazing - please refer to https://osmbuildings.org/developer/.

OSMBuildings Examples

7. Tiles Upon OpenStreetMap

OpenStreetMap may be covered by different tiles in order to show different maps. For instance, Google map vs Google satellite map. Let's take a look at Home Page of WebGLEarth, you'll notice MapTiler at the left bottom corner. Here, we summarize 3 widely used tiles in the following:

Generally, there are the following types of common tiles:

  • OpenStreetMap vector tiles
  • Contour lines vector tiles
  • Hillshading raster tiles
  • Landcover vector tiles
  • Satellite raster tiles
  • Terrain - quantized mesh tiles
  • etc.

8. OpenLayers

Besides tiles, there might be several layers (including one layer of tile) upon OpenStreetMap, here comes OpenLayers: "OpenLayers makes it easy to put a dynamic map in any web page. It can display map tiles, vector data and markers loaded from any source." (cited from https://openlayers.org/)

A simple example of is in OpenLayers QuickStart. The code is cited as:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
<!doctype html>
<html lang="en">
  <head>
    <link rel="stylesheet" href="https://cdn.rawgit.com/openlayers/openlayers.github.io/master/en/v5.3.0/css/ol.css" type="text/css">
    <style>
      .map {
        height: 400px;
        width: 100%;
      }
    </style>
    <script src="https://cdn.rawgit.com/openlayers/openlayers.github.io/master/en/v5.3.0/build/ol.js"></script>
    <title>OpenLayers example</title>
  </head>
  <body>
    <h2>My Map</h2>
    <div id="map" class="map"></div>
    <script type="text/javascript">
      var map = new ol.Map({
        target: 'map',
        layers: [
          new ol.layer.Tile({
            source: new ol.source.OSM()
          })
        ],
        view: new ol.View({
          center: ol.proj.fromLonLat([37.41, 8.82]),
          zoom: 4
        })
      });
    </script>
  </body>
</html>

Please click on the following image and have a try.

OpenLayers Quick Start

9. CesiumJS

There is a complete example from Cesium Tutorial, which is cited directly as:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <script src="https://cesium.com/downloads/cesiumjs/releases/1.68/Build/Cesium/Cesium.js"></script>
  <link href="https://cesium.com/downloads/cesiumjs/releases/1.68/Build/Cesium/Widgets/widgets.css" rel="stylesheet">
</head>
<body>
  <div id="cesiumContainer" style="width: 700px; height:400px"></div>
  <script>
    Cesium.Ion.defaultAccessToken = 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiI1ZTcwZTRmMS1jNmZkLTQ1NzctYTU3ZC03Mzc1Y2U2YWU5OTMiLCJpZCI6MjY1MzUsInNjb3BlcyI6WyJhc3IiLCJnYyJdLCJpYXQiOjE1ODc5MDM0MDB9.SAx2MQ2F3mUmH2InfoHWJvUoQBhq1CvGwo_2Z8oNC7Y';
    var viewer = new Cesium.Viewer('cesiumContainer', {
        terrainProvider: Cesium.createWorldTerrain()
    });

    var tileset = viewer.scene.primitives.add(
        new Cesium.Cesium3DTileset({
            url: Cesium.IonResource.fromAssetId(your_asset_id)
        })
    );
    viewer.zoomTo(tileset);
  </script>
</body>
</html>

You are welcome to click on the following image and have a try.

A Completed Example of CesiumJS

10. Esri

As the LAST demo of this blog, let's try something interesting: A Live Demo - Wind-JS.

You are of course welcome to click on the following image and have a try:

Esri Wind of the World

11. MapQuest

MapQuest provides Open Search (Nominatim) API, which relies solely on OpenStreetMap data.

Geocoding based on OpenStreetMap is exactly for ME.

😍

Several Geocoding services are provided as follows:

Please take a look at the bottom of my other website https://longervision.cc/.

Posted on Edited on In

Yesterday, we briefly introduced Quantopian and how to ingest Quandl WIKI Bundle data. Today, we are going to briefly summarize the financial data source and how to retrieve Google and Yahoo finance data using pandas_datareader.

1. Google Finance Data and Yahoo Finance Data

State-of-the-art finance data source are partly summarized at Financial Hacker. Our recommended solution is to retrieve stock data by using a SINGLE framework: pandas_datareader. Yahoo Finance Data and Google Finance Data can be easily retrieved by This Learn Data Science Tutorial.

Source Request Link Stock Exchange Symbol (INDEX) Stock Symbol (DATA)
Google Finance https://www.google.ca/finance?q=INDEX:DATA INDEX, ex:
INDEXDJX: Dow Jones
NASDAQ
INDEXNASDAQ
NYSE: NYSE COMPOSITE (DJ)
INDEXSP: S&P 500		 DATA, ex:
APPL
Yahoo Finance https://finance.yahoo.com/quote/DATA?p=DATA Totally, Yahoo provides the following indexes:
%5EGSPC: S&P 500
%5EDJI: Dow 30
%5EIXIC: NASDAQ Composite
%5ERUT: Russell 2000
CL=F: Crude Oil Dec 18
GC=F: Gold Dec 18
SI=F: Silver Dec 18
%5ETNX: CBOE Interest Rate 10 Year T No
%5EVIX: CBOE Volatility Index
BTC-USD: Bitcoin USD
%5EFTSE: FTSE 100
%5EN225: Nikkei 225		 DATA ex:
APPL

2. Use pandas_datareader to Retrieve Finance Data

A comparatively old blog Python for Finance, Part I: Yahoo & Google Finance API, pandas, and matplotlib has already given a fabulous summary. However, with the development of Python, code for finance data loading needs to be modified a bit. For 4 different data resource:

2.1 Google Finance

1
2
3
4
5
6
7
# Define the instruments to download. We would like to see Apple, Microsoft and the S&P500 index.
tickers = ['AAPL', 'MSFT', '^GSPC']

# We would like all available data from 01/01/2017 until 12/31/2017.
start_date = '2017-01-01'
end_date = '2017-12-31'
df = data.DataReader(tickers, data_source='google', start=start_date, end=end_date)

Exception has occurred: pandas_datareader.exceptions.ImmediateDeprecationError Google finance has been immediately deprecated due to large breaks in the API without the introduction of a stable replacement. Pull Requests to re-enable these data connectors are welcome. See https://github.com/pydata/pandas-datareader/issues File "....../test.py", line 16, in df = data.DataReader(tickers, data_source='google', start=start_date, end=end_date)

From pandas-datareader issues 487, we can tell Google Finance API is currently down. Therefore, let's ignore Google Finance for now.

On the other hand, from the following 3 finance data resource, the following printed tables and generated pictures demonstrate that stock data can be successfully loaded.

2.2 Yahoo Finance

1
2
3
4
5
6
7
# Define the instruments to download. We would like to see Apple, Microsoft and the S&P500 index.
tickers = ['AAPL', 'MSFT', '^GSPC']

# We would like all available data from 01/01/2017 until 12/31/2017.
start_date = '2017-01-01'
end_date = '2017-12-31'
df = data.DataReader(tickers, data_source='yahoo', start=start_date, end=end_date)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
Attributes        High                                 Low             \
Symbols           AAPL       MSFT        ^GSPC        AAPL       MSFT
Date
2017-01-03  116.330002  62.840000  2263.879883  114.760002  62.130001
2017-01-04  116.510002  62.750000  2272.820068  115.750000  62.119999
2017-01-05  116.860001  62.660000  2271.500000  115.809998  62.029999
2017-01-06  118.160004  63.150002  2282.100098  116.470001  62.040001
2017-01-09  119.430000  63.080002  2275.489990  117.940002  62.540001
2017-01-10  119.379997  63.070000  2279.270020  118.300003  62.279999
2017-01-11  119.930000  63.230000  2275.320068  118.599998  62.430000
2017-01-12  119.300003  63.400002  2271.780029  118.209999  61.950001
2017-01-13  119.620003  62.869999  2278.679932  118.809998  62.349998

Attributes                     Open                               Close  \
Symbols           ^GSPC        AAPL       MSFT        ^GSPC        AAPL
Date
2017-01-03  2245.129883  115.800003  62.790001  2251.570068  116.150002
2017-01-04  2261.600098  115.849998  62.480000  2261.600098  116.019997
2017-01-05  2260.449951  115.919998  62.189999  2268.179932  116.610001
2017-01-06  2264.060059  116.779999  62.299999  2271.139893  117.910004
2017-01-09  2268.899902  117.949997  62.759998  2273.590088  118.989998
2017-01-10  2265.270020  118.769997  62.730000  2269.719971  119.110001
2017-01-11  2260.830078  118.739998  62.610001  2268.600098  119.750000
2017-01-12  2254.250000  118.900002  63.060001  2271.139893  119.250000
2017-01-13  2271.510010  119.110001  62.619999  2272.739990  119.040001

Attributes                              Volume                            \
Symbols          MSFT        ^GSPC        AAPL        MSFT         ^GSPC
Date
2017-01-03  62.580002  2257.830078  28781900.0  20694100.0  3.770530e+09
2017-01-04  62.299999  2270.750000  21118100.0  21340000.0  3.764890e+09
2017-01-05  62.299999  2269.000000  22193600.0  24876000.0  3.761820e+09
2017-01-06  62.840000  2276.979980  31751900.0  19922900.0  3.339890e+09
2017-01-09  62.639999  2268.899902  33561900.0  20256600.0  3.217610e+09
2017-01-10  62.619999  2268.899902  24462100.0  18593000.0  3.638790e+09
2017-01-11  63.189999  2275.320068  27588600.0  21517300.0  3.620410e+09
2017-01-12  62.610001  2270.439941  27086200.0  20968200.0  3.462130e+09
2017-01-13  62.700001  2274.639893  26111900.0  19422300.0  3.081270e+09

Attributes   Adj Close
Symbols           AAPL       MSFT        ^GSPC
Date
2017-01-03  112.620964  60.431488  2257.830078
2017-01-04  112.494911  60.161095  2270.750000
2017-01-05  113.066986  60.161095  2269.000000
2017-01-06  114.327484  60.682560  2276.979980
2017-01-09  115.374664  60.489429  2268.899902
2017-01-10  115.491020  60.470112  2268.899902
2017-01-11  116.111588  61.020542  2275.320068
2017-01-12  115.626762  60.460461  2270.439941
2017-01-13  115.423149  60.547371  2274.639893
AAPL Closing Price MSFT Closing Price S&P 500 Closing Price
AAPL Closing Price MSFT Closing Price S&P 500 Closing Price

2.3 IEX

1
2
3
4
5
6
7
# Define the instruments to download. We would like to see Apple, Microsoft and the S&P500 index.
tickers = ['AAPL', 'MSFT', 'SPY']

# We would like all available data from 01/01/2017 until 12/31/2017.
start_date = '2017-01-01'
end_date = '2017-12-31'
df = data.DataReader(tickers, data_source="iex", start=start_date, end=end_date)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
Attributes      open                         high                     \
Symbols         AAPL     MSFT       SPY      AAPL     MSFT       SPY   
date                                                                   
2017-01-03  112.6732  60.6343  217.9106  113.1889  60.6825  218.6756   
2017-01-04  112.7219  60.3349  218.4722  113.3641  60.5956  219.5664   
2017-01-05  112.7900  60.0549  219.1016  113.7087  60.5087  219.4018   
2017-01-06  113.6268  60.1611  219.3534  114.9695  60.9819  220.5347   
2017-01-09  114.7652  60.6053  219.7213  116.2052  60.9143  219.8764   
2017-01-10  115.5630  60.5763  219.3050  116.1566  60.9046  220.2442   
2017-01-11  115.5338  60.4604  219.1888  116.6917  61.0591  219.9053   
2017-01-12  115.6847  60.8950  219.3243  116.0787  61.2233  219.5664   
2017-01-13  115.8939  60.4701  219.5470  116.3901  60.7067  220.1958   

Attributes       low                        close                     \
Symbols         AAPL     MSFT       SPY      AAPL     MSFT       SPY   
date                                                                   
2017-01-03  111.6613  59.9921  216.7909  113.0138  60.4315  218.1043   
2017-01-04  112.6246  59.9873  218.4625  112.8873  60.1611  219.4018   
2017-01-05  112.6830  59.9003  218.3366  113.4614  60.1611  219.2275   
2017-01-06  113.3251  59.9100  218.7433  114.7263  60.6825  220.0118   
2017-01-09  114.7554  60.3928  219.2433  115.7771  60.4894  219.2856   
2017-01-10  115.1057  60.1418  218.8499  115.8939  60.4701  219.2856   
2017-01-11  115.3976  60.2866  218.4432  116.5166  61.0205  219.9053   
2017-01-12  115.0182  59.8231  217.8283  116.0301  60.4604  219.3534   
2017-01-13  115.6020  60.2094  219.5083  115.8257  60.5473  219.8569   

Attributes    volume                      
Symbols         AAPL      MSFT       SPY  
date                                      
2017-01-03  28781865  20694101  91366522  
2017-01-04  21118116  21339969  78744433  
2017-01-05  22193587  24875968  78379012  
2017-01-06  31751900  19922919  71559922  
2017-01-09  33561948  20382730  46939676  
2017-01-10  24462051  18593004  63771939  
2017-01-11  27588593  21517335  74650016  
2017-01-12  27086220  20968223  72113181  
2017-01-13  26111948  19422310  62717865

As shown, for IEX, the retrieved data is in reversed order, and there is a hidden ERROR about the time so far. Let's just ignore it at this stage.

2.4 Quandl

Note: for Quandl,

  • you may have to load ONE data at a time
  • you may have to subscribe and pay to obtain S&P 500 data? Anyway, please refer to Quandl MULTPL and multpl.com
1
2
3
4
5
# We would like all available data from 01/01/2017 until 12/31/2017.
start_date = '2017-01-01'
end_date = '2017-12-31'
df = data.DataReader('AAPL', data_source="quandl", start=start_date, end=end_date, access_key="your-quandl-api-key")
# df = data.DataReader('MSFT', data_source="quandl", start=start_date, end=end_date, access_key="your-quandl-api-key")

AAPL

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
              Open     High      Low   Close      Volume  ExDividend  \
Date
2017-12-29  170.52  170.590  169.220  169.23  25643711.0         0.0
2017-12-28  171.00  171.850  170.480  171.08  15997739.0         0.0
2017-12-27  170.10  170.780  169.710  170.60  21672062.0         0.0
2017-12-26  170.80  171.470  169.679  170.57  32968167.0         0.0
2017-12-22  174.68  175.424  174.500  175.01  16052615.0         0.0
2017-12-21  174.17  176.020  174.100  175.01  20356826.0         0.0
2017-12-20  174.87  175.420  173.250  174.35  23000392.0         0.0
2017-12-19  175.03  175.390  174.090  174.54  27078872.0         0.0
2017-12-18  174.88  177.200  174.860  176.42  28831533.0         0.0

            SplitRatio  AdjOpen  AdjHigh   AdjLow  AdjClose   AdjVolume
Date
2017-12-29         1.0   170.52  170.590  169.220    169.23  25643711.0
2017-12-28         1.0   171.00  171.850  170.480    171.08  15997739.0
2017-12-27         1.0   170.10  170.780  169.710    170.60  21672062.0
2017-12-26         1.0   170.80  171.470  169.679    170.57  32968167.0
2017-12-22         1.0   174.68  175.424  174.500    175.01  16052615.0
2017-12-21         1.0   174.17  176.020  174.100    175.01  20356826.0
2017-12-20         1.0   174.87  175.420  173.250    174.35  23000392.0
2017-12-19         1.0   175.03  175.390  174.090    174.54  27078872.0
2017-12-18         1.0   174.88  177.200  174.860    176.42  28831533.0

MSFT

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
             Open     High     Low  Close      Volume  ExDividend  SplitRatio  \
Date                                                                            
2017-12-29  85.63  86.0500  85.500  85.54  18162779.0         0.0         1.0   
2017-12-28  85.90  85.9300  85.550  85.72   9872795.0         0.0         1.0   
2017-12-27  85.65  85.9800  85.215  85.71  13000828.0         0.0         1.0   
2017-12-26  85.31  85.5346  85.030  85.40   9737412.0         0.0         1.0   
2017-12-22  85.40  85.6300  84.920  85.51  14033977.0         0.0         1.0   
2017-12-21  86.05  86.1000  85.400  85.50  16638402.0         0.0         1.0   
2017-12-20  86.20  86.3000  84.710  85.52  23425009.0         0.0         1.0   
2017-12-19  86.35  86.3500  85.270  85.83  23241979.0         0.0         1.0   
2017-12-18  87.12  87.4999  86.230  86.38  21551076.0         0.0         1.0   

            AdjOpen  AdjHigh  AdjLow  AdjClose   AdjVolume  
Date                                                        
2017-12-29    85.63  86.0500  85.500     85.54  18162779.0  
2017-12-28    85.90  85.9300  85.550     85.72   9872795.0  
2017-12-27    85.65  85.9800  85.215     85.71  13000828.0  
2017-12-26    85.31  85.5346  85.030     85.40   9737412.0  
2017-12-22    85.40  85.6300  84.920     85.51  14033977.0  
2017-12-21    86.05  86.1000  85.400     85.50  16638402.0  
2017-12-20    86.20  86.3000  84.710     85.52  23425009.0  
2017-12-19    86.35  86.3500  85.270     85.83  23241979.0  
2017-12-18    87.12  87.4999  86.230     86.38  21551076.0
AAPL Closing Price MSFT Closing Price
AAPL Closing Price MSFT Closing Price

3. Additional Python Tools

3.1 fix-yahoo-finance

fix-yahoo-finance.

3.2 googlefinance.client

googlefinance.client

Posted on Edited on In

Today, let's have some fun of AI in finance. We are going to use a tool named Quantopian. As far as I know, functional programming language Haskell has been widely used in quantitative analysis. That's possibly why a dependent package of Quantopian, namely pandoc, requires The Haskell Tool Stack for its installation. Therefore, Haskell and pandoc need to be installed before Quantopian installation.

1. Preparations

1.1 Haskell Installation

Please refer to https://docs.haskellstack.org/en/stable/install_and_upgrade/:

1
➜  ~ curl -sSL https://get.haskellstack.org/ | sh

1.2 pandoc Installation

Please refer to https://pandoc.org/installing.html:

1
2
3
4
➜  ~ git clone https://github.com/jgm/pandoc
➜  ~ cd pandoc
➜  ~ stack setup
➜  ~ stack install

1.3 Zipline Installation with pip (ERRORS)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
➜  ~ pip install -U zipline --user
Collecting zipline
  Using cached https://files.pythonhosted.org/packages/be/59/8c5802a7897c1095fdc409fb557f04df8f75c37174e80d2ba58c8d8a6488/zipline-1.3.0.tar.gz
Requirement already satisfied, skipping upgrade: pip>=7.1.0 in ~/.local/lib/python3.6/site-packages (from zipline) (18.1)
Requirement already satisfied, skipping upgrade: setuptools>18.0 in ~/.local/lib/python3.6/site-packages (from zipline) (40.5.0)
Collecting Logbook>=0.12.5 (from zipline)
  Using cached https://files.pythonhosted.org/packages/74/fc/3e7557ed1ef1bd4e3ee189fc670416abfc7192b550e8d3c1d858a63f41ab/Logbook-1.4.1.tar.gz
Requirement already satisfied, skipping upgrade: pytz>=2016.4 in ~/.local/lib/python3.6/site-packages (from zipline) (2018.7)
Requirement already satisfied, skipping upgrade: numpy>=1.11.1 in ~/.local/lib/python3.6/site-packages (from zipline) (1.15.4)
Requirement already satisfied, skipping upgrade: requests-file>=1.4.1 in ~/.local/lib/python3.6/site-packages (from zipline) (1.4.3)
Requirement already satisfied, skipping upgrade: scipy>=0.17.1 in ~/.local/lib/python3.6/site-packages (from zipline) (1.1.0)
Collecting pandas<=0.22,>=0.18.1 (from zipline)
  Using cached https://files.pythonhosted.org/packages/da/c6/0936bc5814b429fddb5d6252566fe73a3e40372e6ceaf87de3dec1326f28/pandas-0.22.0-cp36-cp36m-manylinux1_x86_64.whl
Collecting pandas-datareader>=0.2.1 (from zipline)
  Using cached https://files.pythonhosted.org/packages/cc/5c/ea5b6dcfd0f55c5fb1e37fb45335ec01cceca199b8a79339137f5ed269e0/pandas_datareader-0.7.0-py2.py3-none-any.whl
Requirement already satisfied, skipping upgrade: patsy>=0.4.0 in ~/.local/lib/python3.6/site-packages (from zipline) (0.5.1)
Collecting statsmodels>=0.6.1 (from zipline)
  Using cached https://files.pythonhosted.org/packages/85/d1/69ee7e757f657e7f527cbf500ec2d295396e5bcec873cf4eb68962c41024/statsmodels-0.9.0-cp36-cp36m-manylinux1_x86_64.whl
Requirement already satisfied, skipping upgrade: python-dateutil>=2.4.2 in ~/.local/lib/python3.6/site-packages (from zipline) (2.7.5)
Requirement already satisfied, skipping upgrade: six>=1.10.0 in ~/.local/lib/python3.6/site-packages (from zipline) (1.11.0)
Requirement already satisfied, skipping upgrade: requests>=2.9.1 in ~/.local/lib/python3.6/site-packages (from zipline) (2.20.0)
Requirement already satisfied, skipping upgrade: Cython>=0.25.2 in ~/.local/lib/python3.6/site-packages (from zipline) (0.29)
Collecting cyordereddict>=0.2.2 (from zipline)
  Using cached https://files.pythonhosted.org/packages/d1/1a/364cbfd927be1b743c7f0a985a7f1f7e8a51469619f9fefe4ee9240ba210/cyordereddict-1.0.0.tar.gz
Collecting bottleneck>=1.0.0 (from zipline)
  Using cached https://files.pythonhosted.org/packages/05/ae/cedf5323f398ab4e4ff92d6c431a3e1c6a186f9b41ab3e8258dff786a290/Bottleneck-1.2.1.tar.gz
Requirement already satisfied, skipping upgrade: contextlib2>=0.4.0 in ~/.local/lib/python3.6/site-packages/contextlib2-0.5.5-py3.6.egg (from zipline) (0.5.5)
Requirement already satisfied, skipping upgrade: decorator>=4.0.0 in ~/.local/lib/python3.6/site-packages (from zipline) (4.3.0)
Collecting networkx<2.0,>=1.9.1 (from zipline)
  Using cached https://files.pythonhosted.org/packages/d3/2c/e473e54afc9fae58dfa97066ef6709a7e35a1dd1c28c5a3842989322be00/networkx-1.11-py2.py3-none-any.whl
Collecting numexpr>=2.6.1 (from zipline)
  Using cached https://files.pythonhosted.org/packages/db/ea/efd9e16283637eb5b6c0042b6cc3521f1b9a5b47767ac463c88bbd37670c/numexpr-2.6.8-cp36-cp36m-manylinux1_x86_64.whl
Collecting bcolz<1,>=0.12.1 (from zipline)
  Using cached https://files.pythonhosted.org/packages/6c/8b/1ffa01f872cac36173c5eb95b58c01040d8d25f1b242c48577f4104cd3ab/bcolz-0.12.1.tar.gz
Requirement already satisfied, skipping upgrade: click>=4.0.0 in ~/.local/lib/python3.6/site-packages (from zipline) (7.0)
Requirement already satisfied, skipping upgrade: toolz>=0.8.2 in ~/.local/lib/python3.6/site-packages (from zipline) (0.9.0)
Requirement already satisfied, skipping upgrade: multipledispatch>=0.4.8 in ~/.local/lib/python3.6/site-packages (from zipline) (0.6.0)
Requirement already satisfied, skipping upgrade: MarkupSafe>=0.23 in ~/.local/lib/python3.6/site-packages (from zipline) (1.1.0)
Requirement already satisfied, skipping upgrade: Mako>=1.0.1 in /usr/lib/python3/dist-packages (from zipline) (1.0.7)
Collecting sqlalchemy>=1.0.8 (from zipline)
  Using cached https://files.pythonhosted.org/packages/1e/98/4dba86354d271344e25fa01dc38a0bf0e0ba6407ad2d5e8426496a95b568/SQLAlchemy-1.2.13.tar.gz
Collecting alembic>=0.7.7 (from zipline)
  Using cached https://files.pythonhosted.org/packages/c0/f3/e60af9a36ae3b8cafabc7e0834d8df6a2965b3feecf27b9b11352dc05dd4/alembic-1.0.2.tar.gz
Collecting sortedcontainers>=1.4.4 (from zipline)
  Using cached https://files.pythonhosted.org/packages/be/e3/a065de5fdd5849450a8a16a52a96c8db5f498f245e7eda06cc6725d04b80/sortedcontainers-2.0.5-py2.py3-none-any.whl
Collecting intervaltree>=2.1.0 (from zipline)
  Using cached https://files.pythonhosted.org/packages/ca/c1/450d109b70fa58ca9d77972b02f69222412f9175ccf99fdeaf167be9583c/intervaltree-2.1.0.tar.gz
Collecting lru-dict>=1.1.4 (from zipline)
  Using cached https://files.pythonhosted.org/packages/00/a5/32ed6e10246cd341ca8cc205acea5d208e4053f48a4dced2b1b31d45ba3f/lru-dict-1.1.6.tar.gz
Collecting empyrical>=0.5.0 (from zipline)
  Using cached https://files.pythonhosted.org/packages/7b/55/a01b05162b764830dbbac868462f44cd847a5b6523a01ca9f955721819da/empyrical-0.5.0.tar.gz
Collecting tables>=3.3.0 (from zipline)
  Using cached https://files.pythonhosted.org/packages/d7/1b/21f4c7f296b718575c17ef25e61c05742a283c45077b4c8d5a190b3e0b59/tables-3.4.4-cp36-cp36m-manylinux1_x86_64.whl
Collecting trading-calendars>=1.0.1 (from zipline)
  Using cached https://files.pythonhosted.org/packages/6e/e9/e9e2aeadacab58580ed9493e6c252f051cd8ee9d7efbe49d2f041371c45d/trading_calendars-1.5.1.tar.gz
Requirement already satisfied, skipping upgrade: wrapt in ~/.local/lib/python3.6/site-packages (from pandas-datareader>=0.2.1->zipline) (1.10.11)
Requirement already satisfied, skipping upgrade: lxml in ~/.local/lib/python3.6/site-packages (from pandas-datareader>=0.2.1->zipline) (4.2.5)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in ~/.local/lib/python3.6/site-packages (from requests>=2.9.1->zipline) (2018.10.15)
Requirement already satisfied, skipping upgrade: idna<2.8,>=2.5 in ~/.local/lib/python3.6/site-packages (from requests>=2.9.1->zipline) (2.7)
Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in /usr/lib/python3/dist-packages (from requests>=2.9.1->zipline) (3.0.4)
Requirement already satisfied, skipping upgrade: urllib3<1.25,>=1.21.1 in ~/.local/lib/python3.6/site-packages (from requests>=2.9.1->zipline) (1.24.1)
Collecting python-editor>=0.3 (from alembic>=0.7.7->zipline)
  Using cached https://files.pythonhosted.org/packages/65/1e/adf6e000ea5dc909aa420352d6ba37f16434c8a3c2fa030445411a1ed545/python-editor-1.0.3.tar.gz
Building wheels for collected packages: zipline, Logbook, cyordereddict, bottleneck, bcolz, sqlalchemy, alembic, intervaltree, lru-dict, empyrical, trading-calendars, python-editor
  Running setup.py bdist_wheel for zipline ... done
  Stored in directory: ~/.cache/pip/wheels/a4/d6/67/f303ab028b004bf8e00c05b5b04fba83d8ec238b6547becdb7
  Running setup.py bdist_wheel for Logbook ... done
  Stored in directory: ~/.cache/pip/wheels/06/13/e9/88e9e8184d89671ffc754dc80f5eb01dabd72071bdb802c5d1
  Running setup.py bdist_wheel for cyordereddict ... done
  Stored in directory: ~/.cache/pip/wheels/0b/9d/8b/5bf3e22c1edd59b50f11bb19dec9dfcfe5a479fc7ace02b61f
  Running setup.py bdist_wheel for bottleneck ... done
  Stored in directory: ~/.cache/pip/wheels/f2/bf/ec/e0f39aa27001525ad455139ee57ec7d0776fe074dfd78c97e4
  Running setup.py bdist_wheel for bcolz ... done
  Stored in directory: ~/.cache/pip/wheels/c5/cc/1b/2cf1f88959af5d7f4d449b7fc6c9452d0ecbd86fd61a9ee376
  Running setup.py bdist_wheel for sqlalchemy ... done
  Stored in directory: ~/.cache/pip/wheels/ba/2e/47/b194b03e6aeaa658f521dfba094a42f1d37f4ee4018b9e607e
  Running setup.py bdist_wheel for alembic ... done
  Stored in directory: ~/.cache/pip/wheels/95/f3/c2/22155963496514f044049f5c96e2757b9838da4a5cae34e76e
  Running setup.py bdist_wheel for intervaltree ... done
  Stored in directory: ~/.cache/pip/wheels/6b/cf/b0/f7ef2d0f504d26f3e9e70c2369e5725591ccfaf67d528fcbc5
  Running setup.py bdist_wheel for lru-dict ... done
  Stored in directory: ~/.cache/pip/wheels/b7/ef/06/fbdd555907a7d438fb33e4c8675f771ff1cf41917284c51ebf
  Running setup.py bdist_wheel for empyrical ... done
  Stored in directory: ~/.cache/pip/wheels/83/14/73/34fb27552601518d28bd0813d75124be76d94ab29152c69112
  Running setup.py bdist_wheel for trading-calendars ... done
  Stored in directory: ~/.cache/pip/wheels/13/0f/d5/517a7a51d9998c20483d777bf2e35e71ec17445509a9ac4cc1
  Running setup.py bdist_wheel for python-editor ... done
  Stored in directory: ~/.cache/pip/wheels/36/e0/98/ba386b125a00ea9dd52e2c16aa2ec0adbbd639b84bfe2e001d
Successfully built zipline Logbook cyordereddict bottleneck bcolz sqlalchemy alembic intervaltree lru-dict empyrical trading-calendars python-editor
pyro-ppl 0.2.1+fbaa779 has requirement networkx>=2.0.0, but you'll have networkx 1.11 which is incompatible.
pyro-ppl 0.2.1+fbaa779 has requirement torch==0.4.0, but you'll have torch 1.0.0a0+6c8d47f which is incompatible.
Installing collected packages: Logbook, pandas, pandas-datareader, statsmodels, cyordereddict, bottleneck, networkx, numexpr, bcolz, sqlalchemy, python-editor, alembic, sortedcontainers, intervaltree, lru-dict, empyrical, tables, trading-calendars, zipline
  Found existing installation: pandas 0.23.4
    Uninstalling pandas-0.23.4:
      Successfully uninstalled pandas-0.23.4
  Found existing installation: networkx 2.2
    Uninstalling networkx-2.2:
      Successfully uninstalled networkx-2.2
Successfully installed Logbook-1.4.1 alembic-1.0.2 bcolz-0.12.1 bottleneck-1.2.1 cyordereddict-1.0.0 empyrical-0.5.0 intervaltree-2.1.0 lru-dict-1.1.6 networkx-1.11 numexpr-2.6.8 pandas-0.22.0 pandas-datareader-0.7.0 python-editor-1.0.3 sortedcontainers-2.0.5 sqlalchemy-1.2.13 statsmodels-0.9.0 tables-3.4.4 trading-calendars-1.5.1 zipline-1.3.0
➜  ~

You'll possibly meet these ERROR messages:

  • pyro-ppl 0.2.1+fbaa779 has requirement networkx>=2.0.0, but you'll have networkx 1.11 which is incompatible.
  • pyro-ppl 0.2.1+fbaa779 has requirement torch==0.4.0, but you'll have torch 1.0.0a0+6c8d47f which is incompatible.

In such a case, pyro is to be manually installed.

1.4 Manually Install pyro

Please refer to https://github.com/uber/pyro:

1
2
3
4
➜  uber git clone https://github.com/uber/pyro.git
➜  cd pyro
➜  pyro git:(dev) python setup.py build
➜  pyro git:(dev) python setup.py install --user

1.5 Install Zipline Again

Please refer to http://www.zipline.io/install.html:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
➜  ~ pip install -U zipline --user
Collecting zipline
Requirement already satisfied, skipping upgrade: numexpr>=2.6.1 in ./.local/lib/python3.6/site-packages (from zipline) (2.6.8)
Requirement already satisfied, skipping upgrade: cyordereddict>=0.2.2 in ./.local/lib/python3.6/site-packages (from zipline) (1.0.0)
Requirement already satisfied, skipping upgrade: tables>=3.3.0 in ./.local/lib/python3.6/site-packages (from zipline) (3.4.4)
Requirement already satisfied, skipping upgrade: pytz>=2016.4 in ./.local/lib/python3.6/site-packages (from zipline) (2018.7)
Requirement already satisfied, skipping upgrade: scipy>=0.17.1 in ./.local/lib/python3.6/site-packages (from zipline) (1.1.0)
Collecting bcolz<1,>=0.12.1 (from zipline)
Requirement already satisfied, skipping upgrade: patsy>=0.4.0 in ./.local/lib/python3.6/site-packages (from zipline) (0.5.1)
Requirement already satisfied, skipping upgrade: empyrical>=0.5.0 in ./.local/lib/python3.6/site-packages (from zipline) (0.5.0)
Requirement already satisfied, skipping upgrade: setuptools>18.0 in ./.local/lib/python3.6/site-packages (from zipline) (40.5.0)
Requirement already satisfied, skipping upgrade: requests-file>=1.4.1 in ./.local/lib/python3.6/site-packages (from zipline) (1.4.3)
Requirement already satisfied, skipping upgrade: requests>=2.9.1 in ./.local/lib/python3.6/site-packages (from zipline) (2.20.0)
Requirement already satisfied, skipping upgrade: intervaltree>=2.1.0 in ./.local/lib/python3.6/site-packages (from zipline) (2.1.0)
Requirement already satisfied, skipping upgrade: MarkupSafe>=0.23 in ./.local/lib/python3.6/site-packages (from zipline) (1.1.0)
Requirement already satisfied, skipping upgrade: contextlib2>=0.4.0 in ./.local/lib/python3.6/site-packages/contextlib2-0.5.5-py3.6.egg (from zipline) (0.5.5)
Requirement already satisfied, skipping upgrade: sqlalchemy>=1.0.8 in ./.local/lib/python3.6/site-packages (from zipline) (1.2.13)
Requirement already satisfied, skipping upgrade: toolz>=0.8.2 in ./.local/lib/python3.6/site-packages (from zipline) (0.9.0)
Requirement already satisfied, skipping upgrade: six>=1.10.0 in ./.local/lib/python3.6/site-packages (from zipline) (1.11.0)
Requirement already satisfied, skipping upgrade: pandas-datareader>=0.2.1 in ./.local/lib/python3.6/site-packages (from zipline) (0.7.0)
Requirement already satisfied, skipping upgrade: numpy>=1.11.1 in ./.local/lib/python3.6/site-packages (from zipline) (1.15.4)
Collecting pandas<=0.22,>=0.18.1 (from zipline)
  Using cached https://files.pythonhosted.org/packages/da/c6/0936bc5814b429fddb5d6252566fe73a3e40372e6ceaf87de3dec1326f28/pandas-0.22.0-cp36-cp36m-manylinux1_x86_64.whl
Requirement already satisfied, skipping upgrade: click>=4.0.0 in ./.local/lib/python3.6/site-packages (from zipline) (7.0)
Requirement already satisfied, skipping upgrade: Mako>=1.0.1 in /usr/lib/python3/dist-packages (from zipline) (1.0.7)
Requirement already satisfied, skipping upgrade: statsmodels>=0.6.1 in ./.local/lib/python3.6/site-packages (from zipline) (0.9.0)
Requirement already satisfied, skipping upgrade: bottleneck>=1.0.0 in ./.local/lib/python3.6/site-packages (from zipline) (1.2.1)
Requirement already satisfied, skipping upgrade: Logbook>=0.12.5 in ./.local/lib/python3.6/site-packages (from zipline) (1.4.1)
Collecting networkx<2.0,>=1.9.1 (from zipline)
  Using cached https://files.pythonhosted.org/packages/d3/2c/e473e54afc9fae58dfa97066ef6709a7e35a1dd1c28c5a3842989322be00/networkx-1.11-py2.py3-none-any.whl
Requirement already satisfied, skipping upgrade: alembic>=0.7.7 in ./.local/lib/python3.6/site-packages (from zipline) (1.0.2)
Requirement already satisfied, skipping upgrade: sortedcontainers>=1.4.4 in ./.local/lib/python3.6/site-packages (from zipline) (2.0.5)
Requirement already satisfied, skipping upgrade: lru-dict>=1.1.4 in ./.local/lib/python3.6/site-packages (from zipline) (1.1.6)
Requirement already satisfied, skipping upgrade: multipledispatch>=0.4.8 in ./.local/lib/python3.6/site-packages (from zipline) (0.6.0)
Requirement already satisfied, skipping upgrade: pip>=7.1.0 in ./.local/lib/python3.6/site-packages (from zipline) (18.1)
Requirement already satisfied, skipping upgrade: decorator>=4.0.0 in ./.local/lib/python3.6/site-packages (from zipline) (4.3.0)
Requirement already satisfied, skipping upgrade: Cython>=0.25.2 in ./.local/lib/python3.6/site-packages (from zipline) (0.29)
Requirement already satisfied, skipping upgrade: python-dateutil>=2.4.2 in ./.local/lib/python3.6/site-packages (from zipline) (2.7.5)
Requirement already satisfied, skipping upgrade: trading-calendars>=1.0.1 in ./.local/lib/python3.6/site-packages (from zipline) (1.5.1)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in ./.local/lib/python3.6/site-packages (from requests>=2.9.1->zipline) (2018.10.15)
Requirement already satisfied, skipping upgrade: urllib3<1.25,>=1.21.1 in ./.local/lib/python3.6/site-packages (from requests>=2.9.1->zipline) (1.24.1)
Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in /usr/lib/python3/dist-packages (from requests>=2.9.1->zipline) (3.0.4)
Requirement already satisfied, skipping upgrade: idna<2.8,>=2.5 in ./.local/lib/python3.6/site-packages (from requests>=2.9.1->zipline) (2.7)
Requirement already satisfied, skipping upgrade: wrapt in ./.local/lib/python3.6/site-packages (from pandas-datareader>=0.2.1->zipline) (1.10.11)
Requirement already satisfied, skipping upgrade: lxml in ./.local/lib/python3.6/site-packages (from pandas-datareader>=0.2.1->zipline) (4.2.5)
Requirement already satisfied, skipping upgrade: python-editor>=0.3 in ./.local/lib/python3.6/site-packages (from alembic>=0.7.7->zipline) (1.0.3)
pyro-ppl 0.2.1+913618a has requirement networkx>=2.0.0, but you'll have networkx 1.11 which is incompatible.
Installing collected packages: bcolz, pandas, networkx, zipline
  Found existing installation: bcolz 1.2.1
    Uninstalling bcolz-1.2.1:
      Successfully uninstalled bcolz-1.2.1
  Found existing installation: pandas 0.23.4
    Uninstalling pandas-0.23.4:
      Successfully uninstalled pandas-0.23.4
  Found existing installation: networkx 2.2
    Uninstalling networkx-2.2:
      Successfully uninstalled networkx-2.2
  Found existing installation: zipline 1.0.1+1816.gb7f454b7.dirty
    Uninstalling zipline-1.0.1+1816.gb7f454b7.dirty:
      Successfully uninstalled zipline-1.0.1+1816.gb7f454b7.dirty
Successfully installed bcolz-0.12.1 networkx-1.11 pandas-0.22.0 zipline-1.3.0
➜  ~ pip show networkx
Name: networkx
Version: 1.11
Summary: Python package for creating and manipulating graphs and networks
Home-page: http://networkx.github.io/
Author: NetworkX Developers
Author-email: networkx-discuss@googlegroups.com
License: BSD
Location: /home/jiapei/.local/lib/python3.6/site-packages
Requires: decorator
Required-by: zipline, scikit-image, pyro-ppl
➜  ~ pip show pyro-ppl
Name: pyro-ppl
Version: 0.2.1+913618a
Summary: A Python library for probabilistic modeling and inference
Home-page: http://pyro.ai
Author: Uber AI Labs
Author-email: pyro@uber.com
License: MIT License
Location: /home/jiapei/.local/lib/python3.6/site-packages/pyro_ppl-0.2.1+913618a-py3.6.egg
Requires: contextlib2, graphviz, networkx, numpy, opt-einsum, six, torch, tqdm
Required-by:
➜  ~

1.6 Python Package Compatibility

Let's take a look at the compatibility of python packages.

1
2
3
4
5
6
➜  ~ pip list --outdated
Package    Version   Latest Type 
---------- --------- ------ -----
bcolz      0.12.1    1.2.1  sdist
networkx   1.11      2.2    sdist
pandas     0.22.0    0.23.4 wheel

Clearly, Zipline is NOT compatible with the above 3 LATEST python packages.

2. Register Quandl and Ingest Data

As known, Quantopian is a Python collections of various quantitative analysis algorithms. However, where can we obtain the stock data for our testing? Either the real-time data, or the historical data will do.

We can of course use the real-time Yahoo Finance Data. But here, we are going to test on some historical data downloaded from Quandl, which is JUST the data adopted in Quantopian's Beginner's Tutorial.

After you've successfully registered Quandl and obtained an API key, the following command will help you to ingest Quandl WIKI Bundle data easily, please refer to http://www.zipline.io/bundles.html#ingesting-data:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
➜  quantopian QUANDL_API_KEY=<yourkey> zipline ingest -b quandl
[2018-11-07 05:18:37.895463] INFO: zipline.data.bundles.quandl: Downloading WIKI metadata.
Downloading WIKI Prices table from Quandl  [####################################]  100%          
[2018-11-07 05:18:54.369791] INFO: zipline.data.bundles.quandl: Parsing raw data.
[2018-11-07 05:19:24.118879] INFO: zipline.data.bundles.quandl: Generating asset metadata.
Merging daily equity files:  [-----------------------------#------]  1731~/.local/lib/python3.6/site-packages/zipline/data/us_equity_pricing.py:417: UserWarning: Ignoring 1 values because they are out of bounds for uint32:             open  high   low  close        volume  ex_dividend  split_ratio
2011-04-11  1.79  1.84  1.55    1.7  6.674913e+09          0.0          1.0
  winsorise_uint32(raw_data, invalid_data_behavior, 'volume', *OHLC)
Merging daily equity files:  [####################################]      
[2018-11-07 05:21:34.367628] INFO: zipline.data.bundles.quandl: Parsing split data.
[2018-11-07 05:21:34.572459] INFO: zipline.data.bundles.quandl: Parsing dividend data.
~/.local/lib/python3.6/site-packages/zipline/data/us_equity_pricing.py:931: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  if not issubdtype(actual, expected):
➜  quantopian

3. First Try

Details can be easily found at http://www.zipline.io/beginner-tutorial.html. Here, we ONLY test the example ./zipline/examples/buyapple.py, which can also be found on Github.

Following the Beginner's Tutorial, by executing the following command,

1
➜  examples git:(master) ✗ zipline run -f ./buyapple.py --start 2016-1-1 --end 2018-1-1 -o buyapple_out.pickle

we should be able to obtain the resultant figure as follows:

Quantopian Example - Buy Apple

Posted on Edited on In ,

October 31, 2018 is Halloween... ^_^ Happy Halloween everyone...

Happy Halloween

And, Happy Birthday To Myself. Such an old NOBODY... DON'T ask my age. Anyway, enjoy life...

Happy Birthday

Today, we are testing out some pose estimation open source, based on various open source AI frameworks, including: * PyTorch * Tensorflow * Caffe2 * Caffe

1. OpenPose

Body Tracking Body Foot
Tracking Body ONLY Body and Foot
Body Hand Body Face
Body and Hand Body and Face

The related paper can be found at Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields.

2. tf-pose-estimation

Please refer to Posture Recognition by A Single 2D Webcam

3. pose-tensorflow

Single Person Multiple Persons
Single Person Multiple Persons

4. Facebook Detectron

5. Facebook DetectAndTrack

6. Facebook DensePose

6.1 COCO Visualize

Original Image Collected Points Subplots
Original Image Collected Points Subplots

6.2 RCNN Texture Transfer

Texture Mesh Synth UV
Texture Mesh Synth UV
IUV Texture Transfer Original Image
IUV Mesh Original Image

6.3 RCNN Visualize

IUV Mesh Contour
IUV Mesh Contour

7. Facebook maskrcnn-benchmark

Test on maskrcnn-benchmark

Posted on Edited on In ,

Hi, everyone... I've been so busy and haven't written anything for a bit. My previous boss at Lenovo is now in Vancouver for annual executive meeting. She brought me a Huawei HiSilicon Hikey 970.

Several things before testing: * Lenovo and Huawei are two BIGGEST and possibly BEST IT companies from China. * HiSilicon is a chip design and manufacturing company spinned off from Huawei. * 96Boards hosts various SBCs (Single Board Computers), including Hikey 970. * A lot more resource can be found on LeMaker, who has designed several low-cost single board computers (SBCs). - LeMaker Hikey 970 Home Page - LeMaker Hikey 970 Specification - LeMaker Hikey 970 Resource

Now, let's start today's journey.

PART A: Introduction

1. Layout of Hikey 970

Hikey 970 looks like (cited from Hikey 970 ):

HiSilicon Hikey 970

2. Connection of Hikey 970

HiSilicon Hikey 970

Please pay attention to that pink rectangle, where 4 TINY switches are located and will be adjusted while flashing.

PART B: Install Lebuntu onto Hikey 970

1. Download Lebuntu for Hikey 970

We FIRST go visiting the website http://www.lemaker.org/product-hikey970-download-84.html and click Download icon, a file named Hikey 970 Lebuntu.rar will be automatically downloaded. After the extraction, a folder named hikey970-lebuntu-16.04 will be generated.

2. Preparations for Making Full Use of Flash Memony On Hikey 970

By integrating the following 2 contents:

we should be able to successfully install Lebuntu, and make FULL use of the whole flash memory on board.

2.1 Mount Boot Image

Make sure the current path is hikey970-lebuntu-16.04, then:

1
2
sudo mkdir /mnt/boot-hikey970
sudo mount -o loop boot-hikey970.uefi.img /mnt/boot-hikey970

2.2 Edit grub.cfg

1
sudo vim /mnt/boot-hikey970/boot/grub/grub.cfg
  • sdd12 -> sdd15
  • gpt12 -> gpt15

2.3 Edit flash scripts

There are 3 flash scripts:

  • flash-all-binaries.sh
  • flash-minimum-binaries.sh
  • binaries/recovery-flash.sh

Change all cases in the above 3 files:

  • sudo fastboot flash system leubuntu-16.04.img -> sudo fastboot flash userdata leubuntu-16.04.img.

3 Flash Lebuntu onto Hikey 970

3.1 Adjust the Switches

  • Switch 1 : ON
  • Switch 2 : ON
  • Switch 3 : ON
  • Switch 4 : OFF

3.2 Connect Type C USB cable and Power Adapter

  • Connect Type C USB cable between the host computer and Hikey 970.
  • Connect power adapter.
  • Make sure Hikey 970 is configured as ttyUSB0.
1
$ ls -ls /dev/ttyUSB0

3.3 System Installation

1
./binaries/recovery-flash.sh

3.4 Unplug Power Supply then Re-adjust the Switches

Unplug the power adapter, and then re-adjust the switches: * Switch 1 : ON * Switch 2 : OFF * Switch 3 : OFF * Switch 4 : OFF

3.5 Connect HDMI Cable then Power Adapter

  • Connect HDMI cable.
  • Connect the Power Adapter.

NOTE: Plug the HDMI cable before powering up. Lebuntu may NOT boot up without plugging in the HDMI.

3.6 Boot up

You may have to re-plug in the power adapter multiple times, in order to boot up Lebuntu. Username and password are respectively:

  • Username: shunya
  • Password: shunya

Expand the Partition

In order to make full use of the partition, we need to resize the file system as:

1
shunya@hikey970:~$ sudo resize2fs /dev/sdd15

PART C: Waiting for HuaWei HiAI

Longer Vision Technology is waiting for HuaWei Developer account activation:

HuaWei Developer Account Activation

I may talk about Lebian with Pre-built Tensorflow some time later.

So tired these days... Nice dream...

Posted on Edited on In

I've been looking for an IDE for debugging various microcontroller boards, such as OpenMV and PyBoard, etc. There are some fundamental requirements:

  • Open source
  • Python debugger, MicroPython support
  • Any GUI is fine, but Qt preferred

Therefore, after a careful search, I found uPyCraft, which seems to be designed by a Chinese team? Because I found the Chinese version of uPyCraft documentation as well.

However, it seems Linux support for uPyCraft is NOT very good. My working environment:

1
2
3
4
5
6
7
8
9
10
11
12
13
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:        18.04
Codename:       bionic
$ uname -a
Linux jiapei-GT72-6QE 4.18.7-041807-generic #201809090930 SMP Sun Sep 9 09:33:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ python --version
Python 3.6.5
$ qmake --version
QMake version 3.1
Using Qt version 5.11.1 in /opt/Qt/Current/gcc_64/lib

PART A: Test Built uPyCraft

1. Try the Executable

In fact, there is one executable Linux version for uPyCraft. However, to run it after downloading, I obtained the following ERROR messages:

1
2
3
4
5
6
$ ./uPyCraft_linux_V1.0 
Traceback (most recent call last):
  File "uPyCraft.py", line 2, in <module>
  File "/usr/local/lib/python3.5/dist-packages/PyInstaller/loader/pyimod03_importers.py", line 714, in load_module
ImportError: /tmp/_MEIoq8Xi8/libz.so.1: version `ZLIB_1.2.9' not found (required by /usr/lib/x86_64-linux-gnu/libpng16.so.16)
[324] Failed to execute script uPyCraft

2. Try source code

We can also clone uPyCraft source code and build it from scratch. However, clearly, current uPyCraft relies on PyQt4 instead of PyQt5.

Therefore, I've got to modify uPyCraft a bit to have this uPyCraft_PyQt5.

PART B: Build uPyCraft with PyQt5

Before testing uPyCraft_PyQt5, you need to make sure * python3 * Qt5 have been installed.

**What's more important: Please uninstall enum34 but keep the original enum. Please refer to Stack Overflow for more details.

1. Check out and build my modified code of uPyCraft_PyQt5

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
$ git clone git@github.com:jiapei100/uPyCraft_PyQt5.git
$ cd uPyCraft_PyQt5
$ pyinstaller -F uPyCraft.py
47 INFO: PyInstaller: 3.4
47 INFO: Python: 3.6.5
47 INFO: Platform: Linux-4.18.7-041807-generic-x86_64-with-Ubuntu-18.04-bionic
48 INFO: wrote ....../uPyCraft_PyQt5/uPyCraft.spec
50 INFO: UPX is not available.
51 INFO: Extending PYTHONPATH with paths
['....../uPyCraft_PyQt5',
 '....../uPyCraft_PyQt5']
52 INFO: checking Analysis
54 INFO: Building because inputs changed
54 INFO: Initializing module dependency graph...
57 INFO: Initializing module graph hooks...
59 INFO: Analyzing base_library.zip ...
2747 INFO: running Analysis Analysis-00.toc
2776 INFO: Caching module hooks...
2779 INFO: Analyzing ....../uPyCraft_PyQt5/uPyCraft.py
3619 INFO: Loading module hooks...
3619 INFO: Loading module hook "hook-PyQt5.QtGui.py"...
3749 INFO: Loading module hook "hook-pydoc.py"...
3750 INFO: Loading module hook "hook-PyQt5.QtWidgets.py"...
3832 INFO: Loading module hook "hook-xml.py"...
3914 INFO: Loading module hook "hook-PyQt5.QtCore.py"...
3933 INFO: Loading module hook "hook-PyQt5.QtPrintSupport.py"...
4057 INFO: Loading module hook "hook-encodings.py"...
4126 INFO: Loading module hook "hook-PyQt5.py"...
4170 WARNING: Hidden import "sip" not found!
4179 INFO: Looking for ctypes DLLs
4274 WARNING: library setupapi required via ctypes not found
4299 WARNING: library Advapi32 required via ctypes not found
4367 INFO: Analyzing run-time hooks ...
4370 INFO: Including run-time hook 'pyi_rth_qt5.py'
4376 INFO: Looking for dynamic libraries
8225 INFO: Looking for eggs
8225 INFO: Python library not in binary dependencies. Doing additional searching...
8292 INFO: Using Python library /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
8301 INFO: Warnings written to ....../uPyCraft_PyQt5/build/uPyCraft/warn-uPyCraft.txt
8339 INFO: Graph cross-reference written to ....../uPyCraft_PyQt5/build/uPyCraft/xref-uPyCraft.html
8352 INFO: checking PYZ
8354 INFO: Building because name changed
8354 INFO: Building PYZ (ZlibArchive) ....../uPyCraft_PyQt5/build/uPyCraft/PYZ-00.pyz
8762 INFO: Building PYZ (ZlibArchive) ....../uPyCraft_PyQt5/build/uPyCraft/PYZ-00.pyz completed successfully.
8765 INFO: checking PKG
8767 INFO: Building because name changed
8767 INFO: Building PKG (CArchive) PKG-00.pkg
33163 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
33168 INFO: Bootloader ~/.local/lib/python3.6/site-packages/PyInstaller/bootloader/Linux-64bit/run
33168 INFO: checking EXE
33170 INFO: Building because name changed
33170 INFO: Building EXE from EXE-00.toc
33175 INFO: Appending archive to ELF section in EXE ....../uPyCraft_PyQt5/dist/uPyCraft
33753 INFO: Building EXE from EXE-00.toc completed successfully.

2. Run and test uPyCraft_PyQt5

To run uPyCraft_PyQt5

1
2
$ cd dist
$ ./uPyCraft
uPyCraft

So far, 6 boards are particularly supported:

For the sake of our requirements, openmv is going to be supported in particular.

Posted on Edited on In ,

It's been quite a while before I write down this similar blog as my previous Single Board Computer blogs. Banana Pi Pro is a SBC with AllWinner A20 ARM CPU, which is also just of card size (Cited from LeMaker Official website).

Orange Pi Plus 2

Now, we follow our previous blog Install Armbian Debian Server onto NanoPi NEO to install Armbian Ubuntu Desktop onto Banana Pi Pro.

PART A: Install Ubuntu Desktop Built By Armbian onto Banana Pi Pro

1. Download Armbian Ubuntu Desktop for Banana Pi Pro

We FIRST go visiting the website https://www.armbian.com/banana-pi-pro/ and click Armbian Bionic icon, a file named Armbian_5.59_Bananapipro_Ubuntu_bionic_next_4.14.65.7z will be automatically downloaded.

Then, we extract this .7z file by

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
$ 7z x Armbian_5.59_Bananapipro_Ubuntu_bionic_next_4.14.65.7z -oArmbian_5.59_Bananapipro_Ubuntu_bionic_next_4.14.65

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_CA.UTF-8,Utf16=on,HugeFiles=on,64 bits,8 CPUs Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz (506E3),ASM,AES-NI)

Scanning the drive for archives:
1 file, 202610698 bytes (194 MiB)

Extracting archive: Armbian_5.59_Bananapipro_Ubuntu_bionic_next_4.14.65.7z
--
Path = Armbian_5.59_Bananapipro_Ubuntu_bionic_next_4.14.65.7z
Type = 7z
Physical Size = 202610698
Headers Size = 290
Method = LZMA2:25
Solid = +
Blocks = 1

Everything is Ok                                              

Files: 4
Size:       885017676
Compressed: 202610698

2. Install Armbian Ubuntu Desktop for Banana Pi Pro

After the extracted image file is prepared, it's the time to install the Armbian Ubuntu Desktop onto our TF card. We FIRST format the TF card:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ sudo umount /dev/mmcblk0p1 
[sudo] password for jiapei: 
$ sudo mkfs.ext4 /dev/mmcblk0 
mke2fs 1.44.1 (24-Mar-2018)
Found a dos partition table in /dev/mmcblk0
Proceed anyway? (y,N) y
Discarding device blocks: done                            
Creating filesystem with 7791744 4k blocks and 1949696 inodes
Filesystem UUID: 4e7bb315-0db1-4b62-a9d5-40c0ba78c7f0
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
        4096000

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

Afterwards, use dd to install the downloaded Armbian Ubuntu Desktop image.

1
2
3
4
5
$ cd Armbian_5.59_Bananapipro_Ubuntu_bionic_next_4.14.65/
$ sudo dd bs=4M if=Armbian_5.59_Bananapipro_Ubuntu_bionic_next_4.14.65.img of=/dev/mmcblk0 conv=fsync
211+0 records in
211+0 records out
884998144 bytes (885 MB, 844 MiB) copied, 123.525 s, 7.2 MB/s

PART B: Boot Into Armbian, Network Configuration, Locale Configuration and Armbian Upgrading

NOTE: Whenever you met some unsolvable issue after you boot into Armbian, please use the command armbianmonitor -U to log the issue.

1. Boot Into Armbian

We now unplug the TF card from the host and put it into the Banana Pi Pro board, Armbian Ubuntu Desktop boots successfully. The default username and password are respectively: root and 1234.

1
2
3
4
5
6
7
8
9
10
Ubuntu 18.04.1 LTS bananapipro tty1

bananapipro login: root
password:
You are required to change your password immediately (root enforced)
Changing password for root.
(current) UNIX password:
Enter new UNIX password:
Retype new UNIX password:
...

And, for the FIRST boot, we'll be asked to create a NEW user besides root:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
Creating a new user account. Press <Ctrl-C> to abort

Please provide a username (eg. your forename): bananapipro
Trying to add user bananapipro
Adding user 'bananapipro' ...
Adding new group 'bananapipro' (1000) ...
Adding new user 'bananapipro' (1000) with group 'bananapipro' ...
Creating home directory '/home/bananapipro' ...
Copying files from '/etc/ske1' ...
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Changing the user information for orangepiplus2
Eter the new value, or press ENTER for the default
       Full Name []:
       Room Number []:
       Work Phone []:
       Home Phone []:
       Other []:
Is the information correct? [Y/n] Y

Dear bananapipro, your account bananapipro has been created and is sudo enabled.
Please use this account for your daily work from now on.
...

Then, Banana Pi Pro now boots into text mode with account root. 

1
root@bananapipro:~$

We may now re-login with the newly created account bananapipro: 

1
bananapipro@bananapipro:~$

2. Network Configuration

1). How to connect to wireless?

Please refer to Armbian docs - How to connect to wireless?, 

1
bananapipro@bananapipro:~$ nmtui-connect SSID

and then input the corresponding password. Now, you should be able to connect to the Internet with a DHCP IP address.

2). How to set fixed IP?

Please refer to Armbian docs - How to set fixed IP?, and modify the file /etc/network/interfaces as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
bananapipro@bananapipro:~$ sudo vim /etc/network/interfaces
source /etc/network/interfaces.d/*
# Network is managed by Network manager
auto lo
iface lo inet loopback

# Wifi
auto wlan0
iface wlan0 inet static
    address 192.168.0.82
    netmask 255.255.255.0
    network 192.168.0.0
    broadcast 192.168.0.1
    gateway 192.168.0.1
    dns-nameservers 192.168.0.1 8.8.8.8
    wpa-ssid XXXXXXXX
    wpa-psk YYYYYYYY

Afterwards, a reboot (meanwhile, you may have to set up your own router for a fixed IP) will bring your Banana Pi Pro a fixed IP address. Here in my case: 192.168.0.82.

3. SSH Into Banana Pi Pro

We can now access Banana Pi Pro via ssh as follows: 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$ ssh bananapipro@192.168.0.82
The authenticity of host '192.168.0.82 (192.168.0.82)' can't be established.
ECDSA key fingerprint is SHA256:SKJ/eU/a7Rm6T43jzwvNpQfG1w2FqDPg+eieM+sGynA.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.0.82' (ECDSA) to the list of known hosts.
bananapipro@192.168.0.82's password: 
 ____                                  ____  _   ____            
| __ )  __ _ _ __   __ _ _ __   __ _  |  _ \(_) |  _ \ _ __ ___  
|  _ \ / _` | '_ \ / _` | '_ \ / _` | | |_) | | | |_) | '__/ _ \ 
| |_) | (_| | | | | (_| | | | | (_| | |  __/| | |  __/| | | (_) |
|____/ \__,_|_| |_|\__,_|_| |_|\__,_| |_|   |_| |_|   |_|  \___/ 
                                                                 

Welcome to ARMBIAN 5.59 stable Ubuntu 18.04.1 LTS 4.14.65-sunxi   
System load:   0.43 0.32 0.22   Up time:       15 min
Memory usage:  6 % of 992MB     IP:            192.168.0.82
CPU temp:      46°C           
Usage of /:    3% of 29G    

Last login: Tue Sep  4 21:43:26 2018

And, let's have a look at current IP of Banana Pi Pro.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
bananapipro@bananapipro:~$ ip -c address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:0
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 02:86:02:42:2f:4e brd ff:ff:ff:ff:ff:ff
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 98:3b:16:e9:79:43 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.82/24 brd 192.168.0.255 scope global dynamic noprefixroute wlan0
       valid_lft 3069218372sec preferred_lft 3069218372sec
    inet6 fe80::40a5:2ad5:cfa4:a2d8/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

4. Locale Configuration

Clearly, from the above login info, our timezone seems to be wrong. Command dpkg-reconfigure tzdata is able to reset our timezone. 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
bananapipro@bananapipro:~$ sudo dpkg-reconfigure tzdata
[sudo] password for bananapipro: 
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = "en_US.UTF-8",
        LC_ALL = (unset),
        LC_MEASUREMENT = "en_CA.UTF-8",
        LC_PAPER = "en_CA.UTF-8",
        LC_MONETARY = "en_CA.UTF-8",
        LC_NAME = "en_CA.UTF-8",
        LC_ADDRESS = "en_CA.UTF-8",
        LC_NUMERIC = "en_CA.UTF-8",
        LC_MESSAGES = "en_US.UTF-8",
        LC_TELEPHONE = "en_CA.UTF-8",
        LC_IDENTIFICATION = "en_CA.UTF-8",
        LC_TIME = "en_CA.UTF-8",
        LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
locale: Cannot set LC_ALL to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_ALL to default locale: No such file or directory

Current default time zone: 'America/Vancouver'
Local time is now:      Tue Sep  4 15:33:36 PDT 2018.
Universal Time is now:  Tue Sep  4 22:33:36 UTC 2018.

During the above process, you'll set the following 2 pages correspondingly:

System Locale America
System Locale Vancouver

If we logout and login Banana Pi Pro again, we'll see the timezone has been successfully reset. 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
$ exit
logout
Connection to 192.168.0.82 closed.
$ ssh bananapipro@192.168.0.82
bananapipro@192.168.0.82's password: 
 ____                                  ____  _   ____            
| __ )  __ _ _ __   __ _ _ __   __ _  |  _ \(_) |  _ \ _ __ ___  
|  _ \ / _` | '_ \ / _` | '_ \ / _` | | |_) | | | |_) | '__/ _ \ 
| |_) | (_| | | | | (_| | | | | (_| | |  __/| | |  __/| | | (_) |
|____/ \__,_|_| |_|\__,_|_| |_|\__,_| |_|   |_| |_|   |_|  \___/ 
                                                                 

Welcome to ARMBIAN 5.59 stable Ubuntu 18.04.1 LTS 4.14.65-sunxi   
System load:   0.16 0.20 0.18   Up time:       53 min
Memory usage:  6 % of 992MB     IP:            192.168.0.82
CPU temp:      46°C           
Usage of /:    3% of 29G    

Last login: Tue Sep  4 14:58:03 2018 from 192.168.0.60

5. Armbian Upgrading

Always these 3 commands: 

1
2
3
bananapipro@bananapipro:~$ sudo apt update
bananapipro@bananapipro:~$ apt list --upgradable
bananapipro@bananapipro:~$ sudo apt upgrade

6. Kernel Doublechecking

Finally, we have the system and kernel doublechecked. 

1
2
3
4
5
6
7
8
9
10
bananapipro@bananapipro:~$ uname -r
4.14.65-sunxi
bananapipro@bananapipro:~$ uname -a
Linux bananapipro 4.14.65-sunxi #68 SMP Tue Aug 21 19:57:06 CEST 2018 armv7l armv7l armv7l GNU/Linux
bananapipro@bananapipro:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:        18.04
Codename:       bionic

Appendix

We can of course follow our previous blog Install Armbian Ubuntu Desktop with the Newest Supported Mainline Linux Kernel onto Orange Pi Plus 2 to upgrade the NEWEST U-Boot and Linux Kernel for Banana Pi Pro, which is NOT going to be discussed here again.

Posted on Edited on In ,

It seems Tensorflow evolves pretty fast. Today we are testing object tracking based on Tensorflow.

1. Environment

1
2
3
4
5
6
7
8
9
10
➜  ~ python
Python 3.6.6 (default, Sep 12 2018, 18:26:19) 
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> tf.__version__
'1.12.0-rc0'
>>> import cv2
>>> cv2.__version__
'3.4.3'

2. Object Tracking

2.1 Concepts

There are several fundamental concepts to be re-emphasized (Here, we took one single concerned object as our example. There might be multiple concerned objects): - detection: You don't know whethere there is a concerned object in the field of view or not, which you will know after the detection. And, if there is such a concerned object in the view, the object location is to be given. - tracking: You know where the concerned object was. Based on the prior knowledge, you are to determine where this object is going to be next? - location: Both detection and tracking are looked on as locating the concerned object. - recognition: Only after the concerned object has been located, more detailed information may be recognized afterwards.

2.2 Testing

Now, let's test out Deep Object Tracking Implementation in Tensorflow.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
➜  tf-adnet-tracking git:(master) ✗ python runner.py by_dataset  --vid-path=./data/freeman1/
VIDEOIO ERROR: V4L: can't open camera by index 0
Test:  0.0
Ratio:  0.0
Frame Rate:  0.0
Width:  0.0
Height:  0.0
Brightness:  0.0
Contrast:  0.0
Saturation:  0.0
Hue:  0.0
Gain:  0.0
Exposure:  0.0
2018-11-04 19:20:54.997017: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:993] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-11-04 19:20:54.997514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: 
name: GeForce GTX 980M major: 5 minor: 2 memoryClockRate(GHz): 1.1265
pciBusID: 0000:01:00.0
totalMemory: 3.94GiB freeMemory: 3.14GiB
2018-11-04 19:20:54.997565: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2018-11-04 19:20:55.222053: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-11-04 19:20:55.222092: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 
2018-11-04 19:20:55.222100: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N 
2018-11-04 19:20:55.222278: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2854 MB memory) -> physical GPU (device: 0, name: GeForce GTX 980M, pci bus id: 0000:01:00.0, compute capability: 5.2)
[2018-11-04 19:20:55,738] [networks] [INFO] all global variables initialized
[2018-11-04 19:20:55,865] [networks] [INFO] conv1/weights:0 : original weights assigned. [0]=[[[-0.08093996 -0.03
[2018-11-04 19:20:55,877] [networks] [INFO] conv1/biases:0 : original weights assigned. [0]=-1.5706328
[2018-11-04 19:20:55,922] [networks] [INFO] conv2/weights:0 : original weights assigned. [0]=[[[-0.00856966 -0.00
[2018-11-04 19:20:55,936] [networks] [INFO] conv2/biases:0 : original weights assigned. [0]=-0.06253582
[2018-11-04 19:20:56,016] [networks] [INFO] conv3/weights:0 : original weights assigned. [0]=[[[ 0.00339902  0.00
[2018-11-04 19:20:56,033] [networks] [INFO] conv3/biases:0 : original weights assigned. [0]=-0.071603894
[2018-11-04 19:20:56,236] [networks] [INFO] fc4/weights:0 : original weights assigned. [0]=[[[ 2.6850728e-03  2
[2018-11-04 19:20:56,254] [networks] [INFO] fc4/biases:0 : original weights assigned. [0]=0.10009623
[2018-11-04 19:20:56,286] [networks] [INFO] fc5/weights:0 : original weights assigned. [0]=[[[-0.01758034  0.01
[2018-11-04 19:20:56,310] [networks] [INFO] fc5/biases:0 : original weights assigned. [0]=0.13894661
[2018-11-04 19:20:56,332] [networks] [INFO] fc6_1/weights:0 : original weights assigned. [0]=[[[ 0.001359    0.01
[2018-11-04 19:20:56,358] [networks] [INFO] fc6_1/biases:0 : original weights assigned. [0]=0.01
[2018-11-04 19:20:56,379] [networks] [INFO] fc6_2/weights:0 : original weights assigned. [0]=[[[-1.23225665e-02  
[2018-11-04 19:20:56,400] [networks] [INFO] fc6_2/biases:0 : original weights assigned. [0]=0.0
[]
[2018-11-04 19:20:56,482] [ADNetRunner] [INFO] ---- start dataset l=326
[2018-11-04 19:20:58,451] [ADNetRunner] [INFO] ADNetRunner.initial_finetune t=1541388056.484
[2018-11-04 19:20:59,194] [ADNetRunner] [DEBUG] redetection success=True
[2018-11-04 19:20:59,487] [ADNetRunner] [DEBUG] finetuned
[2018-11-04 19:21:01,750] [ADNetRunner] [DEBUG] finetuned
[2018-11-04 19:21:03,691] [ADNetRunner] [DEBUG] redetection success=False
[2018-11-04 19:21:03,975] [ADNetRunner] [DEBUG] finetuned
[2018-11-04 19:21:04,878] [ADNetRunner] [DEBUG] finetuned
[2018-11-04 19:21:07,524] [ADNetRunner] [DEBUG] finetuned
[2018-11-04 19:21:08,925] [ADNetRunner] [DEBUG] redetection success=True
[2018-11-04 19:21:09,249] [ADNetRunner] [DEBUG] finetuned
[2018-11-04 19:21:10,825] [ADNetRunner] [DEBUG] finetuned
[2018-11-04 19:21:11,768] [ADNetRunner] [DEBUG] redetection success=True
[2018-11-04 19:21:12,096] [ADNetRunner] [DEBUG] finetuned
[2018-11-04 19:21:14,021] [ADNetRunner] [DEBUG] finetuned
[2018-11-04 19:21:16,867] [ADNetRunner] [DEBUG] finetuned
[2018-11-04 19:21:19,710] [ADNetRunner] [DEBUG] finetuned
[2018-11-04 19:21:22,677] [ADNetRunner] [DEBUG] finetuned
[2018-11-04 19:21:25,711] [ADNetRunner] [DEBUG] finetuned
[2018-11-04 19:21:26,202] [ADNetRunner] [DEBUG] redetection success=True
[2018-11-04 19:21:26,567] [ADNetRunner] [DEBUG] finetuned
[2018-11-04 19:21:26,837] [ADNetRunner] [DEBUG] redetection success=True
[2018-11-04 19:21:27,152] [ADNetRunner] [DEBUG] finetuned
[2018-11-04 19:21:29,387] [ADNetRunner] [DEBUG] finetuned
[2018-11-04 19:21:31,371] [ADNetRunner] [INFO] ----
[2018-11-04 19:21:31,371] [ADNetRunner] [INFO] total: 34.8886
initial_finetune: 1.9675
tracking.do_action: 5.4896
tracking.save_samples.roi: 0.5116
tracking.save_samples.feat: 17.9738
tracking.online_finetune: 6.2583
[2018-11-04 19:21:31,371] [ADNetRunner] [INFO] 9.344 FPS

Face Tracking Based on Tensorflow

0%