With the most inspiring speech by Arnold Schwarzenegger, 2019 arrived… I’ve got something in my mind as well…
Everybody has a dream/goal. NEVER EVER blow it out… Here comes the preface of my PhD’s thesis… It’s been 10 years already… This is NOT ONLY my attitude towards science, BUT ALSO towards those so-called professors(叫兽) and specialists(砖家).
So, today, I’m NOT going to repeat any cliches, BUT happily encouraged by Arnold Schwarzenegger’s speech. We will definitely have a fruitful 2019. Let’s have some pizza.
import cv2 import numpy as np import matplotlib.pyplot as plt import pydicom as pdicom import os import glob import pandas as pd import scipy.ndimage from skimage import measure, morphology from mpl_toolkits.mplot3d.art3d import Poly3DCollection
# Collect all dicom images lstFilesDCM = [] # create an empty list defload_scan2(path): for dirName, subdirList, fileList in os.walk(path): for filename in fileList: if".dcm"in filename.lower(): lstFilesDCM.append(os.path.join(dirName, filename)) return lstFilesDCM
first_patient = load_scan2(INPUT_FOLDER)
# Get ref file print (lstFilesDCM[0]) RefDs = pdicom.read_file(lstFilesDCM[0])
# Load dimensions based on the number of rows, columns and slices (along z axis) ConstPixelDims = (int(RefDs.Rows), int(RefDs.Columns), len(lstFilesDCM))
x = np.arange(0.0, (ConstPixelDims[0]+1)*ConstPixelSpacing[0], ConstPixelSpacing[0]) y = np.arange(0.0, (ConstPixelDims[1]+1)*ConstPixelSpacing[1], ConstPixelSpacing[1]) z = np.arange(0.0, (ConstPixelDims[2]+1)*ConstPixelSpacing[2], ConstPixelSpacing[2])
# loop through all the DICOM files for filenameDCM in lstFilesDCM: # read the file ds = pdicom.read_file(filenameDCM) # store the raw image data ArrayDicom[:, :, lstFilesDCM.index(filenameDCM)] = ds.pixel_array
I did a MRI scan in 2015, in SIAT, for the purpose of research. The scan resolution is NOT very high. Anyway, a bunch of DICOM images can be viewed as follows:
It’s been quite a while without writing anything. Today, we are going to introduce PyFlux for time seriers analysis. Cannonical models are to be directly adopted from PyFlux Documents and tested in this blog.
Get Started
Copy and paste that piece of code from PyFlux Documents with trivial modifications as follows:
import pandas as pd import numpy as np from pandas_datareader.data import DataReader from datetime import datetime import pyflux as pf import matplotlib.pyplot as plt
Index Latent Variable Prior Prior Hyperparameters V.I. Dist Transform ======== ========================= =============== ========================= ========== ========== 0 Constant Normal mu0: 0, sigma0: 3 Normal None 1 AR(1) Normal mu0: 0, sigma0: 0.5 Normal None 2 AR(2) Normal mu0: 0, sigma0: 0.5 Normal None 3 AR(3) Normal mu0: 0, sigma0: 0.5 Normal None 4 AR(4) Normal mu0: 0, sigma0: 0.5 Normal None 5 MA(1) Normal mu0: 0, sigma0: 0.5 Normal None 6 MA(2) Normal mu0: 0, sigma0: 0.5 Normal None 7 MA(3) Normal mu0: 0, sigma0: 0.5 Normal None 8 MA(4) Normal mu0: 0, sigma0: 0.5 Normal None 9 Normal Scale Flat n/a (non-informative) Normal exp
~/.local/lib/python3.6/site-packages/numdifftools/limits.py:126: UserWarning: All-NaN slice encountered warnings.warn(str(msg)) Acceptance rate of Metropolis-Hastings is 0.000125 Acceptance rate of Metropolis-Hastings is 0.00075 Acceptance rate of Metropolis-Hastings is 0.105525 Acceptance rate of Metropolis-Hastings is 0.13335 Acceptance rate of Metropolis-Hastings is 0.1907 Acceptance rate of Metropolis-Hastings is 0.232 Acceptance rate of Metropolis-Hastings is 0.299
Tuning complete! Now sampling. Acceptance rate of Metropolis-Hastings is 0.36655 GARCH(1,1) ======================================================= ================================================== Dependent Variable: JPM Returns Method: Metropolis Hastings Start Date: 2006-06-05 00:00:00 Unnormalized Log Posterior: -2671.5492 End Date: 2016-06-02 00:00:00 AIC: 5351.717896880396 Number of observations: 2517 BIC: 5375.041188860938 ========================================================================================================== Latent Variable Median Mean 95% Credibility Interval ======================================== ================== ================== ========================= Vol Constant 0.0059 0.0057 (0.004 | 0.0076) q(1) 0.0721 0.0728 (0.0556 | 0.0921) p(1) 0.9202 0.9199 (0.9013 | 0.9373) Returns Constant 0.0311 0.0311 (0.0109 | 0.0511) ==========================================================================================================
This blog is just to solve some bugs while using MySQL as a server. Assuming you’ve already installed mysql-server, mysql-server-5.7, and mysql-server-core-5.7.
1. Errors
1.1 Failed to start mysql.service: Unit mysql.service not found.
1 2 3 4 5 6 7
➜ ~ sudo service mysqld start [sudo] password for jiapei: Failed to start mysqld.service: Unit mysqld.service not found. ➜ ~ /etc/init.d/mysql stop [ ok ] Stopping mysql (via systemctl): mysql.service. ➜ ~ /etc/init.d/mysql start [ ok ] Starting mysql (via systemctl): mysql.service.
Conclusion: use /etc/init.d/mysql start instead of service mysqld start.
1.2 ERROR 1045 (28000): Access denied for user ‘root‘@’localhost’ (using password: YES)
1 2 3
➜ ~ mysql -u root -p Enter password: ERROR 1698 (28000): Access denied for user 'root'@'localhost'
We FIRST list all existing MySQL processes and kill them all.
1 2 3
➜ ~ ps aux | grep mysql ... ➜ ~ sudokill PID
Util now, there should be NO MySQL process running.
1.3 ERROR 2002 (HY000): Can’t connect to local MySQL server through socket ‘/var/run/mysqld/mysqld.sock’
Since there is NO MySQL process running, of course we cannot connect to MySQL server.
1 2 3 4 5 6 7
➜ ~ mysql -u root -p Enter password: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) ➜ ~ /etc/init.d/mysql start [....] Starting mysql (via systemctl): mysql.serviceJob for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details. failed!
Then, we will have to have these 3 packages reinstalled: mysql-server, mysql-server-5.7, and mysql-server-core-5.7.
/var/run/mysqld/mysqld.sock is NOW back, and MySQL seems to run automatically.
1 2 3 4 5
➜ ~ ls -ls /var/run/mysqld/mysqld.sock 0 srwxrwxrwx 1 mysql mysql 0 Nov 30 03:08 /var/run/mysqld/mysqld.sock ➜ ~ ps aux | grep mysql mysql 13785 0.1 0.3 1323300 172680 ? Sl 03:08 0:00 /usr/sbin/mysqld --daemonize --pid-file=/run/mysqld/mysqld.pid jiapei 14453 0.0 0.0 21556 2560 pts/0 R+ 03:16 0:00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn mysql
Then, use chown and chmod to set up the owners and permissions suitably as follows.
1 2 3 4 5 6 7 8 9 10 11 12 13
➜ run pwd /var/run ➜ run sudochown mysql:mysql -R mysql ➜ run sudochmod 755 -R mysql ➜ run ls -lsd mysql 0 drwxr-xr-x 2 mysql mysql 100 Nov 30 02:38 mysqld ➜ run cd ../lib ➜ lib pwd /var/lib ➜ lib sudochown mysql:mysql -R mysql ➜ lib sudochmod 755 -R mysql ➜ lib ls -lsd mysql 4 drwxr-xr-x 6 mysql mysql 4096 Nov 30 02:38 mysql
1.4 ERROR 1698 (28000): Access denied for user ‘root‘@’localhost’
➜ ~ mysql -u root -p Enter password: ERROR 1698 (28000): Access denied for user 'root'@'localhost' ➜ ~ sudo mysql -u root Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 3 Server version: 5.7.24-0ubuntu0.18.04.1 (Ubuntu)
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.
Type 'help;' or '\h'forhelp. Type '\c' to clear the current input statement.
mysql> USE lvrsql; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A
Database changed mysql> CREATE USER 'YOUR_SYSTEM_USER'@'localhost' IDENTIFIED BY 'YOUR_PASSWORD'; mysql> GRANT ALL PRIVILEGES ON *.* TO 'YOUR_SYSTEM_USER'@'localhost'; mysql> FLUSH PRIVILEGES; mysql> exit;
➜ ~ service mysql restart ➜ ~
1.5 [ERROR] InnoDB: Unable to lock ./ibdata1 (Additional)
If you meet the above ERROR message, what you can do is to restart MySQL by:
1 2 3
➜ ~ /etc/init.d/mysql restart [ ok ] Restarting mysql (via systemctl): mysql.service. ➜ ~
var view = new SceneView({ map: map, container: "viewDiv" });
var osmLayer = new OpenStreetMapLayer(); map.add(osmLayer); }); </script> </head>
<body> <div id="viewDiv"></div> </body>
</html>
Please click on the following image and have a try.
ArcGIS is like the CROWN of GIS. Its SDK/API seems NOT fully open source but commercial. Therefore, I’d love to breifly test out some other open source.
“OpenGlobus is a javascript library designed to display interactive 3d maps and planets with map tiles, imagery and vector data, markers and 3d objects. It uses the WebGL technology, open source and completely free.” (Cited from OpenGlobus Github). Let’s just take the very FIRST example Og - examples: Attribution as our example. Code is cited as follows:
OpenStreetMap may be covered by different tiles in order to show different maps. For instance, Google map vs Google satellite map. Let’s take a look at Home Page of WebGLEarth, you’ll notice MapTiler at the left bottom corner. Here, we summarize 3 widely used tiles in the following:
Besides tiles, there might be several layers (including one layer of tile) upon OpenStreetMap, here comes OpenLayers: “OpenLayers makes it easy to put a dynamic map in any web page. It can display map tiles, vector data and markers loaded from any source.” (cited from https://openlayers.org/)
Yesterday, we briefly introduced Quantopian and how to ingest Quandl WIKI Bundle data. Today, we are going to briefly summarize the financial data source and how to retrieve Google and Yahoo finance data using pandas_datareader.
# Define the instruments to download. We would like to see Apple, Microsoft and the S&P500 index. tickers = ['AAPL', 'MSFT', '^GSPC']
# We would like all available data from 01/01/2017 until 12/31/2017. start_date = '2017-01-01' end_date = '2017-12-31' df = data.DataReader(tickers, data_source='google', start=start_date, end=end_date)
Exception has occurred: pandas_datareader.exceptions.ImmediateDeprecationError
Google finance has been immediately deprecated due to large breaks in the API without the introduction of a stable replacement. Pull Requests to re-enable these data connectors are welcome. See https://github.com/pydata/pandas-datareader/issues
File "....../test.py", line 16, in
df = data.DataReader(tickers, data_source='google', start=start_date, end=end_date)
On the other hand, from the following 3 finance data resource, the following printed tables and generated pictures demonstrate that stock data can be successfully loaded.
# Define the instruments to download. We would like to see Apple, Microsoft and the S&P500 index. tickers = ['AAPL', 'MSFT', '^GSPC']
# We would like all available data from 01/01/2017 until 12/31/2017. start_date = '2017-01-01' end_date = '2017-12-31' df = data.DataReader(tickers, data_source='yahoo', start=start_date, end=end_date)
# Define the instruments to download. We would like to see Apple, Microsoft and the S&P500 index. tickers = ['AAPL', 'MSFT', 'SPY']
# We would like all available data from 01/01/2017 until 12/31/2017. start_date = '2017-01-01' end_date = '2017-12-31' df = data.DataReader(tickers, data_source="iex", start=start_date, end=end_date)
# We would like all available data from 01/01/2017 until 12/31/2017. start_date = '2017-01-01' end_date = '2017-12-31' df = data.DataReader('AAPL', data_source="quandl", start=start_date, end=end_date, access_key="your-quandl-api-key") # df = data.DataReader('MSFT', data_source="quandl", start=start_date, end=end_date, access_key="your-quandl-api-key")
Today, let’s have some fun of AI in finance. We are going to use a tool named Quantopian. As far as I know, functional programming language Haskell has been widely used in quantitative analysis. That’s possibly why a dependent package of Quantopian, namely pandoc, requires The Haskell Tool Stack for its installation. Therefore, Haskell and pandoc need to be installed before Quantopian installation.
As known, Quantopian is a Python collections of various quantitative analysis algorithms. However, where can we obtain the stock data for our testing? Either the real-time data, or the historical data will do.
➜ quantopian QUANDL_API_KEY=<yourkey> zipline ingest -b quandl [2018-11-07 05:18:37.895463] INFO: zipline.data.bundles.quandl: Downloading WIKI metadata. Downloading WIKI Prices table from Quandl [####################################] 100% [2018-11-07 05:18:54.369791] INFO: zipline.data.bundles.quandl: Parsing raw data. [2018-11-07 05:19:24.118879] INFO: zipline.data.bundles.quandl: Generating asset metadata. Merging daily equity files: [-----------------------------#------] 1731~/.local/lib/python3.6/site-packages/zipline/data/us_equity_pricing.py:417: UserWarning: Ignoring 1 values because they are out of bounds for uint32: open high low close volume ex_dividend split_ratio 2011-04-11 1.79 1.84 1.55 1.7 6.674913e+09 0.0 1.0 winsorise_uint32(raw_data, invalid_data_behavior, 'volume', *OHLC) Merging daily equity files: [####################################] [2018-11-07 05:21:34.367628] INFO: zipline.data.bundles.quandl: Parsing split data. [2018-11-07 05:21:34.572459] INFO: zipline.data.bundles.quandl: Parsing dividend data. ~/.local/lib/python3.6/site-packages/zipline/data/us_equity_pricing.py:931: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`. if not issubdtype(actual, expected): ➜ quantopian
Hi, everyone… I’ve been so busy and haven’t written anything for a bit. My previous boss at Lenovo is now in Vancouver for annual executive meeting. She brought me a HuaweiHiSiliconHikey 970.
Several things before testing:
Lenovo and Huawei are two BIGGEST and possibly BEST IT companies from China.
HiSilicon is a chip design and manufacturing company spinned off from Huawei.
96Boards hosts various SBCs (Single Board Computers), including Hikey 970.
A lot more resource can be found on LeMaker, who has designed several low-cost single board computers (SBCs).
However, it seems Linux support for uPyCraft is NOT very good. My working environment:
1 2 3 4 5 6 7 8 9 10 11 12 13
$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.1 LTS Release: 18.04 Codename: bionic $ uname -a Linux jiapei-GT72-6QE 4.18.7-041807-generic #201809090930 SMP Sun Sep 9 09:33:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux $ python --version Python 3.6.5 $ qmake --version QMake version 3.1 Using Qt version 5.11.1 in /opt/Qt/Current/gcc_64/lib
PART A: Test Built uPyCraft
1. Try the Executable
In fact, there is one executable Linux version for uPyCraft. However, to run it after downloading, I obtained the following ERROR messages:
1 2 3 4 5 6
$ ./uPyCraft_linux_V1.0 Traceback (most recent call last): File "uPyCraft.py", line 2, in <module> File "/usr/local/lib/python3.5/dist-packages/PyInstaller/loader/pyimod03_importers.py", line 714, in load_module ImportError: /tmp/_MEIoq8Xi8/libz.so.1: version `ZLIB_1.2.9' not found (required by /usr/lib/x86_64-linux-gnu/libpng16.so.16) [324] Failed to execute script uPyCraft