Amund Tveit's Blog

Artificial Intelligence, AutoML Tables, BigQuery, BigQuery ML

Predicting Bitcoin price with BigQuery ML

June 1, 2019 No Comments

In the previous blog I described how to fetch a bitcoin data set and prepare it for AutoML Tables to create a (highly accurate) machine learning model to predict Bitcoin price.

This blog post describes a complementary (or alternative) approach – how to use BigQuery ML to create a (simpler) regression model using a SQL like syntax. The advantage of this approach is that training time is very rapid and that it is very easy to use for batch-based predictions also using SQL like syntax. The blog assumes that you have a table with data in BigQuery (see previous blog post for how to do that)

Test the query that is the core part of the model – training data

Most of the data needed is already in the table, but in addition need to create the label to predict using LEAD() method and since BigQuery ML requires non-NULL data (e.g. not having NaN values) that is set to 0 for the label with IFNULL.

SELECT
IFNULL(lead(close) OVER (ORDER BY formatted_timestamp),0) AS label,
*
FROM
`bitcoindata.bcdata`

SELECT

IFNULL(lead(close) OVER (ORDER BY formatted_timestamp),0) AS label,

FROM

`bitcoindata.bcdata`

**Figure 1 – BigQuery SQL query generating training data**

2. Generate the regression model for predicting Bitcoin price

#standardSQL
CREATE MODEL `bitcoindata.bitcoinlinpred`
OPTIONS(model_type='linear_reg') AS
SELECT
IFNULL(lead(close) OVER (ORDER BY formatted_timestamp), 0) AS label,
*
FROM
`bitcoindata.bcdata`

#standardSQL

CREATE MODEL `bitcoindata.bitcoinlinpred`

OPTIONS(model_type='linear_reg') AS

SELECT

IFNULL(lead(close) OVER (ORDER BY formatted_timestamp), 0) AS label,

FROM

`bitcoindata.bcdata`

3. Evaluate model – get predicted_label as new value

SELECT ABS(predicted_label-label) as delta FROM ML.PREDICT(MODEL `predicting.bitcoindata.bitcoinlinpred`, (
SELECT
IFNULL(lead(close) OVER (ORDER BY formatted_timestamp),0) AS label,
*
FROM
`bitcoindata.bcdata`
))

SELECT ABS(predicted_label-label) as delta FROM ML.PREDICT(MODEL `predicting.bitcoindata.bitcoinlinpred`, (

SELECT

IFNULL(lead(close) OVER (ORDER BY formatted_timestamp),0) AS label,

FROM

`bitcoindata.bcdata`

))

Conclusion

Have shown how to use BigQuery ML regression on a BitCoin dataset to predict Bitcoin price, given how easy this is to use even at large scale (e.g. several hundred billion rows) this can be good start when doing predictions in on tabular data.

Best regards,

Amund Tveit

Artificial Intelligence, AutoML Tables, Finance

Predicting Bitcoin Price with AutoML Tables

May 30, 2019 No Comments

Applying Artificial Intelligence (AI) frequently require surprising amount of (tedious) manual work. Tools for automation can make AI available to more people and to rapidly solve many more important challenges. This blog posts tests such a tool – AutoML Tables.

**Figure 1 – Bitcoin price prediction – is it going up, down or sideways?**

This blog post generates a data set from an API and applies automated AI – AutoML Tables for regression to predict numbers – in this case Bitcoin closing price next hour based on data from the current hour.

1. Introduction

AutoML Tables – can used on tabular data (e.g. from databases or spreadsheets) for either: classification (e.g. classify whether it is an Valyrian steel Sword or not – as shown in Figure 2a) or regression (predict a particular number, e.g. reach of Scorpion canon aiming at dragons as shown in the top figure 2b)

**Figure 2a- Classification Example – Is Aria’s Needle Sword of Valyrian Steel?**

**Figure 2b – Regression Example – reach of Euron’s arrow**

2. Choosing and Building Dataset

However, I didn’t find data sets that I could use for Valerian steel classification or Scorpion arrow reach regression (let me know if such data exists), but instead found a free API to get bitcoin related data over time instead and since I assume Bitcoin is completely unrelated to Valyrian steel and Scorpion (however, I might be wrong about that given that Valyrian steel furnaces might compete with Bitcoin about energy – perhaps a potential confounding variable to explain a potential relationship between prices of Valyrian swords and Bitcoin?) .

Scientific selection of Bitcoin data API: Since I am not an expert in cryptocurrency I just searched for free bitcoin api (or something in that direction) and found/selected cryptocompare.com

2.1 Python code to fetch and prepare API data

Materials & Methods

I used a colab (colab.research.google.com) to fetch and prepare API data, in combination with AutoML web UI and a few Google Cloud command line commands (gsutil and gcloud methods). Also used Bigquery for storing results and AutoML stored some output related to evaluation in Bigquery

Imports and authentication (Google Cloud)

CRYPTOCOMPARE_API_KEY = ""  # get your own on cryptocompare.com
import requests # for API request to cryptocompare.com
from datetime import datetime as dt
from google.colab import auth, drive 
import json 
# storing Bitcoin API data in bigquery
from google.cloud import bigquery
from google.api_core.exceptions import BadRequest
auth.authenticate_user() # authentication on Google Cloud

CRYPTOCOMPARE_API_KEY = "" # get your own on cryptocompare.com

import requests # for API request to cryptocompare.com

from datetime import datetime as dt

from google.colab import auth, drive

import json

# storing Bitcoin API data in bigquery

from google.cloud import bigquery

from google.api_core.exceptions import BadRequest

auth.authenticate_user() # authentication on Google Cloud

Method to fetch Bitcoin related trade data

def apicall_prices(crypto_currency="BTC", currency="USD", 
per_api_call_limit=1, to_timestamp=None, 
headers={'authorization':'Apikey %s' % (CRYPTOCOMPARE_API_KEY)}):
params={'fsym': crypto_currency,
'tsym': currency,
'limit': str(per_api_call_limit)}
if to_timestamp is not None:
params["toTs"] = to_timestamp
print("price api", params)
response = requests.get(
'https://min-api.cryptocompare.com/data/histohour',
params=params,
headers=headers,
)
return response

def apicall_prices(crypto_currency="BTC", currency="USD",

per_api_call_limit=1, to_timestamp=None,

headers={'authorization':'Apikey %s' % (CRYPTOCOMPARE_API_KEY)}):

params={'fsym': crypto_currency,

'tsym': currency,

'limit': str(per_api_call_limit)}

if to_timestamp is not None:

params["toTs"] = to_timestamp

print("price api", params)

response = requests.get(

'https://min-api.cryptocompare.com/data/histohour',

params=params,

headers=headers,

)

return response

Method to fetch Bitcoin related social & activity data

(code duplication isn’t wrong – or is it? – leave refactoring of this and previous method as an exercise for the reader)

def apicall_socialsignals(crypto_currency_coin_id=7605, 
per_api_call_limit=1, 
to_timestamp=None,
headers={'authorization':'Apikey %s' % (CRYPTOCOMPARE_API_KEY)}):
params={'coinId': crypto_currency_coin_id, #coinId=7605
'limit': str(per_api_call_limit)}
if to_timestamp is not None:
params["toTs"] = to_timestamp
print("social api", params)
response = requests.get(
'https://min-api.cryptocompare.com/data/social/coin/histo/hour',
params=params,
headers=headers,
)
return response

def apicall_socialsignals(crypto_currency_coin_id=7605,

per_api_call_limit=1,

to_timestamp=None,

headers={'authorization':'Apikey %s' % (CRYPTOCOMPARE_API_KEY)}):

params={'coinId': crypto_currency_coin_id, #coinId=7605

'limit': str(per_api_call_limit)}

if to_timestamp is not None:

params["toTs"] = to_timestamp

print("social api", params)

response = requests.get(

'https://min-api.cryptocompare.com/data/social/coin/histo/hour',

params=params,

headers=headers,

)

return response

Method to combine the 2 types of API data

def join_pricing_and_social_signals(json_price_response, 
json_social_signal_response):
price_data = json_price_response["Data"]
social_signal_data = json_social_signal_response["Data"]
joined_on_timestamp = {}
for item in price_data:
timestamp = item["time"]
if not timestamp in joined_on_timestamp:
joined_on_timestamp[timestamp] = item
#print(joined_on_timestamp.keys())
for item in social_signal_data:
timestamp = item["time"]
if not timestamp in joined_on_timestamp:
print("SHOULDN'T OCCUR!", timestamp)
#print(item)
else:
tmp = joined_on_timestamp[timestamp].copy()
formatted_timestamp = dt.utcfromtimestamp(timestamp).strftime("%Y-%m-%d %H:%M:%S")
item["formatted_timestamp"] = formatted_timestamp
joined_on_timestamp[timestamp] = {**tmp, **item}
return joined_on_timestamp

def join_pricing_and_social_signals(json_price_response,

json_social_signal_response):

price_data = json_price_response["Data"]

social_signal_data = json_social_signal_response["Data"]

joined_on_timestamp = {}

for item in price_data:

timestamp = item["time"]

if not timestamp in joined_on_timestamp:

joined_on_timestamp[timestamp] = item

#print(joined_on_timestamp.keys())

for item in social_signal_data:

timestamp = item["time"]

if not timestamp in joined_on_timestamp:

print("SHOULDN'T OCCUR!", timestamp)

#print(item)

else:

tmp = joined_on_timestamp[timestamp].copy()

formatted_timestamp = dt.utcfromtimestamp(timestamp).strftime("%Y-%m-%d %H:%M:%S")

item["formatted_timestamp"] = formatted_timestamp

joined_on_timestamp[timestamp] = {**tmp, **item}

return joined_on_timestamp

Method for fetching and preprocessing data from API

def fetch_and_preprocess(min_num_results=1,
per_api_call_limit=1, 
crypto_currency="BTC", 
currency="USD"):
num_results = 0
to_timestamp = None 
json_results = {}
while num_results &lt;= min_num_results:
print("num results, min_num_results", num_results, min_num_results)
price_response = apicall_prices(crypto_currency=crypto_currency, currency=currency, 
per_api_call_limit=per_api_call_limit, 
to_timestamp=to_timestamp)
social_signal_response = apicall_socialsignals(per_api_call_limit=per_api_call_limit,
to_timestamp=to_timestamp)
json_price_response = price_response.json()
num_items = len(json_price_response["Data"])
num_results += num_items
to_timestamp = int(json_price_response['TimeFrom'])
formatted_timestamp = dt.utcfromtimestamp(to_timestamp).strftime("%Y/%m/%d %H:%M:%S")
json_price_response["formatted_timestamp"] = formatted_timestamp
json_social_signal_response = social_signal_response.json()
json_social_signal_response["formatted_timestamp"] = formatted_timestamp
combined_json = join_pricing_and_social_signals(json_price_response, 
json_social_signal_response)
for key in combined_json:
json_results[key] = combined_json[key]
return json_results

def fetch_and_preprocess(min_num_results=1,

per_api_call_limit=1,

crypto_currency="BTC",

currency="USD"):

num_results = 0

to_timestamp = None

json_results = {}

while num_results <= min_num_results:

print("num results, min_num_results", num_results, min_num_results)

price_response = apicall_prices(crypto_currency=crypto_currency, currency=currency,

per_api_call_limit=per_api_call_limit,

to_timestamp=to_timestamp)

social_signal_response = apicall_socialsignals(per_api_call_limit=per_api_call_limit,

to_timestamp=to_timestamp)

json_price_response = price_response.json()

num_items = len(json_price_response["Data"])

num_results += num_items

to_timestamp = int(json_price_response['TimeFrom'])

formatted_timestamp = dt.utcfromtimestamp(to_timestamp).strftime("%Y/%m/%d %H:%M:%S")

json_price_response["formatted_timestamp"] = formatted_timestamp

json_social_signal_response = social_signal_response.json()

json_social_signal_response["formatted_timestamp"] = formatted_timestamp

combined_json = join_pricing_and_social_signals(json_price_response,

json_social_signal_response)

for key in combined_json:

json_results[key] = combined_json[key]

return json_results

2.2 Python Code to prepare Bitcoin data for BigQuery and AutoML Tables

Actually fetch some results (16000 hours = 1.82 years)

json_results = fetch_and_preprocess(16000, per_api_call_limit=2000)

1	json_results = fetch_and_preprocess(16000, per_api_call_limit=2000)

Write as 1 json per line to a file

fh = open("bitcoindata.json","w")
for line in sorted(json_results):
fh.write(json.dumps(json_results[line]) + "\n")
fh.close()

fh = open("bitcoindata.json","w")

for line in sorted(json_results):

fh.write(json.dumps(json_results[line]) + "\n")

fh.close()

Set active Google Cloud project (! in front in colab means shell command line command)

project_id="" # the project id you have in google cloud (Web UI)
!gcloud config set project {project_id}

1 2	project_id="" # the project id you have in google cloud (Web UI) !gcloud config set project {project_id}

Creating a Google Cloud storage bucket to store data

bucket_name = "" # the google cloud bucket name you want
!gsutil mb -c regional -l us-central1  -p {project_id} gs://{bucket_name}

1 2	bucket_name = "" # the google cloud bucket name you want !gsutil mb -c regional -l us-central1 -p {project_id} gs://{bucket_name}

Create a Bigquery schema based on the API data fetched

Note: bigquery-schema-generator was a nice tool, but had to change INTEGER to FLOAT in the generated schema in addition to prepare data (ref perl oneliner)

!pip install bigquery-schema-generator
!generate-schema bitcoindata.json > bitcoindata.schema
!perl -npi -e 's/INTEGER/FLOAT/gi' *.schema
!gsutil cp *.schema gs://{bucket_name}

!pip install bigquery-schema-generator

!generate-schema bitcoindata.json > bitcoindata.schema

!perl -npi -e 's/INTEGER/FLOAT/gi' *.schema

!gsutil cp *.schema gs://{bucket_name}

Generate (or fetch existing) Bigquery data set & create Bigquery Table

Note: I used the project id ‘predicting’, replace with your – ref: bq command further down.

client = bigquery.Client(project=project_id) # 
dataset_id = 'bitcoindata' # or whatever you choose
dataset_ref = client.dataset(dataset_id)
try:
client.get_dataset(dataset_ref)
except:
dataset = bigquery.Dataset(dataset_ref)
dataset = client.create_dataset(dataset)
print('Dataset {} created.'.format(dataset.dataset_id))
dataset_ref = client.dataset(dataset_id)
print(dataset_ref)

client = bigquery.Client(project=project_id) #

dataset_id = 'bitcoindata' # or whatever you choose

dataset_ref = client.dataset(dataset_id)

try:

client.get_dataset(dataset_ref)

except:

dataset = bigquery.Dataset(dataset_ref)

dataset = client.create_dataset(dataset)

print('Dataset {} created.'.format(dataset.dataset_id))

dataset_ref = client.dataset(dataset_id)

print(dataset_ref)

!bq mk --table --expiration 3600 --description "This is my table" --label predicting:bitcoindata bitcoindata.bctrainingdata bitcoindata.schema

1	!bq mk --table --expiration 3600 --description "This is my table" --label predicting:bitcoindata bitcoindata.bctrainingdata bitcoindata.schema

Load API data into (new) Bigquery Table

!bq load  --source_format NEWLINE_DELIMITED_JSON \
--ignore_unknown_values \
bitcoindata.bctrainingdata \
gs://predicting/bitcoindata.json \
bitcoindata.schema

!bq load --source_format NEWLINE_DELIMITED_JSON \

--ignore_unknown_values \

bitcoindata.bctrainingdata \

gs://predicting/bitcoindata.json \

bitcoindata.schema

Check that the table exists and query it

!bq show predicting:bitcoindata

1	!bq show predicting:bitcoindata

%%bigquery --project predicting bitcoindata
select * from bitcoindata.bctrainingdata

1 2	%%bigquery --project predicting bitcoindata select * from bitcoindata.bctrainingdata

**Figure 3 – output from select query towards Bitcoin data in Bigquery**

We have input (x) features, but not a feature (y) to predict(!)

Create a column to predict can be done by creating a new column that is time shifted, e.g. for a time t=0 there is a particular row that require a t=1 feature to train – the feature we want to predict is the Bitcoin close price next hour (e.g. not exactly quant/high-frequency trading – but a more soothing once-per-hour experience, if it works out ok it can be automated – for the risk taking?). This can be generated either in Bigquery with select and LEAD() method or with a Python Pandas Dataframe shift – showing both approached underneath.

%%bigquery --project predicting bitcoindata
select lead(close) OVER (ORDER BY formatted_timestamp) as NEXTCLOSE, * from bitcoindata.bctrainingdata ORDER by formatted_timestamp desc

1 2	%%bigquery --project predicting bitcoindata select lead(close) OVER (ORDER BY formatted_timestamp) as NEXTCLOSE, * from bitcoindata.bctrainingdata ORDER by formatted_timestamp desc

df = df.assign(NEXTCLOSE=tst.loc[:,'close'].shift(1)) 
# ADDING TIMESTAMP CAN BE USEFUL FOR DEBUGGING, TO MAKE SURE YOU ARE SHIFTING IN THE RIGHT DIRECTION!
df = df.assign(NEXT_TIMESTAMP=tst.loc[:,'formatted_timestamp'].shift(1))

df = df.assign(NEXTCLOSE=tst.loc[:,'close'].shift(1))

# ADDING TIMESTAMP CAN BE USEFUL FOR DEBUGGING, TO MAKE SURE YOU ARE SHIFTING IN THE RIGHT DIRECTION!

df = df.assign(NEXT_TIMESTAMP=tst.loc[:,'formatted_timestamp'].shift(1))

Prepare final data with NEXTCLOSE column (as csv) for AutoML and copy to Google Cloud bucket

csv_data = bitcoindata_w_prediction_column.to_csv(index=False)
fh = open("automl_import_data.csv","w")
fh.write(data)
fh.close()

csv_data = bitcoindata_w_prediction_column.to_csv(index=False)

fh = open("automl_import_data.csv","w")

fh.write(data)

fh.close()

!gsutil cp bq_import_data.csv gs://{bucket_name}

1	!gsutil cp bq_import_data.csv gs://{bucket_name}

3. AutoML prediction

Now the data is ready for AutoML (Note that the step with Bigquery could have been avoided in this case, but could also be another direction since AutoML can import directly from Bigquery). Underneath you can see an example of a created dataset in AutoML Console.

Figure 4 – AutoML Console – with an example data set named bitcoindata

Creating a new dataset

Figure 5 – create new AutoML Tables dataset

Importing data from Google Cloud Bucket

Figure 6 – Import data to AutoML from Google Cloud Storage

Set target variable (NEXTCLOSE) and look at statistics of features

Figure 8 – select target column and data split (train/validation/testing)

Figure 9 – inspect correlation with target variable

Train Model with AutoML

Figure 10 – select budget for resources to generate model

Look at core metrics regarding accuracy

Figure 11 – Metrics from training (MAE, RMSE,R^2, MAPE) and important features

Deploy and Use the AutoML model – Batch Mode

Conclusion

Have shown an example of using AutoML – the main part was about getting data from an API and preparing to use it (section 2), and using it in AutoML to train a model and look into evaluation. This model aims to predict the next hour bitcoin closing based on data from the current hour, but can probably be extended in several ways – how would you extend it?

Best regards,

Amund Tveit

DISCLAIMER: this blog only represent my PERSONAL opinions and views

Uncategorized

Poll – Software Engineering from Academic to Craft?

March 16, 2019 No Comments

My impression is that the software engineer profession is slowly transitioning from academic to craft (closer to being e.g a carpenter or electrician) perhaps due to changes in online learning opportunities, cloud computing and industry needs. What do you think?

— Amund Tveit (@atveit) March 16, 2019

AI, Deep Learning

Thoughts on AI replacing coders by 2040

December 12, 2017 No Comments

This blog post looks into which methods and technologies that can potentially lead to the replacement of coders in the future, some are of futuristic nature but some are more “low-hanging” wrt automation of (parts of) coding.

The background for this blog post is that researchers Jay Jay Billings, Alexander J. McCaskey, Geoffroy Vallee, and Greg Watson at Oak Ridge Laboratory wrote a paper: Will humans even write code in 2040 and what would that mean for extreme heterogeneity in computing?

Also related to this is (Tesla AI director) Andrej Karpathy’s article: Software 2.0 where he looks back at how AI (primarily Deep Learning) has replaced custom methods for e.g. image and speech recognition, machine translation (++) and generalizes how Deep Learning can further replace a lot of other software in years to come (note: examplified by Google’s recent paper The Case for Learned Index Structures)

1. FACT: Programming environments (IDEs) have barely changed the last 30 years

One of primary purposes of programming is to provide efficient automation, however programming itself is still a highly manual and labour intensive process – except for refactoring the difference between modern IDEs compared to e.g. Turbo Pascal in 1989 is surprisingly small? (Turbo Pascal came out more than 30 years ago and improved gradually towards 1989)

Turbo Pascal 5.5 in 1989	Eclipse in 2017

2. FACT: For (close to) all FUNCTIONS written there exists one or several TESTS for it

For any method already written (or to be written) in any (of the most popular) languages currently used in programming there already exists a test for it – in the same language or in a similar language (e.g. a C# test for a Java function). The obvious (big) data source for this is all private and public repositories in Github (100M pull requests merged so far in 2017)

So why are most developers still writing unit tests instead of having an IDE/service find and translate the appropriate tests to the functions they write? (e.g. something along the lines of IntelliTest)

3. FACT: as 2 – For (close to) all TESTS there exists a (set of ) FUNCTIONs they test

Assuming e.g. with Test Driven Development (TDD) – where you are roughly writing the (test of the new) API first – and then alternating between creating in the code to (just enough) fulfill the API.

This seems like it has 2 potential ways of being increasingly automated – on the function writing part.

Search for a set of function that matches a set of tests – instead of writing the functions just write the tests
Automatically generate the code (fragments) to fulfill the test, this can potentially be done in many ways, e.g.
1. 1. Brute force with a lot of computers (e.g. a few thousand CPUs in the cloud should be more than capable of quickly generating and selecting the best of maybe up to 30-50 increments needed per test writing iteration, this resource could be shared by a large set of programmers). See also the science of brute force.
  2. Using sophisticated AI methods with GPUs – e.g. Program Synthesis with Deep Learning and earlier AI methods such as variants of John Koza’s Genetic Programming.
  3. Quantum Computer – e.g. Logic Synthesis with Quantum Computing (see also Quantum Development Kit)

4. FACT: (Many?) Data Structures can now be replaced by AI (e.g. Bloom Filters & B-Trees)

Google – with Tim Kraska, Alex Beutel, Ed H. Chi, Jeffrey Dean, Neoklis Polyzotis published an interesting paper: The Case for Learned Index Structures where they showed that traditional and popular data structures such as B-Tree, Hash Index and Bloom Filter can be with advantage be replaced by AI Trained Index Structures. This is from my perspective pretty groundbreaking research and will be interesting to see what the use cases can be towards program synthesis (e.g for other categories of data structures and also logic operating on data structures). Key result figures from their paper:

5. FACT: Formal Methods are starting to work and can be used to support automation of code

Amazon has used formal methods since 2011 for Amazon Web Services, e.g. to ensure the quality of AWS S3 – see Use of Formal Methods at Amazon for an overview. Facebook is using formal methods for mobile application quality assurance, see Moving Fast with Software Verification. Formal methods has also been used to validate various blockchain technologies (potentially Bitcoin), see e.g. A Temporal Blockchain: A Formal Analysis, Blockchain Protocol Analysis and Security Engineering 2017, Validation of Decentralised Smart Contracts Through Game Theory and Formal Methods

A few years back I participated in a research project – Envisage (Engineering Virtualized Services) – where formal methods were used to prove that the TimSort algorithm was buggy in a lot of languages (See Hacker News story below). Formal methods can potentially be used together with code generating methods to ensure that what is generated is correct.

Conclusion

Have presented various technologies that might play a role in automating coding forwards. A potential interesting direction on the generative side for coding is to use sequence to sequence deep learning in combination with GAN for synthesis, see e.g. Text2Action: Generative Adversarial Synthesis from Language to Action – for program synthesis this looks like a

Best regards,

Amund Tveit

AI, Artificial Intelligence, Deep Learning, GAN

New Publications in Deep Learning Publication Navigator

October 8, 2017 No Comments

Long overdue update of new publications in Deep Learning Publication Navigator (ai.amundtveit.com) – for now the easiest way to discover new publications is probably to convert screenshots (number of papers) per category in the before and after update screenshots below.

Examples of keywords (from publication title) with (several) new Deep Learning publications are:

3D
Acoustic
Active learning
Adaptive
Adversarial (123 new papers since last update, due to significant activity in GAN Research)
Alzheimer’s (22 new papers related to a disease that cost more than a quarter trillion US$ annually to treat in the USA)
Anomaly detection
Autoencoders
Bayesian
Biomedical
Chinese
Clinical
Collaborative filtering (e.g. for recommender systems)
Dataset
EEG (electric brain signals)
Ensemble
+++++ (many more!)

If you have feature ideas or other requests for Deep Learning Publication Navigator, feel free to reach out.

Best regards,

Amund Tveit

After update (with new papers):

Before update (without new papers):

API, AWS, AWS API Gateway, AWS Lambda, Azure, Cloud Computing, FaaS, Functions as a Service, Google Cloud Functions, IBM Bluemix Openwhisk, Microsoft Azure Functions, RPC, Serverless, Thrift

Serverless Thrift APIs in Python on AWS Lambda

September 16, 2017 No Comments

I recently wrote a blog post called Serverless Thrift APIs in Python with AWS Lambda on the Zedge corporate blog, underneath is a reposting of it:

This blog post shows a basic example of a Serverless Thrift API with Python for AWS Lambda and AWS API Gateway.

1. Serverless Computing for Thrift APIs?

Serverless computing – also called Cloud Functions or Functions as a Service (FaaS) – is an interesting type of cloud service due to its simplicity. An interpretation of serverless computing is that you (with relatively low effort):

Deploy only the function needing to do the work
Only pay per request to the function
1. With the notable exception of other cloud resources used, e.g. storage
Get some security setup automation/support (e.g. SSL and API keys)
Get support for request throttling (e.g. QPS) and quotas (e.g. per month)
Get (reasonably) low latency – when the serverless function is kept warm
Get support for easily setting up caching
Get support for setting up custom domain name
Lower direct (cloud costs) and indirect (management) costs?

These are characteristics that in my mind make Serverless computing an interesting infrastructure to develop and deploy Thrift APIs (or other types of APIs) for.
Perhaps over time even Serverless will be preferred over (more complex) container (Kubernetes/Docker) or virtual machine based (IaaS) or PaaS solutions for APIs?

2. Example Cloud Vendors providing Serverless Services

AWS Lambda in combination with AWS API Gateway
Google Cloud Functions
IBM Bluemix Openwhisk (Apache Openwhisk)
Microsoft Azure Functions

Since Python is a key language in my team, for this initial test I choose the AWS option also since I am most familiar with AWS and the open source tooling for AWS was best wrt Python (runner up was Microsoft Azure Functions).

3. Thrift (over HTTPS) on AWS Lambda and API Gateway with Python

This shows an example of the (classic) Apache Thrift tutorial Calculator API running on AWS Lambda and API Gateway, the service requires 2 thrift files:

3.1 Development Environment and Tools

The tool used for deployment in this blog post is Zappa, I recommend using Zappa together with Docker for Python 3.6 as described in this blog post, with a slight change of the Dockerfile if you want to build and compile Apache thrift Python library yourself, here is the altered Dockerfile. There hasn’t been official releases of Apache Thrift since 0.10.0 January 6th 2017, and there has been important improvement related to its Python support since last release – in particular the fix for supporting recursive thrift structs in Python

a. Dockerfile – for creating a Zappashell (same as Lambda runtime ) and builds Thrift

# build this with command:
#   docker build -t myzappa .
FROM lambci/lambda:build-python3.6
WORKDIR /var/task
# Fancy prompt to remind you are in zappashell                                                                                                             
RUN echo 'export PS1="\[\e[36m\]zappashell&gt;\[\e[m\] "' &gt;&gt; /root/.bashrc
# Build Apache thrift Python library                                                                                                                           
RUN yum clean all &amp;&amp; \                                                                                                                                   
yum -y install emacs boost* gcc 
RUN git clone https://github.com/apache/thrift.git &amp;&amp; \
cd thrift &amp;&amp; \
./bootstrap.sh &amp;&amp; \
./configure &amp;&amp; \
make &amp;&amp; make install &amp;&amp; \
cd lib/py &amp;&amp; python setup.py install &amp;&amp; \
python setup.py sdist # Builds a thriftSomeVersion.tar.gz
CMD ["bash"]

# build this with command:

# docker build -t myzappa .

FROM lambci/lambda:build-python3.6

WORKDIR /var/task

# Fancy prompt to remind you are in zappashell

RUN echo 'export PS1="\[\e[36m\]zappashell>\[\e[m\] "' >> /root/.bashrc

# Build Apache thrift Python library

RUN yum clean all && \

yum -y install emacs boost* gcc

RUN git clone https://github.com/apache/thrift.git && \

cd thrift && \

./bootstrap.sh && \

./configure && \

make && make install && \

cd lib/py && python setup.py install && \

python setup.py sdist # Builds a thriftSomeVersion.tar.gz

CMD ["bash"]

After building this Dockerfile (see command on top of file) and adding zappashell to your .bash_profile like this (source: the above mentioned blog post)

alias zappashell='docker run -ti -e AWS_PROFILE=zappa -v $(pwd):/var/task -v ~/.aws/:/root/.aws  --rm myzappa'
alias zappashell &gt;&gt; ~/.bash_profile

1 2	alias zappashell='docker run -ti -e AWS_PROFILE=zappa -v $(pwd):/var/task -v ~/.aws/:/root/.aws --rm myzappa' alias zappashell >> ~/.bash_profile

You can start your serverless deployment environment with the command zappashell (inside an new empty directory on your host platform e.g. a mac), this gives something like this – with an empty directory.

username@MyMac$ mkdir my_thrift_app
username@MyMac$ cd my_thrift_app
username@MyMac$ zappashell
[zappashell&gt; pwd
/var/task
[zappashell: ls 
[zappashell:

username@MyMac$ mkdir my_thrift_app

username@MyMac$ cd my_thrift_app

username@MyMac$ zappashell

[zappashell> pwd

/var/task

[zappashell: ls

[zappashell:

Install virtualenv and create/activate an environment(and assuming you installed thrift as shown in Dockerfile above)

[zappashell: pip install virtualenv
[zappashell: virtualenv serverlessdeployenv
[zappashell: source serverlessdeployenv/bin/activate
(serverlessdeployenv)[zappashell&gt;

[zappashell: pip install virtualenv

[zappashell: virtualenv serverlessdeployenv

[zappashell: source serverlessdeployenv/bin/activate

(serverlessdeployenv)[zappashell>

Use thrift to generate python code for tutorial.thrift and shared.thrift

(serverlessdeployenv)[zappashell: thrift --gen py tutorial.thrift
(serverlessdeployenv)[zappashell: thrift --gen py shared.thrift
(serverlessdeployenv)[zappashell: ls gen-py
__init__.py  shared  tutorial

(serverlessdeployenv)[zappashell: thrift --gen py tutorial.thrift

(serverlessdeployenv)[zappashell: thrift --gen py shared.thrift

(serverlessdeployenv)[zappashell: ls gen-py

__init__.py shared tutorial

Convert the gen-py package into a python library (for convenient packaging) with a setup.py file as below (change version according to your wants)

setup.py

from setuptools import setup, find_packages
# https://stackoverflow.com/questions/10924885/is-it-possible-to-include-subdirectories-using-dist-utils-setup-py-as-part-of
setup(
name='genpy',
version='1.1',
packages=find_packages(),
license='Creative Commons Attribution-Noncommercial-Share Alike license',
long_description=open('README.txt').read(),
)

from setuptools import setup, find_packages

# https://stackoverflow.com/questions/10924885/is-it-possible-to-include-subdirectories-using-dist-utils-setup-py-as-part-of

setup(

name='genpy',

version='1.1',

packages=find_packages(),

license='Creative Commons Attribution-Noncommercial-Share Alike license',

long_description=open('README.txt').read(),

)

(serverlessdeployenv)[zappashell: mv gen-py genpy
(serverlessdeployenv)[zappashell: cd genpy
(serverlessdeployenv)[zappashell: touch README.txt # or create one
(serverlessdeployenv)[zappashell: # create setup.py as shown above
(serverlessdeployenv)[zappashell: python setup.py sdist
(serverlessdeployenv)[zappashell: ls dist
genpy-1.1.tar.gz
(serverlessdeployenv)[zappashell: cp dist/genpy-1.1.tar.gz ..

(serverlessdeployenv)[zappashell: mv gen-py genpy

(serverlessdeployenv)[zappashell: cd genpy

(serverlessdeployenv)[zappashell: touch README.txt # or create one

(serverlessdeployenv)[zappashell: # create setup.py as shown above

(serverlessdeployenv)[zappashell: python setup.py sdist

(serverlessdeployenv)[zappashell: ls dist

genpy-1.1.tar.gz

(serverlessdeployenv)[zappashell: cp dist/genpy-1.1.tar.gz ..

Copy the generated thrift library – note: thrift itself not the tutorial code – (ref thriftSomeVersion.tar.gz generated by python setup.by sdist in Dockerfile) to the same directory and add it to requirements.txt

requirements.txt should look something like this:

zappa
flask
thriftSomeVersion.tar.gz
genpy-1.1.tar.gz

zappa

flask

thriftSomeVersion.tar.gz

genpy-1.1.tar.gz

Run pip install -r requirements.txt

Create app.py that has code for calculator thrift

import io
from flask import Flask
from flask import make_response, send_file, request
from tutorial import Calculator
import tutorial.ttypes
from shared.ttypes import SharedStruct
from thrift.protocol import TBinaryProtocol
from thrift.server import THttpServer
from thrift.server import TServer
from thrift.transport import TTransport
app=Flask(__name__)
class CalculatorHandler(object):
def __init__(self):
self.log = {}
def ping(self):
print("ping()")
def add(self, n1, n2):
print("add({}, {})".format(n1, n2))
return n1 + n2
@app.route('/thr', methods=['POST'])
def thr():
# get the thrift package from HTTP POST body
body = request.get_data()
content_length = int(request.headers["Content-Length"])
# thrift setup
itrans = TTransport.TMemoryBuffer(body)
itrans = TTransport.TBufferedTransport(itrans, content_length)
otrans = TTransport.TMemoryBuffer()
handler = CalculatorHandler()
processor = Calculator.Processor(handler)
inputProtocolFactory = TBinaryProtocol.TBinaryProtocolFactory()
outputProtocolFactory = inputProtocolFactory
iprot = inputProtocolFactory.getProtocol(itrans)
oprot = outputProtocolFactory.getProtocol(otrans)
processor.process(iprot,oprot)
response = make_response(otrans.getvalue())
response.headers['Content-Type'] = 'application/x-thrift'
return response, 200
if __name__ == '__main__':
app.run()

import io

from flask import Flask

from flask import make_response, send_file, request

from tutorial import Calculator

import tutorial.ttypes

from shared.ttypes import SharedStruct

from thrift.protocol import TBinaryProtocol

from thrift.server import THttpServer

from thrift.server import TServer

from thrift.transport import TTransport

app=Flask(__name__)

class CalculatorHandler(object):

def __init__(self):

self.log = {}

def ping(self):

print("ping()")

def add(self, n1, n2):

print("add({}, {})".format(n1, n2))

return n1 + n2

@app.route('/thr', methods=['POST'])

def thr():

# get the thrift package from HTTP POST body

body = request.get_data()

content_length = int(request.headers["Content-Length"])

# thrift setup

itrans = TTransport.TMemoryBuffer(body)

itrans = TTransport.TBufferedTransport(itrans, content_length)

otrans = TTransport.TMemoryBuffer()

handler = CalculatorHandler()

processor = Calculator.Processor(handler)

inputProtocolFactory = TBinaryProtocol.TBinaryProtocolFactory()

outputProtocolFactory = inputProtocolFactory

iprot = inputProtocolFactory.getProtocol(itrans)

oprot = outputProtocolFactory.getProtocol(otrans)

processor.process(iprot,oprot)

response = make_response(otrans.getvalue())

response.headers['Content-Type'] = 'application/x-thrift'

return response, 200

if __name__ == '__main__':

app.run()

Create .aws directory with files:

credentials

[default]
aws_access_key_id=SOMETHINGSECRET
aws_access_access_key=SOMETHINGELSESECRET

[default]

aws_access_key_id=SOMETHINGSECRET

aws_access_access_key=SOMETHINGELSESECRET

config

[default]
region=eu-west-1 # or the region of your choice
output=text

[default]

region=eu-west-1 # or the region of your choice

output=text

Run zappa init and answers questions, it should look something like the image below:

you should now be able to deploy the API with

zappa deploy beta

1	zappa deploy beta

You can test the deployed API with the following client, remember to change the https address to the address that the deploy gave you

from tutorial import Calculator
from tutorial.ttypes import Operation, Work, InvalidOperation
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol
from thrift import Thrift
from thrift.transport import TSocket
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol
from thrift.transport import THttpClient
import logging
def main():
transport = THttpClient.THttpClient('https://SOMETHING.amazonaws.com/beta/thr')
#transport = THttpClient.THttpClient('http://127.0.0.1:5000/thr')                                                                                                                                                   
# NEW - set API key                                                                                                                                                                                                 
transport.setCustomHeaders({"x-api-key":"SomeGeneratedKey"})
# Buffering is critical. Raw sockets are very slow                                                                                                                                                                  
transport = TTransport.TBufferedTransport(transport)
# Wrap in a protocol                                                                                                                                                                                                
protocol = TBinaryProtocol.TBinaryProtocol(transport)
client = Calculator.Client(protocol)
transport.open()
client.ping()
result = client.add(11,77)
print("11+77 = ", result)
transport.close()
if __name__ == "__main__":
main()

from tutorial import Calculator

from tutorial.ttypes import Operation, Work, InvalidOperation

from thrift.transport import TTransport

from thrift.protocol import TBinaryProtocol

from thrift import Thrift

from thrift.transport import TSocket

from thrift.transport import TTransport

from thrift.protocol import TBinaryProtocol

from thrift.transport import THttpClient

import logging

def main():

transport = THttpClient.THttpClient('https://SOMETHING.amazonaws.com/beta/thr')

#transport = THttpClient.THttpClient('http://127.0.0.1:5000/thr')

# NEW - set API key

transport.setCustomHeaders({"x-api-key":"SomeGeneratedKey"})

# Buffering is critical. Raw sockets are very slow

transport = TTransport.TBufferedTransport(transport)

# Wrap in a protocol

protocol = TBinaryProtocol.TBinaryProtocol(transport)

client = Calculator.Client(protocol)

transport.open()

client.ping()

result = client.add(11,77)

print("11+77 = ", result)

transport.close()

if __name__ == "__main__":

main()

But wait, something is missing, this API is reachable by anyone. Let us add an API key (and update the client with the x-api-key). This can be done through AWS Console (and perhaps with Zappa itself through automation soon?) with the following steps:

Go to Amazon API Gateway Console and click on the generated API (perhaps named task-beta due to the Docker file path and the selected stage during zappa init)

Create a Usage Plan and associate it with the API (e.g. task-beta), then create an API Key (on the left side menu) and attach the API Key to the Usage Plan

Do a zappa update dev and and uncomment/update the transport.setCustomHeaders with x-api-key in the python client above to get authentication and throttling in place.

4. Conclusion

Have shown an example of getting thrift API running on Serverless that can relatively easily be automated, and when the API is initially created it is very little effort to update it (e.g. through continuous deployment).

A final note on roundtrip time performance, based on a few rough tests it looks like the roundtrip time for calls to API is around 300-400 milliseconds (with the test client based in Trondheim, Norway and accessing API Gateway in AWS and AWS Lambda in Germany), which is quite good. Believe that with an AWS Route53 Routing Policy one could have automatic selection of the closest AWS API Gateway/Lambda to get the lowest latency (note that one of the selections in zappa init was to deploy globally, but default was one availability zone).

Believe personally that Serverless computing has a strong future ahead wrt API development, and look forward to what cloud vendors software engineers/product managers add of new features, my wish list is:

Strong Python support
Built-in Thrift support and service discovery, as well as support for other RPC systems, e.g. gRPC, messagepack,++
Improved software tooling for automation (e.g. simplified SSL/domain name setup/maintenance handling – get deep partnerships with letsencrypt.com for SSL certificates?)
Increase caching support at various levels

Best regards,

Amund Tveit

Convolutional Neural Network, CoreML, Deep Learning, GAN, generative adversarial networks, Mobile

Creative AI on the iPhone (with GAN) and Dynamic Loading of CoreML models

September 16, 2017 No Comments

Zedge summer interns developed a very cool app using ARKit and CoreML (on iOS11). As parts of their journey the published 2 blog posts on the Zedge corporate web site related to:

How to develop and run Generative Adversarial Networks (GAN) for Creative AI on the iPhone using Apple’s CoreML tools, check out their blog post about it.
Deep Learning models (e.g. for GAN) can take a lot of space on a mobile device (tens of Megabytes to perhaps even Gigabytes), in order to keep initial app download size relatively low it can be useful to dynamically load only the models you need. Check out their blog post about various approaches for hotswapping CoreML models.

Best regards,

Amund Tveit

AI, Computer Vision, Deep Learning, Self-Driving Cars

Convolutional Neural Networks for Self-Driving Cars

August 27, 2017 No Comments

This blog post are my notes from project 3 in the term 1 of the Udacity Nanodegree in Self Driving cars. The project is about developing and training a convolutional neural network of camera input (3 different camera angles) from a simulated car.

Best regards,

Amund Tveit

1. Modelling Convolutional Neural Network for Self Driving Car

Used the NVIDIA Autopilot Deep Learning model for self-driving as inspiration (ref: paper “End to End Learning for Self-Driving Cars” – https://arxiv.org/abs/1604.07316 and implementation of it: https://github.com/0bserver07/Nvidia-Autopilot-Keras), but did some changes to it:

Added normalization in the model itself (ref Lambda(lambda x: x/255.0 – 0.5, input_shape=img_input_shape)), since it is likely to be faster than doing it in pure Python.
Added Max Pooling after the first convolution layers, i.e. making the model a more “traditional” conv.net wrt being capable of detecting low level features such as edges (similar to classic networks such as LeNet).
Added Batch Normalization in early layers to be more robust wrt different learning rates
Used he_normal normalization (truncated normal distribution) since this type of normalization with TensorFlow has earlier mattered a lot
Used L2 regularizer (ref: “rule of thumb” – https://www.quora.com/What-is-the-difference-between-L1-and-L2-regularization-How-does-it-solve-the-problem-of-overfitting-Which-regularizer-to-use-and-when )
Made the model (much) smaller by reducing the fully connected layers (got problems running larger model on 1070 card, but in retrospect it was not the model size but my misunderstandings of Keras 2 that caused this trouble)
Used selu (ref: paper “Self-Normalizing Neural Networks” https://arxiv.org/abs/1706.02515) instead of relu as rectifier functions in later layers (fully connected) – since previous experience have shown (with traffic sign classification and tensorflow) showed that using selu gave faster convergence rates (though not better final result).
Used dropout in later layers to avoid overfitting
Used l1 regularization on the final layer, since I’ve seen that it is good for regression problems (better than l2)

Image of Model

Detailed model

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
lambda_1 (Lambda)            (None, 160, 320, 3)       0         
_________________________________________________________________
cropping2d_1 (Cropping2D)    (None, 85, 320, 3)        0         
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 41, 158, 24)       1824      
_________________________________________________________________
batch_normalization_1 (Batch (None, 41, 158, 24)       96        
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 21, 79, 24)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 9, 38, 36)         21636     
_________________________________________________________________
batch_normalization_2 (Batch (None, 9, 38, 36)         144       
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 3, 17, 48)         43248     
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 1, 15, 64)         27712     
_________________________________________________________________
flatten_1 (Flatten)          (None, 960)               0         
_________________________________________________________________
dense_1 (Dense)              (None, 400)               384400    
_________________________________________________________________
dropout_1 (Dropout)          (None, 400)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 100)               40100     
_________________________________________________________________
dropout_2 (Dropout)          (None, 100)               0         
_________________________________________________________________
dense_3 (Dense)              (None, 50)                5050      
_________________________________________________________________
dense_4 (Dense)              (None, 10)                510       
_________________________________________________________________
dense_5 (Dense)              (None, 1)                 11        
=================================================================
Total params: 524,731
Trainable params: 524,611
Non-trainable params: 120

_________________________________________________________________

Layer (type) Output Shape Param #

=================================================================

lambda_1 (Lambda) (None, 160, 320, 3) 0

_________________________________________________________________

cropping2d_1 (Cropping2D) (None, 85, 320, 3) 0

_________________________________________________________________

conv2d_1 (Conv2D) (None, 41, 158, 24) 1824

_________________________________________________________________

batch_normalization_1 (Batch (None, 41, 158, 24) 96

_________________________________________________________________

max_pooling2d_1 (MaxPooling2 (None, 21, 79, 24) 0

_________________________________________________________________

conv2d_2 (Conv2D) (None, 9, 38, 36) 21636

_________________________________________________________________

batch_normalization_2 (Batch (None, 9, 38, 36) 144

_________________________________________________________________

conv2d_3 (Conv2D) (None, 3, 17, 48) 43248

_________________________________________________________________

conv2d_4 (Conv2D) (None, 1, 15, 64) 27712

_________________________________________________________________

flatten_1 (Flatten) (None, 960) 0

_________________________________________________________________

dense_1 (Dense) (None, 400) 384400

_________________________________________________________________

dropout_1 (Dropout) (None, 400) 0

_________________________________________________________________

dense_2 (Dense) (None, 100) 40100

_________________________________________________________________

dropout_2 (Dropout) (None, 100) 0

_________________________________________________________________

dense_3 (Dense) (None, 50) 5050

_________________________________________________________________

dense_4 (Dense) (None, 10) 510

_________________________________________________________________

dense_5 (Dense) (None, 1) 11

=================================================================

Total params: 524,731

Trainable params: 524,611

Non-trainable params: 120

####2. Attempts to reduce overfitting in the model

The model contains dropout layers in order to reduce overfitting (ref dropout_1 and dropout_2 in figure above and train_car_to_drive.ipynb).

Partially related: Used also balancing of data sets in the generator, see sample_weight in generator function and snippet below

sample_weight = sklearn.utils.class_weight.compute_sample_weight('balanced', y_train)
...
yield sklearn.utils.shuffle(X_train, y_train, sample_weight)

sample_weight = sklearn.utils.class_weight.compute_sample_weight('balanced', y_train)

...

yield sklearn.utils.shuffle(X_train, y_train, sample_weight)

The model was tested by running it through the simulator and ensuring that the vehicle could stay on the track. See modelthatworked.mp4 file in this github repository.

####3. Model parameter tuning

The model used an adam optimizer, so the learning rate was not tuned manually

####4. Appropriate training data

Used the training data that was provided as part of the project, and in addition added two runs of data to avoid problems (e.g. curve without lane line on the right side – until the bridge started and also a separate training set driving on the bridge). Data is available on https://amundtveit.com/DATA0.tgz).

###Model Architecture and Training Strategy

####1. Solution Design Approach

The overall strategy for deriving a model architecture was to use a conv.net, first tried the previous one I used for Traffic Sign detection based on LeNet, but it didn’t work (probably too big images as input), and then started with the Nvidia model (see above for details about changes to it).

In order to gauge how well the model was working, I split my image and steering angle data into a training and validation set. Primary finding was that numerical performance of the models I tried was not a good predictor of how well it would it perform on actual driving in the simulator. Perhaps overfitting could be good for this task (i.e. memorize track), but I attempted to get a correctly trained model without overfitting (ref. dropout/selu and batch normalization). There were many failed runs before the car actually could drive around the first track.

2. Creation of the Training Set & Training Process

I redrove and captured training data for the sections that were problematic (as mentioned the curve without lane lines on right and the bridge and part just before bridge). Regarding center-driving I didn’t get much success adding data for that, but perhaps my rebalancing (ref. generator output above) actually was counter-productive?

For each example line in the training data I generated 6 variants (for data augmentetation), i.e. flipped image (along center vertical axis) + also used the 3 different cameras (left, center and right) with adjustments for the angle.

After the collection process, I had 10485 lines in driving_log.csv, i.e. number of data points = 62430 (6*10485). Preprocessing used to flip image, convert images to numpy arrays and also (as part of Keras model) to scale values. Also did cropping of the image as part of the model. I finally randomly shuffled the data set and put 20 of the data into a validation set, see generator for details. Examples of images (before cropping inside model) is shown below:

Example of center camera image

Example of flipped center camera image

Example of left camera image

Example of right camera image

generator

# dealing with unbalanced data with class_weight in Keras
# https://datascience.stackexchange.com/questions/13490/how-to-set-class-weights-for-imbalanced-classes-in-keras
import sklearn.utils.class_weight
def generator(samples,batch_size=32, image_prefix_path="../DATA0/IMG/"):
num_samples = len(samples)
while 1: # Loop forever so the generator never terminates
random.shuffle(samples)
for offset in range(0, num_samples, batch_size):
batch_samples = samples[offset:offset+batch_size]
images = []
angles = []
angle_offsets = [0.0, 0.20, -0.20]
for batch_sample in batch_samples:
center_angle = float(batch_sample[3])
for image_position in range(3):
image_subpath = batch_sample[image_position].split('/')[-1]
image_path = image_prefix_path + image_subpath
image = cv2.imread(image_path)
images.append(image)
angle = center_angle + angle_offsets[image_position]
angles.append(angle)
# also add flipped image and angle
flipped_image = np.fliplr(image)
flipped_angle_for_image = -angle
images.append(flipped_image)
angles.append(flipped_angle_for_image)
X_train = np.array(images)
y_train = np.array(angles)
sample_weight = sklearn.utils.class_weight.compute_sample_weight('balanced', y_train)
yield sklearn.utils.shuffle(X_train, y_train, sample_weight)

# dealing with unbalanced data with class_weight in Keras

# https://datascience.stackexchange.com/questions/13490/how-to-set-class-weights-for-imbalanced-classes-in-keras

import sklearn.utils.class_weight

def generator(samples,batch_size=32, image_prefix_path="../DATA0/IMG/"):

num_samples = len(samples)

while 1: # Loop forever so the generator never terminates

random.shuffle(samples)

for offset in range(0, num_samples, batch_size):

batch_samples = samples[offset:offset+batch_size]

images = []

angles = []

angle_offsets = [0.0, 0.20, -0.20]

for batch_sample in batch_samples:

center_angle = float(batch_sample[3])

for image_position in range(3):

image_subpath = batch_sample[image_position].split('/')[-1]

image_path = image_prefix_path + image_subpath

image = cv2.imread(image_path)

images.append(image)

angle = center_angle + angle_offsets[image_position]

angles.append(angle)

# also add flipped image and angle

flipped_image = np.fliplr(image)

flipped_angle_for_image = -angle

images.append(flipped_image)

angles.append(flipped_angle_for_image)

X_train = np.array(images)

y_train = np.array(angles)

sample_weight = sklearn.utils.class_weight.compute_sample_weight('balanced', y_train)

yield sklearn.utils.shuffle(X_train, y_train, sample_weight)

I used this training data for training the model. The validation helped determine if the model was over or under fitting. The ideal number of epochs was 5 as evidenced by the quick flattening of loss and validation loss (to around 0.03), in earlier runs validation loss increased above training loss when having more epochs. I used an adam optimizer so that manually training the learning rate wasn’t necessary.

Epoch 1/5
178s - loss: 0.2916 - val_loss: 0.0603
Epoch 2/5
180s - loss: 0.0530 - val_loss: 0.0463
Epoch 3/5
181s - loss: 0.0398 - val_loss: 0.0330
Epoch 4/5
179s - loss: 0.0309 - val_loss: 0.0326
Epoch 5/5
178s - loss: 0.0302 - val_loss: 0.0312

Epoch 1/5

178s - loss: 0.2916 - val_loss: 0.0603

Epoch 2/5

180s - loss: 0.0530 - val_loss: 0.0463

Epoch 3/5

181s - loss: 0.0398 - val_loss: 0.0330

Epoch 4/5

179s - loss: 0.0309 - val_loss: 0.0326

Epoch 5/5

178s - loss: 0.0302 - val_loss: 0.0312

3. Challenges

Challenges along the way – found it to be a very hard task, since the model loss and validation loss weren’t good predictors for actual driving performance, also had cases when adding more training data with nice driving data (at the center and far from the edges) actually gave worse results and made the car drive off the road. Other challenges were Keras 2 related, the semantics of parameters in Keras 1 and Keras 2 fooled me a bit using Keras 2, ref the steps_per_epoch. Also had issues with the progress bar not working in Keras 2 in Jupyter notebook, so had to use 3rd party library https://pypi.python.org/pypi/keras-tqdm/2.0.1

AI, Deep Learning, Energy, Gas, Oil, Turbine, Wind

Deep Learning in Energy Production

July 21, 2017 No Comments

This blog post has recent publications about use of Deep Learning in Energy Production context (wind, gas and oil), e.g. wind power prediction, turbine risk assessment, reservoir discovery and price forecasting.

Best regards,

Amund Tveit

Wind

Year	Title	Author
2017	Short-term Wind Energy Prediction Algorithm Based on SAGA-DBNs	W Fei, WU Zhong
2017	Wind Power Prediction using Deep Neural Network based Meta Regression and Transfer Learning	AS Qureshi, A Khan, A Zameer, A Usman
2017	Wind Turbine Failure Risk Assessment Model Based on DBN	C Fei, F Zhongguang
2017	The optimization of wind power interval forecast	X Yu, H Zang
2016	Deep Learning for Wind Speed Forecasting in Northeastern Region of Brazil	AT Sergio, TB Ludermir
2016	A very short term wind power prediction approach based on Multilayer Restricted Boltzmann Machine	X Peng, L Xiong, J Wen, Y Xu, W Fan, S Feng, B Wang
2016	Short-term prediction of wind power based on deep Long Short-Term Memory	Q Xiaoyun, K Xiaoning, Z Chao, J Shuai, M Xiuda
2016	Deep belief network based deterministic and probabilistic wind speed forecasting approach	HZ Wang, GB Wang, GQ Li, JC Peng, YT Liu
2016	A hybrid wind power prediction method	Y Tao, H Chen
2016	Deep learning based ensemble approach for probabilistic wind power forecasting	H Wang, G Li, G Wang, J Peng, H Jiang, Y Liu
2016	A hybrid wind power forecasting model based on data mining and wavelets analysis	R Azimi, M Ghofrani, M Ghayekhloo
2016	ELM Based Representational Learning for Fault Diagnosis of Wind Turbine Equipment	Z Yang, X Wang, PK Wong, J Zhong
2015	Deep Neural Networks for Wind Energy Prediction	D Díaz, A Torres, JR Dorronsoro
2015	Predictive Deep Boltzmann Machine for Multiperiod Wind Speed Forecasting	CY Zhang, CLP Chen, M Gan, L Chen
2015	Resilient Propagation for Multivariate Wind Power Prediction	J Stubbemann, NA Treiber, O Kramer
2015	Transfer learning for short-term wind speed prediction with deep neural networks	Q Hu, R Zhang, Y Zhou
2014	Wind Power Prediction and Pattern Feature Based on Deep Learning Method	Y Tao, H Chen, C Qiu

Gas

Year	Title	Author
2017	Sample Document–Inversion Of The Permeability Of A Tight Gas Reservoir With The Combination Of A Deep Boltzmann Kernel …	L Zhu, C Zhang, Y Wei, X Zhou, Y Huang, C Zhang
2017	Deep Learning: Chance and Challenge for Deep Gas Reservoir Identification	C Junxing, W Shikai
2016	Finite-sensor fault-diagnosis simulation study of gas turbine engine using information entropy and deep belief networks	D Feng, M Xiao, Y Liu, H Song, Z Yang, Z Hu
2015	On Accurate and Reliable Anomaly Detection for Gas Turbine Combustors: A Deep Learning Approach	W Yan, L Yu
2015	A Review of Datasets and Load Forecasting Techniques for Smart Natural Gas and Water Grids: Analysis and Experiments.	M Fagiani, S Squartini, L Gabrielli, S Spinsante
2015	Short-term load forecasting for smart water and gas grids: A comparative evaluation	M Fagiani, S Squartini, R Bonfigli, F Piazza
2015	The early-warning model of equipment chain in gas pipeline based on DNN-HMM	J Qiu, W Liang, X Yu, M Zhang, L Zhang

Oil

Year	Title	Author
2017	Development of a New Correlation for Bubble Point Pressure in Oil Reservoirs Using Artificial Intelligent Technique	S Elkatatny, M Mahmoud
2017	A deep learning ensemble approach for crude oil price forecasting	Y Zhao, J Li, L Yu
2016	Automatic Detection and Classification of Oil Tanks in Optical Satellite Images Based on Convolutional Neural Network	Q Wang, J Zhang, X Hu, Y Wang
2015	A Hierarchical Oil Tank Detector With Deep Surrounding Features for High-Resolution Optical Satellite Imagery	L Zhang, Z Shi, J Wu

Computer Vision, Lane Finding, OpenCV, Python, Self-Driving Cars

Lane Finding (on Roads) for Self Driving Cars with OpenCV

July 14, 2017 No Comments

This blog post is a (basic) approach of how to potentially use OpenCV for Lane Finding for self-driving cars (i.e. the yellow and white stripes along the road) – did this as one of the projects of term 1 of Udacity’s self-driving car nanodegree (highly recommended online education!).

Disclaimer: the approach presented in this blog post is way to simple to use for an actual self-driving car, but was a good way (for me) to learn more about (non-deep learning based) computer vision and the lane finding problem.

See github.com/atveit/LaneFindingForSelfDrivingCars for more details about the approach (python code)

Best regards,

Amund Tveit

Lane Finding (On Roads) for Self Driving Cars with OpenCV

1. First I selected the region of interest (with hand-made vertices)

vertices = np.array([[(w//15,h), # LOWER LEFT
(w//2-w//50, h//2+h//15), # UPPER LEFT
(w//2+w//50, h//2+h//15), # UPPER RIGHT
(w-w//15,h)]], # LOWER RIGHT
dtype=np.int32)
region_of_interest_image = region_of_interest(image, vertices)

vertices = np.array([[(w//15,h), # LOWER LEFT

(w//2-w//50, h//2+h//15), # UPPER LEFT

(w//2+w//50, h//2+h//15), # UPPER RIGHT

(w-w//15,h)]], # LOWER RIGHT

dtype=np.int32)

region_of_interest_image = region_of_interest(image, vertices)

2. Converted the image to grayscale

grayscale_image = grayscale(region_of_interest_image)

1	grayscale_image = grayscale(region_of_interest_image)

3. Extracted likely white lane information from the grayscale image.

Used 220 as limit (255 is 100% white, but 220 is close enough)

grayscale_select_image = np.copy(grayscale_image)
white_threshold = (grayscale_image[:,:] < 220)
grayscale_select_image[white_threshold] = 0

grayscale_select_image = np.copy(grayscale_image)

white_threshold = (grayscale_image[:,:] < 220)

grayscale_select_image[white_threshold] = 0

4. Extracted likely yellow lane information from the (colorized) region of interest image.

RGB for Yellow is [255,255,0] but found [220,220,30] to be close enough

yellow_threshold = [220,220,30]
yellow_select_image = np.copy(region_of_interest_image)
yellow_thresholds = (image[:,:,0] > yellow_threshold[0]) \
| (image[:,:,1] > yellow_threshold[1]) \
| (image[:,:,2] < yellow_threshold[2])
yellow_select_image[yellow_thresholds] = [0,0,0]

yellow_threshold = [220,220,30]

yellow_select_image = np.copy(region_of_interest_image)

yellow_thresholds = (image[:,:,0] > yellow_threshold[0]) \

| (image[:,:,1] > yellow_threshold[1]) \

| (image[:,:,2] < yellow_threshold[2])

yellow_select_image[yellow_thresholds] = [0,0,0]

5. Converted the yellow lane information image to grayscale

grayscale_from_yellow = grayscale(yellow_select_image)

1	grayscale_from_yellow = grayscale(yellow_select_image)

6. Combined the likely yellow and white lane grayscale images into a new grayscale image (using max value)

combined_gray_image = np.maximum(grayscale_select_image, grayscale_from_yellow)

1	combined_gray_image = np.maximum(grayscale_select_image, grayscale_from_yellow)

7. Did a gaussian blur (with kernel size 3) followed by canny edge detection

Gaussian blur smooths out the image using Convolution, this is reduce false signalling to the (canny) edge detector

blur_kernel_size = 3
blur_gray = cv2.GaussianBlur(combined_gray_image,(blur_kernel_size, blur_kernel_size),0)
canny_edge_image = canny(blur_gray, low_threshold=10, high_threshold=150)

blur_kernel_size = 3

blur_gray = cv2.GaussianBlur(combined_gray_image,(blur_kernel_size, blur_kernel_size),0)

canny_edge_image = canny(blur_gray, low_threshold=10, high_threshold=150)

8. Did a hough (transform) image creation, I also modified the draw_lines function (see GitHub link above) by calculating average derivative and b value (i.e. calculating y = x-b for all the hough lines to find a and b, and then average over them).

For more information about Hough Transform, check out this hough transformation tutorial.

(side note: believe it perhaps could have been smarter to use hough line center points instead of hough lines, since the directions of them seem sometimes a bit unstable, and then use average of derivatives between center points instead)

rho = 1 # distance resolution in pixels of the Hough grid
theta = np.pi/180 # angular resolution in radians of the Hough grid
threshold = 1 # minimum number of votes (intersections in Hough grid cell)
min_line_len = 3 #minimum number of pixels making up a line
max_line_gap = 1 # maximum gap in pixels between connectable line segments
line_image = np.copy(image)*0 # creating a blank to draw lines on
hough_image, lines = hough_lines(canny_edge_image, rho, theta, threshold, min_line_len, max_line_gap)

rho = 1 # distance resolution in pixels of the Hough grid

theta = np.pi/180 # angular resolution in radians of the Hough grid

threshold = 1 # minimum number of votes (intersections in Hough grid cell)

min_line_len = 3 #minimum number of pixels making up a line

max_line_gap = 1 # maximum gap in pixels between connectable line segments

line_image = np.copy(image)*0 # creating a blank to draw lines on

hough_image, lines = hough_lines(canny_edge_image, rho, theta, threshold, min_line_len, max_line_gap)

9. Used the weighted image to overlay the hough image with lane detection on top of the original image

weighted_image = weighted_img(hough_image, image, α=0.8, β=1., λ=0.)

1	weighted_image = weighted_img(hough_image, image, α=0.8, β=1., λ=0.)

1 2 3 … 6 Next »