Cloud Tutorial & Hackathon – Why a Software development Community? – Welcome



Cloud Tutorial & Hackathon – Why a Software development Community? – Welcome

1 0


2015-Q3-DeveloperCommunity-Hackathons

Slides ++ for the Q3 Hackathon in Trondheim

On Github Statoil / 2015-Q3-DeveloperCommunity-Hackathons

Statoil Software Developer Community

Cloud Tutorial & Hackathon

Q3/4 2015

Explore, Experiment, Learn & Have Fun

Why a Software development Community?

To create an arena for competence building, sharing, exploration, experimentation ++

Software Eats The World

Software Development And Data Science will become a business value/enabler for Statoil

Even if this not is broadly accepted yet

Where will we be doing a lot more software development?

Change in IT

In my humble opinion

From To IT Enables the commercial strategy IT Drives the commercial strategy IT is a cost center IT is a value driver Outsourcing for the lowest price Hiring the best talent Buy before build Build all the way to understand

The Quality of the IT Organization is defined by the The Quality of IT's engineer

Quality Software Engineers

  • Delivers software really really fast .. always!
  • Focus on business outcome
  • Deeply into technologies - on the edge
  • Proud Craftsmen

Software Engineer != Software Developer?

The Software Development Community represents one great opportunity for you to become the Quality Software Engineer that Staoil needs

Welcome

  • HSE
  • The Agenda
  • The Statoil Development Community
  • Ground rules
  • Be social
  • What is a Hackathon?
  • How will this Hackathon be?

Health, Safety & Environment

  • Fire exits
  • The are noe fire drills planned for today
  • It is ok to take photograps
  • We write in English - and speak Norwegian if it's ok for everyone

The Agenda

is defined on the events wiki page - and give an idea of what we planned, but it will change as we go along

The Statoil Development Community

{	"local_community" : [ 
		{
			"location": "Bergen",
			"playground": "Monthly",
			"ambassador": "Son"
		},
		{
			"location": "Trondheim",
			"playground": "Weekly",
			"ambassador": "Ellen"
		},
		{
			"location": "Stavanger",
			"playground": "Weekly",
			"ambassador": "Ingvild"
		}],
	"company_community": {
		"Organizer": "Lars Kåre / ITV-DIS",
		"Events": ["On The Wall Sessions",
		"Annual Statoil Developer Days"]	
	}
}

Places to monitor

The home on the Wiki

The Yammer Groups (local and common)

The Community list

Ground rules

  • Don't be an asshole
  • Be open minded
  • Partisipate
  • Share
  • Listen
  • All ideas are good ideas
  • Other's that we should mention?

Be social

Be social, take pictures, tweet, post to our Yammer Groups

Don't be afraid of saying something wrong, or doing someting stupid

Be the good examples, that dare :)

What is a Hackathon?

It is an event where software developers (and others) get together and collaborates intensively on various projects

Hackathons can be social, educational and could create actual software

Hackathons can have a specific focus

The Wiki has an eloborative definition

How will this Hackathon be?

A Tutorial - to give more people a bit more common ground

Tutorial: Introduce a "modern" technology and a small life cycle into a cloud solution

A Explorative Hackaton (on proposed tasks or the ones you select)

If you do not want to follow the tutorial that is ok too (but I hope you will and help others if you find it too easy)

..

The Hackathon, it's value for yourself and others, becomes what You make it!!

Tutorial part #1

Building a small NodeJs "Hello world"

Get the code under version control

Verifying the development environment

Verify that git is installed

c:\>git --version

Verify that node (NodeJS) is installed

C:\>node -v

Verify access to github Login tohttps://github.com/swtrh

Verify proxy settings HTTP_PROXY=http://www-proxy.statoil.no:80 HTTPS_PROXY=http://www-proxy.statoil.no:80

C:\>set

Update NPM

Npm install npm -g
Npm -v (should give version 2.5.1 or newer)

Creating the project

Create local project directory

mkdir lkskMyApp1 (use your own initial)
cd lkskMyApp1

Init the NodeJs project

C:\>npm init

name: (lkskMyApp1) lkskMyApp1
version: (1.0.0)
description: A NodeJs test app
entry point: (index.js)
test command:
git repository:
keywords:
author: Lars Kåre Skjorestad
license: (ISC)
About to write to D:\lksk\dev\projects\lkskMyApp1\package.json:

{
  "name": "lkskMyApp1",
  "version": "1.0.0",
  "description": "A NodeJs test app",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "Lars Kåre Skjorestad",
  "license": "ISC"
}


Is this ok? (yes) yes

Create index.js in the project directory

Add the following code

var express = require('express')
var app = express()
		 
app.get('/', function (req, res) {
  res.send('Hello World')
})

app.listen(3000)

Run the code and verify result

C:\>node index.js
Error: Cannot find module 'express'
    at Function.Module._resolveFilename (module.js:336:15)
    at Function.Module._load (module.js:278:25)
    at Module.require (module.js:365:17)
    at require (module.js:384:17)
	..
	..

We are missing a the node module: express

Let's install the node module: express

>npm install express --save

npm WARN package.json lkskMyApp1@1.0.0 No repository field.
npm WARN package.json lkskMyApp1@1.0.0 No README data
express@4.13.3 node_modules\express
├── escape-html@1.0.2
├── merge-descriptors@1.0.0
├── array-flatten@1.1.1
├── cookie@0.1.3
├── cookie-signature@1.0.6
├── utils-merge@1.0.0
├── range-parser@1.0.2
├── methods@1.1.1
├── vary@1.0.1
├── fresh@0.3.0
├── path-to-regexp@0.1.7
├── etag@1.7.0
├── content-type@1.0.1
├── parseurl@1.3.0
├── serve-static@1.10.0
├── content-disposition@0.5.0
├── depd@1.0.1
├── qs@4.0.0
├── finalhandler@0.4.0 (unpipe@1.0.0)
├── debug@2.2.0 (ms@0.7.1)
├── proxy-addr@1.0.8 (forwarded@0.1.0, ipaddr.js@1.0.1)
├── send@0.13.0 (destroy@1.0.3, ms@0.7.1, statuses@1.2.1, mime@1.3.4, http-errors@1.3.1)
├── accepts@1.2.13 (negotiator@0.5.3, mime-types@2.1.6)
├── type-is@1.6.8 (media-typer@0.3.0, mime-types@2.1.6)
└── on-finished@2.3.0 (ee-first@1.1.1)

Run the code again and verify in browser (localhost:3000)

Initializing a local git repository

Creating git repository

git init

Initialized empty Git repository in ./lkskMyApp1/.git/

Create .gitignore file

This file will contain all the content that should not be part of version control. The first version should look like this:

node_modules

Add files to local git repository and do a commit

git add index.js
git add package.json
git add .gitignore
git commit -a -m "Initial version of myapp1"

Task:

Add a line of code to your app that prints a line to the console telling that the app has started and which ports it is listening to

..
..
app.listen(3000);

console.log ('Server started on port 3000');

Connecting to github

Create a repository on https://github.com/swtrh

Remember to select the correct organisation (swtrh) Give the repository the same name as your app. Example: "lkskMyApp1" Make the repository public Make a note of the github url for you project

https://github.com/swtrh/lkskMyApp1.git

Push the project to the github repository you just created

Verify the results

(Remember to create a personal access token if you use two-factor authentication)

git remote add origin https://github.com/swtrh/lkskMyApp1.git
git push -u origin master

Tutorial part #2

Connecting to Azure

Enabling auto deploy from version control

Creating the webApp in Microsoft Azure

Steps

Enter the MS Azure Portal at https://portal.azure.com Log on using your statoil account, choose "work- & School" account Choose "+ New" -> "Web + Mobile" -> Web App Enter the following information
  • Name: lkskMyApp1
  • Subscription: LKSK-Hackathonsubscripton
  • Resource Group: Default
  • App Service Plan: DefaultAppService...
  • Press "Create" - could take minutes to finish

Test the url for the new app <

Task: Connect Azure & Github

Play around in Azure and see if you find how to connect it to gitHub

Find "deployment" in Azure webapp (Settings->Continous Deployment) Configure required settings Source: GitHub Enter the following information
  • Authorization: Your GitHub Account
  • Organization: swtrh
  • Project: Your GitHub project
  • Branch: master
  • Press "Ok" - Azure will not fecth from GitHub and Deploy

It is expected that you are asked to Authorize Azure to connect to your GitHub repositories

Verify the results - using the URL from Azure

Houston - we have a problem

Thing looks ok on Azure but the browser reports

You do not have permission to view this directory or page.

Task:Identify problem(s)

Play around and investigate. Keyword: (deployment)logs & logging

Do we know which file Azure want to start for the webapp? Server ports

Specifying startup file for the web application

(This can be done in many ways - we show one "common")

Edit your local package.json and add the following line

"scripts": {
  "start": "node index.js"
}

Test locally

npm start

Push changes to github and verify Azure redeployment

git commit -a -m "Adding startscript to package.json"
git push

Defining serverport

The quick fix and dirty fix would be to set the server to listen on port 80 :). That will not work on your Statoil PC due to firewall issues

Solution: Dynamic allocation of server port

Reading server port from the Environment

(This can be done in many ways - we show one "common" solution)

Alter index.js to read port from environment

..
var port = (process.env.PORT || '3000');
..
app.listen(port);
console.log ('Server started on port '+ port);

Test locally (Set port=3000)

Commit locally, push to github and verify results on Azure

Sweet Success

Tutorial part #3

The express framework

You probably will be using framworks. Express is one of many. Software Developent is often about connecting the dots, gluing things together rather than the hard core development people think of.

Some words of caution on Cloud solutions

  • Creating is free
  • Using cost money
  • It is really hard to figure out what the cost will be in advance
  • It is as important to stop as to start a service :)

The Express framework

It's home is on npmjs.com

Let's install the generator (link)- a tool to quickly set up a skeleton for the framework

npm install express-generator --save

npm WARN prefer global express-generator@4.13.1 should be installed with -g
express-generator@4.13.1 node_modules\express-generator
├── sorted-object@1.0.0
├── commander@2.7.1 (graceful-readlink@1.0.1)
└── mkdirp@0.5.1 (minimist@0.0.8)

The "-g" option means global, not only for your local project

Let's generate a webapp skeleton

express

destination is not empty, continue? [y/N] y

   create : .
   create : ./package.json
   create : ./app.js
   create : ./public
   create : ./public/javascripts
   create : ./views
   create : ./views/index.jade
   create : ./views/layout.jade
   create : ./views/error.jade
   create : ./routes
   create : ./routes/index.js
   create : ./routes/users.js
   create : ./public/images
   create : ./public/stylesheets
   create : ./public/stylesheets/style.css
   create : ./bin
   create : ./bin/www

   install dependencies:
     > cd . && npm install

   run the app:
     > SET DEBUG=lkskMyApp1:* & npm start

Test app locally

npm start

> lkskMyApp1@0.0.0 start lkskMyApp1
> node ./bin/www

module.js:338
    throw err;
          ^
Error: Cannot find module 'serve-favicon'
    at Function.Module._resolveFilename (module.js:336:15)
    at Function.Module._load (module.js:278:25)
    at Module.require (module.js:365:17)
    at require (module.js:384:17)
   ....
					

We are missing modules

Install missing modules

npm install

debug@2.2.0 node_modules\debug
└── ms@0.7.1

serve-favicon@2.3.0 node_modules\serve-favicon
├── ms@0.7.1
├── fresh@0.3.0
├── parseurl@1.3.0
└── etag@1.7.0

cookie-parser@1.3.5 node_modules\cookie-parser
├── cookie@0.1.3
└── cookie-signature@1.0.6

morgan@1.6.1 node_modules\morgan
├── basic-auth@1.0.3
├── on-headers@1.0.0
├── depd@1.0.1
└── on-finished@2.3.0 (ee-first@1.1.1)

body-parser@1.13.3 node_modules\body-parser
├── bytes@2.1.0
├── content-type@1.0.1
├── depd@1.0.1
├── qs@4.0.0
├── http-errors@1.3.1 (statuses@1.2.1, inherits@2.0.1)
├── raw-body@2.1.3 (unpipe@1.0.0)
├── iconv-lite@0.4.11
├── on-finished@2.3.0 (ee-first@1.1.1)
└── type-is@1.6.8 (media-typer@0.3.0, mime-types@2.1.6)

jade@1.11.0 node_modules\jade
├── character-parser@1.2.1
├── void-elements@2.0.1
├── commander@2.6.0
├── mkdirp@0.5.1 (minimist@0.0.8)
├── jstransformer@0.0.2 (is-promise@2.1.0, promise@6.1.0)
├── constantinople@3.0.2 (acorn@2.4.0)
├── with@4.0.3 (acorn-globals@1.0.6, acorn@1.2.2)
├── clean-css@3.4.3 (commander@2.8.1, source-map@0.4.4)
├── transformers@2.1.0 (promise@2.0.0, css@1.0.8, uglify-js@2.2.5)
└── uglify-js@2.4.24 (uglify-to-browserify@1.0.2, async@0.2.10, source-map@0.1.34, yargs@3.5.4)
					

Modules are updated based on dependencies in package.json. Test app locally

Startfile is updated (package.json). New default port.

Task:

Add files to local git Push changes to github Verify deployment and result on Azure
git stataus
...
git add * --all
..
git status
..
git commit -a -m "Added Express framework"
..
git push
					

Task:

Change the startpage to say "Greetings, welcome to Express" Verify cange locally Push changes to github Verify deployment and result on Azure
edit "views\index.jade"
npm start
git status
git commit -a -m "Alter welcome page"
git push
					

You have now established a short development lifecycle

Task: Routing

Extend the app to respond to "http://..../store" get requests Verify cange locally. Push changes to github. Verify deployment and result on Azure

Google is your friend. So is npmjs.com and http://stackoverflow.com/

Changes in app.js
	var store = require('./routes/store');
	app.use('/store', store);
New file "store.js" in ./routes (copy index.js)
	Edit store.js to render "store.jade"
New file "store.jade" in view (copy index.jade)
					

Jade is a node template engine

Let's enhance the store with a message form

Extend store.jade to include a message form
 form(action='http://localhost:3500/store/message',method='post')
  input(type="text",name="message", placeholder="Enter a message",size="40")
  input(type='submit',name='submit',value='Submit')

(jade is sensitive on indents & white spaces)
Verify locally
					

Let's enhance the store with the ability to receive a submitted message

Extend store.js to receive message
	router.route('/message')
		.post(function (req, res, next) {
		console.log('Receiving message ' + req.body.message);
		res.send('Message was ' + req.body.message);
	});

Verify locally. Then commit, push to github and verify on Azure
					

It did not work on Azure (unless you had the local appserver running)

The "submit" adress was hardcoded

How should we handle this kind of configuration?Configuration that depends on the runtime environment?

We will environment variables
	NODE_ENV=development(local) | production (azure default)
Some "helper code" in ./bin/www
	server.listen(port);
	server.on('error', onError);
	server.on('listening', onListening);
Altering ./routes/store.js to pass on URL to template
	router.get('/', function(req, res, next) {
 		 res.render('store', { title: 'Store', messageURL: process.env.MESSAGE_URL });
	});
Using value passed from store.js in store.jade
	form(action='#{messageURL}',method='post')
Commit, Push to Azure ad Verify!
					

It almost worked :)

We must define the MESSAGE_URL in Azure

In Azure, select settings Select Application Settings In the "App settings section" add the MESSAGE_URL and the correct Azure url Remeber to save! Azure restarts app automatically Verify

..

Tutorial part #4

Connection to a databases (mongoDB)

Creating a mongoDB service in Azure

Creating a mongoDB service in Azure

In Azure select "+ New" -> "Data + Storage" -> "Mongolab"

You are taken to the Azure Marketplace. Select MongoLab again

Add information
  • Select the SandBox plan
  • Subsxcription: LKSK-Hackatonsubscription
  • Name: lksk_MongoLab (use your initials)
  • Region: North Europe
  • Select "Next" arrow and the the "Purchase" button

    Can take a while to procure

Creating a document collection in the database

Goto the "old" Azure portal https://manage.windowsazure.com/ Find your mongoLab in the Azure Service List Go to the MongoLab website for the next steps Select "+ Add collection". Create a "messages" collection Create a new database user "messagewriter". Create a password Make a note of the connection URI

mongodb://dbuser:dbpassword@ds042898.mongolab.com:42898/lksk_MongoLab

Let's add support for mongoDB to our project

We start by searching for mongodb in https://www.npmjs.com/

Looks like we should install the "mongodb" modules

npm install mongodb --save
..
mongodb@2.0.43 node_modules\mongodb
├── readable-stream@1.0.31 (isarray@0.0.1, string_decoder@0.10.31, inherits@2.0.1, core-util-is@1.0.1)
├── es6-promise@2.1.1
└── mongodb-core@1.2.12 (bson@0.4.11, kerberos@0.0.14)
					

Let's add support for mongoDB in our code

We will add code to connect and store messages in store.js

var MongoClient = require('mongodb').MongoClient;
var mongoURL = '  mongodb://messagewriter:password@ds042898.mongolab.com:42898/lksk_MongoLab';

router.route('/message')
	.post(function (req, res, next) {

	var txtMessage = (req.body.message || 'empty message');
	// Storing message in database
	MongoClient.connect(mongoURL, function(err, db) {
		console.log("Connected to database");
	
		db.collection('messages').insert({'message': txtMessage}, {w: 1 }, function (err, item) {
						if (err) {
							console.log('Error storing message in database: ' + err);
							db.close();
							res.status(400).send('Error, unable to store message: ' + txtMessage);
						} else {
							db.close();
							console.log('Message stored ok in database: ' + txtMessage)
							res.status(200).send('Message stored: "' + txtMessage + '"');
						}});
	});   
	// End storing message in database
});
					

Test locally and find document in mongoLab

We are facing 1 or 2 problems now, which?

Connectivity issues to the database if you are on the internal network

Risk of storing password in git and then githubThe mean time for this to be exploited is now around 8 minutes

Task:

Find and implement a solution to the password problem

Alter store.js to read the password in an env variable: DBPASSWORDIt's not perfect but it works

var mongoURL = 'mongodb://messagewriter:' + (process.env.DBPASSWORD || 'empty') + '@ds042898.mongolab.com:42898/lksk_MongoLab';

Testing locally, committing and pushing to Azure. Verifying in mongoLab. Remeber the env variable in Azure

Hackathon

  • Find tasks / projects
  • Work alone or in pairs
  • We will experiment with stand-ups
  • ... and summarize and share in the end
Statoil Software Developer Community Cloud Tutorial & Hackathon Q3/4 2015