Tuesday, October 10, 2017

Build a Node.js RESTful API and deploy it to an App Service in Azure

Introduction

This post is a rework of the Microsoft documentation Build a Node.js RESTful API and deploy it to an API app in Azure linked with this git hub sample Azure-Samples/app-service-api-node-contact-list
In comparison with the Microsoft documentation, I focused more on the detailed steps with a lot of screen shots and didn't change nothing to the git hub sample so that the reader (even a newbie) can obtain a node.js api running in Azure within a few minutes. This quickstart shows how to :
  • create an Azure App Service to host your API
  • create locally a REST API, written with Node.js Express, using a Swagger definition
  • deploying it on Azure.

Configuring your environment

  • Create your Azure free account
  • Download and install Visual Studio Code
  • Download and install Git The most official build is available for download on the Git website. Just go to http://git-scm.com/download/win and the download will start automatically. Once git is installed open a command prompt or powershell and type:
    git -- version
    you should get a similar result:

  • Download node.js and install it
    On this step again you can test installation by asking for the node.js and npm version



Creating an Azure App Service to host the REST API

Connect to Azure portal We are going to host our Node.js REST API into an Azure Mobile APP. From 2015, all the repositories for Web App, mobile App backend, Web API have been gathered in an Azure offering called App Service in order all enjoy the same capabilities (behavior, insights, features,etc.). That means also there are quite the same and you can use a Web API App Service for a mobile backend and vice versa or a Web App App Service for hosting a Web API. So let's create a Mobile App App Service.



Click on App Service in the left menu (1)
Then on Add on the top menu (2)
Finally on the create button for the Mobile App (3) (You notice that Microsoft makes a special advertisment for this type of App Service)

You are then invited to enter the first settings of your App Service before launching its creation within the Azure portal.
Note carefully the App Service name and the Resource Group Name because we need them later for the deployment.
In my case: App Service Name : Contact-list Resource Group: tutorials



Once the App Service is created, locate the deployment credentials blade and fill the required fields then click on save on the top menu.



For having the complete information to deploy we need the Url for Git deployment.
We are first going to initialize the Git capability using the Azure CLI
Open the Azure CLI by clicking the top icon (1)
Accept the creation of the storage for the persistance
Choose Bash mode
When connected to CLI type the following line :

az webapp deployment source config-local-git --name <your app service name> --resource-group <the associated resource group name> --query url --output tsv

in my case

az webapp deployment source config-local-git --name Contact-list --resource-group tutorials --query url --output tsv



The Azure CLI provides you with the Git Url and this Url now appears in the UI in the properties blade:



Testing the Node.js API Locally



We are now going to dowload the Git Hub sample and make it work on our local machine.

Go to the Git Hub sample and download it.



open the sample with Visual Studio Code



Notice that the key files of the project, package.json and server.js are located int the following path: project/end/ContactList

Open a command prompt and navigate to this location, then type the following instruction:

npm install



You should have more lines because it would be your first installation of the npm packages linked to the project.
Once it's done you can test locally by typing this instruction in your command prompt:

npm start



Node is starting a virtual web server and excuting the project on the 8000 port as specified in the server.js file.
So for testing, open Chrome (IE will download the json result as a file) ans type the following Url in the browser:

http://localhost:8000

The browser will display the contact list in a json format:



Deploying to the Azure App Service and testing



Within Visual Studio code, open the integrated terminal



Navigate to the contactlist folder and to initialize your git,type:

git init .



You will notice that a .git folder was created on the contactlist repositery:



Create then a gitignore txt file in the contactlist repositery and type the folowing instruction in it:

node_module/

Save it and we are going to use a workaround to create a .gitignore file (because windows requires a file to have a name and not just an extension). While holding the right shift key of your keyboard, right click the file and click on "open a windows powershell (or command depending on your windows version) from here:



Then, type:

ren gitignore.txt .gitignore



Then you can notice in VS Code that the gitignore file has the proper icon and that the files under the node_module directory won't be deployed by typing:

git status

on the Visual Studio Code Terminal.



then type

git add .



and

git commit -m "initial version"



then connect to the Azure App Service by typing:

git remote add azure <git url that you can copy from the porperty blade of your app service>

in my case:

git remote add azure https://deploy-account-02@contact-list.scm.azurewebsites.net:443/Contact-list.git

The terminal just jumps to another line (my screen shot is wrong)
Then push your project files to your App Service by typing:

git push azure master

You are asked for your deployment account credentials,



then your files are deployed:



You can then test by calling your api. Your Azure App Service Url is at the top of the properties blade:



So, in my case one of my API url is:

http://contact-list.azurewebsites.net/contacts



If you want to have a more complete testing experience using swagger, type this Url:

http://<your app service url>/docs/?url=%2Fswagger#/Contacts

in my case:

http://contact-list.azurewebsites.net/docs/?url=%2Fswagger#/Contacts



Keep developping online within Azure


Last, you can enjoy the preview of Visual Studio Code online for keeping developping your API without having to deploy anymore from local environment.





By the way, it is an excellent way to check what was exactly deployed into the Azure App Service. The deployment has drammatically simplified the solution.

Hope that helps :-)

Friday, October 6, 2017

Azure App Service



Objective of this post : It is a bit tricky to understand well the new App Service offering for Azure. Here are some explanations details and useful links (I gave up using favourites in my browser to save the Azure documentation. Too big, too rich, too complex, meandering) I will start a serie of Sheets/Wiki on several topics, first for myself and make them public because it could help somebody else. This is the first one of the serie.

A little bit of history:

For Azure classic portal, there was 2 offerings - Azure Websites and Azure Mobile services. Since the beginning of 2015 and in Azure new portal, both of them got rolled into something called Azure App Service which also included two more kinds of apps - API Apps and Logic Apps.
Azure Web App is new name for Azure Websites that you can use to build websites in multiple languages where as Azure Mobile App is the new name for Azure Mobile Service that provides backend infrastructure for mobile applications (or in other words, Azure Mobile App is Mobile Backend-as-a-Service [MBaaS] offering from Azure).
source: stackoverflow discussion on the topic
You will have a clear and friendly explanation from Kirill Gavrylyuk in this Azure Friday session on Mobile App Service (03-19-2015)

General presentation

App Service is a new offering for Azure wich pull in bunch of services into a single offering called App Service including Web Apps, mobile, API and Logic. It is a new way in Azure to gather different sevices so as they can enjoy the same capabilities. All the new features what we had in Azure Web Sites are available across all for new type of services (mobile, API and Logic). That includes:
  • How they are scaled
  • How they behave
  • Things like application insights
Link to Microsoft App Service Portal
Fantastic training courses on Azure from Microsoft


Mobile App


Accelerate your mobile app development with this turnkey way to structure storage, authenticate users, and send push notifications. With native and cross-platform SDKs for iOS, Android, Windows, and HTML, as well as a powerful and flexible REST API, Mobile Apps empowers you to build connected applications for any platform and deliver a consistent experience across devices.

Integrate with SQL, Oracle, SAP, MongoDB, and more.
Make your app work offline and sync.
Connect to on-premises data.
Leverage enterprise single sign-on with Active Directory.
Integrate with social providers like Facebook, Twitter, and Google.
Broadcast push notifications across platforms, with customer segmentation.
Gain insights with mobile analytics.
Auto-scale to millions of devices.
source : Azure Portal Description when attempting to create

Link to Azure App Service Mobile Apps documentation
Link to Azure Friday Video - Azure App Service Mobile Apps with Kirill Gavrylyuk
Link to Mobile App Microsoft Portal
Link to Azure portal Mobile App Creation

Link to node.js backend with Mongo DB

Creation:

you can use this link or
On Azure Portal Click on App Service, then on Add:



Then click on the create button of the Mobile App picture



Web App


Create and deploy web sites in seconds, as powerful as you need them
Leverage your existing tools to create and deploy applications without the hassle of managing infrastructure. Microsoft Azure Web Sites offers secure and flexible development, deployment, and scaling options for any sized web application. Use frameworks and templates to create web sites in seconds. Choose from source control options like TFS, GitHub, and BitBucket. Use any tool or OS to develop your site with .NET, PHP, Node.js or Python.

Fastest way to build for the cloud
Provision and deploy fast
Secure platform that scales automatically
Great experience for Visual Studio developers
Open and flexible for everyone
Monitor, alert, and auto scale (preview)
source : Azure Portal Description when attempting to create


Web App are part of App Service and App Service is a new offering for Azure wich pull in bunch of services into a single offering called App Service including Web Apps, mobile, API and Logic. Web App is actually a web site.
Type Url
Choose Service Plan and Hosting (depending on the location of the main users)
source : Azure Friday Video: Azure App Service Web Apps with Yochay Kiriaty - 2015 May 24

API App


Create and deploy RESTful APIs in seconds, as powerful as you need them

Leverage your existing tools to create and deploy RESTful APIs without the hassle of managing infrastructure. Microsoft Azure App Service API Apps offers secure and flexible development, deployment, and scaling options for any sized RESTful API application. Use frameworks and templates to create RESTful APIs in seconds. Choose from source control options like TFS, GitHub, and BitBucket. Use any tool or OS to develop your RESTful API with .NET, Java, PHP, Node.js or Python.

Fastest way to build for the cloud
Provision and deploy fast
Simple access control and authentication
Secure platform that scales automatically
Great experience for Visual Studio developers with automatic SDK generation
Open and flexible for everyone
Monitor, alert, and auto scale (preview)

source : Azure Portal Description when attempting to create

Link to Microsoft Azure API App Service Portal
Link to API App Service Video

Creation

Click on All Resources (1), then Add (2). Type API APP into the search zone (3).



Then, clisk on the API APP button to access to the creation blade.



Logic App



Logic Apps allow developers to design workflows that articulate intent via a trigger and series of steps, each invoking an App Service API app whilst securely taking care of authentication and best practices like durable execution.
Easy to use design tools - Logic Apps can be designed end-to-end in the browser. Start with a trigger - from a simple schedule to whenever a tweet appears about your company. Then orchestrate any number of actions using the rich gallery of connectors.

Compose SaaS easily - Even composition tasks that are easy to describe are difficult to implement in code. Logic Apps make it a cinch to connect disparate systems. Want to create a task in CRM based on activity on your Facebook or Twitter accounts? Want to connect your cloud marketing solution to your on-premises billing system? Logic apps are the fastest, most reliable way to deliver solutions to these problems.

Extensibility baked in - Don't see the connector you need? Logic Apps are part of the App Service suite and designed to work with API apps; you can easily create your own API app to use as a connector. Build a new app just for you, or share and monetize in the marketplace.

Real integration horsepower - Start easy and grow as you need. Logic Apps can easily leverage the power of BizTalk, Microsoft's industry leading integration solution to enable integration professionals to build the solutions they need.

source : Azure Portal Description when attempting to create

Creation:

On Azure Portal Click on App Service, then on Add:



Then, scroll down to display the Logic App Section, and on the "More" link to display all the options regarding Logic Apps:




Tuesday, May 31, 2016

Improve SharePoint 2013 search relevance

You want to ensure that the search results that are returned to the user match what the user wanted to find and that the results that are returned on the first page are the most relevant, so the user does not have to look through several pages of results to find the best matches for their search ?

This is called Search Relevancy!

In this post we are going to learn how to customize SharePoint 2013 search relevance. We will see how to create and deploy custom ranking models and how to test them.




1 - What you need to do this tutorial
You need:
  • A SharePoint 2013 Enterprise on premise environment
  • Visual Studio 2013 installed on this environment

2 - Definition
The ranking model is an XML file that SharePoint uses to assign a rank for each search result . The display of results is made according to this rank. To adjust the relevance of research in SharePoint 2013 you thus need to write a custom ranking model and deploy it so that it takes into account the preferences of the user in matter of display of results .

3 - Start your ranking model customization based on a SharePoint ranking model template


To get the default ranking model template, log on any server in the SharePoint 2013 server farm with Administrator rights.
Create a directory, for example c: \ rankModel
Open a Powershell for SharePoint command prompt and enter the following lines:

$ssa = Get-SPEnterpriseSearchServiceApplication
$owner = Get-SPenterpriseSearchOwner -Level ssa
$defaultRankingModel = Get-SPEnterpriseSearchRankingModel -SearchApplication $ssa -Owner $owner | Where-Object { $_.IsDefault -eq $True }
$twoLinearStagesRankingModel.RankingModelXML > c:\rankmodel\defaultRankingModel.xml
Note: Microsoft recommends to use a two linear ranking model (it is a different and newer xml structure). To use it, you have, at least, to update your SharePoint Farm with the SharePoint Server 2013 cumulative update that was published in March 2014.



After having executed the previous SherPoint Powershell instructions, you will see appear the requested ranking model as an xml file in the c:\rankmodel folder.
Then, open the defaultRankingModel.xml, xml file into a text editor.


Note that the ranking model template is identified by an unique GUID.
4 - Test the ranking model template (first tool)
I developed a tool, the SPsearchRankingModelTester that can test the relevance of the search results according to a given ranking model (Thanks to Johan Olivier).
It is a console application.
This tool allows to easily test an application after having deployed a custom ranking model without having to register each time the new rankig model in search results web part which is tedious for testing.
You can find the package ready to use in CodePlex: SPSearchRankingModelTester-package .
You can find the source code of this application in the codePlex project, or in my GitHub repositery: SP2013searchRankingModelTester

4.1 Deploying SP2013searchRankingModelTester

Copy the downloaded package on a server of the SharePoint 2013 Farm, unzip it.
Modify the exe.config file: change the "value" attribute of the SPsiteAddress key and use the site collection where you want to test the search with a specific ranking model. For exemple, you can use the Url of a your SharePoint 2013 search center.

4.2 Using the SP2013searchRankingModelTester

Double click the .exe file.
The console application is opening and displaying your site collection Url (the one referenced in the exe.config file).
You are then, prompted for a request. Type your request.
Then, you are prompted for the GUID of your ranking model.

Make a right-click on the top of the window to paste the GUID you have previously copied from the xml file:



In the next screen shot, I use the GUID of the SharePoint 2013 default ranking model.
Once the ranking model GUID typed and the "enter" key pressed, the console application displays the 50th first relevant results according to the ranking model you specified.



5 - Tune your ranking model with rank features

There is several ways to customize a ranking model. The xml syntax and the ways to focus on a special point for relevance is not easy to understand. You can refer first to the official Microsoft documentation. I am first giving a quick overview of it, then I will explain how to test your changes with several tools.

Roughly you can act on:
  • BM25 rank feature ranks items
    You specify a property like "body" (the content of the crawled or indexed item) or "Title" (the title of the crawled or indexed item) and give that property a weight in order the relevance to more or less take into account this property.
    You can also specify the document length normalization for each property: a title is usually composed with few words although the body is rather long and the query is also usually composed with a few words. So you can tell your ranking model that a single occurence of one of the keywords of the query in the body has the same importance than a single occurence of one of the keywords of the query in the title

  • Weight groups
    You tune this in the Advanced Search Settings, using the search schema feature in the Search service application.
    For example, you create a new managed property of the type string that contains about ten words or less. You consider this new managed property to be about as important as the existing managed property Title. In that case, you should map the new managed property to context 1.
    Another example. You create a managed property of the type string that contains lots of words, for example a description of something. You should map this new managed property to context 7 because it is similar to the managed property Body, both in length as well as in importance.

  • static rank feature ranks items
    The static rank feature ranks items based on numeric managed properties that are stored in the search index. The numeric managed properties used for relevance rank calculation in static rank features must be of type Integer and set to Refinable or Sortable in the search schema. You can’t use multivalued managed properties in combination with the static rank feature.
    For example, UrlDepth is a typical static rank: the less url to reach the result is long, the more that result is relevant.
    Another example: you can make your content to be rated by users and make a static rank with this rating.

  • Bucketed static
    The bucketed static rank feature ranks documents based on their file type and language. You can tell your model that a .pdf in english is more relevant than a .docx in french.

  • Proximity rank feature
    The proximity rank feature ranks items depending on the distance between query terms inside the full-text index. The rank score is boosted if two query terms appear in the same managed properties within the full-text index. Proximity calculations are expensive in terms of disk activity and CPU consumption; as a result, proximity boost is carried out only during the second stage of the default SharePoint Server 2013 rank model (if available).

  • The dynamic rank feature
    It seems to use this if you want to boost a specific property:
    The dynamic rank feature ranks an item depending on whether the query property matches a given managed property. If there is a match, the item’s rank score is multiplied with a specific value to distinguish that particular item. The weight attribute is used to control how much this feature affects the overall rank score.
    Note
    The dynamic rank feature is not customizable; it’s for internal use only. However, if you install the SharePoint Server 2013 cumulative update of August 2013, the AnchortextComplete rank feature is a customizable dynamic rank feature that is part of the default ranking model.

  • Freshness
    The default SharePoint 2013 ranking model doesn’t boost the rank of search results based on their freshness. You can achieve this by adding a new static rank feature that combines information from the LastModifiedTime managed property with the DateTimeUtcNow query property, using the freshness transform function. The freshness transform function is the only transform that you can use for this freshness rank feature, because it converts the age of the item from an internal representation into days.


6 - Testing a custom ranking model

6.1 Description of the custom ranking model

In the Microsoft documentation, you will find two samples of customized ranking model. Let's show how to deploy and test the second one.

This ranking model with one linear stage contains these four rank features:
  • BM25 This rank feature is based on managed properties Title and body; the w attribute for title is set so that hits of query terms in Title are two times (2x) more important than hits of query terms in body.

  • UrlDepth This rank feature is based on the UrlDepth managed property, which is available by default in SharePoint installations. UrlDepth contains the number of backslashes (\) in the URL of a document. The inverse rational (InvRational) transform ensures that documents with shorter URLs receive higher rank scores.

  • TitleProximity This rank feature boosts documents if some of the query terms occur close to each other in the title of these documents.
    InternalFileType This rank feature boosts documents of type HTML, DOC, XLS, or PPT. The names of the buckets in the definition of the rank model are provided for readability only.
<?xml version="1.0"?>
<rankingmodel2stage name=" RankModel2"
                    description="Rank model -- example 2"
                    id="DE48A3A1-67CE-44A2-9712-E8A5128787CF"
                    xmlns="urn:Microsoft.Search.Ranking.Model.2NN">
    <rankingmodel2nn id="A0A030D1-805D-437E-A001-CC151ED7473A" precalcEnabled="0">
        <hiddennodes count="1">
            <thresholds>
                <threshold>0</Threshold>
            </Thresholds>
            <layer2weights>
                <weight>1</Weight>
            </Layer2Weights>
        </HiddenNodes>
        <rankingfeatures>
            <bm25main name="BM25" k1="1">
                <layer1weights>
                    <weight>1</Weight>
                </Layer1Weights>
                <properties>
                    <property name="Title" propertyName="Title" w="2" b="0.5" />
                    <property name="body" propertyName="body" w="1" b="0.5" />
                </Properties>
            </BM25Main>
            <static name="UrlDepth" propertyName="UrlDepth" default="1">
                <transform type="InvRational" k="1.5"/>
                <layer1weights>
                    <weight>0.5</Weight>
                </Layer1Weights>
            </Static>
            <minspan name="TitleProximity" propertyName="Title" default="0" maxMinSpan="1" isExact="0" isDiscounted="0">
                <normalize SDev="1" Mean="0"/>
                <transform type="Linear" a="1" b="-0.5" maxx="2"/>
                <layer1weights>
                    <weight>1.2</Weight>
                </Layer1Weights>
            </MinSpan>
            <bucketedstatic name="InternalFileType" propertyName="InternalFileType" default="0">
                <bucket name="http" value="0">
                    <hiddennodesadds>
                        <add>1.5</Add>
                    </HiddenNodesAdds>
                </Bucket>
                <bucket name="doc" value="1">
                    <hiddennodesadds>
                        <add>2.5</Add>
                    </HiddenNodesAdds>
                </Bucket>
                <bucket name="ppt" value="2">
                    <hiddennodesadds>
                        <add>0.5</Add>
                    </HiddenNodesAdds>
                </Bucket>
                <bucket name="xls" value="3">
                    <hiddennodesadds>
                        <add>-3.5</Add>
                    </HiddenNodesAdds>
                </Bucket>
            </BucketedStatic>
        </RankingFeatures>
    </RankingModel2NN>
</RankingModel2Stage>

6.2 Custom ranking model deployment

Copy the previous xml in a file and name this file RankingModel2.xml. Copy this file in the previous folder c:\rankmodel.
Then, execute the following Powershell for SharePoint instructions:

$myRankingModel = Get-Content c:\rankmodel\ RankingModel2.xml
$myRankingModel = [String]$myRankingModel
$ssa = Get-SPEnterpriseSearchServiceApplication
$owner = Get-SPenterpriseSearchOwner -Level ssa
$newrm = New-SPEnterpriseSearchRankingModel -SearchApplication $ssa -Owner $owner -RankingModelXML $myRankingModel

Here is the screenshot for control:




6.3 testing the custom ranking model

To test the custom ranking model, we just have to use the previous tool. So double click on the .exe and this time after having typed the query, type the GUID of the custom ranking model:

DE48A3A1-67CE-44A2-9712-E8A5128787CF

You will see that the results are sorted in a completely different way accorded to the new and custom ranking model:



As I wrote before, understanding the syntax of the ranking model is a difficult work to achieve. By being able to test quickly the modifications of a custom ranking model you can improve your knowledge of the syntax, rules, formula and so on.

We are going to see more tools to do this...

7 - More tools to learn how to customize a SharePoint 2013 search ranking model

7.1 Tools overview and prerequisites

The two tools that we are going to demonstrate are Add-Ins for SharePoint ( formerly called Apps) .
I will not deny that the requirements to use these tools are difficult to carry out even for a SharePoint developer.
It takes a very good knowledge of SharePoint administration to use these tools because:
  • The Apps Settings for a SharePoint environment is difficult (especially on a local environment because of, for example, the creation the subdomain) and does not always work at the first time. You have sometimes to delete the service and recreate it for making it work.
  • One of these tools requires at least ( without it being mentioned in its documentation!), the activation of the User Profile Service Application that requires a lot of expertise ( especially for a local SharePoint 2013 environment)
However, here is a demonstration of tools and assets that can be drawn.

7.2 Mavention Explain Rank

Overview
SharePoint 2013 has the ExplainRank.aspx page which is a truly hidden gem, that displays the rankdetail for a single item in a more comprehensible way. However, this page is not working! (see Waldek Mastykarz's post on the subject.
Fortunately, there is a similar and free(!) tool (a SharePoint Add-In) that allows you to do the same.
This Add-In allows to query and for each result to understand the different values ​​assigned by the ranking model and explains its place in the ordering of results.
This is very educational when we want to understand how the ranking model is working.

Installation
Log in as Administrator on the research center of the SharePoint environment. Use the upper left menu to add the Add-In (app)




In the "Apps" search box enter the name of the publisher " Mavention " to display the catalog.



Locate « Explain Rank » Add-In.



Click the Add-In to deploy it and give it the appropriate permissions.



Once deployed, the add-in appears in usable the Add-In ( Apps) and can be used .





Using the "Explain Rank" Mavention Add-In

Open the Add- In home page and enter a query.



Start the query



Before each result a radio button helps to understand how the ranking model has awarded this rank for this result : what used properties and weight related to the presence of a particular word of the request in a particular property.



This allows us to better understand how a standard or a custom ranking model is acting in the ordering of results.

7.3 Ranking Model Tuning

Overview

This Add-in is more difficult to use than the previous one. It helps doing what we have done previously that is to mean "creating a custom ranking model", but in a guided manner , and without having to manipulate the xml.

Installation

Proceed as for the Add-In " Explain Rank " of Mavention , but instead of entering Mavention in the search box, enter the name of the Add-In " Ranking Model Tuning ".
Then install the Add-In .




Using the "Ranking Model Tuning" Add-In

The home page of the Add-In presents different ranking models that can serve as a basis to create a customized one. You just have to copy one to work on it.



Once the copy made we can start customizing the copied ranking model.




There are two macro- features : "Adding a judgment sets " or "Adding a rank features"





When you copy an existing ranking model, the new ranking model contains the same rank features and weights as in the base model. You can add more managed properties as additional rank features, remove existing features, or tune the weight of existing features.






Conclusion

This tool helps improve understanding of the xml of a ranking model since he automatically generates the xml based on instructions given by the Add-In UI. Then simply retrieving the new ranking model in xml format to understand how to transform the rules in a well formatted and consistent ranking model xml.
At best, the tool may be suitable for customisations needs of a ranking model without any need to work on the xml.
These various tools and practices can be used to define several approaches to constitute the ranking model appropriate for your needs, although it seems to require a large amount of time for testing and researching to achieve real expertise and mastery of the subject.


Aknowledgements