tag:blogger.com,1999:blog-79396260099066440842024-03-18T02:48:03.006-07:00The Data InstructorsAnonymoushttp://www.blogger.com/profile/00885379426770757526noreply@blogger.comBlogger14125tag:blogger.com,1999:blog-7939626009906644084.post-4940026749983036432017-05-07T16:48:00.002-07:002017-05-07T16:48:35.646-07:00How to get free Azure credits for more than one month and use them efficiently<b><span style="font-size: large;">Introduction</span></b><br />
<div style="text-align: justify;">
<div style="text-align: left;">
A good way to learn about cloud development is building prototypes. The most important thing that you need when you want you want to build prototypes in the cloud is a cloud environment and preferably at an affordable price. How does the cost of a cloud environment compare to building prototypes on your local machine?</div>
<div style="text-align: left;">
When you are building prototypes on your local machine, you have first invested in your hardware, your computer and depending on the prototype you want to develop you have invested a certain amount of money in software. Next you are still have your monthly costs like your internet connection and your electricity bill. This means that you immediately know how much you are spending.</div>
<div style="text-align: left;">
When you are building prototypes in the cloud, you initially start with a low bill, because you have more a pay as you go model, but if you don't pay attention this bill can grow faster than the amount that you would have invested in your bare metal locally. You have however the possibility to play with technologies that are not easily available on your desktop environment. Examples of these are machine learning software and business intelligence solutions.</div>
<div style="text-align: left;">
In this blog post I will go more into detail about how you can get free cloud credits for Azure and things to consider when you are building prototypes with virtual machines.</div>
<br />
<b style="font-size: x-large;">On a quest for credits</b><br />
<div style="text-align: left;">
Initially, when I started with building prototypes for the cloud I used a mix of emulators, Azure Machine Learning Studio and the Azure free trial credits for one month of $200. However to keep building prototypes on a continuous basis I needed a more long term supply of credits. Luckily after an extra search on the internet I found the <a href="https://www.visualstudio.com/dev-essentials/" target="_blank">Visual Studio Dev Essentials</a> program. This program provides a lot of free resources for developers but the most important aspect for me were the $25 or 40 CAD of Azure credits a month for one year. </div>
</div>
<div style="text-align: justify;">
<div style="text-align: left;">
In the rest of this blog post I will describe some lessons learnt using these credits in the most optimal way. I will focus on Virtual Machines in this blog post and how the Azure Portal also will help you to identify the highest costs of your cloud consumption.</div>
</div>
<b><span style="font-size: large;"><br /></span></b>
<b><span style="font-size: large;">How the portal tracks the consumption of your monthly credits</span></b><br />
<div style="text-align: justify;">
<div style="text-align: left;">
The Azure Portal tracks the amount of credits that you have used and makes a forecast about the credits that you will use by the end of the billing period. It also provides a breakdown about the resources that are using the most.</div>
<div style="text-align: left;">
By inspecting these costs on a daily basis you can adjust your spending costs. Next you can also validate whether you really understand how your are spending your credits. For example if you have deleted or stopped a resource and your spending is still not significantly lower, you might still have some ghost parts of this resource living in your Azure account and eating away your credits.</div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-4x2m10DP4pE/WNxbexOezuI/AAAAAAAAAUc/hiblFwdpDD8UOg2kvdqETcvddrO5FHR1gCLcB/s1600/cost_breakdown.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="145" src="https://3.bp.blogspot.com/-4x2m10DP4pE/WNxbexOezuI/AAAAAAAAAUc/hiblFwdpDD8UOg2kvdqETcvddrO5FHR1gCLcB/s400/cost_breakdown.png" width="400" /></a></div>
<b><span style="font-size: large;"><br /></span></b>
<span style="font-size: large;"><b>How to optimize the cost of Virtual Machines</b></span><br />
<b>What do we want to learn from experimenting with Virtual Machines in the cloud?</b><br />
<div style="text-align: left;">
<span style="text-align: justify;">In Azure you can choose from various versions of both </span><a href="https://azure.microsoft.com/en-us/services/virtual-machines/?b=17.08a" style="text-align: justify;" target="_blank">Windows</a><span style="text-align: justify;"> and </span><a href="http://blog.datainstructors.com/2017/03/setting-up-linux-virtual-machines-in.html" style="text-align: justify;" target="_blank">Linux</a> operating systems<span style="text-align: justify;">. You can also select the flavor that you want and you need to select different options like the type of drives for the virtual machine, the region and the size and the cost of the virtual machine. Finally you can still select some extra options like the use of managed disks, the availability set and a diagnostic storage account that might add in extra costs.</span></div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
<b>What is the most cost effective option?</b></div>
<div style="text-align: left;">
<span style="text-align: justify;">It is important that you have a clear goal about what you want to achieve with the prototype that you are building when you are selecting you virtual machines. For example if you want to understand the various ways about how to connect with a virtual machine or want to experiment with Network interfaces, you don't need a machine with 14 GB of memory, 2 cores and 4 data disks. In the beginning of the billing period you also want to be more more conservative than the end of the billing period. If you however have a more computing intensive project in mind working with more computing power might reduce the number of resources that you are spending.</span></div>
<div style="text-align: left;">
<br /></div>
<i>Be careful when selecting the size of your VM</i><br />
One of the first things that I learnt was that when you are selecting the size of your virtual machine, don't select one of the recommended options as you can see below. However select the View all option in the upper right corner.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-cjV00erEzZA/WNxuCgYvgZI/AAAAAAAAAUs/ohJvGncjsM8pQ1uF0OgQTt-jXZfP0y45gCLcB/s1600/choose_size.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="200" src="https://3.bp.blogspot.com/-cjV00erEzZA/WNxuCgYvgZI/AAAAAAAAAUs/ohJvGncjsM8pQ1uF0OgQTt-jXZfP0y45gCLcB/s320/choose_size.png" width="320" /></a></div>
Next scroll down and you will be presented with the A0 Basic and the A1 Basic which are more affordable options to play with. If these cheap options are not available. Select another region for your virtual machine till you have found the A0 and A1 options.<br />
<b><br /></b>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-jVWrOUKKTaw/WNxvHJ85IaI/AAAAAAAAAU0/FEAiqD0LEIAhQr7Ug8-l5AXOyOG-ZgALwCLcB/s1600/cost_breakdown2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://2.bp.blogspot.com/-jVWrOUKKTaw/WNxvHJ85IaI/AAAAAAAAAU0/FEAiqD0LEIAhQr7Ug8-l5AXOyOG-ZgALwCLcB/s320/cost_breakdown2.png" width="280" /></a></div>
<br />
<i><br /></i>
<i>Understand the charges when a virtual machine is turned off</i><br />
<div style="text-align: justify;">
<div style="text-align: left;">
You can turn off a virtual machine from the Azure Portal and in this case nothing is being charged for the compute hours. When you turn on and off your local machine, you expect it still to work. This means that your operating system and all the the software is still left on your drives. In the cloud this means that you need to keep paying for the storage of your virtual machine. </div>
<div style="text-align: left;">
There are different types of availability and redundancy that can influence the cost significantly. When you also want to make sure that your disk are safe from disaster, you also will need to add in extra backup options.</div>
</div>
<div style="text-align: justify;">
<div style="text-align: left;">
Next you can also add extra data disks to your virtual machine, these disks are persistent when your virtual machine has been deleted. So make sure also to delete these disks in case that you don't need them any more. Therefore when you have deleted a virtual, make sure that you keep monitoring the costs and that you are seeing the cost reduction that you expected.</div>
</div>
<div style="text-align: justify;">
<div style="text-align: left;">
Finally there are also still some costs involved with the public ips for the various virtual machines. However the costs of these are small in comparison with the disks. But when you want to remove all the traces from a virtual machine, you will need to delete these ones separately. </div>
</div>
<br />
<b><span style="font-size: large;">Conclusion</span></b><br />
In this blog post we discussed a strategy that gives you free cloud credits for a year. Next we discussed how you can use these cloud credits in an optimal way when you are using virtual machines in the cloud. Good luck with your journey to the cloud.Anonymoushttp://www.blogger.com/profile/00885379426770757526noreply@blogger.com6tag:blogger.com,1999:blog-7939626009906644084.post-16213462782217341342017-04-02T15:21:00.000-07:002017-04-02T15:26:52.732-07:00An easy trick to get a csv-file in Azure SQL Server with Azure Machine Learning Studio without coding<b><span style="font-size: large;">Introduction</span></b><br />
<div style="text-align: justify;">
In this blog post I will describe a method that you can use to import a local csv-file to the cloud in Azure SQL Server without writing any code. Although you can also do this with SQL Server Management Studio (SSMS), this is not always an easy option for example if you don't have SQL Server Management Studio available because you are working on a Mac or a Linux computer. Another reason might be that you are working from another location than usual and your IP hasn't been granted permission to access the Azure SQL Server database.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-8Po9h7-IkwI/WOERiNvz3OI/AAAAAAAAAXs/e_bVDp8r7LcRNnbjZeZ93UrBbhKsplYzQCEw/s1600/azureml.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://1.bp.blogspot.com/-8Po9h7-IkwI/WOERiNvz3OI/AAAAAAAAAXs/e_bVDp8r7LcRNnbjZeZ93UrBbhKsplYzQCEw/s320/azureml.png" width="293" /></a></div>
<br />
<br />
<div style="text-align: justify;">
In the rest of this post, I will explain how you can execute this trick yourself. The most important things that you will need are a csv file, <a href="https://studio.azureml.net/" target="_blank">Azure Machine Learning Studio</a> and a <a href="https://azure.microsoft.com/en-us/pricing/member-offers/vs-dev-essentials/" target="_blank">subscription to Azure</a>. If you have already deployed an Azure SQL Database in the cloud, you can fast forward to forward to Preparing Azure Machine Learning Studio.</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br />
<br />
<b><span style="font-size: large;">Make a SQL Database in the Azure Portal</span></b><br />
<b><span style="font-size: large;"><br /></span></b>
<br />
<div style="text-align: justify;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-_2Uln8aHmhw/WOAYmJQzPlI/AAAAAAAAAVc/FHMj7CE3odooe2wig1Wv1l2hfTx-6rbHACLcB/s1600/database_selector.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="90" src="https://3.bp.blogspot.com/-_2Uln8aHmhw/WOAYmJQzPlI/AAAAAAAAAVc/FHMj7CE3odooe2wig1Wv1l2hfTx-6rbHACLcB/s400/database_selector.png" width="400" /></a></div>
<br />
We first make a database from the Azure Portal. We are going to use this database to build a prototype and the main functionality that we need is to access the database, to perform an INSERT and a SELECT statement. Therefore, we will generate the least expensive database. </div>
<div style="text-align: justify;">
We login to the Azure Portal and we select SQL databases from the left column in the portal. This brings us to the configuration screen as we can see below and we make the choices to define the name of the database and define a server where the database will live.</div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-R79AcROksDg/WOAZR04Zv-I/AAAAAAAAAVk/9eQocRK0FNYKT59VWMnMoK7EbLEFA2nOgCLcB/s1600/configuration_part1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="379" src="https://4.bp.blogspot.com/-R79AcROksDg/WOAZR04Zv-I/AAAAAAAAAVk/9eQocRK0FNYKT59VWMnMoK7EbLEFA2nOgCLcB/s640/configuration_part1.png" width="640" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Next we will need to make a choice about the cost of the database. You can make a choice between Basic, Standard, Premium and PremiumRS. You can select both the <a href="https://docs.microsoft.com/en-us/azure/sql-database/sql-database-what-is-a-dtu" target="_blank">Database Transaction Units </a>(DTU) which determine the computational speed and the storage which determines the space that your database will use. For this experiment we will work with the Basic option and we select the lowest DTU and we will keep the suggested storage space. The cost of this database is 6.07 CAD a month. </div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-Cq4bIYEq97E/WOAZe0CZDxI/AAAAAAAAAVo/FjoDz6mVKjoqPyte_6-8-oNcdom6yO0VwCLcB/s1600/performance.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="233" src="https://1.bp.blogspot.com/-Cq4bIYEq97E/WOAZe0CZDxI/AAAAAAAAAVo/FjoDz6mVKjoqPyte_6-8-oNcdom6yO0VwCLcB/s640/performance.png" width="640" /></a></div>
<br />
<br />
<div style="text-align: justify;">
When you have put in all these configurations it is a good idea to select the Pin to Dashboard checkbox. This allows you to easily monitor the progress of the deployment of a resource or another action that you initiated in Azure. Just wait a few minutes and your database will be deployed to Azure.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-Ppf2x_Sj11k/WOAaylIizzI/AAAAAAAAAV0/jupkRRnC7ZgKj6h3t0Ypp9nxbK8VF8P7QCLcB/s1600/pin_to_dashboard.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="100" src="https://1.bp.blogspot.com/-Ppf2x_Sj11k/WOAaylIizzI/AAAAAAAAAV0/jupkRRnC7ZgKj6h3t0Ypp9nxbK8VF8P7QCLcB/s320/pin_to_dashboard.png" width="320" /></a></div>
<br />
<b><span style="font-size: large;">Preparing Azure Machine Learning Studio</span></b><br />
<b>Uploading your local file to the cloud as a dataset</b><br />
<div style="text-align: justify;">
First, you will need to login to <a href="http://studio.azureml.net/" target="_blank">Azure Machine Learning Studio</a>. In case you haven't used this before you can use the same Microsoft account that you use for your Azure account. If you select the Free Workspace, there will be no extra charges involved for using Azure Machine Learning Studio.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-99VMo3kzygE/WOAixWEec5I/AAAAAAAAAWc/f2ZkSORqYzIgkpcQm5z_vDBknK8QfU2awCLcB/s1600/DatasetSelector.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://3.bp.blogspot.com/-99VMo3kzygE/WOAixWEec5I/AAAAAAAAAWc/f2ZkSORqYzIgkpcQm5z_vDBknK8QfU2awCLcB/s320/DatasetSelector.png" width="169" /></a></div>
<br />
<div style="text-align: justify;">
First you want to make a dataset in Azure Machine Learning Studio from your csv-file. This will mean that your dataset will be stored in your datadrive that belongs to your Azure Machine Learning account Studio. Therefore, select <b>DATASETS</b> in the left column. Next you can make a new dataset by selecting the +<b> NEW</b> button on the bottom of the page. After you have clicked this button, you will be able to upload your csv-file to the cloud.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-Uo0YGvYQSwU/WOAkeRUCeAI/AAAAAAAAAWo/wKoZVC-tqrQmURhnVSQm3a8WWFbZBELTgCLcB/s1600/available_datasets.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="142" src="https://4.bp.blogspot.com/-Uo0YGvYQSwU/WOAkeRUCeAI/AAAAAAAAAWo/wKoZVC-tqrQmURhnVSQm3a8WWFbZBELTgCLcB/s400/available_datasets.png" width="400" /></a></div>
<br />
<br />
<b>Building the Experiment that uploads the csv file to the database</b><br />
Now, let's switch in Azure Machine Learning Studio to Experiments in the left column and create a new Experiment by clicking on the <b>+ NEW</b> button on the bottom of the screen. In the wizard that follows, you can select "Blank experiment".<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-lrh_llJeNSg/WOAlV_XFJVI/AAAAAAAAAWw/dZXFZeDbp4AUXZ1oAmo7X2IPmhaUyFlqACLcB/s1600/StartExperiment.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="147" src="https://4.bp.blogspot.com/-lrh_llJeNSg/WOAlV_XFJVI/AAAAAAAAAWw/dZXFZeDbp4AUXZ1oAmo7X2IPmhaUyFlqACLcB/s400/StartExperiment.png" width="400" /></a></div>
<br />
<br />
You can drag over your csv-file to the middle of the page. If you right click on the circled 1 you can select Visualize.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-hsPHnnN4vJ8/WOAoVcSgLxI/AAAAAAAAAW8/VOga3kxv7oMquAUvEv6RXwoAaczK5yEFwCLcB/s1600/visualize.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="180" src="https://3.bp.blogspot.com/-hsPHnnN4vJ8/WOAoVcSgLxI/AAAAAAAAAW8/VOga3kxv7oMquAUvEv6RXwoAaczK5yEFwCLcB/s320/visualize.png" width="320" /></a></div>
<br />
We now see an overview of the dataset and we can visually inspect whether the data has been imported correctly.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-5HJ-ipAvW7I/WOAptFN8k2I/AAAAAAAAAXI/1VT8EO8wyUMU02T8u_xNZyF7hihIzQUKwCLcB/s1600/column_selection.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="220" src="https://2.bp.blogspot.com/-5HJ-ipAvW7I/WOAptFN8k2I/AAAAAAAAAXI/1VT8EO8wyUMU02T8u_xNZyF7hihIzQUKwCLcB/s400/column_selection.png" width="400" /></a></div>
<br />
Now we will add in a <b>Column Selecto</b>r that will select all the columns that we want to transfer to the database.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-KsSDYBarOLU/WOAqd2vQQ8I/AAAAAAAAAXQ/jQN7Kkwf-nAHYc_v6JMP6zh75yCPAUL6ACLcB/s1600/SelectColumns.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="212" src="https://2.bp.blogspot.com/-KsSDYBarOLU/WOAqd2vQQ8I/AAAAAAAAAXQ/jQN7Kkwf-nAHYc_v6JMP6zh75yCPAUL6ACLcB/s400/SelectColumns.png" width="400" /></a></div>
<br />
Finally we will select an Export Data box and our experiment will look as below.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-AKGeXo7awzk/WOArFhC0oZI/AAAAAAAAAXY/ozEPaE4l4wIGbHxXPyjmcTlYSpOFAk2MwCLcB/s1600/ExportDataExperiment.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://4.bp.blogspot.com/-AKGeXo7awzk/WOArFhC0oZI/AAAAAAAAAXY/ozEPaE4l4wIGbHxXPyjmcTlYSpOFAk2MwCLcB/s1600/ExportDataExperiment.png" /></a></div>
<br />
To make the red exclamation mark disappear you will need to provide the information of your database. With Data table name you can name the resulting table name in the database, in case this table is not available yet it will be automatically generated.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-4nDQjiYzcr0/WOEdEBZx8SI/AAAAAAAAAX4/UZ52NpZqtL0OAumEnJDtmVNZTPylrjA2wCLcB/s1600/ExportDataExperiment2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="https://3.bp.blogspot.com/-4nDQjiYzcr0/WOEdEBZx8SI/AAAAAAAAAX4/UZ52NpZqtL0OAumEnJDtmVNZTPylrjA2wCLcB/s400/ExportDataExperiment2.png" width="352" /></a></div>
<br />
When the red exclamation mark has disappeared, you can run the experiment by selecting the <b>RUN</b> button in Azure Machine Learning Studio. In case you run into errors, make sure to check whether your have written down the column names correctly or whether you have made some typos in your password.<br />
<br />
<b>Validation that the data has arrived in your database</b><br />
Create now a new experiment and drag an import button to this experiment and configure this again that you will be able to access your Azure SQL Database. Type a simple select statement as your database query.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-V2gkIeB97wY/WOF2K6qNhwI/AAAAAAAAAYQ/UKluC_Xly6kTT459VGgA8z4m29XWs-hdACLcB/s1600/validation_experiment.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="262" src="https://3.bp.blogspot.com/-V2gkIeB97wY/WOF2K6qNhwI/AAAAAAAAAYQ/UKluC_Xly6kTT459VGgA8z4m29XWs-hdACLcB/s400/validation_experiment.png" width="400" /></a></div>
<br />
Press the RUN button again and select next Visualize from he Circled 1. You will see that you now can retrieve the information from your csv-file from the database.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-dQqDWqAZFGI/WOF23wd1pnI/AAAAAAAAAYY/Gkv1VQtiprMOkqxBbAVWzv8kh8rz7Pc3gCLcB/s1600/result_dataset.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="233" src="https://1.bp.blogspot.com/-dQqDWqAZFGI/WOF23wd1pnI/AAAAAAAAAYY/Gkv1VQtiprMOkqxBbAVWzv8kh8rz7Pc3gCLcB/s400/result_dataset.png" width="400" /></a></div>
<br />
<br />
<span style="font-size: large;"><b>Conclusion</b></span><br />
Today, we discussed today a strategy to upload a local csv-file into a table in Azure SQL Server with an uncommon use of Azure Machine Learning Studio. For another Azure Machine Learning Studio hack see<br />
<br />
<ul>
<li><a href="http://blog.datainstructors.com/2017/01/how-to-extract-images-that-are.html" target="_blank">How to extract images that are generated by Azure Machine Learning Studio</a></li>
<li><a href="http://blog.datainstructors.com/2017/03/how-to-call-web-service-generated-from.html" target="_blank">How to call a web service generated from Azure Machine Learning Studio from Ruby</a>.</li>
</ul>
<br />
<br />
<br />Anonymoushttp://www.blogger.com/profile/00885379426770757526noreply@blogger.com3tag:blogger.com,1999:blog-7939626009906644084.post-20515537142834965912017-03-26T09:32:00.000-07:002017-04-02T15:23:09.315-07:00Some Azure security settings don't really mean what you think they mean<b><span style="font-size: large;">Introduction</span></b><br />
<div style="text-align: justify;">
In today's blog we will focus on how to reset a public key for a Linux VM in Azure. Spoiler Alert!!! Using the <i><b>Reset password</b></i> functionality from the Azure Portal doesn't have the functionality that you would expect from a password reset. We will explain this further in this blog.</div>
<div style="text-align: justify;">
First, we will explain how you generate a VM with public key access in the Azure portal. Next, we will explain what happens when you reset your public key for a VM using the <b><i>Reset Password </i></b>functionality in the portal. Next we will explain how you can actually remove the previous public key to get a complete reset functionality. Finally we will also explain what happens with the public keys when a VM gets redeployed.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-RtQr4soYQhI/WNlb04Lwx1I/AAAAAAAAAUE/Tymg409FbZAESYisacvmsdcjUKpbKC0CACLcB/s1600/reset_password.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="56" src="https://2.bp.blogspot.com/-RtQr4soYQhI/WNlb04Lwx1I/AAAAAAAAAUE/Tymg409FbZAESYisacvmsdcjUKpbKC0CACLcB/s320/reset_password.png" width="320" /></a></div>
<br /></div>
<br />
<span style="font-size: large;"><b>Setting up a Linux VM in Azure through the portal</b></span><br />
<b>How to setup the VM</b><br />
<div style="text-align: justify;">
To setup a Linux VM in Azure, you can go to the Azure Portal. For the Authentication Type you can choose <i><b>SSH public key</b></i>. To generate a public/private key pair you can use for example <a href="http://www.chiark.greenend.org.uk/~sgtatham/putty/" target="_blank">PuTTYgen</a>. You can next choose all the other parts of the VM. In the portal you also need the username that will use the VM. In this example we will use the username <b><i>mary</i></b> and Mary will provide her first public key <b><i>mary1</i></b>.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-rMFiFqTrFCY/WM6fhA8t2UI/AAAAAAAAASI/nLVyl3iino4UAl4iAp2ofaAQRLsSSi48wCLcB/s1600/generate_vm1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="360" src="https://2.bp.blogspot.com/-rMFiFqTrFCY/WM6fhA8t2UI/AAAAAAAAASI/nLVyl3iino4UAl4iAp2ofaAQRLsSSi48wCLcB/s400/generate_vm1.png" width="400" /></a></div>
<br />
<b>Logging in to the VM</b><br />
<div style="text-align: justify;">
You can next choose the ssh client of your choice to access the VM when you provide the IP of the machine together with the private key that corresponds with the public key pair. In this way we can successfully log into the machine.</div>
<br />
<b>What happened behind the scenes?</b><br />
<div style="text-align: justify;">
When you use the <b><i>SSH public key</i></b> authentication type when you generate your VM, two things happen behind the scene.</div>
<div style="text-align: justify;">
1. In your VM, a .<b><i>ssh</i></b> directory has been generated. In this .ssh directory, a file <i><b>authorized_keys</b></i> has been generated. In this file you will see the public key that you provided in the Portal.</div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-SFs5kt3MLmQ/WM7rRHaiiCI/AAAAAAAAASY/gvfQLh69aF0MDSh4MHKcwCF1UCA2GHeGQCLcB/s1600/authorized_keys1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="403" src="https://2.bp.blogspot.com/-SFs5kt3MLmQ/WM7rRHaiiCI/AAAAAAAAASY/gvfQLh69aF0MDSh4MHKcwCF1UCA2GHeGQCLcB/s640/authorized_keys1.png" width="640" /></a></div>
<br />
<br />
<div style="text-align: justify;">
2. In the Portal under the <i><b>Automation script</b></i> tab, various templates were generated. You can use these templates later on when you want to automate the deployment of your VMs. In the picture below you can see a fragment from the Template file and you can see in the highlighted <i><b>osProfile</b></i> section that the user mary can log in with the provided public key and that password authentication has been disabled.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-IKHZfIaJnR0/WM7uU_Yk87I/AAAAAAAAASk/tIy6XPozEuACF8TYDzddAfTOScbexgB9QCLcB/s1600/where_to_find_login_information_cropped.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="209" src="https://4.bp.blogspot.com/-IKHZfIaJnR0/WM7uU_Yk87I/AAAAAAAAASk/tIy6XPozEuACF8TYDzddAfTOScbexgB9QCLcB/s640/where_to_find_login_information_cropped.png" width="640" /></a></div>
<br />
<b><span style="font-size: large;">Sometime bad things happen</span></b><br />
<div style="text-align: justify;">
Unfortunately your laptop on which you had installed your keys got stolen and therefore you need to reset all your passwords and off course you also need to revoke public key access for your public keys that you had on your VM.</div>
<br />
<span style="font-size: large;"><b>Resetting the public key through the Azure Portal </b></span><br />
<b>The steps to follow in the portal</b><br />
<div style="text-align: justify;">
When you go to the Azure Portal and go to your VM, you can select <b><i>Reset password</i></b> from the left column. In the screenshot below you can see the information that is provided to you and the information that you need to fill in to reset your public key. In this case we have added in public key mary2.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-4JDZOqEjYWY/WM7xo613MWI/AAAAAAAAAS4/SCWLGZe4xbgpelitP1e8mF0jmr--fjDAgCLcB/s1600/cropped_resetkey2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="216" src="https://2.bp.blogspot.com/-4JDZOqEjYWY/WM7xo613MWI/AAAAAAAAAS4/SCWLGZe4xbgpelitP1e8mF0jmr--fjDAgCLcB/s640/cropped_resetkey2.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br />
<b>Logging in with key mary2</b><br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
We use the mary2 key to login and we can happily login to our VM.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-pkeRftSe754/WM7ywUOmmyI/AAAAAAAAATI/3UYZgtS_cyY1tilPEDJZH_EENkwBwDkUgCLcB/s1600/login2_cut.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="358" src="https://3.bp.blogspot.com/-pkeRftSe754/WM7ywUOmmyI/AAAAAAAAATI/3UYZgtS_cyY1tilPEDJZH_EENkwBwDkUgCLcB/s640/login2_cut.png" width="640" /></a></div>
<b><br /></b>
<b><br /></b>
<b>Logging in with key mary1</b><br />
<div style="text-align: justify;">
Unfortunately, the person who stole your laptop was able to extract the key because no full disk encryption was used. Contrary to what you expect the key still works and he has access to your vm in the cloud.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-YYZt7I9uE98/WM7zWfDxFPI/AAAAAAAAATQ/Z8ZXzgfx8IM68lUYd5qqx3SVXSsNb04jACLcB/s1600/relogin1_cut.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="404" src="https://1.bp.blogspot.com/-YYZt7I9uE98/WM7zWfDxFPI/AAAAAAAAATQ/Z8ZXzgfx8IM68lUYd5qqx3SVXSsNb04jACLcB/s640/relogin1_cut.png" width="640" /></a></div>
<br />
<b>What happened behind the scenes?</b><br />
<div style="text-align: justify;">
1. The public key that you provide through the Azure Portal gets added to the authorized_keys file. However, the original public key has not been removed. Therefore you can keep using this key to access the VM.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-GnQg-seAS1A/WM70Y82F_5I/AAAAAAAAATY/-suY3bWBUlIXnEV_X3jQ0CneuYleOZOYACLcB/s1600/authorized_keys2_cropped.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="188" src="https://1.bp.blogspot.com/-GnQg-seAS1A/WM70Y82F_5I/AAAAAAAAATY/-suY3bWBUlIXnEV_X3jQ0CneuYleOZOYACLcB/s640/authorized_keys2_cropped.png" width="640" /></a></div>
<br />
<div style="text-align: justify;">
2. In the Portal in the <i style="font-weight: bold;">Automation script </i>you will see that there is still being referred to the first public key. Notice that just copying over this information to deploy a new virtual machine might allow the laptop thief again to gain access.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-QvE8KNdpABw/WM78M6JD6-I/AAAAAAAAATo/L_caUKIbCokzi8JK0oklMoQQPEreUjwDQCLcB/s1600/where_to_find_login_information_afterkey_reset_cropped.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="126" src="https://4.bp.blogspot.com/-QvE8KNdpABw/WM78M6JD6-I/AAAAAAAAATo/L_caUKIbCokzi8JK0oklMoQQPEreUjwDQCLcB/s640/where_to_find_login_information_afterkey_reset_cropped.png" width="640" /></a></div>
<br />
<b>How to do a "real" reset?</b><br />
<div style="text-align: justify;">
The easiest way to do a real reset of the ssh key, is to log in to the VM and remove the previous public key from the authorized key file. The automated script will however still look the same as above. Also make sure that you put in comments that start with # in your authorized_key file so that you know which public keys are used for what purpose.</div>
<br />
<span style="font-size: large;"><b>What happens after a redeploy of the virtual machine?</b></span><br />
<div style="text-align: justify;">
When you redeploy your virtual machine, the active authorized_key file is copied over during deployment. So if you could login with key1, key2 and key3 before deployment. You can also still login with key1, key2 and key3. These keys can either be added in the authorized_key file through the VM itself or they can be added to the VM by the Reset functionality of the portal. </div>
<div style="text-align: justify;">
However deploying of your VM might result in downtime of the machine and in loss of data. Therefore we will go in further detail about what happens with your VM when you redeploy in a later blog post.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b><span style="font-size: large;">Conclusion</span></b></div>
<div style="text-align: justify;">
In this blog post we have learnt how you can reset the public key that you use to gain access through a Linux VM in Azure. This consists of two steps. The first one is going to VM in the portal and use the Reset password functionality to add an extra public key. If you want to disable the previous public key from the authorized_keys file. You next will need to log in into the VM and remove the previous public key. When a VM is redeployed all the public keys from the authorized_keys file will provide access after the redeployment. In a later blog post we will focus on what happens behind the scenes when a VM is redeployed in Azure. In another upcoming blog post we will discuss how you can switch from password access to a VM to access to a VM with a public key.</div>
<br />
<br />
<br />
<b><br /></b>Anonymoushttp://www.blogger.com/profile/00885379426770757526noreply@blogger.com4tag:blogger.com,1999:blog-7939626009906644084.post-19809854058489207122017-03-10T04:19:00.000-08:002017-03-10T04:19:39.387-08:00How to call a web service generated from Azure Machine Learning Studio From Ruby<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">Background Note</span><br />
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small; font-weight: normal;">This blog post is the last post in a series about how to get started to build a data architecture in the cloud. The architecture is described in the first blog post. This series features two people, Dana Simpsons the data scientist and Frank Steve Davidson the full stack developer, who are building their own SaaS company that will build games based on data science. In this blog post we will describe how Frank can call the web service that Dana generated in Azure Machine Learning Studio from Ruby.</span></div>
</h2>
<div>
<b><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">Position in the architecture</span></b><br />
<div style="text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Frank wants to run the web service that Dana built in Azure Machine Learning Studio. To be able to use this web service, the files that he is providing as input and the files that he gets as output need to be stored in the cloud, so in blob storage. He already knows how to handle these files in the cloud, as we described in a <a href="http://blog.datainstructors.com/2017/03/how-to-access-blob-storage-with-ruby.html" target="_blank">previous post</a>. So therefore he only will need to focus on how to feed and call the web service and how to extract the results from the web service. The major challenge for Frank was also that there was no automatically generated stub available to call a web service from Azure ML for Ruby. This is the case for Python, C# and R. </span></div>
<b><br /></b></div>
<div style="text-align: justify;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-wdLt1vgrij0/WLuKPO2G5cI/AAAAAAAAAOM/_9rKUPZTV2kfMMBWSE5YkD7X5QLqcSt_gCPcB/s1600/goal3_new.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-size: large;"><img border="0" height="231" src="https://2.bp.blogspot.com/-wdLt1vgrij0/WLuKPO2G5cI/AAAAAAAAAOM/_9rKUPZTV2kfMMBWSE5YkD7X5QLqcSt_gCPcB/s400/goal3_new.png" width="400" /></span></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<b><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">Understanding the payload</span></b></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">In the picture below, we see again how Dana sees her experiment in Azure ML. On top, we have the input file in blue and on the bottom we have all the different output files.</span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-pLwHMskNXFI/WLxrE91efSI/AAAAAAAAAOo/pgCSqnwZnf8xIjo3RMJcHulAfcvRoFTyACLcB/s1600/AzureMLExperiment.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="297" src="https://1.bp.blogspot.com/-pLwHMskNXFI/WLxrE91efSI/AAAAAAAAAOo/pgCSqnwZnf8xIjo3RMJcHulAfcvRoFTyACLcB/s640/AzureMLExperiment.png" width="640" /></a></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Frank will need to build a <i>json</i> object that looks like the Sample Request Payload below. The top blue box presents the input file. The bottom blue boxes present the output files. For the example the output label for the word cloud from the previous blog and the final location of where this file will be stored are annotated in the orange boxes in the picture below. The main goal will be now to generate this json object.</span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-SjlFWJGZdVc/WLxrTDtVxeI/AAAAAAAAAOs/nRlkNlfv8jsxYB9a_IUfGT3t0goiGTCBQCLcB/s1600/drawn_requestpayload.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="256" src="https://3.bp.blogspot.com/-SjlFWJGZdVc/WLxrTDtVxeI/AAAAAAAAAOs/nRlkNlfv8jsxYB9a_IUfGT3t0goiGTCBQCLcB/s640/drawn_requestpayload.png" width="640" /></a></div>
<br />
<b><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">Defining the payload in Ruby</span></b><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Frank implements now a hash structure, <i>payload_hash</i> in ruby. For the <a href="http://blog.datainstructors.com/2017/03/how-to-access-blob-storage-with-ruby.html" target="_blank">ConnectionString </a>he looks up again the name of the storage account, and the account key. To be able to distinguish between different runs of the web service he will use a timestamp to have unique names.</span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-WsEn3I299NI/WLxrljkl3TI/AAAAAAAAAOw/PqvPlxG2X9EfinYvAniMcOfBPDTWoEnWgCLcB/s1600/input_hash2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="185" src="https://3.bp.blogspot.com/-WsEn3I299NI/WLxrljkl3TI/AAAAAAAAAOw/PqvPlxG2X9EfinYvAniMcOfBPDTWoEnWgCLcB/s640/input_hash2.png" width="640" /></a></div>
<br />
<b><span style="font-size: large;">Start the web service</span></b><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">To start the web service, Frank needs the API key from the web service and the URI. The API key can be found on the main page of the web service as shown below.</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-gvysQVKO0ao/WLxr3IJDgYI/AAAAAAAAAO0/x4Q_DwxUM5UR_hTg3Obi1rYMTs9XyQ4TQCLcB/s1600/webservice.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="201" src="https://2.bp.blogspot.com/-gvysQVKO0ao/WLxr3IJDgYI/AAAAAAAAAO0/x4Q_DwxUM5UR_hTg3Obi1rYMTs9XyQ4TQCLcB/s640/webservice.png" width="640" /></a></div>
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">When Dana clicks next on Batch Execution she gets the picture below and there she finds also the Request URI. This is what she will give at Frank.</span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-gp83k-DrVuw/WLxsI6lZhvI/AAAAAAAAAO4/oRBRejmaNwMRhu3a6SRcX_rLb0ffv5LogCLcB/s1600/uri_selector.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="314" src="https://4.bp.blogspot.com/-gp83k-DrVuw/WLxsI6lZhvI/AAAAAAAAAO4/oRBRejmaNwMRhu3a6SRcX_rLb0ffv5LogCLcB/s640/uri_selector.png" width="640" /></a></div>
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Armed with all these pieces of information, Frank has now built the Ruby code to call the web service that Dana has built.</span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-WXswrmKaR40/WLxse6zTdQI/AAAAAAAAAO8/AgvAriGJ8EstQQCrV7WOyuV1NMdh88z2QCLcB/s1600/starting_webservice.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="218" src="https://2.bp.blogspot.com/-WXswrmKaR40/WLxse6zTdQI/AAAAAAAAAO8/AgvAriGJ8EstQQCrV7WOyuV1NMdh88z2QCLcB/s640/starting_webservice.png" width="640" /></a></div>
<br />
<b><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">While the web service is running</span></b><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">The web service hasn't finished immediately. Therefore, Frank will need to check when the web service is done. You can see the code for this in the code piece below.</span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-H9KoI_sdVh8/WLxw2Tii8NI/AAAAAAAAAPM/S0R82BHlIA0oCczjqlASj7Q_0_iOUZ3ZwCLcB/s1600/webservice_running.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="161" src="https://3.bp.blogspot.com/-H9KoI_sdVh8/WLxw2Tii8NI/AAAAAAAAAPM/S0R82BHlIA0oCczjqlASj7Q_0_iOUZ3ZwCLcB/s640/webservice_running.png" width="640" /></a></div>
<br />
<b><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">I was promised an image?</span></b><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">The goal of this blog was to extract an image from the web service. But currently Frank has only extracted csv files. As you may remember we have connected the right dot to get the output. This is accidentally still called a csv file. But the input itself looks like below.</span><br />
<b><br /></b>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-PUmnBgz5Q_Y/WLxxh8qHAOI/AAAAAAAAAPQ/3tfy_cdTO8UtZF8WIyayQI05CQ7asPj4wCLcB/s1600/csv_example.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="134" src="https://4.bp.blogspot.com/-PUmnBgz5Q_Y/WLxxh8qHAOI/AAAAAAAAAPQ/3tfy_cdTO8UtZF8WIyayQI05CQ7asPj4wCLcB/s640/csv_example.png" width="640" /></a></div>
<b><br /></b>
<br />
<span lang="EN-US" style="font-family: "arial" , "helvetica" , sans-serif; line-height: 107%;">We
now will need to extract the piece with all the funny letter and number combinations
that has a graphics title, this will actually be the image file. </span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-T-02JQSIkaM/WLxx8Yt54YI/AAAAAAAAAPY/C1wRDFf2m9gcM4Ejm-K2FWXC55S6wCZlACLcB/s1600/image_selection.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="338" src="https://1.bp.blogspot.com/-T-02JQSIkaM/WLxx8Yt54YI/AAAAAAAAAPY/C1wRDFf2m9gcM4Ejm-K2FWXC55S6wCZlACLcB/s640/image_selection.png" width="640" /></a></div>
<div style="text-align: left;">
<br /></div>
<span lang="EN-US" style="font-family: "calibri" , sans-serif; font-size: large; line-height: 107%;"><b>Conclusion</b></span><br />
<span style="font-family: "calibri" , sans-serif;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: 14.6667px;">If you have read all the different blogs of this series you have now an idea about the architecture that is involved for building the architecture of a data application in the cloud and what the different tasks are that people need to do that are building this architecture. Thank you very much for reading my blog.</span></span><br />
<br /></div>
Anonymoushttp://www.blogger.com/profile/00885379426770757526noreply@blogger.com5tag:blogger.com,1999:blog-7939626009906644084.post-46679395003239902992017-03-09T03:55:00.000-08:002017-03-10T04:21:32.517-08:00How to use R scripts in Azure Machine Learning Studio and generate a webservice<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif; font-size: large;"><b>Background Note</b></span></div>
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif;">This blog post is the fourth post in a series about how to get started to build a data architecture in the cloud. The architecture is described in the <a href="http://blog.datainstructors.com/2017/03/how-to-build-architecture-for-your.html" target="_blank">first blog post</a>. This series features two people, Dana Simpsons the data scientist and Frank Steve Davidson the full stack developer, who are building their own SaaS company that will build games based on data science. In this blog post we will describe how Dana will use Azure Machine Learning Studio combined with R to generate the images that Frank will use in the <a href="http://datainstructors.com/titlegame" target="_blank">gaming website</a>. In the <a href="http://blog.datainstructors.com/2017/03/how-to-call-web-service-generated-from.html" target="_blank">last blog </a>of this series we will learn how Frank calls this web service.</span></div>
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif; font-size: large;"><b>Introduction to Azure Machine Learning Studio</b></span></div>
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif;">A good resource to get started with Azure Machine Learning Studio is the free ebook: <a href="https://blogs.msdn.microsoft.com/microsoft_press/2015/04/15/free-ebook-microsoft-azure-essentials-azure-machine-learning/" target="_blank">Microsoft Azure Essentials: Azure Machine Learning</a>. In this blogpost we will focus on the way Dana is working on generating the images for the game website that Dana and Frank are developing. To evaluate Azure Machine Learning Studio, you will need to have a Microsoft Account. You can use this account to log in to <a href="http://studio.azureml.net/">http://studio.azureml.net</a>. Exploring Azure Machine Learning Studio is completely free.</span></div>
<div>
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif; font-size: x-small;"><br /></span></div>
</div>
<div>
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif; font-size: large;"><b>Position of this blog in the architecture</b></span></div>
</div>
<div>
<div class="MsoNormal">
<div style="text-align: justify;">
<span lang="EN-US" style="font-family: "georgia" , "times new roman" , serif;">Now we will
describe the steps that Dana performs to embed her R scripts into Azure
Machine Learning Studio and how she will be able to convert the experiments
that she makes into a web service so that Frank will be able to use them in the
Ruby layer. So the focus of this blog is more on how only a few things are
needed so that both Frank and Dana will be able to work with the development
tools that they both feel most comfortable with.<o:p></o:p></span></div>
</div>
<div class="MsoNormal">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-HteEFtX9aQI/WMC2SOjxDQI/AAAAAAAAARs/DIvTFya107IFiwyy8810g4CKLBu8MlzwQCLcB/s1600/goal3_new.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="231" src="https://3.bp.blogspot.com/-HteEFtX9aQI/WMC2SOjxDQI/AAAAAAAAARs/DIvTFya107IFiwyy8810g4CKLBu8MlzwQCLcB/s400/goal3_new.png" width="400" /></a></div>
<div style="text-align: left;">
<br /></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif; font-size: large;"><b>Preparing the text to be in the correct format</b></span></div>
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif;">For this application, Dana started from a book in text format which she converted to a csv file that consists of one column and each row represents one sentence of the text. She did this formatting on her own computer but this can later be automated further in the green block above. </span></div>
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif;"><br /></span></div>
</div>
<div class="MsoNormal">
<span lang="EN-US" style="font-family: "georgia" , "times new roman" , serif; font-size: x-small;"></span></div>
<div class="MsoNormal">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-CML8sAnzN5A/WLuKisCfKaI/AAAAAAAAANw/KZ0VgBYunZ8MIAszWJkE1VNh6q3WA5dHACLcB/s1600/AzureMLDataSet.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="282" src="https://4.bp.blogspot.com/-CML8sAnzN5A/WLuKisCfKaI/AAAAAAAAANw/KZ0VgBYunZ8MIAszWJkE1VNh6q3WA5dHACLcB/s320/AzureMLDataSet.png" width="320" /></a></div>
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif; font-size: x-small;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif;">To be able to use this dataset further, she will upload this csv file as a dataset in Azure ML. So after she has logged in into Azure Ml, she has selected DATASETS in the left column. Next, she clicks on + NEW in the left lower corner and she can upload her dataset to her available datasets in Azure ML. In the picture below you can see all the datasets that she has uploaded in this way.</span></div>
</div>
</div>
<div>
<div style="text-align: justify;">
<br /></div>
</div>
<div>
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif; font-size: large;"><b>A high level overview of the experiment</b></span></div>
</div>
<div>
<div>
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif;">The experiment that she built looks as the picture below. You see that there are two type of boxes, white ones and blue ones. First you only will need to focus on the white boxes. The box containing the text “chapter1_to_5_list.csv” is the box that is selecting the input data set. This dataset is fed to a “Select Column in Dataset”. Next, the output of this data is being fed to several “Execute R Scripts”. These R scripts will be generating the data analysis and the different images that are provided to Frank by the Web service in a next step. </span></div>
</div>
<div>
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif;">Now, focus on the blue boxes. You will see that there is one blue box on top which is called Web service input and six blue boxes on the bottom which are called Web service output. When in a next step the web service will be generated from this experiment, random csv files that only have one column can be fed to this web service and the different images can be automatically generated.</span></div>
</div>
<div style="font-weight: bold;">
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif; font-size: x-small;"><br /></span></div>
</div>
</div>
<div class="separator" style="clear: both; text-align: justify;">
<a href="https://3.bp.blogspot.com/-DH5CWObYdoE/WLuLDE8FrkI/AAAAAAAAAN0/sYMN7WUI7wsZ34uMsjD2dR4IW5PJgU6AgCLcB/s1600/AzureML2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: "georgia" , "times new roman" , serif; font-size: x-small;"><img border="0" height="313" src="https://3.bp.blogspot.com/-DH5CWObYdoE/WLuLDE8FrkI/AAAAAAAAAN0/sYMN7WUI7wsZ34uMsjD2dR4IW5PJgU6AgCLcB/s640/AzureML2.png" width="640" /></span></a></div>
<div>
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif; font-size: x-small;"><br /></span></div>
</div>
<div>
<div style="text-align: justify;">
<b><span style="font-family: "georgia" , "times new roman" , serif; font-size: large;">A deeper dive into on Execute R Script box</span></b></div>
</div>
<div>
<div>
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif;">We now look a bit deeper into the calculation of the word cloud that is being performed in the box with the blue border. As you can see in the R code on the right, first the dataset is being selected in dataset1. Next Dana is deleting some common English stopwords to provide a clearer picture about the special words from the book that she will be displaying. Next, she is working further to build a picture of the wordcloud. </span></div>
</div>
<div>
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif;">She will right click on the number 2 and will select Visualize. This provides her the output of her R script. Which will look like the picture below. It is important to notice the Graphics title here. In this experiment, she will be generating several graphics and also some extra datasets. When she is happy with her results. She will generate a webservice from this experiment.</span></div>
</div>
<div style="font-weight: bold;">
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif; font-size: x-small;"><br /></span></div>
</div>
</div>
<div class="separator" style="clear: both; text-align: justify;">
<a href="https://4.bp.blogspot.com/-j5oM_NQaU9g/WLuLhF-zGgI/AAAAAAAAAN4/_znXszPz7Vo9zZcaHe08H9NVu0ZojFi4QCLcB/s1600/wordcloud_calculator2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: "georgia" , "times new roman" , serif; font-size: x-small;"><img border="0" height="411" src="https://4.bp.blogspot.com/-j5oM_NQaU9g/WLuLhF-zGgI/AAAAAAAAAN4/_znXszPz7Vo9zZcaHe08H9NVu0ZojFi4QCLcB/s640/wordcloud_calculator2.png" width="640" /></span></a></div>
<div>
<div style="text-align: justify;">
<b><span style="font-family: "georgia" , "times new roman" , serif; font-size: x-small;"><br /></span></b></div>
</div>
<div class="separator" style="clear: both; text-align: justify;">
<a href="https://3.bp.blogspot.com/-eJwcpHmwYZc/WLuLncvgv5I/AAAAAAAAAN8/y8kfvmbnsoQUnUh7bxhdYBySGGHF9pdcQCLcB/s1600/graphics_wordcloud.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: "georgia" , "times new roman" , serif; font-size: x-small;"><img border="0" height="320" src="https://3.bp.blogspot.com/-eJwcpHmwYZc/WLuLncvgv5I/AAAAAAAAAN8/y8kfvmbnsoQUnUh7bxhdYBySGGHF9pdcQCLcB/s320/graphics_wordcloud.png" width="302" /></span></a></div>
<div>
<div style="text-align: justify;">
<b><span style="font-family: "georgia" , "times new roman" , serif; font-size: large;">The generation of the web service</span></b></div>
</div>
<div>
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif;">To generate a web service, she selected, <i>Deploy Web service</i> from the bottom. She also switched the view to the web service view by switching over the slider on the bottom to show the globe. You will see that now the blue boxes of the web service have turned dark blue. There is a curved line from an Execute R Script box to a blue box of the Web service output. When this line starts from the <i>right dot</i>, you will be able to export an image, <i>left dot</i> you will be able to export a csv dataset. Also when you click on a blue box you will be able to provide a meaningful name for the output. </span></div>
</div>
<div>
<div style="text-align: justify;">
<b><span style="font-family: "georgia" , "times new roman" , serif; font-size: x-small;"><br /></span></b></div>
</div>
<div class="separator" style="clear: both; text-align: justify;">
<a href="https://3.bp.blogspot.com/-I3iMldt--UE/WLuL3aVcX5I/AAAAAAAAAOA/hevDKQquhAIlsHmES_fqwEDbodEJ8TIbwCLcB/s1600/deploy_webservice.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: "georgia" , "times new roman" , serif; font-size: x-small;"><img border="0" height="417" src="https://3.bp.blogspot.com/-I3iMldt--UE/WLuL3aVcX5I/AAAAAAAAAOA/hevDKQquhAIlsHmES_fqwEDbodEJ8TIbwCLcB/s640/deploy_webservice.png" width="640" /></span></a></div>
<div>
<div style="text-align: justify;">
<b><span style="font-family: "georgia" , "times new roman" , serif; font-size: x-small;"><br /></span></b></div>
</div>
<div class="separator" style="clear: both; text-align: justify;">
<a href="https://1.bp.blogspot.com/-n61HffdKKwg/WLuMBjRwBfI/AAAAAAAAAOE/TBgN6Ny8bpg-NoCy0WXDe2OjE-_94v9IwCLcB/s1600/webservice_view1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: "georgia" , "times new roman" , serif; font-size: x-small;"><img border="0" height="432" src="https://1.bp.blogspot.com/-n61HffdKKwg/WLuMBjRwBfI/AAAAAAAAAOE/TBgN6Ny8bpg-NoCy0WXDe2OjE-_94v9IwCLcB/s640/webservice_view1.png" width="640" /></span></a></div>
<div>
<div style="text-align: justify;">
<b><span style="font-family: "georgia" , "times new roman" , serif; font-size: x-small;"><br /></span></b></div>
</div>
<div>
<div class="MsoNormal">
<div style="text-align: justify;">
<span lang="EN-US" style="font-family: "georgia" , "times new roman" , serif;">For their project,
Dana and Frank will be working with the <i>Batch Execution</i> mode because they are
working with the csv files that will be uploaded. On this page, she will find
the API key that she will need to give to Frank.<o:p></o:p></span></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<span lang="EN-US" style="font-family: "georgia" , "times new roman" , serif;">When she
clicks on the Batch Execution mode. She will find the Request URI for the web
service that she will need to provide to Frank.<o:p></o:p></span></div>
</div>
<div style="text-align: justify;">
<span lang="EN-US" style="font-family: "georgia" , "times new roman" , serif; line-height: 107%;">Next, she scrolls down to <i>Sample Request
Payload</i>, she can validate that all her different inputs and outputs have been
defined properly.</span></div>
</div>
<div>
<div style="text-align: justify;">
<span lang="EN-US" style="font-family: "georgia" , "times new roman" , serif; font-size: x-small; line-height: 107%;"><br /></span></div>
</div>
<div class="separator" style="clear: both; text-align: justify;">
<a href="https://4.bp.blogspot.com/-D61mpJjloDU/WLuMXNh60vI/AAAAAAAAAOQ/0O4-2xZCRkY-G4E7jTlyCz02F25gJN8iACLcB/s1600/webservice2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: "georgia" , "times new roman" , serif; font-size: x-small;"><img border="0" height="256" src="https://4.bp.blogspot.com/-D61mpJjloDU/WLuMXNh60vI/AAAAAAAAAOQ/0O4-2xZCRkY-G4E7jTlyCz02F25gJN8iACLcB/s640/webservice2.png" width="640" /></span></a></div>
<div class="separator" style="clear: both; text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif; font-size: x-small;"><br /></span></div>
<div>
<div style="text-align: justify;">
<span lang="EN-US" style="line-height: 107%;"><b><span style="font-family: "georgia" , "times new roman" , serif; font-size: large;">Cost Analysis</span></b></span></div>
</div>
<div>
<div style="text-align: justify;">
<span lang="EN-US" style="line-height: 107%;"><span style="font-family: "georgia" , "times new roman" , serif;">Azure Machine Learning Studio has a Free tier that Dana can use for building her current data solutions. For the Web Services, she also still belongs within the DEV/Test limits. All the pricing details can be seen <a href="https://azure.microsoft.com/en-us/pricing/details/machine-learning/" target="_blank">here</a> . The only thing were there will be a cost involved is in the blob storage for the files that will be used as the input and the output for the web service. These are currently 0.01 CAD for using the web service for one book for a bit more than twee weeks.</span></span></div>
</div>
<div>
<div style="text-align: justify;">
<span lang="EN-US" style="line-height: 107%;"><span style="font-family: "georgia" , "times new roman" , serif; font-size: x-small;"><br /></span></span></div>
</div>
<div>
<div style="text-align: justify;">
<span lang="EN-US" style="line-height: 107%;"><span style="font-family: "georgia" , "times new roman" , serif; font-size: large;"><b>Conclusion</b></span></span></div>
</div>
<div>
<div style="text-align: justify;">
<span lang="EN-US" style="line-height: 107%;"><span style="font-family: "georgia" , "times new roman" , serif;">In this blog post we showed how Azure ML can be used to generate web services on top of the data science experiments that Dana has built. In the <a href="http://blog.datainstructors.com/2017/03/how-to-call-web-service-generated-from.html" target="_blank">next blog</a>, we will show how Frank will call this web service to extract all the different images.</span></span></div>
</div>
<div>
<span lang="EN-US" style="line-height: 107%;"><o:p></o:p></span></div>
Anonymoushttp://www.blogger.com/profile/00885379426770757526noreply@blogger.com21tag:blogger.com,1999:blog-7939626009906644084.post-70365159199348934892017-03-08T04:34:00.000-08:002017-03-09T03:58:38.652-08:00How to access blob storage with Ruby<span style="font-size: large;"><span style="text-align: justify;"><b>Background Note</b></span></span><br />
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif;">This blog post is the third post in a series about how to get started to build a data architecture in the cloud. The architecture is described in the <a href="http://blog.datainstructors.com/2017/03/how-to-build-architecture-for-your.html" target="_blank">first blog</a> post. This series features two people, Dana Simpsons the data scientist and Frank Steve Davidson the full stack developer, who are building their own SaaS company that will build <a href="http://datainstructors.com/titlegame" target="_blank">games</a> based on data science. In this blog post we will describe how you can access blob storage in Ruby. In the <a href="http://blog.datainstructors.com/2017/03/how-to-use-r-scripts-in-azure-machine.html" target="_blank">next blog pos</a>t of this series you will learn how to use R scripts in Azure Machine Learning Studio.</span></div>
<span style="font-size: large;"><b><br /></b></span>
<span style="font-size: large;"><b>Position in the architectu</b>re</span><br />
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif;">All programs need data at some point. That data can either come from files or from a database. This is not different when your program lives in the cloud. In this case, it also might want to access data that is stored in the cloud. For Azure these files are stored in Blob Storage. For Frank and Dana's application from Dana and Frank, Frank will need to know how we will be able to access Blob Storage from Ruby. Because eventually you might want to store large pieces of information in the cloud, it is important to also again understand the cost.</span></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-oFto935yzak/WLx06wC7iNI/AAAAAAAAAPg/NTMdGOpnjasnEB89NMHlRqXjBWOuPgl5wCLcB/s1600/goal2_new.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="225" src="https://4.bp.blogspot.com/-oFto935yzak/WLx06wC7iNI/AAAAAAAAAPg/NTMdGOpnjasnEB89NMHlRqXjBWOuPgl5wCLcB/s400/goal2_new.png" style="cursor: move;" width="400" /></a></div>
<br />
<br />
<b><span style="font-size: large;">How to get the needed pieces from the Azure Portal</span></b><br />
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif;">To be able to access a piece of blob storage you will need to know the account name, the access keys and the container name. When you go to the Azure portal, you can select Storage accounts to get an overview of the storage accounts. When you click on one of the storage accounts you will get a view similar to the view below. This will help you to identify the pieces needed for your own example.</span></div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-5J7VX8KKnQU/WLx1hEH2mKI/AAAAAAAAAPk/UdGKi1fR_W4crBUz15gKJ0HYav8HCKAjACLcB/s1600/parts_selection.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="224" src="https://3.bp.blogspot.com/-5J7VX8KKnQU/WLx1hEH2mKI/AAAAAAAAAPk/UdGKi1fR_W4crBUz15gKJ0HYav8HCKAjACLcB/s640/parts_selection.png" width="640" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<span style="font-size: large;"><b>Initializing blob storage object from Ruby</b></span></div>
<div>
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif;">When you want to access blob storage, make sure that you have the Azure gem installed. Next write require ‘azure’ at the the beginning of your script. When you have gathered all these pieces of information, you will be able to initialize your blob storage object and you will also be able to define your connectionstring.</span></div>
</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-3r0WZ-rV6MY/WLx2neO_r2I/AAAAAAAAAPo/BUQHqGPK3Tc7kSTOtipZ4lPnYoe9-wy7QCLcB/s1600/connect_to_blob_storage2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="72" src="https://2.bp.blogspot.com/-3r0WZ-rV6MY/WLx2neO_r2I/AAAAAAAAAPo/BUQHqGPK3Tc7kSTOtipZ4lPnYoe9-wy7QCLcB/s640/connect_to_blob_storage2.png" width="640" /></a></div>
<div>
<br /></div>
<div>
<b><span style="font-size: large;">List the content of your blob storage</span></b></div>
<div>
<span style="font-family: "georgia" , "times new roman" , serif;">The easiest way to list your content from a blob container is by using the Azure portal. You can find an example of this below.</span><br />
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-oyysaTkHMok/WLx3MGV8V3I/AAAAAAAAAPs/dRPMXcky1rkaDgnAPPYIg45_yG8-n1-yQCLcB/s1600/blob_content.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="286" src="https://2.bp.blogspot.com/-oyysaTkHMok/WLx3MGV8V3I/AAAAAAAAAPs/dRPMXcky1rkaDgnAPPYIg45_yG8-n1-yQCLcB/s640/blob_content.png" width="640" /></a></div>
<div>
<br /></div>
<div>
<span style="font-family: "georgia" , "times new roman" , serif;">Accessing the content from a container from Ruby however is also straight forward.</span></div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-mcOZzywMgTE/WLx3kF_wBSI/AAAAAAAAAP0/16qTnC3UF18wbWHfqT2H404NDFoboN94QCLcB/s1600/list_the_blobs.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="124" src="https://3.bp.blogspot.com/-mcOZzywMgTE/WLx3kF_wBSI/AAAAAAAAAP0/16qTnC3UF18wbWHfqT2H404NDFoboN94QCLcB/s640/list_the_blobs.png" width="640" /></a></div>
<div>
<br /></div>
<div>
<span style="font-family: "georgia" , "times new roman" , serif;">The first </span><span style="font-family: "courier new" , "courier" , monospace;">puts</span><span style="font-family: "georgia" , "times new roman" , serif;"> will just write the blob storage objects and the second </span><span style="font-family: "courier new" , "courier" , monospace;">puts</span><span style="font-family: "georgia" , "times new roman" , serif;"> will actually write down the file names like you can see below.</span></div>
<div>
<span lang="EN-US" style="font-family: "calibri" , sans-serif; font-size: 11.0pt; line-height: 107%;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-ZTOdGKgxtuU/WLx3vpLFkEI/AAAAAAAAAP4/uft9S0B0j4oIXmOW6PM1lzT9NzNJcTFoQCLcB/s1600/blob_storage_list2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="289" src="https://1.bp.blogspot.com/-ZTOdGKgxtuU/WLx3vpLFkEI/AAAAAAAAAP4/uft9S0B0j4oIXmOW6PM1lzT9NzNJcTFoQCLcB/s640/blob_storage_list2.png" width="640" /></a></div>
<div>
<span lang="EN-US" style="font-family: "calibri" , sans-serif; font-size: 11.0pt; line-height: 107%;"><br /></span></div>
<div>
<span lang="EN-US" style="font-family: "calibri" , sans-serif; line-height: 107%;"><b><span style="font-size: large;">Downloading the files</span></b></span></div>
<div>
<div style="text-align: justify;">
<span lang="EN-US" style="line-height: 107%;"><span style="font-family: "georgia" , "times new roman" , serif;">Finally, it also might be the case that you want to download the files to a local drive or to another VM in the cloud. Below you can see the code for this.</span></span></div>
<div style="text-align: justify;">
<span lang="EN-US" style="line-height: 107%;"><span style="font-family: "calibri" , sans-serif;"><br /></span></span></div>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-lKt8G0MPlJs/WLx4IVWPfyI/AAAAAAAAAP8/FOufjTg9Hg81bWa-HJlxicJivUQDHBFLwCLcB/s1600/download_blob_storage_files.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="65" src="https://2.bp.blogspot.com/-lKt8G0MPlJs/WLx4IVWPfyI/AAAAAAAAAP8/FOufjTg9Hg81bWa-HJlxicJivUQDHBFLwCLcB/s640/download_blob_storage_files.png" width="640" /></a></div>
<div>
<span lang="EN-US" style="line-height: 107%;"><span style="font-family: "calibri" , sans-serif;"><br /></span></span></div>
<div>
<span lang="EN-US" style="font-family: "calibri" , sans-serif; line-height: 107%;"><b><span style="font-size: large;">Cost Analysis</span></b></span></div>
<div>
<span lang="EN-US" style="font-family: "georgia" , "times new roman" , serif; line-height: 107%;"><b>Storage Cost</b></span></div>
<div>
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif;">An overview of the blob storage prices can be found <a href="https://azure.microsoft.com/en-us/pricing/details/storage/blobs/" target="_blank">here</a>. There is again a fixed cost and a variable cost for using blob storage. The fixed cost is just for hosting your data. The second cost is for accessing your data. The price for storing your data still depends on the amount of redundancy that you require and the amount of data that you are storing in blob storage. The last aspect is whether you want fast or hot access or slower or cool access to your data. All these different combinations, result in the price for storing your data ranging from 0.01 USD per gigabyte per month till 0.046 USD per gigabyte per month.</span></div>
</div>
<div>
<span lang="EN-US" style="font-family: "calibri" , sans-serif; line-height: 107%;"><b><span style="font-size: large;"><br /></span></b></span></div>
<div>
<span lang="EN-US" style="font-family: "georgia" , "times new roman" , serif; line-height: 107%;"><b>Access Prices</b></span></div>
<div>
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif;">For accessing your data there are differences between the blob/block operations and data retrieval and data writes. For the blob/block operations the cost is counted in number of operations except for the delete which is free. The data retrial is counted in gigabyte. But it is interesting to notice here that data retrieval and data writing from hot data storage is free. Finally if you would want to import or export large amount of data, there are also options using Azure hard drive disks.</span></div>
</div>
<div>
<span lang="EN-US" style="line-height: 107%;"><span style="font-family: "calibri" , sans-serif;"><br /></span></span></div>
<div>
<span lang="EN-US" style="line-height: 107%;"><span style="font-family: "georgia" , "times new roman" , serif;"><b>Conclusion</b></span></span></div>
<div>
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif;">We have learnt more in this blog about using blob storage with Ruby and the costs that are involved in it. It is important to understand the usage pattern for your data to make the best decision about which type of blob storage to use. In one of the next blogs we will show how blob storage can be used to call the web service that Dana generated from Azure Machine Learning Studio.</span></div>
</div>
Anonymoushttp://www.blogger.com/profile/00885379426770757526noreply@blogger.com3tag:blogger.com,1999:blog-7939626009906644084.post-47863477812910287632017-03-07T04:42:00.000-08:002017-03-08T04:37:16.211-08:00Setting up Linux Virtual Machines in Azure For Ruby Development.<div>
<span style="font-size: large;"><b>Background note</b></span><br />
<div style="text-align: justify;">
This blog post is the second post in a series about how to get started to build a data architecture in the cloud. The architecture is described in the <a href="http://blog.datainstructors.com/2017/03/how-to-build-architecture-for-your.html" target="_blank">first blog post</a>. This series features two people, Dana Simpsons the data scientist and Frank Steve Davidson the full stack developer, who are building their own SaaS company that will build <a href="http://datainstructors.com/titlegame/" target="_blank">games based on data science</a>. In this blog post we will describe how you can create a Linux Virtual Machine (VM) in Azure. We also will investigate the different components of hosting a VM in the cloud. <a href="http://blog.datainstructors.com/2017/03/how-to-access-blob-storage-with-ruby.html" target="_blank">This</a> is the next blog in the series.</div>
<b><span style="font-size: large;"><br /></span></b>
<b><span style="font-size: large;">Introduction</span></b></div>
<div>
<div style="text-align: justify;">
In this blog, Frank has written down his notes that he uses to deploy virtual machines. He has added in lots of screenshots so that it is easy for him to remember the different steps he needs to take in the Azure Portal. These screenshots will also make it easier in the future to hand the creation of VMs off to someone else. While Frank is figuring out how to create the VMs, Dana is investigating the costs and the durability of the VMs when they are not powered on.</div>
</div>
<div>
<br />
<span style="font-size: large;"><b>Position of this blog in the architecture</b></span></div>
<div>
<div style="text-align: justify;">
Currently Frank will have two VMs on which he will host the different components like the Ruby layer and the Apache web server to host the websites for the different games.<br />
In what follows he will explain first how he creates the first Debian VM in the Azure cloud that is used for Ruby development. Next he will explain which links he uses to install Ruby on this VM.<br />
The second VM he will use as a web server. Therefore he will need to install apache on this VM.<br />
Finally, Dana will give her analysis of the costs that are involved in hosting a VM in the cloud. She will look both at the fixed and the variable costs of hosting a virtual machine in the cloud.</div>
<br /></div>
<div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-0H2A7mqXIMU/WLsfKv0qTiI/AAAAAAAAANE/MvP_9X3g7P0OyrVJdYB1D8JBJBQDmRMRACLcB/s1600/goal1_new2.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="409" src="https://1.bp.blogspot.com/-0H2A7mqXIMU/WLsfKv0qTiI/AAAAAAAAANE/MvP_9X3g7P0OyrVJdYB1D8JBJBQDmRMRACLcB/s640/goal1_new2.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<b><span style="font-size: large;">Creating a Debian Linux VM for Ruby development</span></b></div>
<div class="separator" style="clear: both; text-align: left;">
<b><span style="font-size: large;"><br /></span></b></div>
<div class="separator" style="clear: both; text-align: left;">
<b>1. Log into the Azure Portal</b></div>
<div class="separator" style="clear: both; text-align: left;">
Go to the Virtual Machine Section by selecting Virtual Machines in the left column.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
</div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-QtxQW-QMdQc/WLtgqcOXesI/AAAAAAAAANU/_Gq7krXNdBc8zYstTidA1ZikC2UroUpwwCLcB/s1600/select_virtual_machines.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="404" src="https://3.bp.blogspot.com/-QtxQW-QMdQc/WLtgqcOXesI/AAAAAAAAANU/_Gq7krXNdBc8zYstTidA1ZikC2UroUpwwCLcB/s640/select_virtual_machines.png" width="640" /></a></div>
<br /></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<b><br /></b>
<b><span style="font-family: inherit;">2. Add a virtual machine</span></b><br />
<b><span style="font-family: inherit;"><br /></span></b>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-9sxGgsunE-M/WLthZYd5-oI/AAAAAAAAANY/3o75HIiy_XMP2_2WWLnYoy1lhLkd2zOUgCLcB/s1600/add_virtual_machine.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="314" src="https://1.bp.blogspot.com/-9sxGgsunE-M/WLthZYd5-oI/AAAAAAAAANY/3o75HIiy_XMP2_2WWLnYoy1lhLkd2zOUgCLcB/s320/add_virtual_machine.png" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<b><span style="font-family: inherit;">3. Select Debian Jessie</span></b><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-gWxz9qNqgCk/WLthmabdEZI/AAAAAAAAANc/fjTKTP8PP18RwNWeGuhjgY51tcV1EIopwCLcB/s1600/select_jessie2.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="187" src="https://3.bp.blogspot.com/-gWxz9qNqgCk/WLthmabdEZI/AAAAAAAAANc/fjTKTP8PP18RwNWeGuhjgY51tcV1EIopwCLcB/s320/select_jessie2.png" width="320" /></a></div>
<br />
<br />
<div style="clear: both; text-align: left;">
<b>4. Configure the basic options.</b> </div>
<div class="separator" style="clear: both; text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: justify;">
You can setup your virtual machine so that you can login with your public private key pair. To generate this pair, have a look <a href="https://support.rackspace.com/how-to/generating-rsa-keys-with-ssh-puttygen/" target="_blank">here</a>. For the rest you can create or use an existing <a href="http://windowsitpro.com/azure/azure-resource-groups" target="_blank">resource group</a>. Next select the geographical location where you want to store your virtual machine.</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
</div>
<div>
<div class="separator" style="clear: both; text-align: left;">
<a href="https://3.bp.blogspot.com/-V439_KVXUSQ/WLd7SN4kesI/AAAAAAAAALg/BHSobgSFeGogqP9Iz_8VXsMlWTp6Ev55gCEw/s1600/vm_setup1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="377" src="https://3.bp.blogspot.com/-V439_KVXUSQ/WLd7SN4kesI/AAAAAAAAALg/BHSobgSFeGogqP9Iz_8VXsMlWTp6Ev55gCEw/s400/vm_setup1.png" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br />
<div style="clear: both; text-align: left;">
<b><span style="font-family: inherit;">5. Select the size of the virtual machine</span></b></div>
<div class="separator" style="clear: both; text-align: justify;">
<span style="font-family: inherit;"><br /></span></div>
<div class="separator" style="clear: both; text-align: justify;">
<span style="font-family: inherit;">Now you can select the requirements for your virtual machine. When you are creating a virtual Linux Machine <a href="https://docs.microsoft.com/en-us/azure/storage/storage-about-disks-and-vhds-linux" target="_blank">you will have an Operating system disk (OS disk) and a temporary disk attached to it</a>. </span></div>
<div class="separator" style="clear: both; text-align: justify;">
<span style="font-family: inherit;">The Operating system disk is labeled /dev/sda and the temporary disk is typically /dev/sdb. The data of the OS disk is persistent in case you stop, deallocate, pause, reboot, shut-down or resize the VM. However, in case of a delete or a failure of the VM this disk is not persistent. In case that you want to have a case that your disk is persistent in the case of a failure or a deletion of your vm you need to attach an extra data disk. </span></div>
<div class="separator" style="clear: both; text-align: justify;">
<span style="font-family: inherit;">So in case that you are not working with a data disk make sure that you are making backups of your system in a different way. A good strategy for this is two make sure your code is in github and to study the Automation script that has been generated to create this VM combined with your .bash_history when you installed the needed software packages.</span></div>
<div class="separator" style="clear: both; text-align: justify;">
<span style="font-family: inherit;">For the development VM we will select the DS1_V2. However in case you need more computing power, you just can change the size of your VM. In cases that you need less computing power, you just can scale down your VM. </span></div>
<div>
<br /></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-0TnvDMjfpH8/WLd7SECdnPI/AAAAAAAAALo/aeI3j4CBR7QyhUnS8MotxoAz2sMhBNCzACEw/s1600/vm_setup2.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="369" src="https://4.bp.blogspot.com/-0TnvDMjfpH8/WLd7SECdnPI/AAAAAAAAALo/aeI3j4CBR7QyhUnS8MotxoAz2sMhBNCzACEw/s640/vm_setup2.png" width="640" /></a></div>
<div>
<div class="separator" style="clear: both;">
<b><span style="font-size: medium;"><br /></span></b></div>
<div style="clear: both;">
<b><span style="font-family: inherit; font-size: medium;">6. Creating an attached data disk </span></b></div>
<div class="separator" style="clear: both;">
<br /></div>
<div class="separator" style="clear: both;">
Because Frank wants to make sure that his data and his code is still safe in case there happens a failure of the virtual machine or if it would get accidentally deleted, he also attaches a data disk to his VM. Therefore he goes to his VM in the azure portal and selects Disks. Next he will click Add data disk.</div>
<div class="separator" style="clear: both;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-VFerI9H-rOA/WL4WzBBe3yI/AAAAAAAAARA/d0YnhNCeCXMxDxmXnB5YJ1eji8zLqkiTwCLcB/s1600/add_datadisk.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="225" src="https://1.bp.blogspot.com/-VFerI9H-rOA/WL4WzBBe3yI/AAAAAAAAARA/d0YnhNCeCXMxDxmXnB5YJ1eji8zLqkiTwCLcB/s640/add_datadisk.png" width="640" /></a></div>
<div class="separator" style="clear: both;">
<br /></div>
<div class="separator" style="clear: both;">
Now he will select a storage container where he wants to locate his attached data disk. He also has the possibility to select an existing blob. Make sure that you carefully watch the size of the disk that you are making. To have an idea of the prices you can have a look <a href="https://azure.microsoft.com/en-us/pricing/details/storage/unmanaged-disks/" target="_blank">here</a>. We also will discuss this further in the cost analysis section.</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/--u3xUrgTIHU/WL4aBkQ3MBI/AAAAAAAAARM/CozfmDr0SHEgEUahol8CWg1lMbSaDP6pwCLcB/s1600/attach_datadisk2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="392" src="https://2.bp.blogspot.com/--u3xUrgTIHU/WL4aBkQ3MBI/AAAAAAAAARM/CozfmDr0SHEgEUahol8CWg1lMbSaDP6pwCLcB/s640/attach_datadisk2.png" width="640" /></a></div>
<div class="separator" style="clear: both;">
<br /></div>
<div class="separator" style="clear: both;">
Based on the information that you can find <a href="https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-linux-classic-attach-disk#initialize-a-new-data-disk-in-linux" target="_blank">here</a>, you will be able now to initialize the new data disk in your VM and mount it as a separate drive. Carefully execute the different steps. </div>
<div class="separator" style="clear: both;">
<b><span style="font-size: medium;"><br /></span></b></div>
<div style="clear: both;">
<span style="font-size: medium;"><b>7. Installing the needed software and packages</b></span></div>
</div>
<div>
<div style="text-align: justify;">
<br />
Now we need to install Ruby. If we just would install Ruby with sudo-apt-get install we wouldn’t get the newest version. Therefore you can follow the steps as described on <a href="https://www.digitalocean.com/community/tutorials/how-to-install-ruby-on-rails-with-rbenv-on-debian-7-wheezy" target="_blank">this site</a> to install Ruby. For this application ruby 2.4.0 was installed. </div>
<div style="text-align: justify;">
To be able to work with Ruby and Azure you still will need to install the Azure gem. To call the web services generated from Azure ML, you will need to install the httparty gem for Ruby. Therefore, you will need to type the following two commands.</div>
<div style="text-align: justify;">
<br />
<ul>
<li>sudo gem install azure</li>
<li>sudo gem install httparty</li>
</ul>
</div>
</div>
<b><span style="font-size: large;"><br /></span></b>
<b><span style="font-size: large;">Creating a Debian Linux VM as a Web Server</span></b><br />
<b><span style="font-size: large;"><br /></span></b>
<b><span style="font-family: inherit; font-size: medium;">1-4. Creating a VM from the portal</span></b><br />
<br />
1 till 4 are the same as the previous. However in this case we will select a smaller VM and we will not attach a data disk. You can see the specifications below. But remember again in case you need more computer power, you easily can scale up your VM.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-UosTos7wGn0/WL4er93Y6fI/AAAAAAAAARU/6iz6D_VzYBU0F2edAgc__OX684WDKCTwwCLcB/s1600/webserver.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://2.bp.blogspot.com/-UosTos7wGn0/WL4er93Y6fI/AAAAAAAAARU/6iz6D_VzYBU0F2edAgc__OX684WDKCTwwCLcB/s320/webserver.png" width="178" /></a></div>
<br />
<b><span style="font-size: medium;"><br /></span></b>
<b>5. Defining a public IP address</b><br />
<b><br /></b>
Because this VM will be used to host the webserver, a public IP is needed. You can select a static when you are creating the VM. Because this is an extra feature that you are selecting, there is an extra cost added for this.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-QTvmP3t88Mc/WLd7SlvUhzI/AAAAAAAAAL0/FhLDxtd3X2cEJJVuHBc3VlSpMeDc_8lvQCEw/s1600/vm_setup6.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="193" src="https://2.bp.blogspot.com/-QTvmP3t88Mc/WLd7SlvUhzI/AAAAAAAAAL0/FhLDxtd3X2cEJJVuHBc3VlSpMeDc_8lvQCEw/s640/vm_setup6.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div style="clear: both; text-align: left;">
<span style="font-family: inherit;"><b>6. Defining a Network Security Group</b></span></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Because your VM will be used to host your websites, it also will need to be accessible through http. Therefore an extra Network Security Group will need to be defined as you can see below.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-s6bZhrjINW0/WLd7STBTynI/AAAAAAAAALw/xcbpkEJmAFQmM9zzn4TsX2--jFqbYfD_gCEw/s1600/vm_setup5.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="216" src="https://4.bp.blogspot.com/-s6bZhrjINW0/WLd7STBTynI/AAAAAAAAALw/xcbpkEJmAFQmM9zzn4TsX2--jFqbYfD_gCEw/s640/vm_setup5.png" width="640" /></a></div>
<div style="clear: both; text-align: left;">
<span style="font-family: inherit;"><b>7. Installing Apache2</b></span></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Finally, to install apache2, you need to type the following command.</div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<ul>
<li>sudo apt-get install apache2</li>
</ul>
<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
</div>
<div>
<div style="text-align: justify;">
<b><span style="font-size: large;">Cost Analysis</span></b><br />
As the last part of this post, Dana will describe the cost of creating and using the virtual machines in Azure. This consists of a combination of fixed costs and variable costs.<br />
<br />
<b>The cost of the web server</b><br />
<br />
The web server will need to be up all the time. For the rest this VM also needs a static IP and http access. Therefore you can consider all the costs of this VM as fixed. The cost for a public static IP is 0.11 CAD per day. Therefore the cost of the web server will be 48.3 CAD a month.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-UosTos7wGn0/WL4er93Y6fI/AAAAAAAAARU/YZfvrbAMxtwzXEAwsmOw3tLIS_l4fqNTwCEw/s1600/webserver.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://4.bp.blogspot.com/-UosTos7wGn0/WL4er93Y6fI/AAAAAAAAARU/YZfvrbAMxtwzXEAwsmOw3tLIS_l4fqNTwCEw/s320/webserver.png" width="178" /></a></div>
<br />
<br />
<b>The cost of the Ruby development VM</b><br />
<br />
The development machine doesn't need to be powered on all the time. Therefore we can make a distinction between fixed costs and variable costs for the development VM.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-vxpP5eL2wkA/WL4Jw9RZZlI/AAAAAAAAAQw/j2zo0SIPzB8pNsDizCK2ypNwYDlau_Z6ACLcB/s1600/vm_specification.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://2.bp.blogspot.com/-vxpP5eL2wkA/WL4Jw9RZZlI/AAAAAAAAAQw/j2zo0SIPzB8pNsDizCK2ypNwYDlau_Z6ACLcB/s320/vm_specification.png" width="180" /></a></div>
<br />
The fixed costs are the costs that you need to pay whether you machine is running or not. In this case these costs consist of the Operating System Disk of the VM and the attached data disk. For this VM, the cost of the disks is 0.06 CAD per day. Next Frank also decided to attach a standard unmanaged data disk of 128 GB that <a href="https://azure.microsoft.com/en-us/pricing/details/storage/unmanaged-disks/" target="_blank">costs</a> him 8.6 CAD a month.<br />
<br />
Let's have a look now at the variable costs. In case the virtual machine is running the whole day, the machine costs 2.13 CAD per day. So in this way you see that you can save a lot by turning off your development VM in case you are not using it. To facilitate this, there exist auto-shutdown options that you don't need to remember each day to turn off your VM. It is good to know that your data stays on the operating disk in case your VM is in the deallocated state. Remember however that there is no automatic back up of this drive in case of failure. If you want to make daily backups of your system. You can check out this <a href="https://docs.microsoft.com/en-in/azure/backup/backup-azure-vms-first-look-arm" target="_blank">resource</a> to understand the costs.<br />
If you are using a standard unmanaged disk, you also will need to take into account the <a href="https://azure.microsoft.com/en-us/pricing/details/storage/unmanaged-disks/" target="_blank">access price cost</a>. This is a cost that you will be able to estimate further after your first months of development.<br />
<br />
Combining everything together. We assume that Frank has 20 work days of eight hours in a month. This means that he will be using his VM 160 hours a month. Therefore the cost of running his VM for a month is 14.2 CAD during his working days. Next he still needs to pay for this fixed costs of the OS disk (1.2 CAD a month) and the attached data disk (8.6 CAD a month). In this way the cost of the total development VM is 24 CAD a month. By using the auto-shut down option, Dana has implemented cost savings and used some of these cost savings to add in extra piece of mind for Frank by adding the attached data disk.<br />
<br /></div>
</div>
<div>
<span style="font-size: large;"><b>Conclusion</b></span></div>
<div>
<div style="text-align: justify;">
In this blog post Dana and Frank have described all the different steps that you need to follow to create a Linux VM in Azure and to use Ruby on this VM. We also discussed further how to setup a public IP address and how you can allow http access so that the VM can be used as a web server. Finally Dana discussed the cost of hosting such a VM in Azure and we discussed the types of disk that are available on the VM. In the<a href="http://blog.datainstructors.com/2017/03/how-to-access-blob-storage-with-ruby.html" target="_blank"> next blog post</a> we will learn more about how to use this VM to access blob storage in Ruby.</div>
</div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/00885379426770757526noreply@blogger.com16tag:blogger.com,1999:blog-7939626009906644084.post-14998632113779490742017-03-06T04:59:00.000-08:002017-03-10T04:23:42.332-08:00How to build an architecture for your cloud data application<b><span style="font-size: large;">Introduction</span></b><br />
<div style="text-align: justify;">
<span style="font-family: inherit;"><a href="http://www.innovativeii.com/2015-2017-forecast-cloud-computing-skyrocket-rule-delivery/">In 2017 about 35% of new applications will use cloud-enabled, continuous delivery and DevOps life cycles</a>. The use of data is becoming more and more important in the current enterprise with terms like <a href="http://www.forbes.com/sites/bernardmarr/2016/06/14/data-driven-decision-making-10-simple-steps-for-any-business/#4f5e6fdd7283">Data-Driven Decision Making</a> and <a href="http://whatis.techtarget.com/definition/data-driven-decision-management-DDDM">Data-Driven Decision Management</a>.</span></div>
<div style="text-align: justify;">
<span style="font-family: inherit;">But how do you actually start with building data applications in the cloud? Which known technologies can you use? What are new technologies? How do all these technologies work together? What will the cost be of such a solution? </span></div>
<div style="text-align: justify;">
<span style="font-family: inherit;">We will try to answer these questions in this series of blog posts. We will show how two people, Dana and Frank (see picture below) will start building the next Unicorn using cloud based data technology. We will first present their product, next introduce Dana and Frank and finally explain how they started building their architecture.</span></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
The next blogs in the series can be found here<br />
<ul>
<li><a href="http://blog.datainstructors.com/2017/03/setting-up-linux-virtual-machines-in.html" target="_blank">Setting up Linux Virtual Machines in Azure For Ruby Development.</a></li>
<li><a href="http://blog.datainstructors.com/2017/03/how-to-access-blob-storage-with-ruby.html" target="_blank">How to access blob storage with Ruby</a></li>
<li><a href="http://blog.datainstructors.com/2017/03/how-to-use-r-scripts-in-azure-machine.html" target="_blank">How to use R scripts in Azure Machine Learning Studio and how to generate a webservice</a></li>
<li><a href="http://blog.datainstructors.com/2017/03/how-to-call-web-service-generated-from.html" target="_blank">How to call a web service generated from Azure Machine Learning Studio from Ruby</a></li>
</ul>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://2.bp.blogspot.com/-OmpK8VaRedk/WLG5OK4qqEI/AAAAAAAAAKQ/O69wFZtYQt83X2fX8gYIWprbj90kXB_tQCLcB/s1600/dana_and_frank.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="230" src="https://2.bp.blogspot.com/-OmpK8VaRedk/WLG5OK4qqEI/AAAAAAAAAKQ/O69wFZtYQt83X2fX8gYIWprbj90kXB_tQCLcB/s320/dana_and_frank.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The Usbourne first book of the Computer, Usbourn Publishing Ltd, 1985</td></tr>
</tbody></table>
<br />
<br />
<b><span style="font-size: large;">The game changing idea</span></b><br />
<div style="color: black; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: justify; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<div style="margin: 0px;">
<span style="font-family: inherit;">Together they are building a SaaS company that makes online games with data science. Their first game that they are working on is called <b><i>What's the title?</i></b>. They show automatically generated data graphs of classic novels. The user needs to guess the title. A real game changer. Below you can see their first game and you can also check out the link to <a href="http://datainstructors.com/titlegame" target="_blank">their website</a>.</span></div>
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-MmhCG8Cq-Cw/WLx7B4wJIKI/AAAAAAAAAQE/svmzKV080yMOqBB6IwWfCv7jgABEma9egCLcB/s1600/titlegame.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="411" src="https://1.bp.blogspot.com/-MmhCG8Cq-Cw/WLx7B4wJIKI/AAAAAAAAAQE/svmzKV080yMOqBB6IwWfCv7jgABEma9egCLcB/s640/titlegame.png" width="640" /></a></div>
<br />
<br />
<span style="font-size: large;"><b>Meet the team</b></span><br />
<div style="text-align: justify;">
<span style="font-family: inherit;">For now, the team consists of only two people, but they hope to grow fast in the near future. Therefore they both agreed that they will need to acquire extra skill sets besides their current skills to make the full application. Their focus initially is to get all the components connected to have a global overview and see how the different pieces are working together. Later on they can then elaborate further on the different components and move on to using more advanced technologies.</span></div>
<br />
<b><span style="font-size: medium;">
Dana Simpsons, the Data Scientist</span></b><br />
<br />
<div style="text-align: justify;">
Dana loves everything about mathematics and data and she has a decent background in computer science. Her favourite programming languages are R and Python. Finally she also has some SQL and NoSQL database skills. As part of her tasks she will also watch the cloud costs and look for potential savings. She will also be responsible for customer success and will analyze the usage patterns.</div>
<br />
<span style="font-size: medium;"><b>
Frank Steve Davidson, the Full Stack Developer</b></span><br />
<div>
<br /></div>
<div style="text-align: justify;">
Frank loves everything about programming and has a decent background in mathematics. His favorite programming languages are Java and Ruby. He will be responsible for the architecture design and will make sure that the solution gets deployed in the cloud.</div>
<br />
<span style="font-size: large;"><b>The generic architecture</b></span><br />
Dana and Frank did some brainstorming meetings and came up with the following architecture. With this architecture they want to achieve the three goals explained below.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-WPMYglhXhPM/WLx7-dWZgnI/AAAAAAAAAQI/mWfNnNIN9_YKkqz1D4kr6kq-MqmGWSwoACLcB/s1600/global_architctecture.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="326" src="https://4.bp.blogspot.com/-WPMYglhXhPM/WLx7-dWZgnI/AAAAAAAAAQI/mWfNnNIN9_YKkqz1D4kr6kq-MqmGWSwoACLcB/s400/global_architctecture.png" width="400" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<b><br />Goal 1</b><br />
<div style="text-align: justify;">
Their game will be hosted in the cloud. This means that both their development and their production servers will live in the cloud together with the data storage. This means that they will need to look at how virtual machines are being set up by their chosen cloud provider and what kind of cloud storage solutions there are available. Because they are hosting a website, they also still will need to investigate this further.</div>
<br />
<b>Goal 2</b><br />
<div style="text-align: justify;">
Frank has chosen to work with Ruby as programming language to build out the back-end further. Some responsibilities of the back-end will be the storing of different novels in the cloud in a data lake and transforming these novels into a data format consumable by the web services that Dana is building. He will extract the images that Dana generates and will store them back in the cloud. For now they only will use a simple front-end because the data solution will already provide detailed images. In the future they will look further into more advanced front-end technology.</div>
<br />
<b>Goal 3</b><br />
<div style="text-align: justify;">
Dana will need data science related software to be able to make her analysis and generate the graphs. She also will need to have an easy way to access different data sources when she is performing her analysis. She prefers to have access to python and R. Finally she wants to have an easy way to expose her graphs and analysis to Frank. When she is happy with her data science solutions she wants to build an automation layer on top of this.</div>
<br />
<div class="separator" style="clear: both; text-align: left;">
<b><span style="font-size: large;">The architecture in Azure</span></b></div>
<div style="text-align: justify;">
Dana and Frank compared the different cloud providers and they came to the conclusion that Azure suited their needs best. The architecture in Azure is shown below. In what follows we will explain these components more. In the next blogs you will learn how to setup and use these components.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-qN2gAaDtCE8/WLx9AT9gdbI/AAAAAAAAAQQ/rHRYR0Ge9nskdbAF8Ik1FDMbZyWbG728wCLcB/s1600/azure_arhitecture.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="325" src="https://2.bp.blogspot.com/-qN2gAaDtCE8/WLx9AT9gdbI/AAAAAAAAAQQ/rHRYR0Ge9nskdbAF8Ik1FDMbZyWbG728wCLcB/s400/azure_arhitecture.png" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br />
<br />
<b>Goal 1: How to deploy and develop</b><br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-PrtE96FNagM/WLx9J-10pHI/AAAAAAAAAQU/aU66fRyTtaEr_7mY6rFEncVym-aeYGYDwCLcB/s1600/goal1_new2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="256" src="https://3.bp.blogspot.com/-PrtE96FNagM/WLx9J-10pHI/AAAAAAAAAQU/aU66fRyTtaEr_7mY6rFEncVym-aeYGYDwCLcB/s400/goal1_new2.png" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<b><br /></b>
Frank uses a<a href="https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-linux-quick-create-portal"> Linux Virtual Machine in Azure</a> on which he has installed Ruby. He next uses a virtual machine to host the web server Apache2. To save costs he will put his Ruby development machine on auto-shut down mode. To be able to host this virtual machine as a web server he will need to select a public ip address and allow http access on the server. We go into more detail in these steps in the <a href="http://blog.datainstructors.com/2017/03/setting-up-linux-virtual-machines-in.html" target="_blank">next blog post of this series</a>.<br />
<br />
<b>Goal 2: How to talk to and feed the data storage solution</b><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-oFto935yzak/WLx06wC7iNI/AAAAAAAAAPw/ZeIjNo6GpHEisZZ2SLt_EZHv88nzB4gGQCEw/s1600/goal2_new.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="225" src="https://3.bp.blogspot.com/-oFto935yzak/WLx06wC7iNI/AAAAAAAAAPw/ZeIjNo6GpHEisZZ2SLt_EZHv88nzB4gGQCEw/s400/goal2_new.png" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br />
Frank and Dana use <a href="http://blog.datainstructors.com/2016/11/how-to-get-started-with-development-in.html">blob storage</a> as their cloud storage solution. You can imagine Blob storage as your own big drive in the cloud. When you are developing you can use several APIs to access blob storage. Frank will use the <a href="https://rubygems.org/gems/azure/versions/0.7.1">azure gem</a> for Ruby. Dana will analyze the costs that are involved with using blob storage. We go into more detail in this in the <a href="http://blog.datainstructors.com/2017/03/how-to-access-blob-storage-with-ruby.html" target="_blank">third blog post of this series</a>.<br />
<b><br /></b>
<b><br /></b>
<b>Goal 3: How to talk with and build the Data Science Solution</b><br />
<b><br /></b>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-pApky-F97uA/WLx9jLfIAhI/AAAAAAAAAQY/onJ3-4l_qq0V4TkLFtDOOiVjCkiGDoQ4QCLcB/s1600/goal3_new.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="231" src="https://3.bp.blogspot.com/-pApky-F97uA/WLx9jLfIAhI/AAAAAAAAAQY/onJ3-4l_qq0V4TkLFtDOOiVjCkiGDoQ4QCLcB/s400/goal3_new.png" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br />
Dana used <a href="https://studio.azureml.net/">Azure Machine Learning Studi</a>o to build an experiment that generates the needed graphs. Next she deployed this experiment as a web service. This web service consumes the formatted text of a classic novel that is stored in Blob storage and then produces six different graphs. These graphs are then stored in Blob storage. She will describe her strategy in the <a href="http://blog.datainstructors.com/2017/03/how-to-use-r-scripts-in-azure-machine.html" target="_blank">fourth blog post</a>. Next Frank will describe in the last blog post <a href="http://blog.datainstructors.com/2017/03/how-to-call-web-service-generated-from.html" target="_blank">how he feeds and consumes the data from this web service</a>.</div>
<div>
<br /></div>
<b><span style="font-size: large;">Conclusion </span></b><br />
<div style="text-align: justify;">
In this blog post we have introduced two people, Dana and Frank, on their journey to build their SaaS company in the cloud. We briefly mentioned some of the technologies that they needed. In the next blog posts we will explain these technologies more in depth so that you will be able to use them for your own application.</div>
<br />
<br />Anonymoushttp://www.blogger.com/profile/00885379426770757526noreply@blogger.com45tag:blogger.com,1999:blog-7939626009906644084.post-2226044550283865322017-01-16T19:10:00.003-08:002017-01-16T19:11:50.633-08:00How to extract images that are generated by Azure Machine Learning Studio<span style="font-size: large;"><b>
Motivation for this post</b></span><br />
<div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Azure Machine Learning Studio from Microsoft can be used to easily build your data science solutions component by component. One of the major benefits is that in each component you can easily add in some visualizations. When you are satisfied with your data solution you can build a web service from this solution. In this blog post I will show you how you also can extract the visualizations from this web service so that you don't lose the visualizations when you are automating your data science solution.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<span style="font-size: large;"><b>
Building the ML Experiment</b></span></div>
<br />
We will build a simple experiment in Azure ML that imports a data file. Next, the input is connected to an R Module. The main goal of this R module is to draw a graph with a bar chart.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-ShCqP5KAdx0/WH12GQhHYJI/AAAAAAAAAHY/4_Y-fIUSk4U3pArKk1xU3lrjHlzWGmScwCLcB/s1600/AzureMLExperiment.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://3.bp.blogspot.com/-ShCqP5KAdx0/WH12GQhHYJI/AAAAAAAAAHY/4_Y-fIUSk4U3pArKk1xU3lrjHlzWGmScwCLcB/s320/AzureMLExperiment.png" width="258" /></a></div>
<br />
<br />
To see this graph chart you can right click on the right dot of the module and select visualize. <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-moXAq6oZr58/WH12fBz1JOI/AAAAAAAAAHg/fR9aOkTvIqIp5MxIXSdl57lISugpXzn9ACLcB/s1600/VisualizeSelection.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="216" src="https://1.bp.blogspot.com/-moXAq6oZr58/WH12fBz1JOI/AAAAAAAAAHg/fR9aOkTvIqIp5MxIXSdl57lISugpXzn9ACLcB/s320/VisualizeSelection.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
In the resulting window under the title Graphics you see the image that you want to extract under the graphics title. This will be an example graph that we want to extract in an automated way.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-9KWQZVqcItM/WH12MpIdlHI/AAAAAAAAAHc/E_t6ktmm4vQwpns5xd8UBcRuIxRkrZtAQCEw/s1600/Visualization.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://1.bp.blogspot.com/-9KWQZVqcItM/WH12MpIdlHI/AAAAAAAAAHc/E_t6ktmm4vQwpns5xd8UBcRuIxRkrZtAQCEw/s320/Visualization.png" width="271" /></a></div>
<br />
<b><span style="font-size: large;">
Building the Web service</span></b></div>
<div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Next, we will build the web service that corresponds to this experiment. For this web service we need an input and an output. With the input we can read in a blob storage file and the output we are going to connect with the right dot from the R module. This is the one that you also used to generate the visualization. By making this the output we are going to save the content of the window that we visualized in the previous paragraph in a blob storage file. </div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-tvj5i1gCnzQ/WH15tjoRQ6I/AAAAAAAAAH4/T4Kgypr--Fw1XUuI0KWQ8xMuTtLUlTC_gCLcB/s1600/OutputSelection.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="175" src="https://2.bp.blogspot.com/-tvj5i1gCnzQ/WH15tjoRQ6I/AAAAAAAAAH4/T4Kgypr--Fw1XUuI0KWQ8xMuTtLUlTC_gCLcB/s320/OutputSelection.png" width="320" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
So the major two tasks that are still left for us are to execute the web service to generate the output blob and next to parse this output blob storage file. The web service will be generated in a way similar to the way that you you normally generate a web service from Azure ML. We will describe these steps further in an upcoming blog post. </div>
<div style="text-align: justify;">
<br /></div>
</div>
<b><span style="font-size: large;">
Executing the Web service</span></b><br />
<div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
We will execute the web service in the same way that we normally execute a web service generated from Azure ML. The major difference will now be on h<span style="text-align: center;">ow we will parse the output file. We will only be focussing on the latter part. First we read in this file and print it's contents to the console to have a better idea of what the file looks like.</span></div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-5MBDTydHjuQ/WH17WvPhQWI/AAAAAAAAAII/1S-L3ZjHwbksVpNTZa4ePwRBN9iJ8iohwCEw/s1600/OutputFragment.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="116" src="https://3.bp.blogspot.com/-5MBDTydHjuQ/WH17WvPhQWI/AAAAAAAAAII/1S-L3ZjHwbksVpNTZa4ePwRBN9iJ8iohwCEw/s640/OutputFragment.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
We see that this file starts with R Output JSON, we recognize some JSON formatting and we also see a larger part of text written in random letters and numbers following each other. This part is actually the image represented in PNG encoding in this JSON file.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Based on this analysis we know that we need to perform the following tasks to extract this picture. </div>
</div>
<div>
<br /></div>
<div>
1. Find the location of the first "{</div>
<div>
2. Extract the JSON text from the file</div>
<div>
3. Parse the result as a JSON object</div>
<div>
3. Loop over the sections in the JSON object and locate the section that has graphic as title</div>
<div>
4. Decode the resulting values for the image as a PNG and write to a file</div>
<div>
5. Open the file that you have saved and admire your work.<br />
<br />
The code that you need for this you can find in the picture below.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-24kZkvCz1dI/WH1zscQ68nI/AAAAAAAAAHI/Avzrby17EwwzrJM0MZCRmmYxHl_B_om_wCLcB/s1600/code_fragment.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://1.bp.blogspot.com/-24kZkvCz1dI/WH1zscQ68nI/AAAAAAAAAHI/Avzrby17EwwzrJM0MZCRmmYxHl_B_om_wCLcB/s640/code_fragment.png" width="608" /></a></div>
<br /></div>
<span style="font-size: large;"><b>Resulting graph</b></span><br />
<span style="font-size: large;"><b><br /></b></span>
<br />
<div>
Below you can find the extracted images from the web service. If you want to use this image directly in html you actually don't need to execute this extra step. </div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-nDeQTymvyoE/WH2JrP0YVMI/AAAAAAAAAIg/-4ELa2I5ahgzOCDroFeFbhKoQufgbQ0TwCLcB/s1600/my_output_new_blog.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://1.bp.blogspot.com/-nDeQTymvyoE/WH2JrP0YVMI/AAAAAAAAAIg/-4ELa2I5ahgzOCDroFeFbhKoQufgbQ0TwCLcB/s320/my_output_new_blog.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div>
<b><span style="font-size: large;">Conclusion</span></b></div>
<b><span style="font-size: large;"><br /></span></b>
<br />
<div>
In this post I explained how you can extract images that are generated in Azure Machine Learning Studio. This is a step that might help you out in speeding up your data science prototypes. </div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/00885379426770757526noreply@blogger.com9tag:blogger.com,1999:blog-7939626009906644084.post-34986834616498365272017-01-01T15:50:00.002-08:002017-01-01T18:15:36.394-08:00Building your first cloud application using blob storage<span style="font-size: large;"><b>
Introduction</b></span><br />
<br />
<div style="text-align: justify;">
In this blog post I will teach you how you can build your first application with blob storage and C#. The tools that you need to get started are described in the previous <a href="http://blog.datainstructors.com/2016/11/how-to-get-started-with-development-in.html">blog post</a>. In this blog post I will first repeat the use case and next I will start building the use case step by step. Good luck with the different exercises and if you get stuck, please leave a comment below.</div>
<br />
<b><span style="font-size: large;">
The Use Case</span></b><br />
<div>
<br /></div>
<div style="text-align: justify;">
We are building the application that is shown in the picture below. We have locally on our computer the file <i>C:\test_files\Awesome_local_file.txt</i>. We have a web service in the cloud that performs a special operation on this file that we can't perform on our file locally. Therefore we will need to upload the Awesome_local_file.txt to <i>my_container</i> as the blob <i>larf_YYYYMMDDHHmm.txt. </i>In this way the Web service will be able to access the file and perform his operation on it. The web service will provide as output the <i>olarf_YYYYMMDDHHmm.txt</i> file which it will store in the container<i> mycontainer</i>. Afterwards you will be able to download this file back to your computer as <i>Awesome_output.txt</i>. </div>
<h2 style="text-align: center;">
<img height="265" src="https://3.bp.blogspot.com/-rsYUbuP5Mlk/WCfXRSq49QI/AAAAAAAAAB8/8nUpN7v13XUIo6CHAO-z36aMHg85eiXIwCLcB/s640/architecture.png" width="640" /></h2>
<span style="font-size: large;"><b>The structure of the exercises.</b></span><br />
You can find <a href="https://github.com/CSharpDevConnect/GetStartedWithBlobStorage/blob/master/ConsoleApplication1/Program.cs">here</a> the overview of all the exercises, so that you can each time copy/paste what you need when you are making the exercises.<br />
<br />
<div style="text-align: justify;">
Build the Visual Studio Project as described in the <a href="http://blog.datainstructors.com/2016/11/how-to-get-started-with-development-in.html">previous blog</a> and make sure that you have started the Microsoft Azure Storage Emulator. Change your <i>Main</i> function in the <i>Program.cs</i> file so that it looks like the code below. When you are working on an exercise, you need to delete the <i>// </i>from the fourth line and adjust the number from the exercise that you are working on and next you can execute your exercise.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-xlE-vCMWvVA/WGlhGvlWUZI/AAAAAAAAADo/tSXk8_lRuWoMlFZ5UJLlmIWbPEECv_KPgCLcB/s1600/main_function_initial.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="140" src="https://2.bp.blogspot.com/-xlE-vCMWvVA/WGlhGvlWUZI/AAAAAAAAADo/tSXk8_lRuWoMlFZ5UJLlmIWbPEECv_KPgCLcB/s400/main_function_initial.png" width="400" /></a></div>
<div>
<br /></div>
<div style="text-align: justify;">
The following libraries will be used during the exercises. Make sure to add them in the beginning of the file <i>Program.cs</i>.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-MB2uAmBIHC8/WGl-b4k7AhI/AAAAAAAAAGs/iejrD01e8PIe4v4KJYg4TTCWo7NmS8TFwCLcB/s1600/libraries3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="70" src="https://3.bp.blogspot.com/-MB2uAmBIHC8/WGl-b4k7AhI/AAAAAAAAAGs/iejrD01e8PIe4v4KJYg4TTCWo7NmS8TFwCLcB/s640/libraries3.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Press Start from the toolbar and a Console window will appear, hit two times <i>Enter</i> and your Console Window will close again.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-9aJyhNCllo8/WGl7lnY6L0I/AAAAAAAAAGg/9eJuwP4Vg18trfCTqK_e3vo2wv0d8B_hACLcB/s1600/result_exercise_0_2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="44" src="https://2.bp.blogspot.com/-9aJyhNCllo8/WGl7lnY6L0I/AAAAAAAAAGg/9eJuwP4Vg18trfCTqK_e3vo2wv0d8B_hACLcB/s640/result_exercise_0_2.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: justify;">
Make a folder on your <i>C</i> drive that is called <i>test_files</i>. Make in this folder a text file that you give the name <i>Awesome_local_file.txt.</i> In what follows we will teach you the code how you can copy this file to blob storage, so that a potential web service in the cloud can use this.</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-oZur0gfEfLk/WGllbSbdLeI/AAAAAAAAAEI/qokIUdIEmpYxbl2L164QBYyk4J_m5QhBwCLcB/s1600/picture_initial.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="124" src="https://2.bp.blogspot.com/-oZur0gfEfLk/WGllbSbdLeI/AAAAAAAAAEI/qokIUdIEmpYxbl2L164QBYyk4J_m5QhBwCLcB/s640/picture_initial.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<span style="font-size: large;"><b>
Exercise_0</b></span><br />
<div>
<br /></div>
<div style="text-align: justify;">
Before we can upload <i>Awesome_local_file.txt</i> to the cloud, we need to be able to connect to our blob storage account and create a container in the cloud, the equivalent of a directory on your computer. We can achieve this by using the code of <i>Exercise_0</i> below. Add this function to the <i>Program.cs</i> file and make sure that you have uncommented <i>Exercise_0</i> in the <i>Main</i> file. Run your program.</div>
<h2>
<div style="direction: ltr; line-height: 90%; margin-bottom: 0pt; margin-left: 0in; margin-top: 10pt; text-indent: 0in; unicode-bidi: embed; word-break: normal;">
<div class="MsoNormal" style="line-height: 90%; margin-bottom: .0001pt; margin-bottom: 0cm; margin-left: 0cm; margin-right: 0cm; margin-top: 10.0pt;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-5vauXIsFcHU/WGll5fsbMmI/AAAAAAAAAEM/V-WKLHvunu4w8QiBrqxcJS9wTbfuseG0QCLcB/s1600/exercise_0_code.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="251" src="https://3.bp.blogspot.com/-5vauXIsFcHU/WGll5fsbMmI/AAAAAAAAAEM/V-WKLHvunu4w8QiBrqxcJS9wTbfuseG0QCLcB/s640/exercise_0_code.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-weight: normal;"><span style="font-size: small;"><br /></span></span></div>
<div class="separator" style="clear: both; text-align: justify;">
<span style="font-weight: normal;"><span style="font-size: small;">When you have executed your program. The container<i> mycontainer </i>has been created and this looks schematically like the following picture.</span></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-weight: normal;"><span style="font-size: small;"><br /></span></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-a2AaXJ20zQg/WGlnaJ5F7DI/AAAAAAAAAEU/Ds6DJtRP0XM7qy4YDb2n3sZpVJw0d8-1wCLcB/s1600/exercise_0_result.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="290" src="https://2.bp.blogspot.com/-a2AaXJ20zQg/WGlnaJ5F7DI/AAAAAAAAAEU/Ds6DJtRP0XM7qy4YDb2n3sZpVJw0d8-1wCLcB/s640/exercise_0_result.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: justify;">
<span style="font-size: small;"><span style="font-weight: normal;">If you want to check whether <i>mycontainer</i> has been created successfully, you can open the <a href="http://storageexplorer.com/">Microsoft Azure Storage Explorer </a>that we have installed in the previous blog. You will see that the container <i>mycontainer</i> is now part of your <i>Blob Containers</i> in the <i>(Development) Storage Account</i>.</span></span></div>
</div>
</div>
</h2>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-gbPF1S1cJzw/WGlpUPq47fI/AAAAAAAAAEg/YF0ggw91ZVUd0UdEt0OiY0bQNCEwroRFwCLcB/s1600/exercise_0_explorer.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="297" src="https://1.bp.blogspot.com/-gbPF1S1cJzw/WGlpUPq47fI/AAAAAAAAAEg/YF0ggw91ZVUd0UdEt0OiY0bQNCEwroRFwCLcB/s320/exercise_0_explorer.png" width="320" /></a></div>
<h2>
</h2>
<b><span style="font-size: large;"><span style="font-size: xx-small;">
Exercise_1 Se</span>nd Awesome_local_file.txt to the cloud</span></b><br />
<br />
<div style="text-align: justify;">
We will now copy <i>Awesome_local_file.txt</i> to the cloud. Because the name of this file might not be that unique in the cloud, we are adding in a unique identifier to this file in the cloud and we will call the file in the cloud <i>larf_YYYYMMDDHHmm.txt </i>for which we will derive YYYYMMDDHHmm from the current date and time. </div>
<div style="text-align: justify;">
When we have generated the file name, we will upload <i>Awesome_local_file.txt</i> to the cloud. At the end of the function we will return the name of the newly generated file so that we can use this function in the next exercises. </div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-E_fOXKoxBok/WGlrCQepgYI/AAAAAAAAAEs/8Qi8EXuZbqMHVq1xaSwhD-egWOWuxDLdwCLcB/s1600/exercise_1_code.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="426" src="https://4.bp.blogspot.com/-E_fOXKoxBok/WGlrCQepgYI/AAAAAAAAAEs/8Qi8EXuZbqMHVq1xaSwhD-egWOWuxDLdwCLcB/s640/exercise_1_code.png" width="640" /></a></div>
<br />
<div style="text-align: justify;">
Make sure to change in the <i>Main</i> function the fourth line to <i>Exercise_1</i> and run the program.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
We have schematically achieved the following result. In the next exercise we will list all the different files that are available in <i>mycontainer</i> using code. </div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<a href="https://3.bp.blogspot.com/-kz0xydIs0VQ/WGlsoWkVg3I/AAAAAAAAAE4/v4HNaP0dueEHfzi7My0uWMuUJrvudZF5ACLcB/s1600/exercise_1_result.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="294" src="https://3.bp.blogspot.com/-kz0xydIs0VQ/WGlsoWkVg3I/AAAAAAAAAE4/v4HNaP0dueEHfzi7My0uWMuUJrvudZF5ACLcB/s640/exercise_1_result.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<span style="text-align: start;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<span style="text-align: start;"><br /></span></div>
<div class="separator" style="clear: both; text-align: justify;">
If you can't wait till the end of Exercise_2 to see whether your exercise was successful, you can also use the Microsoft Azure Storage Explore. Make sure that you hit the refresh icon and when you click on <i>mycontainer</i>, you will see the <i>larf_YYYYMMDDHHmm.txt</i> that you have just created.</div>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-rWvfSpF1AjI/WGltLeYvUZI/AAAAAAAAAE8/qO5FYt6g2YQ6vLRE0zOLOk4AyoFWoeSuACLcB/s1600/larf_in_explorer.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="108" src="https://3.bp.blogspot.com/-rWvfSpF1AjI/WGltLeYvUZI/AAAAAAAAAE8/qO5FYt6g2YQ6vLRE0zOLOk4AyoFWoeSuACLcB/s640/larf_in_explorer.png" width="640" /></a></div>
<h2>
<span style="font-size: large;"><br /></span></h2>
<h2>
<span style="font-size: large;">
Exercise_2 Getting a listing from the files in mycontainer</span></h2>
<div style="text-align: justify;">
Now we will learn how you can get the list of all the files in <i>mycontainer</i> using code. First you will need to connect to your storage account and ask for a reference to <i>mycontainer</i>. We will get all the different blob items in the container and will print the information of the <i>CloudBlockBlob</i>s, this is the way that a text file is stored in Azure Blob storage. </div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-F9mM0wbn0Pc/WGlvwXxhrHI/AAAAAAAAAFQ/WqouWybAJ3wHKUL6lalTtr0gTRPo1-tFwCLcB/s1600/exercise_2_code.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="278" src="https://2.bp.blogspot.com/-F9mM0wbn0Pc/WGlvwXxhrHI/AAAAAAAAAFQ/WqouWybAJ3wHKUL6lalTtr0gTRPo1-tFwCLcB/s640/exercise_2_code.png" width="640" /></a></div>
<br />
<div style="text-align: justify;">
Change in the main function <i>Exercise_1</i> to <i>Exercise_2</i> and run the program. You will see something similar as below in your console window. </div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-ho2D7atL2Qo/WGlwPF66t_I/AAAAAAAAAFU/kBQH9F1XbR0M0h752wlmwIw2jyPAqrAjwCLcB/s1600/result_exercise_2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="58" src="https://1.bp.blogspot.com/-ho2D7atL2Qo/WGlwPF66t_I/AAAAAAAAAFU/kBQH9F1XbR0M0h752wlmwIw2jyPAqrAjwCLcB/s640/result_exercise_2.png" width="640" /></a></div>
<br />
<div style="text-align: justify;">
As you may notice from the console window, the (Development) storage account is also called the <i>devstoreaccount1</i>. When you look back at the Microsoft Azure Storage Explorer, you also will see the <i>devstoreaccount1. </i></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
As an extra exercise, you can also add in the sequence of <i>Exericse_2</i>, <i>Exercise_1</i> and <i>Exercise_2</i> in the main file and you can see that <i>Exercise_1</i> has added in an extra file in <i>mycontainer</i>. </div>
<br />
<span style="font-size: large;"><b>
Exercise_3 Mimic the web service and download the result</b></span><br />
<br />
<div style="text-align: justify;">
Because we have now a copy of <i>Awesome_local_file.txt</i> in the cloud, the Web service will be able to reach it. We will reuse <i>Exercise_1 </i>for this<i>. </i>The Web service will perform some awesome manipulation on this file and will return the result as <i>olarf_YYYYMMDDHHmm.txt. </i>We will mimic this by making a copy of our file in the cloud. Finally we want to have a copy of the result locally and we will therefore download this file to a local file <i>AwesomeOutput.txt </i>in the <i>test_files</i> directory. </div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-l3kILCX8ONw/WGlzvP8KsEI/AAAAAAAAAFg/WLkuvf69DIMAfGddPyURGGn0XfXgEXOWwCLcB/s1600/exercise_3_code.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="https://2.bp.blogspot.com/-l3kILCX8ONw/WGlzvP8KsEI/AAAAAAAAAFg/WLkuvf69DIMAfGddPyURGGn0XfXgEXOWwCLcB/s640/exercise_3_code.png" width="640" /></a></div>
<br />
<div style="text-align: justify;">
Make sure again that you are referencing <i>Exercise_3</i> in the main function and run your program. Go to your <i>test_files</i> directory and you should have now the <i>AwesomeOutput.txt</i> file. </div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-UJ2oy03gopA/WGl5PV17a1I/AAAAAAAAAGA/vw0_mf_qMyoOh1Dg1g_VEgrHsdFukSYZQCLcB/s1600/exercise_3_result.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="290" src="https://3.bp.blogspot.com/-UJ2oy03gopA/WGl5PV17a1I/AAAAAAAAAGA/vw0_mf_qMyoOh1Dg1g_VEgrHsdFukSYZQCLcB/s640/exercise_3_result.png" width="640" /></a></div>
<br />
<span style="font-size: large;"><b>
Exercise_4 Make sure to have a nice clean up routine</b></span><br />
<span style="font-size: large;"><b><br /></b></span>
<br />
<div style="text-align: justify;">
Use your preference to check which files are available in <i>mycontainer</i>. You will see that for each time that you are executing <i>Exercise_1</i> or <i>Exercise_3</i>, you will be generating extra files in the cloud. When you are using the cloud, you should make sure that you are nicely cleaning up your unneeded files because you are paying for these unneeded files. Therefore we will be adding in code now to remove the two cloud files that are generated when you are executing the use case. </div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-qRhkp0WbtHs/WGl1l50aQiI/AAAAAAAAAFs/QlMEP5jUBmARDqPTn3Tf6BHIQu3jGvs3ACLcB/s1600/exercise_4_code.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="424" src="https://3.bp.blogspot.com/-qRhkp0WbtHs/WGl1l50aQiI/AAAAAAAAAFs/QlMEP5jUBmARDqPTn3Tf6BHIQu3jGvs3ACLcB/s640/exercise_4_code.png" width="640" /></a></div>
<br />
<div style="text-align: justify;">
Execute the following sequence <i>Exercise_2</i>, <i>Exercise_4</i>, <i>Exercise_2</i> in your main file. You will now see that no extra file has been generated. Similar to the schematic view below.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-DTC8ha5Mgbw/WGl5lyWHItI/AAAAAAAAAGE/UOWW803rsO41eXkk0YmJxOO3pddSTMq-gCLcB/s1600/exercise_4_result.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="290" src="https://1.bp.blogspot.com/-DTC8ha5Mgbw/WGl5lyWHItI/AAAAAAAAAGE/UOWW803rsO41eXkk0YmJxOO3pddSTMq-gCLcB/s640/exercise_4_result.png" width="640" /></a></div>
<br />
<span style="font-size: large;"><b>
Exercise_5 Clean up everything what was left from the initial exercises</b></span><br />
<span style="font-size: large;"><b><br /></b></span>
<br />
<div style="text-align: justify;">
When we are now executing our use case, we are not leaving any files behind in the cloud. Great. But what happens with the old files that we have generated and what happens with <i>mycontainer</i> when we eventually don't need it any more? Therefore we have a final clean up routine. We now will be removing all the extra files in <i>mycontainer</i> and finally will also remove the container. </div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-9ENfpoxk-e8/WGl3NCml_SI/AAAAAAAAAF0/wk2jWKj_7zI9GCCthyGRIHL8XvihqUz-gCLcB/s1600/exercise_code_5.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="264" src="https://1.bp.blogspot.com/-9ENfpoxk-e8/WGl3NCml_SI/AAAAAAAAAF0/wk2jWKj_7zI9GCCthyGRIHL8XvihqUz-gCLcB/s640/exercise_code_5.png" width="640" /></a></div>
<div>
<br /></div>
<div style="text-align: justify;">
Execute <i>Exercise_5</i> in your main file. You can next go to the Microsoft Azure Storage Explorer and when you refresh, you will see that <i>mycontainer</i> doesn't exist any more. Schematically this will look as below.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-NoArkCsqe74/WGl6QnjsIbI/AAAAAAAAAGM/AGHK7doNBak7jWbB8zaxVgp5WynCn7-8QCLcB/s1600/exercise_5_result.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="236" src="https://2.bp.blogspot.com/-NoArkCsqe74/WGl6QnjsIbI/AAAAAAAAAGM/AGHK7doNBak7jWbB8zaxVgp5WynCn7-8QCLcB/s640/exercise_5_result.png" width="640" /></a></div>
<br />
<br />
<span style="font-size: large;"><b>Conclusion</b></span><br />
<span style="font-size: large;"><b><br /></b></span>
<br />
<div style="text-align: justify;">
Congratulations if you have reached this part of my blog, that means that you have gained some extra knowledge about blob storage. You can find the code also on my <a href="https://github.com/CSharpDevConnect/GetStartedWithBlobStorage">github </a>account. In a next blog I will explain how a web service generated from Azure Machine Learning Studio can use these files from blob storage.</div>
Anonymoushttp://www.blogger.com/profile/00885379426770757526noreply@blogger.com4tag:blogger.com,1999:blog-7939626009906644084.post-74269043177710329762016-11-20T16:51:00.000-08:002017-01-01T17:43:41.307-08:00How to get started with development in the cloud for free: tools for blob storage development<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;"><b>Introduction</b></span><br />
<div>
<div style="text-align: justify;">
<span style="font-family: inherit;">Getting started in the cloud might be an overwhelming experience. When you are learning to work with the cloud, you also want to keep your development costs as low as possible. Therefore I will present with this and the following blog a use case that will allow you to write your first cloud blog storage program at no cost. We will use for this the cloud offering of Microsoft which is called Azure. </span></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<span style="font-family: inherit;">The use case will exist of uploading a file to the cloud so that a special cloud service would be able to process this file and return the processed result which you eventually can download again.</span></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<span style="font-family: inherit;">In this first blog I will present all the different development tools that you need to develop this use case. We will use an emulator for the cloud in this way you can postpone starting to use your free </span><a href="https://azure.microsoft.com/en-us/offers/ms-azr-0044p/" style="font-family: inherit;">Azure credits</a><span style="font-family: inherit;"> when you have a better idea about your first cloud project. </span></div>
<br />
<span style="font-family: inherit;"><br /></span>
<br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;"><b>What is Blob Storage</b></span></div>
<div>
<div style="text-align: justify;">
<span style="font-family: inherit;">Blob storage is a service for storing large amounts of data or files in the cloud. Such files are called blobs in the cloud. You can see this as your personal hard drive in the cloud. These files can be made accessible through http or https and you can make them publicly available to the world or you can use it to store private application data. </span></div>
<div style="text-align: justify;">
<span style="font-family: inherit;"><br /></span></div>
</div>
<div>
<div style="text-align: justify;">
<span style="font-family: inherit;">To be able to use blob storage in the cloud, you need a storage account. In your storage account you will have containers what you can compare with directories on your computer that are storing the blobs. If you would store a blob file <i>myblob</i> on the account <i>myaccount</i>, in container <i>mycontainer</i>. </span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">The url of this blob looks as follows: <a href="https://myaccount.blob.core.windows.net/mycontainer/myblob">https://myaccount.blob.core.windows.net/mycontainer/myblob</a> . The equivalent of this on your windows account <i>myuser</i> is a file <i>myfile</i> in directory <i>mydirectory</i> is the file <i>C:\Users\myuser\Documents\mydirectory\myblob</i>.</span></div>
<br />
<div style="text-align: justify;">
<span style="font-family: inherit;">The blob storage emulator will in this case also emulate the storage account. </span></div>
<br />
<span style="font-family: inherit;"><br /></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;"><b>Use case</b></span><br />
<div>
<div style="text-align: justify;">
<span style="font-family: inherit;">We are building the application that is shown in the picture below. We have locally on our computer the file <i>C:\test_files\Awesome_local_file.txt</i>. We have a web service in the cloud that performs a special operation on this file that we can't perform on our file locally. Therefore we will need to upload the <i>Awesome_local_file.txt</i> to <i>my_container</i> as the blob <i>larf_YYYYMMDDHHmm.txt</i> . In this way the Web service will be able to access the file and perform his operation on it. The web service will provide as output the <i>olarf_YYYYMMDDHHmm.txt</i> file which it will store in the container<i> mycontainer</i>. Afterwards you will be able to download this file back to your computer as <i>Awesome_output.txt</i>. </span></div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-rsYUbuP5Mlk/WCfXRSq49QI/AAAAAAAAAB8/8nUpN7v13XUIo6CHAO-z36aMHg85eiXIwCLcB/s1600/architecture.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: inherit;"><img border="0" height="165" src="https://3.bp.blogspot.com/-rsYUbuP5Mlk/WCfXRSq49QI/AAAAAAAAAB8/8nUpN7v13XUIo6CHAO-z36aMHg85eiXIwCLcB/s400/architecture.png" width="400" /></span></a></div>
<span style="font-family: inherit;"><br /></span></div>
<span style="font-family: inherit;"><br /></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;"><b>
Needed free resources for blob storage development</b></span><br />
<span style="font-family: inherit;"><b>Development Environment</b></span><br />
<div>
<div style="text-align: justify;">
<span style="font-family: inherit;">This part is based on <a href="https://docs.microsoft.com/en-us/azure/storage/storage-dotnet-how-to-use-blobs">Get started with Azure Blob storage using .NET</a> As the development environment we will use Visual Studio. <a href="https://www.visualstudio.com/vs/community/">Visual Studio Community</a> is a free version of Visual Studio. </span></div>
</div>
<div>
<div style="text-align: justify;">
<span style="font-family: inherit;"><br /></span></div>
</div>
<div>
<div style="text-align: justify;">
<span style="font-family: inherit;">When you have installed Visual Studio, open Visual Studio and select <b>New Project</b> in the left column and select a new Visual C# Console Application as demonstrated in the picture below and press <b>OK</b>.</span></div>
</div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-wav43t0uydE/WDDyHspnA_I/AAAAAAAAACY/UUY_UrqrDg8g3n3VNzZvdJgNt2TYT6v6ACLcB/s1600/MakeApplication.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: inherit;"><img border="0" height="443" src="https://1.bp.blogspot.com/-wav43t0uydE/WDDyHspnA_I/AAAAAAAAACY/UUY_UrqrDg8g3n3VNzZvdJgNt2TYT6v6ACLcB/s640/MakeApplication.png" width="640" /></span></a></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<span style="font-family: inherit;"><br /></span>
<br />
<div style="text-align: justify;">
<span style="font-family: inherit;"><span style="font-family: inherit;">When you have created your project, you will need to add some extra libraries. You will be able to do this with the </span><b style="font-family: inherit;">Nuget Package Manager</b><span style="font-family: inherit;">. You can launch it by Selecting </span><b style="font-family: inherit;">Nuget Package Manager</b><span style="font-family: inherit;"> from the </span><b style="font-family: inherit;">Tools</b><span style="font-family: inherit;"> menu. Next you can select the </span><b style="font-family: inherit;">Nuget Package Manager Console</b><span style="font-family: inherit;">.</span></span></div>
<br />
<span style="font-family: inherit;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-SvlC4W-CqPM/WDHBKcG4BkI/AAAAAAAAADM/gIrm-r9E0iwSvPrmWOiqNyRSccWVeMS9wCLcB/s1600/NugetSetup_part1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="299" src="https://1.bp.blogspot.com/-SvlC4W-CqPM/WDHBKcG4BkI/AAAAAAAAADM/gIrm-r9E0iwSvPrmWOiqNyRSccWVeMS9wCLcB/s640/NugetSetup_part1.png" width="640" /></a></div>
<span style="font-family: inherit;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div style="text-align: justify;">
<span style="font-family: inherit;"><span style="font-family: inherit;">In the console in the bottom type the following two commands</span></span></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-taruU1uPk0w/WDHBhl2urpI/AAAAAAAAADQ/18EN_7RdLig7WzwB6OZrRcrPgSW3QglawCLcB/s1600/NugetSetup_part2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="163" src="https://4.bp.blogspot.com/-taruU1uPk0w/WDHBhl2urpI/AAAAAAAAADQ/18EN_7RdLig7WzwB6OZrRcrPgSW3QglawCLcB/s640/NugetSetup_part2.png" width="640" /></a></div>
<span style="font-family: inherit;"><br /></span>
<br />
<ul>
<li>Install-Package WindowsAzure.Storage</li>
<li>Install-Package Microsoft.WindowsAzure.ConfigurationManager</li>
</ul>
<div style="text-align: justify;">
<span style="font-family: inherit;"><b style="font-family: inherit;">Azure Storage Emulator</b></span></div>
<br />
<div>
<div style="text-align: justify;">
<span style="font-family: inherit;">For this use case we will use a blob storage emulator, this will mean that your computer will emulate the blob storage environment in the cloud locally and you don't need to generate an Azure account yet. To be able to use the Azure Storage Emulator, you first will need to have an instance of SQL Server installed. A free option is offered in <a href="https://www.microsoft.com/en-us/download/details.aspx?id=52679">SQL Server 2016 Express</a>. </span></div>
</div>
<div>
<div style="text-align: justify;">
<span style="font-family: inherit;"><br /></span></div>
</div>
<div>
<div style="text-align: justify;">
<span style="font-family: inherit;">When SQL Server has been successfully installed, you will be able to install the <a href="https://go.microsoft.com/fwlink/?linkid=717179&clcid=0x409">Azure Storage Emulator.</a> When you have installed the storage emulator you can start it from the program menu. You will see a command window similar such as the one below when everything has run correctly.</span></div>
</div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-JsYHMBGcroA/WDD7ItfETfI/AAAAAAAAACw/e-MD4ALxpbQdiqeurpvXltFqzRvMeTw7wCEw/s1600/CmdStorageEmulator.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: inherit;"><img border="0" height="336" src="https://4.bp.blogspot.com/-JsYHMBGcroA/WDD7ItfETfI/AAAAAAAAACw/e-MD4ALxpbQdiqeurpvXltFqzRvMeTw7wCEw/s640/CmdStorageEmulator.png" width="640" /></span></a></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<div style="text-align: justify;">
<span style="font-family: inherit;"><b>Authenticating requests against the storage emulator</b></span></div>
</div>
<div>
<div style="text-align: justify;">
<span style="font-family: inherit;">The benefit of using the storage emulator is that the account name and the account key are the same for all the developers using the storage emulator, so these ones don't need to be kept secret. These values are: </span></div>
</div>
<div>
<div style="text-align: justify;">
<div style="text-align: left;">
<span style="font-family: inherit;"><span style="font-weight: bold; text-indent: 0in; vertical-align: baseline;">Account
name</span><span style="text-indent: 0in; vertical-align: baseline;">:
devstoreaccount1</span></span></div>
</div>
</div>
<div style="text-align: justify;">
<div style="text-align: left;">
<span style="font-family: inherit;"><span style="font-weight: bold; vertical-align: baseline;">Account key</span><span style="vertical-align: baseline;">:
Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/</span><span style="vertical-align: baseline;">KBHBeksoGMGw</span><span style="vertical-align: baseline;">==</span></span></div>
</div>
<div>
<br />
<div style="text-align: justify;">
<span style="font-family: inherit;">When you want to use this emulator in Visual Studio, you need to put the following values in the app.config file. But we will cover this more in the next blog when we are building up the use case.</span></div>
<br />
<div style="text-align: justify;">
<span style="font-family: inherit;"><br /></span>
</div>
<div class="separator" style="clear: both; text-align: justify;">
<a href="https://3.bp.blogspot.com/-dHuPq-YQGh8/WDHAfGkx_vI/AAAAAAAAADI/fV3vIVOsGikgms1B1CAJRUOzoCCZabB_wCLcB/s1600/app_config.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="59" src="https://3.bp.blogspot.com/-dHuPq-YQGh8/WDHAfGkx_vI/AAAAAAAAADI/fV3vIVOsGikgms1B1CAJRUOzoCCZabB_wCLcB/s640/app_config.png" width="640" /></a></div>
<div style="direction: ltr; line-height: normal; margin: 0pt 0in; text-indent: 0in; unicode-bidi: embed; vertical-align: baseline; word-break: normal;">
<div style="text-align: justify;">
<br /></div>
</div>
</div>
<div>
<br />
<div style="text-align: justify;">
<b style="font-family: inherit;">Microsoft Azure Storage Explorer</b></div>
</div>
<div style="text-align: justify;">
<span style="font-family: inherit;">To be able to access your files in storage with a desktop application similar to Windows Explorer, you can install <a href="http://storageexplorer.com/">Microsoft Azure Storage Explorer</a>, To connect to your storage emulator, use the <i>Account key</i> <b><span style="vertical-align: baseline;">Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/</span><span style="vertical-align: baseline;">KBHBeksoGMGw</span></b><span style="vertical-align: baseline;"><b>== </b>and the <i>account name</i> </span><b>devstoreaccount1</b>. </span></div>
<br />
<div style="text-align: justify;">
<span style="font-family: inherit;">The Microsoft Azure Storage Explore will look now like the picture below. </span></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-DcTxALn4nas/WDD-3Y01Q2I/AAAAAAAAAC4/hjty1iH4ihcVSHRWfn71eIzkVgbClqaUwCLcB/s1600/StorageAccount.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: inherit;"><img border="0" height="494" src="https://2.bp.blogspot.com/-DcTxALn4nas/WDD-3Y01Q2I/AAAAAAAAAC4/hjty1iH4ihcVSHRWfn71eIzkVgbClqaUwCLcB/s640/StorageAccount.png" width="640" /></span></a></div>
<br />
<div>
<div style="text-align: justify;">
<span style="font-family: inherit; font-size: large;"><b>Conclusion</b></span></div>
<div style="text-align: justify;">
<span style="font-family: inherit;">Congratulations, now you have set up all the different parts to be able to get started with blog storage. In the next blog we will show step by step how you can build the use case. Good luck with your journey to the cloud.</span></div>
</div>
Anonymoushttp://www.blogger.com/profile/00885379426770757526noreply@blogger.com2tag:blogger.com,1999:blog-7939626009906644084.post-44323613224140227122016-11-06T16:44:00.000-08:002017-06-22T17:45:11.888-07:00Efficient debugging of cloud applications using an Azure case study<div class="graf graf--p graf-after--h3" id="6c7a" name="6c7a" style="background-color: white; color: rgba(0, 0, 0, 0.8); letter-spacing: -0.003em; line-height: 1.58; margin-top: 10px;">
In the current world of software development cloud computing has taken an important place. The triage of software crashes however has become more challenging because of the use of the cloud. Software crashes never happen at a good time, therefore it is important that you beforehand know where you can get all your logs. So that you can build an efficient triage strategy.</div>
<div class="graf graf--p graf-after--p" id="470c" name="470c" style="background-color: white; color: rgba(0, 0, 0, 0.8); letter-spacing: -0.003em; line-height: 1.58; margin-top: 29px;">
In this blog post I will explain some tips learnt from analyzing crashes from a cloud data solution that I recently developed. This data solution was a web service derived from <a class="markup--anchor markup--p-anchor" data-href="https://blogs.msdn.microsoft.com/microsoft_press/2015/04/15/free-ebook-microsoft-azure-essentials-azure-machine-learning/" href="https://blogs.msdn.microsoft.com/microsoft_press/2015/04/15/free-ebook-microsoft-azure-essentials-azure-machine-learning/" rel="nofollow" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0.439216); background-color: transparent; background-image: linear-gradient(rgba(0, 0, 0, 0.6) 50%, rgba(0, 0, 0, 0) 50%); background-position: 0px 1.07em; background-repeat: repeat-x; background-size: 2px 0.1em; text-decoration: none;" target="_blank">Azure Machine Learning Studio</a> (Azure ML) that consumes and produces <a class="markup--anchor markup--p-anchor" data-href="https://azure.microsoft.com/en-us/documentation/articles/storage-dotnet-how-to-use-blobs/" href="https://azure.microsoft.com/en-us/documentation/articles/storage-dotnet-how-to-use-blobs/" rel="nofollow" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0.439216); background-color: transparent; background-image: linear-gradient(rgba(0, 0, 0, 0.6) 50%, rgba(0, 0, 0, 0) 50%); background-position: 0px 1.07em; background-repeat: repeat-x; background-size: 2px 0.1em; text-decoration: none;" target="_blank">blob storage files</a>. Blob storage is the data storage solution from Azure. Although some of the tips are only related to Azure ML and blob storage, they can also help you out to build an efficient triage strategy for other cloud solutions.</div>
<div class="graf graf--p graf-after--p" id="739e" name="739e" style="background-color: white; color: rgba(0, 0, 0, 0.8); letter-spacing: -0.003em; line-height: 1.58; margin-top: 29px;">
In the next part of this blog I will first describe the cloud architecture, then the different logs that are available and how you can access them and finally, I will discuss some bug triage strategies.</div>
<div class="graf graf--p graf-after--p" id="739e" name="739e" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-serif-font, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 21px; letter-spacing: -0.003em; line-height: 1.58; margin-top: 29px;">
<span style="font-family: , "lucida grande" , "lucida sans unicode" , "lucida sans" , "geneva" , "arial" , sans-serif; font-size: 32px; letter-spacing: -0.02em;">High level overview of the cloud architecture</span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-tbQxk_YETk4/WCe2rfM-vmI/AAAAAAAAABo/PZPcGaMy-m06iWNbDVL3AEdtLETeg-GCQCLcB/s1600/new_architecture.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="203" src="https://1.bp.blogspot.com/-tbQxk_YETk4/WCe2rfM-vmI/AAAAAAAAABo/PZPcGaMy-m06iWNbDVL3AEdtLETeg-GCQCLcB/s320/new_architecture.jpg" width="320" /></a></div>
<h4 class="graf graf--h4 graf-after--figure" id="7208" name="7208" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-sans-serif-font, "Lucida Grande", "Lucida Sans Unicode", "Lucida Sans", Geneva, Arial, sans-serif; font-size: 24px; letter-spacing: -0.018em; line-height: 1.22; margin: 39px 0px 0px -1.5px;">
Web service from Azure Machine ML</h4>
<div class="graf graf--p graf-after--h4" id="d846" name="d846" style="background-color: white; color: rgba(0, 0, 0, 0.8); letter-spacing: -0.003em; line-height: 1.58; margin-top: 6px;">
In the picture above, you can see an overview of the architecture that I built around the web service derived from Azure ML. The only important thing that you need to know about Azure ML is that it provides you a lot of different data science and machine learning algorithms in an easy way to solve complex data problems. Such a solution is called an experiment in Azure ML. An experiment will consist of different components that are connected to each other. When your experiment satisfies the requirements, <a class="markup--anchor markup--p-anchor" data-href="https://azure.microsoft.com/en-us/documentation/articles/machine-learning-walkthrough-5-publish-web-service/" href="https://azure.microsoft.com/en-us/documentation/articles/machine-learning-walkthrough-5-publish-web-service/" rel="nofollow" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0.439216); background-color: transparent; background-image: linear-gradient(rgba(0, 0, 0, 0.6) 50%, rgba(0, 0, 0, 0) 50%); background-position: 0px 1.07em; background-repeat: repeat-x; background-size: 2px 0.1em; text-decoration: none;" target="_blank">you can deploy it as a web service</a> and make it part of your cloud solution.</div>
<div class="graf graf--p graf-after--p" id="316e" name="316e" style="background-color: white; color: rgba(0, 0, 0, 0.8); letter-spacing: -0.003em; line-height: 1.58; margin-top: 29px;">
Web services can both be consumed in a Request-Response Service and in a Batch Execution Service (BES) mode. The first one means that you only provide one data point to be evaluated by the web service and the second one means that you provide larger volumes of data to be consumed by the web service. In this example I use the BES mode.</div>
<h4 class="graf graf--h4 graf-after--p" id="2d45" name="2d45" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-sans-serif-font, "Lucida Grande", "Lucida Sans Unicode", "Lucida Sans", Geneva, Arial, sans-serif; font-size: 24px; letter-spacing: -0.018em; line-height: 1.22; margin: 30px 0px 0px -1.5px;">
Feeding the web service from blob storage</h4>
<div class="graf graf--p graf-after--h4" id="9cc2" name="9cc2" style="background-color: white; color: rgba(0, 0, 0, 0.8); letter-spacing: -0.003em; line-height: 1.58; margin-top: 6px;">
The web service is fed with files from <a class="markup--anchor markup--p-anchor" data-href="https://azure.microsoft.com/en-us/documentation/articles/storage-dotnet-how-to-use-blobs/" href="https://azure.microsoft.com/en-us/documentation/articles/storage-dotnet-how-to-use-blobs/" rel="nofollow" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0.439216); background-color: transparent; background-image: linear-gradient(rgba(0, 0, 0, 0.6) 50%, rgba(0, 0, 0, 0) 50%); background-position: 0px 1.07em; background-repeat: repeat-x; background-size: 2px 0.1em; text-decoration: none;" target="_blank">Azure blog storage</a>. Azure blob storage is a service for easily storing files in the cloud. In the current architecture, I make changes of my files locally, then these files are automatically uploaded to blob storage with a script. The web service consumes these blobs and produces new blobs. The blobs are finally downloaded to my local machine.</div>
<div class="graf graf--p graf-after--p" id="e75e" name="e75e" style="background-color: white; color: rgba(0, 0, 0, 0.8); letter-spacing: -0.003em; line-height: 1.58; margin-top: 29px;">
Multiple instances of this web service can run at the same time and if the maximum number of simultaneous instances has been reached, the instances are queued till one has been finished. In a perfect world, everything runs perfectly and my blog post would end here. However, bugs and crashes are part of real life. Therefore, I will provide now some tips that can help with efficient triage of bugs so that you can deal with them when they appear in production.</div>
<h3 class="graf graf--h3 graf-after--p" id="7fcb" name="7fcb" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-sans-serif-font, "Lucida Grande", "Lucida Sans Unicode", "Lucida Sans", Geneva, Arial, sans-serif; font-size: 32px; letter-spacing: -0.02em; line-height: 1.15; margin: 56px 0px 0px -2px;">
Have access to all the evidence</h3>
<h4 class="graf graf--h4 graf-after--h3" id="a06d" name="a06d" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-sans-serif-font, "Lucida Grande", "Lucida Sans Unicode", "Lucida Sans", Geneva, Arial, sans-serif; font-size: 24px; letter-spacing: -0.018em; line-height: 1.22; margin: 31px 0px 0px -1.5px;">
Know which logs are available and how you can acquire them</h4>
<div class="graf graf--p graf-after--h4" id="7481" name="7481" style="background-color: white; color: rgba(0, 0, 0, 0.8); letter-spacing: -0.003em; line-height: 1.58; margin-top: 6px;">
One of the most important things before you can start the triage process is having access to all available logs. On the one hand, the cloud components that you are using might be generating client side logs that you can easily access from your cloud portal. They might however not be turned on by default. On the other hand, your cloud provider also might be generating server side logs that you might get access to in case of need. Finally, you also must make sure that your own application generates useful logs that you also save in a consistent way.</div>
<h4 class="graf graf--h4 graf-after--p" id="6715" name="6715" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-sans-serif-font, "Lucida Grande", "Lucida Sans Unicode", "Lucida Sans", Geneva, Arial, sans-serif; font-size: 24px; letter-spacing: -0.018em; line-height: 1.22; margin: 30px 0px 0px -1.5px;">
Turn on client side logging in Azure ML</h4>
<div class="graf graf--p graf-after--h4" id="9cc5" name="9cc5" style="background-color: white; color: rgba(0, 0, 0, 0.8); letter-spacing: -0.003em; line-height: 1.58; margin-top: 6px;">
To access your logs, you need an Azure account. The client side logs are not automatically turned on in Azure ML. <a class="markup--anchor markup--p-anchor" data-href="https://azure.microsoft.com/en-us/documentation/articles/machine-learning-web-services-logging" href="https://azure.microsoft.com/en-us/documentation/articles/machine-learning-web-services-logging" rel="nofollow" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0.439216); background-color: transparent; background-image: linear-gradient(rgba(0, 0, 0, 0.6) 50%, rgba(0, 0, 0, 0) 50%); background-position: 0px 1.07em; background-repeat: repeat-x; background-size: 2px 0.1em; text-decoration: none;" target="_blank">You can turn on these logs through the Classic portal or through the Azure Machine Learning Web Services portal.</a> I prefer to use the Azure Classic Portal. You will be able to find these logs in the <em class="markup--em markup--p-em" style="font-feature-settings: 'liga' 1, 'salt' 1;">ml-diagnostics</em> folder in your blob storage.</div>
<h4 class="graf graf--h4 graf-after--p" id="f683" name="f683" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-sans-serif-font, "Lucida Grande", "Lucida Sans Unicode", "Lucida Sans", Geneva, Arial, sans-serif; font-size: 24px; letter-spacing: -0.018em; line-height: 1.22; margin: 30px 0px 0px -1.5px;">
Get familiar with the log file structure</h4>
<div class="graf graf--p graf-after--h4" id="04b8" name="04b8" style="background-color: white; color: rgba(0, 0, 0, 0.8); letter-spacing: -0.003em; line-height: 1.58; margin-top: 6px;">
Your log files will be stored in a blob called <em class="markup--em markup--p-em" style="font-feature-settings: 'liga' 1, 'salt' 1;">ml-diagnostics</em> and each web service will have a separate unique identifier. When you are running your web service, you will see folders with each run of your web service.</div>
<div class="graf graf--p graf-after--p" id="cbef" name="cbef" style="background-color: white; color: rgba(0, 0, 0, 0.8); letter-spacing: -0.003em; line-height: 1.58; margin-top: 29px;">
If you look into the folder belonging to the run of a web service. You will find files that are structured in the following way <em class="markup--em markup--p-em" style="font-feature-settings: 'liga' 1, 'salt' 1;">‘COMPONENT_TYPE’_’NB’.stdout</em>and <em class="markup--em markup--p-em" style="font-feature-settings: 'liga' 1, 'salt' 1;">‘COMPONENT_TYPE’_NB.stderr</em>. In the <em class="markup--em markup--p-em" style="font-feature-settings: 'liga' 1, 'salt' 1;">stdout</em> file you will find normal output information and in the<em class="markup--em markup--p-em" style="font-feature-settings: 'liga' 1, 'salt' 1;"> stderr</em> file you will find the error information.</div>
<div class="graf graf--p graf-after--p" id="8640" name="8640" style="background-color: white; color: rgba(0, 0, 0, 0.8); letter-spacing: -0.003em; line-height: 1.58; margin-top: 29px;">
Examples for the <em class="markup--em markup--p-em" style="font-feature-settings: 'liga' 1, 'salt' 1;">‘COMPONENT_TYPE’</em> are<em class="markup--em markup--p-em" style="font-feature-settings: 'liga' 1, 'salt' 1;"> Apply%20SQL%20Tranformation</em>, <em class="markup--em markup--p-em" style="font-feature-settings: 'liga' 1, 'salt' 1;">Execute%20R%20Script</em>, <em class="markup--em markup--p-em" style="font-feature-settings: 'liga' 1, 'salt' 1;">Join%20Dat</em>a. Which refer to different components like SQL Transformations, R scripts and Join components that you can add in Azure ML.</div>
<div class="graf graf--p graf-after--p" id="fd03" name="fd03" style="background-color: white; color: rgba(0, 0, 0, 0.8); letter-spacing: -0.003em; line-height: 1.58; margin-top: 29px;">
It is however still a challenge to know which <em class="markup--em markup--p-em" style="font-feature-settings: 'liga' 1, 'salt' 1;">‘NB’</em> corresponds to which component in Azure ML. For the Python script and the R script components you can solve this easily by adding in an extra print command with a useful identifier that makes it easier to determine which of the components you are looking at. For the other ones you need to dig deeper into the file structure.</div>
<h4 class="graf graf--h4 graf-after--p" id="3261" name="3261" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-sans-serif-font, "Lucida Grande", "Lucida Sans Unicode", "Lucida Sans", Geneva, Arial, sans-serif; font-size: 24px; letter-spacing: -0.018em; line-height: 1.22; margin: 30px 0px 0px -1.5px;">
Save your local logs</h4>
<div class="graf graf--p graf-after--h4" id="be93" name="be93" style="background-color: white; color: rgba(0, 0, 0, 0.8); letter-spacing: -0.003em; line-height: 1.58; margin-top: 6px;">
Also, make sure that you keep track of your local logs in which you are calling the web service. This will help you out later on with debugging issues. Some information that is useful here are: time stamps, unique identifiers, files that you are saving, errors thrown by the web service. Also make sure that you keep the information that you need to revert changes in case of a crash.</div>
<h4 class="graf graf--h4 graf-after--p" id="05f1" name="05f1" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-sans-serif-font, "Lucida Grande", "Lucida Sans Unicode", "Lucida Sans", Geneva, Arial, sans-serif; font-size: 24px; letter-spacing: -0.018em; line-height: 1.22; margin: 30px 0px 0px -1.5px;">
Save your local logs</h4>
<div class="graf graf--p graf-after--h4" id="1ae7" name="1ae7" style="background-color: white; color: rgba(0, 0, 0, 0.8); letter-spacing: -0.003em; line-height: 1.58; margin-top: 6px;">
Last but not least, make also sure that you keep track of your local logs in which you are calling the web service. This will help you out later on with debugging issues. Some information that is useful here are: time stamps, unique identifiers, files that you are saving, errors thrown by the web service. Also make sure that you keep the information that you need to revert changes in case of a crash.</div>
<h4 class="graf graf--h4 graf-after--p" id="aff2" name="aff2" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-sans-serif-font, "Lucida Grande", "Lucida Sans Unicode", "Lucida Sans", Geneva, Arial, sans-serif; font-size: 24px; letter-spacing: -0.018em; line-height: 1.22; margin: 30px 0px 0px -1.5px;">
Ask an example of a cloud side log in a normal case</h4>
<div class="graf graf--p graf-after--h4" id="f454" name="f454" style="background-color: white; color: rgba(0, 0, 0, 0.8); letter-spacing: -0.003em; line-height: 1.58; margin-top: 6px;">
At the cloud side there also might be extra log information available that you can’t access yourself directly. If your web service is a critical component of your data solution it might be a good idea to ask support for these log files. That way you have an idea what information you can derive from these log files. You can then also ask for these logs when you run into a problem.<br />
<span style="font-size: 32px; letter-spacing: -0.02em;"><br /></span>
<span style="font-size: 32px; letter-spacing: -0.02em;">Catching the bugs</span></div>
<h4 class="graf graf--h4 graf-after--h3" id="e359" name="e359" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-sans-serif-font, "Lucida Grande", "Lucida Sans Unicode", "Lucida Sans", Geneva, Arial, sans-serif; font-size: 24px; letter-spacing: -0.018em; line-height: 1.22; margin: 31px 0px 0px -1.5px;">
Is it a missing data issue, a data formatting issue or something else?</h4>
<div class="graf graf--p graf-after--h4" id="dfe9" name="dfe9" style="background-color: white; color: rgba(0, 0, 0, 0.8); letter-spacing: -0.003em; line-height: 1.58; margin-top: 6px;">
The difference between a data issue and a formatting error is tricky. A formatting error means that there is an error in one of the input files which means that the web service couldn’t run till the end. A data issue means that the data was uploaded to blob storage but that a component of the web service started running before the data was available. This also means that the web service couldn’t run till the end. Besides this you also can have a cloud side error which also will mean that your web service didn’t run till the end.</div>
<div class="graf graf--p graf-after--p" id="814d" name="814d" style="background-color: white; color: rgba(0, 0, 0, 0.8); letter-spacing: -0.003em; line-height: 1.58; margin-top: 29px;">
After some trial and error, I have established the following steps to determine the root cause of the issue.</div>
<ol class="postList" style="background-color: white; color: rgba(0, 0, 0, 0.8); list-style: none none; margin: 29px 0px 0px; padding: 0px;">
<li class="graf graf--li graf-after--p" id="8a3e" name="8a3e" style="letter-spacing: -0.003em; line-height: 1.58; margin-bottom: 14px; margin-left: 30px;">1. I go in Azure Machine Learning Studio and I run the Azure Machine Learning experiment with the input blobs that caused the bugs.</li>
<li class="graf graf--li graf-after--li" id="c969" name="c969" style="letter-spacing: -0.003em; line-height: 1.58; margin-bottom: 14px; margin-left: 30px;">2. If the experiment runs till the end smoothly I will examine the log files from the last component and will look for something that says 0 rows. If I find this, this means that a component started to run before the data was available. In the other case there might be a cloud side issue which I will discuss later.</li>
<li class="graf graf--li graf-after--li" id="8e67" name="8e67" style="letter-spacing: -0.003em; line-height: 1.58; margin-bottom: 0px; margin-left: 30px;">3. If the experiment fails, I now know on which component it failed. During development of the R and Python components I made sure to add in enough logging information and this will easily help me to track down the issue and resolve it. The fix will mean fixing the local input file and uploading the fixed file to the cloud.</li>
</ol>
<div class="graf graf--p graf-after--li" id="f713" name="f713" style="background-color: white; color: rgba(0, 0, 0, 0.8); letter-spacing: -0.003em; line-height: 1.58; margin-top: 29px;">
In case these two strategies don’t work you might have run into a cloud side error.</div>
<h3 style="background-color: white; color: rgba(0 , 0 , 0 , 0.8); font-family: , "lucida grande" , "lucida sans unicode" , "lucida sans" , "geneva" , "arial" , sans-serif; font-size: 24px; letter-spacing: -0.018em; line-height: 1.22; margin: 30px 0px 0px -1.5px;">
Cloud side error</h3>
<div class="graf graf--p graf-after--h4" id="695a" name="695a" style="background-color: white; color: rgba(0, 0, 0, 0.8); letter-spacing: -0.003em; line-height: 1.58; margin-top: 6px;">
There exist two types of cloud side errors, the glitches and the systematic errors. A cloud side error will always start as a glitch because there always will need to be a first time that something bad happens. Make sure that you store the information of this error and try to memorize the error. If you don’t see the error happening again, then it was a glitch.</div>
<div class="graf graf--p graf-after--p" id="9499" name="9499" style="background-color: white; color: rgba(0, 0, 0, 0.8); letter-spacing: -0.003em; line-height: 1.58; margin-top: 29px;">
However, when you see the same error happening more than once you might have run into a systematic error and it is time to call the Azure ML support team for help. Make sure in this case you provide all your local logs and your client side logs which will enable you to nail down the issue in the fastest possible way.</div>
<h3 class="graf graf--h3 graf-after--p" id="3c47" name="3c47" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-sans-serif-font, "Lucida Grande", "Lucida Sans Unicode", "Lucida Sans", Geneva, Arial, sans-serif; font-size: 32px; letter-spacing: -0.02em; line-height: 1.15; margin: 56px 0px 0px -2px;">
Conclusion</h3>
<div class="graf graf--p graf-after--h3 graf--last" id="d459" name="d459" style="background-color: white; color: rgba(0, 0, 0, 0.8); letter-spacing: -0.003em; line-height: 1.58; margin-top: 8px;">
I hope with these tips you will be able to discover new pieces of logging information of your cloud architecture. Hopefully, you will never need them, but that they will help you out in the unfortunate case that you run into a crash of your cloud application.</div>
Anonymoushttp://www.blogger.com/profile/00885379426770757526noreply@blogger.com5tag:blogger.com,1999:blog-7939626009906644084.post-66277936237008071432016-09-20T16:39:00.000-07:002016-11-12T18:53:42.669-08:00Four reasons not to go to a technical Meetup<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-g_YT5HPzeo0/WCe2ImLPq8I/AAAAAAAAABk/aec7IDN4k80LtWMUVa-zEwH8TwqpaAfDgCLcB/s1600/labirinth.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="300" src="https://1.bp.blogspot.com/-g_YT5HPzeo0/WCe2ImLPq8I/AAAAAAAAABk/aec7IDN4k80LtWMUVa-zEwH8TwqpaAfDgCLcB/s400/labirinth.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-size: small;">Don’t let practical obstacles prevent you from going to a technical Meetup. </span><a class="markup--anchor markup--figure-anchor" data-href="https://commons.wikimedia.org/wiki/File:Labirinth_schonbrunn.JPG" href="https://commons.wikimedia.org/wiki/File:Labirinth_schonbrunn.JPG" rel="nofollow" style="background-color: white; background-image: linear-gradient(rgb(255, 255, 255) 50%, rgba(0, 0, 0, 0.439216) 50%); background-position: 0px 0.98em; background-repeat: repeat-x; background-size: 2px 2px; text-decoration: none;" target="_blank"><span style="font-size: xx-small;">https://commons.wikimedia.org/wiki/File:Labirinth_schonbrunn.JPG</span></a></td></tr>
</tbody></table>
<div>
<div class="graf graf--p" name="79f6">
Professional development is something that we all need to spend a lot of time on if we want to keep up with the advances in technology and try to advance in our career. Most of the time this is a pretty individual and lonely task. Did you know there are many technical groups that meet up on a regular basis? This could be a great way to learn and also network with others.</div>
<div class="graf graf--p" name="79f6">
<br /></div>
<div class="graf graf--p" name="f1c5">
<a class="markup--anchor markup--p-anchor" data-href="http://www.meetup.com" href="http://www.meetup.com/" target="_blank">Meetup</a> is an easy and straight forward place to start looking. Make an account on Meetup, add in your interests, go to some interesting meetings and thanks to the machine learning algorithm, a lot of interesting suggestions will come to your inbox every week.</div>
<div class="graf graf--p" name="f1c5">
<br /></div>
<div class="graf graf--p" name="05ef">
Fear, time commitments, not exactly knowing what is available and whether it will interest you are all barriers that might be holding you back. I will share four fears that might be holding you back from going to a meetup and will explain how to overcome these fears.</div>
<div class="graf graf--p" name="05ef">
<br /></div>
<h3 class="graf graf--h3" name="b949">
But … I don’t know anyone!</h3>
<div class="graf graf--p" name="f7d5">
For everybody there has always been a first time going to a Meetup and changes are high that you won’t be the only new person going. You can also just ask whether a friend can join you.</div>
<div class="graf graf--p" name="24bb">
<br /></div>
<div class="graf graf--p" name="24bb">
If you are not fond of hours of networking, you can pick a Meetup with a lot of talks in it. In case you really fear talking with a lot of unknown people, look at the agenda and arrive just a few minutes before the talk is scheduled to start. In case there is no real agenda available for the meeting, talks normally start 10 minutes after the start of the meeting and in case the talk is started when you arrive just sneak in at the back. If you enjoyed the talk and would consider going a second time, there will be certainly already a lot of familiar faces.</div>
<div class="graf graf--p" name="24bb">
<br /></div>
<h3 class="graf graf--h3" name="d8b3">
But … I don’t know enough about the topic!</h3>
<div class="graf graf--p" name="52c5">
On the internet there is probably months worth of reading material available on your favorite topic. Every week a few extra months worth of reading material will be added and there are a lot of books in the library and e-books for your tablet available too. I even forgot to mention all the online courses that you can take as well as all the podcasts that are available. If you are studying alone and try to read everything available, you don’t really have the feeling that you are progressing, because indeed there will be always that extra piece of information that you haven’t consumed yet. Do you think the people at the meeting have really read every piece of available information? Nope!</div>
<div class="graf graf--p" name="388d">
<br /></div>
<div class="graf graf--p" name="388d">
Therefore, it is also important to talk with people who share interests in a similar topic. In this way you can share ideas about what you have read and compare your knowledge levels. You can determine what you are good at and which parts you still need to improve on. You also can get hints about material that you should explore further.</div>
<div class="graf graf--p" name="a036">
<br /></div>
<div class="graf graf--p" name="a036">
Listening to a presentation or watching a demo, will allow you to remember more about the topic. You also have the possibility of immediate interaction if something is not really clear.</div>
<div class="graf graf--p" name="7edb">
<br /></div>
<div class="graf graf--p" name="7edb">
You will also learn about various speaking styles. These you can try out at your job when you are discussing your own work. In case the meeting is not in your mother tongue, it will also allow you to learn these terms in your new language. It will also help you to learn the correct pronunciation of the terminology.</div>
<div class="graf graf--p" name="7edb">
<br /></div>
<h3 class="graf graf--h3" name="dfe9">
But … I won’t find the location of the Meetup!</h3>
<div class="graf graf--p" name="803b">
It might indeed be a challenge to find the location of the Meetup. There might be construction going on. The signs to the correct location might be missing. But please don’t let this be a reason for not going to a Meetup, because you already have gotten this far. If you have a hard time finding the location of the Meetup, you are probably not the only one. Just briefly mention to the organizers when you eventually arrive that the place is not that easy to find. In this way they can make sure that they give better directions for other people still coming and they can make accommodations for future meetings. Also make sure that you have your smartphone charged with you in case you get lost, Google Maps is your friend.</div>
<div class="graf graf--p" name="803b">
<br /></div>
<h3 class="graf graf--h3" name="20ea">
But … I will be the only woman there!</h3>
<div class="graf graf--p" name="7a9f">
This is a reason that might be holding back a lot of women unfortunately and will eventually become a self fulfilling prophecy. You actually might be surprised about how many technical meetups have at least one woman in their organizing team. In these Meetups you never need to feel alone as a woman. Also, by going to such a Meetup, you are supporting them.</div>
<div class="graf graf--p" name="4ec1">
<br /></div>
<div class="graf graf--p" name="4ec1">
But also don’t be scared if you are going to a Meetup where you might be the only woman, the men will be really happy that you are going and will be really friendly and welcoming. Imagine the power that you have in this case, just by you attending, you enable gender diversity in the group.</div>
<div class="graf graf--p" name="862e">
<br /></div>
<div class="graf graf--p" name="862e">
Another approach might be to send a message to one of the women in tech initiatives in your area and ask whether there is anyone who wants to join you to a certain meetup. In this case you can convince other women to join who were struggling with the same objection.</div>
<div class="graf graf--p" name="617e">
<br /></div>
<div class="graf graf--p" name="617e">
…</div>
<div class="graf graf--p" name="70d3">
<br /></div>
<div class="graf graf--p" name="70d3">
Fast forward a few months …</div>
<div class="graf graf--p" name="70d3">
<br /></div>
<h3 class="graf graf--h3" name="9c1c">
I am convinced, I went, loved the meetup, can I give a talk myself?</h3>
<div class="graf graf--p" name="3635">
Congratulations! You certainly can give a talk at your favorite Meetup. Let the organizers know and they will give you a time slot. Good luck with your talk.</div>
</div>
<div>
<h3 class="graf graf--h3 graf-after--p" id="b949" name="b949" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-sans-serif-font, "Lucida Grande", "Lucida Sans Unicode", "Lucida Sans", Geneva, Arial, sans-serif; font-size: 32px; letter-spacing: -0.02em; line-height: 1.15; margin: 56px 0px 0px -2px;">
<br /></h3>
<h3 class="graf graf--h3 graf-after--p" id="d8b3" name="d8b3" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-sans-serif-font, "Lucida Grande", "Lucida Sans Unicode", "Lucida Sans", Geneva, Arial, sans-serif; font-size: 32px; letter-spacing: -0.02em; line-height: 1.15; margin: 56px 0px 0px -2px;">
<br /></h3>
</div>
Anonymoushttp://www.blogger.com/profile/00885379426770757526noreply@blogger.com3tag:blogger.com,1999:blog-7939626009906644084.post-14682459358216108262016-01-30T16:34:00.000-08:002016-11-12T16:51:03.150-08:00<h1 class="graf graf--h3 graf--leading graf--title" id="bfbe" name="bfbe" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-sans-serif-font, "Lucida Grande", "Lucida Sans Unicode", "Lucida Sans", Geneva, Arial, sans-serif; font-size: 40px; letter-spacing: -0.028em; line-height: 1.04; margin: 0px 0px 0px -2.5px;">
How to (re)light the mathematical flame in you.</h1>
<div>
Originally posted on linkedin.com </div>
<div>
<br /></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-mVBiotbsZSQ/WCe06KgtcxI/AAAAAAAAABg/FBCxJ-iEw-UEljfVBLpspGGx5tCduDktgCLcB/s1600/graph_blog.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://1.bp.blogspot.com/-mVBiotbsZSQ/WCe06KgtcxI/AAAAAAAAABg/FBCxJ-iEw-UEljfVBLpspGGx5tCduDktgCLcB/s640/graph_blog.png" width="640" /></a></div>
<br /></div>
<div>
<div class="graf graf--p graf-after--figure" id="7a92" name="7a92" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-serif-font, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 21px; letter-spacing: -0.003em; line-height: 1.58; margin-top: 38px;">
Nowadays there is a massive growth of data which is leading to mathematical modeling appearing in a growing number of industries. This is the perfect time to become curious about all those things you learnt in your mathematics classes and courses. To help you with this, I’m sharing some fun mathematical resources to explore.</div>
<h4 class="graf graf--h4 graf-after--p" id="cdff" name="cdff" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-sans-serif-font, "Lucida Grande", "Lucida Sans Unicode", "Lucida Sans", Geneva, Arial, sans-serif; font-size: 24px; letter-spacing: -0.018em; line-height: 1.22; margin: 30px 0px 0px -1.5px;">
Study of some surprising social networks</h4>
<div class="graf graf--p graf-after--h4" id="68e5" name="68e5" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-serif-font, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 21px; letter-spacing: -0.003em; line-height: 1.58; margin-top: 6px;">
Everyone knows the standard social networks LinkedIn and Facebook. A lot of fancy mathematics is done behind the scenes to help you to find people you may know, to check how you are ranked between your contacts and<br />
your company.</div>
<div class="graf graf--p graf-after--p" id="b66f" name="b66f" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-serif-font, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 21px; letter-spacing: -0.003em; line-height: 1.58; margin-top: 29px;">
This type of social network analysis can also be applied to some surprising situations, for example: the <a class="markup--anchor markup--p-anchor" data-href="http://arxiv.org/pdf/cond-mat/0202174v1.pdf" href="http://arxiv.org/pdf/cond-mat/0202174v1.pdf" rel="nofollow" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0.439216); background-color: transparent; background-image: linear-gradient(rgba(0, 0, 0, 0.6) 50%, rgba(0, 0, 0, 0) 50%); background-position: 0px 1.07em; background-repeat: repeat-x; background-size: 2px 0.1em; text-decoration: none;" target="_blank">Marvel Superhero network</a>, the <a class="markup--anchor markup--p-anchor" data-href="http://arxiv.org/pdf/cond-mat/0001458.pdf" href="http://arxiv.org/pdf/cond-mat/0001458.pdf" rel="nofollow" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0.439216); background-color: transparent; background-image: linear-gradient(rgba(0, 0, 0, 0.6) 50%, rgba(0, 0, 0, 0) 50%); background-position: 0px 1.07em; background-repeat: repeat-x; background-size: 2px 0.1em; text-decoration: none;" target="_blank">Movie Actors Network</a> and a <a class="markup--anchor markup--p-anchor" data-href="http://evelinag.com/blog/2015/12-15-star-wars-social-network/" href="http://evelinag.com/blog/2015/12-15-star-wars-social-network/" rel="nofollow" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0.439216); background-color: transparent; background-image: linear-gradient(rgba(0, 0, 0, 0.6) 50%, rgba(0, 0, 0, 0) 50%); background-position: 0px 1.07em; background-repeat: repeat-x; background-size: 2px 0.1em; text-decoration: none;" target="_blank">social network from the characters in Star Wars</a>. For all of these networks the heroes, the actors or the characters will represent the nodes of the graphs. Connecting the various nodes is different for each of these networks. For the superhero network, there will be a line between two characters when they occurred significantly in the same comic book. For the actors if they played in the same movie. Finally, for the Star Wars characters if they spoke to each other in a movie.</div>
<div class="graf graf--p graf-after--p" id="b11a" name="b11a" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-serif-font, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 21px; letter-spacing: -0.003em; line-height: 1.58; margin-top: 29px;">
In the links that I mentioned you can see the colorful visualizations of the derived networks and some insightful mathematical analysis. If you want to play with the super hero network, <a class="markup--anchor markup--p-anchor" data-href="https://www.udemy.com/taming-big-data-with-apache-spark-hands-on/learn/#/" href="https://www.udemy.com/taming-big-data-with-apache-spark-hands-on/learn/#/" rel="nofollow" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0.439216); background-color: transparent; background-image: linear-gradient(rgba(0, 0, 0, 0.6) 50%, rgba(0, 0, 0, 0) 50%); background-position: 0px 1.07em; background-repeat: repeat-x; background-size: 2px 0.1em; text-decoration: none;" target="_blank">this</a> is a useful resource. The latter will also<br />
help you to gain some hands-on experience with <a class="markup--anchor markup--p-anchor" data-href="http://spark.apache.org/" href="http://spark.apache.org/" rel="nofollow" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0.439216); background-color: transparent; background-image: linear-gradient(rgba(0, 0, 0, 0.6) 50%, rgba(0, 0, 0, 0) 50%); background-position: 0px 1.07em; background-repeat: repeat-x; background-size: 2px 0.1em; text-decoration: none;" target="_blank">Spark</a>. Spark is an engine for processing very large sets of data. It has a machine learning library, this means a lot of mathematical programs that can be used to find patterns in large sets of data. If you want to make beautiful graph visualizations, you can use <a class="markup--anchor markup--p-anchor" data-href="https://gephi.org/" href="https://gephi.org/" rel="nofollow" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0.439216); background-color: transparent; background-image: linear-gradient(rgba(0, 0, 0, 0.6) 50%, rgba(0, 0, 0, 0) 50%); background-position: 0px 1.07em; background-repeat: repeat-x; background-size: 2px 0.1em; text-decoration: none;" target="_blank">gephi</a>. Gephi is an open-source software for visualizing and analyzing large networks graphs.</div>
<h4 class="graf graf--h4 graf-after--p" id="73f4" name="73f4" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-sans-serif-font, "Lucida Grande", "Lucida Sans Unicode", "Lucida Sans", Geneva, Arial, sans-serif; font-size: 24px; letter-spacing: -0.018em; line-height: 1.22; margin: 30px 0px 0px -1.5px;">
Recreational Mathematics</h4>
<div class="graf graf--p graf-after--h4" id="b5b4" name="b5b4" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-serif-font, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 21px; letter-spacing: -0.003em; line-height: 1.58; margin-top: 6px;">
Do you just want to play with mathematics for fun or want to dust off your math skills? <a class="markup--anchor markup--p-anchor" data-href="https://en.wikipedia.org/wiki/Recreational_mathematics" href="https://en.wikipedia.org/wiki/Recreational_mathematics" rel="nofollow" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0.439216); background-color: transparent; background-image: linear-gradient(rgba(0, 0, 0, 0.6) 50%, rgba(0, 0, 0, 0) 50%); background-position: 0px 1.07em; background-repeat: repeat-x; background-size: 2px 0.1em; text-decoration: none;" target="_blank">Recreational mathematics</a> might be something for you and you can pick from various flavors: <a class="markup--anchor markup--p-anchor" data-href="https://en.wikipedia.org/wiki/Mathematical_game" href="https://en.wikipedia.org/wiki/Mathematical_game" rel="nofollow" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0.439216); background-color: transparent; background-image: linear-gradient(rgba(0, 0, 0, 0.6) 50%, rgba(0, 0, 0, 0) 50%); background-position: 0px 1.07em; background-repeat: repeat-x; background-size: 2px 0.1em; text-decoration: none;" target="_blank">mathematical games</a>,<br />
<a class="markup--anchor markup--p-anchor" data-href="https://en.wikipedia.org/wiki/Mathematical_puzzle" href="https://en.wikipedia.org/wiki/Mathematical_puzzle" rel="nofollow" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0.439216); background-color: transparent; background-image: linear-gradient(rgba(0, 0, 0, 0.6) 50%, rgba(0, 0, 0, 0) 50%); background-position: 0px 1.07em; background-repeat: repeat-x; background-size: 2px 0.1em; text-decoration: none;" target="_blank">mathematical puzzles</a>, <a class="markup--anchor markup--p-anchor" data-href="https://en.wikipedia.org/wiki/Tessellation" href="https://en.wikipedia.org/wiki/Tessellation" rel="nofollow" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0.439216); background-color: transparent; background-image: linear-gradient(rgba(0, 0, 0, 0.6) 50%, rgba(0, 0, 0, 0) 50%); background-position: 0px 1.07em; background-repeat: repeat-x; background-size: 2px 0.1em; text-decoration: none;" target="_blank">tessellations</a>, <a class="markup--anchor markup--p-anchor" data-href="https://en.wikipedia.org/wiki/Fractal" href="https://en.wikipedia.org/wiki/Fractal" rel="nofollow" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0.439216); background-color: transparent; background-image: linear-gradient(rgba(0, 0, 0, 0.6) 50%, rgba(0, 0, 0, 0) 50%); background-position: 0px 1.07em; background-repeat: repeat-x; background-size: 2px 0.1em; text-decoration: none;" target="_blank">fractals</a>, <a class="markup--anchor markup--p-anchor" data-href="http://rubiks.com/blog/how-to-solve-the-rubiks-cube" href="http://rubiks.com/blog/how-to-solve-the-rubiks-cube" rel="nofollow" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0.439216); background-color: transparent; background-image: linear-gradient(rgba(0, 0, 0, 0.6) 50%, rgba(0, 0, 0, 0) 50%); background-position: 0px 1.07em; background-repeat: repeat-x; background-size: 2px 0.1em; text-decoration: none;" target="_blank">Rubik’s cube solving</a>, ….</div>
<div class="graf graf--p graf-after--p" id="4418" name="4418" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-serif-font, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 21px; letter-spacing: -0.003em; line-height: 1.58; margin-top: 29px;">
Various people write blogs about their fun with recreational mathematics. Here are some examples a <a class="markup--anchor markup--p-anchor" data-href="http://andrewtweddle.blogspot.ca/" href="http://andrewtweddle.blogspot.ca/" rel="nofollow" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0.439216); background-color: transparent; background-image: linear-gradient(rgba(0, 0, 0, 0.6) 50%, rgba(0, 0, 0, 0) 50%); background-position: 0px 1.07em; background-repeat: repeat-x; background-size: 2px 0.1em; text-decoration: none;" target="_blank">blog of a friend</a>, <a class="markup--anchor markup--p-anchor" data-href="http://www.amotlpaa.org/math/" href="http://www.amotlpaa.org/math/" rel="nofollow" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0.439216); background-color: transparent; background-image: linear-gradient(rgba(0, 0, 0, 0.6) 50%, rgba(0, 0, 0, 0) 50%); background-position: 0px 1.07em; background-repeat: repeat-x; background-size: 2px 0.1em; text-decoration: none;" target="_blank">another friend</a> and a <a class="markup--anchor markup--p-anchor" data-href="http://blogs.ams.org/blogonmathblogs/category/recreational-mathematics/#sthash.hcAFQHLB.J8Op7m2g.dpbs" href="http://blogs.ams.org/blogonmathblogs/category/recreational-mathematics/#sthash.hcAFQHLB.J8Op7m2g.dpbs" rel="nofollow" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0.439216); background-color: transparent; background-image: linear-gradient(rgba(0, 0, 0, 0.6) 50%, rgba(0, 0, 0, 0) 50%); background-position: 0px 1.07em; background-repeat: repeat-x; background-size: 2px 0.1em; text-decoration: none;" target="_blank">blog on math blogs</a>. Maybe you’ll even be inspired to write your own recreational math blog.</div>
<h4 class="graf graf--h4 graf-after--p" id="cbf1" name="cbf1" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-sans-serif-font, "Lucida Grande", "Lucida Sans Unicode", "Lucida Sans", Geneva, Arial, sans-serif; font-size: 24px; letter-spacing: -0.018em; line-height: 1.22; margin: 30px 0px 0px -1.5px;">
Mathematics clubs</h4>
<div class="graf graf--p graf-after--h4" id="b96f" name="b96f" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-serif-font, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 21px; letter-spacing: -0.003em; line-height: 1.58; margin-top: 6px;">
Do you want to discover new fields of mathematics and grow your professional network? You might have been part of a mathematics club when you were at university. Similar versions of those clubs still exist for math lovers in the working world.</div>
<div class="graf graf--p graf-after--p" id="2571" name="2571" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-serif-font, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 21px; letter-spacing: -0.003em; line-height: 1.58; margin-top: 29px;">
The <a class="markup--anchor markup--p-anchor" data-href="http://www.meetup.com/Intersections-KW/" href="http://www.meetup.com/Intersections-KW/" rel="nofollow" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0.439216); background-color: transparent; background-image: linear-gradient(rgba(0, 0, 0, 0.6) 50%, rgba(0, 0, 0, 0) 50%); background-position: 0px 1.07em; background-repeat: repeat-x; background-size: 2px 0.1em; text-decoration: none;" target="_blank">Meetup group KW Intersections</a> is an example of such a group. They talk about various topics in research in mathematics and computer science. <a class="markup--anchor markup--p-anchor" data-href="http://www.amotlpaa.org/kcapw/" href="http://www.amotlpaa.org/kcapw/" rel="nofollow" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0.439216); background-color: transparent; background-image: linear-gradient(rgba(0, 0, 0, 0.6) 50%, rgba(0, 0, 0, 0) 50%); background-position: 0px 1.07em; background-repeat: repeat-x; background-size: 2px 0.1em; text-decoration: none;" target="_blank">Some topics</a> of the last two years are a geometrical proof of π, quantum computing, neural networks, and type theory. The Meetup offers so many different colours and flavours of mathematics.</div>
<div class="graf graf--p graf-after--p" id="9004" name="9004" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-serif-font, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 21px; letter-spacing: -0.003em; line-height: 1.58; margin-top: 29px;">
It is amazing about how much energy comes out of such a group. Hearing someone giving a talk about their passion, makes you look again into your own projects, or let’s you dive deeper into a paper so that you can explain the paper yourself in front of a group. You don’t have time to join this Meetup? No worries, one of hour members films a lot of the talks and puts them on <a class="markup--anchor markup--p-anchor" data-href="https://www.youtube.com/results?search_query=kw+intersections+kw+intersections" href="https://www.youtube.com/results?search_query=kw+intersections+kw+intersections" rel="nofollow" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0.439216); background-color: transparent; background-image: linear-gradient(rgba(0, 0, 0, 0.6) 50%, rgba(0, 0, 0, 0) 50%); background-position: 0px 1.07em; background-repeat: repeat-x; background-size: 2px 0.1em; text-decoration: none;" target="_blank">his channel</a>.</div>
<h4 class="graf graf--h4 graf-after--p" id="e6ac" name="e6ac" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-sans-serif-font, "Lucida Grande", "Lucida Sans Unicode", "Lucida Sans", Geneva, Arial, sans-serif; font-size: 24px; letter-spacing: -0.018em; line-height: 1.22; margin: 30px 0px 0px -1.5px;">
Conclusion</h4>
<div class="graf graf--p graf-after--h4" id="caea" name="caea" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-serif-font, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 21px; letter-spacing: -0.003em; line-height: 1.58; margin-top: 6px;">
I hope you found these resources fun and interesting and that you have learnt the many places math can pop up — and when that happens it can be fun! Not all mathematics have to be useful, sometimes it can just be fun.</div>
<div class="graf graf--p graf-after--p graf--last" id="1983" name="1983" style="background-color: white; color: rgba(0, 0, 0, 0.8); font-family: medium-content-serif-font, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 21px; letter-spacing: -0.003em; line-height: 1.58; margin-top: 29px;">
Do you have any fun mathematical resources to share? I’d love to hear all about them. Just add a response below this article.</div>
</div>
Anonymoushttp://www.blogger.com/profile/00885379426770757526noreply@blogger.com3