Monday, 6 March 2017

How to build an architecture for your cloud data application

Introduction
But how do you actually start with building data applications in the cloud? Which known technologies can you use? What are new technologies? How do all these technologies work together? What will the cost be of such a solution? 
We will try to answer these questions in this series of blog posts. We will show how two people, Dana and Frank (see picture below) will start building the next Unicorn using cloud based data technology. We will first present their product, next introduce Dana and Frank and finally explain how they started building their architecture.
The next blogs in the series can be found here

The Usbourne first book of the Computer, Usbourn Publishing Ltd, 1985


The game changing idea
Together they are building a SaaS company that makes online games with data science. Their first game that they are working on is called What's the title?. They show automatically generated data graphs of classic novels. The user needs to guess the title. A real game changer. Below you can see their first game and you can also check out the link to their website.



Meet the team
For now, the team consists of only two people, but they hope to grow fast in the near future. Therefore they both agreed that they will need to acquire extra skill sets besides their current skills to make the full application. Their focus initially is to get all the components connected to have a global overview and see how the different pieces are working together. Later on they can then elaborate further on the different components and move on to using more advanced technologies.

Dana Simpsons, the Data Scientist

Dana loves everything about mathematics and data and she has a decent background in computer science. Her favourite programming languages are R and Python. Finally she also has some SQL and NoSQL database skills. As part of her tasks she will also watch the cloud costs and look for potential savings. She will also be responsible for customer success and will analyze the usage patterns.

Frank Steve Davidson, the Full Stack Developer

Frank loves everything about programming and has a decent background in mathematics. His favorite programming languages are Java and Ruby. He will be responsible for the architecture design and will make sure that the solution gets deployed in the cloud.

The generic architecture
Dana and Frank did some brainstorming meetings and came up with the following architecture. With this architecture they want to achieve the three goals explained below.




Goal 1

Their game will be hosted in the cloud. This means that both their development and their production servers will live in the cloud together with the data storage. This means that they will need to look at how virtual machines are being set up by their chosen cloud provider and what kind of cloud storage solutions there are available. Because they are hosting a website, they also still will need to investigate this further.

Goal 2
Frank has chosen to work with Ruby as programming language to build out the back-end further. Some responsibilities of the back-end will be the storing of different novels in the cloud in a data lake and transforming these novels into a data format consumable by the web services that Dana is building. He will extract the images that Dana generates and will store them back in the cloud. For now they only will use a simple front-end because the data solution will already provide detailed images. In the future they will look further into more advanced front-end technology.

Goal 3
Dana will need data science related software to be able to make her analysis and generate the graphs. She also will need to have an easy way to access different data sources when she is performing her analysis. She prefers to have access to python and R. Finally she wants to have an easy way to expose her graphs and analysis to Frank. When she is happy with her data science solutions she wants to build an automation layer on top of this.

The architecture in Azure
Dana and Frank compared the different cloud providers and they came to the conclusion that Azure suited their needs best. The architecture in Azure is shown below. In what follows we will explain these components more. In the next blogs you will learn how to setup and use these components.




Goal 1: How to deploy and develop



Frank uses a Linux Virtual Machine in Azure on which he has installed Ruby. He next uses a virtual machine to host the web server Apache2.  To save costs he will put his Ruby development machine on auto-shut down mode. To be able to host this virtual machine as a web server he will need to select a public ip address and allow http access on the server. We go into more detail in these steps in the next blog post of this series.

Goal 2: How to talk to and feed the data storage solution



Frank and Dana use blob storage as their cloud storage solution. You can imagine Blob storage as your own big drive in the cloud. When you are developing you can use several APIs to access blob storage. Frank will use the azure gem for Ruby. Dana will analyze the costs that are involved with using blob storage. We go into more detail in this in the third blog post of this series.


Goal 3: How to talk with and build the Data Science Solution




Dana used Azure Machine Learning Studio to build an experiment that generates the needed graphs. Next she deployed this experiment as a web service. This web service consumes the formatted text of a classic novel that is stored in Blob storage and then produces six different graphs. These graphs are then stored in Blob storage. She will describe her strategy in the fourth blog post. Next Frank will describe in the last blog post how he feeds and consumes the data from this web service.

Conclusion 
In this blog post we have introduced two people, Dana and Frank, on their journey to build their SaaS company in the cloud. We briefly mentioned some of the technologies that they needed. In the next blog posts we will explain these technologies more in depth so that you will be able to use them for your own application.


67 comments:

  1. For the finding of best data solution person go to this web-site activewizards.com/ , from here you can be able to find and compare the huge number of data handling expert who can easily handle or solve your big data handling problems in a short time.

    ReplyDelete
  2. Superb. I really enjoyed very much with this article here. Really it is an amazing article I had ever read. I hope it will help a lot for all.
    Thank you so much for this amazing posts and please keep update like this excellent article.
    thank you for sharing such a great blog with us. expecting for your.

    python Training in Bangalore | Python Training institute in Bangalore

    ReplyDelete
  3. This article is fantastic! Noise in charts and data visualization is such a problem, and this gave me some ideas for how to better approach it, such as greying out the less-important categories in a graph. The slopegraph was still hard for me to understand, ever re-worked.


    Java training in Chennai | Java training institute in Chennai | Java course in Chennai

    ReplyDelete
  4. This article is fantastic! Noise in charts and data visualization is such a problem, and this gave me some ideas for how to better approach it, such as greying out the less-important categories in a graph. The slopegraph was still hard for me to understand, ever re-worked.


    Java training in Chennai | Java training institute in Chennai | Java course in Chennai

    ReplyDelete
  5. I am often to blogging and i really appreciate your content. The article has really peaks my interest. I am going to bookmark your site and keep checking for new information. the impossible quiz unblocked

    ReplyDelete
  6. My friend got hooked on this site and got me hooked modern all online casinos There are great slot machines and various gambling games. You can chat and earn extra money

    ReplyDelete
  7. I'm really impressed about the info you provide in your articles. If you want to invest in suits and be stress-free, then you should shake hands with us. We are providing great clothing in the most reasonable price range that you wouldn’t find anywhere else. Put your trust in us as your supplier since we are the prominent custom stickers printing You can rely on us for the clothing problems we assure you we will never let our customers down.

    ReplyDelete
  8. I'm really impressed about the info you provide in your articles. If you want to invest in suits and be stress-free, then you should shake hands with us. custom window cake boxes

    ReplyDelete
  9. Attractive information on your blog, thank you for taking the time and share with us.
    custom toy boxes

    ReplyDelete
  10. Hello, I want to subscribe for this web site to take most up-to-date updates, thus where can i do it please help out.
    custom lipstick boxes

    ReplyDelete
  11. Nice post, keep up with this interesting work. It really is good to know that this topic is being covered also on this web site so cheers for taking time to discuss this! Healthy lunch Ideas

    ReplyDelete
  12. On this page, you'll see my profile, please read this information. buy medical cannabis online uk

    ReplyDelete
  13. For e.g., the correct inquiry to pose for a car organization could be what is the advertising spend to income proportion in various postal districts for various models. This will lead them to adjust their showcasing dollars stream.machine learning course

    ReplyDelete
  14. Not a single second surrendered relinquishing me in total stun.
    Microsoft project alternative

    ReplyDelete
  15. Thank you for sharing superb information. Your blog is very cool. I am impressed by your blog and detail that you have on this site. Thank you…
    Custom Printed Boxes
    Custom Boxes with Logo
    Custom CBD Boxes

    ReplyDelete
  16. Crypto-currency as a modern form of the digital asset has received a worldwide acclaim for easy and faster financial transactions and its awareness among people have allowed them to take more interest in the field thus opening up new and advanced ways of making payments. Crypto.com Referral Code the growing demand of this global phenomenon more,new traders and business owners are now willing to invest in this currency platform despite its fluctuating prices however it is quite difficult to choose the best one when the market is full. In the list of crypto-currencies bit-coins is one of the oldest and more popular Crypto.com Referral Code for the last few years. It is basically used for trading goods and services and has become the part of the so-called computerized block-chain system allowing anyone to use it thus increasing the craze among the public, Crypto.com Referral Code.

    Common people who are willing to purchase BTC can use an online wallet system for buying them safely in exchange of cash or credit cards and in a comfortable way from the thousands of BTC foundations around the world and keep them as assets for the future. Due to its popularity, many corporate investors are now accepting them as cross-border payments and the rise is unstoppable. With the advent of the internet and mobile devices,information gathering has become quite easy as a result the BTC financial transactions are accessible and its price is set in accordance with people’s choice and preferences thus leading to a profitable investment with Crypto.com Referral Code. Recent surveys have also proved that instability is good for BTC exchange as if there is instability and political unrest in the country due to which banks suffer then investing in BTC can surely be a better option. Again bit-coin transaction fees are pretty cheaper and a more convenient technology for making contracts thus attracting the crowd. The BTC can also be converted into different fiat currencies and is used for trading of securities, for land titles, document stamping, public rewards and vice versa.

    Another advanced block-chain project is Ethereumor the ETH which has served much more than just a digital form of crypto-currency Crypto.com Referral Code and its popularity in the last few decades have allowed billions of people to hold wallets for them. With the ease of the online world,the ETH have allowed the retailers and business organizations to accept them for trading purposes, therefore, can serve as the future of the financial system.

    ReplyDelete
  17. I would like to thank you for getting my neurons conspicuous with this brilliant article that you have written which contains every potential points which needs to considered on the given topic. Thanks for chipping in such a brilliant writing!
    Data Science training in Mumbai
    Data Science course in Mumbai
    SAP training in Mumbai

    ReplyDelete
  18. This blog is amazing keep sharing quality custom boxes
    very fast and great service Customized Packing Box Company
    best birthdays wishes i like it new birthday wishes

    ReplyDelete
  19. The information given in this article is very good and I like it. I have also written this kind of blog you can also read for more knowledge.
    how to improve google
    search results

    how to improve seo ranking on google
    artificial intelligence latest developments

    ReplyDelete
  20. I would like to thank you for getting my neurons conspicuous with this brilliant article that you have written which contains every potential points which needs to considered on the given topic. Thanks for chipping in such a brilliant writing!
    Custom Packaging Boxes

    ReplyDelete
  21. Health Experts have proven that regular exercise coupled with a good diet allow you to live longer and healthier. In this busy day and age, not everyone has the time to go to the gym - resulting to a lot of overweight people that desperately need to exercise. A healthy alternative is for you to Buy Home Gym Equipments that you can store in your own home or even at your office. Here are some tips when buying home gym equipment.

    First, know your fitness goals and keep these goals in mind when you are buying home gym equipment. One of the biggest mistakes that people make is buying the biggest or trendiest fitness machine simply because they like how it looks. More often than not, these end up gathering dust in your storage rooms or garage because you never end up using them. It is important to take note of what particular type of physical activity you want or enjoy doing before you buy your exercise machine. If you are looking to loose a few pounds and you enjoy walking or hiking, a treadmill is the best option for you. If you are looking to tone your lower body while burning calories a stationary bike is your obvious choice. Similarly, Special Equipments for Core Strength Exercises, Strength Training Weight Vests, Core Strength & Abdominal Trainers, Home Cardio Training, Strength Training Power Cages, Strength Training Racks & More.

    Second, set aside a budget before Buying Home Gym Equipments. Quality exercise machines do not come cheap. They are constantly exposed to wear and tear when they are used properly. So, pick machines that are built to last and have passed quality certifications to get the most out of your money. If you are operating on a tight budget, think about investing in several weights, We can Provide you High Quality Home Gym Equipments at Very Low Prices for your Complete Uses: Core Strength Exercises, Strength Training Weight Vests, Core Strength & Abdominal Trainers, Home Cardio Training, Strength Training Power Cages, Strength Training Racks & More.

    Its the Right Time to Buy Home Gym Equipments for you at Very Low Prices.

    ReplyDelete
  22. Archie 420 Dispensary is a trusted Cannabis dispensary base in Los Angeles California USA. It is one of the top dispensary in this part of the country. They do deliver Marijuana in the USA and to over 25 countries in the world. Buy Purple haze online You can always visit their dispensary in Los Angeles using the address on their website. Place your order and get served by the best dispensary in the planet. Have fun.

    ReplyDelete
  23. Crystal online pharmacy is a trusted online drug store with a wide range of products to suit the needs of our clients. Buy Fish oil online Crystal Pharmacy do strive to offer the best service and ship products world wide. All the products listed on our website are Ava in stock. Expect your order to be processed Immediately when you send us your request. We deal with varieties of drugs for our customers satisfaction. We cross barriers with our products and struggle hard to meet human satisfaction. When shopping with us, Be safe and secured and you will realize how swift we are with our services.

    ReplyDelete
  24. Universal Gun sales is a trusted Firearm company base in Los Angeles California USA. It is one of the top Firearms Company in this part of the country. Buy Big sky racks UTV Bar online They do offer the best firearms deal in the USA and to over 25 countries in the world. You can always visit their shop in Los Angeles using the address on their website. Place your order and get served by the best Firearm Company in the planet. Have fun.

    ReplyDelete
  25. Health Experts have proven that regular exercise coupled with a good diet allow you to live longer and healthier. In this busy day and age, not everyone has the time to go to the gym - resulting to a lot of overweight people that desperately need to exercise. A healthy alternative is for you to Buy Home Gym Equipments that you can store in your own home or even at your office. Here are some tips when buying home gym equipment.

    First, know your fitness goals and keep these goals in mind when you are buying home gym equipment. One of the biggest mistakes that people make is buying the biggest or trendiest fitness machine simply because they like how it looks. More often than not, these end up gathering dust in your storage rooms or garage because you never end up using them. It is important to take note of what particular type of physical activity you want or enjoy doing before you buy your exercise machine. If you are looking to loose a few pounds and you enjoy walking or hiking, a treadmill is the best option for you. If you are looking to tone your lower body while burning calories a stationary bike is your obvious choice. Similarly, Special Equipments for Core Strength Exercises, Strength Training Weight Vests, Core Strength & Abdominal Trainers, Home Cardio Training, Strength Training Power Cages, Strength Training Racks & More.

    Second, set aside a budget before Buying Home Gym Equipments. Quality exercise machines do not come cheap. They are constantly exposed to wear and tear when they are used properly. So, pick machines that are built to last and have passed quality certifications to get the most out of your money. If you are operating on a tight budget, think about investing in several weights, We can Provide you High Quality Home Gym Equipment at Very Low Prices for your Complete Uses: Core Strength Exercises, Strength Training Weight Vests, Core Strength & Abdominal Trainers, Home Cardio Training, Strength Training Power Cages, Strength Training Racks & More.

    Its the Right Time to Buy Home Gym Equipment for you at Very Low Prices.

    ReplyDelete
  26. 국내 최고 스포츠 토토, 바카라, 우리카지노, 바이너리 옵션 등 검증완료된 메이져 온라인게임 사이트 추천해 드립니다. 공식인증업체, 먹튀 검증 완료된 오라인 사이트만 한 곳에 모아 추천해 드립니다 - 카지노 사이트 - 바카라 사이트 - 안전 놀이터 - 사설 토토 - 카지노 솔루션.

    온라인 카지노, 바카라, 스포츠 토토, 바이너리 옵션 등 온라인 게임의 최신 정보를 제공해 드립니다.

    탑 카지노 게임즈에서는 이용자 분들의 안전한 이용을 약속드리며 100% 신뢰할 수 있고 엄선된 바카라, 스포츠 토토, 온라인 카지노, 바이너리 옵션 게임 사이트 만을 추천해 드립니다.

    ReplyDelete
  27. Thanks for posting the best information and the blog is very helpful.python course in Bangalore

    ReplyDelete
  28. 먹튀검증 Because the admin of this web site is working, no doubt very quickly it will be renowned,
    due to its feature contents.


    ReplyDelete
  29. 먹튀검증 I think this is among the most important information for me.
    And i am glad reading your article. But should remark on few general things,
    The website style is ideal, the articles is really nice : D.
    Good job, cheers

    ReplyDelete
  30. 온라인카지노 Howdy! I could have sworn I've been to this site before but after checking through some of the post I realized
    it's new to me. Anyhow, I'm definitely delighted I found it and
    I'll be book-marking and checking back frequently!


    ReplyDelete
  31. 온라인카지노 Hi there very cool web site!! Guy .. Excellent ..
    Superb .. I will bookmark your web site and take
    the feeds additionally? I'm glad to search out a
    lot of useful info hede wituin thhe submit, we'd like work oout more strategies


    ReplyDelete
  32. Oferecemos os melhores serviços para seguidores, curtidas, comentários e visualizações no Instagram. Os serviços são entregues de forma rápida, segura e por um preço honesto e barato.
    Todos os serviços são iniciados de forma automática e você pode fazer um teste grátis de seguidores, curtidas e visualizações no Instagram.

    O Go Followers surgiu em 2015 e durante alguns anos foi o melhor e maior site para comprar seguidores, curtidas e comentários no instagram. Com o passar dos anos evoluímos e em 2021 lançamos um novo site simples e intuitivo para aumentar a experiência dos mais de 150.000 clientes atendidos.

    Nossa ferramenta envia seguidores brasileiros reais e ativos em sua grande maioria para qualquer perfil do Instagram. Escolha o melhor pacote para comprar seguidores instagram de alta qualidade. Você poderá escolher alguns pacotes como seguidores com curtidas, seguidores masculinos, seguidores femininos e seguidores automáticos.
    Todos os pedidos são feitos de forma automática e iniciam em poucos minutos após a compra, tá esperando o que?

    ReplyDelete
  33. Thanks for taking the time to discuss this, I feel strongly about it and love learning more on this topic. If possible, as you gain expertise, would you mind updating your blog with extra information? It is extremely helpful for me. lip gloss box sets | lip gloss box sets

    ReplyDelete
  34. I’m really enjoying the design and layout of your website.
    It’s a very easy on the eyes which makes it much more pleasant for me to come
    here and visit more often. Did you hire out a developer to create your theme?
    Exceptional work! 온라인카지노

    ReplyDelete
  35. 카지노사이트 I was studying some of your content on this internet site and I think this internet site is very informative!
    Keep on posting.

    ReplyDelete
  36. I am actually happy to read this website posts which carries plenty of helpful data, thanks for providing these kinds
    of data. 토토


    ReplyDelete
  37. Your good writing makes many people feel happy.
    With a lot of writing, you are moved a lot. I also have a lot of interactions with you
    I want to feel alive and be happy, saying that I will be a good friend with you, and I will be very happy too. I hope there are always good things and happy days. I always cheered for you.
    Thank you 스포츠토토

    ReplyDelete
  38. I am glad to discover this post extremely valuable for me, as it contains part of data. I generally like to peruse the
    quality substance and this thing I found in you post. A debt of gratitude is in order for sharing. 토토사이트


    ReplyDelete
  39. I'm going to show you the methods and rules of the toto game, such as sports toto, football, baseball. Visit my blog right now to check it out.
    here is my web site : ブルレクジェク


    ReplyDelete
  40. Custom Printing Services co.uk. it is the best printing services and the follow my website new technology printing in the best works try the link spoken. thanks popcorn boxes bulk | popcorn boxes bulk

    ReplyDelete