Monday 6 March 2017

How to build an architecture for your cloud data application

Introduction
But how do you actually start with building data applications in the cloud? Which known technologies can you use? What are new technologies? How do all these technologies work together? What will the cost be of such a solution? 
We will try to answer these questions in this series of blog posts. We will show how two people, Dana and Frank (see picture below) will start building the next Unicorn using cloud based data technology. We will first present their product, next introduce Dana and Frank and finally explain how they started building their architecture.
The next blogs in the series can be found here

The Usbourne first book of the Computer, Usbourn Publishing Ltd, 1985


The game changing idea
Together they are building a SaaS company that makes online games with data science. Their first game that they are working on is called What's the title?. They show automatically generated data graphs of classic novels. The user needs to guess the title. A real game changer. Below you can see their first game and you can also check out the link to their website.



Meet the team
For now, the team consists of only two people, but they hope to grow fast in the near future. Therefore they both agreed that they will need to acquire extra skill sets besides their current skills to make the full application. Their focus initially is to get all the components connected to have a global overview and see how the different pieces are working together. Later on they can then elaborate further on the different components and move on to using more advanced technologies.

Dana Simpsons, the Data Scientist

Dana loves everything about mathematics and data and she has a decent background in computer science. Her favourite programming languages are R and Python. Finally she also has some SQL and NoSQL database skills. As part of her tasks she will also watch the cloud costs and look for potential savings. She will also be responsible for customer success and will analyze the usage patterns.

Frank Steve Davidson, the Full Stack Developer

Frank loves everything about programming and has a decent background in mathematics. His favorite programming languages are Java and Ruby. He will be responsible for the architecture design and will make sure that the solution gets deployed in the cloud.

The generic architecture
Dana and Frank did some brainstorming meetings and came up with the following architecture. With this architecture they want to achieve the three goals explained below.




Goal 1

Their game will be hosted in the cloud. This means that both their development and their production servers will live in the cloud together with the data storage. This means that they will need to look at how virtual machines are being set up by their chosen cloud provider and what kind of cloud storage solutions there are available. Because they are hosting a website, they also still will need to investigate this further.

Goal 2
Frank has chosen to work with Ruby as programming language to build out the back-end further. Some responsibilities of the back-end will be the storing of different novels in the cloud in a data lake and transforming these novels into a data format consumable by the web services that Dana is building. He will extract the images that Dana generates and will store them back in the cloud. For now they only will use a simple front-end because the data solution will already provide detailed images. In the future they will look further into more advanced front-end technology.

Goal 3
Dana will need data science related software to be able to make her analysis and generate the graphs. She also will need to have an easy way to access different data sources when she is performing her analysis. She prefers to have access to python and R. Finally she wants to have an easy way to expose her graphs and analysis to Frank. When she is happy with her data science solutions she wants to build an automation layer on top of this.

The architecture in Azure
Dana and Frank compared the different cloud providers and they came to the conclusion that Azure suited their needs best. The architecture in Azure is shown below. In what follows we will explain these components more. In the next blogs you will learn how to setup and use these components.




Goal 1: How to deploy and develop



Frank uses a Linux Virtual Machine in Azure on which he has installed Ruby. He next uses a virtual machine to host the web server Apache2.  To save costs he will put his Ruby development machine on auto-shut down mode. To be able to host this virtual machine as a web server he will need to select a public ip address and allow http access on the server. We go into more detail in these steps in the next blog post of this series.

Goal 2: How to talk to and feed the data storage solution



Frank and Dana use blob storage as their cloud storage solution. You can imagine Blob storage as your own big drive in the cloud. When you are developing you can use several APIs to access blob storage. Frank will use the azure gem for Ruby. Dana will analyze the costs that are involved with using blob storage. We go into more detail in this in the third blog post of this series.


Goal 3: How to talk with and build the Data Science Solution




Dana used Azure Machine Learning Studio to build an experiment that generates the needed graphs. Next she deployed this experiment as a web service. This web service consumes the formatted text of a classic novel that is stored in Blob storage and then produces six different graphs. These graphs are then stored in Blob storage. She will describe her strategy in the fourth blog post. Next Frank will describe in the last blog post how he feeds and consumes the data from this web service.

Conclusion 
In this blog post we have introduced two people, Dana and Frank, on their journey to build their SaaS company in the cloud. We briefly mentioned some of the technologies that they needed. In the next blog posts we will explain these technologies more in depth so that you will be able to use them for your own application.


45 comments:

  1. For the finding of best data solution person go to this web-site activewizards.com/ , from here you can be able to find and compare the huge number of data handling expert who can easily handle or solve your big data handling problems in a short time.

    ReplyDelete
  2. Superb. I really enjoyed very much with this article here. Really it is an amazing article I had ever read. I hope it will help a lot for all.
    Thank you so much for this amazing posts and please keep update like this excellent article.
    thank you for sharing such a great blog with us. expecting for your.

    python Training in Bangalore | Python Training institute in Bangalore

    ReplyDelete
  3. This article is fantastic! Noise in charts and data visualization is such a problem, and this gave me some ideas for how to better approach it, such as greying out the less-important categories in a graph. The slopegraph was still hard for me to understand, ever re-worked.


    Java training in Chennai | Java training institute in Chennai | Java course in Chennai

    ReplyDelete
  4. This article is fantastic! Noise in charts and data visualization is such a problem, and this gave me some ideas for how to better approach it, such as greying out the less-important categories in a graph. The slopegraph was still hard for me to understand, ever re-worked.


    Java training in Chennai | Java training institute in Chennai | Java course in Chennai

    ReplyDelete
  5. I am often to blogging and i really appreciate your content. The article has really peaks my interest. I am going to bookmark your site and keep checking for new information. the impossible quiz unblocked

    ReplyDelete
  6. I'm really impressed about the info you provide in your articles. If you want to invest in suits and be stress-free, then you should shake hands with us. custom window cake boxes

    ReplyDelete
  7. Hello, I want to subscribe for this web site to take most up-to-date updates, thus where can i do it please help out.
    custom lipstick boxes

    ReplyDelete
  8. Nice post, keep up with this interesting work. It really is good to know that this topic is being covered also on this web site so cheers for taking time to discuss this! Healthy lunch Ideas

    ReplyDelete
  9. On this page, you'll see my profile, please read this information. buy medical cannabis online uk

    ReplyDelete
  10. For e.g., the correct inquiry to pose for a car organization could be what is the advertising spend to income proportion in various postal districts for various models. This will lead them to adjust their showcasing dollars stream.machine learning course

    ReplyDelete
  11. Not a single second surrendered relinquishing me in total stun.
    Microsoft project alternative

    ReplyDelete
  12. I would like to thank you for getting my neurons conspicuous with this brilliant article that you have written which contains every potential points which needs to considered on the given topic. Thanks for chipping in such a brilliant writing!
    Custom Packaging Boxes

    ReplyDelete
  13. Crystal online pharmacy is a trusted online drug store with a wide range of products to suit the needs of our clients. Buy Fish oil online Crystal Pharmacy do strive to offer the best service and ship products world wide. All the products listed on our website are Ava in stock. Expect your order to be processed Immediately when you send us your request. We deal with varieties of drugs for our customers satisfaction. We cross barriers with our products and struggle hard to meet human satisfaction. When shopping with us, Be safe and secured and you will realize how swift we are with our services.

    ReplyDelete
  14. Universal Gun sales is a trusted Firearm company base in Los Angeles California USA. It is one of the top Firearms Company in this part of the country. Buy Big sky racks UTV Bar online They do offer the best firearms deal in the USA and to over 25 countries in the world. You can always visit their shop in Los Angeles using the address on their website. Place your order and get served by the best Firearm Company in the planet. Have fun.

    ReplyDelete
  15. 국내 최고 스포츠 토토, 바카라, 우리카지노, 바이너리 옵션 등 검증완료된 메이져 온라인게임 사이트 추천해 드립니다. 공식인증업체, 먹튀 검증 완료된 오라인 사이트만 한 곳에 모아 추천해 드립니다 - 카지노 사이트 - 바카라 사이트 - 안전 놀이터 - 사설 토토 - 카지노 솔루션.

    온라인 카지노, 바카라, 스포츠 토토, 바이너리 옵션 등 온라인 게임의 최신 정보를 제공해 드립니다.

    탑 카지노 게임즈에서는 이용자 분들의 안전한 이용을 약속드리며 100% 신뢰할 수 있고 엄선된 바카라, 스포츠 토토, 온라인 카지노, 바이너리 옵션 게임 사이트 만을 추천해 드립니다.

    ReplyDelete
  16. 먹튀검증 Because the admin of this web site is working, no doubt very quickly it will be renowned,
    due to its feature contents.


    ReplyDelete
  17. 먹튀검증 I think this is among the most important information for me.
    And i am glad reading your article. But should remark on few general things,
    The website style is ideal, the articles is really nice : D.
    Good job, cheers

    ReplyDelete
  18. 온라인카지노 Howdy! I could have sworn I've been to this site before but after checking through some of the post I realized
    it's new to me. Anyhow, I'm definitely delighted I found it and
    I'll be book-marking and checking back frequently!


    ReplyDelete
  19. Oferecemos os melhores serviços para seguidores, curtidas, comentários e visualizações no Instagram. Os serviços são entregues de forma rápida, segura e por um preço honesto e barato.
    Todos os serviços são iniciados de forma automática e você pode fazer um teste grátis de seguidores, curtidas e visualizações no Instagram.

    O Go Followers surgiu em 2015 e durante alguns anos foi o melhor e maior site para comprar seguidores, curtidas e comentários no instagram. Com o passar dos anos evoluímos e em 2021 lançamos um novo site simples e intuitivo para aumentar a experiência dos mais de 150.000 clientes atendidos.

    Nossa ferramenta envia seguidores brasileiros reais e ativos em sua grande maioria para qualquer perfil do Instagram. Escolha o melhor pacote para comprar seguidores instagram de alta qualidade. Você poderá escolher alguns pacotes como seguidores com curtidas, seguidores masculinos, seguidores femininos e seguidores automáticos.
    Todos os pedidos são feitos de forma automática e iniciam em poucos minutos após a compra, tá esperando o que?

    ReplyDelete
  20. Thanks for taking the time to discuss this, I feel strongly about it and love learning more on this topic. If possible, as you gain expertise, would you mind updating your blog with extra information? It is extremely helpful for me. lip gloss box sets | lip gloss box sets

    ReplyDelete
  21. I’m really enjoying the design and layout of your website.
    It’s a very easy on the eyes which makes it much more pleasant for me to come
    here and visit more often. Did you hire out a developer to create your theme?
    Exceptional work! 온라인카지노

    ReplyDelete
  22. I am actually happy to read this website posts which carries plenty of helpful data, thanks for providing these kinds
    of data. 토토


    ReplyDelete
  23. Your good writing makes many people feel happy.
    With a lot of writing, you are moved a lot. I also have a lot of interactions with you
    I want to feel alive and be happy, saying that I will be a good friend with you, and I will be very happy too. I hope there are always good things and happy days. I always cheered for you.
    Thank you 스포츠토토

    ReplyDelete
  24. I am glad to discover this post extremely valuable for me, as it contains part of data. I generally like to peruse the
    quality substance and this thing I found in you post. A debt of gratitude is in order for sharing. 토토사이트


    ReplyDelete
  25. I'm going to show you the methods and rules of the toto game, such as sports toto, football, baseball. Visit my blog right now to check it out.
    here is my web site : ブルレクジェク


    ReplyDelete
  26. Custom Printing Services co.uk. it is the best printing services and the follow my website new technology printing in the best works try the link spoken. thanks popcorn boxes bulk | popcorn boxes bulk

    ReplyDelete
  27. One Spin on EVERY Slot Machine in HIGH LIMIT at Coushatta 파워볼중계 파워볼 먹튀 검증

    ReplyDelete
  28. Im Immo Cashflow Booster erfahren Sie, wie Sie durch smarte Immobilienanlagestrategien passiven Cashflow generieren. Sie müssen keine Bankgeschäfte tätigen, Geld leihen oder etwas kaufen.

    Eric erklärte 6 verschiedene Strategien, die verwendet werden können, um durch Leasing und Leasing einen Cashflow zu generieren. Diese Methode ist völlig neu und eigentlich eine interne Strategie.

    Anfangs war ich dieser Strategie etwas skeptisch gegenüber, denn wenn man die vermietete Wohnung wieder untervermieten möchte, wie überzeugt man dann den Vermieter.

    Aber ich habe eine bessere Ausbildung und Eric hat klare Anweisungen, wie man den Vermieter leicht überreden kann. Der Immo Cashflow Booster bietet dir Top aktuelle Strategien, welche auch jahrelang funktionieren. Und in diesem Erfahrungsbericht, werde ich dir meine persönlichen Immo Cashflow Booster Erfahrungen mitteilen.

    ReplyDelete
  29. Just 99 Web Design offers affordable web design packages for every business, from basic one-page websites to full eCommerce website design services. low cost website designing

    ReplyDelete
  30. Ýnternet üzerinden gerçek bir casino hizmetine ulaþabilirsiniz. Gerçek casino salonlarýndan canlý casino hizmeti veren özel siteleri bulunmakta. Canlý casino siteleri gerçek casino salonlarýndan kullanýcýlarýna gerçek paralar üzerinden casino oyunlarý sunmakta canlı casino. Oyuncular internete baðlý olduklarý her yerden casino oyunlarý oynayabilmekte. Lakin bahsettiðimiz ve bahsedeceðimiz hizmetleri almak için doðru casino siteleri seçmeniz gerekmekte. Doðru siteleri yani güvenilir casino siteleri tercih ederek hem gerçek bir hizmet alýn hem de kazançlarýnýzý sorunsuz bir þekilde çekin.
    Sektörün en gözde casino þirketleri sayfamýzda paylaþýlmakta. Casino siteleri giriþ butonlarý üzerinden sitelere ulaþabilirsiniz. Sizlere sitelerin hizmetleri hakkýnda bilgiler vereceðiz. Sitelerin sunduklarý hizmetleri, bonuslarý ve finansal araçlarý hakkýnda bilgileri sizlere sunacaðýnýz. En iyi canlı casino sayfamýzdan ulaþabilirsiniz.
    Canlý casino
    Paylaþtýðýmýz siteler oyuncularýna gerçek casino salonlarýndan hizmet sunmakta. Oyuncular BlackJack, rulet, poker gibi casino masa oyunlarýný 4K görüntü kalitesi üzerinden gerçek casino salonlarýndan oynamakta. Ayrýca masa oyunlarý yanýnda özel oyunlarda bulunmakta. Canlý olarak sunulan oyunlar 4K görüntü kalitesi üzerinden oyunculara aktarýlmakta. Oyunlarýn sunumlarý canlý kurpiyerlerin sunumlarýyla yapýlmakta. Canlý casino oyunlarý kurpiyerlerin sunumlarýyla oynanmakta. Ayrýca masalarda bulunan herkes sizler gibi gerçek oyuncudur. Asla sitelerde bot hesaplarý bulunmaz. Herkes gerçek oyuncudur. Ayrýca odalarda sohbet bölümleri bulunmakta. Canlý casino
    oyunlarý oynanýrken ayný zamanda diðer oyuncular ile sohbet edebilirsiniz.
    Sitelerde poker, blackjack, rulet gibi onlarca farklý casino masa oyunlarý. Özel olarak hazýrlanan crayz time gibi canlý casino oyunlarý. Aviator, zeplin gibi popüler oyunlar ve binlerce farklý slot oyunlarý bulunmakta. Oyuncular istedikleri oyunlarý istedikleri bet üzerinden oynayabilmekte. Casino siteleri gerçek kumarhanelerdeki oyunlarý ve hatta daha fazlasýný kullanýcýlarýna sunmakta canlı casino.
    Lakin unutmayýn bahsettiðimiz hizmetleri en iyi casino siteleri için geçerli. Her canlý casino sitesi ayný hizmeti vermemekte. Bundan dolayý önünüze gelen her casino sitesi giriþ yapmayýn. Güvenilir casino siteleri giriþ yaparak gerçek bir hizmete ulaþýn.
    Oyuncular casino oyunlarýndan saðladýklarý kazançlarýný sorun yaþamadan çekebilmekte. Ayrýca istedikleri finansal yöntemi kullanarak seçebilmekte. Siteler oyuncular onlarca farklý finansal araç sunmakta. Kullanýcýlar istedikleri yöntemleri kullanarak para yatýrma ve çekme iþlemlerini kolay bir þekilde canlı casino yapabilmekte.
    Casino siteleri onlarca farklý bonus da vermekte. Yeni kayýt olan kullanýcýlar deneme bonusu ve free spin alarak bedava casino oyunlarý oynama fýrsatý yakalamakta. Ýlk yatýrýmlarýna özel casino hoþgeldin bonusu. Her yatýrýmlarýna özel casino yatýrým bonusu. Þansýz günlerinde ise casino kayýp bonusu alabilmekte. Ayrýca bahsettiðimiz bonuslar dýþýnda farklý casino bonuslarý da bulunmakta canlı casino.
    Deneme bonusu veren casino siteleri sayesinde kayýt olarak yatýrým yapamadan bedava casino oyunlarý oynayabilirsiniz. Siteler yeni üyelerine özel olarak bedava bonus vermekte.
    Bonusu alan kullanýcýlar bedava bir þekilde casino oyunlarý oynayarak ayrýca kazançlarýnýzý çekebilirsiniz.
    Güvenilir canlý casino siteleri sayesinde avantajlý bir hizmete ulaþýn.

    ReplyDelete
  31. It has an incredibly well-organized cell tailored browser, the layout and design of the cell version are exceptionally good. The show is crystal clear and it really works} intuitively on completely different display sizes to deliver great gameplay. Microgaming is the unique casino provider, founded in 1994. The best of their slots are Immortal Romance and Mega Moolah nicely as|in addition to} Thunderstruck 2 . Classic Slot/Slot Machine - These are the unique classic slots with 카지노 사이트 3 reels and a single payline. You can win should you get 3 matching symbols on the display's center.

    ReplyDelete