Sunday, December 29, 2013

Consumer Need and Data Driving the Next Generation of Technology

A quick look into the evolution of Information Technology tells us how the needs from the focused groups have shaped up the innovation in Information Technology. 1950s saw scientists in the labs coming up with first generation of technology for the scientific research, it was then lapped up by the Defense departments in 1960s for the military needs. In 1970s, very large Corporations started making use of Information Technology, primarily for batch processing, file transfer in electronic format etc. 1980s saw large organizations automating internal processes and running the batch processes on mainframes. With 1990s, medium business started adopting Technology for automation and B2B communication. Starting 1990 computing started entering into the houses. By 1995 with Windows, Microsoft changed the face of personal computing for ever. This fueled the growth of consumer internet during late 1990s and early 2000. By now technology cycles became shorter and things started evolving much faster.

Fast forward to early 2000s, we see the next generation of Technology innovation was primarily driven by end consumer needs , the volume of traffic and the humongous data that was getting generated due to sudden rise in websites and web users.  By early 2010s Smart phone usage started taking over other forms of personal computing. During early 2000s, Google made many path breaking technological innovation (Big Table, distributed file system etc) while trying to handle millions of web searchers. Those technologies in turn set the direction of next generation Technology. Launch of Apple App Store in 2008 can be considered as another landmark event in this phase.

So, it was no more the demands from Business or Defense Depts that were pushing the limits. It was all about consumers and Facebook, Apple, Google and Amazon became the flag bearers.

Amazon, Facebook, Google, Apple -are all very different companies. If one has to take a guess on their primary identity, it would probably look like - Amazon is the seller, Apple is the product guy, Facebook is the marketer and Google helps you sell. While Amazon and Apple sell their produce directly to the end consumers, Facebook and Google are the market exchange where the sellers & marketers get to meet potential buyers and they primarily sell Ad space.

Each of these companies are pushing the limits of traditional Engineering & Technology backed by insight into end consumer needs or by creating end consumer need. They are redefining computing as it existed.

Although they did not start as Technology companies, but today they control the new frontiers of the Technology and pose serious threat to traditional technology vendors. Apple and Google control Mobile App technologies and the ecosystems. Apple along with Google defines the new age computing devices. Facebook leads the API economy and literally owns a parallel Web (45 million Facebook Fan pages vs 350 million websites). Amazon and Google control the new age computing environments - AWS and GCE. Google controls the World wide web.

What distinguishes them from the traditional Technology players like Microsoft, IBM, Oracle, SAP is their control over the data or the content and their strategy to go deeper into that. This leaves the traditional technology vendors with no option but to do a catch up job in the new world order. Of the traditional Tech vendors, Microsoft probably stands the best chance to remain a dominant player in the new world with their Azure and Windows Mobile offerings. Lets take a quick look into their evolution.

Amazon started as an online book seller, diversified to add many new product categories to eventually become the largest eCommerce company (and poses serious threat to traditional brick and mortar merchants like Walmart). Amazon used their early mover's advantage and insight into end consumer need to start doing the backward integration. They identified newer areas that can scale, they moved to e-books then to Audio books. Like Steve Jobs, Jeff Bezos also knew this very well, it is the superior consumer experience that matters the most and the most important piece here is the reader. This enables Amazon to control end-to-end quality and creates an entry barrier that most of the competitors will find difficult to break through. In the whole process, very diligently Amazon built unique capabilities and leverage that to enter into a new market.  Amazon used Technology to drive the cost down and build an impenetrable entry barrier as a Cost Leader. With their strategy of cost leadership and the burgeoning sales, Amazon had their own need to build a Technology infrastructure that can scale, scale and scale.They invested heavily there and made use of the unused capacity to create a new market called "Cloud Computing". Since then Cloud based computing has empowered scores of developers and fueled the next generation of Technology innovation.

If superior product design and engineering was one major factor behind Apple's success the other one was their ability to serve the content for those devices through iTunes. iTunes also created an entry barrier that most of the other players like Sony etc could not break through just by superior product design and Engineering. Apple duplicated the success theory of iPod + iTunes again with iPhone + App Store. iTunes enabled Apple to control the digital entertainment content and App Store helped them control a new form of content "Apps". Apple's Technology Platform for the mobile Apps still leads the pack. Another example of how content and consumer IT shaping the future of technology.

Google's case is more obvious given the fact that they are necessarily the entry point to world wide web and controls what content a web surfer gets to see. With Google+, Gmail, photo sharing and other consumer specific Apps (Google Now, Google Awesome), they are gradually zeroing into every consumer and pretty much knows what a consumer is doing or looking for. Through Android OS and Play Store, Google remained a dominant player in the new world order and created amazing assets for future revenue. With their control over the traditional web coupled with the growth of Android OS and g+, they probably control most of the digital content in the world. Google followed Amazon's footstep to rent out their computing infrastructure through Google Compute Engine. These technology offerings have used Google's need for scale and consumer centric data as their test bed and probably miles ahead in terms of Engineering excellence and tooling as compared to traditional tech vendors. Google's Technology offering spans from back-end computing (what users can not see) to front-end OS, Apps (what users experience) and most probably the strongest of all.

Facebook's case is probably not as obvious as others. But remember, Facebook is probably the strongest when it comes to consumer data and API access to that data. Their control over data is not just limited to consumer data, Facebook hosts more than 45 million Fan pages, these pages are far more dynamic in terms content & updates when compared to regular web. User engagement of each post talks about the quality of the content. Overall there are 200 million+ Fan pages, Groups and Communities in Facebook. API access to those data would be absolutely critical for the future of the Web and Apps. I will not be surprised (in fact I will expect) if Facebook start charging the developers for the API access.

Of rest, Microsoft probably stands the best chance but their access to content and user data is shallow. A lot depends on how their Mobile platform does. It also depends on how Microsoft can make use of other consumer centric offerings like Skype, Outlook email, Bing Search, Xbox, Zune and the most important their App Store. Microsoft is still the single biggest entity when it comes to desktop computing and owns MS Office. A clear focus and tight integration among them would be critical for MS's success and to remain relevant.

It's evident that Google, Amazon, Apple, Facebook are going to control the world of technology for foreseeable future and Microsoft stands a good chance as well.

So, what's next? What else is seen on the horizon? Well, the answer is - no one knows. However, if I have to take a shot, here is what I think is going happen in next few years.

A critical & big changing pattern is really around smarter tools - development as well as runtime environments and how accessible they have become to the developers. AWS, Azure, Rackspace, and GCE have democratized the Software development process. Now an individual developer has access to all the resources at pittance. To add to it, the developer happens to be the consumer as well and can relate to consumers need personally. Powered by all the resources of the world, developers are going ahead and developing what they love.

This is very different when compared to traditional enterprise world software development where often developers are clueless about what they are developing adding to the high failure rate of the software projects and leading to higher average cost. That doesn't mean projects don't fail outside the boundary of companies. In fact, there the failure rate is higher, albeit with few distinct differences - there is no one to assume the risks for the developers and when they fail, they fail fast and that helps keep the sunk cost to a lower amount. Second, large part of the software development challenges are not only about the business problem, in fact, it is more about the underlying infrastructure, the architecture & design required to achieve HA, DR, Scalability, Serviceability etc. Now, all these new age cloud based computing environments provide all these things and more at the same place. This has reduced the need for big project management, corporate discipline and the necessity for system, structure and process, usually required for big scale software development.

This empowerment of common developers resulted into development of thousands of consumer focused Apps in last few years. However, earning money in that space has been far more challenging when compared to Enterprise world. So, in coming days we can expect to see some of those developers turning their head towards the Enterprise Software Market and start disrupting the enterprises, industry after industry and some would start challenging the control of large Tech Companies over Enterprise software world with their expertise and all the cool technologies that they have access to.

So, what we would eventually see is very large but limited number Tech Companies providing the computing platform, environment, technology, APIs and large number of independent smart developers to small but agile organizations making use of them but mostly outside the boundaries of intermediate large tech companies.
Follow me @satya_paul


Monday, August 5, 2013

10 Tools that every web developer must know while developing a website


In this post I will cover the tools every web developer or webmaster must know for running a website. These tools & resources will help you to ensure that the users have a superior experience when they visit your website. Superior user experience is not just about the functionality or the cool User Interface, it's more than that. Functionality and UI will help get users for first time but it's the page performance that ensures that users actually spend time or come back again. Next, in case users like your website, how easily and effectively users can share it in their network. In today's world just sharing a url is not sufficient unless it shows the necessary preview info that most of the Social Networks (facebook, linkedin, Google+, twitter et. al) support.

Here are few points that I plan to cover in the rest of the post - 
  • Website Performance & Availability
  • Web Analytics
  • Social Media Sharing
  • Search Engine Optimization
  • Protection of Intellectual Property
1. Website Performance & Availability

There are two aspects of the performance measurement of a website 


a. Number of concurrent users it can support while ensuring an accepted level of response time 
b. Page latency or response time of the page.

Let's spend a minute trying to understand what each of these really means. Support of concurrent users can be interpreted as Throughput without deteriorating the SLA (i.e. Response Time). However, the definition Response Time itself is little fuzzy. First, it's not an absolute number, rather it's a range. Lower range is the time taken to receive the bare-bone HTML content. Once HTML content is loaded in browser's memory, it starts downloading other resources like js, css, images etc. "Page Load time" is the overall time taken to download all the resources that a particular page requires "in order to render it". You can understand it quickly by looking into "Chrome > Right click > Inspect Element > Click 'Network' in the lower panel > Load the website > Observe the results", you can do the same in Firefox as well. Typically the bare bone html is generated in your web server while other static resources are served by the CDN networks.

So, it's important that your webserver is able to scale and serve the base page really fast (~300 ms) even under high load and you make use of a good CDN service that can serve the static content blazing fast.

Coming to the tools, JMeter is a good choice for doing the load testing on the website. In fact, JMeter needs to be used even during the development phase to fix low hanging issues. However, for professional grade results, I will recommend a cloud based solution - LoadImpact. Here is a report generated on www.fanffair.com using loadimpact.com - goo.gl/MQK0R . Note both the tools measure the time taken to load the bare-bone HTML content. Two other tools that you may like to evaluate are - Anturis & Pingdom . They do a good job of monitoring the site performance (along with monitoring availability).


JMeter - http://jmeter.apache.org/
LoadImpact - http://loadimpact.com 

Anturis - https://anturis.com/
Pingdom - https://www.pingdom.com 

Now, how to measure the Page Loading Time or the latency to serve the page i.e. what your users finally experience. Google Page Speed is probably a good starting point to measure, use the service to get an overall score and a report with all the recommendations. Two other tools that would come handy are Chrome Developer Tool and Yahoo! Slow. It will take some time to act on all the recommendations but you will be able to bring noticeable difference in overall page load time by acting on all the recommendations.


Google Page Speed - https://developers.google.com/speed/pagespeed 
Yahoo! Slow - http://developer.yahoo.com/yslow/
Yahoo! Slow Performance rule - http://developer.yahoo.com/performance/rules.html
Chrome Developer Tool - https://developers.google.com/chrome-developer-tool

Next is Availability, it is far more important a factor than Website performance. However, it is probably more basic a problem to solve and most of the Data Centers and Cloud based computing environments (e.g. AWS, GAE, Azure etc) provide a very high rate of availability and today this has become more or less a non-issue. But nonetheless, it's always a good idea that you monitor your website for availability. Better to get an alert from a automated monitor than a user. Both Anturis & Pingdom do a pretty good job on that front as well.

Anturis - https://anturis.com/
Pingdom - https://www.pingdom.com 

Personally I would prefer Anturis over Pingdom.

2. Web Analytics


This is the most important measurement for your website. Based of number of users, usage, location of users, devices used and other demographic info, you may need to alter your website design or the content. Google Analytics is probably the most popular and widely used Web Analytics tool, it's available for free. Few other alternative would include Adobe Marketing Cloud (Omniture) (paid), Core Metrics from IBM(paid).


Google Analytics - http://www.google.com/analytics/
Adobe (Omniture) - http://www.adobe.com/solutions/digital-marketing.html
IBM (Coremetrics) -  http://www-01.ibm.com/software/marketing-solutions/coremetrics/


If your website is registered as a Facebook App then you can benefit a great deal from Facebook App Insight.

Facebook Insights - https://www.facebook.com/insights/
Facebook Insights Documentation -  https://developers.facebook.com/docs/insights/

3. Web Masters tool 

Web Masters tool allows the website owners to submit their websites to the Search engines and configure few other associated parameters. Obviously, of all Google's Web Masters Tool is the most important. However, there are more e.g. Bing's Webmaster's Toolbox

Google - https://www.google.com/webmasters/tools
Microsoft - http://www.bing.com/toolbox/webmaster 

In addition, you may like to submit the website url to various directory listing services 

Open Directory Project - http://www.dmoz.org/

Yahoo! Directory Listings - http://dir.yahoo.com/ 

They still have importance from Search Engine point of view.

4. Chrome Dev tools 

I am calling it out separately as Chrome Dev Tool deserves this. This is probably the one tool that will give you a holistic picture of your website's performance as seen by end users. It helps you to optimize the html, css, javascript, images contents of your website as well as improve the network and rendering performances. It has lot of debugging tools as well. It's a complete solution in itself.

Chrome Dev Tools - https://developers.google.com/chrome-developer-tools

A good alternative would be - https://developer.mozilla.org/en-US/docs/Tools 

5. Cross Browser Support

Testing your website for cross browser support is a long and often a tiring process. In fact, testing your website on multiple browsers across different versions and OS combinations is not going to be a simple task. And with every significant UI change, you need to repeat these testing. It may slow down your development speed as well as your ability to release the new versions significantly. But not any more once you know about these tools - http://browsershots.org 

Browser Shots - http://browsershots.org 
BrowserStack http://www.browserstack.com/


6. Obfuscate/Minify your Javascript

Last thing you want is your competition to copy your javascript code. So, go for obfuscation. Apart from protecting your intellectual property, it also helps you to "minify" (or reduce the size) the javascript content. Here are few -

Closure Complier: http://code.google.com/closure/compiler/
YUI Compressor: http://developer.yahoo.com/yui/compressor/
UglifyJS - http://marijnhaverbeke.nl/uglifyjs 
JSMin by Douglas Crockford - http://www.crockford.com/javascript/jsmin.html

Personally, I liked UglifyJS. 

7. Validate your Open Graph Parameters 

In this age, its important that you understand and ensure that the users & potential users see the right information when a url from your website is shared in popular social media. Managing this preview information is probably as important as your content if not more. This factor may become a make or break point for a website.
Open Graph parameters (Reference: http://ogp.me/) are the meta information about the page (similar to html header info) that goes into every page. Popular social media like Facebook et al, make use of this info to render the preview window when a url is shared.

You can test the OG meta info for a url here - http://developers.facebook.com/tools/debug 

8. SEO 

Search Engine Optimization helps websites to be more searchable and for achieving this, SEO involves implementing certain best practices e.g. setting the meta info of page. SEO is a subject and an industry by itself, here I am not going to cover what is SEO and how it is done. I will leave with certain resources that you may find useful. Webmaster's tools from Google and Bing are the starting points and here are some resources for serious business.

Google - Search Engine Optimization Starter Guide
SEOBook - http://www.seobook.com/ - tons of information available here.
Moz (SEOMoz) - http://moz.com/beginners-guide-to-seo 
Woorank - http://www.woorank.com - Get your website evaluated here and then work on the recommendations.
SEO WORKERS - http://www.seoworkers.com

You can use following two websites for quickly checking backlinks for your website
Majestic SEO - http://www.majesticseo.com
Open Site Explorer - http://www.opensiteexplorer.org


9. Social Media Integration 

Social Media Integration means instrumenting your website so that social sharing becomes easier for the users. A part of it is covered by implementing proper Open Graph parameters for the pages (urls). However, another significant part involves configuring your website for various popular Social Plugins from Facebook, Twitter, Linkedin, Google+, Pinterest etc. Depending on the nature of content and your target audience, you may choose not to integrate all of them but this requires careful thinking before you finalize which plugins should be integrated.

Facebook - http://developers.facebook.com/docs/plugins/ 
Linkedin - http://developer.linkedin.com/plugins
Twitter - https://twitter.com/about/resources/buttons
Google+ - https://developers.google.com/+/web/share/
Pinterest - http://business.pinterest.com/widget-builder/#do_pin_it_button


10. Alexa Account

Finally, as the adage goes, "if you cannot measure it you cannot improve it". Google Analytics gives you a very good insight into how your website is doing with or without all the changes. But that is in isolation, you need to measure how you are doing as compared to your peers and competition. And that's why configuring your web site for Alexa  is very important. 
Alexa measures the overall ranking of the websites based on website traffic. Once your website is configured, you can view the global and country specific ranking for your website. This will enable you to measure how your website is doing in terms of number of visitors and few other parameters and will highlight any change in ranking.

Alexa - http://www.alexa.com/


Although I had started with the plan to cover exactly top 10 tools from different categories but eventually I realized many of the tools are not mutually exclusive and complimentary in nature. So, decided to retain all the important tools and resources. Hope that makes sense.



Thursday, May 2, 2013

Planning a new App development? How about the Technology Stack?

I guess anyone doing any sort of Software Development work will agree that Technology Stack plays the most critical part during the development life cycle as well when the product is in use which often requires modifications. A bad choice could add millions of dollars of additional expenses during the development as well as higher maintenance expenses when it's in use resulting into higher TCO (Total Cost of Ownership) for the customers. But unfortunately there is no simple answer to this question when you start building a new app. In this post and the follow up posts I will try to cover  some of those Tech Stacks and pros & cons of them. 

Let's take a look at what are the different pieces that influence the choice of Technology Stack:

1. Content Rendering or UI Layer
2. Content Generation Layer
3. Core Business Logic
4. Back-end batch processing
5. Data Storage

Next, some of the priorities that influence the choice of Technology Stack.

1. Is UI layer a native App or a regular Web App?
2. High availability 
3. Scalability
4. Disaster recovery
5. Data Consistency - transactional consistency or eventual consistency? 
6. Serviceability 
7. Extensibility 

And last but not the least low cost of development and maintenance. 

In this first case, we will consider developing a basic Web Application meant for consumers using Java/J2EE Technology.

Content Rendering or UI Layer: Use of HTML5, CSS3 and Javascript/jQuery is probably given. There is not much scope for debate. However, an important consideration here would be to think if HTML5 based rendering is going to be the final one. This is an important consideration as increasingly consumers are moving towards Mobile devices for Internet access and regular HTML5 doesn't really give the best experience there. So, this consideration would influence what Technology to be used for the Content Generation LayerFor UI layer, time spent to identify the right jQuery plugins is worth every second of it.

Content Generation Layer: Here our pick is RESTful APIs. That allows you to keep this layer agnostic of the final UI Layer. The RESTful APIs could feed to your HTML5+CSS3+jQuery based UI or to your iOS or Android based native App equally comfortably. Considering this App is getting developed using Java/J2EE, we will recommend Jersey for RESTful API implementation.

Core Business Logic: Here our pick is one of the most commonly used patterns by the Java Developers - use POJO (Plain Old Java Objects) coupled with DAO layer (Data Access Layer APIs) and for persistence use JPA (Java Persistence API) coupled with some ORM tool. Hibernate is pretty universal choice for an ORM tool.

Back-end batch processing: Even simplest of the Software Applications will have some sort of back-end batch processing where the priority is necessarily the "throughtput" i.e. how many records/data/file processed per unit time. For simple workflow and business process, we will recommend having one single step of processing and run it in a scheduled manner. One option is to use Java Concurrency APIs but it requires deeper skill sets. So, to keep the things simple, we will recommend using either Unix cron job or Windows Scheduler. However, if you don't want to depend on the OS, then go for Quartz

Data Storage: Any standard RDBMS should fit the bill here. For the simple app in our discussion, we will recommend MySQL. However, when it comes to creation of the Database tables etc, we will recommend that be done through regular Java objects/beans and let your JPA/ORM layer take care of managing the DB tables.

Till this point, we really did not talk about which Web Server or App Server to be used or where to host the Application and they play a very critical role in the choice of overall Technology Stack.

So, as mentioned earlier two of our major considerations are - 
1. Keep things simple that requires as less skill sets as possible 
2. Keep the cost low - both development and running cost.

So, keeping those goals in perspective, our recommendation would be to go for Google App Engine. Google App Engine makes the development really simple, you can work out your Eclipse environment - use the environment for both development and deployment (release management) activities. If you are not already familiar with GAE, GAE is necessarily a PaaS offering from Google that gives you  Hosting Environment (HW), Software Dev Env (GAE Eclipse Plugin) as well as Software infrastructure layer like Web Server, App Server, Scheduler, Database and multiple value added Services for running the Application in production. And imagine you get to run your app on the same infrastructure as Google Search.

Finally, for the code repository and version control, go for GitHub.

So, here is how it finally looks like -

1. Eclipse 4.2 (Juno) for Development/Deployment (free)
2. GitHub as code repository and version control. (free for public repository otherwise $7/month)
3. Google App Engine as Hosting Env as well as Platform.(free for limited quota otherwise $10/month is sufficient)
4. HTML5 + CSS3 + jQuery for UI
5. RESTful APIs using Jersey
6. JPA for persistence layer
7. Google Big Table for storage (comes with GAE)
8. Make use of other GAE services like cron job etc.