CloudCrypto – Crypto Currency Trend Analysis Using AWS Serverless Architecture

CloudCrypto – Crypto Currency Trend Analysis Using AWS Serverless Architecture.

Indian economic policies have opened the door for financial institutions to support accounts for user-based investments in Crypto Currencies. We want to provide a user-friendly platform to the Indian retail users to view the trends and real-time statistics related to cryptocurrencies and enabling them to make an informed investment decision.

This solution is deployed using AWS services. The product has the following features:

Trends & real-time statistics
User-friendly platform
Analysis of Cryptocurrencies
Data for informed investment decisions
User Sentiments Data

Baseline

CryptoCurrency dashboard platform is deployed using Fargate and data is being handled using CodeCommit, CodeBuild and Lambda at major. CryptoCurrency Analyzer.

The Architecture above shows the deployment and service connectivity within AWS and to services outside AWS.

WordPress is hosted on AWS Fargate and is handling compute and Load Balancing internally and Elastic Load Balancer is mapped to a domain for public access.

Fargate is distributed across 2 availability group and hosted in a particular region to provided minimum latency to end-user.

CodeCommit and CodeBuild are used to control deployments and includes data fetch from Public APIs.

AWS CodeBuild is handling the web API calls from public APIs and its data transition to S3 buckets which are processed at QuickSight for generating required data.

Lambda is handling API calls from Twitter and follows the same path for data and generate sentiment related data.

The data flow across the Lambda Development Pipeline is as follows:

Amazon Forecast is used for predictions for coin prices in future based on the historical data price.

Using Fargate to have a scalable application.

Pre-Production deployment was based on EC2 AMIs and was running on a single instance. We faced an issue while moving this application to a serverless architecture. We used a separate Pre-Prod cluster for deployment with lots of iterations to have a final instance working. We faced issues in passing definitions from DockerFile to Container definitions. We were able to solve this by using AWS documentation to some extent. We used a simple methodology to map every DockerFile syntax to Fargate UI and worked through iterations by adding a service at each step like EFS, RDS, Modified Docker image, security certificate, auto-scaling.

Meaningful Data Interpretation

We extracted variables from source APIs and pulled them into QuickSight. However, the whole data was not creating a meaningful analysis and we had to be selective on the variables to be pulled from APIs and have it processed directly in QuickSight to have the relevant data for end-user perspective.

Analysis of Sentiment Data

Twitter is a good source for collecting sentiments for any product. We observed lots of irrelevant data being processed for our sentiment analysis objective. We identified that this would need an ML model to train the data and fetch more realistic sense-ful sentiments. To overcome this, we changed our source to a different Public News API and limited the sentiment data generation out of twitter. Also, the tweet length was a concern at Twitter API and bulk of sentiments were resulting in neutral polarity.

Lambda

For fetching news and tweets, different lambda functions were developed using Python. These lambda functions fetch news, tweets, calculate sentiments and save to S3 buckets. Results from S3 are then rendered in QuickSight to display useful graphs for the user. These lambda functions are maintained and deployed using CodeCommit and CodeBuild. For refreshing data on hourly basis, cron job was using CloudWatch.

QuickSight & Forecast

This provided us with a platform for analysis at AWS, It fetches data from S3 and processes the data to present graphs to be displayed on the website. The forecast was used for creating prediction based charts showing price variations for a currency.

Correlation of Crypto News Sentiments and Price Values

Lambda was again used to add one more layer to amplify on the price trends and observer if the fluctuations were related to the news inflow at any level. The trend was somehow related going up and down but not a linear relationship that can be established. Data without any filter had a Pearson correlation coefficient of 0.148737, however, we improved to 0.329084 using all the filters.