Blueprint of Live Redis Event

Agenda

The Hashedin and Redis meetup brings together the cloud computing community to connect, collaborate and learn


  • How to scale-up and boost the performance using Redis
  • How to architect internet scaled high availability applications in the cloud using Redis
  • Potential to network with other organizations to learn how they are leveraging Redis Labs capabilities


All sessions will be delivered by subject matter experts from Redis Labs and HashedIn, Partners or expert customers can share their real world experiences and lessons learned.


Timeline

  • 09:00 - 09:30: Registration and light snacks
  • 09:30 - 09:40: Welcome Note
  • 09:40 - 09:50: HashedIn Keynotes
  • 09:50 - 10:50: Introduction to Redis Labs
  • 10:50 - 11:15: Scale-up and Performance Enhancement using Redis
  • 11:15 - 11:30: Break - Tea / Coffee, Networking
  • 11:30 - 11:50: MeetNotes: Redis for Real-time Collaboration
  • 11:50 - 12:30: Journey of Redis at Myntra
  • 12:30 - 12:45 Redis for Cyber Security Enterprises Products


HashedIn Keynote

Welcome Note

Harshit Singhal, CBO, HashedIn Technologies


Welcome note of Redis-Hashedin meetup


A lot of effort went into convincing Redis that India is a prominent market for their product.

Participants must showcase their skills and convince Redis that the market is ripe for their product.

The simile that is used to describe Redis is the swiss army knife

A swiss army knife is a survival tool kit, similarly Redis also has numerous applications.

How one uses Redis is important and it is important to examine whether Redis is used effectively.


Keynote

Himanshu Varshney, CEO, HashedIn Technologies


Hashedin is wired for speed


  • A brief introduction into how HashedIn came into play. The first thing that is written in C code is  “# include” and “hash out” is often is used to describe “solving a problem: and that is where the name HashedIn comes from.
  • HashedIn is “Wired for Speed” and we translate ideas into products in weeks, not years.
  • Founded in 2010 with the mission to help businesses in Product development..
  • In 7+ years, we have built, transformed and launched 90+ products for customers in US and APAC.


HashedIn is involved in:

  • Digital Transformation
  • IoT Enabler
  • Artificial Intelligence  
  • Machine Learning
  • Smart Cloud
  • Voice Systems


What we do:

  • Digital Strategy Consulting
  • DevOps Automation
  • Smart Platforms
  • Customer Experience
  • Data Engineering
  • IoT Driven Systems


HashedIn and Redis - Connecting the Dots

  • 2009 - Sripathi (HashedIn, CTO) started contributing to Redis
  • 2013 - HashedIn created Redis Center of Excellence (CoE)
  • Redis has been an integral part of all Digital Solutions
  • Tech Consulting - Helping businesses in using Redis the right way
  • 2017 HashedIn became Redis Labs Partner
  • 25+ Experienced Redis Engineers
  • Only Emerging Tech Partner for Redis Labs in APAC market


Introduction to Redis Labs

Amir Levy, Director of Sales, Redis Labs


Introduction to Redis Labs


  • Redis is Open source. The leader in-memory database platform, supporting any high performance operational, analytics or hybrid use case.
  • Redis Labs is the open source home and commercial provider of Redis Enterprise (Redise) technology, platform, products & services.
  • Redis tops database popularity rankings
  • Redis 4.0 allows you to extend Redis functionality and most commonly used Modules are Neural Redis, RedisML and RediSearch.
  • RediSearch is 5X faster than Elasticsearch because it uses Redis in the underlying architecture.
  • GST calculations used Redis cache for intensive calculations.
  • Redis is state of the art database, opensource and proprietary tech built in multi tenancy, offering effortless scaling.
  • ACID compliant, high availability.


Tal Shkolnik, Solution Architect Expert, Redis Labs


Demo of Redis use cases


Tal Shkolnik, gives a demo of Redis enterprise version. 


Scale-up and Performance Enhancement using Redis

Sripathi Krishnan, CTO, HashedIn Technologies


Hashedin performance enhancement using Redis


Redis is mostly used as cache. Several use cases exist where features can be built on top of Redis. It can solve scalability and performance issues.


Use Case #1: Real Time Analytics-Dashboard

  • Track KPIs in real time
  • Cohort analysis
  • Leader boards


Use Case #2: Session / API Tokens

  • Logout if idle v/s logout after fixed time
  • Redis Log out based on inactivity
  • Force logout if necessary
  • Reduce load on your database


Use Case #3: Task Queues for Background Jobs

  • Distribute load to background jobs


Use Case #4: De-duplication Within Time Window

  • Duplication occurring within short time window because of retries.


Use Case #5: Rate Limiting

  • Redis can track number of requests per minute/per hour, this can be used to prevent overloading resources.


Use Case #6: IP Blacklisting

  • If you detect rogue ip addresses, you can blacklist them in Redis. Nginx can then access Redis and ban the IP addresses.
  • Achieved by use of Lua.


Use Case #7: Feature Gates

  • Store settings on Redis which can be accessed instantaneously.
  • This avoids the need to restart server.


Use Case #8: Real-time Notifications

  • Redis pub/sub let's you eliminate polling in JS.
  • Using libraries like socket.io you can build real-time collaborative apps


Use Case #9: Autocomplete API

  • Redis Sorted Set is the perfect tool to build autocomplete APIs.


Q&A

Q. ActiveMQ for user notifications vs Redis

MQ based products RabbitMQ and ActiveMQ are more robust but Redis has most of the features of AMQP and can be used for numerous scenarios.


Q. Does expiration work on nested keys?

Nested keys cannot be expired, only top level keys have expiration.


MeetNotes: Redis for Real-time Collaboration

Anshuman Singh, Founder, MeetNotes


Meetnotes uses Redis for real time collaboration


Introduction to MeetNotes and How Redis is used in MeetNotes.

37 billion dollars are wasted on meetings in the US alone and 15% of organization time is wasted in meetings. This is the problem that MeetNotes is trying to solve.


Technical Summary

  • MeetNotes is a real time collaborative document.
  • It parses information as you type or speak.
  • It creates action items with due dates, tags and enables relentless follow-ups.


Redis @ MeetNotes.co

Redis is used

  • As a cache
  • As a Task Queue
  • For Real time collaboration


Operational Transforms

  • Operational Transforms are used to provide consistent outcomes and to merge changes
  • It Preserves user intent and converge to common state
  • Redis Pub-sub helps in inter-user communication
  • Relay changes
  • Do it reliably & fast


Q&A

Q. How are concurrency issues handled?

Operational Transform layers take care of that.


Q. How scalable is the infra?

A single aws m2.xlarge machine is capable of handling upto half a million socket connections but that does not take care of data operations(operational transform).


Q. Long Polling vs Sockets

Sockets help in bidirectional flow. Single socket takes care of both pub and sub. Further sockets deliver results instantaneously.


Journey of Redis at Myntra

Sanjay Yadav, Principal Architect, Myntra


Redis at Myntra


Myntra uses redis for more than 200+ services and there are over 400 instances of redis (Cluster / Sentinel / Twemproxy).


How it Began in 2011

  • PHP Stack


Impact

  • Checkout experience was shaky.
  • Customer location serviceability - Unpredictable
  • High latency across multiple pages.
  • Customer will not stick if low latency is not guaranteed.


What Was Missing

  • A distributed fast access store.


Enter Redis

  • Stores everything in memory.
  • Key value store with expiry.
  • Data structure store.
  • Popularly knows as "Memcached on steroids".


Initial Setup

  • Master slave
  • Key value store
  • JSON Values
  • 10,000-20,000 concurrent users.
  • Up to 100K rpm with ~25 ms response time


Auto Failover

  • In-house client built with auto failover mechanism.
  • Switches slave to master when master is detected as down upon 3 successive pings.
  • The issues of horizontal scaling were not yet addressed.
  • All request were going to a single box.


In 2013

  • Scale is increasing .
  • Customer base is increasing.
  • In pursuit of growth events are being organized - load increases further.
  • Frequent outage on many servers including redis boxes.


Twemproxy

  • Multiple Nodes
  • Sharding logic
  • One proxy and 4 master


Pros

  • Blazing fast.
  • Able to support high concurrency - Twempoxy claimed billion request/sec on single instance.
  • Horizontally scale redis.
  • 1 Million rpm achieved without much efforts.


Cons

  • It became the single point of failure.
  • When throughput increases, bandwidth on twemproxy boxes gets consumed.
  • Mget was missing from twemproxy.
  • In order to remove SPOF, we created multiple proxies but that leads to inconsistencies.


2015 Redis cluster

  • Seamless multi-master setup.
  • Seamless fail-over from master to slave on failures.
  • We need not worry about shard management. no inconsistencies, intelligent clients.
  • Truly horizontal scaling.
  • Served 30M rpm at 95th percentile 25ms in a recent event Single instance gave us 50K rpm

Session - Biggest Use Case

  • Every request that reaches us has to be authenticated.
  • Cumulative of all request we serve.
  • Single key, value store.
  • 600K concurrent users come to Myntra at event peaks.


Sentinel for Master Slave Setups

  • When you don't need too many write servers and need multiple read servers you need horizontal scaling only for reads.
  • Cases where management is very important, need all keys, to be available in a single box.
  • Setup one master and multiple slaves as per throughput requirements.
  • Reads were sent to slaves.


LGP

  • Myntra social platform - Look good platform
  • Users, objects and actions - first class citizens
  • Objects can be published  by any external apps to the platform
  • Users perform actions on other user/objects.
  • Feed is created for users based on interest.
  • 10 Million Weekly Active Users 500K objects added everyday 100K actions per minute..


Redis Has Always Helped

  • Powerful data structures.
  • Group multiple keys under one key.
  • Store unique sorted values.
  • Simple increment/decrement operations.
  • Feed generation using Celery - RabbitMQ + Redis.


Big Events

  • Style chat with celebs like Hritik/ Deepika/ Virat Kohli.
  • Launch of new brands with brand engaging users


Redis for Cyber Security Enterprises Products

Anindya Sharma, Co-Founder and CTO, Cyware Labs


Cyber security enterprise products


How Redis helped solve critical issues and improved performance for Cyware and its customers.


Performance

  • Main caching choice for all layers.
  • ElastiCache -> Local Cache -> Global Cache
  • Persistence (across deployments and restarts).
  • Smart invalidation's.
  • RPUSHX Update only existing keys.
  • Overall RDS cost went down by 30%.
  • Reduced dashboard load time.


Cyware Feeds Broadcasting Pipeline.

  • Crawler to clients pipeline.
  • Cushioning based on component capacity.
  • Cy-bank channels.
  • Client apps can subscribe to the channels.
  • Auto publish based on client’s wish.
  • Better capacity handling at clients.


Near Realtime Security News Crawling

  • Real time Security
  • New Crawling
  • Crawling more than 10,000 security news websites and blogs every 30 mins
  • Repetitive crawling of news that has been published a few minutes back
  • Real time updating